CN113687934A - Data acquisition method, task scheduling method, device, apparatus, medium, and program - Google Patents
Data acquisition method, task scheduling method, device, apparatus, medium, and program Download PDFInfo
- Publication number
- CN113687934A CN113687934A CN202111056690.9A CN202111056690A CN113687934A CN 113687934 A CN113687934 A CN 113687934A CN 202111056690 A CN202111056690 A CN 202111056690A CN 113687934 A CN113687934 A CN 113687934A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- processing
- acquisition
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000012545 processing Methods 0.000 claims abstract description 46
- 238000013500 data storage Methods 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 10
- 238000007405 data analysis Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 3
- 238000013480 data collection Methods 0.000 abstract description 30
- 238000010586 diagram Methods 0.000 description 17
- 238000004891 communication Methods 0.000 description 9
- 238000011161 development Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000004140 cleaning Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012358 sourcing Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
- G06F9/4887—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues involving deadlines, e.g. rate based, periodic
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the disclosure provides a data acquisition method, a task scheduling method, a device, equipment, a medium and a program. The data acquisition method comprises the following steps: receiving a first message from a self-service task platform; collecting first target data based on the first message; sending the first target data to a data storage system after first processing; and sending the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different. Therefore, after data are collected based on the first message, the first message is processed and fed back to the self-service task platform, so that the self-service task platform can generate a subsequent collection instruction based on the information, the data collection task can be arranged based on service dimensionality, limited data collection can be carried out according to service requirements, the reasonability and continuity of collected data are guaranteed, and the obtained data are complete and reliable.
Description
Technical Field
The present disclosure relates to the field of computers, and more particularly, to a data acquisition method, a task scheduling method, an apparatus, a device, a medium, and a program.
Background
With the rapid development of information technology, virtual service lines have been advanced into various industries, and the data volume of each service line has also been rapidly increased. At present, data of each business data source party are comparatively cracked, and the processing and analysis are difficult to be completely carried out. Some data synchronization technologies require a large amount of network and storage resources due to the high synchronization pressure, and are high in cost.
Disclosure of Invention
In order to solve the problems in the related art, embodiments of the present disclosure provide a data acquisition method, a task scheduling method, a device, an apparatus, a device, a medium, and a program.
One aspect of the present disclosure provides a data acquisition method, including: receiving a first message from a self-service task platform; collecting first target data based on the first message; sending the first target data to a data storage system after first processing; and sending the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different.
Optionally, the collecting first target data based on the first message comprises: acquiring first target data based on acquisition parameters under the condition that the acquisition parameters are included in the first message; or, collecting first target data based on default parameters under the condition that the first message is a null message.
Optionally, the first processing includes some or all of data cleansing, data analysis, and data summarization, and the second processing includes data screening.
Another aspect of the present disclosure provides a task scheduling method, including: sending a first message, wherein the first message is used for triggering a data acquisition system to execute a first acquisition task; receiving a feedback message from the data acquisition system; generating a second message based on the feedback message; and sending the second message, wherein the second message is used for triggering the data acquisition system to execute a second acquisition task.
Optionally, the sending the first message includes: and generating and sending a first message in response to the satisfaction of a trigger condition, wherein the trigger condition comprises at least one of the number of tasks being executed being less than a first threshold value, the resource occupancy rate being lower than a second threshold value and the arrival at a preset time, and the first message is a message carrying acquisition parameters or an empty message.
Another aspect of the present disclosure provides a data acquisition apparatus including a first receiving module, an acquisition module, a first sending module, and a second sending module. Wherein the first receiving module is configured to receive a first message from the self-service task platform; an acquisition module configured to acquire first target data based on the first message; the first sending module is configured to send the first target data to a data storage system after first processing; and the second sending module is configured to send the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different.
Another aspect of the present disclosure provides a task scheduling apparatus, including a third sending module, a second receiving module, a generating module, and a fourth sending module. The third sending module is configured to send a first message, and the first message is used for triggering the data acquisition system to execute a first acquisition task; a second receiving module configured to receive a feedback message from the data acquisition system; a generating module configured to generate a second message based on the feedback message; a fourth sending module configured to send the second message, where the second message is used to trigger the data acquisition system to execute a second acquisition task.
Another aspect of the disclosure provides an electronic device comprising at least one processor and at least one memory storing one or more computer-readable instructions, wherein the one or more computer-readable instructions, when executed by the at least one processor, cause the processor to perform the method as described above.
Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.
Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.
According to the technical scheme provided by the embodiment of the disclosure, after data are collected based on the first message, the first message is processed and fed back to the self-service task platform, so that the self-service task platform can generate a subsequent collection instruction based on the information, the data collection task can be arranged based on service dimensionality, limited data collection can be carried out according to service requirements, the reasonability and continuity of collected data are ensured, and the obtained data are complete and reliable.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments when taken in conjunction with the accompanying drawings. In the drawings:
fig. 1 schematically shows a schematic diagram of an application scenario of a data acquisition method and a task scheduling method according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a data acquisition method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a task scheduling method according to an embodiment of the present disclosure;
FIG. 4 schematically illustrates a timing diagram of a data collection method and a task scheduling method according to an embodiment of the disclosure;
FIG. 5 schematically illustrates a block diagram of a data acquisition device according to an embodiment of the present disclosure;
FIG. 6 schematically shows a block diagram of a task scheduling apparatus according to an embodiment of the present disclosure;
FIG. 7 schematically shows a block diagram of an electronic device according to an embodiment of the present disclosure; and
FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing the methods and apparatus of embodiments of the present disclosure.
Detailed Description
Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts not relevant to the description of the exemplary embodiments are omitted in the drawings.
In the present disclosure, it is to be understood that terms such as "including" or "having," etc., are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or combinations thereof, and are not intended to preclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or combinations thereof may be present or added.
It should be further noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
It should be noted that the acquisition or presentation of data in this disclosure is authorized, confirmed, or actively selected by the user.
An embodiment of the present disclosure provides a data acquisition method, including: receiving a first message from a self-service task platform; collecting first target data based on the first message; sending the first target data to a data storage system after first processing; and sending the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different.
Fig. 1 schematically shows a schematic diagram of an application scenario of a data acquisition method and a task scheduling method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the scenario may be a self-service task platform 110, a data collection system 120, a plurality of business data sourcing parties 130 (e.g., business data sourcing parties or other third party data centers, etc.), and a data storage system 140.
According to the embodiment of the disclosure, the self-service task platform 110 may be deployed in a server or a terminal device, and is configured to initiate a data collection command, for example, send a message to the data collection system 120, so that the data collection system 120 performs a data collection task according to the received message. The data collection system 120 may collect data for business data as well as behavioral data for multiple systems. The service data source 130 is usually a plurality of, and may include, for example, one or more self-owned service systems, and may also include one or more third-party data centers. The data storage system 140 may be implemented as a data server deployed with a database or data warehouse for storing the collected various types of business data.
According to the embodiment of the disclosure, data interaction between the self-service task platform 110, the data acquisition system 120, the plurality of business data sourcing parties 130 and the data storage system 140 can be performed by using a message queue or HTTP communication mode. Any data collection task based on message queues or HTTP communication can access the data collection architecture, but requires task registration at the self-service task platform 110.
The data collection method described in the embodiment of the present disclosure may be executed by the data collection system 120, and the task scheduling method described in the embodiment of the present disclosure may be executed by the self-service task platform 110. Various methods provided by embodiments of the present disclosure are described below with reference to fig. 2-4.
Fig. 2 schematically shows a flow chart of a data acquisition method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S210 to S240.
In operation S210, a first message is received from a self-service task platform.
Acquiring first target data based on the first message in operation S220;
in operation S230, the first target data is sent to a data storage system after being subjected to the first processing;
in operation S240, the first target data is sent to the self-service task platform after being subjected to a second process, where the first process and the second process are the same or different.
According to the technical scheme provided by the embodiment of the disclosure, after data are collected based on the first message, the first message is processed and fed back to the self-service task platform, so that the self-service task platform can generate a subsequent collection instruction based on the information, the data collection task can be arranged based on service dimensionality, limited data collection can be carried out according to service requirements, the reasonability and continuity of collected data are ensured, and the obtained data are complete and reliable.
Fig. 3 schematically shows a flow chart of a task scheduling method according to an embodiment of the present disclosure.
As shown in fig. 3, the method includes operations S310 to S340.
In operation S310, a first message is sent, where the first message is used to trigger a data acquisition system to execute a first acquisition task;
receiving a feedback message from the data acquisition system in operation S320;
generating a second message based on the feedback message in operation S330;
in operation S340, the second message is sent, where the second message is used to trigger the data acquisition system to execute a second acquisition task.
According to the technical scheme provided by the embodiment of the disclosure, the subsequent acquisition instruction can be generated based on the feedback message of the first message, and the data acquisition task can be arranged based on the service dimension, so that limited data acquisition can be performed according to the service requirement, the reasonability and continuity of the acquired data are ensured, and the acquired data are complete and reliable.
The method of the disclosed embodiment is described below with reference to the timing diagram illustrated in fig. 4.
FIG. 4 schematically shows a timing diagram of a data collection method and a task scheduling method according to an embodiment of the disclosure.
As shown in fig. 4, the method includes operations S401 to S410.
In operation S401, the self-service task platform 110 sends a first message to the data collection system 120. According to embodiments of the present disclosure, scheduling may be initiated through a message queue or HTTP communication.
According to embodiments of the present disclosure, the first message may be sent at various occasions, for example, the sending the first message includes generating and sending the first message in response to a trigger condition being met. That is, the first message may be automatically generated and transmitted if the trigger condition is satisfied. The triggering condition may include at least one of the number of tasks being executed being less than a first threshold, the resource occupancy being below a second threshold, and the predetermined time being reached, for example. When the number of tasks is less than the first threshold or the resource occupancy rate is lower than the second threshold, the system is in a relatively idle state, and the method of the embodiment of the disclosure may perform the data acquisition task by using the idle period of the system. And the task platform schedules the collection tasks on time according to the requirements. The predetermined time of day may be a certain time of night, for example 2:00 am each day, which is typically in an idle period of the system. In addition to specifying nighttime periods, it may be ensured by way of a predetermined time of day that data is collected at least once per certain period, for example, in the case of a 2:00 trigger each morning, at least once per day.
In other embodiments of the present disclosure, the triggering may also be manually by an administrator. The self-service task platform 110 generates and sends a first message in response to the administrator's input action.
According to the embodiment of the present disclosure, the first message may be a message carrying an acquisition parameter or an empty message. The self-service task platform 110 allows the administrator to manually enter collection parameters, such as which data needs to be collected, through what form to collect, in what form to store, etc., and also allows the administrator not to fill in collection parameters, but to send an empty message as the first message. Similarly, in case the first message is triggered automatically, a null message may also be sent without filling in any acquisition parameters.
In operation S402, the data collection system 120 processes the first message and performs a first collection task to initiate data collection to the service data source 130. Similarly, data collection may be initiated through a message queue or HTTP communication.
According to the embodiment of the present disclosure, the data acquisition system 120 stores default parameters, and acquires the first target data based on the acquisition parameters when the acquisition parameters are included in the first message. Or, collecting first target data based on default parameters under the condition that the first message is a null message.
In operation S403, the data collection system 120 obtains the collected business data, i.e., the first target data.
In operation S404, the data acquisition system 120 sends the first target data to the data storage system 140 for storage after the first processing.
According to an embodiment of the present disclosure, the first processing may include data cleaning, data analysis, and data summarization, and the collected data is stored in the data storage system 140 after the data cleaning, the data analysis, and the data summarization, so as to be used by other systems. In other embodiments of the present disclosure, the first process may include only data cleansing, or only data cleansing and data analysis, with the remaining processes being handed to data storage system 140 for completion. Even further, the data acquisition system 120 may send the data directly to the data storage system 140 without any processing so that the data storage system 140 stores the complete raw data.
In operation S405, the data acquisition system 120 performs a second processing on the first target data to obtain a feedback message, and sends the feedback message to the self-service task platform 110. The second process may include a data filtering process operation, in which data required by the self-service task platform 110 is filtered according to a predetermined rule for feedback, so that the self-service task platform 110 initiates scheduling of next data collection according to the feedback message.
Here, the order of operations S404 and S405 may be exchanged and may even be performed simultaneously.
According to the embodiment of the disclosure, after a round of data acquisition tasks is completed, other acquisition tasks which depend on the data returned by the task can be scheduled again in a message queue or HTTP communication mode, so that task arrangement is realized, and the sequence and the reasonability of service logic are ensured. I.e., the following operations S406 to S410 are continuously performed.
In operation S406, the self-service task platform 110 generates a second message based on the feedback message and sends the second message to the data collection system 120.
Operations S407-S410 are similar to S402-S405 described above.
In operation S407, the data collection system 120 processes the second message and performs a second collection task, initiating data collection to the service data source 130.
In operation S408, the data collection system 120 obtains the collected business data, i.e., the second target data.
In operation S409, the data acquisition system 120 sends the first target data to the data storage system 140 for storage after the first processing.
In operation S410, the data acquisition system 120 performs a second processing on the first target data to obtain a feedback message, and sends the feedback message to the self-service task platform 110. The order of operations S409 and S410 may be switched or performed simultaneously.
According to the technical scheme of the embodiment of the disclosure, two or more data acquisition processes can be continuously executed, and the data acquisition tasks can be arranged based on the service dimensionality, so that limited data acquisition can be carried out according to the service requirements, the reasonability and continuity of the acquired data are ensured, and the acquired data are complete and reliable. Meanwhile, the method can effectively reduce the workload of developers and operation and maintenance personnel, reduce the development and operation and maintenance cost, improve the overall development efficiency of a team under the condition of multi-person collaborative development, and effectively maintain and manage the projects of the server side.
Fig. 5 schematically illustrates a block diagram of a data acquisition device 500 according to an embodiment of the present disclosure. The apparatus 500 may be implemented as part or all of an electronic device through software, hardware, or a combination of both.
As shown in fig. 5, the data acquisition apparatus 500 includes a first receiving module 510, an acquisition module 520, a first transmitting module 530, and a second transmitting module 540. The data acquisition device 500 may perform the method described above with respect to fig. 2.
A first receiving module 510 configured to receive a first message from a self-service task platform.
An acquisition module 520 configured to acquire first target data based on the first message.
A first sending module 530 configured to send the first target data to a data storage system after the first processing.
A second sending module 540, configured to send the first target data to the self-service task platform after a second process, where the first process and the second process are the same or different.
According to the technical scheme provided by the embodiment of the disclosure, after data are collected based on the first message, the first message is processed and fed back to the self-service task platform, so that the self-service task platform can generate a subsequent collection instruction based on the information, the data collection task can be arranged based on service dimensionality, limited data collection can be carried out according to service requirements, the reasonability and continuity of collected data are ensured, and the obtained data are complete and reliable.
Fig. 6 schematically shows a block diagram of a task scheduling device 600 according to an embodiment of the present disclosure. The apparatus 600 may be implemented as part or all of an electronic device through software, hardware, or a combination of both.
As shown in fig. 6, the task scheduler 600 includes a third sending module 610, a second receiving module 620, a generating module 630, and a fourth sending module 640. The data acquisition device 600 may perform the method described above with respect to fig. 3.
A third sending module 610 configured to send a first message for triggering the data acquisition system to perform the first acquisition task.
A second receiving module 620 configured to receive a feedback message from the data acquisition system.
A generating module 630 configured to generate a second message based on the feedback message.
A fourth sending module 640 configured to send the second message, where the second message is used to trigger the data acquisition system to perform a second acquisition task.
According to the technical scheme provided by the embodiment of the disclosure, the subsequent acquisition instruction can be generated based on the feedback message of the first message, and the data acquisition task can be arranged based on the service dimension, so that limited data acquisition can be performed according to the service requirement, the reasonability and continuity of the acquired data are ensured, and the acquired data are complete and reliable.
The present disclosure also discloses an electronic device, and fig. 7 shows a block diagram of an electronic device according to an embodiment of the present disclosure.
As shown in fig. 7, the electronic device 700 includes a memory 701 and a processor 702, where the memory 701 is used for storing a program that supports the electronic device to execute the data acquisition method or the code generation method in any of the above embodiments, and the processor 702 is configured to execute the program stored in the memory 701.
The memory 701 is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 702 to implement the steps of:
receiving a first message from a self-service task platform;
collecting first target data based on the first message;
sending the first target data to a data storage system after first processing; and
and sending the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different.
According to an embodiment of the present disclosure, the collecting first target data based on the first message includes:
acquiring first target data based on acquisition parameters under the condition that the acquisition parameters are included in the first message; or
And acquiring first target data based on default parameters under the condition that the first message is a null message.
According to the embodiment of the disclosure, the first processing comprises part or all of data cleaning, data analysis and data summarization, and the second processing comprises data screening.
Alternatively, the one or more computer instructions are executed by the processor 702 to implement the steps of:
sending a first message, wherein the first message is used for triggering a data acquisition system to execute a first acquisition task;
receiving a feedback message from the data acquisition system;
generating a second message based on the feedback message; and
and sending the second message, wherein the second message is used for triggering the data acquisition system to execute a second acquisition task.
According to an embodiment of the present disclosure, the sending the first message includes: in response to a trigger condition being met, generating and transmitting a first message, wherein the trigger condition includes at least one of a number of tasks being executed being less than a first threshold, a resource occupancy being below a second threshold, and a predetermined time being reached.
According to the embodiment of the disclosure, the first message is a message carrying acquisition parameters or an empty message.
FIG. 8 schematically illustrates a block diagram of a computer system 800 suitable for implementing the methods and apparatus of embodiments of the present disclosure.
As shown in fig. 8, the computer system 800 includes a processing unit 801 which can execute various processes in the above-described embodiments according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The processing unit 801, the ROM802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary. The processing unit 801 may be implemented as a CPU, a GPU, a TPU, an FPGA, an NPU, or other processing units.
In particular, the above described methods may be implemented as computer software programs according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a medium readable thereby, the computer program comprising program code for performing the above-described method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present disclosure may be implemented by software or by programmable hardware. The units or modules described may also be provided in a processor, and the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.
As another aspect, the present disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or the computer system in the above embodiments; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the present disclosure.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is possible without departing from the inventive concept. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Claims (10)
1. A method of data acquisition, comprising:
receiving a first message from a self-service task platform;
collecting first target data based on the first message;
sending the first target data to a data storage system after first processing; and
and sending the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different.
2. The method of claim 1, wherein said collecting first target data based on the first message comprises:
acquiring first target data based on acquisition parameters under the condition that the acquisition parameters are included in the first message; or
And acquiring first target data based on default parameters under the condition that the first message is a null message.
3. The method of claim 1 or 2, wherein the first process comprises some or all of data cleansing, data analysis, and data summarization, and the second process comprises data screening.
4. A task scheduling method comprises the following steps:
sending a first message, wherein the first message is used for triggering a data acquisition system to execute a first acquisition task;
receiving a feedback message from the data acquisition system;
generating a second message based on the feedback message; and
and sending the second message, wherein the second message is used for triggering the data acquisition system to execute a second acquisition task.
5. The method of claim 4, wherein the sending the first message comprises:
and generating and sending a first message in response to the satisfaction of a trigger condition, wherein the trigger condition comprises at least one of the number of tasks being executed being less than a first threshold value, the resource occupancy rate being lower than a second threshold value and the arrival at a preset time, and the first message is a message carrying acquisition parameters or an empty message.
6. A data acquisition device comprising:
a first receiving module configured to receive a first message from a self-service task platform;
an acquisition module configured to acquire first target data based on the first message;
the first sending module is configured to send the first target data to a data storage system after first processing; and
and the second sending module is configured to send the first target data to the self-service task platform after second processing, wherein the first processing and the second processing are the same or different.
7. A task scheduling apparatus comprising:
a third sending module configured to send a first message, the first message being used to trigger the data acquisition system to execute a first acquisition task;
a second receiving module configured to receive a feedback message from the data acquisition system;
a generating module configured to generate a second message based on the feedback message; and
a fourth sending module configured to send the second message, where the second message is used to trigger the data acquisition system to execute a second acquisition task.
8. An electronic device, comprising:
one or more processors;
a memory for storing one or more computer programs,
wherein the one or more computer programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-5.
9. A computer readable medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 5.
10. A computer program which, when executed by a processor, causes the processor to carry out the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111056690.9A CN113687934A (en) | 2021-09-09 | 2021-09-09 | Data acquisition method, task scheduling method, device, apparatus, medium, and program |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111056690.9A CN113687934A (en) | 2021-09-09 | 2021-09-09 | Data acquisition method, task scheduling method, device, apparatus, medium, and program |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113687934A true CN113687934A (en) | 2021-11-23 |
Family
ID=78586189
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111056690.9A Pending CN113687934A (en) | 2021-09-09 | 2021-09-09 | Data acquisition method, task scheduling method, device, apparatus, medium, and program |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113687934A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202302A (en) * | 2016-07-01 | 2016-12-07 | 龙官波 | Collecting method, Apparatus and system |
| CN106549830A (en) * | 2016-11-09 | 2017-03-29 | 上海最会保网络科技有限公司 | A kind of Mobile solution data acquisition unit and method |
| CN108897607A (en) * | 2018-06-05 | 2018-11-27 | 中国平安人寿保险股份有限公司 | Message treatment method, device, electronic equipment and storage medium |
| CN109951326A (en) * | 2019-02-28 | 2019-06-28 | 北京互金新融科技有限公司 | Data processing method, device, storage medium and processor |
| CN113064740A (en) * | 2021-04-02 | 2021-07-02 | 北京京东拓先科技有限公司 | Message processing method and device |
-
2021
- 2021-09-09 CN CN202111056690.9A patent/CN113687934A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202302A (en) * | 2016-07-01 | 2016-12-07 | 龙官波 | Collecting method, Apparatus and system |
| CN106549830A (en) * | 2016-11-09 | 2017-03-29 | 上海最会保网络科技有限公司 | A kind of Mobile solution data acquisition unit and method |
| CN108897607A (en) * | 2018-06-05 | 2018-11-27 | 中国平安人寿保险股份有限公司 | Message treatment method, device, electronic equipment and storage medium |
| CN109951326A (en) * | 2019-02-28 | 2019-06-28 | 北京互金新融科技有限公司 | Data processing method, device, storage medium and processor |
| CN113064740A (en) * | 2021-04-02 | 2021-07-02 | 北京京东拓先科技有限公司 | Message processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109343941B (en) | Task processing method and device, electronic equipment and computer readable storage medium | |
| CN108733461B (en) | Distributed task scheduling method and device | |
| CN111526049B (en) | Operation and maintenance system, operation and maintenance method, electronic device and storage medium | |
| CN113672934A (en) | Security vulnerability scanning system and method, terminal and storage medium | |
| CN111309395A (en) | Object keep-alive method and device, electronic equipment and computer readable storage medium | |
| CN111507257A (en) | Picture processing method, apparatus, system, medium, and program | |
| CN111258741A (en) | Warehouse task execution method, distributed server cluster and computer equipment | |
| CN111210340A (en) | Automatic task processing method and device, server and storage medium | |
| CN114168297A (en) | Method, device, equipment and medium for scheduling collection tasks | |
| CN112783629A (en) | Service calling method and device, electronic equipment and storage medium | |
| US20200293370A1 (en) | Usecase specification and runtime execution to serve on-demand queries and dynamically scale resources | |
| CN113032125A (en) | Job scheduling method, device, computer system and computer-readable storage medium | |
| CN115361382B (en) | Data processing method, device, equipment and storage medium based on data group | |
| CN113687934A (en) | Data acquisition method, task scheduling method, device, apparatus, medium, and program | |
| CN110009273B (en) | Information processing method and device, storage medium, and electronic device | |
| US20230061780A1 (en) | Automated end-to-end testing platform with dynamic container configuration | |
| CN115378937B (en) | Distributed concurrency method, device, equipment and readable storage medium for tasks | |
| CN112825525A (en) | Method and apparatus for processing transactions | |
| CN110008034A (en) | Task automated execution method, apparatus, electronic equipment and storage medium | |
| CN110545296A (en) | Log data acquisition method, device and equipment | |
| CN117112121A (en) | Distributed task processing system, method, apparatus and computer program product | |
| CN114996081A (en) | Batch job progress monitoring method and device, electronic equipment and storage medium | |
| CN111611123B (en) | Data processing method, data processing system and equipment | |
| CN113268365A (en) | Method, device, equipment and storage medium for realizing delay message in distributed system | |
| CN108073450B (en) | Task scheduling method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20211123 |
|
| WD01 | Invention patent application deemed withdrawn after publication |