WO2020232870A1 - 一种基于Actor模型的数据采集方法、装置及存储介质 - Google Patents

一种基于Actor模型的数据采集方法、装置及存储介质 Download PDF

Info

Publication number
WO2020232870A1
WO2020232870A1 PCT/CN2019/102705 CN2019102705W WO2020232870A1 WO 2020232870 A1 WO2020232870 A1 WO 2020232870A1 CN 2019102705 W CN2019102705 W CN 2019102705W WO 2020232870 A1 WO2020232870 A1 WO 2020232870A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
collectors
task
triggered
data collector
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/102705
Other languages
English (en)
French (fr)
Inventor
袁康
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2020232870A1 publication Critical patent/WO2020232870A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
    • G06F9/524—Deadlock detection or avoidance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues

Definitions

  • This application relates to the field of data collection, and in particular to a data collection method, device and storage medium based on an Actor model.
  • the embodiment of the present application provides a data collection method based on an Actor model, which can realize distributed data collection without using a lock.
  • an embodiment of the present application provides a data collection method based on an Actor model, which includes:
  • an Actor model is created for the target task.
  • the Actor model includes a plurality of data collectors, and the data collectors are used to collect data. Communication between collectors through message transmission;
  • the multiple data collectors are divided into multiple stages, and the trigger sequence of the multiple stages is set.
  • Each stage includes at least one data collector, and different data collectors of the same stage are located in different terminals, respectively Used to collect data in different terminals;
  • the trigger sequence of the multiple stages sequentially trigger the data collectors of the multiple stages to collect data
  • an embodiment of the present application provides a data collection device, the data collection device includes a unit for executing the data collection method of the first aspect, the data collection device includes:
  • the detection unit is used to detect whether there is a new target task in the task cluster
  • the creation unit is configured to create an Actor model for the target task when it is detected that there is a newly added target task in the task cluster, the Actor model includes a plurality of data collectors, and the data collector is used to collect data, The multiple data collectors communicate by means of message transfer;
  • the dividing unit is used to divide the multiple data collectors into multiple stages and set the trigger sequence of the multiple stages.
  • Each stage contains at least one data collector, and different data collectors of the same stage are located in different In the terminals, they are used to collect data from different terminals;
  • the execution unit is configured to sequentially trigger the data collectors of the multiple stages to collect data according to the trigger sequence of the multiple stages;
  • the deleting unit is configured to delete the target task in the task cluster after detecting that the data collectors of the multiple stages have all collected data.
  • an embodiment of the present application provides another data collection device, including a processor and a memory, the processor and the memory are connected to each other, wherein the memory is used to store data that supports the data collection device to perform the above data collection method
  • a computer program, the computer program including program instructions, and the processor is configured to call the program instructions to execute:
  • an Actor model is created for the target task.
  • the Actor model includes a plurality of data collectors, and the data collectors are used to collect data. Communication between collectors through message transmission;
  • the multiple data collectors are divided into multiple stages, and the trigger sequence of the multiple stages is set.
  • Each stage includes at least one data collector, and different data collectors of the same stage are located in different terminals, respectively Used to collect data in different terminals;
  • the trigger sequence of the multiple stages sequentially trigger the data collectors of the multiple stages to collect data
  • an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the program instructions are executed by a processor to The data collection method of any one of the foregoing first aspect to the first aspect is implemented.
  • the data collection method provided in this application ensures that data competition and deadlock will not occur during the data collection process.
  • This application can realize complex distributed data collection without the help of locks, and greatly improve the concurrent performance of distributed data collection .
  • FIG. 1 is a schematic flowchart of a data collection method provided by an embodiment of the present application
  • FIG. 2 is a schematic flowchart of a data collection method provided by another embodiment of the present application.
  • FIG. 3 is a schematic block diagram of a data collection device provided by an embodiment of the present application.
  • Fig. 4 is a structural block diagram of a data acquisition device provided by an embodiment of the present application.
  • This application is mainly applied to data collection devices, which can be traditional data collection devices, large storage systems, desktop computers, notebook computers, tablet computers, handheld computers, smart phones, portable digital players, smart watches, and smart bracelets Wait, this application does not restrict this.
  • the terminals described in the embodiments of this application include, but are not limited to, devices with communication functions, smart phones, tablet computers, notebook computers, desktop computers, portable digital players, smart bracelets, smart watches, etc.
  • the terminal sends data to the data collection device, it records and transmits the characteristics of the data according to a preset format, where the characteristics of the data include time, location, type, etc.
  • FIG. 1 is a schematic flowchart of a data collection method provided by an embodiment of the present application.
  • the data collection method may include:
  • the task cluster refers to the collection of tasks to be processed, and different tasks in the task cluster may be proposed by different terminals, and the tasks in the task cluster may be distributed tasks processed by multiple terminals. It may be a stand-alone task processed by only one terminal.
  • the detection process refers to whether the local data acquisition device regularly detects whether there is a new task in the task cluster, or if a new task is added to the task cluster, the local device can receive it. To a message indicating a new task in the task cluster.
  • the above-mentioned target task is not an arbitrary task newly added in the task cluster, but a distributed task newly added in the task cluster. Specifically, whether there is a newly-added target task in the above-mentioned detection task cluster It refers to detecting a newly added task in the task cluster, and if it is determined that the newly added task is a distributed task, it is determined that the newly added target task is detected in the task cluster.
  • the process of detecting that the newly added task in the task cluster is a distributed task refers to obtaining the type of the newly added task in the task cluster, if the type of the newly added task is a distributed task , It means that the type of the newly added task is a distributed task, or the difficulty level of the newly added task is obtained. If it is determined that the difficulty level of the newly added task exceeds the preset level, it means that the newly added task requires Take up more computing resources, and determine that the newly added task is a distributed task, that is, the task whose difficulty level exceeds the preset level is regarded as the target task, and then the distributed task processing method is used to process it .
  • the embodiment of the present application determines that there are newly added target tasks in the task cluster only when it is determined that a distributed task is added to the task cluster, and the newly added distributed task in the task cluster detected above is the target task.
  • an Actor model is created for the target task, and the Actor model includes multiple data collectors.
  • the Actor model is a model for task processing.
  • the Actor model is a conceptual model for processing concurrent computing. It defines a series of general rules for how system components should act and interact.
  • the Actor model can be used to complete any type of task, but the Actor model constructed in the embodiment of the present application is used to perform data collection.
  • the Actor model contains multiple actors. Actors are called data collectors in the embodiments of this application. Each data collector is the most basic unit for data collection in the Actor model. It can receive a message and execute it based on it. data collection.
  • an Actor model containing multiple data collectors based on the execution of the target task refers to the decomposition of the target task, and then based on the collection task of the target task after the decomposition, multiple data collectors are created to execute the corresponding
  • For the collection task it should be noted that when the data collector is collecting data, multiple data collectors may be executed in parallel, but it will not cause resource competition, because the data collectors communicate through message passing. So there will be no resource competition.
  • Actors share resources through message passing.
  • an Actor needs another Actor's resources, it only needs to send the message to another Actor, but does not care when the other Actor will process it. Therefore, the Actor
  • the messages between are transmitted asynchronously, similar to mail communication.
  • each actor has an address, so they can send messages to each other.
  • an actor can only process messages sequentially. In other words, the other three actors have sent a message to an actor. This actor can only process one message at a time.
  • Actors separate the accessible resources of different Actors through message passing, and guarantee the asynchronous processing of messages between Actors, thereby avoiding the use of locks to deal with data race issues.
  • the biggest advantage of the message passing mechanism is that it will not produce data race.
  • the above-mentioned multiple data collectors are divided into multiple stages, and the trigger sequence of the multiple stages is set.
  • Each stage includes at least one data collector, and different data collectors of the same stage are located in different In the terminals, they are used to collect data from different terminals.
  • the embodiment of this application is applied to distributed data collection. It can be seen that the data collectors located on different terminals are collecting data. , It will only occupy the computing resources of the terminal where it is located, so multiple terminals in a terminal cluster jointly process a distributed collection task. Compared with single-machine processing, distributed collection can greatly improve computing power and computing speed. It should be noted that since the Actor in the Actor model is used as the data collector in the embodiment of this application, distributed data collection can be realized without the need of a distributed lock. This is because the Actor only pays attention to the content of the received message.
  • the data collectors of the foregoing multiple stages are sequentially triggered to collect data.
  • the current phase to be triggered is determined according to the trigger sequence of the above multiple phases; if it is determined that the phase to be triggered is at the top of the trigger sequence, or it is determined that all data collectors in the previous phase of the phase to be triggered have completed collection Information, all data collectors in the phase to be triggered are triggered to collect data; when all data collectors in the phase to be triggered are received, all data collectors in the next phase of the phase to be triggered are triggered to collect data , Until the information collected by all data collectors in the multiple stages is received.
  • the above Actor model contains three stages of data collectors, which are sorted in the order of execution.
  • the three stages are the first stage, the second stage and the third stage. So I think about all the data collectors in the first stage. Send data collection instructions to make all data collectors in this stage perform data collection concurrently. After all data collectors in the first stage have collected data, the local device will receive the data sent by each data collector in the first stage. The collected information is completed. When the collected information of all the data collectors in the first phase is received, it means that all the data collectors in the first phase have completed data collection, and the data collectors in the second phase can be triggered to execute the second phase. All data collectors in the second stage perform data collection concurrently.
  • the data collector in the third stage After receiving the information that all the data collectors in the second stage have completed collection, the data collector in the third stage is triggered to perform data collection.
  • the third stage After the information collected by all data collectors is completed, it means that all data collection is completed, that is, the target task is completed.
  • this application does not use an Actor as a scheduling node to schedule data collectors for data collection, this application divides the data collectors into multiple stages, so that data collectors that will cause data competition are divided At different stages, even if this application does not use Actor as a scheduling node, distributed data collection can be achieved, and locks will not be used.
  • the data collected by the data collector of the previous stage can be transmitted to the data collector of the next stage for use by the data collector of the next stage.
  • the data collected by all data collectors in the previous phase of the phase to be triggered are acquired, and all data collectors in the previous phase of the phase to be triggered are collected.
  • the collected data is saved in the local database, and then the data request instruction of the target data collector in the phase to be triggered is received.
  • the data request instruction is used to request the target data in the local database, and then retrieve it from the local database. Target data, and send the target data to the target data collector.
  • data exchange can be carried out between data collectors in different stages, and the data collected in the previous stage can be used by the data collector in the next stage, so that the data collector in the next stage can better complete data collection. Therefore, data exchange between different stages is beneficial to improve the efficiency of data collection.
  • the down data collector when a down data collector is detected, the down data collector is deleted, and the initialization information of the down data collector is obtained, and then the initialization information is used Create a new data collector to replace the down data collector.
  • a new data collector in the case of detecting that the data collector is down, a new data collector is created according to the initialization information of the down data collector, and then the new data collector replaces the aforementioned downtime
  • the new data collector can collect data from the beginning, or according to the information collected before the down data collector and the progress of the collection, the new data collector can continue to perform the collection according to the progress of the collection .
  • the above Actor model also includes at least one monitoring node.
  • the monitoring node is used to initialize the down data collector. Therefore, when the down data collector is detected, the monitoring node is used to monitor the downtime.
  • the data collector of the machine is initialized to control the down data collector to return to normal.
  • the embodiment of the application describes another method for solving the downtime problem of the data collector, that is, if the data collector is down, the monitoring device is used to initialize the down data collector. Specifically, after detecting that the data collector is down, the monitoring node is used to initialize the down data collector according to the initialization information of the data collector to control the down data collector to return to normal. Alternatively, the monitoring node is used to restore the downtime data collector to the normal state at the moment before the downtime.
  • the monitoring node After detecting that the above-mentioned data collector is down, obtain the downtime data collection recorded in the database
  • the state information of the data collector at the moment before the downtime, and the monitoring node is used to restore the data collector according to the state information of the data collector at the moment before the downtime, so that the data collector is restored to the normal state corresponding to the above state information.
  • the tag of the target task is obtained.
  • the tag is used to indicate the importance level of the target task. If the importance level of the target task exceeds the preset level, then when creating the Actor model , Create the above monitoring node, if the importance level of the target task does not exceed the preset level, then do not create the above monitoring node.
  • the target task is deleted from the task cluster, and the next target task in the task cluster is detected and executed.
  • the number of created data collectors is detected, and the creation time of the created data collector is obtained; if the created data If the number of collectors is greater than the preset number, the created data collectors will be deleted in order of creation time from early to late until the number of remaining created data collectors is less than or equal to the above preset Number.
  • the number of data collectors that have been created is detected. If the number of data collectors exceeds the preset number, the data collectors will be deleted until the number of remaining data collectors is less than or equal to the preset number. Among them, delete the data collectors in the order of creation time from early to late. This is because the data collectors created early are less likely to be used again than the data collectors created later.
  • the detailed process of deleting the created data collectors is to detect the number of created data collectors and obtain the creation time of the created data collectors. If the number of created data collectors is greater than the preset Set the number to determine the time node with the earliest creation time in the created data collector, and delete the data collector with the earliest creation time among the created data collectors, and then if you delete the created data collector If the number of created data collectors is still greater than the preset number after the data collector with the latest creation time in the selected data collector, it will trigger the execution of the data collector with the earliest creation time among the created data collectors. The steps of creating the oldest data collector among the created data collectors until the number of created data collectors is less than or equal to the preset number after deleting the data collector with the latest creation time.
  • an Actor model including multiple data collectors is established for the target task, and the multiple data collectors are used to complete the data collection task indicated by the target task Specifically, the above-mentioned multiple data collectors are divided into multiple stages, and the execution order of the multiple stages is set, and then according to the execution order, the data collectors of the multiple stages are triggered in turn to collect data, and finally in the detection After all data collectors in all stages have collected data, the target task is deleted in the task cluster. It can be seen that in the data collection method provided by the embodiment of the present application, since multiple data collectors are all Actors and Actors Communication is carried out by means of information transmission.
  • the advantage of information transmission is that there is no data competition problem. There is no need to synchronize and block waiting when processing tasks, and although the embodiment of this application does not use Actors to schedule the above multiple data collectors Data collection is performed, but the embodiment of the present application divides the above-mentioned multiple data collectors into different stages, thereby setting the execution order of the above-mentioned data collectors, thereby further ensuring that data competition and death will not occur during the data collection process. Therefore, the embodiment of the present application can realize complex distributed data collection without the use of a lock, and greatly improve the concurrent performance of distributed data collection. It can be seen that the embodiment of the present application provides an efficient data collection method.
  • the data collection method may include:
  • the specific detection process refers to whether the local data collection device regularly detects whether there is a new task in the task cluster, or whether there is a new task in the task cluster.
  • Task the local device can receive a message indicating a new task in the task cluster.
  • the task cluster refers to the collection of tasks to be processed, and different tasks in the task cluster may be proposed by different terminals, and the tasks in the task cluster may be distributed tasks processed by multiple terminals. It may be a stand-alone task handled by only one terminal.
  • the newly added task is a target task. Specifically, it is determined whether the newly added task is a distributed task, and if so, it is determined that the newly added target task is detected in the task cluster.
  • the process of detecting that the newly added task in the task cluster is a distributed task refers to obtaining the type of the newly added task in the task cluster, if the type of the newly added task is a distributed task , It means that the type of the newly added task is a distributed task, or the difficulty level of the newly added task is obtained. If it is determined that the difficulty level of the newly added task exceeds the preset level, it means that the newly added task requires Take up more computing resources, and determine that the newly added task is a distributed task, that is, the task whose difficulty level exceeds the preset level is regarded as the target task, and then the distributed task processing method is used to process it .
  • the embodiment of the present application only determines that there is a newly added target task in the task cluster when it is determined that a distributed task is added to the task cluster, and the newly added distributed task in the task cluster detected above is the target task.
  • an Actor model is created for the target task, and the Actor model includes multiple data collectors.
  • the Actor model is a model for task processing.
  • the Actor model is a conceptual model for processing concurrent computing. It defines a series of general rules for how system components should act and interact.
  • the Actor model can be used to complete any type of task, but the Actor model constructed in the embodiment of the present application is used to perform data collection.
  • the Actor model contains multiple actors. Actors are called data collectors in the embodiments of this application. Each data collector is the most basic unit for data collection in the Actor model. It can receive a message and execute it based on it. data collection.
  • an Actor model containing multiple data collectors based on the execution of the target task refers to the decomposition of the target task, and then based on the collection task of the target task after the decomposition, multiple data collectors are created to execute the corresponding
  • For the collection task it should be noted that when the data collector is collecting data, multiple data collectors may be executed in parallel, but it will not cause resource competition, because the data collectors communicate through message passing. So there will be no resource competition.
  • Actors share resources through message passing.
  • an Actor needs another Actor's resources, it only needs to send the message to another Actor, but does not care when the other Actor will process it. Therefore, the Actor
  • the messages between are transmitted asynchronously, similar to mail communication.
  • each actor has an address, so they can send messages to each other.
  • an actor can only process messages sequentially. In other words, the other three actors have sent a message to an actor. This actor can only process one message at a time.
  • Actors separate the accessible resources of different Actors through message passing, and guarantee the asynchronous processing of messages between Actors, thereby avoiding the use of locks to deal with data race issues.
  • the biggest advantage of the message passing mechanism is that it will not produce data race.
  • the above-mentioned multiple data collectors are divided into multiple stages, and the trigger sequence of the multiple stages is set.
  • Each stage includes at least one data collector, and different data collectors of the same stage are located in different In the terminals, they are used to collect data from different terminals.
  • the embodiment of this application is applied to distributed data collection. It can be seen that the data collectors located on different terminals are collecting data. , It will only occupy the computing resources of the terminal where it is located, so multiple terminals in a terminal cluster jointly process a distributed collection task. Compared with single-machine processing, distributed collection can greatly improve computing power and computing speed. It should be noted that since the Actor in the Actor model is used as the data collector in the embodiment of this application, distributed data collection can be realized without the need of a distributed lock. This is because the Actor only pays attention to the content of the received message.
  • the data collectors of the foregoing multiple stages are sequentially triggered to collect data.
  • the current phase to be triggered is determined according to the trigger sequence of the above multiple phases; if it is determined that the phase to be triggered is at the top of the trigger sequence, or it is determined that all data collectors in the previous phase of the phase to be triggered have completed collection Information, all data collectors in the phase to be triggered are triggered to collect data; when all data collectors in the phase to be triggered are received, all data collectors in the next phase of the phase to be triggered are triggered to collect data , Until the information collected by all data collectors in the multiple stages is received.
  • the above Actor model contains three stages of data collectors, which are sorted in the order of execution.
  • the three stages are the first stage, the second stage and the third stage. So I think about all the data collectors in the first stage. Send data collection instructions to make all data collectors in this stage perform data collection concurrently. After all data collectors in the first stage have collected data, the local device will receive the data sent by each data collector in the first stage. The collected information is completed. When the collected information of all the data collectors in the first phase is received, it means that all the data collectors in the first phase have completed data collection, and the data collectors in the second phase can be triggered to execute the second phase. All data collectors in the second stage perform data collection concurrently.
  • the data collector in the third stage After receiving the information that all the data collectors in the second stage have completed collection, the data collector in the third stage is triggered to perform data collection.
  • the third stage After the information collected by all data collectors is completed, it means that all data collection is completed, that is, the target task is completed.
  • this application does not use an Actor as a scheduling node to schedule data collectors for data collection, this application divides the data collectors into multiple stages, so that data collectors that will cause data competition are divided At different stages, even if this application does not use Actor as a scheduling node, distributed data collection can be achieved, and locks will not be used.
  • the data collected by the data collector of the previous stage can be transmitted to the data collector of the next stage for use by the data collector of the next stage.
  • the data collected by all data collectors in the previous phase of the phase to be triggered are acquired, and all data collectors in the previous phase of the phase to be triggered are collected.
  • the collected data is saved in the local database, and then the data request instruction of the target data collector in the phase to be triggered is received.
  • the data request instruction is used to request the target data in the local database, and then retrieve it from the local database. Target data, and send the target data to the target data collector.
  • data exchange can be carried out between data collectors in different stages, and the data collected in the previous stage can be used by the data collector in the next stage, so that the data collector in the next stage can better complete data collection. Therefore, data exchange between different stages is beneficial to improve the efficiency of data collection.
  • the down data collector when a down data collector is detected, the down data collector is deleted, and the initialization information of the down data collector is obtained, and then the initialization information is used Create a new data collector to replace the down data collector.
  • a new data collector in the case of detecting that the data collector is down, a new data collector is created according to the initialization information of the down data collector, and then the new data collector replaces the aforementioned downtime
  • the new data collector can collect data from the beginning, or according to the information collected before the down data collector and the progress of the collection, the new data collector can continue to perform the collection according to the progress of the collection .
  • the above Actor model also includes at least one monitoring node.
  • the monitoring node is used to initialize the down data collector. Therefore, when the down data collector is detected, the monitoring node is used to monitor the downtime.
  • the data collector of the machine is initialized to control the down data collector to return to normal.
  • the embodiment of the application describes another method for solving the downtime problem of the data collector, that is, if the data collector is down, the monitoring device is used to initialize the down data collector. Specifically, after detecting that the data collector is down, the monitoring node is used to initialize the down data collector according to the initialization information of the data collector to control the down data collector to return to normal. Alternatively, the monitoring node is used to restore the downtime data collector to the normal state at the moment before the downtime.
  • the monitoring node After detecting that the above-mentioned data collector is down, obtain the downtime data collection recorded in the database
  • the state information of the data collector at the moment before the downtime, and the monitoring node is used to restore the data collector according to the state information of the data collector at the moment before the downtime, so that the data collector is restored to the normal state corresponding to the above state information.
  • the tag of the target task is obtained. The tag is used to indicate the importance level of the target task. If the importance level of the target task exceeds the preset level, then when creating the Actor model , Create the above-mentioned monitoring node, if the importance level of the target task exceeds the preset level, the above-mentioned monitoring node is not created.
  • the local end since the local end can detect whether the target task is completed by detecting whether all data collectors have completed data collection, it can detect whether the target task has been completed, and if it is detected that the target task is completed , Delete the target task in the task cluster, and detect and execute the next target task in the task cluster.
  • 207 Detect the number of created data collectors, and obtain the creation time of the created data collectors.
  • the above-mentioned created data collectors are detected, the number of data collectors are counted, and the creation time of each created data collector. Sort the created data collectors based on the order of creation time from morning to night.
  • the above created data collectors will be deleted in order according to the creation time from early to late until the remaining created data collectors The number of is less than or equal to the preset number.
  • the number of data collectors that have been created is detected. If the number of data collectors exceeds the preset number, the data collectors will be deleted until the number of remaining data collectors is less than or equal to the preset number. Among them, delete the data collectors in the order of creation time from early to late. This is because the data collectors created early are less likely to be used again than the data collectors created later.
  • the detailed process of deleting the created data collectors is to detect the number of created data collectors and obtain the creation time of the created data collectors. If the number of created data collectors is greater than the preset Set the number to determine the time node with the earliest creation time in the created data collector, and delete the data collector with the earliest creation time among the created data collectors, and then if you delete the created data collector If the number of created data collectors is still greater than the preset number after the data collector with the latest creation time in the selected data collector, it will trigger the execution of the data collector with the earliest creation time among the created data collectors. The steps of creating the oldest data collector among the created data collectors until the number of created data collectors is less than or equal to the preset number after deleting the data collector with the latest creation time.
  • the embodiment of this application is more detailed than the previous application embodiment, and describes in detail that the distributed data collection method can be executed only when the newly added task in the task cluster is a distributed task. In addition, it also describes in detail After the target task is completed, if the number of created data collectors exceeds the preset number, the redundant data collectors will be cleared, and the data collectors with an earlier creation time will be deleted, so that the data collectors created after the deletion will be deleted. The number is less than or equal to the preset number. Therefore, on the one hand, the embodiments of this application are only suitable for distributed tasks. On the other hand, the embodiments of this application ensure the smooth operation of the system of the device by clearing redundant data collectors in time after the target task is completed. And to ensure the processing of the next target task. In general, the efficiency of data collection has been further improved.
  • FIG. 3 is a schematic block diagram of a data acquisition device provided by an embodiment of the present application.
  • the data collection device of this embodiment includes: a detection unit 301, a creation unit 302, a division unit 303, an execution unit 304, and a deletion unit 305. specific:
  • the detection unit 301 is used to detect whether there is a newly added target task in the task cluster;
  • the creation unit 302 is configured to create an Actor model for the target task when it is detected that there is a newly added target task in the task cluster.
  • the Actor model includes multiple data collectors.
  • the data collector is used to collect data. Communication between two data collectors through message passing;
  • the dividing unit 303 is configured to divide the multiple data collectors into multiple stages and set the trigger sequence of the multiple stages. Each stage contains at least one data collector, and different data collectors of the same stage are located in different stages. In the terminal, they are used to collect data in different terminals;
  • the execution unit 304 is configured to sequentially trigger the data collectors of the multiple stages to collect data according to the trigger sequence of the multiple stages;
  • the deleting unit 305 is configured to delete the target task in the task cluster after detecting that the data collectors in the multiple stages have all collected data.
  • the aforementioned data acquisition device further includes a determining unit 306, configured to determine the current phase to be triggered according to the trigger sequence of the aforementioned multiple phases; the aforementioned executing unit 304 is specifically configured to determine if the aforementioned pending trigger is determined The phase is at the first place in the trigger sequence, or if it is determined that all data collectors in the previous phase of the phase to be triggered are received, then all data collectors in the phase to be triggered are triggered to collect data; the execution unit 304, It is also used to trigger all data collectors in the next stage of the above-mentioned to-be-triggered stage to collect data in the case of receiving the information that all the data collectors in the above-mentioned to-be-triggered stage have completed collection, until all of the above-mentioned multiple stages are received The information collected by the data collector is completed.
  • the above-mentioned data acquisition device further includes an acquiring unit 307, configured to acquire all data of the previous stage of the above-mentioned stage to be triggered before all data collectors that trigger the stage to be triggered above collect data
  • the data collected by the collector the above-mentioned data collection device also includes a storage unit 308, which is used to save the data collected by all the data collectors in the previous stage of the above-mentioned to-be-triggered phase into a local database;
  • the unit 309 is configured to receive the data request instruction of the target data collector in the to-be-triggered stage.
  • the data request instruction is used to request the target data in the local database; the acquisition unit 307 is also used to retrieve the above-mentioned target data; the above-mentioned data collection device further includes a sending unit 310 for sending the above-mentioned target data to the above-mentioned target data collector; the above-mentioned saving unit 308 is also used for receiving all data collectors in the above-mentioned to-be-triggered stage In the case of the collection completion message, save the data collected by all the data collectors in the above-mentioned to-be-triggered stage into the above-mentioned local database.
  • the detection unit 301 is also used to detect newly added tasks in the task cluster; the determination unit 306 is also used to determine that the newly added tasks are distributed tasks, and to determine the detection There are new target tasks in the above task cluster.
  • the detection unit 301 is also used to detect the number of created data collectors; the acquisition unit 307 is also used to acquire the creation time of the created data collectors; The deleting unit 305 is further configured to delete the created data collectors in sequence according to the order of the creation time from early to late if the number of the created data collectors is greater than the preset number, until the remaining The number of created data collectors is less than or equal to the above preset number.
  • the detection unit 301 is also used to detect a data collector that is down; the deletion unit 305 is also used to delete the data collector that is down; the acquisition unit 307 is also used to The initialization information of the above-mentioned down data collector is obtained; the above-mentioned data collection device further includes a replacement unit 311, which uses the above-mentioned initialization information to create a new data collector to replace the above-mentioned down data collector.
  • the aforementioned Actor model further includes at least one monitoring node, the aforementioned monitoring node is used to initialize a data collector that is down, and the detection unit 301 is also used to detect a data collector that is down;
  • the data collection device further includes a recovery unit 312, which is used to initialize the above-mentioned down data collector by using the above-mentioned monitoring node to control the above-mentioned down data collector to return to normal.
  • the creation unit builds an Actor model including multiple data collectors for the target task, and the execution unit uses the multiple data collectors to complete the target task
  • the dividing unit divides the multiple data collectors into multiple stages, and sets the execution order of the multiple stages, and then the execution unit triggers the multiple stages in sequence according to the execution order
  • the data collector collects data, and finally after the detection unit detects that all data collectors in all stages have collected the data, the deletion unit deletes the target task in the task cluster.
  • the data collection method provided by the embodiment of the application In, because multiple data collectors are actors, and actors communicate through information transmission, the advantage of information transmission is that there will be no data competition problems, and there is no need to synchronize blocking waiting when processing tasks, and this application
  • the embodiment does not use Actors to schedule the multiple data collectors for data collection
  • the embodiment of the present application divides the multiple data collectors into different stages, thereby setting the execution sequence of the data collectors. It is further ensured that data competition and deadlock will not occur during the data collection process. Therefore, the embodiment of the present application can realize complex distributed data collection without the use of locks, and greatly improve the concurrent performance of distributed data collection.
  • the application embodiment provides an efficient data collection method.
  • FIG. 4 is a schematic block diagram of a data acquisition device provided by another embodiment of the present application.
  • the data collection device in this embodiment may include: one or more processors 410 and a memory 420.
  • the aforementioned processor 410 and memory 420 are connected through a bus 430.
  • the memory 420 is used to store a computer program.
  • the computer program includes program instructions.
  • the processor 410 is used to execute the program instructions stored in the memory 420. specific:
  • the processor 410 is used to perform the function of the detection unit 301, which is used to detect whether there is a newly added target task in the task cluster; it is also used to perform the function of the creation unit 302, which is used to detect the presence of a new target in the task cluster
  • create an Actor model for the above target task In the case of a task, create an Actor model for the above target task.
  • the above Actor model includes multiple data collectors, the above data collectors are used to collect data, and the above multiple data collectors communicate through message passing; also use execution division
  • the function of the unit 303 is used to divide the multiple data collectors into multiple stages and set the trigger sequence of the multiple stages. Each stage contains at least one data collector.
  • Different data collectors of the same stage are located in different In the terminal, it is used to collect data in different terminals; it is also used to perform the function of the execution unit 304, which is used to trigger the data collectors of the multiple stages to collect data in sequence according to the trigger sequence of the multiple stages;
  • the function of the execution deleting unit 305 is used to delete the target task in the task cluster after detecting that the data collectors of the multiple stages have all collected data.
  • the aforementioned processor 410 is further configured to perform the function of the determining unit 306, and is configured to determine the current phase to be triggered according to the trigger sequence of the aforementioned multiple phases;
  • the phase is in the first place of the above trigger sequence, or if it is determined that all data collectors in the previous phase of the phase to be triggered are received, all data collectors in the phase to be triggered are triggered to collect data;
  • all data collectors in the next stage of the above-mentioned to-be-triggered stage are triggered to collect data until the completion of all data collectors in the above-mentioned multiple stages is received Information collected.
  • the above-mentioned processor 410 is further configured to perform the function of the acquiring unit 307, which is used to acquire the previous data of the above-mentioned stage to be triggered before all the data collectors that trigger the above-mentioned stage to be triggered collect data.
  • the data collected by all the data collectors in the phase; the processor 410 is also used for the storage unit 308, which is used to save the data collected by all the data collectors in the previous phase of the phase to be triggered to the local database; After receiving the data request instruction of the target data collector in the to-be-triggered stage, the data request instruction is used to request the target data in the local database; it is also used to retrieve the target data in the local database; the data collection
  • the device also includes a communication unit 430, which is used to perform the function of the sending unit 310, and is used to send the above-mentioned target data to the above-mentioned target data collector; the above-mentioned processor 410 is also used to receive all the data collectors in the phase to be triggered In the case of the collection completion message, save the data collected by all the data collectors in the above-mentioned to-be-triggered stage into the above-mentioned local database.
  • the above-mentioned processor 410 is also used to detect newly added tasks in the above-mentioned task cluster; the above-mentioned processor 410 is also used to determine that the above-mentioned newly added tasks are distributed tasks, and to determine the detection There are new target tasks in the above task cluster.
  • the above-mentioned processor 410 is also used to detect the number of created data collectors; the above-mentioned processor 410 is also used to obtain the creation time of the above-mentioned created data collectors; The processor 410 is further configured to, if the number of the created data collectors is greater than the preset number, according to the order of the creation time from early to late, delete the created data collectors in sequence until the remaining The number of created data collectors is less than or equal to the above preset number.
  • the above-mentioned processor 410 is also used to detect the data collector that is down; the above-mentioned processor 410 is also used to delete the above-mentioned down data collector; the above-mentioned processor 410 is also used to Acquire the initialization information of the above-mentioned down data collector; the above-mentioned processor 410 is also configured to perform the function of the replacement unit 311, and use the above-mentioned initialization information to create a new data collector to replace the above-mentioned down data collector.
  • the aforementioned Actor model further includes at least one monitoring node, the aforementioned monitoring node is used to initialize a data collector that is down, and the processor 410 is also used to detect a data collector that is down;
  • the device 410 is also used to perform the function of the recovery unit 312, and is used to initialize the above-mentioned down data collector by using the above-mentioned monitoring node to control the above-mentioned down data collector to return to normal.
  • the processor 410 may be a central processing unit (CPU), and the processor may also be other general-purpose processors or digital signal processors (DSP). , Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the memory 420 may include a read-only memory and a random access memory, and provides instructions and data to the processor 410. A part of the memory 420 may also include a non-volatile random access memory. For example, the memory 420 may also store device type information.
  • the processor 410 described in the embodiment of this application can execute the implementation described in the first embodiment and the second embodiment of the data collection method provided in the embodiment of this application, and can also execute the embodiment of this application.
  • the implementation of the described data acquisition device will not be repeated here.
  • a computer-readable storage medium stores a computer program, the computer program includes program instructions, and the program instructions are executed by a processor.
  • the computer-readable storage medium may be an internal storage unit of the data acquisition device of any of the foregoing embodiments, such as the hard disk or memory of the data acquisition device.
  • the computer-readable storage medium can also be an external storage device of the data acquisition device, such as a plug-in hard disk equipped on the data acquisition device, a Smart Media Card (SMC), a Secure Digital (SD) card, and a flash memory Card (Flash Card) etc.
  • the computer-readable storage medium may also include both an internal storage unit of the data collection device and an external storage device.
  • the computer-readable storage medium is used to store computer programs and other programs and data required by the data acquisition device.
  • the computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请公开了一种基于Actor模型的数据采集方法、装置及存储介质，应用于数据处理领域，其中方法包括：检测任务集群中是否存在新增的目标任务；在检测到任务集群中存在新增的目标任务的情况下，为目标任务创建Actor模型，Actor模型包括多个数据采集器；将多个数据采集器划分到多个阶段，并设定多个阶段的触发顺序；按照多个阶段的触发顺序，依次触发多个阶段的数据采集器采集数据；在检测到多个阶段的数据采集器全部采集完数据之后，在任务集群中删除目标任务。本申请中的多个数据采集器都为Actor并且设定了不同数据采集器的执行顺序，使得不需要锁便能调度多个数据采集器完成数据采集。

Description

一种基于Actor模型的数据采集方法、装置及存储介质
本申请要求于2019年05月20日提交中国专利局、申请号为201910428815.2、申请名称为“一种基于Actor模型的数据采集方法、装置及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据采集领域，尤其涉及一种基于Actor模型的数据采集方法、装置及存储介质。
背景技术
数据采集常常因为多线程或多进程并发访问某个临界资源而陷入死锁，这时可通过对该临界资源进行加锁，来建立线程或进程之间的互斥同步机制，以保障临界资源的完整性和一致性。单机的进程同步和线程同步有很多可用的互斥方案，比如互斥锁、信号量和条件变量等。但这些单机下的互斥方案在多机上是无法使用的，因为分布式系统的组件运行在不同的机器上，使得其不在统一的运行环境中。
分布式数据采集中，常常通过分布式锁解决对临界资源的互斥访问，比如Redis分布式锁和Zookeeper分布式锁的等。
但是分布式锁的实现比较复杂，且维护比较麻烦，需要较多维护成本，因此分布式锁不是一个解决分布式数据采集的死锁问题的理想方案，还是缺少了一种可以更高效的分布式数据采集方法。
发明内容
本申请实施例提供一种基于Actor模型数据采集方法，可以不使用锁便实现分布式数据采集。
第一方面，本申请实施例提供了一种基于Actor模型数据采集方法，该方法包括：
检测任务集群中是否存在新增的目标任务；
在检测到任务集群中存在新增的目标任务的情况下，为所述目标任务创建Actor模型，所述Actor模型包括多个数据采集器，所述数据采集器用于采集数据，所述多个数据采集器之间通过消息传递的方式通信；
将所述多个数据采集器划分到多个阶段，并设定所述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据；
在检测到所述多个阶段的所有数据采集器全部采集完数据之后，在所述任务集群中删除所述目标任务。
第二方面，本申请实施例提供了一种数据采集装置，该数据采集装置包括用于执行上述第一方面的数据采集方法的单元，该数据采集装置包括：
检测单元，用于检测任务集群中是否存在新增的目标任务；
创建单元，用于在检测到任务集群中存在新增的目标任务的情况下，为所述目标任务创建Actor模型，所述Actor模型包括多个数据采集器，所述数据采集器用于采集数据，所述多个数据采集器之间通过消息传递的方式通信；
划分单元，用于将所述多个数据采集器划分到多个阶段，并设定所述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
执行单元，用于按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据；
删除单元，用于在检测到所述多个阶段的数据采集器全部采集完数据之后，在所述任务集群中删除所述目标任务。
第三方面，本申请实施例提供了另一种数据采集装置，包括处理器和存储器，所述处理器和存储器相互连接，其中，所述存储器用于存储支持数据采集装置执行上述数据采集方法的计算机程序，所述计算机程序包括程序指令，所述处理器被配置用于调用所述程序指令，用以执行：
检测任务集群中是否存在新增的目标任务；
在检测到任务集群中存在新增的目标任务的情况下，为所述目标任务创建Actor模型，所述Actor模型包括多个数据采集器，所述数据采集器用于采集数据，所述多个数据采集器之间通过消息传递的方式通信；
将所述多个数据采集器划分到多个阶段，并设定所述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据；
在检测到所述多个阶段的所有数据采集器全部采集完数据之后，在所述任务集群中删除所述目标任务。
第四方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行，用以执行上述第一方面至第一方面的任意一种实现方式的数据采集方法。
本申请所提供的数据采集方法保证了数据采集过程中不会产生数据竞争和死锁，本申请不需要借助于锁便可以实现复杂的分布式数据采集，并且大大提高分布式数据采集的并发性能。
附图说明
为了更清楚地说明本申请实施例技术方案，下面将对实施例描述中所需要使用的附图 作简单地介绍。
图1是本申请实施例提供的一种数据采集方法的示意流程图；
图2是本申请另一实施例提供的一种数据采集方法的示意流程图；
图3是本申请实施例提供的一种数据采集装置的示意性框图；
图4是本申请实施例提供的一种数据采集装置的结构性框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述。
本申请主要应用于数据采集装置，该数据采集装置可以是传统数据采集装置、大型存储系统、台式电脑、笔记本电脑、平板电脑、掌上电脑、智能手机、便携式数字播放器、智能手表以及智能手环等等，本申请对此不做限制。
本申请实施例中描述的终端包括但不限于带通讯功能的设备、智能手机、平板电脑、笔记本电脑、台式电脑、便携式数字播放器、智能手环以及智能手表等。当终端向数据采集装置发送数据的时候，按照预设格式进行对数据的特性进行记录并传送，其中，数据的特性包括时间、地点、类型等。
参见图1，是本申请实施例提供一种数据采集方法的示意流程图，如图1所示数据采集方法可包括：
101：检测任务集群中是否存在新增的目标任务。
在本申请实施例中，检测任务集群中是否存在新增的任务，若存在新增的任务，则将该新增的任务作为目标任务。其中，任务集群指的是待处理的任务的集合，且该任务集群中的不同任务可能由不同的终端所提出，而任务集群中的任务可能是由多个终端共同处理的分布式任务，也可能是仅由一个终端处理的单机任务，检测的过程指的是，本端的数据采集装置定时检测任务集群中是否存在新增的任务，或者任务集群中新增一个任务，本端装置便能接收到用于指示任务集群中新增任务的消息。
在另一种可实施的方式中，上述目标任务不是任务集群中新增的任意任务，而是任务集群中新增的分布式任务，具体的，上述检测任务集群中是否存在新增的目标任务指的是，检测任务集群中的新增的任务，若确定该新增的任务为分布式任务，则确定检测到任务集群中存在新增的目标任务。
在本申请实施例中，上述检测任务集群中新增的任务为分布式任务的过程指的是，获取上述任务集群中新增的任务的类型，若该新增的任务的类型为分布式任务，则表示该新增的任务的类型为分布式任务，或者，获取上述新增的任务的难度等级，若确定该新增的任务的难度等级超过预设级别，则说明该新增的任务需要占用较多的计算资源，并确定该新增的任务为分布式任务，也即是将该难度级别超过预设级别的任务作为目标任务，并接下来采用分布式任务的处理方法对其进行处理。
可见，本申请实施例只有在确定任务集群新增了分布式任务的时候，才确定任务集群 中有新增的目标任务，上述检测到的任务集群中新增的分布式任务即目标任务。
102：在检测到任务集群中存在新增的目标任务的情况下，为该目标任务创建Actor模型，Actor模型包括多个数据采集器。
在本申请实施例中，Actor模型为一个用于任务处理的模型，具体来说Actor模型是一个概念模型，用于处理并发计算，它定义了一系列系统组件应该如何动作和交互的通用规则，Actor模型可以用于完成任意类型的任务，但在本申请实施例中构建的Actor模型用于执行数据采集。Actor模型中包含了多个Actor，Actor在本申请实施例中被称为数据采集器，每个数据采集器是Actor模型中进行数据采集的最基本的单元，它能接收一个消息并且基于其执行数据采集。于是上述根据该目标任务的执行内容创建包含多个数据采集器的Actor模型指的是，对目标任务进行分解，然后基于分解之后的目标任务的采集任务，创建多个数据采集器来执行对应的采集任务，需要注意的是，数据采集器在采集数据时，多个数据采集器可能是并行执行的，但也不会造成资源竞争，因为数据采集器之间是通过消息传递的方式进行通信，于是不会造成资源竞争。
需要说明的是，由于本申请实施例采用的所有数据采集器都是Actor，而Actor之间是通过消息传递的方式进行通信，因此数据采集器进行数据采集的时候，不会产生资源竞争的问题。其中，消息传递的方式指的是在消息传递系统中，进程间的数据交换是以格式化的消息(Message)为单位的，Actors的一大重要特征在于Actors之间相互隔离，它们并不互相共享内存，一个Actor能维持一个私有的状态，并且这个状态不可能被另一个Actor所改变，而消息传递也保证了Actor之间的异步处理，因此Actor之间也不会存在数据竞争。若通信的进程之间不存在可直接访问的共享空间，则必须利用操作系统提供的消息传递方法实现进程通信，进程通过系统提供的发送消息和接收消息两个原语进行数据交换。简单来说，Actor之间通过消息传递的方式来进行资源共享，当Actor需要另一个Actor的资源的时候，只需要将消息发送给另一个Actor，但不关心另一个Actor什么时候处理，因此Actor之间的消息是异步传输的，类似邮件通信。具体的，在Actor模型里每个Actor都有地址，所以它们才能够相互发送消息，尽管许多Actor同时运行，但是一个Actor只能顺序地处理消息。也就是说其它三个Actor发送了给一个Actor分别发送了一条消息，这个Actor只能一次处理一条，如果需要并行处理三条消息，则需要三个Actor分别对这三条消息进行处理，并且如果Actor在处理消息的过程中又接收到一条新的消息，则该新的消息会被存储到Actor的邮箱Mailbox中，每个Actor对应有一个Mailbox。
可见，Actor通过消息传递的方式将不同Actor的可访问的资源进行了分隔，并保障了Actor之间的消息的异步处理，从而避免了使用锁来处理数据条件竞争(data race)的问题。消息传递机制最大的优点就是不会产生数据竞争状态(data race)。
103：将上述多个数据采集器划分到多个阶段，并设定该多个阶段的触发顺序。
在本申请实施例中，将上述多个数据采集器划分到多个阶段，并设定该多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据。
需要说明的是，由于相同阶段的不同数据采集器分别分布于不同的终端上，于是本申请实施例应用于分布式数据采集，可见，由于位于不同的终端上的数据采集器在采集数据的时候，只会占用其所在终端的计算资源，于是在一个终端集群中的多个终端共同处理一个分布式采集任务，相比单机处理，分布式采集可以大大提升了计算能力和计算速度。需要注意的是，由于本申请实施例采用了Actor模型中的Actor来作为数据采集器，才可以不需要分布式锁便能实现分布式数据采集，这是因为Actor只关注接收到的消息的内容，而不关注消息的发送方是本地的Actor还是终端上的Actor，于是只要保证多个Actor之间的消息能够顺利达到即可，也也允许了本申请实施例可以不及住锁便能在包含多个终端的终端集群中构建Actor模型，并实现分布式数据采集。
104：按照上述多个阶段的触发顺序，依次触发上述多个阶段的数据采集器采集数据。
在本申请实施例中，按照上述多个阶段的触发顺序，依次触发上述多个阶段的数据采集器采集数据。具体的，按照上述多个阶段的触发顺序，确定当前待触发阶段；若确定待触发阶段位于上述触发顺序的首位，或者确定接收到待触发阶段的前一阶段的所有数据采集器的完成采集的信息，则触发待触发阶段的所有数据采集器采集数据；在接收到待触发阶段的所有数据采集器的完成采集的信息的情况下，触发待触发阶段的下一阶段的所有数据采集器采集数据，直至接收到所述多个阶段的所有数据采集器的完成采集的信息。
举例来说，上述Actor模型包含三个阶段的数据采集器，按照执行顺序排序，该三个阶段分别为第一阶段、第二阶段和第三阶段，于是首先想第一阶段的所有数据采集器发送数据采集指令，使得该阶段的所有数据采集器并发执行数据采集，当第一阶段的所有数据采集器采集完数据之后，本端装置将会收到第一阶段的每个数据采集器发送的完成采集的信息，当接受到第一阶段的所有数据采集器的完成采集的信息之后，表示第一阶段的所有数据采集器完成数据采集，可以触发执行第二阶段的数据采集器，于是触发第二阶段的所有数据采集器并发执行数据采集，当接收到第二阶段的所有数据采集器的完成采集的信息之后，触发执行第三阶段的数据采集器执行数据采集，当接收到第三阶段的所有数据采集器的完成采集的信息之后，表示数据全部采集完成，即目标任务完成。
需要说明的是，本申请实施例虽然没有用Actor来作为调度节点来调度数据采集器进行数据采集，但是本申请通过将数据采集器划分到多个阶段，使得会产生数据竞争的数据采集器划分到不同的阶段，于是本申请即使不使用Actor作为调度节点，也可以实现分布式数据采集，并且也不会用到锁。
在另一种可实施的方式中，上一阶段的数据采集器采集到的数据可以传输给下一阶段的数据采集器，以供下一阶段的数据采集器使用。具体的，在上述触发待触发阶段的所有数据采集器采集数据之前，获取待触发阶段的前一阶段的所有数据采集器采集到的数据，并将待触发阶段的前一阶段的所有数据采集器采集到的数据保存到本地数据库中，然后接收到待触发阶段的目标数据采集器的数据请求指令，该数据请求指令用于请求所述本地数据库中的目标数据，于是在上述本地数据库中调取目标数据，并将该目标数据发送给目标数据采集器，最后在接收到待触发阶段的所有数据采集器的采集完成消息的情况下，将待 触发阶段的所有数据采集器采集到的数据保存到上述本地数据库中。简单来说，就是每当一个数据采集器完成了数据采集之后，便将该数据采集器采集到的A数据保存到本地数据库中，然后在下一阶段的数据采集器请求该A数据的时候，便从本地数据库中提取该A数据发送给上述下一阶段的数据采集器。
可见，不同阶段的数据采集器之间是可以进行数据交流的，上一阶段采集到的数据可以供下一阶段的数据采集器使用，使得下一阶段的数据采集器更好完成数据采集。于是不同阶段之间的数据交流有利于提高数据采集的效率。
在另一种可实施的方式中，在检测到宕机的数据采集器的情况下，删除该宕机的数据采集器，并获取该宕机的数据采集器的初始化信息，然后利用该初始化信息创建新的数据采集器，以替代所述宕机的数据数据采集器。
在本申请实施例中，在检测到数据采集器宕机的情况下，按照该宕机的数据采集器的初始化信息创建的新的数据采集器，然后将该新的数据采集器替换上述宕机的数据采集器，然后新的数据采集器可以从头开始采集数据，或者按照宕机的数据采集器之前采集到的信息，以及采集的进度，新的数据采集器按照该采集的进度，继续执行采集。
在另一种可实施的方式中，上述Actor模型还包括至少一个监控节点，监控节点用于初始化宕机的数据采集器，于是在检测到宕机的数据采集器的时候，利用监控节点对宕机的数据采集器进行初始化，以控制宕机的数据采集器恢复正常。
本申请实施例描述了另一种解决数据采集器宕机问题的方法，即如果数据采集器宕机，则使用该监控设备对宕机的数据采集器进行初始化。具体的，在检测到数据采集器宕机之后，使用监控节点根据数据采集器的初始化信息对宕机的数据采集器进行初始化，以控制宕机的数据采集器恢复正常。或者，监控节点用于将宕机的数据采集器恢复到宕机前一个时刻的正常状态，具体的，在检测到上述数据采集器宕机之后，获取记录在数据库中的该宕机的数据采集器在宕机前一个时刻的状态信息，并使用监控节点根据上述数据采集器在宕机前一个时刻的状态信息对数据采集器进行恢复，使得数据采集器恢复到上述状态信息对应的正常状态。
在另一种可实施的方式中，在为目标任务创建Actor模型的时候，根据目标任务的重要程度决定是否在上述Actor模型中创建监控节点。具体的，在任务集群中检测到目标任务之后，获取该目标任务的标签，该标签用于指示目标任务的重要等级，若目标任务的重要等级超过了预设等级，则在创建Actor模型的时候，创建上述监控节点，若目标任务的重要等级未超过预设等级，则不创建上述监控节点。
105：在检测到上述多个阶段的所有数据采集器全部采集完数据之后，在上述任务集群中删除所述目标任务。
在本申请实施例中，在检测到上述目标任务完成之后，便在上述任务集群中删除上述目标任务，并检测和执行任务集群中的下一个目标任务。
在另一种可实施的方式中，上述在任务集群中删除目标任务之后，检测已创建的数据采集器的个数，并获取所述已创建的数据采集器的创建时间；若已创建的数据采集器的个 数大于预设个数，则按照创建时间从早到晚的排序，依次删除已创建的数据采集器，直到剩下的已创建的数据采集器的个数小于或等于上述预设个数。
在本申请实施例中，系统中的数据采集器可能太多而导致本端装置存储空间不足或者系统运行缓慢，于是在完成目标任务之后，检测已创建的数据采集器的个数，如果已创建的数据采集器的个数超过预设个数，就对数据采集器进行删减，直到剩下的数据采集器的个数小于等于预设个数。其中，按照数据采集器的创建时间由早到晚顺序进行删除，这是因为早创建的数据采集器，相较于晚创建的数据采集器而言，被再次使用的几率更小。
上述删除已创建的数据采集器的详细过程为，检测已创建的数据采集器的个数，并获取该已创建的数据采集器的创建时间，若该已创建的数据采集器的个数大于预设个数，则确定该已创建的数据采集器中创建时间最早的时间节点，并删除该已创建的数据采集器中创建时间最早的数据采集器，然后若在删除该已创建的数据采集器中创建时间最晚的数据采集器之后，已创建的数据采集器的个数仍大于预设个数，则触发执行上述确定已创建的数据采集器中创建时间最早的数据采集器，至删除已创建的数据采集器中创建时间最早的数据采集器的步骤，直到已创建的数据采集器的个数在删除了创建时间最晚的数据采集器之后小于或等于预设个数。
本申请实施例每当检测到任务集群中存在新的目标任务，就为该目标任务建立包括多个数据采集器的Actor模型，并利用多个数据采集器来完成目标任务所指示的数据采集任务，具体的，将上述多个数据采集器划分到多个阶段，并设置该多个阶段的执行顺序，然后再按照该执行顺序，依次触发该多个阶段的数据采集器采集数据，最后在检测到所有阶段的所有数据采集器都采集完数据之后，就在任务集群中删除该目标任务，可见，本申请实施例所提供的数据采集方法中，由于多个数据采集器都是Actor，而Actor是通过信息传递的方式进行通信的，信息传递的好处就是不会产生数据竞争的问题，在处理任务的时候无需同步阻塞等待，并且本申请实施例虽然不是通过Actor来调度上述多个数据采集器进行数据采集，但是本申请实施例通过将上述多个数据采集器划分到不同的阶段，从而设定了上述数据采集器的执行顺序，因此进一步保证了数据采集过程中不会产生数据竞争和死锁，于是本申请实施例不需要借助于锁便可以实现复杂的分布式数据采集，并且大大提高分布式数据采集的并发性能，可见，本申请实施例提供了一种高效的数据采集方法。
参见图2，是本申请实施例提供另一种数据采集方法的示意流程图，如图2所示数据采集方法可包括：
201：检测任务集群中新增的任务。
在本申请实施例中，检测任务集群中是否存在新增的任务，具体的检测过程指的是，本端的数据采集装置定时检测任务集群中是否存在新增的任务，或者任务集群中新增一个任务，本端装置便能接收到用于指示任务集群中新增任务的消息。其中，任务集群指的是待处理的任务的集合，且该任务集群中的不同任务可能由不同的终端所提出，而任务集群中的任务可能是由多个终端共同处理的分布式任务，也可能是仅由一个终端处理的单机任 务。
202：若确定上述新增的任务为分布式任务，则确定检测到上述任务集群中存在新增的目标任务。
在本申请实施例中，在检测到任务集群中的新增的任务之后，进一步确定该新增的任务是否为目标任务。具体的，确定该新增的任务是否为分布式任务，若是则确定检测到任务集群中存在新增的目标任务。
在本申请实施例中，上述检测任务集群中新增的任务为分布式任务的过程指的是，获取上述任务集群中新增的任务的类型，若该新增的任务的类型为分布式任务，则表示该新增的任务的类型为分布式任务，或者，获取上述新增的任务的难度等级，若确定该新增的任务的难度等级超过预设级别，则说明该新增的任务需要占用较多的计算资源，并确定该新增的任务为分布式任务，也即是将该难度级别超过预设级别的任务作为目标任务，并接下来采用分布式任务的处理方法对其进行处理。
可见，本申请实施例只有在确定任务集群新增了分布式任务的时候，才确定任务集群中有新增的目标任务，上述检测到的任务集群中新增的分布式任务即目标任务。
203：在检测到任务集群中存在新增的目标任务的情况下，为该目标任务创建Actor模型，Actor模型包括多个数据采集器。
在本申请实施例中，Actor模型为一个用于任务处理的模型，具体来说Actor模型是一个概念模型，用于处理并发计算，它定义了一系列系统组件应该如何动作和交互的通用规则，Actor模型可以用于完成任意类型的任务，但在本申请实施例中构建的Actor模型用于执行数据采集。Actor模型中包含了多个Actor，Actor在本申请实施例中被称为数据采集器，每个数据采集器是Actor模型中进行数据采集的最基本的单元，它能接收一个消息并且基于其执行数据采集。于是上述根据该目标任务的执行内容创建包含多个数据采集器的Actor模型指的是，对目标任务进行分解，然后基于分解之后的目标任务的采集任务，创建多个数据采集器来执行对应的采集任务，需要注意的是，数据采集器在采集数据时，多个数据采集器可能是并行执行的，但也不会造成资源竞争，因为数据采集器之间是通过消息传递的方式进行通信，于是不会造成资源竞争。
需要说明的是，由于本申请实施例采用的所有数据采集器都是Actor，而Actor之间是通过消息传递的方式进行通信，因此数据采集器进行数据采集的时候，不会产生资源竞争的问题。其中，消息传递的方式指的是在消息传递系统中，进程间的数据交换是以格式化的消息(Message)为单位的，Actors的一大重要特征在于Actors之间相互隔离，它们并不互相共享内存，一个Actor能维持一个私有的状态，并且这个状态不可能被另一个Actor所改变，而消息传递也保证了Actor之间的异步处理，因此Actor之间也不会存在数据竞争。若通信的进程之间不存在可直接访问的共享空间，则必须利用操作系统提供的消息传递方法实现进程通信，进程通过系统提供的发送消息和接收消息两个原语进行数据交换。简单来说，Actor之间通过消息传递的方式来进行资源共享，当Actor需要另一个Actor的资源的时候，只需要将消息发送给另一个Actor，但不关心另一个Actor什么时候处理，因此Actor 之间的消息是异步传输的，类似邮件通信。具体的，在Actor模型里每个Actor都有地址，所以它们才能够相互发送消息，尽管许多Actor同时运行，但是一个Actor只能顺序地处理消息。也就是说其它三个Actor发送了给一个Actor分别发送了一条消息，这个Actor只能一次处理一条，如果需要并行处理三条消息，则需要三个Actor分别对这三条消息进行处理，并且如果Actor在处理消息的过程中又接收到一条新的消息，则该新的消息会被存储到Actor的邮箱Mailbox中，每个Actor对应有一个Mailbox。
可见，Actor通过消息传递的方式将不同Actor的可访问的资源进行了分隔，并保障了Actor之间的消息的异步处理，从而避免了使用锁来处理数据条件竞争(data race)的问题。消息传递机制最大的优点就是不会产生数据竞争状态(data race)。
204：将上述多个数据采集器划分到多个阶段，并设定该多个阶段的触发顺序。
在本申请实施例中，将上述多个数据采集器划分到多个阶段，并设定该多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据。
需要说明的是，由于相同阶段的不同数据采集器分别分布于不同的终端上，于是本申请实施例应用于分布式数据采集，可见，由于位于不同的终端上的数据采集器在采集数据的时候，只会占用其所在终端的计算资源，于是在一个终端集群中的多个终端共同处理一个分布式采集任务，相比单机处理，分布式采集可以大大提升了计算能力和计算速度。需要注意的是，由于本申请实施例采用了Actor模型中的Actor来作为数据采集器，才可以不需要分布式锁便能实现分布式数据采集，这是因为Actor只关注接收到的消息的内容，而不关注消息的发送方是本地的Actor还是终端上的Actor，于是只要保证多个Actor之间的消息能够顺利达到即可，也也允许了本申请实施例可以不及住锁便能在包含多个终端的终端集群中构建Actor模型，并实现分布式数据采集。
205：按照上述多个阶段的触发顺序，依次触发上述多个阶段的数据采集器采集数据。
在本申请实施例中，按照上述多个阶段的触发顺序，依次触发上述多个阶段的数据采集器采集数据。具体的，按照上述多个阶段的触发顺序，确定当前待触发阶段；若确定待触发阶段位于上述触发顺序的首位，或者确定接收到待触发阶段的前一阶段的所有数据采集器的完成采集的信息，则触发待触发阶段的所有数据采集器采集数据；在接收到待触发阶段的所有数据采集器的完成采集的信息的情况下，触发待触发阶段的下一阶段的所有数据采集器采集数据，直至接收到所述多个阶段的所有数据采集器的完成采集的信息。
举例来说，上述Actor模型包含三个阶段的数据采集器，按照执行顺序排序，该三个阶段分别为第一阶段、第二阶段和第三阶段，于是首先想第一阶段的所有数据采集器发送数据采集指令，使得该阶段的所有数据采集器并发执行数据采集，当第一阶段的所有数据采集器采集完数据之后，本端装置将会收到第一阶段的每个数据采集器发送的完成采集的信息，当接受到第一阶段的所有数据采集器的完成采集的信息之后，表示第一阶段的所有数据采集器完成数据采集，可以触发执行第二阶段的数据采集器，于是触发第二阶段的所有数据采集器并发执行数据采集，当接收到第二阶段的所有数据采集器的完成采集的信息 之后，触发执行第三阶段的数据采集器执行数据采集，当接收到第三阶段的所有数据采集器的完成采集的信息之后，表示数据全部采集完成，即目标任务完成。
需要说明的是，本申请实施例虽然没有用Actor来作为调度节点来调度数据采集器进行数据采集，但是本申请通过将数据采集器划分到多个阶段，使得会产生数据竞争的数据采集器划分到不同的阶段，于是本申请即使不使用Actor作为调度节点，也可以实现分布式数据采集，并且也不会用到锁。
在另一种可实施的方式中，上一阶段的数据采集器采集到的数据可以传输给下一阶段的数据采集器，以供下一阶段的数据采集器使用。具体的，在上述触发待触发阶段的所有数据采集器采集数据之前，获取待触发阶段的前一阶段的所有数据采集器采集到的数据，并将待触发阶段的前一阶段的所有数据采集器采集到的数据保存到本地数据库中，然后接收到待触发阶段的目标数据采集器的数据请求指令，该数据请求指令用于请求所述本地数据库中的目标数据，于是在上述本地数据库中调取目标数据，并将该目标数据发送给目标数据采集器，最后在接收到待触发阶段的所有数据采集器的采集完成消息的情况下，将待触发阶段的所有数据采集器采集到的数据保存到上述本地数据库中。简单来说，就是每当一个数据采集器完成了数据采集之后，便将该数据采集器采集到的A数据保存到本地数据库中，然后在下一阶段的数据采集器请求该A数据的时候，便从本地数据库中提取该A数据发送给上述下一阶段的数据采集器。
可见，不同阶段的数据采集器之间是可以进行数据交流的，上一阶段采集到的数据可以供下一阶段的数据采集器使用，使得下一阶段的数据采集器更好完成数据采集。于是不同阶段之间的数据交流有利于提高数据采集的效率。
在另一种可实施的方式中，在检测到宕机的数据采集器的情况下，删除该宕机的数据采集器，并获取该宕机的数据采集器的初始化信息，然后利用该初始化信息创建新的数据采集器，以替代所述宕机的数据数据采集器。
在本申请实施例中，在检测到数据采集器宕机的情况下，按照该宕机的数据采集器的初始化信息创建的新的数据采集器，然后将该新的数据采集器替换上述宕机的数据采集器，然后新的数据采集器可以从头开始采集数据，或者按照宕机的数据采集器之前采集到的信息，以及采集的进度，新的数据采集器按照该采集的进度，继续执行采集。
在另一种可实施的方式中，上述Actor模型还包括至少一个监控节点，监控节点用于初始化宕机的数据采集器，于是在检测到宕机的数据采集器的时候，利用监控节点对宕机的数据采集器进行初始化，以控制宕机的数据采集器恢复正常。
本申请实施例描述了另一种解决数据采集器宕机问题的方法，即如果数据采集器宕机，则使用该监控设备对宕机的数据采集器进行初始化。具体的，在检测到数据采集器宕机之后，使用监控节点根据数据采集器的初始化信息对宕机的数据采集器进行初始化，以控制宕机的数据采集器恢复正常。或者，监控节点用于将宕机的数据采集器恢复到宕机前一个时刻的正常状态，具体的，在检测到上述数据采集器宕机之后，获取记录在数据库中的该宕机的数据采集器在宕机前一个时刻的状态信息，并使用监控节点根据上述数据采集器在 宕机前一个时刻的状态信息对数据采集器进行恢复，使得数据采集器恢复到上述状态信息对应的正常状态。
在另一种可实施的方式中，在为目标任务创建Actor模型的时候，根据目标任务的重要程度决定是否在上述Actor模型中创建监控节点。具体的，在任务集群中检测到目标任务之后，获取该目标任务的标签，该标签用于指示目标任务的重要等级，若目标任务的重要等级超过了预设等级，则在创建Actor模型的时候，创建上述监控节点，若目标任务的重要等级超过了预设等级，则不创建上述监控节点。
206：在检测到上述多个阶段的所有数据采集器全部采集完数据之后，在上述任务集群中删除所述目标任务。
在本申请实施例中，由于本端通过检测所有数据采集器是否都执行完数据采集而可以感知到目标任务是否完成，于是可以检测到上述目标任务是否已完成，若在检测到上述目标任务完成，则在上述任务集群中删除上述目标任务，并检测和执行任务集群中的下一个目标任务。
207：检测已创建的数据采集器的个数，并获取该已创建的数据采集器的创建时间。
在本申请实施例中，在上述目标任务完成，检测上述已创建的数据采集器，统计数据采集器的个数，以及每个已创建的数据采集器的创建时间。基于创建时间从早到晚的顺序，对已创建的数据采集器进行排序。
208：若上述已创建的数据采集器的个数大于预设个数，则按照创建时间从早到晚的排序，依次删除上述已创建的数据采集器，直到剩下的已创建的数据采集器的个数小于或等于预设个数。
在本申请实施例中，系统中的数据采集器可能太多而导致本端装置存储空间不足或者系统运行缓慢，于是在完成目标任务之后，检测已创建的数据采集器的个数，如果已创建的数据采集器的个数超过预设个数，就对数据采集器进行删减，直到剩下的数据采集器的个数小于等于预设个数。其中，按照数据采集器的创建时间由早到晚顺序进行删除，这是因为早创建的数据采集器，相较于晚创建的数据采集器而言，被再次使用的几率更小。
上述删除已创建的数据采集器的详细过程为，检测已创建的数据采集器的个数，并获取该已创建的数据采集器的创建时间，若该已创建的数据采集器的个数大于预设个数，则确定该已创建的数据采集器中创建时间最早的时间节点，并删除该已创建的数据采集器中创建时间最早的数据采集器，然后若在删除该已创建的数据采集器中创建时间最晚的数据采集器之后，已创建的数据采集器的个数仍大于预设个数，则触发执行上述确定已创建的数据采集器中创建时间最早的数据采集器，至删除已创建的数据采集器中创建时间最早的数据采集器的步骤，直到已创建的数据采集器的个数在删除了创建时间最晚的数据采集器之后小于或等于预设个数。
本申请实施例相对于上一申请实施例来说更加的详细，且详细描述了在只有当在任务集群中新增的任务是分布式任务才能执行上述分布式数据采集方法，此外，还详细描述了在目标任务完成之后，若已创建的数据采集器超过预设个数，就对多余的数据采集器进行 清除，清除创建时间较早的数据采集器，使得删除之后已创建的数据采集器的个数小于等于预设个数。因此本申请实施例一方面只适用于分布式任务，另一方面，本申请实施例通过在目标任务完成之后及时得对多余的数据采集器进行清除，来保障了本装置的系统的顺畅运行，并且保障了下一个目标任务的处理。总的来说数据采集的效率进一步被提高了。
需要说明的是，上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考，为了简洁，本文不再赘述。
本申请实施例还提供一种数据采集装置，该数据采集装置用于执行前述任一项的数据采集方法的单元。具体地，参见图3，是本申请实施例提供的一种数据采集装置的示意框图。本实施例的数据采集装置包括：检测单元301、创建单元302、划分单元303、执行单元304以及删除单元305。具体的：
检测单元301，用于检测任务集群中是否存在新增的目标任务；
创建单元302，用于在检测到任务集群中存在新增的目标任务的情况下，为上述目标任务创建Actor模型，上述Actor模型包括多个数据采集器，上述数据采集器用于采集数据，上述多个数据采集器之间通过消息传递的方式通信；
划分单元303，用于将上述多个数据采集器划分到多个阶段，并设定上述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
执行单元304，用于按照上述多个阶段的触发顺序，依次触发上述多个阶段的数据采集器采集数据；
删除单元305，用于在检测到上述多个阶段的数据采集器全部采集完数据之后，在上述任务集群中删除上述目标任务。
在另一种可实施的方式中，上述数据采集装置还包括确定单元306，用于按照上述多个阶段的触发顺序，确定当前待触发阶段；上述执行单元304，具体用于若确定上述待触发阶段位于上述触发顺序的首位，或者确定接收到上述待触发阶段的前一阶段的所有数据采集器的完成采集的信息，则触发上述待触发阶段的所有数据采集器采集数据；上述执行单元304，还用于在接收到上述待触发阶段的所有数据采集器的完成采集的信息的情况下，触发上述待触发阶段的下一阶段的所有数据采集器采集数据，直至接收到上述多个阶段的所有数据采集器的完成采集的信息。
在另一种可实施的方式中，上述数据采集装置还包括获取单元307，用于在上述触发上述待触发阶段的所有数据采集器采集数据之前，获取上述待触发阶段的前一阶段的所有数据采集器采集到的数据；上述数据采集装置还包括保存单元308，用于将上述待触发阶段的前一阶段的所有数据采集器采集到的数据保存到本地数据库中；上述数据采集装置还包括接收单元309，用于接收到上述待触发阶段的目标数据采集器的数据请求指令，上述数据请求指令用于请求上述本地数据库中的目标数据；上述获取单元307，还用于上述在上述本地数据库中调取上述目标数据；上述数据采集装置还包括发送单元310，用于将上 述目标数据发送给上述目标数据采集器；上述保存单元308还用于在接收到上述待触发阶段的所有数据采集器的采集完成消息的情况下，将上述待触发阶段的所有数据采集器采集到的数据保存到上述本地数据库中。
在另一种可实施的方式中，上述检测单元301，还用于检测上述任务集群中新增的任务；上述确定单元306，还用于确定上述新增的任务为分布式任务，以及确定检测到上述任务集群中存在新增的目标任务。
在另一种可实施的方式中，上述检测单元301，还用于检测已创建的数据采集器的个数；上述获取单元307，还用于获取上述已创建的数据采集器的创建时间；上述删除单元305，还用于若上述已创建的数据采集器的个数大于预设个数，则按照上述创建时间从早到晚的排序，依次删除上述已创建的数据采集器，直到剩下的已创建的数据采集器的个数小于或等于上述预设个数。
在另一种可实施的方式中，上述检测单元301还用于检测宕机的数据采集器；上述删除单元305，还用于删除上述宕机的数据采集器；上述获取单元307，还用于获取上述宕机的数据采集器的初始化信息；上述数据采集装置还包括替换单元311，利用上述初始化信息创建新的数据采集器，以替代上述宕机的数据数据采集器。
在另一种可实施的方式中，上述Actor模型还包括至少一个监控节点，上述监控节点用于初始化宕机的数据采集器，上述检测单元301，还用于检测宕机的数据采集器；上述数据采集装置还包括恢复单元312，用于利用上述监控节点对上述宕机的数据采集器进行初始化，以控制上述宕机的数据采集器恢复正常。
本申请实施例每当检测单元检测到任务集群中存在新的目标任务，创建单元就为该目标任务建立包括多个数据采集器的Actor模型，并执行单元利用多个数据采集器来完成目标任务所指示的数据采集任务，具体的，划分单元将上述多个数据采集器划分到多个阶段，并设置该多个阶段的执行顺序，然后执行单元再按照该执行顺序，依次触发该多个阶段的数据采集器采集数据，最后在检测单元检测到所有阶段的所有数据采集器都采集完数据之后，删除单元就在任务集群中删除该目标任务，可见，本申请实施例所提供的数据采集方法中，由于多个数据采集器都是Actor，而Actor是通过信息传递的方式进行通信的，信息传递的好处就是不会产生数据竞争的问题，在处理任务的时候无需同步阻塞等待，并且本申请实施例虽然不是通过Actor来调度上述多个数据采集器进行数据采集，但是本申请实施例通过将上述多个数据采集器划分到不同的阶段，从而设定了上述数据采集器的执行顺序，因此进一步保证了数据采集过程中不会产生数据竞争和死锁，于是本申请实施例不需要借助于锁便可以实现复杂的分布式数据采集，并且大大提高分布式数据采集的并发性能，可见，本申请实施例提供了一种高效的数据采集方法。
参见图4，是本申请另一实施例提供的一种数据采集装置示意框图。如图所示的本实施例中的数据采集装置可以包括：一个或多个处理器410和存储器420。上述处理器410和存储器420通过总线430连接。存储器420用于存储计算机程序，计算机程序包括程序 指令，处理器410用于执行存储器420存储的程序指令。具体的：
处理器410，用于执行检测单元301的功能，用于检测任务集群中是否存在新增的目标任务；还用于执行创建单元302的功能，用于在检测到任务集群中存在新增的目标任务的情况下，为上述目标任务创建Actor模型，上述Actor模型包括多个数据采集器，上述数据采集器用于采集数据，上述多个数据采集器之间通过消息传递的方式通信；还用执行划分单元303的功能，用于将上述多个数据采集器划分到多个阶段，并设定上述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；还用于执行执行单元304的功能，用于按照上述多个阶段的触发顺序，依次触发上述多个阶段的数据采集器采集数据；还用于执行删除单元305的功能，用于在检测到上述多个阶段的数据采集器全部采集完数据之后，在上述任务集群中删除上述目标任务。
在另一种可实施的方式中，上述处理器410，还用于执行确定单元306的功能，用于按照上述多个阶段的触发顺序，确定当前待触发阶段；还用于若确定上述待触发阶段位于上述触发顺序的首位，或者确定接收到上述待触发阶段的前一阶段的所有数据采集器的完成采集的信息，则触发上述待触发阶段的所有数据采集器采集数据；还用于在接收到上述待触发阶段的所有数据采集器的完成采集的信息的情况下，触发上述待触发阶段的下一阶段的所有数据采集器采集数据，直至接收到上述多个阶段的所有数据采集器的完成采集的信息。
在另一种可实施的方式中，上述处理器410还用于执行获取单元307的功能，用于在上述触发上述待触发阶段的所有数据采集器采集数据之前，获取上述待触发阶段的前一阶段的所有数据采集器采集到的数据；上述处理器410还用于保存单元308，用于将上述待触发阶段的前一阶段的所有数据采集器采集到的数据保存到本地数据库中；还用于接收到上述待触发阶段的目标数据采集器的数据请求指令，上述数据请求指令用于请求上述本地数据库中的目标数据；还用于上述在上述本地数据库中调取上述目标数据；上述数据采集装置还包括通信单元430，用于执行发送单元310的功能，用于将上述目标数据发送给上述目标数据采集器；上述上述处理器410还用于在接收到上述待触发阶段的所有数据采集器的采集完成消息的情况下，将上述待触发阶段的所有数据采集器采集到的数据保存到上述本地数据库中。
在另一种可实施的方式中，上述处理器410，还用于检测上述任务集群中新增的任务；上述处理器410，还用于确定上述新增的任务为分布式任务，以及确定检测到上述任务集群中存在新增的目标任务。
在另一种可实施的方式中，上述处理器410，还用于检测已创建的数据采集器的个数；上述处理器410，还用于获取上述已创建的数据采集器的创建时间；上述处理器410，还用于若上述已创建的数据采集器的个数大于预设个数，则按照上述创建时间从早到晚的排序，依次删除上述已创建的数据采集器，直到剩下的已创建的数据采集器的个数小于或等于上述预设个数。
在另一种可实施的方式中，上述处理器410还用于检测宕机的数据采集器；上述处理器410，还用于删除上述宕机的数据采集器；上述处理器410，还用于获取上述宕机的数据采集器的初始化信息；上述处理器410还用于执行替换单元311的功能，利用上述初始化信息创建新的数据采集器，以替代上述宕机的数据数据采集器。
在另一种可实施的方式中，上述Actor模型还包括至少一个监控节点，上述监控节点用于初始化宕机的数据采集器，上述处理器410还用于检测宕机的数据采集器；上述处理器410还用于执行恢复单元312的功能，用于利用上述监控节点对上述宕机的数据采集器进行初始化，以控制上述宕机的数据采集器恢复正常。
应当理解，在本申请实施例中，所称处理器410可以是中央处理单元(Central Processing Unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
该存储器420可以包括只读存储器和随机存取存储器，并向处理器410提供指令和数据。存储器420的一部分还可以包括非易失性随机存取存储器。例如，存储器420还可以存储设备类型的信息。
具体实现中，本申请实施例中所描述的处理器410可执行本申请实施例提供的数据采集方法的第一实施例和第二实施例中所描述的实现方式，也可执行本申请实施例所描述的数据采集装置的实现方式，在此不再赘述。
在本申请的另一实施例中提供一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序包括程序指令，程序指令被处理器执行。
计算机可读存储介质可以是前述任一实施例的数据采集装置的内部存储单元，例如数据采集装置的硬盘或内存。计算机可读存储介质也可以是数据采集装置的外部存储设备，例如数据采集装置上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，计算机可读存储介质还可以既包括数据采集装置的内部存储单元也包括外部存储设备。计算机可读存储介质用于存储计算机程序以及数据采集装置所需的其他程序和数据。计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。

Claims (20)

  1. 一种基于Actor模型的数据采集方法，其特征在于，包括：
    检测任务集群中是否存在新增的目标任务；
    在检测到任务集群中存在新增的目标任务的情况下，为所述目标任务创建Actor模型，所述Actor模型包括多个数据采集器，所述数据采集器用于采集数据，所述多个数据采集器之间通过消息传递的方式通信；
    将所述多个数据采集器划分到多个阶段，并设定所述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
    按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据；
    在检测到所述多个阶段的所有数据采集器全部采集完数据之后，在所述任务集群中删除所述目标任务。
  2. 根据权利要求1所述的方法，其特征在于，所述按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据，包括：
    按照所述多个阶段的触发顺序，确定当前待触发阶段；
    若确定所述待触发阶段位于所述触发顺序的首位，或者确定接收到所述待触发阶段的前一阶段的所有数据采集器的完成采集的信息，则触发所述待触发阶段的所有数据采集器采集数据，所述完成采集的信息用于指示所述数据采集器完成了数据采集；
    在接收到所述待触发阶段的所有数据采集器的完成采集的信息的情况下，触发所述待触发阶段的下一阶段的所有数据采集器采集数据，直至接收到所述多个阶段的所有数据采集器的完成采集的信息。
  3. 根据权利要求2所述的方法，其特征在于，在所述触发所述待触发阶段的所有数据采集器采集数据之前，所述方法还包括：
    获取所述待触发阶段的前一阶段的所有数据采集器采集到的数据，并将所述待触发阶段的前一阶段的所有数据采集器采集到的数据保存到本地数据库中；
    接收到所述待触发阶段的目标数据采集器的数据请求指令，所述数据请求指令用于请求所述本地数据库中的目标数据；
    在所述本地数据库中调取所述目标数据，并将所述目标数据发送给所述目标数据采集器；
    在接收到所述待触发阶段的所有数据采集器的采集完成消息的情况下，将所述待触发阶段的所有数据采集器采集到的数据保存到所述本地数据库中。
  4. 根据权利要求1所述的方法，其特征在于，所述检测任务集群中是否存在新增的目标任务，包括：
    检测所述任务集群中新增的任务；
    若确定所述新增的任务为分布式任务，则确定检测到所述任务集群中存在新增的目标 任务。
  5. 根据权利要求1所述的方法，其特征在于，所述在所述任务集群中删除所述目标任务之后，还包括：
    检测已创建的数据采集器的个数，并获取所述已创建的数据采集器的创建时间；
    若所述已创建的数据采集器的个数大于预设个数，则按照所述创建时间从早到晚的排序，依次删除所述已创建的数据采集器，直到剩下的已创建的数据采集器的个数小于或等于所述预设个数。
  6. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    在检测到宕机的数据采集器的情况下，删除所述宕机的数据采集器；
    获取所述宕机的数据采集器的初始化信息；
    利用所述初始化信息创建新的数据采集器，以替代所述宕机的数据数据采集器。
  7. 根据权利要求1所述的方法，其特征在于，所述Actor模型还包括至少一个监控节点，所述监控节点用于初始化宕机的数据采集器，所述方法还包括：
    若检测到宕机的数据采集器，则利用所述监控节点对所述宕机的数据采集器进行初始化，以控制所述宕机的数据采集器恢复正常。
  8. 根据权利要求1所述的方法，其特征在于，所述Actor模型还包括至少一个监控节点，所述监控节点用于初始化宕机的数据采集器，所述方法还包括：
    若检测到宕机的数据采集器，则获取所述宕机的数据采集器在宕机前一个时刻的状态信息；
    利用所述监控节点根据所述宕机的数据采集器在宕机前一个时刻的状态信息，对所述宕机的数据采集器进行恢复，使得所述宕机的数据采集器恢复到所述状态信息对应的正常状态。
  9. 根据权利要求1所述的方法，其特征在于，所述在检测到任务集群中存在新增的目标任务的情况下，所述方法还包括：
    获取所述目标任务的标签，所述标签用于指示所述目标任务的重要等级；
    根据所述目标任务的标签确定所述目标任务的重要等级；
    若所述目标任务的重要等级超过了预设等级，则创建包括有至少一个监控节点的Actor模型，所述监控节点用于初始化宕机的数据采集器。
  10. 一种数据采集装置，其特征在于，包括：
    检测单元，用于检测任务集群中是否存在新增的目标任务；
    创建单元，用于在检测到任务集群中存在新增的目标任务的情况下，为所述目标任务创建Actor模型，所述Actor模型包括多个数据采集器，所述数据采集器用于采集数据，所述多个数据采集器之间通过消息传递的方式通信；
    划分单元，用于将所述多个数据采集器划分到多个阶段，并设定所述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
    执行单元，用于按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据；
    删除单元，用于在检测到所述多个阶段的数据采集器全部采集完数据之后，在所述任务集群中删除所述目标任务。
  11. 一种数据采集装置，其特征在于，包括处理器和存储器，所述处理器和存储器相互连接；所述存储器用于存储计算机程序，所述计算机程序包括程序指令；所述处理器被配置用于调用所述程序指令，用以执行：
    检测任务集群中是否存在新增的目标任务；
    在检测到任务集群中存在新增的目标任务的情况下，为所述目标任务创建Actor模型，所述Actor模型包括多个数据采集器，所述数据采集器用于采集数据，所述多个数据采集器之间通过消息传递的方式通信；
    将所述多个数据采集器划分到多个阶段，并设定所述多个阶段的触发顺序，每个阶段包含至少一个数据采集器，相同阶段的不同数据采集器位于不同的终端中，分别用于采集不同终端中的数据；
    按照所述多个阶段的触发顺序，依次触发所述多个阶段的数据采集器采集数据；
    在检测到所述多个阶段的所有数据采集器全部采集完数据之后，在所述任务集群中删除所述目标任务。
  12. 根据权利要求11所述的装置，其特征在于，所述处理器具体用于：
    按照所述多个阶段的触发顺序，确定当前待触发阶段；
    若确定所述待触发阶段位于所述触发顺序的首位，或者确定接收到所述待触发阶段的前一阶段的所有数据采集器的完成采集的信息，则触发所述待触发阶段的所有数据采集器采集数据，所述完成采集的信息用于指示所述数据采集器完成了数据采集；
    在接收到所述待触发阶段的所有数据采集器的完成采集的信息的情况下，触发所述待触发阶段的下一阶段的所有数据采集器采集数据，直至接收到所述多个阶段的所有数据采集器的完成采集的信息。
  13. 根据权利要求12所述的装置，其特征在于，所述处理器，还用于：
    获取所述待触发阶段的前一阶段的所有数据采集器采集到的数据，并将所述待触发阶段的前一阶段的所有数据采集器采集到的数据保存到本地数据库中；
    接收到所述待触发阶段的目标数据采集器的数据请求指令，所述数据请求指令用于请求所述本地数据库中的目标数据；
    在所述本地数据库中调取所述目标数据，并将所述目标数据发送给所述目标数据采集器；
    在接收到所述待触发阶段的所有数据采集器的采集完成消息的情况下，将所述待触发阶段的所有数据采集器采集到的数据保存到所述本地数据库中。
  14. 根据权利要求11所述的装置，其特征在于，所述处理器具体用于：
    检测所述任务集群中新增的任务；
    若确定所述新增的任务为分布式任务，则确定检测到所述任务集群中存在新增的目标任务。
  15. 根据权利要求11所述的装置，其特征在于，所述处理器还用于：
    检测已创建的数据采集器的个数，并获取所述已创建的数据采集器的创建时间；
    若所述已创建的数据采集器的个数大于预设个数，则按照所述创建时间从早到晚的排序，依次删除所述已创建的数据采集器，直到剩下的已创建的数据采集器的个数小于或等于所述预设个数。
  16. 根据权利要求11所述的装置，其特征在于，所述处理器还用于：
    在检测到宕机的数据采集器的情况下，删除所述宕机的数据采集器；
    获取所述宕机的数据采集器的初始化信息；
    利用所述初始化信息创建新的数据采集器，以替代所述宕机的数据数据采集器。
  17. 根据权利要求11所述的装置，其特征在于，所述Actor模型还包括至少一个监控节点，所述监控节点用于初始化宕机的数据采集器，所述处理器还用于：
    若检测到宕机的数据采集器，则利用所述监控节点对所述宕机的数据采集器进行初始化，以控制所述宕机的数据采集器恢复正常。
  18. 根据权利要求11所述的装置，其特征在于，所述Actor模型还包括至少一个监控节点，所述监控节点用于初始化宕机的数据采集器，所述处理器还用于：
    若检测到宕机的数据采集器，则获取所述宕机的数据采集器在宕机前一个时刻的状态信息；
    利用所述监控节点根据所述宕机的数据采集器在宕机前一个时刻的状态信息，对所述宕机的数据采集器进行恢复，使得所述宕机的数据采集器恢复到所述状态信息对应的正常状态。
  19. 根据权利要求11所述的装置，其特征在于，所述处理器还用于：
    获取所述目标任务的标签，所述标签用于指示所述目标任务的重要等级；
    根据所述目标任务的标签确定所述目标任务的重要等级；
    若所述目标任务的重要等级超过了预设等级，则创建包括有至少一个监控节点的Actor模型，所述监控节点用于初始化宕机的数据采集器。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时使所述处理器执行如权利要求1-9任一项所述的方法。
PCT/CN2019/102705 2019-05-20 2019-08-27 一种基于Actor模型的数据采集方法、装置及存储介质 Ceased WO2020232870A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910428815.2A CN110245025B (zh) 2019-05-20 2019-05-20 一种基于Actor模型的数据采集方法、装置及存储介质
CN201910428815.2 2019-05-20

Publications (1)

Publication Number Publication Date
WO2020232870A1 true WO2020232870A1 (zh) 2020-11-26

Family

ID=67884718

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/102705 Ceased WO2020232870A1 (zh) 2019-05-20 2019-08-27 一种基于Actor模型的数据采集方法、装置及存储介质

Country Status (2)

Country Link
CN (1) CN110245025B (zh)
WO (1) WO2020232870A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118277225A (zh) * 2024-06-04 2024-07-02 杭州吾思智能科技有限公司 一种工业互联网平台的多任务数据采集管理方法及系统

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113961341A (zh) * 2021-10-15 2022-01-21 杭州安恒信息技术股份有限公司 基于Actor模型的并发数据处理方法、系统、设备和存储介质

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120173715A1 (en) * 2010-12-29 2012-07-05 Oracle International Corporation Application server platform for telecom-based applications using an actor container
CN109766326A (zh) * 2019-01-07 2019-05-17 重庆邮电大学 一种以语义化的方式进行智能家居中多智能体任务规划的方法

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9142024B2 (en) * 2008-12-31 2015-09-22 Lucasfilm Entertainment Company Ltd. Visual and physical motion sensing for three-dimensional motion capture
CN104063448B (zh) * 2014-06-18 2017-02-01 华东师范大学 一种视频领域相关的分布式微博数据抓取系统
CN104965761B (zh) * 2015-07-21 2018-11-02 华中科技大学 一种基于gpu/cpu混合架构的流程序多粒度划分与调度方法
US10432471B2 (en) * 2015-12-31 2019-10-01 Microsoft Technology Licensing, Llc Distributed computing dependency management system
CN106649124B (zh) * 2016-12-28 2019-04-02 桂林电子科技大学 一种基于Actor模型的并行动态符号执行方法和系统
CN107135156A (zh) * 2017-06-07 2017-09-05 努比亚技术有限公司 调用链数据采集方法、移动终端及计算机可读存储介质
CN109191103A (zh) * 2018-07-27 2019-01-11 阿里巴巴集团控股有限公司 一种数据梳理方法及装置

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120173715A1 (en) * 2010-12-29 2012-07-05 Oracle International Corporation Application server platform for telecom-based applications using an actor container
CN109766326A (zh) * 2019-01-07 2019-05-17 重庆邮电大学 一种以语义化的方式进行智能家居中多智能体任务规划的方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
朱雨晴 (ZHU, YUQING): "基于流式处理的数据采集系统的设计与实现 (Design and implementation of data acquisition system based on streaming processing)", 中国优秀硕士学位论文全文数据库信息科技辑 (CHINA MASTER’S THESES FULL-TEXT DATABASE, INFORMATION SCIENCE AND TECHNOLOGY), no. 11, 15 November 2018 (2018-11-15), DOI: 20200216095551X *
王世凯 (WANG, SHIKAI): "灾害监测中海量数据的处理方法究 (The research of massive data processing in disaster monitoring)", 中国优秀硕士学位论文全文数据库 信息科技辑 (CHINESE MASTER’S THESES FULL-TEXT DATABASE, INFORMATION SCIENCE & TECHNOLOGY), no. 6, 15 June 2016 (2016-06-15), DOI: 20200216100359A *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118277225A (zh) * 2024-06-04 2024-07-02 杭州吾思智能科技有限公司 一种工业互联网平台的多任务数据采集管理方法及系统

Also Published As

Publication number Publication date
CN110245025B (zh) 2022-05-10
CN110245025A (zh) 2019-09-17

Similar Documents

Publication Publication Date Title
WO2020232875A1 (zh) 一种基于Actor模型的任务调度方法、装置及存储介质
US11860861B2 (en) Growing dynamic shared memory hash table
EP2802109B1 (en) System and method of a shared memory hash table with notifications
JP5888336B2 (ja) データ処理方法、分散処理システムおよびプログラム
CN109271363B (zh) 一种文件存储的方法及设备
WO2019223599A1 (zh) 数据采集系统、方法、节点设备及存储介质
CN116701387A (zh) 数据分段写入方法、数据读取方法及装置
CN117633116A (zh) 数据同步方法、装置、电子设备及存储介质
WO2020232870A1 (zh) 一种基于Actor模型的数据采集方法、装置及存储介质
US20170039143A1 (en) System and method of a shared memory hash table with notifications and reduced memory utilization
CN120639787A (zh) 一种数据双向同步方法、装置、设备及介质
WO2020143181A1 (zh) 数据存储方法、装置、计算机设备及存储介质
US10031859B2 (en) Pulse counters
CN109359005B (zh) 一种跨进程的数据采集处理方法
CN120492231B (zh) 数据库数据处理方法、装置、设备和存储介质
US20140040191A1 (en) Inventorying and copying file system folders and files
US8341368B2 (en) Automatic reallocation of structured external storage structures
JP2628811B2 (ja) マルチプロセッサ・システムにおけるタスクの処理及びメッセージの送受信制御方法
CN101692737A (zh) 一种面向移动rfid的轻量级数据同步架构
CN114064676A (zh) 一种数据处理方法及第一处理单元
CN115185917A (zh) 基于嵌入式Linux系统多程序日志处理方法、装置及终端
RU2490702C1 (ru) Способ ускорения обработки множественных запросов типа select к rdf базе данных с помощью графического процессора
CN115904802A (zh) 一种数据处理的方法及计算设备
CN115269519A (zh) 一种日志检测方法、装置及电子设备
US10613913B1 (en) Funnel locking for normal RCU grace period requests

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19930084

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19930084

Country of ref document: EP

Kind code of ref document: A1