CN113127184A - Data analysis method, device, equipment and medium - Google Patents
Data analysis method, device, equipment and medium Download PDFInfo
- Publication number
- CN113127184A CN113127184A CN201911404909.2A CN201911404909A CN113127184A CN 113127184 A CN113127184 A CN 113127184A CN 201911404909 A CN201911404909 A CN 201911404909A CN 113127184 A CN113127184 A CN 113127184A
- Authority
- CN
- China
- Prior art keywords
- data
- analysis
- overload
- self
- analysis unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 93
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000004458 analytical method Methods 0.000 claims abstract description 243
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 abstract description 22
- 238000010586 diagram Methods 0.000 description 10
- 230000003287 optical effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 3
- 238000004148 unit process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a data analysis method, a data analysis device, data analysis equipment and a data analysis medium. Wherein the method comprises the following steps: determining overload data in the data to be processed according to the available analysis capacity of a local analysis unit and the analysis capacity required by the data to be processed; generating self-explanatory data for the overload data to analyze the overload data from the self-explanatory data by an available analysis unit in a data analysis system. The technical scheme of the embodiment of the invention solves the problems of data load balancing and data time-sharing processing of the analysis unit.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data analysis method, a data analysis device, data analysis equipment and a data analysis medium.
Background
In the security industry, services borne by an intelligent analysis system are large in scale and various in service types, wherein the analysis system is used as a core system of intelligent security, and how to fully utilize the existing analysis capability to process more data becomes a basic appeal of a user.
In the prior art, the most common way for an analysis system to perform data analysis is to use real-time processing, and when the workload of real-time data exceeds the analysis capability of an analysis unit, in order to ensure that subsequent services are not affected, part of data to be analyzed is discarded.
However, the discarding strategy adopted for the real-time data that cannot be processed in time may cause partial loss of the critical data and incomplete data, thereby affecting the analysis and processing of the critical service.
Disclosure of Invention
The invention provides a data analysis method, a data analysis device, data analysis equipment and a data analysis medium, which solve the problems of data load balancing and data time-sharing processing of an analysis unit.
In a first aspect, an embodiment of the present invention provides a data analysis method, which is executed by an analysis unit of a data analysis system, and the method includes:
determining overload data in the data to be processed according to the available analysis capacity of a local analysis unit and the analysis capacity required by the data to be processed;
generating self-explanatory data for the overload data to analyze the overload data from the self-explanatory data by an available analysis unit in a data analysis system.
In a second aspect, an embodiment of the present invention further provides a data analysis method, which is executed by a central server in a data analysis system, and the method includes:
generating overload data and self-interpretation data of the overload data by an analysis unit of a data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed;
and allocating an available analysis unit for the overload data so as to analyze the overload data according to the self-interpretation data through the available analysis unit.
In a third aspect, an embodiment of the present invention further provides a data analysis apparatus configured in an analysis unit of a data analysis system, where the apparatus includes:
the overload data determining module is used for determining overload data in the data to be processed according to the available analysis capacity of the local analysis unit and the analysis capacity required by the data to be processed;
and the self-interpretation data generation module is used for generating self-interpretation data for the overload data so as to analyze the overload data according to the self-interpretation data through the self-interpretation data available in the data analysis system.
In a fourth aspect, an embodiment of the present invention further provides a data analysis apparatus configured in a central server for data analysis, where the apparatus includes:
the generation module is used for generating overload data and self-interpretation data of the overload data through an analysis unit of the data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed; and the distribution module is used for distributing an available analysis unit for the overload data so as to analyze the overload data according to the self-interpretation data through the available analysis unit.
In a fifth aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data analysis method as in any one of the embodiments of the invention.
In a sixth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data analysis method according to any one of the embodiments of the present invention.
According to the method and the device, the overload data in the data to be processed is determined according to the available analysis capacity of the local analysis unit and the analysis capacity required by the data to be processed, and meanwhile, the self-interpretation data is generated for the overload data, so that the overload data is analyzed according to the self-interpretation data through the available analysis unit in the data analysis system. According to the technical scheme of the embodiment of the invention, the overload data is generated from the interpretation data, and the available analysis unit in the data analysis system is used for analyzing and processing the overload data, so that the problems of load balance and data time-sharing processing when the analysis unit processes the data are solved.
Drawings
Fig. 1 is a flowchart of a data analysis method according to an embodiment of the present invention;
fig. 2 is a network diagram of a data analysis system according to an embodiment of the present invention;
FIG. 3 is a flow chart of another data analysis method provided by an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data analysis method according to an embodiment of the present invention;
FIG. 5 is a flow chart of another data analysis method provided by the embodiment of the invention;
fig. 6 is a schematic structural diagram of a data analysis apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of another data analysis apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a flowchart of a data analysis method provided in an embodiment of the present invention, where the present embodiment is applicable to a case of analyzing data, and typically, the method is applicable to a case of analyzing data by an analysis unit in an intelligent analysis system in a security industry. The method can be executed by a data analysis device configured in an analysis unit of the data analysis system, and the device can be realized in a software and/or hardware acquisition mode. Referring to fig. 1, the method may specifically include the following steps:
and step 110, determining overload data in the data to be processed according to the available analysis capability of the local analysis unit and the analysis capability required by the data to be processed.
The analysis unit is an actual calculation unit for performing intelligent analysis, the available analysis capability of a single analysis unit is limited, and generally, due to authorization and hardware design limitations, the single analysis unit can only perform analysis of one service type at the same time.
Referring to fig. 2 in particular, fig. 2 is a networking diagram of a data analysis system according to an embodiment of the present invention. The computing nodes correspond to analysis units in the data analysis system, and the central server is a management server of the analysis unit cluster and is mainly responsible for management of the analysis units and workload scheduling. A user can configure a central server of the data analysis system through a client, task distribution is carried out on the analysis unit through the central server of the analysis system, and the analysis unit carries out real-time analysis on data to be processed distributed by the central server.
If the analysis capability required by the data to be processed is larger than the available analysis capability of the local analysis unit, determining overload data in the data to be processed according to the difference value between the analysis capability and the available analysis capability, and simultaneously analyzing other data except the overload data in real time by the local analysis unit.
And 120, generating self-interpretation data for the overload data so as to analyze the overload data according to the self-interpretation data through an available analysis unit in a data analysis system.
Wherein the self-explanatory data includes a task type of the overload data;
alternatively, the self-explanatory data includes at least one of time information, source information, and forwarding information of the overload data, and a task type.
Illustratively, the task types are mutually independent intelligent analysis services, and may include data analysis types such as face analysis, video structuring, behavior analysis and the like, the time information is the establishment time of the overload data, the source information is an overload data acquisition path, and the priority order of the overload data can be determined according to the time information and/or the source information, so as to determine the processing order of the overload data. Optionally, the self-explanatory data of the overload data further includes information of the analysis unit to which the overload data is initially assigned.
Wherein generating self-explanatory data for the overload data comprises:
generating self-explanatory data for the overload data in at least one of the following forms: xml, json, memory block, text.
The overload data can contain the self-running context information by adding the context information to the overload data to form the self-explained data, so that the overload data can be correctly analyzed and processed on any analysis unit according to the self-explained data.
The available analysis unit is any analysis unit with available analysis capacity greater than or equal to the analysis capacity required by the overload data in the current data analysis system.
After the self-interpretation data of the overload data is generated, as an optional implementation manner, if an available analysis unit exists in the current data processing system, the self-interpretation data of the overload data is directly sent to the available analysis unit, and the available analysis unit analyzes the overload data according to the self-interpretation data of the overload data. As another alternative, if there is no available analysis unit in the current data processing system for a while, the self-interpreted data of the overloaded data may be stored in the cache server, so as to complete the caching of the self-interpreted data of the overloaded data. Optionally, the overload data and the context information are packaged together to form self-explanatory data, and the self-explanatory data of each overload data may be stored in the cache service according to the order of priority.
According to the technical scheme of the embodiment, the overload data in the data to be processed is determined according to the available analysis capability of the local analysis unit and the analysis capability required by the data to be processed, and meanwhile, the self-explanation data is generated for the overload data, so that the overload data is analyzed according to the self-explanation data through the available analysis unit in the data analysis system, and the problems of load balance and data time-sharing processing when the analysis unit processes the data are solved.
On the basis of the above embodiment, the method further includes: receiving self-explanatory data of overload data of any analysis unit;
the overload data is analyzed based on the received self-explanatory data of the overload data.
Optionally, the available analysis unit may directly receive the self-interpretation data of the overload data from another analysis unit, or directly obtain the self-interpretation data of the corresponding overload data from the cache server, and after obtaining the self-interpretation data of the overload data, the available analysis unit analyzes the overload data according to the self-interpretation data of the overload data.
Optionally, the available analysis unit may directly receive the self-interpretation data of the overload data from other analysis units, and specifically, the available analysis units may notify each other of the available analysis capability of the other side, and completely separate from the central server without the central server participating in monitoring, when the self-interpretation data of the overload data is generated on any one of the analysis units, while or before the self-interpretation data of the overload data is generated, if the analysis unit already knows that the available analysis unit meeting the analysis capability required by the overload data exists in the current data processing system, the self-interpretation data of the overload data is directly sent to the available analysis unit, so that the available analysis unit analyzes the overload data according to the self-interpretation data of the overload data; the central server may also be used to monitor the available analysis capability of each analysis unit in real time, and if the central server monitors that there are other analysis units satisfying the analysis capability required by the overload data while or before the generation of the self-explanatory data of the overload data, the self-explanatory data of the overload data may be directly sent to the available analysis units for analysis without being stored.
Specifically, according to the received self-explanatory data of the overload data, the overload data is analyzed, which includes:
and analyzing the overload data based on an analysis algorithm associated with the task type according to the task type of the received overload data.
For example, if the task type of the overload data received by the available analysis unit is face analysis, the available analysis unit may analyze the overload task by using an analysis algorithm of the face analysis.
On the basis of the above embodiment, after analyzing the overload data based on the analysis algorithm associated with the task, the method further includes:
determining a storage server of the overload data according to the received forwarding information of the overload data;
and forwarding the analysis result of the overload data to the storage server.
The available analysis unit can forward the analysis result of the overload data to the corresponding storage server according to the forwarding information of the overload data after processing the overload data.
Fig. 3 is a flowchart of another data analysis method according to an embodiment of the present invention. The embodiment can be suitable for the condition of analyzing the data, and typically, the method can be applied to the condition of analyzing the data by the analysis unit in the intelligent analysis system in the security industry. The method can be executed by a data analysis device configured in a central server of the data analysis system, and the device can be realized in a software and/or hardware acquisition mode. Referring to fig. 3, the method may specifically include the following steps:
step 210, generating overload data and self-interpretation data of the overload data through an analysis unit of a data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed.
And step 220, allocating an available analysis unit to the overload data, so that the available analysis unit analyzes the overload data according to the self-interpretation data.
In this embodiment, when the analysis unit processes the to-be-processed data distributed by the central server, the analysis unit reports the data to the central server in real time, and the central server may monitor the available analysis capability of the analysis unit in real time, and when an analysis unit satisfying the analysis capability required by the overload data is monitored, the analysis unit is used as an available analysis unit for the overload data, and issues a control instruction to the available analysis unit to instruct the available analysis unit to acquire the self-interpretation data of the overload data from the data cache server, so that the available analysis unit analyzes the overload data according to the self-interpretation data.
Optionally, if the central server monitors that other analysis units are available while monitoring the overload data of the analysis unit, the self-interpretation data of the overload data may not be stored, and is directly sent to the available analysis unit for analysis.
For example, referring to fig. 4, in the data analysis system, if the current workload of the analysis unit 1 exceeds the processing capacity of the analysis unit, the analysis unit 1 generates self-interpretation data for the overload data in the data to be analyzed, and sends the self-interpretation data of the overload data to the cache server for storage.
Wherein the self-explanatory data includes a task type of the overload data;
alternatively, the self-explanatory data includes at least one of time information, source information, and forwarding information of the overload data, and a task type.
Specifically, if the central server detects that the remaining analysis capacity of other analysis units 2 in the current data analysis system is greater than or equal to the analysis capacity required by the overload data, a control signal is sent to the analysis unit 2 to instruct the analysis unit 2 to acquire the self-interpretation data of the overload data from the data cache server, and the overload data is analyzed according to the self-interpretation data of the overload data.
When the workload of the analysis unit 1 decreases to the point that the available analysis capacity is greater than the required analysis capacity of the overload data, the analysis unit 1 will automatically connect to the central cache server, obtain the self-interpretation data of the overload data, and continue to analyze and process the overload data according to the self-interpretation data.
According to the technical scheme of the embodiment of the invention, the overload data and the self-interpretation data of the overload data are generated by the analysis unit in the data analysis system, the available analysis unit is distributed to the overload data, and the data to be processed by the analysis unit is scheduled, so that the analysis work can be transferred from the analysis unit with high load to the analysis unit with low load, and the effect of real-time data processing load balancing is achieved. Due to the existence of the cache server, when the workload of the whole analysis system is higher, the self-explained data to be analyzed can still be stored, and after the workload is reduced, the historical data in the cache is analyzed, so that the effect of time-sharing processing of the data is achieved.
Fig. 5 is a flowchart of another data analysis method according to an embodiment of the present invention, and the present embodiment further details the step 220 on the basis of the foregoing embodiment. Referring to fig. 5, the method may specifically include the following steps:
step 310, generating overload data and self-interpretation data of the overload data through an analysis unit of a data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed.
And 320, distributing available analysis units for the overload data according to the available analysis capacity of the analysis units in the data analysis system and the analysis capacity and priority required by the overload data, so that the overload data is analyzed according to the self-interpretation data through the available analysis units.
Optionally, before allocating an available analysis unit for the overload data, the method further includes:
and determining the priority of the overload data according to the time information and/or the source information of the overload data.
In this embodiment, the central server allocates available analysis units for the overloaded data from analysis units whose available analysis capacity is greater than or equal to the analysis capacity required by the overloaded data.
If a plurality of overload data exist in the current cache server, distributing available analysis units for the overload data which has high priority order and the required analysis capacity close to the available analysis capacity of the analysis units according to the required analysis capacity and the priority order of the overload data comprehensively.
According to the technical scheme of the embodiment, the available analysis units are distributed for the overload data according to the available analysis capacity of the analysis units in the data analysis system and the analysis capacity and priority required by the overload data, so that the load balance of the data analysis system can be realized while the priority processing of important data is ensured.
Fig. 6 is a schematic structural diagram of a data analysis apparatus according to an embodiment of the present invention, the apparatus being configured in an analysis unit of a data analysis system. Referring to fig. 6, the apparatus may specifically include:
and an overload data determining module 410, configured to determine overload data in the to-be-processed data according to an available analysis capability of the local analysis unit and an analysis capability required by the to-be-processed data.
A self-interpretation data generation module 420, configured to generate self-interpretation data for the overload data, so as to analyze the overload data according to the self-interpretation data through an available analysis unit in a data analysis system.
Wherein the self-explanatory data includes a task type of the overload data;
alternatively, the self-explanatory data includes at least one of time information, source information, and forwarding information of the overload data, and a task type.
The self-explanatory data generation module 420 is further specifically configured to: generating self-explanatory data for the overload data in at least one of the following forms: xml, json, memory block, text.
On the basis of the embodiment, the device further comprises an overload data analysis module and a storage module.
The overload data analysis module is used for receiving self-interpretation data of the overload data of any analysis unit;
the overload data is analyzed based on the received self-explanatory data of the overload data.
Specifically, the overload data analysis module is further configured to: and analyzing the overload data based on an analysis algorithm associated with the task type according to the task type of the received overload data.
The storage module is used for: determining a storage server of the overload data according to the received forwarding information of the overload data;
and forwarding the analysis result of the overload data to the storage server.
The data analysis device provided by the embodiment of the invention can execute the data analysis method provided by any embodiment of the invention, has corresponding functional modules and beneficial effects of the execution method, and is not repeated.
Fig. 7 is a schematic structural diagram of another data analysis apparatus according to an embodiment of the present invention, which is disposed in a central server of a data analysis system. Referring to fig. 7, the apparatus specifically includes:
a generating module 510, configured to generate overload data and self-interpretation data of the overload data through an analysis unit of the data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed. An assigning module 520, configured to assign an available analyzing unit to the overload data, so that the overload data is analyzed according to the self-explanatory data by the available analyzing unit.
Wherein the self-explanatory data includes a task type of the overload data;
alternatively, the self-explanatory data includes at least one of time information, source information, and forwarding information of the overload data, and a task type.
On the basis of the above embodiment, the apparatus further includes: and the priority determining module is used for determining the priority of the overload data according to the time information and/or the source information of the overload data.
Specifically, the allocating module 520 is configured to allocate an available analyzing unit to the overload data according to the available analyzing capability of an analyzing unit in the data analyzing system, and the analyzing capability and priority required by the overload data, so that the overload data is analyzed according to the self-explanatory data by the available analyzing unit.
The data analysis device provided by the embodiment of the invention can execute the data analysis method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 8 is a schematic structural diagram of an apparatus according to an embodiment of the present invention. FIG. 8 illustrates a block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention. The device 12 shown in fig. 8 is only an example and should not bring any limitation to the function and scope of use of the embodiments of the present invention.
As shown in FIG. 8, device 12 is in the form of a general purpose computing device. The components of device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 8, and commonly referred to as a "hard drive"). Although not shown in FIG. 8, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a data analysis method provided by an embodiment of the present invention.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data analysis method provided in an embodiment of the present invention, and the method includes:
determining overload data in the data to be processed according to the available analysis capacity of a local analysis unit and the analysis capacity required by the data to be processed;
generating self-explanatory data for the overload data to analyze the overload data from the self-explanatory data by an available analysis unit in a data analysis system. And/or:
generating overload data and self-interpretation data of the overload data by an analysis unit of a data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed;
and if so, allocating an available analysis unit for the overload data so as to analyze the overload data according to the self-interpretation data through the available analysis unit.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (14)
1. A data analysis method, performed by an analysis unit of a data analysis system, the method comprising:
determining overload data in the data to be processed according to the available analysis capacity of a local analysis unit and the analysis capacity required by the data to be processed;
generating self-explanatory data for the overload data to analyze the overload data from the self-explanatory data by an available analysis unit in a data analysis system.
2. The method of claim 1, wherein the self-explanatory data includes a task type of the overload data;
alternatively, the self-explanatory data includes at least one of time information, source information, and forwarding information of the overload data, and a task type.
3. The method of claim 1, wherein generating self-explanatory data for the overload data comprises:
generating self-explanatory data for the overload data in at least one of the following forms: xml, json, memory block, text.
4. The method of claim 1, further comprising:
receiving self-explanatory data of overload data of any analysis unit;
the overload data is analyzed based on the received self-explanatory data of the overload data.
5. The method of claim 4, wherein analyzing the overload data based on self-explanatory data of the received overload data comprises:
and analyzing the overload data based on an analysis algorithm associated with the task type according to the task type of the received overload data.
6. The method of claim 5, wherein after analyzing the overload data based on the analysis algorithm associated with the task, further comprising:
determining a storage server of the overload data according to the received forwarding information of the overload data;
and forwarding the analysis result of the overload data to the storage server.
7. A data analysis method, performed by a central server of a data analysis system, the method comprising:
generating overload data and self-interpretation data of the overload data by an analysis unit of a data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed;
and allocating an available analysis unit for the overload data so as to analyze the overload data according to the self-interpretation data through the available analysis unit.
8. The method of claim 7, wherein the self-explanatory data includes a task type of the overload data;
alternatively, the self-explanatory data includes at least one of time information, source information, and forwarding information of the overload data, and a task type.
9. The method of claim 8, further comprising, prior to assigning available analysis units to the overload data:
and determining the priority of the overload data according to the time information and/or the source information of the overload data.
10. The method of claim 9, wherein assigning an available analysis unit to the overload data for analysis of the overload data by the available analysis unit based on the self-explanatory data comprises:
and allocating available analysis units for the overload data according to the available analysis capacity of the analysis units in the data analysis system, and the analysis capacity and the priority required by the overload data, so that the overload data is analyzed according to the self-interpretation data through the available analysis units.
11. A data analysis apparatus, provided in an analysis unit of a data analysis system, the apparatus comprising:
the overload data determining module is used for determining overload data in the data to be processed according to the available analysis capacity of the local analysis unit and the analysis capacity required by the data to be processed;
and the self-interpretation data generation module is used for generating self-interpretation data for the overload data so as to analyze the overload data according to the self-interpretation data through an available analysis unit in a data analysis system.
12. A data analysis apparatus provided in a central server for data analysis, the apparatus comprising:
the generation module is used for generating overload data and self-interpretation data of the overload data through an analysis unit of the data analysis system; wherein the overload data is determined by the analysis unit based on the available analysis capacity of the analysis unit and the required analysis capacity for the data to be processed;
and the distribution module is used for distributing an available analysis unit for the overload data so as to analyze the overload data according to the self-interpretation data through the available analysis unit.
13. An apparatus, characterized in that the apparatus comprises:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data analysis method as claimed in any one of claims 1-6 or claims 7-10.
14. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out a data analysis method as claimed in any one of the claims 1 to 6 or claims 7 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911404909.2A CN113127184A (en) | 2019-12-31 | 2019-12-31 | Data analysis method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911404909.2A CN113127184A (en) | 2019-12-31 | 2019-12-31 | Data analysis method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127184A true CN113127184A (en) | 2021-07-16 |
Family
ID=76768573
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911404909.2A Pending CN113127184A (en) | 2019-12-31 | 2019-12-31 | Data analysis method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127184A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103236168A (en) * | 2013-04-23 | 2013-08-07 | 深圳职业技术学院 | Traffic data online acquisition system and method |
| US20140067920A1 (en) * | 2012-08-31 | 2014-03-06 | International Business Machines Corporation | Data analysis system |
| CN107045461A (en) * | 2017-04-13 | 2017-08-15 | 宇龙计算机通信科技(深圳)有限公司 | The method and device of data processing |
| CN110473015A (en) * | 2019-08-09 | 2019-11-19 | 南京智骋致想电子科技有限公司 | A kind of smart ads system and advertisement placement method |
-
2019
- 2019-12-31 CN CN201911404909.2A patent/CN113127184A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140067920A1 (en) * | 2012-08-31 | 2014-03-06 | International Business Machines Corporation | Data analysis system |
| CN103236168A (en) * | 2013-04-23 | 2013-08-07 | 深圳职业技术学院 | Traffic data online acquisition system and method |
| CN107045461A (en) * | 2017-04-13 | 2017-08-15 | 宇龙计算机通信科技(深圳)有限公司 | The method and device of data processing |
| CN110473015A (en) * | 2019-08-09 | 2019-11-19 | 南京智骋致想电子科技有限公司 | A kind of smart ads system and advertisement placement method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8595722B2 (en) | Preprovisioning virtual machines based on request frequency and current network configuration | |
| US8214843B2 (en) | Framework for distribution of computer workloads based on real-time energy costs | |
| US9497096B2 (en) | Dynamic control over tracing of messages received by a message broker | |
| US9367359B2 (en) | Optimized resource management for map/reduce computing | |
| US8539192B2 (en) | Execution of dataflow jobs | |
| US8397293B2 (en) | Suspicious node detection and recovery in mapreduce computing | |
| CN109284180B (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN106557369A (en) | A kind of management method and system of multithreading | |
| US9870269B1 (en) | Job allocation in a clustered environment | |
| US20140115046A1 (en) | Stream processing using a client-server architecture | |
| CN108595316B (en) | Lifecycle management method, manager, device, and medium for distributed application | |
| US11489735B2 (en) | Dynamic network allocation apparatus, dynamic network allocation method and non-transitory computer-readable medium | |
| US20120158451A1 (en) | Dispatching Tasks in a Business Process Management System | |
| US10884839B2 (en) | Processing system for performing predictive error resolution and dynamic system configuration control | |
| CN109144697B (en) | Task scheduling method and device, electronic equipment and storage medium | |
| US10838798B2 (en) | Processing system for performing predictive error resolution and dynamic system configuration control | |
| CN114090537A (en) | Real-time analysis method, device, system, equipment and medium for satellite in-orbit state | |
| CN110717132A (en) | Data collection method and pushing method for full-link monitoring system and related equipment | |
| EP3633508A1 (en) | Load distribution for integration scenarios | |
| CN109343958B (en) | Computing resource allocation method and device, electronic equipment and storage medium | |
| CN111680869A (en) | Method and device for monitoring release strategy and electronic equipment | |
| CN113127184A (en) | Data analysis method, device, equipment and medium | |
| CN117093335A (en) | Task scheduling method and device for distributed storage system | |
| CN113656161A (en) | Method, device and system for scheduling pressure generation service in performance test | |
| CN114077481A (en) | Task scheduling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |