CN107317722B - data source extensible system and method - Google Patents

data source extensible system and method Download PDF

Info

Publication number
CN107317722B
CN107317722B CN201710392604.9A CN201710392604A CN107317722B CN 107317722 B CN107317722 B CN 107317722B CN 201710392604 A CN201710392604 A CN 201710392604A CN 107317722 B CN107317722 B CN 107317722B
Authority
CN
China
Prior art keywords
data
task
data source
unit
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710392604.9A
Other languages
Chinese (zh)
Other versions
CN107317722A (en
Inventor
冯盼盼
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing QIYI Century Science and Technology Co Ltd
Original Assignee
Beijing QIYI Century Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing QIYI Century Science and Technology Co Ltd filed Critical Beijing QIYI Century Science and Technology Co Ltd
Priority to CN201710392604.9A priority Critical patent/CN107317722B/en
Publication of CN107317722A publication Critical patent/CN107317722A/en
Application granted granted Critical
Publication of CN107317722B publication Critical patent/CN107317722B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/565—Conversion or adaptation of application format or content

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Cardiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides data source extensible systems and methods, which relate to the field of Internet, and the systems comprise a visualization module, a data task management module, a data processing center and a data acquisition adaptation module, wherein the data task management module comprises a th judgment unit, a task creation unit and a distribution unit, the visualization module comprises a request creation unit, and the data acquisition adaptation module comprises a data acquisition unit and a data adaptation unit.

Description

data source extensible system and method
Technical Field
The invention relates to the field of internet, in particular to an data source extensible system and a method.
Background
With the rapid development of internet technology, more and more network services and platforms provide rich and varied functions for users, each company or group has a network service deployment company, and usually maintains a plurality of online services, different services complete different functions, and the services usually depend on each other, and the service quality of any network services affects the quality of the whole network products and services of the company, so how to ensure the service quality of the network services is very important.
Since most teams develop sets of Dashboard (business intelligent Dashboard) systems by themselves, agent agents collect data during service operation and then display the data in the Dashboard system, so that the service operation state is visualized, potential problems of the service can be seen more intuitively, and meanwhile, when the service fails, the failure cause can be analyzed by looking up the historical data trend.
Generally, monitored data is not homogeneous, some data is data of a database, some data is data of machine resources, some data is log data, and various different data need to be presented in the same dashboards, which is not a simple matter , and the common method is to separately present the data, for example, sets of database monitoring systems for data development of the database, sets of log monitoring systems for log data development, sets of machine resource monitoring systems for machine resource data development, so that there are two disadvantages, that is, , if a new data source needs to be supported, developers need to rewrite sets of monitoring systems for presentation, which consumes a large amount of manpower and material resources, and secondly, different teams may have the same requirements, but each development can invest repeated time cost, which wastes the whole manpower and material resources of a company.
Disclosure of Invention
In view of the above, the present invention has been made to provide data source scalable systems and corresponding data source scalable methods that overcome or at least partially solve the above problems.
According to aspects of the invention, a data source extensible system is provided, which comprises a visualization module, a data task management module, a data processing center and a data acquisition adaptation module;
the visualization module comprises a request creation unit;
the request creating unit is used for creating a detection request according to the type of a data source and sending the detection request to the th judging unit of the data task management module;
the data task management module comprises an th judging unit, a task creating unit and a distributing unit;
the th judging unit is configured to, after receiving the probe request sent by the request creating unit, judge whether the type of the data source supports being tested according to the probe request;
the task creating unit is configured to create a probe task after the th judging unit judges that the type support of the data source is tested;
the distribution unit is used for distributing the detection task to the data acquisition unit of the data acquisition adaptation module after the task creation unit creates the detection task;
the data acquisition adaptation module comprises a data acquisition unit and a data adaptation unit;
the data acquisition unit is used for acquiring the data source according to the detection task after receiving the detection task sent by the distribution unit;
the data adaptation unit is used for converting the data source into a general format and then sending the data source to the data processing center after the data acquisition unit acquires the data source;
the data processing center is used for calculating the data source with the general format according to the requirement of the detection task after the data adapting unit sends the data source with the general format; and sending the calculation result to the visualization module.
Optionally, based on the data source extensible system, the task creating unit includes:
the index receiving subunit is used for receiving the test index of the data source sent by the visualization module;
and the task creating subunit is used for creating the detection task according to the test index of the data source.
Optionally, based on the data source scalable system,
the data task management module further comprises:
a non-support feedback unit, configured to return a non-support test message to the visualization module if the data task management module determines, according to the probe request, that the type of the data source does not support the tested data;
the plug-in unit is used for uploading the plug-in corresponding to the type of the data source to the data task management module after the visual module receives the test unsupported message;
the plug-in loading unit is used for loading the plug-in to the data acquisition adapting module by the task management module;
the data acquisition adaptation module further comprises:
and the data conversion unit is used for converting the data source type corresponding to the plug-in into a universal format by the data acquisition adaptation module according to the plug-in, so that the data processing center processes the data source.
Optionally, based on the data source extensible system, the data task management module further includes:
the second judging unit is used for judging the detection task;
and the period starting unit is used for setting a time period and starting the detection task if the detection task is a timing task.
According to aspects of the invention, data source scalable methods are provided, comprising:
the visualization module creates a detection request according to the type of the data source and sends the detection request to the data task management module;
the data task management module judges whether the type of the data source supports to be tested or not according to the detection request;
if the type of the data source supports to be tested, the data task management module creates a detection task;
the data task management module distributes the detection task to a data acquisition adaptation module;
the data acquisition adaptation module acquires the data source according to the detection task;
the data acquisition adaptation module converts the data source into a universal format and then sends the universal format to the data processing center;
the data processing center calculates the data source with the general format according to the requirement of the detection task; and sending the calculation result to the visualization module.
Optionally, the step of creating a probe task if the type of the data source supports being tested includes:
the data task management module returns a message supporting the test to the visualization module;
the visualization module sends the test index of the data source to the data task management module;
and creating the detection task according to the test index of the data source.
Optionally, the method further comprises:
if the data task management module judges that the type of the data source does not support the tested data according to the detection request, returning a test-not-supporting message to the visualization module;
after receiving the test unsupported message, the visualization module uploads a plug-in corresponding to the type of the data source to the data task management module;
the task management module loads the plug-in to the data acquisition adaptation module;
and the data acquisition adaptation module converts the data source type corresponding to the plug-in into a universal format according to the plug-in, so that the data processing center processes the data source.
Optionally, after the step of the data task management module distributing the probe task to the data acquisition adaptation module, the method further includes:
the data task management module judges the detection task;
and if the detection task is a timing task, setting a time period and starting the detection task.
The embodiment of the invention has the following advantages:
when detecting different types of data sources, checking whether the data task management module of the system supports the types of the data sources, if so, establishing a detection task by using the data task management module of the system, so that the system has expansion compatibility for different types of data sources; therefore, the problem that time cost and labor cost are high when index detection is needed to be carried out on different data source types is solved, development labor of different teams for protocol detection is saved, and time cost is reduced.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the alternative embodiments. The drawings are only for purposes of illustrating alternative embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a block diagram illustrating an apparatus embodiment of a data source scalable system according to embodiments of the invention;
FIG. 2 is a block diagram illustrating an apparatus embodiment of a data source scalable system according to another embodiments of the invention;
FIG. 3 is a flowchart illustrating method steps of a data source scalable method in accordance with embodiments of the present invention;
FIG. 4 is a flowchart of method steps for a data source scalable method according to another embodiments of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Example
Referring to FIG. 1, there is shown a structure of an embodiment of a data source extensible system according to embodiments of the present invention, the system including a visualization module 110, a data task management module 120, a data processing center 140, and a data acquisition adaptation module 130;
the visualization module 110 includes a request creation unit 111;
the request creating unit 111 is configured to create a probe request according to a type of a data source, and send the probe request to the th determining unit 121 of the data task management module 120;
after confirming the type of the data source to be detected, the user creates a probe request in the human-computer interaction interface of the visualization module 110 according to the type of the data source, and after the probe request is successfully created, the user sends the probe request to the th judgment unit 121 of the data task management module 120;
optionally, the visualization module 110 provides human-machine interface, such as a web interface, to the user for the user to create the probe request based on the probed data source.
The data task management module 120 includes an th judgment unit 121, a task creation unit 122, and a distribution unit 123;
the -th determining unit 121 is configured to determine, after receiving the probe request sent by the request creating unit 111, whether the type of the data source supports being tested according to the probe request;
the judgment unit 121 extracts a data source type corresponding to the probe request after receiving the probe request sent by the request creation unit 111, searches whether a plug-in corresponding to the data source type is stored in a storage unit of the data task management module 120 according to the data source type, if the plug-in corresponding to the data source is stored in the storage unit, the judgment unit 121 sends a plug-in confirmation command to the visualization module 110, after the visualization module 110 receives the plug-in confirmation command, a user confirms an index for executing the data source probe and a data source acquisition address according to the type of the data source through the visualization module 110, and the visualization module sends the type of the data source, the index for detecting the data source and the data source acquisition address to the task creation unit 122 of the data task management module 120.
The task creating unit 122 is configured to create a probe task after the th determining unit 121 determines that the type support of the data source is tested;
the task creating unit 122 receives the type of the data source and the index detected by the data source and the data source acquiring address sent by the visualization module 110 after the th determining unit 121 determines that the type support of the data source is tested, and creates a detection task according to the type of the data source and the index detected by the data source and the data source acquiring address.
The distributing unit 123 is configured to distribute the probe task to the data acquiring unit 131 of the data acquiring adaptation module 130 after the task creating unit 122 creates the probe task;
the task creating unit 122 creates a probe task, and then sends the probe task to the distributing unit 123; the distribution unit 123 obtains a data source obtaining address in the probe task, and distributes the probe task to the data obtaining unit 131 of the data obtaining adaptation module 130 corresponding to the data source obtaining address.
The data acquisition adaptation module 130 includes a data acquisition unit 131 and a data adaptation unit 132;
the data obtaining unit 131 is configured to obtain the data source according to the probe task after receiving the probe task sent by the distributing unit 123;
the data obtaining unit 131 obtains the data source after receiving the probe task; if the data source is log data, the data obtaining unit 131 queries a log and obtains the data source from the log; if the data source is database data, the data obtaining unit 131 obtains the data source from a database; the format of the data source is the format of the data source in the original memory. The data obtaining unit 131 sends the data source to the adapting unit 132.
The data adapting unit 132 is configured to convert the data source into a general format and send the converted data source to the data processing center 140 after the data obtaining unit 131 obtains the data source;
after receiving the data source, the data adaptation unit 140 converts the data source into a general data format of the system, so that the data source can be smoothly circulated and calculated in the system, and other modules or units can quickly respond to the format of the data source.
For example, the data source a is data from a database, the data format acquired by the data source acquirer is data of multiple columns, for example, two columns including a user name and a user age are provided in the database, the data format acquired by the acquirer is two columns of data, the th column is a user name, the second column is an age, and since the database is data of multiple rows, it would be rows of during storage, and each row is arrays, which is the format taken by the data source acquirer.
The data adapting unit 132 forwards the data source in the generic data format to the data processing center 140.
The data processing center 140 is configured to calculate the data source in the common format according to the requirement of the probe task after the data adapting unit 132 sends the data source in the common format; and sends the calculation result to the visualization module 110.
The data processing center 140 determines whether the data source is compliant according to the probing task, deletes the data that is determined to be non-compliant, and performs index calculation on the remaining compliant data; storing the index calculation result and sending the index calculation result to the visualization module; and displaying the index calculation result to a user through a human-computer interface of the visualization module.
The embodiment of the invention has the following advantages:
when detecting different types of data sources, checking whether the data task management module of the system supports the types of the data sources, if so, establishing a detection task by using the data task management module of the system, so that the system has expansion compatibility for different types of data sources; therefore, the problem that time cost and labor cost are high when index detection is needed to be carried out on different data source types is solved, development labor of different teams for protocol detection is saved, and time cost is reduced.
Example two
Referring to fig. 2, there is shown the structure of an embodiment of data source extensible visual system according to another embodiments of the present invention, the system includes a visualization module 210, a data task management module 220, a data processing center 240 and a data acquisition adaptation module 230;
the visualization module 210 includes a request creation unit 211;
the request creating unit 211 is configured to create a probe request according to a type of a data source, and send the probe request to the th determining unit 221 of the data task management module 220;
after confirming the type of the data source to be detected, the user creates a detection request in the human-computer interaction interface of the visualization module 210 according to the type of the data source, and after the detection request is successfully created, the user sends the detection request to the th judgment unit 221 of the data task management module 220;
optionally, the visualization module 210 provides human-machine interface, such as a web interface, to the user for the user to create the probe request based on the probed data source.
The data task management module 220 includes an th judgment unit 221, a task creation unit 222, and a distribution unit 223;
the -th judging unit 221 is configured to, after receiving the probe request sent by the request creating unit 211, judge whether the type of the data source supports being tested according to the probe request;
the judgment unit 221 extracts a data source type corresponding to the probe request after receiving the probe request sent by the request creation unit 211, searches whether a plug-in corresponding to the data source type is stored in a storage unit of the data task management module 220 according to the data source type, if the plug-in corresponding to the data source is stored in the storage unit, the judgment unit 221 sends a plug-in confirmation command to the visualization module 210, after receiving the plug-in confirmation command, the visualization module 210 confirms an index for executing the data source probe and a data source acquisition address according to the type of the data source by the user, and sends the type of the data source, the index for detecting the data source and the data source acquisition address to the task creation unit 222 of the data task management module 220.
The task creating unit 222 is configured to create a probe task after the th determining unit 221 determines that the type support of the data source is tested;
optionally, the task creating unit 222 includes:
an index receiving subunit 2221, configured to receive the test index of the data source sent by the visualization module;
a task creating subunit 2222, configured to create the probe task according to the test indicator of the data source.
The task creating unit 222 sends a signal that the type support of the data source is tested to the visualization module 210 after the th judging unit 221 judges that the type support of the data source is tested, the visualization module 210 obtains the data source detection index and simultaneously obtains a data field obtaining address selected by a user in a man-machine interaction interface according to the type of the data source after receiving the signal, the index receiving subunit 2221 sends the data source detection index and the data field obtaining address to the task creating unit 222, the index receiving subunit 2221 receives the test index and the data source obtaining address of the data source sent by the visualization module 210, the index receiving subunit 2221 sends the test index and the data source obtaining address of the data source to the task creating subunit 2222, and the task creating subunit 2222 creates the detection task according to the test index of the data source.
The distributing unit 223 is configured to distribute the probe task to the data acquiring unit 231 of the data acquiring adapting module 230 after the task creating unit 222 creates the probe task;
the task creating subunit 2222, after creating the probe task, sends the probe task to the distributing unit 223; the distributing unit 223 acquires a data source acquisition address in the probe task, and distributes the probe task to the data acquiring unit 231 of the data acquisition adapting module 230 corresponding to the data source acquisition address.
Optionally, the data task management module 220 further includes:
a second judging unit 224, configured to judge the probe task;
a period starting unit 225, configured to set a time period and start the probe task if the probe task is a timing task.
For example, after the task creating unit 222 of the data task management module 220 creates the probe task, the task creating unit sends the probe task to the second determining unit 224, determines whether the probe task is a periodic unit, and if so, sends a period command to the period starting unit 225; the period starting unit 225 sets a time period to start the probe task.
For example, if the probe task is a single on-demand data task, the second determining unit 224 sends an instruction of a single task to cause the data distributing unit 223 to distribute the probe task to the data acquiring unit 231 of the data acquiring adaptation module 230, and if the second determining unit 224 determines that the probe task is a periodic task, the second determining unit 224 sends an instruction of a period start to the period starting unit 225, for example, probe tasks are started every 5 minutes to cause the data distributing unit 223 to acquire the data source according to the data acquiring unit 231 every 5 minutes.
The data acquisition adaptation module 230 includes a data acquisition unit 231 and a data adaptation unit 232;
the data acquiring unit 231 is configured to acquire the data source according to the probe task after receiving the probe task sent by the distributing unit 223;
the data obtaining unit 231 obtains the data source after receiving the probe task; if the data source is log data, the data obtaining unit 231 queries the log and obtains the data source from the log; if the data source is database data, the data obtaining unit 231 obtains the data source from a database; the format of the data source is the format of the data source in the original memory. The data obtaining unit 231 sends the data source to the adapting unit 232.
The data adapting unit 232 is configured to convert the data source into a general format and send the converted data source to the data processing center 240 after the data obtaining unit 231 obtains the data source;
after receiving the data source, the data adaptation unit 240 converts the data source into a general data format of the system, so that the data source can be smoothly circulated and calculated in the system, and other modules or units can quickly respond to the format of the data source.
For example, the data source a is data from a database, the data format acquired by the data source acquirer is data of multiple columns, for example, two columns including a user name and a user age are provided in the database, the data format acquired by the acquirer is two columns of data, the th column is a user name, the second column is an age, and since the database is data of multiple rows, it would be rows of during storage, and each row is arrays, which is the format taken by the data source acquirer.
The data adaptation unit 232 forwards the data source in the generic data format to the data processing center 240.
The data processing center 240 is configured to calculate the data source with the common format according to the requirement of the probe task after the data adapting unit 232 sends the data source with the common format; and sends the calculation result to the visualization module 210.
The data processing center 240 determines whether the data source is compliant according to the probe task for the data source with the common format, deletes the data determined to be non-compliant, and performs index calculation for the remaining compliant data; storing the index calculation result and sending the index calculation result to the visualization module; and displaying the index calculation result to a user through a human-computer interface of the visualization module.
Alternatively,
the data task management module 220 further includes:
a non-support feedback unit 226, configured to, if the data task management module determines, according to the probe request, that the type of the data source does not support the tested data, return a non-support test message to the visualization module;
a plug-in obtaining unit 227, configured to upload a plug-in corresponding to the type of the data source to the data task management module after the visualization module receives the test unsupported message;
a plug-in loading unit 228, configured to load the plug-in to the data acquisition adapter module by the task management module;
the data acquisition adaptation module 230 further includes:
a data conversion unit 233, configured to convert, by the data acquisition adaptation module, the data source type corresponding to the plug-in into a general format according to the plug-in, so that the data processing center processes the data source.
If the th determining unit 211 of the data task management module 220 receives the probe request, the th determining unit 211 queries, according to the data source type corresponding to the probe request, that the plug-in corresponding to the data source type is not stored in the data task management module 220, and then the unsupported feedback unit 226 returns an unsupported test message to the visualization module 210.
After the visualization module 210 obtains the test unsupported message, a person skilled in the art encapsulates the data source type into plug-ins, and develops the data source type support, task analysis, task execution, and result delivery steps into plug-in codes in a code manner, where each plug-in has a corresponding interface, such as loading a plug-in, unloading a plug-in, reading probe task configuration, executing a task, and returning task execution result data, and packages the plug-in codes into JAR files, that is, plug-ins corresponding to the probe protocol, and uploads the JAR files to the acquisition plug-in unit 227 of the data task management module 220, where the JAR files (JAR file format is based on popular ZIP file format) and the source code files themselves may exist as plug-in files for scripting language without an additional packaging process.
After receiving the plug-in, the plug-in obtaining unit 227 stores the plug-in through the plug-in loading unit 228 and sends the plug-in to the data obtaining video module 230.
The general format of the system refers to a general format which is manually specified by the system according to the test indexes or learned by using a machine language.
For example, if the index in the probe task is the name of the youngest 10 persons in the data source a, the data acquisition unit 231 acquires the data source a from the database, the data adaptation unit 232 converts the data source a into the common format of the cost system [ [ [ [ 'name', 'age', ], [ 'tension three', 25] ] and then transmits the converted data source a to the data processing center 240, the data processing center 240 determines that the data source a in the common format is out of compliance, deletes the data that is not compliant, if any part is empty, the data is determined to be not compliant, and if the data cannot be calculated to obtain the index, the data is the data that is not compliant.
If the data processing center 240 calculates the data source a to acquire the data source B meeting the compliance of the system after judging and cleaning the data source a, the index of the detection task is calculated according to the data source B, the data processing center 240 calculates required index data in the process of calculating the data source a and generates raw data, which is an index of the viewing times of website videos at time intervals, for example, the viewing times of the 850 website videos at minutes is counted times to obtain the viewing times within minutes, the leader of a company may want to see days of the videos, the data is aggregated, the viewing times of each 38 minutes are accumulated to days, in this case, raw data and metric are , viewing times, the average viewing times are calculated, the average viewing times of each user is calculated by dividing the average viewing times of the data source a into 2 minutes, the average viewing times of each minute, the average viewing times of each user is calculated by dividing the average viewing times of 3510 seconds, the average viewing times of the user is calculated by 636 seconds, the average viewing times of 366 seconds, the average viewing times of the 366 seconds are calculated by the average viewing times of the average times of the viewing times of the theoretical times of theoretical times.
The embodiment of the invention has the following advantages:
the method comprises the steps of checking whether the data task management module of the system supports the type of the data source or not when detecting different types of data sources, if so, establishing a detection task by using the data task management module of the system, and enabling the system to have expansion compatibility for different data source types, so that the problems of high time cost and high labor cost when index detection is needed for different data source types are solved, development labor of different teams for protocol detection is saved, time cost is reduced, meanwhile, plug-ins in the data task management module can be updated based on the system when index detection is needed, the system supports the detection requirement of a new data source type, the plug-ins in the data task management module support the service requirement of protocols of different teams , all parts can perform protocol service detection based on the system, and the whole human resources are saved.
EXAMPLE III
Referring to fig. 3, a flowchart of method steps of the data source scalable method according to embodiments of the present invention is shown, which may specifically include S301-S307:
s301, the visualization module creates a detection request according to the type of the data source and sends the detection request to the data task management module.
After a user confirms the type of a data source to be detected, establishing a detection request in a visual module human-computer interaction interface according to the type of the data source; and after the probe request is successfully created, sending the probe request to the data task management module.
Optionally, the visualization module provides human-machine interface, such as web interface, to the user for the user to create the probe request based on the probed data source.
S302, the data task management module judges whether the type of the data source supports to be tested according to the detection request.
In the embodiment of the present invention, the data task management module searches, according to the data source type, whether a plug-in corresponding to the data source type is stored in a storage unit of the data task management module; if the type support of the data source is tested, S303 is executed.
S303, if the type of the data source supports to be tested, the data task management module creates a probe task.
If the storage unit stores the plug-in corresponding to the data source, a plug-in confirmation command is sent to the visualization module; after the visualization module receives the plug-in confirmation command, a user confirms the index for executing the data source detection and the data source acquisition address according to the type of the data source through the visualization module; and the visualization module sends the type of the data source, the index detected by the data source and the data source acquisition address to the data task management module.
And the data task management module creates a detection task according to the type of the data source, the index detected by the data source and the data source acquisition address.
And S304, the data task management module distributes the detection task to a data acquisition adaptation module.
And the data task management module distributes the detection tasks to corresponding data acquisition adaptation modules.
S305, the data obtaining adaptation module obtains the data source according to the detection task.
After the data acquisition adaptation module receives the detection task, acquiring the data source; if the data source is log data, the data acquisition adaptation module inquires the log and acquires the data source from the log; if the data source is database data, the data acquisition adaptation module acquires the data source from a database; the format of the data source is the format of the data source in the original memory.
And S306, the data acquisition adaptation module converts the data source into a universal format and then sends the universal format to the data processing center.
And after the data acquisition adaptation module acquires the data source, the data source is converted into a universal format and then is sent to the data processing center.
S307, the data processing center calculates the data source with the universal format according to the requirement of the detection task; and sending the calculation result to the visualization module.
The data processing center judges whether the data source in the universal format is in compliance according to the detection task, deletes the data which are judged to be in non-compliance, and calculates indexes of the remaining data which are in compliance; storing the index calculation result and sending the index calculation result to the visualization module; and displaying the index calculation result to a user through a human-computer interface of the visualization module.
The embodiment of the invention has the following advantages:
when detecting different types of data sources, checking whether the data task management module of the system supports the types of the data sources, if so, establishing a detection task by using the data task management module of the system, so that the system has expansion compatibility for different types of data sources; therefore, the problem that time cost and labor cost are high when index detection is needed to be carried out on different data source types is solved, development labor of different teams for protocol detection is saved, and time cost is reduced.
Example four
Referring to fig. 4, a flowchart of method steps of the data source scalable method according to embodiments of the present invention is shown, which may specifically include S401 to S411:
s401, the visualization module creates a detection request according to the type of the data source and sends the detection request to the data task management module.
S402, the data task management module judges whether the type of the data source supports to be tested according to the detection request.
If the type support of the data source is tested, executing S403; if the type of the data source does not support the tested data source, S408 is executed.
And S403, if the type support of the data source is tested, the data task management module creates a detection task.
Optionally, the process of creating a probe task by the data task management module includes: the data task management module returns a message supporting the test to the visualization module; the visualization module sends the test index of the data source to the data task management module; and creating the detection task according to the test indexes of the data source.
The data task management module searches whether a plug-in corresponding to the data source type is stored in a storage unit of the data task management module or not according to the data source type; if the storage unit stores the plug-in corresponding to the data source, a plug-in confirmation command is sent to the visualization module; after the visualization module receives the plug-in confirmation command, a user confirms the index for executing the data source detection and the data source acquisition address according to the type of the data source through the visualization module; and the visualization module sends the type of the data source, the index detected by the data source and the data source acquisition address to the data task management module.
And the data task management module creates a detection task according to the type of the data source, the index detected by the data source and the data source acquisition address.
S404, the data task management module distributes the detection task to a data acquisition adaptation module.
And the data task management module distributes the detection tasks to corresponding data acquisition adaptation modules.
Optionally, the data task management module determines the probe task;
and if the detection task is a timing task, setting a time period and starting the detection task.
For example, if the probe task is a single on-demand data task, the data task management module sends an instruction of a single task to distribute the probe task to the data acquisition adaptation module, and if the data task management module determines that the probe task is a periodic task, the data task management module sets an instruction of periodically starting the probe task, for example, times of starting the probe task every 5 minutes to send an instruction of acquiring the data source every 5 minutes to the data acquisition video module.
S405, the data acquisition adaptation module acquires the data source according to the detection task.
After the data acquisition adaptation module receives the detection task, acquiring the data source; if the data source is log data, the data acquisition adaptation module inquires the log and acquires the data source from the log; if the data source is database data, the data acquisition adaptation module acquires the data source from a database; the format of the data source is the format of the data source in the original memory.
And S406, the data acquisition adaptation module converts the data source into a universal format and then sends the universal format to the data processing center.
And after the data acquisition adaptation module acquires the data source, the data source is converted into a universal format and then is sent to the data processing center.
S407, the data processing center calculates the data source with the universal format according to the requirement of the detection task; and sending the calculation result to the visualization module.
The data processing center judges whether the data source in the universal format is in compliance according to the detection task, deletes the data which are judged to be in non-compliance, and calculates indexes of the remaining data which are in compliance; storing the index calculation result and sending the index calculation result to the visualization module; and displaying the index calculation result to a user through a human-computer interface of the visualization module.
And S408, if the data task management module judges that the type of the data source does not support the tested data according to the detection request, returning a test-not-supporting message to the visualization module.
And if the data task management module receives the detection request, inquiring that the plug-in corresponding to the data source type is not stored in the data task management module according to the data source type corresponding to the detection request, judging that the type of the data source does not support the tested data, and returning a test-not-supporting message to the visualization module.
S409: and after receiving the test-unsupported message, the visualization module uploads a plug-in corresponding to the type of the data source to the data task management module.
After the visualization module obtains the test information which is not supported, technicians in the field package the data source type into plug-ins, develop the data source type support, task analysis, task execution and result delivery steps into plug-in codes in a code mode, wherein each plug-in has a corresponding interface, namely loading plug-ins, unloading plug-ins, reading detection task configuration, executing tasks and returning task execution result data, and package the plug-in codes into JAR files, namely plug-ins corresponding to the detection protocol, and upload the JAR files to an acquisition plug-in unit of the data task management module, wherein the JAR files (the JAR file format is based on a popular ZIP file format) can exist plug-in files for script language, and no additional packaging process is needed.
S410: the task management module loads the plug-in to the data acquisition adaptation module;
in the embodiment of the present invention, after receiving the plug-in, the plug-in obtaining unit of the data task management module stores the plug-in through the plug-in loading unit and sends the plug-in to the data obtaining adaptation module.
S411: and the data acquisition adaptation module converts the data source type corresponding to the plug-in into a universal format according to the plug-in, so that the data processing center processes the data source.
The general format of the system refers to a general format which is manually specified by the system according to the test indexes or learned by using a machine language.
The embodiment of the invention has the following advantages:
the method comprises the steps of checking whether the data task management module of the system supports the type of the data source or not when detecting different types of data sources, if so, establishing a detection task by using the data task management module of the system, and enabling the system to have expansion compatibility for different data source types, so that the problems of high time cost and high labor cost when index detection is needed for different data source types are solved, development labor of different teams for protocol detection is saved, time cost is reduced, meanwhile, plug-ins in the data task management module can be updated based on the system when index detection is needed, the system supports the detection requirement of a new data source type, the plug-ins in the data task management module support the service requirement of protocols of different teams , all parts can perform protocol service detection based on the system, and the whole human resources are saved.
As for the method embodiment, since it is basically similar to the apparatus embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus.
However, it is understood that embodiments of the invention may be practiced without these specific details, and that examples well-known methods, structures, and techniques have not been shown in detail in order not to obscure the understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together by in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of the various inventive aspects, however, the disclosed method is not intended to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.
It will be understood by those skilled in the art that modules in the apparatus of the embodiments may be adaptively changed and arranged in or more apparatuses different from the embodiments, that modules or units or components in the embodiments may be combined into modules or units or components, and further, that they may be divided into sub-modules or sub-units or sub-components, that all features disclosed in this specification (including the accompanying claims, abstract and drawings), and all processes or units of any method or apparatus so disclosed, may be combined in any combination, except at least of such features and/or processes or units are mutually exclusive, unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose.
Furthermore, those of skill in the art will appreciate that while the embodiments described herein include some features included in other embodiments, not others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments.
It should be understood by those skilled in the art that some or all of the functions of or all of the data source scalable system and method according to embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). the present invention may also be embodied as an apparatus or device program (e.g., computer program and computer program product) for performing part or all of the of the method described herein. such program implementing the present invention may be stored on a computer readable medium or may be in the form of or more signals.
The invention may be embodied by means of hardware comprising several distinct elements, and by means of a suitably programmed computer, in a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware, the use of the words , second, third, etc. may indicate any sequence.

Claims (6)

  1. The data source extensible system is characterized by comprising a visualization module, a data task management module, a data processing center and a data acquisition adaptation module;
    the visualization module comprises a request creation unit;
    the request creating unit is used for creating a detection request according to the type of a data source and sending the detection request to the th judging unit of the data task management module;
    the data task management module comprises an th judging unit, a task creating unit, a distributing unit, a non-support feedback unit, a plug-in obtaining unit and a plug-in loading unit;
    the th judging unit is configured to, after receiving the probe request sent by the request creating unit, judge whether the type of the data source supports being tested according to the probe request;
    the task creating unit is configured to create a probe task after the th judging unit judges that the type support of the data source is tested;
    the distribution unit is used for distributing the detection task to the data acquisition unit of the data acquisition adaptation module after the task creation unit creates the detection task;
    the unsupported feedback unit is used for returning an unsupported test message to the visualization module if the data task management module judges that the type of the data source does not support the tested data according to the detection request;
    the plug-in obtaining unit is used for uploading a plug-in corresponding to the type of the data source to the data task management module after the visualization module receives the test unsupported message;
    the plug-in loading unit is used for loading the plug-in to the data acquisition adapting module by the data task management module;
    the data acquisition adaptation module comprises a data acquisition unit, a data adaptation unit and a data conversion unit;
    the data acquisition unit is used for acquiring the data source according to the detection task after receiving the detection task sent by the distribution unit;
    the data adaptation unit is used for converting the data source into a general format and then sending the data source to the data processing center after the data acquisition unit acquires the data source;
    the data conversion unit is used for converting the data source type corresponding to the plug-in into a universal format by the data acquisition adaptation module according to the plug-in, so that the data processing center processes the data source;
    the data processing center is used for calculating the data source with the general format according to the requirement of the detection task after the data adapting unit sends the data source with the general format; and sending the calculation result to the visualization module.
  2. 2. The system of claim 1, wherein the task creation unit comprises:
    the index receiving subunit is used for receiving the test index of the data source sent by the visualization module;
    and the task creating subunit is used for creating the detection task according to the test index of the data source.
  3. 3. The system of claim 1, wherein the data task management module further comprises:
    the second judging unit is used for judging the detection task;
    and the period starting unit is used for setting a time period and starting the detection task if the detection task is a timing task.
  4. The data source extensible method of , comprising:
    the visualization module creates a detection request according to the type of the data source and sends the detection request to the data task management module;
    the data task management module judges whether the type of the data source supports to be tested or not according to the detection request;
    if the type of the data source supports to be tested, the data task management module creates a detection task;
    the data task management module distributes the detection task to a data acquisition adaptation module;
    the data acquisition adaptation module acquires the data source according to the detection task;
    the data acquisition adaptation module converts the data source into a universal format and then sends the universal format to the data processing center;
    the data processing center calculates the data source with the general format according to the requirement of the detection task; and sending the calculation result to the visualization module;
    if the data task management module judges that the type of the data source does not support the tested data according to the detection request, returning a test-not-supporting message to the visualization module;
    after receiving the test unsupported message, the visualization module uploads a plug-in corresponding to the type of the data source to the data task management module;
    the data task management module loads the plug-in to the data acquisition adaptation module;
    and the data acquisition adaptation module converts the data source type corresponding to the plug-in into a universal format according to the plug-in, so that the data processing center processes the data source.
  5. 5. The method of claim 4, wherein the step of creating a probe task if the type of the data source supports being tested comprises:
    the data task management module returns a message supporting the test to the visualization module;
    the visualization module sends the test index of the data source to the data task management module;
    and creating the detection task according to the test index of the data source.
  6. 6. The method of claim 4, wherein after the step of the data task management module distributing the probing task to a data acquisition adaptation module, the method further comprises:
    the data task management module judges the detection task;
    and if the detection task is a timing task, setting a time period and starting the detection task.
CN201710392604.9A 2017-05-27 2017-05-27 data source extensible system and method Active CN107317722B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710392604.9A CN107317722B (en) 2017-05-27 2017-05-27 data source extensible system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710392604.9A CN107317722B (en) 2017-05-27 2017-05-27 data source extensible system and method

Publications (2)

Publication Number Publication Date
CN107317722A CN107317722A (en) 2017-11-03
CN107317722B true CN107317722B (en) 2020-01-31

Family

ID=60181527

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710392604.9A Active CN107317722B (en) 2017-05-27 2017-05-27 data source extensible system and method

Country Status (1)

Country Link
CN (1) CN107317722B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107332730B (en) * 2017-06-19 2020-04-21 北京奇艺世纪科技有限公司 Protocol extensible service availability detection system and method
CN109995703B (en) * 2017-12-29 2021-08-13 中国移动通信集团云南有限公司 Data source security inspection method and edge server
CN111600771B (en) * 2020-04-14 2022-03-08 新浪网技术(中国)有限公司 Network resource detection system and method

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102231686A (en) * 2011-06-24 2011-11-02 北京天融信科技有限公司 System and method for implementing automated test of network security equipment
CN103036736A (en) * 2012-11-30 2013-04-10 航天恒星科技有限公司 Configuration equipment monitoring system and monitoring method based on data sources
CN103279336A (en) * 2013-01-06 2013-09-04 北京慧正通软科技有限公司 Workflow engine multi-data source processing method
CN103902286A (en) * 2014-03-12 2014-07-02 郑州轻工业学院 Hierarchy type multi-source data fusion method based on SOA
CN105243333A (en) * 2015-08-28 2016-01-13 苏州国云数据科技有限公司 Multi-data-source remote access method
CN105260177A (en) * 2015-09-21 2016-01-20 广东大工数值仿真研究院有限公司 SiPESC platform based Python extension module development method
CN105487867A (en) * 2015-11-26 2016-04-13 中国空间技术研究院 Lightweight visual satellite testing program design system and method
CN105512168A (en) * 2015-11-16 2016-04-20 天津南大通用数据技术股份有限公司 Cluster database composite data loading method and apparatus

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102231686A (en) * 2011-06-24 2011-11-02 北京天融信科技有限公司 System and method for implementing automated test of network security equipment
CN103036736A (en) * 2012-11-30 2013-04-10 航天恒星科技有限公司 Configuration equipment monitoring system and monitoring method based on data sources
CN103036736B (en) * 2012-11-30 2015-09-23 航天恒星科技有限公司 A kind of configuration equipment monitoring system based on data source and method
CN103279336A (en) * 2013-01-06 2013-09-04 北京慧正通软科技有限公司 Workflow engine multi-data source processing method
CN103902286A (en) * 2014-03-12 2014-07-02 郑州轻工业学院 Hierarchy type multi-source data fusion method based on SOA
CN105243333A (en) * 2015-08-28 2016-01-13 苏州国云数据科技有限公司 Multi-data-source remote access method
CN105260177A (en) * 2015-09-21 2016-01-20 广东大工数值仿真研究院有限公司 SiPESC platform based Python extension module development method
CN105512168A (en) * 2015-11-16 2016-04-20 天津南大通用数据技术股份有限公司 Cluster database composite data loading method and apparatus
CN105487867A (en) * 2015-11-26 2016-04-13 中国空间技术研究院 Lightweight visual satellite testing program design system and method

Also Published As

Publication number Publication date
CN107317722A (en) 2017-11-03

Similar Documents

Publication Publication Date Title
CN108521353B (en) Processing method and device for positioning performance bottleneck and readable storage medium
CN107688530B (en) Software testing method and device
US11023358B2 (en) Review process for evaluating changes to target code for a software-based product
CN111049705A (en) Method and device for monitoring distributed storage system
CN108959059B (en) Test method and test platform
CN107317722B (en) data source extensible system and method
US20030115511A1 (en) Method, apparatus and program for diagnosing system risk
CN105787364B (en) Automatic testing method, device and system for tasks
US8046638B2 (en) Testing of distributed systems
CN111078567B (en) Report generation method, terminal and storage medium of automatic test platform
CN105607994A (en) Mobile terminal software testing method and system
CN110881009B (en) Method, device, communication equipment and storage medium for receiving test message
CN109408375A (en) The generation method and device of interface document
KR20070080313A (en) Method and system for analyzing performance of providing services to client terminal
CN112333249A (en) Business service system and method
WO2020172569A1 (en) Method, apparatus, and computer-readable medium for maintaining visual consistency
CN117608825A (en) Resource management method based on multi-cloud management platform and related equipment
CN111290951A (en) Test method, terminal, server, system and storage medium
CN107451056B (en) Method and device for monitoring interface test result
CN113807713A (en) Product quality evaluation method and industrial internet identification analysis system
CN107332730B (en) Protocol extensible service availability detection system and method
WO2014085792A1 (en) Systems and methods of assessing software quality for hardware devices
CN111506769A (en) Video file processing method and device, storage medium and electronic device
CN112583660B (en) Main domain and standby domain test comparison method, device and system of recommendation platform
CN111314743B (en) Interface data playback method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant