CN105847446B

CN105847446B - Method, device and system for acquiring network data

Info

Publication number: CN105847446B
Application number: CN201610380227.2A
Authority: CN
Inventors: 林欣烁; 蔡晓鹏; 许泽伟
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2016-05-31
Filing date: 2016-05-31
Publication date: 2020-08-04
Anticipated expiration: 2036-05-31
Also published as: CN105847446A

Abstract

The embodiment of the invention discloses a method, a device and a system for acquiring network data; the embodiment of the invention adopts the steps of acquiring configuration files at fixed time, loading corresponding dynamic library files according to the configuration files, calling corresponding plug-in functions, acquiring parameters related to the plug-in functions from the configuration files, and then executing the plug-in functions according to the parameters to acquire network data; the scheme is simple to realize and update, is easy to expand functions, and can reduce resource consumption.

Description

Method, device and system for acquiring network data

Technical Field

The invention relates to the technical field of communication, in particular to a method, a device and a system for filtering browser page data.

Background

With the development of internet technology, more and more people choose to acquire various information required by themselves through a network, including some electronic readings, such as electronic books, electronic periodicals or electronic newspapers, and the like. The electronic reading material is a publication or non-publication that digitalizes the content of information such as characters, pictures, sounds, and images. Meanwhile, in order to facilitate the user to obtain the electronic reading materials, a batch of terminal applications related to reading, which are referred to as reading applications for short, are also correspondingly released in the prior art.

Data acquired by a user through reading an application can be generally divided into user-related data and user-unrelated data, the user-related data refers to personalized data of the user, such as VIP (VIP) level of the user, service content subscribed by the user, bookmarks of the user, favorites of the user, and the like, and the user-unrelated data refers to data that can be commonly used by all users, such as configuration information of some reading interfaces, illustrations on pages, specific content of electronic readings, and the like. In the prior art, generally, a server to which a reading application belongs acquires user-independent data in batches in a manner of pulling by a timed daemon process (daemon), processes the user-independent data, and stores the processed user-independent data in the local.

In the research and practice process of the prior art, the inventor of the present invention finds that, due to differences of the modules, in the existing scheme, a corresponding daemon process needs to be independently written for each module, and if a new module is added or the function is updated, the corresponding daemon process needs to be reprogrammed and integrated again, that is, the existing scheme adopts a hard coding mode, so that the realization and the update are complex, the function expansion is not easy, and resources are wasted.

Disclosure of Invention

Embodiments of the present invention provide a method, an apparatus, and a system for acquiring network data, which are not only simple to implement and update, and easy to perform function expansion, but also can reduce resource consumption.

The embodiment of the invention provides a method for acquiring network data, which comprises the following steps:

acquiring a configuration file at regular time;

loading a corresponding dynamic library file according to the configuration file, and calling a corresponding plug-in function;

acquiring parameters related to the plug-in function from the configuration file;

and executing the plug-in function according to the parameters to acquire network data.

Correspondingly, an embodiment of the present invention further provides an apparatus for acquiring network data, including:

the file acquisition unit is used for acquiring the configuration file at regular time;

the calling unit is used for loading a corresponding dynamic library file according to the configuration file and calling a corresponding plug-in function;

the parameter acquisition unit is used for acquiring parameters related to the plug-in function from the configuration file;

and the execution unit is used for executing the plug-in function according to the parameters so as to acquire network data.

In addition, the embodiment of the invention also provides a system for acquiring network data, which comprises any one of the devices for acquiring network data provided by the embodiment of the invention.

The embodiment of the invention adopts the steps of acquiring configuration files at fixed time, loading corresponding dynamic library files according to the configuration files, calling corresponding plug-in functions, acquiring parameters related to the plug-in functions from the configuration files, and then executing the plug-in functions according to the parameters to acquire network data; because the scheme can embody some logic processing operations in the process of acquiring network data in a dynamic library file and plug-in function mode, for some basic logic functions shared by different modules, such as acquisition of user-independent data and the like, only the same dynamic library file and plug-in function need to be shared, and a set of corresponding complete codes do not need to be independently written for each module, so that the method is simpler to realize and is convenient for updating and function expansion, for example, when the function expansion is carried out, only the logic development of the expanded function needs to be concentrated, the basic logic function does not need to be concerned, and the like, and the flexibility of the method is greatly improved; furthermore, since the codes realizing the same function can be shared, a large number of repeated codes can be reduced, and the consumption of resources can be reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1a is a schematic scene diagram of a method for acquiring network data according to an embodiment of the present invention;

fig. 1b is a flowchart of a method for acquiring network data according to an embodiment of the present invention;

fig. 2 is another flowchart of a method for acquiring network data according to an embodiment of the present invention;

fig. 3a is a schematic structural diagram of an apparatus for acquiring network data according to an embodiment of the present invention;

fig. 3b is another schematic structural diagram of an apparatus for acquiring network data according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a server according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment of the invention provides a method, a device and a system for acquiring network data.

The network data acquisition system may include any network data acquisition device provided in the embodiments of the present invention, and the network data acquisition device may be specifically integrated in a server.

For example, referring to fig. 1a, the server may periodically obtain a configuration file, load a corresponding dynamic library file, such as a so file, according to the configuration file, and call a corresponding plug-in function, and obtain parameters related to the plug-in function from the configuration file, and then execute the plug-in function according to the parameters to obtain network data; for example, the parameters may be added to a parameter list in sequence, then the parameters in the parameter list are sequentially input to the plug-in function, the plug-in function with the input parameters is executed to call a corresponding external interface, the original network data is pulled from the server through the external interface, and the data processing is performed on the original network data to obtain network data, and so on; thereafter, when the terminal needs to acquire the network data, for example, when the server receives a data acquisition request sent by the terminal, the obtained network data may be provided to the terminal.

The following are detailed below. The numbers in the following examples are not intended to limit the order of preference of the examples.

The first embodiment,

The present embodiment will be described from the perspective of a network data acquisition device, which may be specifically integrated in a server, such as an application server.

A method for acquiring network data comprises the following steps: acquiring a configuration file at regular time; loading a corresponding dynamic library file according to the configuration file, and calling a corresponding plug-in function; acquiring parameters related to the plug-in function from the configuration file; and executing the plug-in function according to the parameters to acquire network data.

As shown in fig. 1b, a specific flow of the network data obtaining method may be as follows:

101. and acquiring the configuration file at fixed time.

For example, an acquisition period may be specifically set, and then the configuration file is periodically acquired according to the period. The period may be set according to the requirements of the practical application, for example, the period may be set to be acquired every 30 seconds, or acquired every 20 seconds, and so on.

The format of the configuration file may be determined according to the requirements of the actual application, for example, the configuration file may be in html or xml format, and the like. For convenience of description, in the embodiments of the present invention, the configuration file is specifically described as an example in an xml format.

When acquiring the configuration file, the configuration file may be acquired from another device, such as a management server or a storage server, or the configuration file may be loaded locally in advance, and the configuration file is acquired locally, that is, before the step "acquiring the configuration file periodically", the method for acquiring the network data may further include:

and when the current operating environment is determined to meet the preset conditions, loading the configuration file.

The current operating environment may include operating time information, and/or state information of each software and hardware, and the preset condition may be set according to the requirement of the actual application, for example, the configuration file may be loaded once every certain period; for another example, it may also be configured to load the configuration file when a process is in a certain state, and so on, which is not described herein again.

Optionally, in order to save the signaling flow and provide processing efficiency, when loading the configuration file, it may be further determined whether the configuration file is changed, and if the configuration file is changed, the configuration file is reloaded, and if the configuration file is not changed, the configuration file loaded last time may be used.

The method for determining whether the configuration file has changed may be in various manners, and besides directly comparing the current configuration file with the configuration file loaded last time, the method may also determine by the modification time of the configuration file, that is, after the step "loading the configuration file", the method for acquiring the network data may further include:

and when the last modification time of the configuration file is determined to be inconsistent with the modification time of the last loading, reloading the configuration file.

Otherwise, if the last modification time of the configuration file is determined to be consistent with the modification time of the last loading, the loaded configuration file is indicated to be unchanged, and the loaded configuration file is the current latest version and can be continuously used.

For example, it may be specifically determined whether the current last modification time of the configuration file is consistent with the modification time of the last loading by reading file state information of the configuration file, such as stat (file state) information, and the like.

102. Loading a corresponding dynamic library file according to the configuration file, and calling a corresponding plug-in function; for example, the following may be specifically mentioned:

reading the daemon process (such as daemon) task in the configuration file to determine the daemon process task which needs to be executed currently, loading the corresponding dynamic library file according to the daemon process task which needs to be executed currently, and calling the corresponding plug-in function.

For example, the identifier of the corresponding dynamic library file and the identifier of the plug-in function may be obtained from the daemon task that needs to be executed currently, the corresponding dynamic library file is called according to the identifier of the dynamic library file, such as the so file, is loaded, and then the corresponding plug-in function is called from the dynamic library file according to the identifier of the plug-in function, and so on.

Among them, the daemon process is a special process running in the background, which is independent of the control terminal and periodically performs some kind of task or waits to process some occurred events.

The identifier of the dynamic library file may include the name or number of the dynamic library file, and the like, which may identify the information of the dynamic library file; the identification of the plug-in function may include information that can identify the plug-in function, such as the name or number of the plug-in function.

For example, if the identifier of the dynamic library file is specifically the name of the dynamic library file, and the identifier of the plug-in function is specifically the name of the plug-in function, the content of the configuration file is as follows:

the handle node represents a plug-in function, and in the above example of the configuration file, specifically refers to a GetColumnBook function (i.e., a plug-in function) in a libdaemon. And the task node represents a daemon (daemon) task to be executed.

Then, after the acquiring device of the network data reads the task record, a handler named "GetColumnBook" is found, and then, according to the information therein, a corresponding dynamic library file is loaded, that is, "libdaemon.

It should be noted that, because the daemon process has less execution times, the execution condition cannot be monitored through module scheduling reporting, therefore, once the process is dead or the access to the external interface fails, the timeliness problem of the external network data is easily caused because the process cannot be processed in time, therefore, in order to avoid the timeliness problem of the extranet data caused by the failure of timely discovering the process death or the failure of accessing the external interface, alternatively, the detection of the property information may be performed each time the logical processing entry of the daemon is triggered, e.g. before the configuration file is acquired, wherein the detection of the characteristic information is similar to a heartbeat mechanism, and is used for checking the survival condition of a daemon process such as a daemon process, whether the daemon process is abnormal or not can be judged based on the characteristic information, and if the daemon process is abnormal, an alarm can be given out so as to inform related personnel of timely processing. That is, optionally, the method for acquiring network data may further include:

acquiring characteristic information of a logic processing entry of the daemon process, and generating alarm information when determining that the daemon process is abnormal according to the characteristic information.

The format of the characteristic information may be specifically determined according to the requirements of the actual application, for example, a numerical characteristic may be used to represent the characteristic information (that is, the characteristic information is represented by the height of a numerical value), meanwhile, an alarm policy that decreases proportionally may be set in a server (such as a network management system), if the numerical value decreases, it indicates that the running condition of the daemon process is abnormal, such as a process is dead or an external interface is failed to access, and the like, and if an alarm information needs to be sent, at this time, the alarm information may be sent to a preset terminal.

The warning information sending channel can comprise mails, short messages and/or messages of a social platform or an instant messaging platform.

103. And acquiring parameters related to the plug-in function from the configuration file.

The parameters related to the plug-in function may include parameters related to data storage, parameters for distinguishing whether the task is a critical task, parameters indicating a message type, and the like. For example, take the following plug-in functions as an example:

the parameters related to the plug-in function may include appid, netid, critical, errmsg, and va _ list, etc., where appid and netid are data storage related parameters; the critical parameter is used for distinguishing whether the task is a key task; the errmsg is a return message, and if the function return value is not 0, the return message is reported by character string type characteristic information; the parameter va _ list expressed in the parameter variation macro mode of the C language is set for processing the problem that the types and the number of parameters required by the plug-in function are different in different service scenes.

104. And executing the plug-in function according to the parameters to acquire network data.

For example, the parameters may be added to a parameter list in sequence, and then the plug-in function is executed according to the parameter list to obtain the network data.

The plug-in function is executed according to the parameter list, so as to obtain network data in various ways, for example, as follows:

and sequentially inputting the parameters in the parameter list into the plug-in function, and executing the plug-in function with the input parameters to acquire the network data.

For example, taking the configuration file illustrated in step 102 as an example, in addition to the "type" parameter ("type" is obtained from the attribute of the task node), other parameters may be sequentially introduced into the plug-in functions in the order of appearance, that is, "1", "2477", and "20" may be sequentially introduced. The type of the parameter may be predefined, for example, it may be unified as char type, and so on.

Wherein, the parameters in the parameter list are sequentially input to the plug-in function, that is, the key code (32-bit machine) of the function parameter stack push operation may specifically be as follows:

after the parameters are input into the plug-in function, the plug-in function with the input parameters can be used for calling a corresponding external interface, then, the original network data is pulled through the external interface, and the data processing is carried out on the original network data, so that the network data is obtained.

Optionally, to avoid the timeliness problem of the extranet data caused by the failure to find the process abnormality in time, when the data processing fails, an alarm may be further performed, that is, after the step "performing data processing on the original network data to obtain the network data", the method for obtaining the network data may further include:

if the data processing fails, generating first failure prompt information, for example, generating character string type first failure prompt information, and giving an alarm according to the first failure prompt information.

For example, if the data processing fails, the plug-in function returns a value indicating an error, such as a value other than 0 (for example, 0 indicates correct, and other than 0 indicates error), at this time, the plug-in function may determine that the data processing fails according to the value, and then generate a first failure prompt message, and perform an alarm according to the first failure prompt message, and so on.

Optionally, after the network data is acquired, the network data may be written into a preset storage space for storage, that is, after the step "data processing is performed on the original network data to obtain the network data", the method for acquiring the network data may further include:

and writing the network data into a preset storage space.

Similarly, in order to avoid the timeliness problem of the extranet data caused by the fact that the process abnormality cannot be found in time, when the writing fails, the warning can be given as follows:

if the writing fails, second failure prompt information is generated, for example, character string type second failure prompt information is generated, and an alarm is given according to the second failure prompt information.

For example, if the writing fails, the plug-in function returns a value indicating an error, such as a value other than 0 (for example, 0 indicates correct, and other than 0 indicates error), and then the writing failure may be determined according to the value, and an alarm may be issued.

The alarm may be in various manners, such as a reminder in the form of text, sound, or picture, and the like.

It should be noted that, when the first failure prompt message or the second failure prompt message is generated, the first failure prompt message or the second failure prompt message may also carry a corresponding plug-in name and/or function name, so that the location where the transmission exception is sent may be quickly located.

As can be seen from the above, in the embodiment, the configuration file is obtained at regular time, the corresponding dynamic library file is loaded according to the configuration file, the corresponding plug-in function is called, the parameters related to the plug-in function are obtained from the configuration file, and then the plug-in function is executed according to the parameters to obtain the network data; because the scheme can embody some logic processing operations in the process of acquiring network data in a dynamic library file and plug-in function mode, for some basic logic functions shared by different modules, such as acquisition of user-independent data and the like, only the same dynamic library file and plug-in function need to be shared, and a set of corresponding complete codes do not need to be independently written for each module, so that the method is simpler to realize and is convenient for updating and function expansion, for example, when function expansion is carried out, only the logic development of the expanded function needs to be concentrated, and the basic logic function does not need to be concerned (the existing development of new daemons needs to redevelop the basic logic function of daemon, the workload is repeated), and the like, and the flexibility of the method is greatly improved; furthermore, since the codes realizing the same function can be shared, a large number of repeated codes can be reduced, and the consumption of resources can be reduced.

Example II,

The method described in the first embodiment is further illustrated by way of example.

In this embodiment, the network data is specifically electronic reading information, and the device for acquiring the network data is specifically integrated in the server.

As shown in fig. 2, a specific process of the method for acquiring network data may be as follows:

201. and when the server determines that the current operating environment meets the preset conditions, loading the configuration file.

The current operating environment may include operating time information and/or state information of each software and hardware, and the preset condition may be set according to the requirement of the actual application.

Optionally, in order to save a signaling flow and provide processing efficiency, when loading a configuration file, it may further be determined whether the configuration file is changed, and if the configuration file is changed, the configuration file is reloaded, and if no change is sent, the configuration file loaded last time may be used, for example, as follows:

the server reads file state information of the configuration file, such as stat (file state) information, determines whether the last modification time of the configuration file is consistent with the modification time of the last loading according to the file state information, if so, the configuration file loaded last can be continuously used, otherwise, the configuration file is reloaded.

Optionally, to avoid a timeliness problem of the extranet data caused by failing to find out that the process dies or fails to access the external interface in time, the detection of the characteristic information may be performed once each time the logic processing entry of the daemon process is triggered (for example, before the configuration file is acquired), for example, the characteristic information of the logic processing entry of the daemon process may be specifically acquired, and then the characteristic information is reported to the corresponding detection module in the server.

The characteristic information report is similar to a heartbeat mechanism and is used for checking the survival condition of a daemon process (such as daemon), then the server judges whether the daemon process is abnormal or not based on the characteristic information, and if the daemon process is abnormal, an alarm can be sent out so as to inform related personnel of timely processing.

It should be noted that the reporting of the property information does not affect the loading of the configuration file, that is, step 201 can be executed regardless of whether the daemon process is abnormal or not.

202. And the server acquires the loaded configuration file at regular time.

For example, taking the case that the server includes a free book module and a ranking list module, and at this time, the relevant electronic reading information of the free book module and the ranking list module needs to be acquired, the corresponding configuration file may be set as:

the handle node represents a plug-in function, and in the above example of the configuration file, the handle node specifically refers to a GetFreeBook function in libfree. The task node represents daemon process (daemon) tasks to be executed, namely, handles named ' GetFreeBook ' and ' getankbook ' need to be found, corresponding dynamic library files and plug-in functions are loaded according to information in the tasks, namely, a ' libfree.

So is used for the free book module, and so is used for the leader board data processing.

203. The server reads the daemon process (such as daemon) task in the configuration file to determine the daemon process task which needs to be executed currently.

For example, taking the configuration file illustrated in step 202 as an example, at this time, it may be determined that the daemon task that needs to be executed currently is:

according to the codes, the daemon task needing to be executed currently indicates that a handle named as ' GetFreeBook ' needs to be found, and a corresponding dynamic library file and a plug-in function are loaded according to information in the daemon task, namely the function ' GetFreebook ' in ' libfree.

For another example, if the daemon task that needs to be executed currently is:

then, according to the codes, the daemon task that needs to be executed currently indicates that a handle named "GetRankBook" needs to be found, and a corresponding dynamic library file and a plug-in function are loaded according to the information in the daemon task, that is, a function "getrankbok" in "librank.

204. And the server loads the corresponding dynamic library file according to the daemon task needing to be executed currently and calls the corresponding plug-in function.

For example, if in step 203, the daemon task that needs to be executed currently indicates that a handle named "GetFreeBook" needs to be found, at this time, a corresponding dynamic library file "libfree.

For another example, if the daemon task that needs to be executed currently indicates that a handle named "getankbook" needs to be found in step 203, at this time, a corresponding dynamic library file "library.so" may be loaded according to information in the handle, and an add-in function named "getankbook" may be found from the dynamic library file "library.so".

205. And the server acquires parameters related to the plug-in function from the configuration file and adds the parameters to the parameter list in sequence.

For example, if the called plug-in function is GetFreeBook in step 204, when the plug-in function "GetFreeBook" is executed, the values of the various parameters, such as "critical", "plat", "columned", and "bloken", in the daemon task "task type" ("GetFreeBook" critical "(" true "plat" ("1") column id "(" 2477 ") and" bloken "(" 20 "/>") that needs to be executed currently may be added as parameters to the parameter list corresponding to the plug-in function "GetFreeBook" in order.

The parameter "column" is a number of a column of a free book (i.e., a free electronic reading), and the parameter "book" indicates the number of electronic readings that need to be pulled, for example, the number of book "20" indicates the number of electronic readings that need to be pulled is 20, and so on.

For another example, if the called plug-in function is "getrankbok" in step 204, when the plug-in function "getrankbok" is executed, the values of the parameters in the daemon task "type" ("getrankbok" critical "(" true "place" ("1" ran kid "(" 2478 ") and" book "(" 100 "/>") that need to be executed currently, such as "critical", "place", "book", and "book", may be added as parameters to the parameter list corresponding to the function "getrankbok" in order.

The parameter "rankid" represents the number of the ranking list, and "bookmark" represents how many electronic readings before the ranking need to be pulled, for example, the bookmark is "100", which represents the information of the electronic readings 100 before the ranking need to be pulled, and so on.

206. The server inputs the parameters in the parameter list into the plug-in function in sequence and executes the parameters to acquire the electronic reading information, and then returns to execute step 203.

For example, taking an example that the current daemon task to be executed is "GetFreeBook" critic "true" place "1" column "═ 2477" book num "20"/> ", at this time," 1"," 2477", and" 20 "may be sequentially transferred to the plug-in function" GetFreeBook "and executed, for example, a corresponding external interface may be called by using the plug-in function with parameters input, and then the original electronic reader information may be pulled through the external interface and subjected to data processing to obtain the electronic reader information.

For example, if the daemon task that needs to be executed at present is "get type" ("get" critical "(" 1"ran kid" ("2478") book "(" 100 "/"), at this time, "1", "2478", and "100" may be sequentially transferred to the plug-in function "get" book "and executed, for example, a corresponding external interface may be called by using the plug-in function into which parameters are input, and then the original electronic reader information may be pulled through the external interface and subjected to data processing to obtain the electronic reader information.

Optionally, to avoid the timeliness problem of the extranet data caused by the failure to find the process abnormality in time, when the data processing fails, an alarm may be given, that is, after the step "data processing is performed on the original electronic reading information to obtain the electronic reading information", the method for acquiring the network data may further include:

and if the data processing fails, generating character string type first failure prompt information, and giving an alarm according to the first failure prompt information.

For example, if the data processing fails, the plug-in function returns a value indicating an error, such as a value other than 0 (for example, 0 indicates correct, and other than 0 indicates error), and then the server may determine that the data processing fails according to the value, and then perform an alarm, and so on.

Optionally, after the electronic reading material information is acquired, the electronic reading material information can be written into a preset storage space for storage; similarly, when the writing fails, the method may also perform an alarm, that is, after the step "perform data processing on the original electronic reading information to obtain the electronic reading information", the method for acquiring the network data may further include:

and writing the electronic reading material information into a preset storage space, if the writing fails, generating second failure prompt information of a character string type, and giving an alarm according to the second failure prompt information.

For example, if the writing fails, the plug-in function returns a value indicating an error, such as a value other than 0 (for example, 0 indicates correct, and other than 0 indicates error), and then the server may determine that the writing fails according to the value, and then perform an alarm, and so on.

It should be noted that, when the first failure prompt message or the second failure prompt message is returned, the first failure prompt message or the second failure prompt message may also carry a corresponding plug-in name and/or function name, so that the location where the transmission exception is sent may be quickly located.

It should be noted that, regardless of the execution condition of the plug-in function of the first daemon task, the next daemon task needs to be executed continuously, that is, regardless of whether the plug-in function of the first "task" returns a return value indicating success, the next "task" continues to be executed, that is, the step 203 needs to be executed again.

For example, if the first daemon task is "get free" type "critical" place "1" column ═ 2477 "bookmark ═ 20"/>, then the second daemon task may be executed continuously, that is, "get free" critical "place ═" true "1" rand ═ "2478" bookmark ═ "100"/>, as the daemon task that needs to be executed currently, see step 203 for details.

It should be noted that after all the daemon tasks (i.e., tasks) are executed, the server may sleep for a specified time, and then re-execute step 201, and this is repeated in a loop, for example, 2 tasks in this embodiment, so that after the two tasks are executed, the server may sleep for the specified time, then re-execute step 201, so as to execute other tasks, and so on.

It should be further noted that, in order to improve the processing efficiency, optionally, a multi-thread processing mode may also be started to improve the parallel processing capability, that is, multiple threads may be adopted to process multiple daemon tasks in parallel, where the execution of each daemon task is similar to the above process, and is not described here again.

As can be seen from the above, in the embodiment, the configuration file is obtained at regular time, the corresponding dynamic library file is loaded according to the configuration file, the corresponding plug-in function is called, the parameters related to the plug-in function are obtained from the configuration file, and then the parameters are sequentially transmitted to the plug-in function and executed to obtain the electronic reading information; because the scheme can embody some logic processing operations in the process of acquiring the electronic reading information in a dynamic library file and plug-in function mode, for some basic logic functions shared by different modules, such as acquisition of user-independent data and the like, only the same dynamic library file and plug-in function need to be shared, and a set of corresponding complete codes do not need to be independently written for each module, so that the realization is simpler, and the updating and function expansion are facilitated; furthermore, since the codes realizing the same function can be shared, a large number of repeated codes can be reduced, and the consumption of resources can be reduced.

Furthermore, the characteristic information of the logic processing entry of the daemon process can be obtained at regular time and reported to the server, and when other processing abnormity is found, such as data processing failure or data writing failure, an alarm is given, so that the problem of timeliness of the extranet data caused by incapability of timely finding out that the process is dead or the external interface is failed to access can be avoided, and the data processing effect is greatly improved.

Example III,

In order to better implement the above method, an embodiment of the present invention further provides an apparatus for acquiring network data, as shown in fig. 3a, the apparatus for acquiring network data includes a file acquiring unit 301, a calling unit, a parameter acquiring unit 303, and an executing unit 304, as follows:

(1) a file acquisition unit 301;

a file acquiring unit 301, configured to acquire the configuration file at regular time.

For example, an acquisition period may be specifically set, and then the configuration file is periodically acquired according to the period. Wherein, the period can be set according to the requirement of practical application.

The format of the configuration file may be determined according to the requirements of the actual application, for example, an xml format may be adopted.

When acquiring the configuration file, the configuration file may be acquired from another device, or the configuration file may be loaded locally in advance, and the configuration file is acquired locally, that is, as shown in fig. 3b, the apparatus for acquiring network data may further include a loading unit 305, as follows:

the loading unit 305 may be configured to load the configuration file when it is determined that the current operating environment meets the preset condition.

The manner of determining whether the configuration file has changed may be various, and besides the current configuration file may be directly compared with the configuration file loaded last time, the determination may also be performed according to the modification time of the configuration file, that is:

the loading unit 305 may be further configured to, after the configuration file is loaded, reload the configuration file if it is determined that the last modification time of the configuration file is inconsistent with the modification time of the last loading.

(2) A calling unit 302;

a calling unit 302, configured to load a corresponding dynamic library file according to the configuration file, and call a corresponding plug-in function;

for example, the call unit 302 may include a determination subunit and a call subunit, as follows:

and the determining subunit is used for reading the daemon task in the configuration file so as to determine the daemon task which needs to be executed currently.

And the calling subunit is used for loading the corresponding dynamic library file according to the daemon task which needs to be executed currently and calling the corresponding plug-in function.

For example, the calling subunit may be specifically configured to obtain an identifier of a corresponding dynamic library file and an identifier of a plug-in function from the daemon task that needs to be executed currently, call the corresponding dynamic library file according to the identifier of the dynamic library file, load the dynamic library file, such as a so file, and then call the corresponding plug-in function from the dynamic library file according to the identifier of the plug-in function, and so on.

(3) A parameter acquisition unit 303;

a parameter obtaining unit 303, configured to obtain a parameter related to the plug-in function from the configuration file.

The parameters related to the plug-in function may include parameters related to data storage, parameters for distinguishing whether the task is a critical task, parameters indicating a message type, and the like.

(4) An execution unit 304;

an executing unit 304, configured to execute the plug-in function according to the parameter, so as to obtain network data.

For example, the execution unit 304 may include an add subunit and an execute subunit, as follows:

the adding subunit is used for adding the parameters to the parameter list in sequence;

and the execution subunit is used for executing the plug-in function according to the parameter list so as to acquire the network data.

For example, the execution subunit may be specifically configured to sequentially input the parameters in the parameter list into the plug-in function, and execute the plug-in function with the input parameters, so as to obtain the network data.

After the parameters are input into the plug-in function, the execution subunit can call a corresponding external interface by using the plug-in function with the input parameters, then, the original network data is pulled through the external interface, and the data processing is carried out on the original network data to obtain the network data.

Optionally, in order to avoid the timeliness problem of the extranet data caused by the fact that the process abnormality cannot be found in time, when the data processing fails, an alarm may be given, that is:

the execution subunit is further configured to generate first failure prompt information when the data processing fails, and perform an alarm according to the first failure prompt information.

Optionally, after the network data is acquired, the network data may be written into a preset storage space for storage, that is, the execution unit 304 may further include a storage subunit, as follows:

the storage subunit is configured to write the network data into a preset storage space, generate a second failure prompt message if the writing fails, and perform an alarm according to the second failure prompt message.

Optionally, to avoid a time-dependent problem of the extranet data caused by failing to find out that the process is dead or failing to access the external interface in time, the detection of the characteristic information may be performed once each time the logic processing entry of the daemon process is triggered (for example, before the configuration file is acquired), that is, as shown in fig. 3b, the apparatus for acquiring the network data may further include an alarm unit 306, as follows:

the alarm unit 306 may be configured to obtain characteristic information of a logic processing entry of the daemon process, and generate alarm information when it is determined that the daemon process is abnormal according to the characteristic information.

The alarm unit 306 may also be configured to send the alarm information to a preset terminal, such as a terminal to which a relevant maintenance worker belongs.

In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.

The network data obtaining device may be specifically integrated in a server, such as an application server.

As can be seen from the above, the file obtaining unit 301 of the network data obtaining apparatus of this embodiment may obtain the configuration file at regular time, then the calling unit 302 loads the corresponding dynamic library file according to the configuration file, and calls the corresponding plug-in function, and the parameter obtaining unit 303 obtains the parameter related to the plug-in function from the configuration file, and then the executing unit 304 executes the plug-in function according to the parameter to obtain the network data; because the scheme can embody some logic processing operations in the process of acquiring network data in a dynamic library file and plug-in function mode, for some basic logic functions shared by different modules, such as acquisition of user-independent data and the like, only the same dynamic library file and plug-in function need to be shared, and a set of corresponding complete codes do not need to be independently written for each module, so that the method is simpler to realize and is convenient for updating and function expansion, for example, when the function expansion is carried out, only the logic development of the expanded function needs to be concentrated, the basic logic function does not need to be concerned, and the like, and the flexibility of the method is greatly improved; furthermore, since the codes realizing the same function can be shared, a large number of repeated codes can be reduced, and the consumption of resources can be reduced.

Example four,

Correspondingly, an embodiment of the present invention further provides a system for acquiring network data, including any one of the apparatuses for acquiring network data provided in the embodiments of the present invention, which can be specifically referred to as embodiment three, for example, as follows:

the network data acquisition device is used for loading a corresponding dynamic library file according to the configuration file and calling a corresponding plug-in function; acquiring parameters related to the plug-in function from the configuration file; and executing the plug-in function according to the parameters to acquire network data.

The device for acquiring network data can also be used for acquiring the characteristic information of the logic processing inlet of the daemon process, determining whether the daemon process is abnormal or not according to the characteristic information, and if so, generating alarm information; if not, generating a response message indicating normal operation, or not generating the response message.

In addition, the system for acquiring network data may further include other devices, such as a terminal, as follows:

and the terminal is used for sending a data acquisition request to the network data acquisition device and receiving the network data returned by the network data acquisition device according to the data acquisition request.

The specific implementation of each device can be referred to the previous embodiment, and is not described herein again.

The network data acquisition system can comprise any network data acquisition device provided by the embodiment of the invention, so that the beneficial effects which can be realized by any network data acquisition device provided by the embodiment of the invention can be realized.

Example V,

An embodiment of the present invention further provides a server, as shown in fig. 4, which shows a schematic structural diagram of the server according to the embodiment of the present invention, specifically:

the server may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, Radio Frequency (RF) circuitry 403, a power supply 404, an input unit 405, and a display unit 406. Those skilled in the art will appreciate that the server architecture shown in FIG. 4 is not meant to be limiting, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 401 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the server. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.

The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the server, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.

The RF circuit 403 may be used for receiving and transmitting signals during transceiving information, and in particular, for receiving and transmitting downlink information of a base station to be processed by the one or more processors 401, and further, for transmitting data related to an uplink to the base station, generally, the RF circuit 403 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low Noise Amplifier (L NA, &l &ttt/t &ttt show Noise Amplifier), a duplexer, etc. furthermore, the RF circuit 403 may also communicate with a network and other devices through wireless communication, which may use any communication standard or protocol, including, but not limited to, Global System for mobile communication (GSM, Global System for mobile communication), General Packet Radio Service (GPRS, General Packet Radio), Division Multiple Access (CDMA, Service), transmission, Short Message Service (SMS), long Term Evolution (time Evolution) and/Short Access (time Evolution) messages (WCDMA), WCDMA, multimedia Access, and so on.

The server also includes a power supply 404 (e.g., a battery) for powering the various components, and preferably, the power supply 404 is logically connected to the processor 401 via a power management system, so that functions such as managing charging, discharging, and power consumption are performed via the power management system. The power supply 404 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The server may further include an input unit 405, and the input unit 405 may be used to receive input numeric or character information and generate a keyboard, mouse, joystick, optical or trackball signal input in relation to user settings and function control. Specifically, in one particular embodiment, input unit 405 may include a touch-sensitive surface as well as other input devices. The touch-sensitive surface, also referred to as a touch display screen or a touch pad, may collect touch operations by a user (e.g., operations by a user on or near the touch-sensitive surface using a finger, a stylus, or any other suitable object or attachment) thereon or nearby, and drive the corresponding connection device according to a predetermined program. Alternatively, the touch sensitive surface may comprise two parts, a touch detection means and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 401, and can receive and execute commands sent by the processor 401. In addition, touch sensitive surfaces may be implemented using various types of resistive, capacitive, infrared, and surface acoustic waves. The input unit 405 may include other input devices in addition to the touch-sensitive surface. In particular, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The server may further include a Display unit 406, the Display unit 406 operable to Display information input by or provided to a user and various graphical user interfaces of the server, the graphical user interfaces may be comprised of graphics, text, icons, video and any combination thereof the Display unit 406 may include a Display panel, optionally the Display panel may be configured in the form of a liquid Crystal Display (L CD, &lttttranslation = L "&tttL &/T &gtgtttiquick Crystal Display), an Organic light emitting diode (O L ED, Organic L sight-emissive diode), or the like.

Although not shown, the server may further include a camera, a bluetooth module, etc., which will not be described herein. Specifically, in this embodiment, the processor 401 in the server loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:

acquiring a configuration file at regular time; loading a corresponding dynamic library file according to the configuration file, and calling a corresponding plug-in function; acquiring parameters related to the plug-in function from the configuration file; and executing the plug-in function according to the parameters to acquire network data.

For example, the processor 401 may be configured to perform the following operations:

adding the parameters into a parameter list in sequence, and then executing the plug-in function according to the parameter list to acquire network data; for example, the parameters in the parameter list may be sequentially input to the plug-in function, the corresponding external interface is called by using the plug-in function to which the parameters are input, then, the original network data is pulled through the external interface, and the data processing is performed on the original network data to obtain the network data, and so on.

Optionally, in order to avoid the timeliness problem of the extranet data caused by the fact that the process exception cannot be found in time, when the data processing fails, an alarm may be further performed, that is, the processor 401 may be further configured to perform the following operations:

and if the data processing fails, generating first failure prompt information, and giving an alarm according to the first failure prompt information.

Optionally, after the network data is acquired, the network data may be written into a preset storage space for storage, and when the writing fails, an alarm is given, that is, the processor 401 may be further configured to perform the following operations:

Optionally, in order to facilitate timely learning of the process status, the processor 401 may be further configured to:

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

As can be seen from the above, the server in this embodiment may obtain a configuration file, load a corresponding dynamic library file according to the configuration file, call a corresponding plug-in function, obtain parameters related to the plug-in function from the configuration file, and then execute the plug-in function according to the parameters to obtain network data; because the scheme can embody some logic processing operations in the process of acquiring network data in a dynamic library file and plug-in function mode, for some basic logic functions shared by different modules, such as acquisition of user-independent data and the like, only the same dynamic library file and plug-in function need to be shared, and a set of corresponding complete codes do not need to be independently written for each module, so that the method is simpler to realize and is convenient for updating and function expansion, for example, when the function expansion is carried out, only the logic development of the expanded function needs to be concentrated, the basic logic function does not need to be concerned, and the like, and the flexibility of the method is greatly improved; furthermore, since the codes realizing the same function can be shared, a large number of repeated codes can be reduced, and the consumption of resources can be reduced.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

The method, the apparatus, and the system for acquiring network data provided by the embodiment of the present invention are described in detail above, and a specific example is applied in the present disclosure to explain the principle and the implementation of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A method for acquiring network data is characterized by comprising the following steps:

acquiring a configuration file at regular time;

executing the plug-in function according to the parameters to acquire network data;

the network data comprises user-related data and user-unrelated data, and different modules in the configuration file only acquire the user-unrelated data and share the same dynamic library file and plug-in function.

2. The method of claim 1, wherein loading the corresponding dynamic library file according to the configuration file and calling the corresponding plug-in function comprises:

reading the daemon task in the configuration file to determine the daemon task which needs to be executed currently;

and loading a corresponding dynamic library file according to the daemon task needing to be executed currently, and calling a corresponding plug-in function.

3. The method according to claim 2, wherein the loading the corresponding dynamic library file according to the daemon task that needs to be executed currently and calling the corresponding plug-in function comprise:

acquiring the identifier of the corresponding dynamic library file and the identifier of the plug-in function from the daemon task which needs to be executed currently;

calling a corresponding dynamic library file according to the identification of the dynamic library file, and loading the dynamic library file;

and calling the corresponding plug-in function from the dynamic library file according to the identification of the plug-in function.

4. The method according to any one of claims 1 to 3, wherein said executing the plug-in function according to the parameter to obtain the network data comprises:

adding the parameters to a parameter list in sequence;

and executing the plug-in function according to the parameter list to acquire network data.

5. The method of claim 4, wherein said executing the plug-in function according to the parameter list to obtain network data comprises:

sequentially inputting the parameters in the parameter list into the plug-in functions;

and executing the plug-in function with the input parameters to acquire the network data.

6. The method of claim 5, wherein executing the plug-in function with the input parameters to obtain the network data comprises:

calling a corresponding external interface by using the plug-in function with the input parameter, and pulling original network data through the external interface;

and processing the original network data to obtain the network data.

7. The method of claim 6, wherein after the data processing of the original network data to obtain the network data, further comprising:

8. The method of claim 6, wherein after the data processing of the original network data to obtain the network data, further comprising:

writing the network data into a preset storage space;

and if the writing fails, generating second failure prompt information, and giving an alarm according to the second failure prompt information.

9. The method according to any one of claims 1 to 3, wherein before the timing the obtaining of the configuration file, further comprising:

10. The method of claim 9, wherein after loading the configuration file, further comprising:

11. The method of claim 2, further comprising:

acquiring characteristic information of a logic processing entry of a daemon process;

and generating alarm information when the daemon process is determined to send abnormity according to the characteristic information.

12. An apparatus for acquiring network data, comprising:

the execution unit is used for executing the plug-in function according to the parameters so as to acquire network data;

13. The apparatus of claim 12, wherein the call unit comprises a determination subunit and a call subunit;

the determining subunit is configured to read the daemon task in the configuration file to determine the daemon task that needs to be executed currently;

and the calling subunit is used for loading the corresponding dynamic library file according to the daemon task which needs to be executed currently, and calling the corresponding plug-in function.

14. The apparatus of claim 13,

the calling subunit is specifically configured to obtain an identifier of a corresponding dynamic library file and an identifier of a plug-in function from the daemon task that needs to be executed currently; calling a corresponding dynamic library file according to the identification of the dynamic library file, and loading the dynamic library file; and calling the corresponding plug-in function from the dynamic library file according to the identification of the plug-in function.

15. The apparatus according to any one of claims 12 to 14, wherein the execution unit comprises an add subunit and an execute subunit;

the adding subunit is configured to add the parameters to a parameter list in order;

and the execution subunit is used for executing the plug-in function according to the parameter list so as to acquire network data.

16. The apparatus of claim 15,

the execution subunit is specifically configured to sequentially input the parameters in the parameter list into the plug-in function; and executing the plug-in function with the input parameters to acquire the network data.

17. The apparatus of claim 16,

the execution subunit is specifically configured to call a corresponding external interface by using the plug-in function to which the parameter is input, and pull the original network data through the external interface; and processing the original network data to obtain the network data.

18. The apparatus of claim 17,

the execution subunit is further configured to generate first failure prompt information when data processing fails, and perform an alarm according to the first failure prompt information.

19. The apparatus of claim 17, wherein the execution unit further comprises a storage subunit;

and the storage subunit is used for writing the network data into a preset storage space, generating second failure prompt information if the writing fails, and giving an alarm according to the second failure prompt information.

20. The apparatus according to any one of claims 12 to 14, further comprising a loading unit;

and the loading unit is used for loading the configuration file when the current operating environment is determined to meet the preset condition.

21. The apparatus of claim 20,

the loading unit is further configured to, after the configuration file is loaded, reload the configuration file if it is determined that the last modification time of the configuration file is inconsistent with the modification time of the configuration file at the last loading.

22. The apparatus of claim 13, further comprising an alert unit;

the alarm unit is used for acquiring the characteristic information of the logic processing entry of the daemon process and generating alarm information when the daemon process is determined to be abnormal according to the characteristic information.

23. A system for acquiring network data, comprising the apparatus for acquiring network data according to any one of claims 12 to 22.

24. A server, comprising a memory and a processor, the memory having stored therein a computer program that, when executed by the processor, causes the processor to perform the steps of:

acquiring a configuration file at regular time;

25. A computer-readable storage medium storing a computer program, wherein when the computer program runs on a computer, the computer is caused to execute the acquisition method of network data according to claim 1.