CN111198859A - Data processing method and device, electronic equipment and computer readable storage medium - Google Patents
Data processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111198859A CN111198859A CN201811369129.4A CN201811369129A CN111198859A CN 111198859 A CN111198859 A CN 111198859A CN 201811369129 A CN201811369129 A CN 201811369129A CN 111198859 A CN111198859 A CN 111198859A
- Authority
- CN
- China
- Prior art keywords
- data
- format
- log
- converted
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000006243 chemical reaction Methods 0.000 claims abstract description 57
- 238000000034 method Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000013500 data storage Methods 0.000 description 6
- 238000011156 evaluation Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000011022 operating instruction Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The present disclosure provides a data processing method, an apparatus, an electronic device and a computer-readable storage medium, the method comprising: acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different; and according to a pre-configured first data format, carrying out format conversion on the log data to obtain converted data. In the embodiment of the disclosure, the data format of the log data acquired from the plurality of service servers can be converted into the pre-configured data format, and the data with the uniform format is acquired and stored.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
For an online product, different service parameters need to be monitored according to different service requirements, so that service performance or user service conditions of the product can be mastered based on data corresponding to each service parameter obtained through monitoring.
In the prior art, when index data corresponding to different service parameters are monitored, index data corresponding to each service parameter needs to be obtained from one or more servers monitoring the index data of the different service parameters, and when the index data of each service scene is monitored by a plurality of servers, the data acquisition efficiency is low because the data volume of the index data corresponding to the service parameters obtained from each server is large and the data needs to be obtained from different servers.
Therefore, in the prior art, the efficiency of obtaining the index data corresponding to the service parameters from different servers is slow.
Disclosure of Invention
The present disclosure aims to solve at least one of the above technical drawbacks and improve data processing efficiency. The technical scheme adopted by the disclosure is as follows:
in a first aspect, the present disclosure provides a data processing method, including:
acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
and according to a pre-configured first data format, carrying out format conversion on the log data to obtain converted data.
In an embodiment of the present disclosure, performing format conversion on log data according to a preconfigured first data format includes:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
In the embodiment of the present disclosure, the log data includes index data of the monitored service parameter, and the converting the log data according to the data conversion rule includes:
extracting index data corresponding to the target service parameters from the log data according to parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and the index data corresponding to the target service parameters according to the data conversion rules.
In the embodiment of the disclosure, the parameter configuration information further includes indirect service parameters, and the index data corresponding to the indirect service parameters is determined by the index data corresponding to at least two monitored service parameters;
the method further comprises the following steps:
and obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In an embodiment of the present disclosure, the method further includes:
and storing the converted data according to the data quantity of the converted data.
In the embodiment of the present disclosure, storing the converted data according to the data size of the converted data includes:
if the data volume of the converted data is larger than a pre-configured data volume threshold value, storing the converted data according to a first storage mode;
if the data volume of the converted data is not larger than the data volume threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the second storage mode for storing the data with the data quantity threshold is larger than the storage space occupied by the first storage mode for storing the data with the data quantity threshold.
In an embodiment of the present disclosure, the method further includes:
receiving a data query request, wherein the data query request comprises a target service parameter, and the request statement format of the data query request corresponds to the first data format;
and according to the data query request, obtaining index data corresponding to the target service parameter from the converted data, and displaying the index data corresponding to the target service parameter.
In a second aspect, the present disclosure provides a data processing apparatus comprising:
the data acquisition module is used for acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
and the data conversion module is used for carrying out format conversion on the log data according to a pre-configured first data format to obtain converted data.
In an embodiment of the present disclosure, when performing format conversion on log data according to a preconfigured first data format, the data conversion module is specifically configured to:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
In the embodiment of the present disclosure, the log data includes index data of the monitored service parameter, and when the data conversion module converts the log data according to the data conversion rule, the data conversion module is specifically configured to:
extracting index data corresponding to the target service parameters from the log data according to parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and the index data corresponding to the target service parameters according to the data conversion rules.
In the embodiment of the disclosure, the parameter configuration information further includes indirect service parameters, and the index data corresponding to the indirect service parameters is determined by the index data corresponding to at least two monitored service parameters;
the device also includes:
and the indirect parameter determining module is used for obtaining the index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In an embodiment of the present disclosure, the apparatus further includes:
and the data storage module is used for storing the converted data according to the data volume of the converted data.
In an embodiment of the present disclosure, the data storage module is specifically configured to:
if the data volume of the converted data is larger than a pre-configured data volume threshold value, storing the converted data according to a first storage mode;
if the data volume of the converted data is not larger than the data volume threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the second storage mode for storing the data with the data quantity threshold is larger than the storage space occupied by the first storage mode for storing the data with the data quantity threshold.
In an embodiment of the present disclosure, the apparatus further includes:
the data processing module is used for receiving a data query request, wherein the data query request comprises target service parameters, and the request statement format of the data query request corresponds to the first data format; and according to the data query request, obtaining index data corresponding to the target service parameter from the converted data, and displaying the index data corresponding to the target service parameter.
In a third aspect, the present disclosure provides an electronic device comprising:
a processor and a memory;
a memory for storing computer operating instructions;
a processor for performing the method as shown in any embodiment of the first aspect of the present disclosure by invoking computer operational instructions.
In a fourth aspect, the present disclosure provides a computer readable storage medium having stored thereon at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by a processor to implement a method as set forth in any one of the embodiments of the first aspect of the disclosure.
The technical scheme provided by the embodiment of the disclosure has the following beneficial effects:
according to the data processing method, the data processing device, the electronic equipment and the computer readable storage medium, the data format of the log data acquired from the plurality of service servers can be converted into the pre-configured data format, the data with the uniform format is acquired and stored, and the log data acquired from the plurality of service servers are the log data with different formats, so that subsequent data processing is not convenient, therefore, by means of the scheme, the log data can be converted into the data with the uniform format, the acquired data with the uniform format can provide a data base for different service requirements in real time, and further, when relevant processing is performed on the basis of the log data with the uniform format, the data processing efficiency can be improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings used in the description of the embodiments of the present disclosure will be briefly described below.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of another data processing method provided by the embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below by referring to the drawings are exemplary only for explaining technical aspects of the present disclosure, and are not construed as limiting the present disclosure.
As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
The following describes the technical solutions of the present disclosure and how to solve the above technical problems in specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure; as shown in fig. 1, the method may include:
step S110, obtaining log data from a plurality of service servers, where data formats of the log data corresponding to at least two service servers in the plurality of service servers are different.
The network device, system and service program will generate an event record called log when operating, the log record is a log file, and the log data in the log file usually records the description of the first-order operation such as date, time, user and action. The specific contents recorded in the log file vary according to the service that the system is started.
Therefore, the service server is used for monitoring the service parameters, wherein in practical application, which service parameters need to be monitored can be configured according to actual service requirements, that is, which monitored service parameters can be specifically configured. In addition, based on the actual application scenario and different service requirements, one service server may correspondingly monitor one or more service parameters, or one service parameter may also be monitored by one or more service servers.
In an example, for an application, for example, it is required to obtain a user access amount UV of a certain time period a of a day, that is, a Unique viewer, where a client accessing a certain application is a guest, a service requirement in this example is to obtain the user access amount UV of the time period a, a service parameter is the access amount UV, and the access amount UV is specifically obtained by accessing a user ID of the user, so in an actual application, the number of user IDs accessing the application program in the time period a can be obtained by monitoring the user ID by a service server, and after performing deduplication and other processing on the obtained user ID, the number of user IDs corresponding to the time period a is finally obtained, that is, the access amount UV of the user.
Step S120, performing format conversion on the log data according to the pre-configured first data format to obtain converted data.
The first data format may be configured according to actual service requirements, and the specific format form of the first data format is not limited in this disclosure.
According to the scheme in the embodiment of the disclosure, the data format of the log data acquired from the plurality of service servers can be converted into the pre-configured data format, the data with the uniform format is acquired and stored, and the log data acquired from the plurality of service servers are the log data with different formats, so that subsequent data processing is not convenient, and therefore, through the scheme, the log data can be converted into the data with the uniform format, the acquired data with the uniform format can provide a data base for different service requirements in real time, and further, when relevant processing is performed on the basis of the log data with the uniform format, the efficiency of data processing can be improved.
In the embodiment of the present disclosure, in step S120, performing format conversion on the log data according to the preconfigured first data format may include:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
The data conversion rule represents a conversion mode between two different data formats, the data conversion rule may be configured in advance or during the data format conversion process, and different data formats may correspond to different data conversion rules, for example, the data conversion rule a corresponding to the data format a is used to convert the data format a into the data format B; the data conversion rule b corresponding to the data format A is used for converting the data format A into the data format C.
In an example, the second data format is an apache format, the first data format is a key value format, and the data conversion rule corresponding to the apache format is c, and then the data conversion rule c is used for converting log data in the apache format into data in the key value format.
In yet another example, the second data format is a json format, the first data format is a key value format, and the data conversion rule corresponding to the json format is d, then the data conversion rule d is used for converting the journal data in the json format into data in the keyvalue format.
In an embodiment of the present disclosure, the log data includes index data of the monitored service parameter, and the converting the log data according to the data conversion rule may include:
extracting index data corresponding to the target service parameters from the log data according to parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and the index data corresponding to the target service parameters according to the data conversion rules.
The parameter configuration information can be configured based on actual service requirements so as to meet different service requirements; specifically, it may be configured to extract the index data corresponding to which specific target service parameter or parameters in the log data, and similarly, it may also be configured to extract the index data corresponding to all target service parameters in the log data. The monitored service parameters in the log data can be configured according to actual service requirements, specifically, one or some service parameters in the log data can be selected as the monitored service parameters according to the service requirements, and similarly, all the service parameters in the log data can also be used as the monitored service parameters.
In an embodiment of the present disclosure, the parameter configuration information may further include an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two monitored service parameters, at this time, the method may further include:
and obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In practical application, a service parameter may be obtained correspondingly based on different monitored service parameters, for example, a service parameter obtained by calculating index data of different monitored service parameters, that is, an indirect service parameter, may be preconfigured with the indirect service parameter in the parameter configuration information, so that index data corresponding to the indirect service parameter may be obtained according to index data corresponding to at least two monitored service parameters corresponding to the indirect service parameter.
In an example, such as an indirect service parameter m, the indirect service parameter m may be determined by averaging index data of a monitored service parameter a and index data of a monitored service parameter b, and then in the parameter configuration information, the index data corresponding to the monitored service parameter a and the monitored service parameter b may be determined as the index data corresponding to the indirect service parameter m.
It can be understood that the data format of the index data corresponding to the obtained indirect service parameter may be the first data format or may be other data formats, and in order to make the data format of the index data corresponding to the indirect service parameter uniform with the first data format, the data format of the index data corresponding to at least two monitored service parameters corresponding to the indirect service parameter may be converted into the first data format before the data format of the index data corresponding to the indirect service parameter is obtained; the data format of the index data corresponding to the indirect service parameter may also be converted into the first data format after the data format of the index data corresponding to the indirect service parameter is obtained.
In an embodiment of the present disclosure, the method may further include:
and storing the converted data according to the data quantity of the converted data.
In order to facilitate subsequent processing of the converted data, such as data display, data analysis, and the like, the converted data may be stored, where the converted data refers to data with a data format of a first data format, and includes index data corresponding to indirect service parameters with the data format of the first data format and index data corresponding to target service parameters; in practical application, the converted data may be stored in a local server or a cloud server, and the storage location of the data is not limited in the disclosure.
In an embodiment of the present disclosure, storing the converted data according to the data size of the converted data may include:
if the data volume of the converted data is larger than a pre-configured data volume threshold value, storing the converted data according to a first storage mode;
if the data volume of the converted data is not larger than the data volume threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the second storage mode for storing the data with the data quantity threshold is larger than the storage space occupied by the first storage mode for storing the data with the data quantity threshold.
In practical application, in order to effectively save storage space when the data amount of the data to be stored is large on the premise of ensuring data storage efficiency, different storage modes can be configured according to the data amount of the converted data, so that in the data storage process, on the premise of ensuring data storage efficiency, the storage mode of the data can be dynamically adjusted based on the size of the data amount.
In an example, the first storage mode is a bitmap structure storage mode, a Bloomfilter bloom filter storage mode and a Hyper Log counter storage mode, the second storage mode is a set structure storage mode, when the data volume of the data to be stored is 1000 ten thousand user volumes, the storage space occupied by the 1000 ten-thousand user volume data stored by the first storage mode is 1.5MB, the storage space occupied by the 1000 ten-thousand user volume data stored by the second storage mode is 100MB, and the data volume threshold is 2000 ten thousand, when the data volume of the data to be stored is 3000 ten thousand, the storage space of 30MB may be needed by the first storage mode, and the storage space of 300MB is needed by the second storage mode, and the storage space of 3000 ten thousand user volume data stored by the first storage mode can be saved more.
In an embodiment of the present disclosure, the method may further include:
receiving a data query request, wherein the data query request comprises a target service parameter, and the request statement format of the data query request corresponds to the first data format;
and according to the data query request, obtaining index data corresponding to the target service parameter from the converted data, and displaying the index data corresponding to the target service parameter.
When the converted data with uniform format is subjected to subsequent processing, the data processing efficiency can be improved, for example, when a data query request is received, index data corresponding to a target business parameter can be quickly obtained from the converted data based on the data query request, and in order to meet business requirements of different users, the queried data can be displayed to the users.
With reference to the foregoing embodiments, this implementation provides a data processing method, as shown in fig. 2, where the data processing method may include:
step S210, obtaining log data from a plurality of service servers, where the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different.
Step S220, determining a data conversion rule for converting the second data format into the first data format according to the preconfigured first data format and the second data format of the log data.
Step S230, converting the log data according to the data conversion rule to obtain converted data.
Step S240, if the data volume of the converted data is larger than a pre-configured data volume threshold value, storing the converted data according to a first storage mode;
if the data volume of the converted data is not larger than the data volume threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the second storage mode for storing the data with the data quantity threshold is larger than the storage space occupied by the first storage mode for storing the data with the data quantity threshold.
In an example, the second data format is an apache format, the first data format is a key value format, the data conversion rule corresponding to the apache format is c, the log data is converted into the data in the key value format based on the data conversion rule c, the data volume of the converted log data is 3000 ten thousand user volume which is larger than a preconfigured data volume threshold value of 2000 ten thousand, the first storage mode is a bitmap structure storage mode, the second storage mode is a set structure storage mode, the storage space of 30MB is needed for storing the converted log data by adopting the first storage mode, and the storage space of 300MB is needed for storing the converted log data by adopting the second storage mode, so when the data with a larger data volume is stored, the storage space is saved by adopting the first storage mode compared with the second storage mode.
In order to improve the data processing efficiency in the subsequent processing process of the converted data based on the format unification, the following examples are provided:
in an example, monitoring an application program in real time through different service servers to obtain monitoring data, wherein the data formats of the monitoring data corresponding to at least two service servers in a plurality of service servers are different, the index data of a monitored service parameter in the monitoring data can reflect the running state of the application program, the monitoring data is data obtained from different service servers and converted according to a first data format, the converted data is data with a uniform format, when the index data corresponding to the monitored service parameter a in the application program is abnormal, the index data corresponding to the monitored service parameter a needs to be checked to find out the reason of data abnormality, so as to update the application program in time, that is, the service requirement is to inquire the index data corresponding to the service parameter a at the current time, the corresponding target service parameter is a monitored service parameter A; then, based on the target service parameter a in the data query request, index data corresponding to the target service parameter a at the current moment is searched in the monitoring data, and a reason for the abnormality of the index data corresponding to the target service parameter a is discovered in time. Because the monitoring data is data with a uniform format, index data of the corresponding target service parameter can be found in the monitoring data only by providing a data query request corresponding to the first data format, and format conversion or other processing is not required to be performed on the monitoring data with different formats during data query, so that the monitoring data with the uniform format provides a data basis for the data query, and the efficiency of the data query is improved.
Based on the same principle as the method shown in fig. 1, an embodiment of the present disclosure also provides a data processing apparatus 20, as shown in fig. 3, where the data processing apparatus 20 may include: a data acquisition module 210 and a data conversion module 220, wherein,
a data obtaining module 210, configured to obtain log data from multiple service servers, where data formats of the log data corresponding to at least two service servers in the multiple service servers are different;
the data conversion module 220 is configured to perform format conversion on the log data according to a preconfigured first data format, so as to obtain converted data.
According to the scheme in the embodiment of the disclosure, the data format of the log data acquired from the plurality of service servers can be converted into the pre-configured data format, the data with the uniform format is acquired and stored, and the log data acquired from the plurality of service servers are the log data with different formats, so that subsequent data processing is not convenient, and therefore, through the scheme, the log data can be converted into the data with the uniform format, the acquired data with the uniform format can provide a data base for different service requirements in real time, and further, when relevant processing is performed on the basis of the log data with the uniform format, the efficiency of data processing can be improved.
In an embodiment of the present disclosure, when performing format conversion on log data according to a preconfigured first data format, the data conversion module 220 is specifically configured to:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
In the embodiment of the present disclosure, the log data includes index data of the monitored service parameter, and when the data conversion module 220 converts the log data according to the data conversion rule, the data conversion module is specifically configured to:
extracting index data corresponding to the target service parameters from the log data according to parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and the index data corresponding to the target service parameters according to the data conversion rules.
In an embodiment of the present disclosure, the parameter configuration information further includes an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two monitored service parameters, at this time, the apparatus may further include:
and the indirect parameter determining module is used for obtaining the index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In an embodiment of the present disclosure, the apparatus may further include:
and the data storage module is used for storing the converted data according to the data volume of the converted data.
In an embodiment of the present disclosure, storing the converted data according to the data size of the converted data may include:
if the data volume of the converted data is larger than a pre-configured data volume threshold value, storing the converted data according to a first storage mode;
if the data volume of the converted data is not larger than the data volume threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the second storage mode for storing the data with the data quantity threshold is larger than the storage space occupied by the first storage mode for storing the data with the data quantity threshold.
In an embodiment of the present disclosure, the apparatus may further include:
the data processing module is used for receiving a data query request, wherein the data query request comprises target service parameters, and the request statement format of the data query request corresponds to the first data format; and according to the data query request, obtaining index data corresponding to the target service parameter from the converted data, and displaying the index data corresponding to the target service parameter.
The data processing apparatus of the embodiment of the present disclosure may execute a data processing method provided by the embodiment of the present disclosure, and the implementation principles thereof are similar, the actions executed by each module in the data processing apparatus in each embodiment of the present disclosure correspond to the steps in the data processing method in each embodiment of the present disclosure, and for the detailed functional description of each module in the data processing apparatus, reference may be specifically made to the description in the corresponding data processing method shown in the foregoing, and details are not repeated here.
Based on the same principle as the data processing method in the embodiment of the present disclosure, the present disclosure provides an electronic device including a processor and a memory; a memory for storing operating instructions; a processor for executing the method as shown in any embodiment of the data processing method of the present disclosure by calling an operation instruction.
Based on the same principle as the data processing method in the embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the method as shown in any one of the embodiments of the data processing method of the present disclosure.
In the embodiment of the present disclosure, as shown in fig. 4, a schematic structural diagram of an electronic device 30 (e.g., a terminal device or a server implementing the method shown in fig. 1) suitable for implementing the embodiment of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 4, the electronic device 30 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic apparatus 30 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Generally, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device 30 to communicate wirelessly or by wire with other devices to exchange data. While fig. 4 illustrates an electronic device 30 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 309, or installed from the storage means 308, or installed from the ROM 302. The computer program, when executed by the processing device 301, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects the internet protocol addresses from the at least two internet protocol addresses and returns the internet protocol addresses; receiving an internet protocol address returned by the node evaluation equipment; wherein the obtained internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first retrieving unit may also be described as a "unit for retrieving at least two internet protocol addresses".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Claims (10)
1. A data processing method, comprising:
acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
and according to a pre-configured first data format, carrying out format conversion on the log data to obtain converted data.
2. The method of claim 1, wherein the format converting the log data in the preconfigured first data format comprises:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and a second data format of the log data;
and converting the log data according to the data conversion rule.
3. The method of claim 2, wherein the log data comprises index data of monitored traffic parameters, and wherein converting the log data according to the data conversion rule comprises:
extracting index data corresponding to a target service parameter from the log data according to parameter configuration information, wherein the parameter configuration information comprises the target service parameter, and the monitored service parameter comprises the target service parameter;
and converting the target service parameters and the index data corresponding to the target service parameters according to the data conversion rules.
4. The method according to claim 3, wherein the parameter configuration information further includes an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two of the monitored service parameters;
the method further comprises the following steps:
and obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
5. The method of any of claims 1 to 4, further comprising:
and storing the converted data according to the data volume of the converted data.
6. The method of claim 5, wherein storing the converted data according to the data amount of the converted data comprises:
if the data volume of the converted data is larger than a pre-configured data volume threshold value, storing the converted data according to a first storage mode;
if the data volume of the converted data is not larger than the data volume threshold, storing the converted data according to a second storage mode;
the storage space occupied by the second storage mode for storing the data of the data quantity threshold is larger than the storage space occupied by the first storage mode for storing the data of the data quantity threshold.
7. The method of any of claims 1 to 4, further comprising:
receiving a data query request, wherein the data query request comprises the target service parameters, and the request statement format of the data query request corresponds to the first data format;
and according to the data query request, obtaining index data corresponding to the target service parameter from the converted data, and displaying the index data corresponding to the target service parameter.
8. A data processing apparatus, comprising:
the data acquisition module is used for acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
and the data conversion module is used for carrying out format conversion on the log data according to a pre-configured first data format to obtain converted data.
9. An electronic device, comprising:
a processor and a memory;
the memory is used for storing computer operation instructions;
the processor is used for executing the method of any one of the claims 1 to 7 by calling the computer operation instruction.
10. A computer readable storage medium having stored thereon a computer program, the storage medium having stored thereon at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by a processor to implement the method of any of the preceding claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811369129.4A CN111198859B (en) | 2018-11-16 | 2018-11-16 | Data processing method, device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811369129.4A CN111198859B (en) | 2018-11-16 | 2018-11-16 | Data processing method, device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111198859A true CN111198859A (en) | 2020-05-26 |
| CN111198859B CN111198859B (en) | 2023-11-03 |
Family
ID=70744179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811369129.4A Active CN111198859B (en) | 2018-11-16 | 2018-11-16 | Data processing method, device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111198859B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112052247A (en) * | 2020-09-29 | 2020-12-08 | 微医云(杭州)控股有限公司 | Index updating system, method and device of search engine, electronic equipment and storage medium |
| CN113297148A (en) * | 2021-06-18 | 2021-08-24 | 杭州安恒信息安全技术有限公司 | Method, device and equipment for acquiring service log data and readable storage medium |
| CN113504896A (en) * | 2021-07-12 | 2021-10-15 | 云南腾云信息产业有限公司 | Service data processing method and device of application program and mobile terminal |
| CN113656445A (en) * | 2021-08-26 | 2021-11-16 | 五八同城信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114241815A (en) * | 2021-11-08 | 2022-03-25 | 深圳市瑞达飞行科技有限公司 | Computing method and device, electronic equipment and computer readable storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178253A1 (en) * | 2001-05-24 | 2002-11-28 | International Business Machines Corporation | Method and apparatus to solve compatibility between heterogeneous web server access logs formats |
| US20040167912A1 (en) * | 2003-02-20 | 2004-08-26 | International Business Machines Corporation | Unified logging service for distributed applications |
| JP2009009448A (en) * | 2007-06-29 | 2009-01-15 | Mitsubishi Electric Corp | Data transmission device, data transmission method, and program |
| CN103235811A (en) * | 2013-04-24 | 2013-08-07 | 微梦创科网络科技（中国）有限公司 | Data storage method and device |
| CN103823811A (en) * | 2012-11-19 | 2014-05-28 | 北京百度网讯科技有限公司 | Method and system for processing journals |
| CN104239538A (en) * | 2014-09-22 | 2014-12-24 | 北京国双科技有限公司 | Method, system and device for compressing snapshot log |
| CN105095518A (en) * | 2015-09-18 | 2015-11-25 | 中国农业银行股份有限公司 | Data conversion method and device |
| CN105262812A (en) * | 2015-10-16 | 2016-01-20 | 浪潮(北京)电子信息产业有限公司 | Log data processing method based on cloud computing platform, log data processing device and log data processing system |
| CN105378716A (en) * | 2014-03-18 | 2016-03-02 | 华为技术有限公司 | Method and apparatus for conversion of data storage formats |
| CN106202235A (en) * | 2016-06-28 | 2016-12-07 | 微梦创科网络科技（中国）有限公司 | A kind of data processing method and device |
| CN107092683A (en) * | 2017-04-21 | 2017-08-25 | 北京奇艺世纪科技有限公司 | A kind of log processing system and method |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| CN108345528A (en) * | 2018-02-13 | 2018-07-31 | 北京天元创新科技有限公司 | A kind of background server monitoring method, device and equipment |
-
2018
- 2018-11-16 CN CN201811369129.4A patent/CN111198859B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178253A1 (en) * | 2001-05-24 | 2002-11-28 | International Business Machines Corporation | Method and apparatus to solve compatibility between heterogeneous web server access logs formats |
| US20040167912A1 (en) * | 2003-02-20 | 2004-08-26 | International Business Machines Corporation | Unified logging service for distributed applications |
| JP2009009448A (en) * | 2007-06-29 | 2009-01-15 | Mitsubishi Electric Corp | Data transmission device, data transmission method, and program |
| CN103823811A (en) * | 2012-11-19 | 2014-05-28 | 北京百度网讯科技有限公司 | Method and system for processing journals |
| CN103235811A (en) * | 2013-04-24 | 2013-08-07 | 微梦创科网络科技（中国）有限公司 | Data storage method and device |
| CN105378716A (en) * | 2014-03-18 | 2016-03-02 | 华为技术有限公司 | Method and apparatus for conversion of data storage formats |
| CN104239538A (en) * | 2014-09-22 | 2014-12-24 | 北京国双科技有限公司 | Method, system and device for compressing snapshot log |
| CN105095518A (en) * | 2015-09-18 | 2015-11-25 | 中国农业银行股份有限公司 | Data conversion method and device |
| CN105262812A (en) * | 2015-10-16 | 2016-01-20 | 浪潮(北京)电子信息产业有限公司 | Log data processing method based on cloud computing platform, log data processing device and log data processing system |
| CN106202235A (en) * | 2016-06-28 | 2016-12-07 | 微梦创科网络科技（中国）有限公司 | A kind of data processing method and device |
| CN107092683A (en) * | 2017-04-21 | 2017-08-25 | 北京奇艺世纪科技有限公司 | A kind of log processing system and method |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| CN108345528A (en) * | 2018-02-13 | 2018-07-31 | 北京天元创新科技有限公司 | A kind of background server monitoring method, device and equipment |
Non-Patent Citations (1)
| Title |
|---|
| 吴涛, 丁伟, 高毓航: "一个开放环境中Client/Server结构的日志审计系统", 计算机工程, no. 1 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112052247A (en) * | 2020-09-29 | 2020-12-08 | 微医云(杭州)控股有限公司 | Index updating system, method and device of search engine, electronic equipment and storage medium |
| CN112052247B (en) * | 2020-09-29 | 2024-05-07 | 微医云(杭州)控股有限公司 | Index updating system, method and device for search engine, electronic equipment and storage medium |
| CN113297148A (en) * | 2021-06-18 | 2021-08-24 | 杭州安恒信息安全技术有限公司 | Method, device and equipment for acquiring service log data and readable storage medium |
| CN113504896A (en) * | 2021-07-12 | 2021-10-15 | 云南腾云信息产业有限公司 | Service data processing method and device of application program and mobile terminal |
| CN113504896B (en) * | 2021-07-12 | 2023-08-18 | 云南腾云信息产业有限公司 | Service data processing method and device of application program and mobile terminal |
| CN113656445A (en) * | 2021-08-26 | 2021-11-16 | 五八同城信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114241815A (en) * | 2021-11-08 | 2022-03-25 | 深圳市瑞达飞行科技有限公司 | Computing method and device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111198859B (en) | 2023-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111198859B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN112291121B (en) | Data processing method and related equipment | |
| CN110908921A (en) | Game problem positioning method and device, electronic equipment and storage medium | |
| CN111198853B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN111274104B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN110727558A (en) | Information prompting method and device, storage medium and electronic equipment | |
| CN112688793B (en) | Data packet obtaining method and device and electronic equipment | |
| CN111460020B (en) | Method, device, electronic equipment and medium for resolving message | |
| CN112260880B (en) | Network access relation display method and related equipment | |
| CN112261176B (en) | Method for acquiring actual network access relationship and related equipment | |
| CN111241368B (en) | Data processing method, device, medium and equipment | |
| CN112379967B (en) | Simulator detection method, device, equipment and medium | |
| CN109614137B (en) | Software version control method, device, equipment and medium | |
| CN111367592B (en) | Information processing method and device | |
| CN109522133B (en) | Data splicing method and device, electronic equipment and storage medium | |
| CN113282455A (en) | Monitoring processing method and device | |
| CN112100159A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN112799863A (en) | Method and apparatus for outputting information | |
| US10516767B2 (en) | Unifying realtime and static data for presenting over a web service | |
| CN116820354B (en) | Data storage method, data storage device and data storage system | |
| CN110099122B (en) | Method and apparatus for sending network request | |
| CN111092929B (en) | File issuing method and device and electronic equipment | |
| CN111176957A (en) | Log browsing method and device including timely search and electronic equipment | |
| CN115877988A (en) | Content resource duplication elimination method and device, electronic equipment and storage medium | |
| CN116204218A (en) | Object processing method, device, electronic equipment, storage medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |