CN110633388A - Real-time index generation method, system and storage medium based on communication XDR - Google Patents
Real-time index generation method, system and storage medium based on communication XDR Download PDFInfo
- Publication number
- CN110633388A CN110633388A CN201910752728.2A CN201910752728A CN110633388A CN 110633388 A CN110633388 A CN 110633388A CN 201910752728 A CN201910752728 A CN 201910752728A CN 110633388 A CN110633388 A CN 110633388A
- Authority
- CN
- China
- Prior art keywords
- index
- dimension
- internal structure
- xdr
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/81—Indexing, e.g. XML tags; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/83—Querying
- G06F16/835—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/83—Querying
- G06F16/838—Presentation of query results
Abstract
The invention discloses a real-time index generation method, a real-time index generation system and a storage medium based on communication XDR, wherein the method comprises the following steps: acquiring attribute data according to the XML file; obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data; performing dimension extraction according to the attribute data and the internal structure message, and performing index operation according to the attribute data and the internal structure message; and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result. The invention can realize the quick generation of the real-time index under the condition of avoiding the partial modification of the dimension and the index algorithm when the communication index is added or deleted and modified or the XDR specification is upgraded. The invention can be widely applied to the technical field of communication as a method, a system and a storage medium.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a real-time index generation method, a real-time index generation system and a storage medium based on communication XDR.
Background
With the continuous development of communication networks and the continuous expansion of user scales, communication operators compete increasingly strongly in the field of mobile voice and data services, the profitability of the communication operators is determined by the control capability of the coverage, quality, guarantee capability and network cost of information networks, communication indexes are important methods for measuring network quality, network capability and the like, and the demands of various cities on communication index models are different. In the prior art, the generation of the communication indexes is generally realized by background complex logic codes, if the communication indexes of the model need to be subjected to addition and deletion modification or XDR specification upgrading, a special technical developer needs to carry out massive modification on algorithm logic codes, the warehousing and storage processes corresponding to the index models also need to be synchronously modified, the modification period is long, and the index change requirements of different projects in various cities cannot be quickly responded.
Disclosure of Invention
In view of the above, in order to solve the above technical problems, an object of the present invention is to provide a method, a system and a storage medium for generating a real-time indicator based on a communication XDR, which can generate a real-time indicator quickly.
The technical scheme adopted by the invention is as follows: the real-time index generation method based on communication XDR comprises the following steps:
acquiring attribute data according to the XML file;
obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data;
performing dimension extraction according to the attribute data and the internal structure message, and performing index operation according to the attribute data and the internal structure message;
and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result.
Further, the step of obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data includes the following steps:
providing a corresponding internal structure for each XDR data;
and mapping the field in the XDR data to a corresponding internal structure body to obtain an internal structure body message.
Further, the step of performing dimension extraction according to the attribute data and the internal structure message, and performing index operation according to the attribute data and the internal structure message includes the following steps:
according to the attribute data, creating an integral dimension object and an integral index object;
extracting an integral dimension object and performing dimension operation according to the internal structure body message to obtain a dimension extraction result;
and according to the internal structure body message, screening and index operation of the whole index object are carried out to obtain an index operation result.
Further, the step of obtaining a real-time index statistical result according to the dimension extraction result and the index operation result includes the following steps:
acquiring statistical granularity time according to the internal structure information;
and obtaining a real-time index statistical result within the statistical granularity time according to the dimension extraction result, the index operation result and the statistical granularity time.
The invention also provides a real-time index generation system based on communication XDR, which comprises:
the model configuration module is used for acquiring attribute data according to the XML file;
the XDR adaptation module is used for obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data;
the dimension index factory is used for dimension extraction and index operation;
and the model data storage and operation module is used for calling the dimension index factory according to the attribute data and the internal structure body information, and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result.
Further, the dimension index factory includes:
the creating unit is used for creating an integral dimension object and an integral index object according to the attribute data;
the extracting unit is used for extracting the whole dimension object and performing dimension operation according to the internal structure body message to obtain a dimension extracting result;
the operation unit is used for screening the whole index object and performing index operation according to the internal structure body message to obtain an index operation result;
and the return unit is used for returning the dimension extraction result and the index operation result to the model data storage and operation module.
Further, the model data storage and operation module comprises a single message index operation result container and a dimension index cache container;
the single message index operation result container is used for caching a single operation result of each index after index operation;
the dimension index cache container is used for caching the dimension extraction result and the index operation result, wherein the index operation result is obtained by accumulating the single operation result.
Further, the system also comprises a model management module and a time control module;
the time control module is used for acquiring the statistical granularity time according to the internal structure body message and notifying the mode management module;
the model management module is used for creating and calling a model data storage and operation module object, processing the internal structure body message, responding to the notification of the time control module to control the model data storage and operation module, and outputting the real-time index statistical result within the statistical granularity time.
The invention also provides a real-time index generation system based on communication XDR, which comprises: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor is enabled to implement the communication XDR-based real-time indicator generation method.
The invention also provides a storage medium which stores processor-executable instructions, and when the processor executes the processor-executable instructions, the communication XDR-based real-time index generation method is executed.
The invention has the beneficial effects that: acquiring attribute data according to an XML file, acquiring an internal structure message according to received XDR data and an internal structure corresponding to the XDR data, and performing dimension extraction and index operation according to the attribute data and the internal structure message to obtain a calculation result and a statistical result; the invention obtains the internal structure body message through the XDR data and the internal structure body corresponding to the XDR data, because the dimension extraction and the index operation use the internal structure body message in the internal structure body, and the XDR data has the corresponding internal structure body, when the XDR specification is upgraded, only the assignment algorithm of the internal structure body needs to be updated according to the XDR data without changing the original structure of the internal structure body, so that the modification of the dimension and the index algorithm part can be simultaneously avoided, the development and test workload when the XDR specification is upgraded is greatly reduced, the modification period is short, and the rapid generation of real-time indexes can be realized; meanwhile, the XML file is used for acquiring the attribute data, dimension extraction and index operation are carried out according to the attribute data and the internal structure body information, when communication indexes are required to be added or deleted or the attribute data are required to be modified, logic codes are not required to be modified, dimension extraction and index operation can be normally carried out only by modifying XML configuration, and the method is fast and efficient.
Drawings
FIG. 1 is a schematic flow chart illustrating steps of a real-time indicator generating method based on communication XDR according to the present invention;
FIG. 2 is a diagram illustrating an XDR data field mapping relationship according to the present invention;
FIG. 3 is a block diagram of a real-time indicator generating system based on communication XDR according to the present invention;
FIG. 4 is a schematic diagram of the data processing relationship between the model data storage and computation module and the dimensional index factory according to the present invention;
FIG. 5 is a schematic diagram of the model data storage and calculation module data processing and relationship to the model management module according to the present invention;
FIG. 6 is a schematic diagram of a growling bitmap algorithm in accordance with the present invention;
FIG. 7 is a flow chart of the stored procedure generating module of the present invention.
Detailed Description
The invention will be further explained and explained with reference to the drawings and the embodiments in the description. The step numbers in the embodiments of the present invention are set for convenience of illustration only, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
As shown in fig. 1, the method for generating real-time indicators based on communication XDR includes the following steps:
acquiring attribute data according to the XML file;
obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data;
performing dimension extraction according to the attribute data and the internal structure message, and performing index operation according to the attribute data and the internal structure message;
and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result.
In this embodiment, the model configuration XML file is read and parsed to obtain the attribute data TableProperty of each model. The design of a model configuration XML file is shown in table 1. An internal structure (also referred to as an internal message structure) refers to a structure (a data set composed of a series of data having the same type or different types) inside a system, and an internal structure message refers to a message formed by the internal structure. Wherein, the dimension extraction result is a dimension combination KEY. In addition, a storage procedure of various database versions of each model can be generated based on the attribute data TableProperty of each model.
TABLE 1
As shown in fig. 2, further as a preferred embodiment, the step of obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data includes the following steps:
providing a corresponding internal structure for each XDR data;
and mapping the field in the XDR data to a corresponding internal structure body to obtain an internal structure body message.
In the present embodiment, there are a plurality of internal structure bodies (also referred to as internal message structure bodies), each XDR data corresponds to one internal message structure body, and extracting required fields in the XDR data (XDR structure bodies) is mapped to the internal message structure body to form a new message, i.e., a new internal structure body message, where the internal structure body message has a plurality of fields (including intMsgType, which indicates the message type of the internal structure body message). In general, the XDR structure field is more than the internal message structure field and is in a fully-contained relationship, so that mapping is needed; mapping refers to assigning a certain field from the XDR structure to a certain field of the internal message structure, which is a one-to-one direct padding value in most cases, and a simple operation for a refill value in few cases, and also supports assigning a result to a field of the internal message structure after performing a certain arithmetic operation on two or more fields of the XDR structure.
Further, as a preferred embodiment, the step of performing dimension extraction according to the attribute data and the internal structure message, and performing index operation according to the attribute data and the internal structure message includes the following steps:
according to the attribute data, creating an integral dimension object and an integral index object;
extracting an integral dimension object and performing dimension operation according to the internal structure body message to obtain a dimension extraction result;
and according to the internal structure body message, screening and index operation of the whole index object are carried out to obtain an index operation result.
In the embodiment, according to the attribute data TableProperty of the model, specifically, according to the field list (including the dimension field and the index field-fieldmame) of the attribute data of the model, all the dimension objects and index objects that the model can use, that is, the whole dimension object and the whole index object, are created;
extracting and carrying out dimension operation on an integral dimension object according to an transmitted internal structure message to obtain a dimension combination KEY, namely a dimension extraction result, wherein the dimension object and a dimension algorithm library are in a one-to-one correspondence relationship, extracting a dimension object required to be used according to a dimension field of the internal structure message, calling an algorithm corresponding to the dimension (for example, two fields are taken from the internal structure message to be added and then divided by 1000) to carry out operation on a certain dimension object to obtain a value of the dimension, and finally splicing values of all the dimensions to obtain the integral dimension, namely the dimension combination KEY (for example, a certain model only has two dimension fields, the value of dimension 1 is 4 bytes, the value of dimension 2 is 8 bytes, and the spliced dimension combination KEY is 12 bytes);
and (3) selecting an index object to be executed from the whole index objects according to the intMsgType of the internal structure body message, distributing the internal structure body message to the selected index object, and performing index operation through algorithm logic preset in an index algorithm class library corresponding to the index object (the index object and the index algorithm library are in one-to-one correspondence, and the algorithm corresponding to the index can be called when a certain index object is executed) to obtain an index operation result. Wherein, the screening step specifically comprises: according to the definition of the index fields in table 1, as can be seen from table 1, each index field has set a "message type (intMsgType) list to be processed" in table 1, so that it is clear which indexes each index field needs to process, when receiving the internal structure message, it is only necessary to traverse the index object (corresponding to the index field) to check whether the "message type list to be processed" of the index object contains the intMsgType, and if so, the index object is selected, and then the preset algorithm logic corresponding to the index algorithm class library of the index object is executed to perform the index operation.
According to different types of indexes, index operation results obtained through the index algorithm class library may include: a common accumulated count index, a certain fixed value (maximum or minimum value index) and an IMSI (Subscriber deduplication index), where the IMSI is an International Mobile Subscriber Identity, which is an Identity that is used to distinguish different subscribers in a cellular network and is not repeated in all cellular networks.
Further as a preferred embodiment, the step of obtaining a real-time index statistical result according to the dimension extraction result and the index operation result includes the following steps:
acquiring statistical granularity time according to the internal structure information;
and obtaining a real-time index statistical result within the statistical granularity time according to the dimension extraction result, the index operation result and the statistical granularity time.
In this embodiment, further logical operations (accumulation operations (including adding 1 operation to count-order indexes such as turn-on times and call drop times, and accumulating any integer greater than 0, such as a delay index and a flow index), maximum or minimum value replacement, counting after the number of IMSI users is deduplicated) are performed on the index operation result to obtain an intermediate result in the model statistical granularity, where the intermediate result includes index values of multiple indexes, and the index value of an index refers to an index value of the index in the model statistical granularity, that is, a result obtained by counting all relevant internal structure messages in the model statistical granularity according to the algorithm of the index; the model statistical granularity can support various statistical granularity time, so that the current message time needs to be acquired according to the time field in the internal structure message, output judgment is carried out by combining the statistical granularity of the model, and the intermediate result and the dimension combination KEY are output as a real-time index statistical result according to the judged statistical granularity time.
And finally, outputting the real-time index statistical result to a data transmission queue DataQueue, acquiring statistical data from the data transmission queue DataQueue, formatting the data according to model attributes (attribute data of the model) and database format requirements, and outputting the data of each model in a classified manner according to the model name and time to obtain a final index statistical result.
In summary, the steps of the real-time indicator generating method based on communication XDR are described as follows:
1) reading and analyzing the model configuration XML file to obtain attribute data TableProperty of each model;
2) extracting required fields from each received XDR data, and mapping the fields to corresponding internal structure bodies to obtain internal structure body messages;
3) according to the attribute data tableProperty of each model, creating all dimension objects and index objects which need to be used by the model, namely an integral dimension object and an integral index object, and extracting the integral dimension object and performing dimension operation according to the transmitted internal structure information to obtain a dimension combination KEY (dimension extraction result);
4) selecting index objects needing to be executed from the whole index objects according to the internal structure body information, distributing the internal structure body information to the selected index objects, and performing index operation through an index algorithm class library to obtain index operation results;
5) performing further logic operation on the index operation result to obtain an intermediate result (including index values of all indexes in the model statistical granularity time);
6) acquiring statistical granularity time according to the internal structure information, and outputting the intermediate result and the dimension combination KEY as a real-time index statistical result;
7) outputting the real-time index statistical result to a data transmission queue DataQueue, acquiring statistical data from the data transmission queue DataQueue, formatting the data according to model attributes (attribute data of the model) and database format requirements, and outputting the data of each model in a classification manner according to the model name and time to obtain a final index statistical result.
Referring to fig. 3, the present invention further provides a real-time indicator generating system based on communication XDR, including:
the model configuration module is used for acquiring attribute data according to the XML file;
the XDR adaptation module is used for obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data;
the dimension index factory is used for dimension extraction and index operation;
and the model data storage and operation module is used for calling the dimension index factory according to the attribute data and the internal structure body information, and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result.
In the present embodiment, the model data storage and calculation module (i.e., the model data storage and calculation module) is provided. When the program is started, a model configuration module (Configer) is first run to read and parse the model configuration XML file to obtain attribute data tableProperty of each model. The design of the model configuration XML file is shown in Table 1 above.
As shown in fig. 2, the XDR adaptation module (Import), which receives XDR data and extracts required fields to map to an internal structure to form a new message (internal structure message). Wherein, a corresponding internal message structure is designed for each XDR data. Therefore, when the XDR specification changes, the XDR structure field changes (the field type, the field length, or the field name changes), and in the XDR adaptation module, the field assignment algorithm of the internal message structure can be updated one by one according to the field mapping relationship between the XDR structure and the internal message structure, without changing the structure of the internal message structure (including the field type, the field length, or the field name).
As shown in fig. 4 and 5, further as a preferred embodiment, the dimension index factory includes:
the creating unit is used for creating an integral dimension object and an integral index object according to the attribute data;
the extracting unit is used for extracting the whole dimension object and performing dimension operation according to the internal structure body message to obtain a dimension extracting result;
the operation unit is used for screening the whole index object and performing index operation according to the internal structure body message to obtain an index operation result;
and the return unit is used for returning the dimension extraction result and the index operation result to the model data storage and operation module.
Further as a preferred embodiment, the model data storage and operation module includes a single message index operation result container and a dimension index cache container;
the single message index operation result container is used for caching a single operation result of each index after index operation;
the dimension index cache container is used for caching the dimension combination and the index operation result, wherein the index operation result is obtained by accumulating the single operation result.
As shown in fig. 4 and 5, in the present embodiment, a model data storage and operation module (DataTable) is used for loading and initializing, specifically: the method comprises the following elements of loading a model attribute object (an object pointed by a pointer of attribute data), establishing a dimension index, creating a dimension index factory object (an object pointed by a pointer of a dimension index factory), creating a dimension index cache container, creating an index operation result container of a single message and the like, wherein the elements and descriptions thereof are shown in a table 2; after receiving the internal structure message, the module calls an index factory object to perform dimension extraction and index operation on the internal structure message to obtain a dimension extraction result and an index operation result, and the dimension extraction result and the index operation result are cached in a dimension index cache container. Wherein, each model corresponds to an independent model data storage and operation module (DataTable).
Wherein, the model data storage and operation module also comprises a message processing module (Msgprocessor), the message processing module is provided with a dimension object container and an index object container, the message processing module is instantiated into an object of the model data storage and operation module when the model data storage and operation module is initialized (instantiation refers to instantiation of C + + class, the message processing module is an independent C + + class, an interface of the model data storage and operation module is called, a specific object needs to be created by the C + + class), 4 calling interfaces are provided for the model data storage and operation module, the 4 calling interfaces are respectively used for extracting dimension combination KEY of the internal structure body message, index operation results of the internal structure body message, dimension creation and index creation. Meanwhile, when the instance of the message processing module is initialized, a pointer of the dimension index factory is obtained (the pointer points to a globally unique dimension index factory object, and all message processing module instances call the same dimension index factory object).
Specifically, during initialization, a 'dimension creation' interface and an 'index creation' interface are called first, so that a dimension object container and an index object container have corresponding contents; and when the internal structure message arrives, calling interfaces of 'extracting dimension combination KEY of the internal structure message' and 'index operation result of the internal structure message'. The vector container (including the dimension object container and the index object container) caches algorithm execution objects, that is, the dimension object container caches dimension objects actually used by the model determined according to the XML file, and the index object container caches index objects actually used by the model determined according to the XML file.
TABLE 2
Referring to fig. 4 and 5, in the present embodiment, a dimension index factory (dimenkpefactory) includes a dimension algorithm class library and an index algorithm class library that are formed by performing code writing based on an internal message structure; wherein, 1) one dimension corresponds to one subclass and is responsible for carrying out dimension value processing on an incoming message, a certain algorithm logic is called to extract the value of the dimension from some fields of an internal structure message, all dimension subclasses form a dimension algorithm class library, and dynamic creation can be carried out in a dimension index factory according to the name of the incoming dimension field; 2) one index also corresponds to one subclass, after receiving the internal structure information, the condition judgment and the value calculation are carried out according to the algorithm logic defined by the index, the single calculation result (index calculation result of the single message) obtained after each index calculation is cached in the single message index calculation result container, and then the single calculation result is transmitted to the dimension index cache container to form the index calculation result obtained by the accumulation of the single calculation result; all the index subclasses form an index algorithm class library, and can be dynamically created in a dimension index factory according to the transmitted index field names.
In this embodiment, the dimension index factory creates all dimension and index algorithm objects used by the model, that is, an overall dimension and overall index object (implemented by a creating unit), according to the attribute data of the model, specifically, a field list (field name) of the attribute data of the model, and when data of different models needs to be read, the unique dimension index factory is also used; extracting an integral dimension object according to the transmitted internal structure body message, performing dimension operation (realized by an extraction unit) through a dimension algorithm class library, and returning the extracted dimension combination KEY to a model data storage and operation module (realized by a return unit); according to the intMsgType of the internal structure body message, screening the whole index object, screening the index object to be executed, distributing the internal structure body message to the screened index object to perform index operation (realized by an operation unit) through an index algorithm class library to obtain an index operation result, wherein the specific realization method is as described in the above embodiment; and returning the dimension combination KEY and the index operation result to a model data storage and operation module (realized by a return unit), inserting the dimension combination KEY and the index operation result into a dimension index cache container in a KEY-value (dimension-index list) mode, and performing summary statistics (accumulating count \ taking maximum or minimum value \ removing the user number) on the dimension index cache container and the cached data.
Referring to FIG. 6, wherein user number deduplication index statistics are reduced by a roaring bitmap (Roaring bitmap) based algorithm to save memory consumption when accounting for such indices, Roaring bitmap improves on the structure of the original bitmap algorithm, which is a high compression ratio and good performance BIT library, specifically: each RoaringBitmap contains a RoaringArray (highLowContainer) as a key-value container; a32-bit integer is divided into upper 16 bits and lower 16 bits, the upper 16 bits are stored as a common key in the short array (defined as short [ ] keys) of highLowContainer, and the lower 16 bits are stored as values in the Container array (defined as Container [ ] values) of highLowContainer. Among them, Container is the core of RoaringBitmap, which has three forms: ArrayCotainer, Bitmapcontainer, and RunCotainer. If the integer quantity to be stored is larger than 4096, a bitmap storage mode needs to be used, namely one key corresponds to an array (bitmap container) of long [1024], so that the principle that 1024 long type numbers represent 2^16 power integers (1024 × 64 ^ 2^16), namely original bitmaps, is used, and otherwise, a short array (array container) needs to be used for storing lower 16 bits. While for consecutive numbers RunContainer storage is used, which only records the initial number and the subsequent number (e.g. for the series 11,12,13,14,15, it will compress to 11,4), there is a better compression effect for consecutive data.
The reason for selecting 4096 this threshold is: because below 4096 bitmap containers may be larger than 16 bits/integer, and above 4096 array containers may exceed 2^16(2^12 ^16 ^ 2^16), occupying space obviously exceeding 2^16, the capacity of the number represented by the lower 16 bits. That is, when the integer radix is small, the use of the array is more space-saving, and when the radix is large, the use of the bitmap is more space-saving. These containers are stored in a dynamic array that shares the 16 most significant bits: they serve as primary indexes. Arrays are used to ensure high 16-bit ordering. We consider the primary index to be generally small. When n is 1000000, it contains up to 16 entities. It should be kept in the CPU cache and the container itself should not use more than 8 KB.
In this embodiment, for each index type with different indexes, after being processed by the index algorithm class library, the return values of the index algorithm classes have the following 3 conditions: 1 or an integer (common accumulated count index) greater than 0, a fixed value (maximum and minimum index), and an IMSI (user de-duplication index); the model data storage and operation module performs further logic operation (accumulation operation (including counting indexes such as 1 addition operation-turn-on times, call drop times and the like, and accumulation of any integer larger than 0, such as a delay index, a flow index and the like), maximum value or minimum value replacement, counting after the number of IMSI users is removed in a duplication mode) on the returned value (a single operation result of each index), so as to obtain the index value of each index, wherein the index value of each index is the index value of the index in the model statistical granularity (namely the index value is the result of counting all relevant internal structure body messages in the model statistical granularity according to the algorithm of the index), and the index values of each index form an intermediate result in the model statistical granularity.
Further as a preferred embodiment, the system also comprises a model management module and a time control module;
the time control module is used for acquiring the statistical granularity time according to the internal structure body message and notifying the mode management module;
the model management module is used for creating and calling a model data storage and operation module object, processing the internal structure body message, responding to the notification of the time control module to control the model data storage and operation module, and outputting the statistical result within the statistical granularity time.
Further, as a preferred embodiment, the system further comprises a stored procedure generation module (generation) and an Export module (Export).
In this embodiment, the model management module (TableManager) creates a model data storage and operation module object (an object pointed by a pointer of the model data storage and operation module) of each model, that is, one model corresponds to one model data storage and operation module; initializing the model data storage and operation module object, storing the initialized model data storage and operation module object into a container, distributing the received internal structure body information to the model data storage and operation module object (equivalent to calling the model data storage and operation module) of each model in the container for processing according to the object and the received internal structure information sent from the XDR adaptation module, responding to the notification of the time control module, and controlling the output of the real-time index statistical result of the model data storage and operation module.
The time control module (Timer) extracts a time field from the internal structure message, performs rounding judgment to obtain the actual statistical granularity time to be output, and informs the model management module when the minute time changes; if the model statistics granularity time is up, the dimension index cache container in the model data storage and operation module can output a real-time index statistics result consisting of the intermediate result and the dimension combination KEY within the statistics granularity time. Then, the time control module outputs the real-time index statistical result to a data transmission queue DataQueue.
Then, through an output module (Export), statistical data is obtained from the data transmission queue DataQueue, the data is formatted according to the model attribute (attribute data TableProperty of the model) and the database format requirement, and the data of each model, namely the final index statistical result, is output according to the classification of the model name and the time.
Referring to fig. 7, a stored procedure generating module (generation) is used for generating stored procedures of various database versions of each model according to the model attribute data tableProperty, wherein the generated stored procedures are called for the database. When the model structure changes (via XML configuration), the database needs to update the stored procedures to load the statistics of each model that the system finally outputs. In this embodiment, a DEELX regular expression engine technology is introduced to realize dynamic generation of storage processes of various database versions of each model, specifically: the system scans the change of the XML file at regular time, if the XML changes, the XML file is loaded and analyzed to obtain attribute data of each model, the attribute data of the model is combined with a template of a database storage process, and a new model storage process is generated through a certain text processing algorithm (DEELX regular expression) such as replacement combination.
The embodiment of the invention also provides a real-time index generation system based on communication XDR, which comprises:
at least one processor;
at least one memory for storing at least one program;
when the at least one program is executed by the at least one processor, the at least one processor is enabled to implement the communication XDR-based real-time indicator generation method.
The contents in the above method embodiments are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above method embodiment, and the beneficial effects achieved by the present system embodiment are also the same as those achieved by the above method embodiment.
In summary, compared with the prior art, the invention has the following advantages:
1) the method comprises the steps that internal structure body information is obtained through XDR data and an internal structure body corresponding to the XDR data, when an XDR specification is upgraded, field assignment algorithms of the internal message structure bodies can be updated one by one according to a field mapping relation between the XDR structure bodies and the internal message structure bodies, the internal message structure bodies corresponding to original XDR data cannot be influenced, an algorithm class library of a dimension index factory is formed by coding based on the internal message structure bodies, the internal message structure bodies are not changed, modification of dimensions and an index algorithm part can be avoided simultaneously, development and testing workload during upgrading of the XDR specification is greatly reduced, the modification period is short, and rapid generation of real-time indexes can be achieved;
2) based on a Model Driven Architecture (MDA) technology, the trouble of the change of an external XDR specification on the development of a dimension index algorithm is eliminated, each XDR is designed with a corresponding internal message structure body, when the XDR specification changes, only an XDR adaptation module needs to be modified, the modification of the dimension and the index algorithm part is avoided, and the development and test workload during the upgrading of the XDR specification is greatly reduced;
3) the method realizes the real-time statistics of various index operation types, not only supports the real-time statistics of accumulation count type and maximum and minimum value type index acquisition, but also realizes the user number deduplication index statistics based on a roaring bitmap (Roaring bitmap) algorithm, and greatly saves the memory consumption when the indexes are counted.
4) The DEELX regular expression engine technology is used for realizing the dynamic generation of the storage process of various database versions of each model, so that the model data can be conveniently led into different databases, and the deployment workload when the model structure changes or the database types change is greatly reduced;
5) the method for configuring the model by using the XML is characterized in that an independent data storage and operation module is designed for each model, the data storage and operation module of each model independently loads the attribute (attribute data of the model) and the dimension index factory, and mapping association is established with the dimension index factory only through the field names (the dimension fields and the index fields-field names), so that the algorithm class of the dimension or the index fields can be called anywhere only by once writing and developing, the XML configuration can be applied to different models by modifying the XML configuration, any program code is not required to be modified for the increase and decrease, field renaming, field data type modification, table division strategy modification and statistical granularity modification of the existing fields of the model, and the requirement for the change of the model structure, the table division strategy and the statistical granularity of a project is quickly responded;
6) the method and the system realize management of the dimension algorithm class library and the index algorithm class library by using a dimension index factory mode, when a caller wants to create or obtain a dimension or index object, a specific object can be created or obtained in a factory as long as the name of the caller is known, the expansibility is strong, and meanwhile, if a specific dimension or index object is added, the addition requirement of the dimension or index can be quickly responded as long as the factory class is expanded.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, while the invention is described in the context of functional modules and illustrated in the form of block diagrams, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated into a single physical device and/or software module or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those skilled in the art can, using ordinary skill, practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The embodiment of the invention also provides a storage medium which stores instructions executable by a processor, and the processor executes the communication XDR-based real-time index generation method when executing the instructions executable by the processor.
It can also be seen that the contents in the above method embodiments are all applicable to the present storage medium embodiment, and the realized functions and advantageous effects are the same as those in the method embodiments.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
In the description herein, references to the description of the term "one embodiment," "the present embodiment," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. The real-time index generation method based on communication XDR is characterized by comprising the following steps:
acquiring attribute data according to the XML file;
obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data;
performing dimension extraction according to the attribute data and the internal structure message, and performing index operation according to the attribute data and the internal structure message;
and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result.
2. The communication XDR-based real-time indicator generation method according to claim 1, wherein: the step of obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data includes the following steps:
providing a corresponding internal structure for each XDR data;
and mapping the field in the XDR data to a corresponding internal structure body to obtain an internal structure body message.
3. The communication XDR-based real-time indicator generation method according to claim 1, wherein: the steps of extracting the dimensionality according to the attribute data and the internal structure information and performing index operation according to the attribute data and the internal structure information comprise the following steps:
according to the attribute data, creating an integral dimension object and an integral index object;
extracting an integral dimension object and performing dimension operation according to the internal structure body message to obtain a dimension extraction result;
and according to the internal structure body message, screening and index operation of the whole index object are carried out to obtain an index operation result.
4. The communication XDR-based real-time index generation method according to claim 3, wherein: the step of obtaining the real-time index statistical result according to the dimension extraction result and the index operation result comprises the following steps:
acquiring statistical granularity time according to the internal structure information;
and obtaining a real-time index statistical result within the statistical granularity time according to the dimension extraction result, the index operation result and the statistical granularity time.
5. Real-time index generation system based on communication XDR is characterized by comprising:
the model configuration module is used for acquiring attribute data according to the XML file;
the XDR adaptation module is used for obtaining an internal structure message according to the received XDR data and an internal structure corresponding to the XDR data;
the dimension index factory is used for dimension extraction and index operation;
and the model data storage and operation module is used for calling the dimension index factory according to the attribute data and the internal structure body information, and obtaining a real-time index statistical result according to the dimension extraction result and the index operation result.
6. The communication XDR-based real-time index generation system according to claim 5, wherein: the dimension index factory includes:
the creating unit is used for creating an integral dimension object and an integral index object according to the attribute data;
the extracting unit is used for extracting the whole dimension object and performing dimension operation according to the internal structure body message to obtain a dimension extracting result;
the operation unit is used for screening the whole index object and performing index operation according to the internal structure body message to obtain an index operation result;
and the return unit is used for returning the dimension extraction result and the index operation result to the model data storage and operation module.
7. The communication XDR-based real-time index generation system according to claim 6, wherein: the model data storage and operation module comprises a single message index operation result container and a dimension index cache container;
the single message index operation result container is used for caching a single operation result of each index after index operation;
the dimension index cache container is used for caching the dimension extraction result and the index operation result, wherein the index operation result is obtained by accumulating the single operation result.
8. The communication XDR-based real-time index generation system according to claim 6, wherein: the system also comprises a model management module and a time control module;
the time control module is used for acquiring the statistical granularity time according to the internal structure body message and notifying the mode management module;
the model management module is used for creating and calling a model data storage and operation module object, processing the internal structure body message, responding to the notification of the time control module to control the model data storage and operation module, and outputting the real-time index statistical result within the statistical granularity time.
9. Real-time index generation system based on communication XDR is characterized by comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, the at least one program causes the at least one processor to implement the method for real-time indicator generation based on communication XDR as claimed in any of claims 1-4.
10. A storage medium storing instructions executable by a processor, wherein: the processor, when executing the processor-executable instructions, performs the communication XDR based real-time indicator generation method as claimed in any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752728.2A CN110633388B (en) | 2019-08-15 | 2019-08-15 | Real-time index generation method, system and storage medium based on communication XDR |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752728.2A CN110633388B (en) | 2019-08-15 | 2019-08-15 | Real-time index generation method, system and storage medium based on communication XDR |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110633388A true CN110633388A (en) | 2019-12-31 |
| CN110633388B CN110633388B (en) | 2021-12-28 |
Family
ID=68969743
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910752728.2A Active CN110633388B (en) | 2019-08-15 | 2019-08-15 | Real-time index generation method, system and storage medium based on communication XDR |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110633388B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111552520A (en) * | 2020-04-10 | 2020-08-18 | 武汉思普崚技术有限公司 | User-defined application identification method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140235306A1 (en) * | 2013-02-19 | 2014-08-21 | David Walls | Systems and methods for a token match game |
| CN105873113A (en) * | 2015-01-21 | 2016-08-17 | 中国移动通信集团福建有限公司 | Method and system for positioning wireless quality problem |
| CN106604224A (en) * | 2016-11-21 | 2017-04-26 | 北京拓明科技有限公司 | XDR-based method and system for obtaining 4G resident ratio index |
| CN108764683A (en) * | 2018-05-21 | 2018-11-06 | 贵州电网有限责任公司 | Operation of power networks data analysis assessment and reporting system |
| CN109241197A (en) * | 2018-06-21 | 2019-01-18 | 深圳市彬讯科技有限公司 | Data processing method, server and the storage medium that index is shown |
-
2019
- 2019-08-15 CN CN201910752728.2A patent/CN110633388B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140235306A1 (en) * | 2013-02-19 | 2014-08-21 | David Walls | Systems and methods for a token match game |
| CN105873113A (en) * | 2015-01-21 | 2016-08-17 | 中国移动通信集团福建有限公司 | Method and system for positioning wireless quality problem |
| CN106604224A (en) * | 2016-11-21 | 2017-04-26 | 北京拓明科技有限公司 | XDR-based method and system for obtaining 4G resident ratio index |
| CN108764683A (en) * | 2018-05-21 | 2018-11-06 | 贵州电网有限责任公司 | Operation of power networks data analysis assessment and reporting system |
| CN109241197A (en) * | 2018-06-21 | 2019-01-18 | 深圳市彬讯科技有限公司 | Data processing method, server and the storage medium that index is shown |
Non-Patent Citations (3)
| Title |
|---|
| ZHAO BING等: "research on xdr bill compression under big data technology", 《INTERNATIONAL CONFERENCE OF PIONEERING COMPUTER SCIENTISTS,ENGINEERS AND EDUCATORS》 * |
| 李训潮等: "信令XDR数据质量智能核查方案研究", 《山东通信技术》 * |
| 郑颖航等: "基于XDR分析互联网业务质量的方案", 《电信快报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111552520A (en) * | 2020-04-10 | 2020-08-18 | 武汉思普崚技术有限公司 | User-defined application identification method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110633388B (en) | 2021-12-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9135572B2 (en) | Method and arrangement for processing data | |
| CN112395293B (en) | Database and table dividing method, database and table dividing device, database and table dividing equipment and storage medium | |
| CN109977099A (en) | Database portion arranging method, user equipment, storage medium and device | |
| CN112464034A (en) | User data extraction method and device, electronic equipment and computer readable medium | |
| CN104216902A (en) | Paging data loading method, device and system based on server paging | |
| CN110955646A (en) | Data storage and query method, device, equipment and medium | |
| CN114328981B (en) | Knowledge graph establishing and data acquiring method and device based on mode mapping | |
| CN112396462A (en) | Crowd circling method and device based on Clickhouse | |
| CN110633388B (en) | Real-time index generation method, system and storage medium based on communication XDR | |
| WO2014137258A1 (en) | Selection of data storage settings for an application | |
| CN111666344A (en) | Heterogeneous data synchronization method and device | |
| CN107526619B (en) | The loading method of format data stream file | |
| CN113254480A (en) | Data query method and device | |
| CN115145552A (en) | Method, device and equipment for executing rule engine and computer readable storage medium | |
| CN113419957A (en) | Rule-based big data offline batch processing performance capacity scanning method and device | |
| CN111639260A (en) | Content recommendation method, device and storage medium thereof | |
| CN112527917A (en) | Data processing method and device | |
| CN116166671B (en) | Memory database table pre-association processing method, system and medium | |
| CN115858699B (en) | Data warehouse construction method and device, electronic equipment and readable storage medium | |
| CN114064716B (en) | Metadata-based automatic WEB report generation method and device | |
| CN115934806B (en) | Statistical method, device, equipment and medium for data deduplication based on RBM | |
| CN115576947A (en) | Data management method and device, combined library, electronic equipment and storage medium | |
| CN115080047A (en) | Function optimization method and system for application program | |
| CN117668060A (en) | Report generation and query method, device, equipment, system and storage medium | |
| CN114676154A (en) | Data processing method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211209 Address after: 510630 No. 6, 1101, building 1, No. 16, Keyun Road, Tianhe District, Guangzhou City, Guangdong Province (office only) Applicant after: Guangdong Yitong Hengrui Technology Co.,Ltd. Address before: 510630 room 1101, building 1, No.16 Keyun Road, Tianhe District, Guangzhou City, Guangdong Province (office use only) Applicant before: YITONG CENTURY INTERNET OF THINGS RESEARCH INSTITUTE (GUANGZHOU) Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |