RU2688229C1 - Method of aggregation and conversion of data and device for its implementation - Google Patents
Method of aggregation and conversion of data and device for its implementation Download PDFInfo
- Publication number
- RU2688229C1 RU2688229C1 RU2017145520A RU2017145520A RU2688229C1 RU 2688229 C1 RU2688229 C1 RU 2688229C1 RU 2017145520 A RU2017145520 A RU 2017145520A RU 2017145520 A RU2017145520 A RU 2017145520A RU 2688229 C1 RU2688229 C1 RU 2688229C1
- Authority
- RU
- Russia
- Prior art keywords
- data
- information
- type
- properties
- aggregation
- Prior art date
Links
- 230000002776 aggregation Effects 0 abstract claims description title 41
- 238000004220 aggregation Methods 0 abstract claims description title 36
- 238000006243 chemical reaction Methods 0 abstract claims description title 10
- 238000000034 methods Methods 0 abstract claims description 67
- 230000001131 transforming Effects 0 abstract claims description 17
- 238000003860 storage Methods 0 abstract claims description 8
- 230000000875 corresponding Effects 0 abstract claims description 4
- 239000003138 indicator Substances 0 claims description 43
- 239000000463 materials Substances 0 claims description 12
- 238000004458 analytical methods Methods 0 claims description 9
- 230000015654 memory Effects 0 claims description 7
- 230000015572 biosynthetic process Effects 0 claims description 4
- 238000005755 formation Methods 0 claims description 4
- 238000009740 moulding (composite fabrication) Methods 0 abstract description 3
- 239000000126 substances Substances 0 abstract description 2
- 230000000694 effects Effects 0 abstract 1
- 239000000203 mixtures Substances 0 description 6
- 239000011248 coating agents Substances 0 description 3
- 238000000576 coating method Methods 0 description 3
- 238000004891 communication Methods 0 description 2
- 230000000670 limiting Effects 0 description 2
- 238000005365 production Methods 0 description 2
- 238000000844 transformation Methods 0 description 2
- 230000000007 visual effect Effects 0 description 2
- 239000004793 Polystyrene Substances 0 description 1
- 238000009825 accumulation Methods 0 description 1
- 230000033228 biological regulation Effects 0 description 1
- 230000001721 combination Effects 0 description 1
- 238000005516 engineering processes Methods 0 description 1
- 239000000446 fuel Substances 0 description 1
- 230000003993 interaction Effects 0 description 1
- 239000011159 matrix materials Substances 0 description 1
- 238000005259 measurements Methods 0 description 1
- 230000003287 optical Effects 0 description 1
- 239000000047 products Substances 0 description 1
- 230000004224 protection Effects 0 description 1
- 238000001228 spectrum Methods 0 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
Abstract
Description
The invention relates to the field of information technology and is intended for aggregation and transformation of data, obtaining their intermediate and consolidated values.
BACKGROUND
Various prior art database management systems (DBMS) are known in the art that control the creation and use of universal data warehouse devices, in particular, databases, as well as various data aggregation systems.
Storage of multidimensional data
Database systems are often designed to support a vast amount of information about various entities or events, and these cases can be described by various characteristics. Even database systems that do not yet contain a huge amount of information are often designed so that they can be scaled, so that database systems can be adapted to accommodate a huge amount of information. Huge tables, which may include every event and every characteristic of every event, may not be possible for analysis if there are not enough resources to store and process significant parts of these tables. Even with sufficient resources, storing and processing large parts of these huge tables can be quite expensive. As a result, when occurrences have many characteristics or are otherwise associated with a variety of information, many database systems divide such event information into several tables.
Database systems often group tables based on categories of characteristics. Much of the information can be descriptive information about the objects, categories, or classes of information (generally referred to as categories) involved in the events. The description of these basic categories may change infrequently compared to other tables that record or measure the events themselves. Dimension tables are tables containing descriptive information about events that other tables (tables) refer to or can refer to. The other table (s) contains a column (s) that refer to the rows (s) of the size table, and each reference column indicates what is called the size of the column (s) that are found in the table (s). Data that is organized into two or more dimensions is referred to here as multidimensional data.
Fact tables are other tables that measure category related events. In other words, fact tables store facts or measured quantitative data, and these measurable data can be related to categories or otherwise fall into these categories. When comparing dimension tables, fact tables should not duplicate all the information contained in dimension tables. As a rule, since fact tables may include several cases (s) that refer to the same category, fact tables are usually larger than dimension tables. In addition, since fact tables measure events and do not record definitions, fact tables are usually updated more often than measurement tables. The organization of multidimensional data in tables (facts) of facts and dimension tables is called a star chart.
Data stored in relational database systems (ROLAP systems), for example, according to a star scheme, is available for all applications that support interaction with such relational systems. Such database applications interact with a relational database, sending commands that correspond to a database language supported by a relational database system, the most common of which is structured query language (SQL).
Alternatively, multidimensional data can be stored in specialized multidimensional database systems (MOLAP systems). Multidimensional database systems provide access structures and methods specifically designed for multidimensional data. When data is stored in specialized multidimensional database systems, only applications that are specifically designed to interact with these multidimensional database systems have access to and control of data.
An alternative approach to managing multidimensional data in a relational database includes storing data in relational files, but storing all multidimensional structures, metadata, administration and access control using multidimensional database system methods. Access to relational-stored data using multidimensional methods creates numerous difficulties. For example, when all administration and access to multidimensional data is managed exclusively through the multidimensional database system mechanism, two database management systems need to be administered. In addition, database applications that access data using traditional relational commands (for example, SQL commands) cannot access multidimensional data.
In particular, in the patent US7680776 B2 (Microsoft Corporation, publ. 03/16/2010) a relational database system (ROLAP system) and a multidimensional database system (MOLAP system) are disclosed.
The disadvantage of such systems is poor performance, because The data is processed by the OLAP server. Another disadvantage is the limitation of functionality due to the use of SQL. The disadvantages of MOLAP systems include limiting the amount of processed data and data redundancy, because for the formation of multidimensional cubes, in various aspects, the data must be duplicated.
DISCLOSURE OF INVENTION
This technical solution is aimed at eliminating the disadvantages inherent in the existing solutions of the prior art.
This technical solution is aimed at expanding the arsenal of technical means for a particular purpose, in our case - expanding the arsenal of technical means of automatic aggregation and data conversion, and as a technical result achieved by the stated solution, the implementation of the specified purpose declared by the invention may be, namely, the implementation of aggregation and data transformations on individual properties of each information object.
To achieve the above technical result, a method for aggregating and converting data has been developed, comprising the steps of using data processing devices and receiving data from a command processing device.
- determine these sources and the format for presenting data on the current state of the values of the properties of objects in the domain that come from at least one source of data for processing in a processing device, in which:
- form the tree structure of the information object, which may include other information objects, with each property included in the information object and each information object has its own unique code;
- form three types of data containers, which are nodes, organized as a combination of sets of interconnected memory cells, wherein:
the first type is the data containers of the collection of primary indicators of information objects and processes performed,
the second type - data containers for collecting intermediate data on information objects and running processes,
the third type - data containers of collecting consolidated information on information objects and processes, with three types of data containers organized in the form of invariant information structures;
- in the processing device receive the primary indicators characterizing the current state of the values of the properties of information objects and processes performed, the metamodel of each of the running processes contains information about the name of the process and its stages, their characteristics, start and end time of the stages and the process as a whole, state description information object at the time of the beginning and completion of the stages of the process, as well as the values of active and latent indicators of the results of their execution;
- write down each property of the information object and the executed processes in the corresponding container of the first type of storage device;
- in the processing device, aggregation and transformation of data is carried out according to individual properties of each information object in accordance with the specified conditions for subsequent comparison and analysis of the incoming values of the properties of information objects for different time periods;
- transfer the aggregated and transformed data to the second type of containers when a new record appears in the container of the first type and assign them the value of intermediate data;
then consolidated processing of these aggregated and transformed data is carried out, and the consolidated data is placed in data containers of the third type.
In some embodiments, the implementation of the technical solution, each container consists of a group of metamodel tables, their primary keys, indexes, and relationships between tables, which are invariant to the type and type of data stored in it.
In some embodiments, the implementation of technical solutions properties of the information object can be assigned to one of two groups:
the first group contains properties that characterize the stable parameters of the state of a material object, the model of which forms an information object;
the second group contains properties, the values of which may be subject to change and are indicators of the state of the controlled material object.
This technical result is achieved through the device aggregation and data conversion, containing:
at least one data processing device designed to implement aggregation and transformation of data on individual properties of each information object in accordance with specified conditions for subsequent comparison and analysis of incoming values of properties of information objects for different time periods with the formation of data containers, each of which represents a node of interconnected memory cells of the data storage node and the recording of each property of the information object in Resp container, and
a command processing device for performing a method of aggregating and transforming data.
BRIEF DESCRIPTION OF THE DRAWINGS
For a better understanding of the invention, and to more clearly show how it can be implemented, reference will be made further, only as an example, to the accompanying drawings, in which:
FIG. 1 - structure of the stream of primary data of the process / stage performance;
FIG. 2 - an example of the aggregation of the values of the state indicator of one of the properties of the control object over time periods (horizontal aggregation);
FIG. 3 is an example of the representation of the structure of data containers for a particular control object by time periods (horizontal aggregation);
FIG. 4 shows an example of aggregation of data over time periods;
FIG. 5 - an example of a visual representation of the aggregated data on the example of the resources used.
IMPLEMENTATION OF THE INVENTION
The device for aggregating and converting data can be performed on the basis of a wide range of electronic computing devices, for example, a personal computer, laptop, server cluster, etc.
The data processing device can be made in the form of a processor that performs the main computational work when implementing the steps of the claimed method.
The storage device can be a hard disk (HDD), solid-state drive (SSD), flash memory (NAND-flash, EEPROM, Secure Digital, etc.), optical disk (CD, DVD, Blue Ray), mini disk or their combination.
Input / Output (I / O) interfaces are standard ports and interface devices for devices and data transmission, selected on the basis of the required configuration of the execution of the aggregation and data conversion device, in particular: USB (2.0, 3.0, USB-C, micro, mini) , Ethernet, PCI, AGP, COM, LPT, PS / 2, SATA, FireWire, Lightning, etc.
I / O facilities are also selected from a known spectrum of various devices, for example, a keyboard, touchpad, touchscreen display, monitor, projector, mouse, joystick, trackball, light pen, stylus, audio output devices (speakers, headphones, built-in speakers, buzzer) etc.
Means of data transmission are selected from devices designed to implement the process of communication between various devices via wired and / or wireless communication, in particular, such devices can be: GSM modem, Wi-Fi transceiver, Bluetooth or BLE module, GPS module, Glonass module, NFC, Ethernet module, etc.
Data on the current state of the values of the properties of the domain objects may contain information about information objects, processes and its stages, as well as indicators (Fig. 1). The property of the object of the domain is understood as a quantitative or qualitative characteristic that determines the current state of the object in the system.
An information object describes the structure of a material object, defining its individual properties or a group of interrelated properties, and can be represented in the form of a tree structure, which may include other information objects. Each property that is part of an information object and each information object has its own unique code as a number from the natural number of integers. All properties of an information object can be assigned to one of two groups.
The first group contains properties that characterize stable (not calculated) state parameters of a material object, the model of which forms an information object and allows viewing it from various points of view (example: size and weight of part, material composition, etc.).
The second group contains properties whose values may be subject to changes (calculations and / or measurements) and are indicators of the state of the controlled material object (for example, the spindle rotation speed, the engine temperature, the number of units produced, etc.).
An example of a set of properties of the domain object:
- object - detail;
- properties - part designation, part name, material grade of the part, overall dimensions of the part, maximum and minimum quality of the characteristic dimensions of the part, part weight, chemical coating, number of parts in stock and work in progress at the current time.
Indicators of the state of the material object are determined by the results of the execution of the process and / or its stages and form data streams.
Each process and each stage, in addition to indicators of the state of the material object, contains indicators of resource support for management objects and indicators of the state and targets of the process / process stages, on the basis of which the performance indicators of the process / process stages and the efficiency of the process performers / process stages are calculated.
Each process or stage of the process contains data streams on one or several information objects involved in it, which can be obtained in automatic mode from instruments, from controllers, from computer memory. This data flow is the primary indicators of information objects and processes performed, the meta model of each of the processes contains information about the name of the process and its stages, their characteristics, the start and end time of the stages and the process as a whole, the description of the state of the information object at the start and completion of the process steps, as well as the values of active and latent indicators of the results of their execution.
Examples of data on the current state of the properties of objects in the domain: the meter reading at the beginning of the shift and at the end of the shift; the speed of the transport unit on the route; the number of tools in the machine shop, requiring replacement, etc.
Depending on the level of detail of the material object, the structure of the information object may contain a different number of properties.
Some of the properties included in the structure of an information object can receive the status “Active”, the other part - the status “Latent”.
The status “Active” is assigned to indicators whose values are subject to further automatic transformation, forming intermediate and consolidated data for a specific point in time or for a specific period of time, necessary for generating statistical samples and / or making management and other decisions.
The status "Latent" is assigned to indicators, the values of which are recorded in the data warehouse without any subsequent processing.
The flow of primary indicators of information objects, namely, each individual property of the information object and the processes being executed is recorded in the corresponding container of the first type.
At the same time, there are three types of data containers:
the first type is the data containers of the collection of primary indicators of information objects and processes performed,
the second type - data containers for collecting intermediate data on information objects and running processes,
the third type is data containers of collecting consolidated information on information objects and processes, with three types of data containers organized in the form of invariant information structures.
Each of the three types of data containers is a node organized as a combination of sets of interconnected memory cells that allow storing information about objects of arbitrary structure and complexity in their relationships, and consisting of a group of table meta-models, their primary keys, indexes and relationships between tables. , are invariant to the type and type of data stored in it, as well as methods of working with this data.
Carry out the aggregation and / or conversion of data on individual properties of each information object in accordance with the specified conditions for the subsequent comparison and analysis of the incoming values of the properties of information objects for different time periods.
There are:
- horizontal aggregation (enlargement) of data along the time axis, taking into account the results of transactions, as well as the range of aggregation of a group of records over time periods, for example, the number of products produced per shift / per day / per week / per decade / per month / per quarter / half-year / year, etc .;
- vertical aggregation (enlargement) of data along the axis of control objects, for example, the amount of energy consumed per shift by a machine / production site / production hall / plant;
- vertical aggregation (enlargement) of data along the axis of the processes performed, for example, the laboriousness of preparing the curriculum of the discipline, the laboriousness of preparing the curriculum of the specialty, etc .;
- vertical aggregation (enlargement) of data along the axis of achieving goals in accordance with a given goal tree, for example, the costs of achieving goals of the third level / second level / first level;
- a combination of horizontal and vertical aggregation (aggregation) of data.
Aggregation and transformation of data is determined by the structure of the process, the structure of the information object and methods of working with data.
With regard to data having the status of “Active” depending on the type of indicators, the invariant information structure of the data container can use the following methods of working with data:
- method of working with indicators of the state of properties of the control object,
- method of working with indicators of resource support of the control object in the process and / or at the stage of the process execution,
- method of working with performance indicators (achievement of goals) of the process and / or stage of the process,
- method of working with performance indicators of the process and / or stage of the process,
- method of working with performance indicators of the process performer and / or stage of the process.
Structuring the data processed by the proposed methods and their subsequent visualization for each indicator is presented in the form of matrices (Fig. 5), each of which contains the following information:
- name and number of the process;
- the name and number of the process step;
- date and time of the beginning of the process stage;
- date and time of completion of the process stage;
- name and code of the control object;
- name and identification code of the indicator;
- status of the indicator (active / latent);
- type of indicator (indicator of the state of properties of the control object, indicator of resource provision of the control object, performance indicator, performance efficiency indicator, performer performance indicator);
- the value of the indicator after the execution of the process stage (the value achieved / the difference between the initial and final value / the resulting value by accrual for the period of the process stage and other data);
- the standard value of the indicator (planned or established as a target);
- absolute deviation (the difference between the actual and the standard value, determined by a given formula);
- the coefficient of deviation (the ratio of the actual value of the indicator to the standard indicator and other data).
The metamodel of the invariant information structure of aggregation and data transformation includes information about the information object, active process, stage of the process, active indicators of its execution, regulation of aggregation and transformations, algorithms for computing (grouping by a time period, grouping by a separate property of the object, etc. ).
The container invariant is designed so that the period of aggregation and data conversion, as well as the formulas by which the data are aggregated and transformed, are set in advance and automatically according to a predefined pattern or can be rearranged in the process of complicating data structures.
The complexity of the information object does not affect the structure of the container.
The number of container types corresponds to the number of indicators -analyzed properties of control objects at each stage of active processes. For example, for the parameter "time" it is the sequence of days, months, quarters, years (see Fig. 2).
When new processes are activated, new data containers are automatically generated.
The accumulation of primary data in data containers is carried out as the processes and / or its steps are executed. The amount of data in the data container corresponds to the number of processes executed and / or the stages of the processes.
As data is accumulated, the need for their aggregation and transformation is formed for the purpose of subsequent comparison and analysis for different time periods for one or several control objects within one or several processes and / or control functions (Fig. 4).
Carry out the aggregation and transformation of data on individual properties of each information object in accordance with the specified conditions for subsequent comparison and analysis of incoming values of information objects properties for different time periods and transfer the aggregated and transformed data to the second type of containers when a new record appears in the first type container and assign them the value of intermediate data.
Next, consolidated processing of these aggregated and transformed data is carried out, and the consolidated data is placed in data containers of the third type. Containers of the third type store consolidated data obtained as a result of aggregation and transformation of intermediate data, which are used to make management and other decisions, as well as to monitor their effectiveness in accordance with the strategic, tactical and operational objectives of the management system.
An example of the format for the presentation of information objects is a tree-like structure. An example of the formation of the structure of an object is a simple and complex reference book. A help system is described as a set of invariant information structures for storing data about information objects with specific content.
An example of the structure of a simple directory: directory name is a table of bolts; properties (attributes): bolt designation, No. of GOST, thread diameter, bolt length, bolt thread length, turnkey size, type of coating, coating thickness, weight 100 pieces.
An example of the structure of a complex directory: directory name - directory of transport units; bus brand, garage number, state registration number, type of fuel used, engine number, drivers' full name, driver's personnel number, driver's license number, time and place of training and internships.
Combinations of tree structures create complex information structures. There are five degrees of difficulty connecting information objects.
Processing and aggregation of data is configured in a convenient user dialogue by using universal processing methods (reading a record from an invariant information structure, writing data to an invariant information structure, correcting data in an invariant information structure, selecting several records from an invariant information structure according to predetermined conditions) and visual representation of a data set in the form of data mapping in matrix form, in the form of a multidimensional cube, in gr aphic view (diagrams, dependency graphs and the like).
The proposed device allows you to aggregate and convert data on individual properties of each information object in accordance with the specified conditions for subsequent comparison and analysis of incoming values of properties of information objects for different time periods, as well as automatically organize invariant information structures for storing these data, while invariant information structures made in the form of three types of containers, each of which is an interconnection node nnyh memory cell storage node.
The description of the claimed invention discloses the preferred versions of the claimed solution and should not be construed as limiting other, private implementation options, not beyond the scope of the requested legal protection, which should be clear to a person skilled in the technical field.
Claims (19)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| RU2017145520A RU2688229C1 (en) | 2017-12-25 | 2017-12-25 | Method of aggregation and conversion of data and device for its implementation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| RU2017145520A RU2688229C1 (en) | 2017-12-25 | 2017-12-25 | Method of aggregation and conversion of data and device for its implementation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| RU2688229C1 true RU2688229C1 (en) | 2019-05-21 |
Family
ID=66636556
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| RU2017145520A RU2688229C1 (en) | 2017-12-25 | 2017-12-25 | Method of aggregation and conversion of data and device for its implementation |
Country Status (1)
| Country | Link |
|---|---|
| RU (1) | RU2688229C1 (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2103728C1 (en) * | 1995-10-24 | 1998-01-27 | Владимир Олегович Сафонов | Method for converting input program of translator and device which implements said method |
| US20020029207A1 (en) * | 2000-02-28 | 2002-03-07 | Hyperroll, Inc. | Data aggregation server for managing a multi-dimensional database and database management system having data aggregation server integrated therein |
| US20060129597A1 (en) * | 2003-03-28 | 2006-06-15 | Microsoft Corporation | Systems and methods for proactive caching utilizing olap variants |
-
2017
- 2017-12-25 RU RU2017145520A patent/RU2688229C1/en active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2103728C1 (en) * | 1995-10-24 | 1998-01-27 | Владимир Олегович Сафонов | Method for converting input program of translator and device which implements said method |
| US20020029207A1 (en) * | 2000-02-28 | 2002-03-07 | Hyperroll, Inc. | Data aggregation server for managing a multi-dimensional database and database management system having data aggregation server integrated therein |
| US20060129597A1 (en) * | 2003-03-28 | 2006-06-15 | Microsoft Corporation | Systems and methods for proactive caching utilizing olap variants |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Holzschuher et al. | Performance of graph query languages: comparison of cypher, gremlin and native access in Neo4j | |
| Vaisman et al. | Data warehouse systems | |
| CN101916261B (en) | Data partitioning method for distributed parallel database system | |
| Carey et al. | Data-Centric Systems and Applications | |
| US8244743B2 (en) | Scalable rendering of large spatial databases | |
| Folk et al. | An overview of the HDF5 technology suite and its applications | |
| CN104205039B (en) | Drive data using interest pipelines for data analysis and business intelligence systems drive interest method | |
| US20050278346A1 (en) | System Providing Methodology for Replication Subscription Resolution | |
| US9361340B2 (en) | Processing database queries using format conversion | |
| US7774371B2 (en) | Data processing system | |
| US9009201B2 (en) | Extended database search | |
| US20050065952A1 (en) | Extensible framework supporting deposit of heterogenous data sources into a target data repository | |
| CA2603901C (en) | System and methods for facilitating a linear grid database with data organization by dimension | |
| Sumbaly et al. | The big data ecosystem at linkedin | |
| US8341131B2 (en) | Systems and methods for master data management using record and field based rules | |
| US20050055289A1 (en) | Multi-dimensional business information accounting software engine | |
| US20050010550A1 (en) | System and method of modelling of a multi-dimensional data source in an entity-relationship model | |
| US7730059B2 (en) | Cube faceted data analysis | |
| US7805465B2 (en) | Metadata management for a data abstraction model | |
| US7925658B2 (en) | Methods and apparatus for mapping a hierarchical data structure to a flat data structure for use in generating a report | |
| AU2014201593B2 (en) | Shared cache used to provide zero copy memory mapped database | |
| US20150286681A1 (en) | Techniques for partition pruning based on aggregated zone map information | |
| US8185509B2 (en) | Association of semantic objects with linguistic entity categories | |
| EP1394696A2 (en) | Query interface for OLAP cubes | |
| CA2551030A1 (en) | System and method for translating between relational database queries and multidimensional database queries |