EA036384B1 - Method and system for integrated management of big data - Google Patents

Abstract

The invention relates to management of large volumes of data, in particular, to a system and a method for their processing and unified storage. The achieved technical result matches the technical problem solved, and consists in increasing the efficiency of storage of large volumes of data due to application of a distributed storage architecture with ensuring input data processing for their unification. The claimed solution is implemented using a system for integrated management of big data (Big Data), comprising a subsystem for input data transportation and checking (hereinafter - TFS), subsystem for external data loading (hereinafter - xStream), functioning on the basis of the data description standard, wherein: TFS receives, checks and transfers data received from data sources to xStream, TFS receiving data in an archived form and transferring data via a transport protocol; xStream contains a subsystem for file loading, a subsystem for data processing and archiving, a subsystem for access to the data (hereinafter - hCatalog), data loading control module and access control module, wherein in xStream: the subsystem for file loading interrogates TFS for transfer of data received from sources to the data processing and archiving subsystem, data sources are pre-registered in the data loading subsystem, the data loading subsystem carries out primary copying of files received from TFS to the processing and archiving subsystem, the processing and archiving subsystem contains data storage areas performing structural storage of primary data, intermediate data and published data; files from the data loading subsystem corresponding to the set parameters of archiving are transferred to the primary data storage area, and stored in primary archived form; the processing and archiving subsystem is able to transfer de-archived primary data transformed to the parquet format for further conversions to the intermediate data area; prepared, transformed, unified data reduced to xStream standard are transferred to the published data area, and unified data from the published data area are used to form a structured catalog for access to the mentioned data by means of hCatalog.

Description

Technology area

The invention relates to the management of large amounts of data, in particular to a system and method for processing and unified storage.

State of the art

From the prior art, various solutions for organizing data storage systems based on distributed file systems (HDFS / Hadoop File System) are known, which, in particular, are used for working with big data. The term Big Data refers to a huge amount of data entering the storage system. Data usually comes from many sources of information in an unstructured form. Big Data also includes technologies for their processing and use, methods for finding the necessary information in large arrays.

The main problem of big data is associated with the large amount of information that needs to be processed and, accordingly, stored in a trusted way, minimizing the risk of data loss. In this regard, there is a need for data backup, which implies the organization of voluminous data storage structures. However, with the increase in the amount of information, the difficulties with its reservation grow.

HDFS, or Hadoop Distributed File System, is the primary storage system used by Hadoop applications. HDFS copies blocks of data many times and distributes these copies among the compute nodes of the cluster, thereby ensuring high reliability and speed of computations.

The prior art knows the architecture of a data warehouse for providing shared access to data stored in it (US 20130262615, 03.10.2013). A data warehouse can be implemented using HDFS and provide a filter for incoming information for its primary processing while directly saving for later use. The storage system functions by forming copies of incoming data from heterogeneous sources for their subsequent linking using metadata for the purpose of accessing them.

CN 106407309 (Wei et al., 15.02.2017) describes a cluster storage mechanism for retrieving information from many different data sources and providing an authentication process between the database and data sources.

The known solution has significant drawbacks, which consist in the lack of processing of incoming data in order to transform them into a unified form, which does not allow to promptly update information from sources, and reduces the efficiency of the data warehouse. Also, the absence of a data unification process leads to low fault tolerance of the storage system in the event of data arriving in an incorrect or unsupported format (transport format errors, archiving errors, copy errors, data structure errors, data type errors).

Disclosure of invention

The technical problem to be solved using the claimed method and system is to eliminate the disadvantages inherent in known analogues, as well as to create a new principle of efficient storage of large amounts of data.

The achieved technical result coincides with the technical problem being solved and consists in increasing the efficiency of storing large amounts of data through the use of a distributed storage architecture with the provision of processing input data in order to unify them.

An additional technical effect is to increase the fault tolerance of the data storage system due to the architecture of the big data storage management system and ensure the unification of stored data.

The invention is carried out using a system of integrated management of big data (Big Data), containing a subsystem for transporting and checking input data (hereinafter TFS), a subsystem for loading external data (hereinafter xStream), operating on the basis of a data description standard, in which

TFS receives, checks and transmits data in xStream from data sources, and TFS receives data in an archived form and transfers data using a transport protocol;

xStream contains a file loading subsystem, a data processing and archiving subsystem, a data access subsystem (hereinafter referred to as hCatalog), a data loading control module and an access control module, and in xStream, the file download subsystem polls TFS to transfer data received from sources to the processing subsystem and archiving the data, the data sources being pre-registered with the data loading subsystem;

data loading subsystem performs primary copying of files from TFS to the processing and archiving subsystem;

the processing and archiving subsystem contains data storage areas that carry out structured storage of primary data, intermediate data and published data;

files from the data loading subsystem are transferred to the primary data storage area, which

- 1,036384 correspond to the set archiving parameters and are stored in the originally archived form;

the processing and archiving subsystem is configured to transfer to the intermediate data area the unzipped primary data converted into the parquet format for further transformations;

prepared, transformed, unified data converted to the xStream standard are transferred to the published data area, in which the names of tables and fields are converted to the xStream standard, the data types are converted to the xStream standard, the tables are supplemented with mandatory technical fields describing the delivery of data - dates of the beginning and end of the period the relevance of the data supply, the numbers of the streams that performed the loading and conversion;

and from the unified data of the published data area, a structured catalog is formed for accessing said data through hCatalog.

In a particular implementation of the TFS system, it checks the integrity of the received archived data.

In another particular implementation of the system for registered sources, the ID of said sources is stored in the download control module.

In another particular embodiment of the system, the download control module controls the data flow of the plurality of sources according to the corresponding stored IDs.

In another particular embodiment of the system, for each data source, the load control module contains data load parameters.

In another particular embodiment of the system, the data loading subsystem loads files into the processing and archiving subsystem based on the file loading mask.

In another particular embodiment of the system, the file download mask is formed based on at least the name of the downloaded archive file.

In another particular embodiment of the system, a directory is formed in the processing and archiving subsystem in each of the data storage areas for storing data of the corresponding source.

In another particular implementation of the system, the download control module checks the availability of information for all sources in TFS.

In another particular implementation of the system, full or incremental data loading from TFS is performed.

In another particular embodiment of the system, the incremental loading is performed by the data loading module when it detects new data in TFS that differs from the previously supplied files by the archive delivery date.

In another particular implementation of the system, the processing and archiving subsystem processes parquet files to bring them into conformity with Hive-SQL types.

In another particular embodiment of the system, files are checked for the presence of previously stored similar data when processed by the processing and archiving subsystem.

In another particular embodiment of the system, in the presence of earlier data, the current version of the data is transferred to the area of published data, with the transfer of the previous version to the history storage directory with partitioning by the date of loading.

In another particular implementation of the system, hCatalog provides a mapping of the data structure of the data publication area.

In another particular implementation of the system, the data structure is divided into databases corresponding to the data sources.

In another particular embodiment of the system, the data processing and archiving subsystem additionally provides an automated function of loading data for any date in the past.

The claimed solution is also implemented using a method for managing Big Data using the input data transport and verification subsystem (hereinafter TFS) and the external data loading subsystem (hereinafter xStream), and xStream consists of a file upload subsystem, a processing and archiving subsystem, a subsystem access to data, a data download control module and an access control module, the method comprising the steps of interacting with the TFS using the xStream download control module to receive data from said sources, the data sources being pre-registered with the data download control module;

receive data from said sources using TFS, which receives data in an archived form and accumulates and verifies the data; in case of successful verification of said data, they are transferred to the data loading subsystem via a transport protocol;

using a subsystem for processing and archiving data, processing the received data, which includes accumulating files, checking files, unpacking archived files that have been checked and converting the unpacked files to parquet format;

structure the converted files by placing them in directories,

- 2,036384 each of which is associated with data sources registered in said data download control module;

control and delete duplicate data, control and delete data with a broken structure, convert data types to Hive-SQL, control and update the current data catalog, control and update the data change history catalog, control and manage the depth of the data archive;

provide users with access to data located in the data access subsystem using the access control module.

In one of the particular examples of the implementation of the method, the access control module determines a set of functionality based on the user's access level.

In another particular example of the implementation of the method, the data processing and archiving subsystem processes files of the parquet format for their correspondence to the Hive-SQL type.

In another particular example of implementation of the method, data sources are registered using the source ID entry in the download control module.

In another particular example of the implementation of the method, the download control module controls the data flow of the plurality of sources according to the corresponding stored IDs.

In another particular example of the implementation of the method, data flow control includes checking the availability of information from a data source in TFS, processing messages from TFS, and performing actions based on message processing.

In another particular example of the implementation of the method, full or incremental data loading from the input data primary processing subsystem is performed.

In another particular example of implementation of the method, the incremental loading is performed when the loading module detects the presence of new data.

In another particular example of the implementation of the method, for each data source, the load control module contains parameters for loading data.

In another particular example of implementation of the method for uploading files to the processing and archiving subsystem, it is carried out based on the file upload mask.

In another particular example of the implementation of the method, the file download mask is formed based on at least the name of the downloaded archive file.

Description of drawings

The features and advantages of the technical solution will become apparent from the following detailed description and the accompanying drawings, in which: FIG. 1 illustrates a claimed integrated big data management system;

fig. 2 is an example of a source identifier;

fig. 3 is a general view of the user's computing device;

fig. 4 is a general diagram of networking.

Implementation of the invention

FIG. 1 shows the general scheme of implementation of the claimed system (100) for managing big data. The main functional element of the claimed system (100) is a subsystem (170) for loading external data - xStream, which interacts (or is part of) with a cloud platform (OP) (180) for storing and managing data. XStream is a framework for providing a centralized infrastructure for receiving, processing and storing external data in a Hadoop-based OP for further distribution to all stakeholders for use in business processes and in research for value extraction. Data sources (110) can represent various resources and channels for providing information, for example, social networks, accounting systems, CRM systems, registers of government agencies (USRLE, ROSSTAT, FFMS, etc.), etc. The information coming from the data sources (110) is initially processed by the input data transport and validation subsystem (hereinafter TFS / Transport File System) (120). TFS (120) receives, checks and transmits data from data sources (110) into xStream (170) via a transport protocol. As a protocol for transferring information from TFS (120) to xStream (170), the Network File System (NFS) protocol can be used.

The data arrives at the TFS input (120) in an archived form.

The xStream subsystem (170), in turn, consists of a file loading subsystem (130), a data processing and archiving subsystem (140), a data access subsystem (150), and a technological core (160).

The data received at the TFS input (120) is transferred to the file upload subsystem (130), which transports data archives using the TFS integration module (131) (120) to the primary data storage layer, the subsystem (140).

For each of the data sources (110), two types of loading are assumed.

Initial download is an initialization download that runs once and grabs all archives available in TFS (120) whose filenames meet vendor specifications.

- 3 036384

Regular download - performed incrementally, during which only archives that have not been downloaded earlier are captured.

The capture of archives occurs from the TFS catalogs. When accessing TFS (120), authentication is performed using a technical account and a private key. Copying data from TFS (120) to the primary storage layer xStream (170) is done by the download control module (161) initiating requests for data from the TFS (120). TFS (120) specifies the download routes for data and control messages between the module (161) and the TFS (120). The module (161) can initiate the download of one or several data streams simultaneously, while the streams will be processed in parallel.

Each data source (110) is registered in the module (161). For each source, its identifier (ID) is created and stored. In the process of registering a data source (110), the following procedure is performed;

assigning a number to the data source (110);

defining schemas within the source (110) and assigning a number to them. Schema is optional and is used to logically group tables;

assigning a number to all tables within the source (110) within the schema (if used).

Source (110), schemas, tables correspond in module (161) to entities with unique identifiers. The identifier is an integer of the format shown in FIG. 2.

The source number is the internal surrogate source identifier (110) in xStream (170), which is generated at the stage of connecting the source (110) to xStream (170).

The data area indicates the storage area of the received information in xStream (170):

- area of published data (143);

- raw data area (141);

- area of intermediate data (142);

- basic entity identifying source data. A schema is an identifier for a schema in a source or subsystem. Generated at the stage of connecting source (110) to xStream (170). The table number represents the number of the table in the schema, also generated at the stage of connecting the source (110) to xStream (170).

The data processing and archiving subsystem (140) is a storage of data received from external sources (110) and is a logical area in HDFS. The access to the stored data is carried out through the access subsystem (150) hCatalog, which is also designed to publish metadata about data. The data processing and archiving subsystem (140) contains data storage areas that carry out structured storage of primary (raw) data (141), intermediate data (142) and published data (143). Archives of information received from TFS (120) from data sources (110) registered in module (161) are transferred to the primary data storage area (141). The received data is checked for integrity in TFS (120). In case of violation of the integrity of archives received from data sources (110), such information is not transferred to the xStream subsystem (170). Upon successful copying of information from the TFS (120) to the primary data area (141), xStream (170) notifies of a successful operation.

When loading data into the intermediate data storage area (142), the data archives are initially unzipped from the primary data area (141). The transfer of files from the subsystem (130) to the storage subsystem (140) can be carried out using a file download mask, which can be generated based on, for example, the name (title) of the downloaded archive file. System (100) stores information about all successfully loaded archives from data sources (110).

The ETL (Extract, Transform, Load) module (132) transfers data to the required areas (142) - (143) of the storage and archiving subsystem (140), and also prepares and converts unzipped data into parquet format when it enters the subsystem ( 130). Parquet is a binary, column-oriented data storage format (see, for example, Apache Parquet Performance. //Https://habr.com/post/282552/).

When converting, source data (110) is cast to Hive-SQL types. An example of casting data to the Hive-AQL type is presented in Table. 1. If the name of the file, for example * json format, does not match the mask, then the data of archives containing such files will not be converted to parquet. This technique provides a stable process of accessing data through Hive using ordinary SQL queries, which leads to an increase in the reliability of access to information.

- 4 036384

Table 1. Data type conversion matrix

Data types, tech. fields Hive-SQL data types D (10) STRING F BOOLEAN N (l) BIGINT N (5.2) DECIMAL (22.2) N (6) BIGINT N (8) BIGINT N (9) BIGINT N (13) BIGINT N (15) BIGINT N (18.2) DECIMAL (22.2) N (19) DECIMAL (22.0) N (22.2) DECIMAL (22.2) N (22) DECIMAL (22.0) N (1000) DECIMAL (38.0) T (l) STRING T (3) STRING T (4) STRING T (5) STRING T (6) STRING T (7) STRING T (8) STRING T (9) STRING T (10) STRING T (ll) STRING T (12) STRING T (13) STRING T (15) STRING

In the area of published data (143), prepared, transformed, unified data are transferred, which are reduced to the standard of the xStream subsystem (170). When preparing unified data, the names of tables and fields are brought to the xStream standard, the data types are brought to the xStream standard, the tables are supplemented with mandatory technical fields describing the data delivery - the start and end dates of the data delivery actuality period, the numbers of the streams that performed the loading and transformation.

The implementation of the above data processing procedures in the subsystem (140) allows you to organize and update the actual data slice (for example, the historical slice) for each source (110) in the primary xStream layer (170), while maintaining the required time depth of the data archive. Loading data into the publication area (143) is carried out in several stages. The first step is to get new archives. To obtain a new list of archives, the raw data storage area (141) is scanned by the name of the corresponding source (110) (for example, for the abc source, the directory in area (141) / data / core / external / abc / src) using the archive name template. The resulting list excludes archives that have already been loaded into the publication area (143) of the source, which have been successfully processed by the storage and archiving subsystem (140). Also, archives that were processed with an error are excluded from the resulting list. Source archives (110) from the remaining list are transferred to the raw data area (141). Next, the resulting archives are unpacked. For unpacking, select only those archives from area (141), the name of which matches the template. The resulting list excludes the source archives (110) that were previously loaded into the publication area (143) of this source. Archives that were processed with an error are excluded from the resulting list. The archives from the remaining list are unpacked into the intermediate data area (142) into the appropriate subdirectory, for example, src / <loading_id> / unpack / <archive_name>.

Each archive is unpacked into a separate subdirectory in the intermediate data area (142) corresponding to the name of the archive. If the archive is successfully unpacked, its name is registered in the src / <loading_id> /unpack/.success file, which is a log of successful download, if an error occurs while unpacking the archive, its name will be registered in the src / <loading_id> /unpack/.fail file , which is, accordingly, the log of the failed download. The processing of other archives is not interrupted when receiving data. After saving the data from the archives in the intermediate data area (142), they are converted to parquet format. Only those data files from the archives that have been registered in the subsystem can be converted to the parquet format.

- 5 036384 subject (140) as successfully unpacked. In this case, files from different archives are processed separately. Each file from the archive corresponds to only one entity, for example, the source table, if the data was loaded from a table view (determined by the file name mask), the files of one archive corresponding to one table are processed together. The tables are supplemented with service attributes (see Table 2), which allow you to maintain an up-to-date snapshot (snapshot) and form the history of file changes received from the data source (110).

Table 2. An example of service attributes

Attribute A type Description ctlloading Int Load ID of the module (161) within which the record was inserted or updated ctl_pa_loadi ng Int The value of the ctl_loading attribute from the Replica before transferring the record to History. A null value means that there was no record with this state in the Replica. ctl_pa_loading <ctl loading ctlaction String Indicator of the last operation on a record in the source: I - record has been inserted U - updated ctlvalidFro m String Insert / update timestamp. Time of confirmation (commit-a) of the transaction, within which the record got into snp. ctlvalidTo String The timestamp of the transfer of the record from snp to history. The ctl validTo field can be greater than or equal to ctl validFrom. Equality is possible if from the source the data came for the previous period (retroactively), in which case the record immediately goes to history. archivenam e String The name of the supplier's archive containing the file containing the data for this table (For example: abc_fb_pagesgeneralinfo_ <date> _ <time> _ nnn.tar.gz). file_name String The name of the file that was in the supplier's archive containing the data for this table (For example: abc_fb_pagesgeneralinfo_20171031 19115 8_001.json).

The above attributes are passed as job parameters and are provided by the functionality of the download control module (161). As a result of processing, the corresponding directory structure is created. If all files from archives are successfully converted to parquet format, the name of the archive is recorded in the log, which is necessary to control the process of access to information and ensure automatic rollback of the system to an earlier point. If an error occurs while processing at least one archive file, subsequent processing of the archive is terminated and the archive name is marked as erroneous.

The processed and unified data from the published data area (143) is transferred to the data access subsystem (150) to ensure that end users receive data for work.

In the data access subsystem (150), two sections are formed - the actual data slice (151), with

- 6 036384 containing a fingerprint of the current data, and a data history section (152) containing information about data changes.

Each entity containing the source data (110) is processed separately, while processing the data corresponding to the entity from all archives that were registered as successfully processed. There are two methods of data processing - the choice of the method depends on the presence of historicity in the data for the mentioned entity.

In the absence of highlighting of historicity (availability of new data), the following is performed. For all new data in the archive table stored in the intermediate data area (142), the attribute is set with the current value of the start time of the data transfer stream. The data is saved with partitioning in the published data area (143) stg / <loadingId> / pa / snp / <table_name>, as a result of which only one partition (partition) is created.

If there is a historicity allocation (the presence of new data) for all new data in the table from the intermediate data area (142), an attribute is set with the start time of the data loading stream by the module (161). The obtained data is combined with the table data from the published data area (143), and new data is extracted from the resulting union, which are placed in areas (151) and (152).

Data to be displayed in area (151) is saved with partitioning by the ctl_loading field in the intermediate data directory (142) stg / <loadшgШ> / pa / snp / <table_name> Ctl_loading is a stream (technical entity) that is activated by the load control module (161 ). A separate stream is created and registered for each source (110).

For data in the area (152) of the data change history, the fields ctl _pa_loading (from the ctl_loading field), ctl_loading (with the current <loadingId> value) and ctl_validTo (the start time of the data download stream) are filled. The data for display in area (152) is saved with partitioning by the ctl_loading field in the intermediate data directory (142) stg / <loadingId> / pa / hist / <table_name>. As a result, only one partition (partition) is created for area (152) stg / <loadingId> / pa / hist / <table_name> / ctl_loading = <loadingId>

At the end of this action, the stg / status / .fail file is copied to the directory of the intermediate data storage area (142) stg / <loadingId> / pa /. At the end of the operation, the intermediate data storage area (142) is cleared. The log data of the operation performed in the area (142) must be added to the general log of the published data area (143).

Next, the files from area (142) stg / status / .success and area (141) src / <loading_id> /parquet/.success are merged into the file stg / <loadingId> /pa/.success. The stage is considered successful if all data was successfully processed, otherwise the stream will end with an error.

As part of the implementation of the claimed solution, two methods are provided for publishing data obtained from the source (110), the choice of the method depends on the presence of the historicity of changes in the data in this source (110).

If there is no change information for the source (110), then the stg / <loadingId> / pa / snp / <table_name> / ctl_loading = <loadingId> directory from the intermediate data storage area (142) is moved to the published data area directory (143) pa / snp / <table_name> as a new section that is subsequently registered for display in hCatalog (150) in area (151).

If there is an allocation of historicity (availability of new data) for the data source (110), the following is performed. The data directory for source (110) pa / snp / <table_name> from area (143) is moved to stg / <loadingId> / reserve / pa / snp in intermediate data area (142). The stg / <loadingId> / pa / snp / <table_name> directory from the intermediate data area (142) is moved to the published data area directory (142) as pa / snp / <table_name> New section in area (143) pa / snp / < table_name> / ctl_loading = <loadingId> is registered in hCatalog (150) to provide access to information. The directory in area (142) stg / <loadingId> / pa / hist / <table_name> / ctl_loading = <loadingId> created earlier is moved to the directory in area (143) pa / hist / <table_name> as a new section, which is also is registered in subsystem (150). At the end of this process, the files stg / <loadingId> /pa/.success and stg / <loadingId> /pa/.fail are copied to the directory of the intermediate data area (142) stg / status /, which display the status of the data loading operations. This stage is considered successful if during its execution there were no exceptions and data loading errors. The data processing and archiving subsystem (140) clears / archives the published data area (143) according to the parameter of the information download stream identifier (ctlloading and ctl_pa_loading), which is set by the download control module (161). The depth of the history to be cleared / archived is set by a parameter, for example, 5 years.

To connect a new data source (110) or change the parameters of an existing subsystem xStream (170) has configuration files, as well as profiles specified when starting the XStream subsystem (170) containing a set of parameters. Configuration files assume setting the following parameters:

source folder address;

type of input files;

- 7 036384 names of archives and files in the archive;

type of transformations;

the need for history;

role model;

scripts for creating a database and tables for the source, etc.

Files from the sender - the data source (110) - are usually published to the sender's shared folder in TFS (120), so for this it is necessary to generate a unique file name. The files are delivered packed in archives. The valid extension types can be different, for example tar.gz, zip, etc. The archive file name can be formed, for example, using the following mask: <saller> _ <source> _ <table> _ <inc> _ <ver> _ <date> _ <time> _ <nnn>. <extension>, where saller - the ultimate data provider (or proxy provider);

source - data source (110);

table is the name of the data type or entity. If the delivery in one archive comes with a schema consisting of several interconnected entities that must be supplied simultaneously, for example, tables, then - kit is put in this section, and the direct names of the entities are already indicated in the files inside the archive;

inc - contains full if the archive refers to the delivery / re-delivery of the complete data archive, incr - if the archive refers to the delivery of the increment;

ver - delivery version. If the data composition or delivery format changes, the version must be incremented upwards. May consist of the letter v and 3 digits starting with v001, for example v001, v002, etc .;

date - the date of the file formation in YYYYMMDD format;

time - time of file formation in the format HCHMMSS;

nnn is a sequential number starting from 001, if several delivery files are within the coincidence of the rest of the archive file name. Always 3 digits starting at 001, for example 001, 002, etc. If there is only 1 file in the delivery, then this section indicates - 001; extension - archive extension. Example archive name: abc_def_org1_incr_v002_20171106_112400_001.tar.gz. The presence of an archive in the archive is not allowed. The file name in the archive is formed in the same way as the name of the archive file, except for the file extension and the <table> section for the set.

Archive name: abc_def_kit_incr_v002_20171106_112400_001.tar.gz, files in the archive: abc_def_kit1_incr_v002_20171106_112400_001.xml, abc_def_orgl_incr_v002_20171106112400 etc. Allowed file types and extensions, for example, csv, tsv, txt, json, avro, xml, etc. A maximum of 128 characters (including the extension) are allowed in the name of the transferred file.

The data model of the external source (110) corresponds to the structure of the supplied data and is determined at the stage of analysis and preparation of the source (110) for loading into xStream (170). Apache Hive serves as the target for data access. User access to the published data contained in area (150) is based on the control of the access level of each of the users. For each user, when interacting with xStream (170), the permitted functionality for performing operations with data is checked, in particular, such operations can be: viewing, editing, obtaining an analytical cut, combined viewing, etc. A user with the roles Administrator and Auditor have access to xStream logs (170) through a centralized log management system available in the cloud platform (180).

FIG. 3 shows a general view of a computing device (200), using which the claimed method and system is implemented in the cluster. In the general case, the computing device (200) contains one or more processors (201) united by a common bus, memory means such as RAM (202) and ROM (203), input / output interfaces (204), input / output means (205 ), and a means for networking (206).

The processor (201) (or multiple processors, multi-core processor) can be selected from a wide range of devices currently in widespread use, such as Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™, etc. ...

RAM (202) is a random access memory and is intended for storing machine-readable instructions executed by the processor (201) for performing the necessary operations for logical data processing. RAM (202), as a rule, contains executable instructions of the operating system and corresponding software components (applications, software modules, etc.).

ROM (203) is one or more persistent storage devices such as a hard disk drive (HDD), solid state data storage device (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

Various types of I / O interfaces (204) are used to organize the operation of device components (200) and to organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific design of the computing device, which can be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, com, SATA, IDE, Lightning, USB (2.0, 3.0,

- 8 036384

3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc. To ensure user interaction with the computing device (200), various means (205) of I / O information are used, for example, a keyboard, display (monitor), touch display, touch pad, joystick, mouse manipulator, light pen, stylus, touch panel, trackball , speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retina scanner, fingerprint scanner, voice recognition module), etc.

The means of networking (206) allows the device (200) to transmit data via an internal or external computer network, for example, Intranet, Internet, LAN, etc. One or more means (206) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module, and dr.

Additionally, satellite navigation aids can also be used, for example GPS, GLONASS, BeiDou, Galileo.

FIG. 4 shows an example of a network environment in the implementation of the operation of the claimed system (100). The organization of work with data when using FIDFS consists in the formation of appropriate levels of abstraction in clustered or virtual environments. Each system stack includes a plurality of computing devices, such as computers and / or servers, that communicate with a cloud platform (180) containing an xStream (170) through switches. This architecture allows you to quickly increase the required computing power with a significant increase in the amount of stored and processed data. The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation, which do not go beyond the scope of the claimed legal protection, which are obvious to specialists in the relevant field of technology.

Claims (28)

CLAIM 1. The system of integrated management of Big Data (Big Data), containing a subsystem for transportation and verification of input data (hereinafter TFS), a subsystem for loading external data (hereinafter xStream), operating on the basis of the data description standard, in which TFS receives, checks and transmits data in xStream from data sources, and TFS receives data in an archived form and transfers data using a transport protocol; xStream contains a file loading subsystem, a data processing and archiving subsystem, a data access subsystem (hereinafter referred to as hCatalog), a data loading control module and an access control module, and in xStream, the file download subsystem polls TFS to transfer data received from sources to the processing subsystem and archiving data, wherein the data sources are pre-registered with the download control module; data loading subsystem performs primary copying of files from TFS to the processing and archiving subsystem; the processing and archiving subsystem contains data storage areas that carry out structured storage of primary data, intermediate data and published data; files from the data loading subsystem are transferred to the primary data storage area, which correspond to the established archiving parameters and are stored in the originally archived form; the processing and archiving subsystem is configured to transfer to the intermediate data area the unzipped primary data converted into the parquet format for further transformations; prepared, transformed, unified data converted to the xStream standard are transferred to the published data area, in which the names of tables and fields are converted to the xStream standard, the data types are converted to the xStream standard, the tables are supplemented with mandatory technical fields describing the delivery of data - dates of the beginning and end of the period the relevance of the data supply, the numbers of the streams that performed the loading and conversion; and from the unified data of the published data area, a structured catalog is formed for accessing said data through hCatalog. 2. The system of claim 1, characterized in that TFS checks the integrity of the archived data received. 3. A system according to claim 1, characterized in that the ID of said sources is stored in the download control module for registered sources. 4. The system according to claim 3, characterized in that the download control module controls the data flow of the plurality of sources according to the respective stored IDs. - 9 036384 5. The system according to claim 4, characterized in that for each data source, the download control module contains data download parameters. 6. The system according to claim 1, characterized in that the data loading subsystem loads files into the processing and archiving subsystem based on the file loading mask. 7. The system according to claim 6, characterized in that the file download mask is formed based on at least the name of the downloaded archive file. 8. The system according to claim 1, characterized in that in the processing and archiving subsystem in each of the data storage areas, a directory is formed to store the data of the corresponding source. 9. The system according to claim 7, characterized in that the download control module checks the availability of information for all sources in the TFS. 10. The system according to claim 9, characterized in that full or incremental loading of data from TFS is performed. 11. The system according to claim 10, characterized in that the incremental loading is performed by the data loading module when new data is found in the TFS that differ from those previously supplied by the date of delivery of the archive files. 12. The system according to claim 1, characterized in that the processing and archiving subsystem processes parquet files to bring them into conformity with the Hive-SQL types. 13. The system according to claim 1, characterized in that for files when they are processed by the processing and archiving subsystem, a check is performed for the presence of similar data saved earlier. 14. The system according to claim 13, characterized in that, in the presence of earlier data, the current version of the data is transferred to the area of published data, and the previous version is moved to the history storage directory with partitioning by the date of loading. 15. The system according to claim 1, characterized in that hCatalog provides a display of the data structure of the data publication area. 16. The system according to claim 15, characterized in that the data structure is divided into databases corresponding to the sources of data provision. 17. The system according to claim 1, characterized in that the data processing and archiving subsystem additionally provides an automated function for loading data for any date in the past. 18. A method for managing Big Data using a subsystem for transporting and checking input data (hereinafter TFS) and a subsystem for loading external data (hereinafter xStream), and xStream consists of a file upload subsystem, a processing and archiving subsystem, a data access subsystem, a data download control module and an access control module, the method comprising the steps of interacting with the TFS using the xStream download control module to receive data from said sources, the data sources being pre-registered with the data download control module; receive data from said sources using TFS, which receives data in an archived form and accumulates and verifies the data; in case of successful verification of said data, they are transferred to the data loading subsystem via a transport protocol; using a subsystem for processing and archiving data, processing the received data, which includes accumulating files, checking files, unpacking archived files that have been checked and converting the unpacked files to parquet format; structuring the converted files by placing them in directories, each of which is associated with data sources registered in said data loading control module; control and delete duplicate data, control and delete data with a broken structure, convert data types to Hive-SQL, control the update of the current data catalog, control and update the data change history catalog, control and manage the depth of the data archive; provide users with access to data located in the data access subsystem using the access control module. 19. The method of claim 18, wherein the access control module determines a set of functionality based on the user's access level. 20. The method according to claim 18, characterized in that the data processing and archiving subsystem processes files of the parquet format for their correspondence to the Hive-SQL type. 21. The method according to claim 18, characterized in that the data sources are registered by recording the source ID in the download control module. 22. The method according to claim 18, characterized in that the download control module controls the data flow of the plurality of sources according to the respective stored IDs. 23. The method according to claim 22, characterized in that data flow control includes checking for the availability of information from a data source in TFS, processing messages from TFS, performing actions on - 10 036384 based on message processing. 24. The method according to claim 22, characterized in that full or incremental data loading is performed from the input data primary processing subsystem. 25. The method according to claim 23, characterized in that the incremental download is performed when the download module detects the presence of new data. 26. The method according to claim 18, characterized in that for each data source, the download control module contains data download parameters. 27. The method according to claim 18, characterized in that the files are loaded into the processing and archiving subsystem based on the file download mask. 28. The method according to claim 27, characterized in that the file download mask is generated based at least on the name of the downloaded archive file.