RU2690777C1 - Method and system for complex control of big data - Google Patents

Abstract

FIELD: computer equipment.SUBSTANCE: invention relates to computer engineering for controlling large amounts of data. File download subsystem polls TFS for transmitting data from sources to data processing and archiving subsystem, wherein data sources are pre-registered in the data loading subsystem, the data loading subsystem performs primary copying of files coming from the TFS into the processing and archiving subsystem; processing and archiving subsystem comprises data storage areas, which perform structured storage of primary data, intermediate data and published data; files are transferred from the data loading subsystem into the primary data storage area, which correspond to the archiving parameters and are stored in the originally archived form.EFFECT: technical result consists in improvement of efficiency of storing large amounts of data by using distributed storage architecture with provision of input data processing for their unification.28 cl, 2 tbl, 4 dwg

Description

TECHNICAL FIELD

[0001] The claimed solution relates to the management of large amounts of data, in particular, to a system and method for processing and unified storage.

BACKGROUND

[0002] From the prior art various solutions are known for organizing data storage systems based on distributed file systems (HDFS / Hadoop File System), which, in particular, are used for working with big data.

[0003] The term "Big Data" (Big Data) refers to a huge amount of data entering the storage system. Data, as a rule, comes from a variety of information sources in an unstructured form. Big Data also includes technologies for their processing and use, methods for finding the necessary information in large arrays.

[0004] The main problem of big data is associated with a large amount of information that needs to be processed and, accordingly, stored in a trusted manner while minimizing the risks of data loss. In this regard, there is a need for data backup, which implies the organization of bulk data storage structures. However, with the increase in the amount of information increases the complexity of its reservation.

[0005] HDFS, or Hadoop Distributed File System, is the main storage system used by Hadoop.HDFS applications multiple copies of data blocks and distributes these copies across the compute nodes of the cluster, thereby ensuring high reliability and speed of calculations.

[0006] From the prior art, a data warehouse architecture is known for providing shared access to the data stored therein (US20130262615, 03.10.2013). Data storage can be implemented using HDFS and provide a filter for incoming information for its primary processing while directly storing it for later use. The storage system operates by generating copies of incoming data from disparate sources for subsequent linking them with metadata for access purposes.

[0007] The application CN 106407309 (Wei et al., 15.02.2017) describes the cluster storage mechanism for obtaining information from many different data sources and providing an authentication process between the database and data sources.

[0008] Known solutions have significant drawbacks, which are the lack of processing of incoming data in order to convert them into a unified form, which does not allow for the prompt updating of information from sources, and reduces the efficiency of the data warehouse. Also, the lack of a data unification process leads to a low resiliency of the storage system in case of data arrival in the wrong or unsupported format (transport format errors, archiving errors, copy errors, data structure errors, data type errors).

DISCLOSURE OF INVENTION

[0009] The technical problem to be solved with the help of the claimed method and system is the elimination of the drawbacks inherent in the known analogues, as well as the creation of a new principle of efficient storage of large amounts of data.

[0010] The technical result achieved coincides with the technical problem being solved and consists in increasing the efficiency of storing large amounts of data through the use of a distributed storage architecture, ensuring the processing of input data with a view to their unification.

[0011 An additional technical effect is to increase the fault tolerance of the data storage system, due to the architecture of the big data storage management system and ensuring unification of the stored data.

[0012] The decision is made using the Integrated Big Data Management System (Big Data), which contains the transportation and input data verification subsystem (hereafter referred to as TFS), the external data upload subsystem (hereafter xStream), which operates on the basis of a data description standard, in which: TFS performs reception, verification and transmission in xStream of data coming from data sources, with TFS receiving data in an archived form and transmitting data over a transport protocol;

xStream contains the file download subsystem, the data processing and archiving subsystem, the data access subsystem (hereinafter hCatalog), the data download control module and the access control module, and in xStream: the file upload subsystem interrogates TFS for transferring data from sources to the subsystem processing and archiving data, with data sources pre-recorded in the data loading subsystem; the data loading subsystem performs the primary copying of files from TFS to the processing and archiving subsystem;

processing and archiving subsystem contains data storage areas that carry out structured storage of primary data, intermediate data and published data;

files are transferred to the primary data storage area from the data download subsystem, which correspond to the established archiving parameters and are stored in the originally archived form;

the processing and archiving subsystem is designed with the possibility of transferring unarchived primary data into the region of intermediate data, converted into parquet format for further transformations;

The prepared data, converted, unified data converted to the xStream standard are transferred to the published data area, in which the table and field names are given to the xStream standard, the data types are referred to the xStream standard, the tables are supplemented with the required technical fields describing the data supply - dates of the beginning and end of the period the relevance of the data delivery, the numbers of the threads that performed the download and conversion;

and from the unified data of the published data area a structured catalog is formed for access to the mentioned data by means of hCatalog.

[0013] In one of the private implementation options, the TFS system checks the integrity of the archived data received.

[0014] In another particular implementation of the system for registered sources, the ID of said sources is stored in the download control module.

[0015] In another particular implementation of the system, the load control module manages the flow of data from a plurality of sources by the corresponding stored IDs.

[0016] In another particular implementation of the system, for each data source in the load control module, data loading parameters are contained.

[0017] In another particular implementation of the system, the data loading subsystem uploads files to the processing and archiving subsystem based on the file loading mask.

[0018] In another particular implementation of the system, a file download mask is formed based on at least the name of the downloaded archive file.

[0019] In another private implementation of the system, in the processing and archiving subsystem, in each of the data storage areas, a directory is created for storing data of the corresponding source.

[0020] In another particular implementation of the system, the load control module checks the availability of information from all sources in TFS.

[0021] In another particular implementation of the system, full or incremental data loading from TFS is performed.

[0022] In another particular implementation of the system, incremental loading is performed by the data loading module when new data in the TFS is detected that differs from the previously set by the delivery date of the archive of files.

[0023] In another particular implementation of the system, the processing and archiving subsystem processes parquet files to match them to Hive-SQL types.

[0024] In another particular implementation of the system, for files being processed, the processing and archiving subsystem checks for the presence of similar data previously stored.

[0025] In another particular implementation of the system, if there are earlier data, the current version of the data is transferred to the published data area, moving the previous version to the history storage directory with partitioning by download date.

[0026] In another particular implementation of the system, hCatalog provides a mapping of the data publication data structure.

[0027] In another particular implementation of the system, the data structure is broken down into databases corresponding to the sources of the data provided.

[0028] In another particular implementation of the system, the data processing and archiving subsystem additionally provides an automated feature for loading data on any date in the past.

[0029] The claimed solution is also implemented using a Big Data (Big Data) management method using a transport subsystem and input data validation (hereinafter referred to as TFS) and external data loading subsystem (hereafter xStream), and xStream consists of a file upload subsystem, processing subsystem and archiving, data access subsystem, data loading control module and access control module, the method including the steps of: interacting with TFS with the help of the xStream loading control module to obtain data from mentioned sources, with the data sources being pre-registered in the data load control module;

get data from the mentioned sources using TFS, which receives the data in an archived form and accumulates and verifies the data; in case of successful verification of the mentioned data, they are transferred to the data loading subsystem using the transport protocol;

using the subsystem of processing and archiving data, they process the received data, which includes accumulating files, checking files, unpacking archive files that have been checked and converting unpacked files into parquet format;

structured converted files by placing them in directories, each of which is associated with data sources registered in said data load control module;

monitor and delete duplicate data, monitor and delete data with broken structure, convert data types to Hive-SQL, monitor update of the actual data catalog, monitor and update the data change history catalog, monitor and control the depth of the data archive;

access users to the data hosted in the data access subsystem using the access control module.

[0030] In one particular example implementation of the method, the access control module determines the functionality set based on the user's access level.

[0031] In another particular example of the implementation of the method, the data processing and archiving subsystem processes parquet files to match the Hive-SQL type.

[0032] In another particular example of the method implementation, data sources are registered by recording the source ID in the load control module.

[0033] In another particular example of the method implementation, the load control module controls the flow of data from a plurality of sources by the corresponding stored IDs.

[0034] In another particular example of the implementation of the method, data flow control includes checking the availability of information from a data source in TFS, processing messages from TFS, and performing actions based on message processing.

[0035] In another particular example of the method implementation, full or incremental data loading is performed from the primary processing subsystem of the input data.

[0036] In another particular example of the method implementation, incremental loading is performed when the loading module detects the presence of new data.

[0037] In another particular example of the method implementation for each data source in the load control module contains data loading parameters.

[0038] In another particular example of the implementation of the method, files are loaded into the processing and archiving subsystem based on the file loading mask.

[0039] In another particular example of the method implementation, a file download mask is formed based on at least the name of the downloaded archive file.

DESCRIPTION OF THE DRAWINGS

[0040] The features and advantages of this technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0041] FIG. Figure 1 illustrates the claimed system for integrated big data management.

[0042] FIG. 2 illustrates an example source identifier.

[0043] FIG. 3 illustrates a general view of a user's computing device.

[0044] FIG. 4 illustrates a general network interaction diagram.

IMPLEMENTATION OF THE INVENTION

[0045] FIG. 1 shows a general scheme for implementing the claimed big data management system (100). The main functional element of the declared system (100) is the subsystem (170) of external data loading - xStream, which interacts (or is part of) with the cloud platform (OP) (180) of data storage and management. XStream is a framework for providing a centralized infrastructure for receiving, processing and storing external data in the EP on Hadoop for further distribution to all interested parties for use in business processes and in research for the extraction of value.

[0046] Data sources (110) can represent various resources and channels for providing information, for example, social networks, accounting systems, CRM systems, registries of state bodies (Unified State Register of Legal Entities, ROSSTAT, FFMS, etc.), etc.

[0047] Information from data sources (110) is initially processed by the transportation and input data verification subsystem (hereinafter referred to as the TFS / Transport File System) (120). TFS (120) receives, checks and transmits through the transport protocol to xStream (170) data from data sources (110). As a protocol for transferring information from TFS (120) to xStream (170), the Network File System (NFS) protocol can be used. The data is fed to the TFS (120) input in archived form.

[0048] The xStream subsystem (170) in turn consists of: a file download subsystem (130), a data processing and archiving subsystem (140), a data access subsystem (150) and a technology core (160).

[0049] The data received at the TFS input (120) is transferred to the file upload subsystem (130), which transports data archives using the integration module (131) from TFS (120) to the primary data storage layer - subsystem (140) .

[0050] For each of the data sources (110), two types of loading are assumed:

Первичная загрузка - является инициализирующей загрузкой, которая выполняется один раз и захватывает все архивы доступные в TFS (120), названия файлов которых удовлетворяют спецификациям на поставку данных от поставщиков.

Primary Download is an initialization download that runs once and captures all the archives available in TFS (120), the file names of which meet the specifications for supplying data from suppliers.

Регулярная загрузка - выполняется инкрементально, при которой захватываются только архивы, которые не были загружены ранее.

Regular download - performed incrementally, which captures only archives that have not been loaded before.

[0051] The capture of archives comes from DFS directories. When accessing TFS (120), authentication is performed using a technical account and private key. Data is copied from TFS (120) to the primary storage layer xStream (170) by initiating by the load control module (161) data requests from TFS (120). TFS (120) indicates the routes for loading data and control messages between the module (161) and TFS (120). Module (161) can initiate loading of one or several data stream simultaneously, while the streams will be processed in parallel.

[0052] Each data source (110) is registered in a module (161). For each source, its identifier (ID) is created and stored. During the registration of the data source (110), the following procedure is performed:

Присвоение номера источнику данных (110);

Assigning a number to the data source (110);

Определение схем внутри источника (110) и присвоения им номера. Схема является необязательным элементом и применяется для логической группировки таблиц.

Identification of circuits inside the source (110) and assigning them numbers. The schema is an optional element and is used for logical grouping of tables.

Присвоение номера всем таблицам внутри источника (110) в пределах схемы (при условии ее использования).

Assigning numbers to all tables within the source (110) within the scheme (subject to its use).

[0053] Source (110), schemas, tables correspond in module (161) to entities with unique identifiers. The identifier is an integer of the format shown in FIG. 2

[0054] The source number is the internal surrogate source identifier (110) in xStream (170), which is generated at the stage of connecting the source (SW) to xStream (170).

[0055] The data area indicates the storage area of the received information in xStream (170):

1 - область публикуемых данных (143);

1 - the area of published data (143);

2 - область сырых данных (141);

2 - area of raw data (141);

3 - область промежуточных данных (142);

3 - intermediate data area (142);

0 - базовая сущность, идентифицирующая данные источника.

0 is the base entity that identifies the source data.

[0056] A schema is a schema identifier at the source or subsystem. It is generated at the stage of connecting the source (110) to xStream (170). The table number represents the table number in the schema. It is also generated at the stage of connecting the source (110) to xStream (170).

[0057] The data processing and archiving subsystem (140) is a repository of data received from external sources (110), and is a logical area in HDFS. Access to the stored data is carried out through the access subsystem (150) hCatalog, which is also intended for publishing meta information about the data.

[0058] The data processing and archiving subsystem (140) contains data storage areas that carry out structured storage of primary (raw) data (141), intermediate data (142) and published data (143).

[0059] Archives of information obtained from TFS (120) from data sources (110) registered in module (161) are transferred to the primary data storage area (141). Received data is checked for integrity in TFS (120). In case of violation of the integrity of archives received from data sources (110), such information is not transmitted to the xStream subsystem (170). Upon successful copying of information from TFS (120) to the primary data area (141), xStream (170) notifies of a successful operation.

[0060] When data is loaded into the intermediate data storage area (142), unarchiving data archives from the primary data area (141) is initially performed. Transferring files from subsystem (130) to storage subsystem (140) can be accomplished using a file upload mask, which can be generated based on, for example, the name (name) of the downloaded archive file. The system (100) stores information about all successfully downloaded archives from data sources (110).

[0061] The ETL module (Extract, Transform, Load) (132) transfers data to the required areas (142) - (143) of the storage and archiving subsystem (140), and also prepares and converts the unzipped data into parquet format when it arrives to the subsystem (130). Parquet is a binary, column-oriented data storage format (see, for example, Apache Parquet Performance. // https://habr.com/post/282552A.

[0062] When converting, source data (110) is converted to Hive-SQL types. An example of converting data to the Hive-AQL type is presented in Table 1. If the file name, for example, the * .json format, does not match the mask, then the data of the archives containing such files will not be converted to parquet. This technique provides a steady process of accessing data through Hive using ordinary SQL queries, which leads to increased reliability of access to information.

[0063] In the area of published data (143), prepared, transformed, unified data are transmitted, which are converted to the standard of the xStream subsystem (170). When preparing unified data, the names of the tables and fields are given to the xStream standard, the data types are given to the xStream standard, the tables are supplemented with the required technical fields describing the data supply — the start and end dates of the data supply time period, the load and conversion numbers

[0064] The execution of the above data processing procedures in the subsystem (140) allows you to organize and update the actual data slice (for example, historical slice) for each source (110) in the xStream primary layer (170), while maintaining the required archive time depth data.

[0065] The data is loaded into the publication area (143) in several steps. At the first stage, new archives are received. To obtain a new list of archives, the raw data storage area (141) is scanned by the name of the corresponding source (110) (for example, for the source abc, the directory in the area (141) / data / core / external / abc / src) by the archive name template. The archives that have already been downloaded to the publication area (143) of the source, which were successfully processed by the storage and archiving subsystem (140), are excluded from the list. Also, archives that were processed with an error are excluded from the resulting list. The source archives (110) from the remaining list are transferred to the raw data area (141).

[0066] Next is the unpacking of the received archives. For unpacking, only those archives are selected from the area (141), the name of which matches the pattern. The source archives (110) that were previously downloaded to the publication area (143) of this source are excluded from the resulting list. Archives that were processed with an error are excluded from the list. Archives from the remaining list are unpacked into the intermediate data area (142) into the appropriate subdirectory, for example, src / <loading_id> / unpack / <archive_name>.

[0067] Each archive is unpacked into a separate subdirectory in the intermediate data area (142), corresponding to the name of the archive. If the archive is successfully unpacked, its name is registered in the src / <loading_id> /unpack/.success file, which is a successful download log, if an error occurs when unpacking the archive, its name is registered in the src / <loading_id> / unpack / file. fail, which is, respectively, the log of unsuccessful downloads. Other archives are not interrupted when data is received.

[0068] After saving the data from the archives in the area of intermediate data (142), they are converted into parquet format. Only data files from archives that were registered in subsystem (140) as successfully unpacked are subject to translation into parquet format. In this case, files from different archives are processed separately. Each file from the archive corresponds to only one entity, for example, the source table, in case the data were loaded from a table view (determined by the file name mask), files of one archive corresponding to one table are processed together. The tables are supplemented with service attributes (see Table 2), which allow you to maintain an up-to-date snapshot (snapshot) and generate a change history of files received from the data source (110).

[0069] The above attributes are passed as job parameters and provided with the functionality of the load control module (161). As a result of processing the corresponding directory structure is created. In case of successful conversion of all files from archives to parquet format, the name of the archive is recorded in the log, which is necessary to control the process of access to information and ensure automatic rollback of the system to an earlier point. If an error occurs while processing at least one archive file, the subsequent processing of the archive stops and the archive name is marked as erroneous.

[0070] The processed and unified data from the published data area (143) is transmitted to the data access subsystem (150) to ensure that the data is received for work by end users.

[0071] In the data access subsystem (150), two sections are formed — the actual data slice (151), containing the imprint of the current data, and the data history section (152), which contains information about changes in the data.

[0072] Each entity containing data from source (110) is processed separately, and the data corresponding to the entity is processed from all archives that were registered as successfully processed. There are two ways of data processing - the choice of method depends on the availability of historical data in the data on the above entity.

[0073] In the case of non-allocation of historicity (availability of new data) the following is done. For all new data on the archive table stored in the intermediate data area (142), the attribute is set with the current start time of the data stream. Data is saved with partitioning in the published data area (143) stg / <1oadingId / pa / snp / <table_name>, as a result of which only one partition (partition) is created.

[0074] If there is a selection of historicity (availability of new data) for all new data on the table from the intermediate data area (142), an attribute is set with the start time of the data load stream by the module (161). The obtained data is combined with the data of the table from the published data area (143) and new data are separated from the resulting union, which are located in areas (151) and (152).

[0075] The data for display in the area (151) is stored with partitioning by the ctl_loading field in the intermediate data directory (142) stg / <loadingId> / pa / snp / <table_name>. Ctl_loading is a stream (technical entity) that is activated by the load control module (161). Under each source (110) a separate stream is created and registered.

[0076] For the data in the data change history area (152), the ctl_pa_loading fields (from the ctl_loading field), ctl_loading (with the current <loadingId> value) and ctl_validTo (start time of the data loading stream) are filled. The data for display in the area (152) is saved with partitioning by the ctl_loading field in the intermediate data directory (142) stg / <loadingId> / pa / hist / <table_name>. As a result, only one partition (section) stg / <loadingId> / pa / hist / <table_name> / ctl_loading = <loadingId> is created for the area (152).

[0077] At the end of this action, the stg / status / .fail file is copied to the intermediate data storage area directory (142) stg / <loadingId> / pa /. At the end of the operation, the intermediate data storage area (142) is cleared. The log data of the operation performed in the field (142) must be added to the general log of the published data area (143).

[0078] Next, the files from the (142) stg / status / .success and the (141) src / <loading_id> /parquet/.success to the stg / <loadingId> /pa/.success file are merged. A stage is considered successful if all the data were processed successfully, otherwise the stream will end with an error.

[0079] As part of the implementation of the claimed solution, there are two ways to publish data obtained from the source (110), the choice of method depends on the presence of historical changes in the data in this source (110).

[0080] If there is no change information for source (110), then the stg / <loadingId> / pa / snp / <table_name> / ctl_loading = <loadingId> directory is moved from the intermediate data storage area (142) to the directory of the published data area ( 143) ra / snp / <table_name> as a new section, which is subsequently registered for display in hCatalog (150) in the area (151).

[0081] In the case of the presence of a selection of historicity (availability of new data) for the data source (110), the following is performed. The data directory for the source (110) pa / snp / <table_name> from the area (143) is moved to stg / <loadingId> / reserve / pa / snp intermediate data area (142). The stg / <loadingId> / pa / snp / <table_name> directory from the intermediate data area (142) is moved to the directory of the published data area (142) as ra / snp / <table_name>. A new section in the area (143) pa / snp / <table_name> / ctl_loading = <loadingId> is registered in hCatalog (150) to provide access to information.

[0082] The directory in the area (142) stg / <loadingId> / pa / hist / <table_name> / ctl_loadmg = <loadingId> created earlier is moved to the directory in the area (143) ra / hist / <table_name> as a new section which is also registered in the subsystem (150). At the end of this process, the stg / <loadingId> /pa/.success and stg / <loadingId> /pa/.fail files are copied to the intermediate data area directory (142) stg / status /, which display the status of data load operations. This stage is considered successful if during its execution there were no exceptions and data loading errors.

[0083] The data processing and archiving subsystem (140) clears / archives the published data area (143) by the parameter of the information flow identifier (ctl_loading and ctl_pa_loading), which is set by the load control module (161). The depth of the history to be cleaned / archived is set by the parameter, for example, 5 years.

[0084] To connect a new data source (110) or modify the parameters of an existing subsystem, xStream (170) has configuration files, as well as profiles, specified when the XStream subsystem (170) starts, containing a set of parameters. Configuration files assume the following settings:

адрес папки источника

source folder address

тип входных файлов,

type of input files

имена архивов и файлов в архиве,

names of archives and files in the archive,

тип преобразований,

type of transformations

необходимость ведения истории

the need for a history

ролевая модель

role model

скрипты создания базы данных и таблиц для источника

scripts to create a database and tables for the source

и др.

and etc.

[0085] Files from the sender - the data source (110) are usually published to the common folder of the sender in TFS (120), therefore, for this it is necessary to generate a unique file name. Files are delivered packed in archives. Valid extension types can be different, for example, tar.gz, zip, etc.

[0086] The archive file name can be generated, for example, by the following mask: <saller> _ <source> _ <table> _ <inc> _ <ver> _ <date> _ <time> _ <nnn>. <Extension> where

sailer - конечный поставщик данных (или прокси-поставщик);

sailer - the ultimate data provider (or proxy provider);

source - источник данных (110);

source - data source (110);

table - название типа данных или сущности. В случае если в поставке одним архивом приходит схема, состоящая из нескольких взаимосвязанных сущностей, которые необходимо поставлять одновременно, например, таблиц, то в этой секции ставится - kit, а в самих файлах внутри архива уже указываются непосредственные имена сущностей;

table - the name of the data type or entity. If the delivery comes in a single archive with a schema consisting of several interrelated entities that need to be delivered simultaneously, for example, tables, then the kit is placed in this section, and the actual names of the entities are already indicated inside the archive;

inc - содержит full если архив относится к поставке/перепоставке полного архива данных, incr - если архив относится к поставке инкремента;

inc - contains full if the archive refers to the delivery / re-delivery of a complete data archive, incr - if the archive refers to the delivery of an increment;

ver - версия поставки. В случае изменения состава данных или формата поставки необходимо инкрементировать версию вверх. Может состоять из литеры v и 3-х цифр, начиная с v001, например, v001, v002 и т.д.;

ver - delivery version. In case of a change in the composition of the data or the delivery format, it is necessary to increment the version upwards. It may consist of the letter v and 3 digits, starting with v001, for example, v001, v002, etc .;

date - дата формирования файла в формате ГГГГММДД;

date - date of file formation in the format YYYYMMDD;

time - время формирования файла в формате ЧЧММСС;

time - the time of formation of the file in the format HHMMSS;

nnn - номер по порядку, начинающийся с 001, если несколько файлов поставки в рамках совпадений остальных частей названия файла архива. Всегда состоит из 3-х цифр начиная с 001, например, 001, 002 и т.д. Если в поставке только 1 файл, то в этой секции указано - 001;

nnn is a number in order, starting with 001, if several delivery files are within the scope of the remaining parts of the archive file name. It always consists of 3 digits starting from 001, for example, 001, 002, etc. If there is only 1 file in the delivery, then this section indicates - 001;

extension - расширение архива.

extension - archive extension.

[0087] An example of the archive name is abc_def_orgl_incr_v002_20171106_112400_001.tar.gz. The presence of an archive in the archive is not allowed. The file name in the archive is formed in the same way as the file name of the archive, with the exception of the file extension and the <table> section for dialing.

[0088] Archive name: abc_def_kit_incr_v002_20171106_112400_001.tar.gz, the files in the archive: abc_def_kitl_incr_v002_20171106_112400_001.xml, abc_def_orgl_incr_v002_20171106_112400_001.xml etc. Valid file types and extensions, for example, csv, tsv, txt, json, avro, xml, etc. A maximum of 128 characters are allowed in the name of the file to be transferred (including the extension).

[0089] The external source data model (110) corresponds to the structure of the supplied data and is determined at the stage of analysis and preparation of the source (110) for loading into xStream (170). The target tool for accessing data is Apache Hive.

[0090] User access to published data contained in area (150) is carried out on the basis of controlling the level of access of each user. When interacting with xStream (170), each user is checked for allowed functionality for performing data operations, in particular, such operations can be: viewing, editing, obtaining an analytical slice, combined viewing, etc. A user with the Administrator and Auditor roles has access to xStream logs (170) through a centralized log management system available in the cloud platform (180).

[0091] FIG. 3 shows a general view of a computing device (200), using which the claimed method and system is implemented as part of a cluster.

[0092] In general, the computing device (200) contains one or more processors (201) combined with a common bus, memory means such as RAM (202) and ROM (203), input / output interfaces (204), input / output (205), and a means for networking (206).

[0093] A processor (201) (or multiple processors, a multi-core processor) may be selected from a variety of devices currently widely used, such as Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™, and etc.

[0094] The RAM (202) is a RAM and is intended for storing computer-readable instructions executed by the processor (201) to perform the necessary logical data processing operations. RAM (202), as a rule, contains executable instructions of the operating system and the corresponding software components (applications, program modules, etc.).

[0095] A ROM (203) is one or more persistent data storage devices, for example, a hard disk (HDD), solid-state data storage (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0096] Different types of I / O interfaces (204) are used to organize the operation of the components of the device (200) and to organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific implementation of the computing device, which can be, without limitation: PCI, AGP, PS / 2, IrDa, Fire Wire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro , mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0097] Various means (205) I / O information are used to ensure user interaction with the computing device (200), for example, a keyboard, a display (monitor), a touchscreen display, a touchpad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retina scanner, fingerprint scanner, voice recognition module), etc.

[0098] The network interaction tool (206) provides data transmission by the device (200) via an internal or external computer network, for example, Intranet, Internet, LAN, etc. As one or more means (206), it can be used but not limited to : Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module, etc.

[0099] In addition, satellite navigation tools, such as GPS, GLONASS, BeiDou, Galileo, can also be used.

[0100] FIG. 4 shows an example of the network environment in the implementation of the work of the claimed system (100). The organization of work with data using HDFS is to form the appropriate levels of abstraction in cluster or virtual environments. Each system stack includes multiple computing devices, such as computers and / or servers, that communicate with the cloud platform (180) containing xstream (170) through switches. This architecture allows you to quickly increase the necessary computing power with a significant increase in the volume of stored and processed data.

[0101] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation, not going beyond the requested legal protection, which are obvious to specialists of the relevant field of technology.

Claims (43)

1. An integrated big data management system (Big Data) containing a subsystem for transporting and validating input data (hereinafter referred to as TFS), an external data loading subsystem (hereafter referred to as xStream), operating on the basis of a data description standard in which:

TFS receives, checks and transmits in xStream data from data sources, with TFS receiving data in an archived form and transmitting data over a transport protocol;

xStream contains a file download subsystem, a data processing and archiving subsystem, a data access subsystem (hereinafter hCatalog), a data download control module and an access control module, and in xStream:

the file download subsystem polls TFS to transfer data from sources to the data processing and archiving subsystem, with the data sources being pre-registered in the download control module;

the data loading subsystem performs the primary copying of files from TFS to the processing and archiving subsystem;

processing and archiving subsystem contains data storage areas that carry out structured storage of primary data, intermediate data and published data;

files are transferred to the primary data storage area from the data download subsystem, which correspond to the established archiving parameters and are stored in the originally archived form;

the processing and archiving subsystem is designed with the possibility of transferring unarchived primary data into the region of intermediate data, converted into parquet format for further transformations;

The prepared data, converted, unified data converted to the xStream standard are transferred to the published data area, in which the table and field names are given to the xStream standard, the data types are referred to the xStream standard, the tables are supplemented with the required technical fields describing the data supply - dates of the beginning and end of the period the relevance of the data delivery, the numbers of the threads that performed the download and conversion;

and from the unified data of the published data area a structured catalog is formed for access to the mentioned data by means of hCatalog. 2. The system of claim 1, characterized in that TFS checks the integrity of the archived data received. 3. The system of claim 1, characterized in that, for registered sources, the ID of said sources is stored in the download control module. 4. The system of claim 3, characterized in that the load control module controls the data flow of a plurality of sources according to the corresponding stored ID. 5. The system according to claim 4, characterized in that for each data source in the load control module contains data loading parameters. 6. The system of claim 1, characterized in that the data loading subsystem uploads files to the processing and archiving subsystem based on the file upload mask. 7. The system of claim 6, characterized in that the file upload mask is formed based on at least the name of the downloaded archive file. 8. The system of claim 1, characterized in that in the processing and archiving subsystem, in each of the data storage areas, a directory is created for storing the data of the corresponding source. 9. The system of claim 7, characterized in that the load control module checks the availability of information on all sources in the TFS. 10. The system of claim 9, characterized in that the full or incremental data loading from TFS is performed. 11. The system of claim 10, characterized in that the incremental loading is performed by the data loading module when new data is detected in TFS that differs from the previously set by the delivery date of the archive of files. 12. The system of claim 1, characterized in that the processing and archiving subsystem processes parquet files to match them to Hive-SQL types. 13. The system of claim 1, characterized in that for files being processed by the processing and archiving subsystem, a check is performed for the presence of similar data previously stored. 14. The system of claim 13, characterized in that if there are earlier data, the current version of the data is transferred to the published data area with the previous version moved to the history storage directory with partitioning by download date. 15. The system of claim 1, characterized in that hCatalog provides a mapping of the data publication data structure. 16. The system according to claim 15, characterized in that the data structure is divided into databases, corresponding to the sources of data. 17. The system of claim 1, characterized in that the data processing and archiving subsystem additionally provides an automated function to download data on any date in the past. 18. A method for managing big data (Big Data) using the transport subsystem and checking input data (hereafter referred to as TFS) and external data loading subsystem (hereafter referred to as xStream), where xStream consists of a file loading subsystem, a processing and archiving subsystem the data load control module and the access control module, the method including the steps of:

using the load control module, xStream interacts with TFS to obtain data from the sources mentioned, the data sources being pre-registered in the data load control module;

get data from the mentioned sources using TFS, which receives the data in an archived form and accumulates and verifies the data; in case of successful verification of the mentioned data, they are transferred to the data loading subsystem using the transport protocol;

using the subsystem of processing and archiving data, they process the received data, which includes accumulating files, checking files, unpacking archive files that have been checked and converting unpacked files into parquet format;

structured converted files by placing them in directories, each of which is associated with data sources registered in said data load control module;

monitor and delete duplicate data, monitor and delete data with broken structure, convert data types to Hive-SQL, monitor update of the actual data catalog, monitor and update the data change history catalog, monitor and control the depth of the data archive;

access users to the data hosted in the data access subsystem using the access control module. 19. The method according to claim 18, characterized in that the access control module determines the set of functionality based on the level of user access. 20. The method according to claim 18, characterized in that the data processing and archiving subsystem processes parquet files to match their Hive-SQL type. 21. The method according to claim 18, characterized in that the registration of data sources is carried out by recording the source ID in the download control module. 22. The method according to claim 18, characterized in that the load control module performs the data flow control of a plurality of sources according to the corresponding stored ID. 23. A method according to claim 22, characterized in that the flow control data includes checking the availability of information from the data source in TFS, processing messages from TFS, performing actions based on message processing. 24. The method according to p. 22, characterized in that the full or incremental data download from the primary input data processing subsystem is performed. 25. The method according to claim 23, characterized in that the incremental loading is performed when the loading module detects the presence of new data. 26. The method according to claim 18, characterized in that for each data source in the load control module contains data loading parameters. 27. The method according to claim 18, characterized in that the loading of files into the processing and archiving subsystem is carried out on the basis of the file loading mask. 28. The method according to p. 27, characterized in that the mask download files is formed on the basis of at least the name of the downloaded archive file.

Images