CN115391413A - Multi-source data access aggregation method and device - Google Patents
Multi-source data access aggregation method and device Download PDFInfo
- Publication number
- CN115391413A CN115391413A CN202211320059.XA CN202211320059A CN115391413A CN 115391413 A CN115391413 A CN 115391413A CN 202211320059 A CN202211320059 A CN 202211320059A CN 115391413 A CN115391413 A CN 115391413A
- Authority
- CN
- China
- Prior art keywords
- data
- data access
- source data
- subscription
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 230000002776 aggregation Effects 0.000 title claims description 11
- 238000004220 aggregation Methods 0.000 title claims description 11
- 230000008569 process Effects 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 8
- 238000007726 management method Methods 0.000 abstract description 30
- 238000013523 data management Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 13
- 238000003860 storage Methods 0.000 description 12
- 230000006872 improvement Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 238000013461 design Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 5
- 230000006978 adaptation Effects 0.000 description 2
- 230000007123 defense Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
Abstract
The embodiment of the specification discloses a multi-source data access convergence method and a device, and the method comprises the steps of writing start configuration information of a data access system into a startup script; in the starting process, the data access system is automatically started; accessing multi-source data after starting the data access system; establishing a metadata management model to convert the accessed multi-source data into a preset data format to obtain different types of theme data; and constructing a topic data subscription template, so that the user selects topic data needing subscription based on the topic data subscription template. The multi-source data access convergence method provided by the embodiment of the specification can realize self-starting of a data access system, carry out unified format management on accessed multi-source data, and simplify the complexity of data center configuration and data management.
Description
Technical Field
The embodiment of the specification relates to the technical field of data processing, in particular to a multi-source data access convergence method and device.
Background
With the deep integration of industrialization and informatization, the demand for interconnection and intercommunication between a production control system and a production management system inside an enterprise and between different departments of the enterprise is increasing, or interconnection and intercommunication of information such as service data and the like are required between different organizations of the same service.
The service data, the internet of things data, or other file data come from different devices, different mechanisms, etc., the data format structures are completely different, and in order to facilitate the data query of users needing the data, the prior art provides a data access convergence system, which converges and accesses the data from different sources into one system, so that the users can directly query the needed data through the data access convergence system.
However, the conventional data access aggregation system has complex system configuration and high management difficulty, the accessed multi-source data cannot be in a uniform format, the data query is difficult, and the operation of the data access aggregation system is easily influenced by the environment.
Disclosure of Invention
In view of this, embodiments of the present disclosure provide a multi-source data access convergence method and apparatus, so as to solve the problems in the prior art that a conventional data access convergence system is difficult to operate and configure and is easily influenced by the environment during operation.
The embodiment of the specification adopts the following technical scheme:
an embodiment of the present specification provides a multi-source data access aggregation method, including:
writing the starting configuration information of the data access system into a starting script;
in the starting process, the data access system is automatically started;
accessing multi-source data after starting the data access system;
establishing a metadata management model to convert the accessed multi-source data into a preset data format to obtain different types of theme data;
and constructing a topic data subscription template, so that the user selects topic data needing subscription based on the topic data subscription template.
Further, accessing multi-source data, comprising:
a data access task is newly established;
configuring a data link;
and accessing multi-source data, wherein the multi-source data comprises data of an internet of things device, data of a business information system and data of other files.
Further, after accessing the multi-source data, the method further comprises:
scanning the accessed multi-source data to obtain a scanning result;
judging whether the multi-source data is transmitted successfully or not according to the scanning result to obtain a first judgment result;
if the first judgment result is yes, writing the accessed multi-source data into a file record table;
and if the first judgment result is negative, continuing to perform multi-source data transmission.
Further, establishing a metadata management model, including:
according to a preset dimension, performing data classification on the accessed multi-source data;
and performing label processing on the multi-source data of different types to obtain the subject data of different label types.
Further, constructing a topic data subscription template, so that a user selects topic data to be subscribed based on the topic data subscription template, including:
selecting a subscription element using the topic data subscription template;
and generating an offline subscription task according to the selected subscription element, wherein the offline subscription task carries a user identifier.
Further, the subscription element at least comprises theme data to be subscribed, a data updating period and a triggering condition.
Further, the topic data subscription template is displayed in an interface of the data access system in a view mode.
Further, the startup script is a startup script of the data access all-in-one machine, and the data access system is loaded in the data access all-in-one machine;
the method further comprises the following steps:
and if the data access all-in-one machine has equipment failure, restarting the data access all-in-one machine so as to automatically restart the data access system and automatically recover the data access operation.
An embodiment of the present specification further provides a multi-source data access convergence device, including:
the configuration module writes the starting configuration information of the data access system into the starting script;
the starting module is used for self-starting the data access system in the starting process;
the data access module accesses multi-source data after starting the data access system;
the model establishing module is used for establishing a metadata management model so as to convert the accessed multi-source data into a preset data format and obtain different types of theme data;
and the data subscription module is used for constructing a topic data subscription template so that a user selects topic data needing to be subscribed based on the topic data subscription template.
The embodiment of the specification adopts at least one technical scheme which can achieve the following beneficial effects:
the method comprises the steps of writing start configuration information of a data access system into a start-up script, automatically starting the data access system in the start-up process, accessing multi-source data after the data access system is started, establishing a metadata management model to convert the accessed multi-source data into a preset data format, obtaining theme data of different categories, and constructing a theme data subscription template, so that a user selects the theme data to be subscribed based on the theme data subscription template.
In addition, the accessed multi-source data is managed in a unified format by establishing a metadata management model and a topic data subscription template, so that the operation configuration of the data access system can be greatly simplified, and a user can conveniently inquire and subscribe the multi-source data.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the specification and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the specification and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic flowchart of a multi-source data access aggregation method provided in an embodiment of the present disclosure;
fig. 2 is a schematic operation flow diagram of a multi-source data access aggregation system according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a multi-source data access convergence device according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be clearly and completely described below with reference to the specific embodiments of the present specification and the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step are within the scope of the present application.
The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings.
As shown in fig. 1, a schematic flow diagram of a multi-source data access aggregation method provided in an embodiment of the present specification is provided, and an execution subject of the multi-source data access aggregation method provided in the embodiment of the present specification may be a server.
The multi-source data access convergence method is applied to a data access system, and specifically comprises the following steps:
s101: writing the starting configuration information of the data access system into a starting script;
in this embodiment of the present specification, the data access system may be a data system that aggregates data from different sources, and the data access system may aggregate multi-source data together and perform unified management, so as to facilitate query and subscription of a user who needs to query data.
The startup configuration information of the data access system may specifically be information such as a code and a configuration file required for starting the data access system.
In the embodiment of the present specification, the startup script is a startup script of a data access all-in-one machine, and the data access system is loaded in the data access all-in-one machine;
the method may further comprise:
and if the data access all-in-one machine has equipment failure, restarting the data access all-in-one machine so as to automatically restart the data access system and automatically recover data access operation.
Under the condition, when the data access all-in-one machine has equipment faults such as network faults and the like, and the data access system can be automatically restarted when the data access all-in-one machine is restarted, the data access operation before the equipment faults is automatically recovered, the data access task does not need to be restarted and reconfigured additionally, and the data access operation under the complex environment is more convenient.
The data access system may be mounted in the data access all-in-one machine, that is, the startup configuration information of the data access system is specifically written in a startup script of the data access all-in-one machine. The data access all-in-one machine is nationwide production equipment and adopts a Feiteng chip and an kylin operating system.
For the step S101, the start configuration information of the data access system is written into the start script by performing integrated deployment on the data access all-in-one machine, and when the data access all-in-one machine is started, the data access system is self-started, so that the problems of complex environment setup and deployment configuration of the data access system are solved, and the start-up self-start function of the data access system is realized.
S102: in the starting process, the data access system is automatically started;
as an application embodiment, before booting, the multi-source data access aggregation method may further include:
and adapting the data access system with the data access all-in-one machine.
In the adaptation process, a software architecture and a version adapted to a domestic system are adopted, and the configuration of a start script of a data access system is deployed in combination, so that the adaptation and the rapid deployment of the data access all-in-one machine are completed, and the normal use of the data access system is ensured.
S103: accessing multi-source data after starting the data access system;
in this embodiment of this specification, for the step S103, accessing multi-source data may include:
a data access task is newly established;
configuring a data link;
multi-source data is accessed.
The multi-source data may specifically include data from different sources, such as internet of things data, business information data, device operation data, and other file data, which are not described herein again.
The configuration data link may be performed manually, and the configured data may specifically include information such as a host name of a data source, a network address, a port number, a user name, and a password.
Further, after accessing the multi-source data, the method may further include:
scanning the accessed multi-source data to obtain a scanning result;
judging whether the multi-source data is transmitted successfully or not according to the scanning result to obtain a first judgment result;
if the first judgment result is yes, writing the accessed multi-source data into a file record table;
if the first judgment result is negative, multi-source data transmission is continued.
In the embodiment of the specification, whether the multi-source data is transmitted successfully is judged by scanning the accessed multi-source data, if the multi-source data is transmitted successfully, the accessed multi-source data is written into the file record table, if interruption occurs in the data transmission process, the multi-source data is not output successfully, and the multi-source data is not written into the file record table during the next file scanning, so that the data is transmitted continuously after the multi-source data is scanned continuously until the multi-source data is transmitted successfully, and the data is recorded into the file record table.
S104: establishing a metadata management model to convert the accessed multi-source data into a preset data format to obtain different types of subject data;
as an application example, for the step S104, the metadata management model may include basic information such as a data ID, a data name, a file size, a file path, a file type, a creation time, a creator, and the like.
And performing unified representation, data integration and authority management on the accessed multi-source data by establishing a metadata management model. On the basis of ensuring the integrity and consistency of metadata, a unified view of each level of elements is constructed through a logic model, and data services such as data access and data downloading are supported.
Further, establishing the metadata management model may specifically include:
according to a preset dimension, performing data classification on the accessed multi-source data;
and performing label processing on the multi-source data of different types to obtain the subject data of different label types.
In the embodiment of the specification, the accessed multi-source data is organized and managed by adopting hierarchical classification and labels, wherein the preset dimensionality can comprise a business angle, a data management angle and the like, the multi-source data with the same attribute and/or the same characteristic is classified into one class, different classes such as a data source dimensionality, a business dimensionality, a data open dimensionality and the like are formed, the multi-source data is convenient to inquire, identify, manage, protect and utilize, and the multi-source data classification is realized.
S105: and constructing a topic data subscription template, so that the user selects topic data needing subscription based on the topic data subscription template.
In this embodiment of the present specification, the topic data subscription template may be a window displayed on a display interface of the data access system, and the topic data generated in step S104 is subscribed and distributed by constructing the topic data subscription template, so as to provide a topic data subscription entry for the user, and support the user to flexibly establish a subscription rule by using the topic data subscription template.
Specifically, the method for supporting the user to flexibly establish the subscription rule by using the topic data subscription template may include:
selecting subscription elements by using the topic data subscription template, wherein the subscription elements at least comprise topic data to be subscribed, a data updating period and a triggering condition;
and generating an offline subscription task according to the selected subscription element, wherein the offline subscription task carries a user identifier.
The triggering condition may be a triggering condition for generating an offline subscription task, for example, after the user selects a subscription element, the user may click a completion button, as the triggering condition, the data access system automatically generates the offline subscription task carrying the user identifier, and subsequently, according to the subscription element selected by the user, regularly pushes the subscribed theme data to the user as needed.
Furthermore, the data access system can also provide functions of task state query, message abstract pushing, result pushing authority control and the like for the user, and the subscribed data content is stored in the storage space with authority.
As an application embodiment, the topic data subscription template is displayed in an interface of the data access system in a view mode.
In the data access convergence method provided in the embodiments of the present description, the start configuration information of the data access system is written into the start script, during the start process, the data access system may be automatically started, after the data access system is started, the multi-source data is accessed, and the metadata management model is established, so as to convert the accessed multi-source data into the preset data format, obtain the subject data of different types, and construct the subject data subscription template, so that the user selects the subject data to be subscribed based on the subject data subscription template.
In addition, the accessed multi-source data is managed in a unified format by establishing a metadata management model and a topic data subscription template, so that the operation configuration of the data access system can be greatly simplified, and a user can conveniently inquire and subscribe the multi-source data.
Fig. 2 is a schematic operation flow diagram of a data access convergence system provided in an embodiment of the present disclosure.
S201: starting a multi-source data access and convergence all-in-one machine;
in the embodiment of the specification, the multi-source data access and convergence all-in-one machine adopts a home-made design, and core devices such as a CPU, an FPGA and storage particles all adopt home-made chips.
Furthermore, the machine body of the multi-source data access and aggregation all-in-one machine adopts the innate strengthening design technology, the derating design is carried out according to the relevant regulations of GJB/Z35-1993 component derating criterion, and the omnibearing severe environment resistance design is carried out from multiple dimensions such as temperature resistance, air pressure resistance, vibration impact resistance, three prevention, electromagnetic compatibility and the like, so that the data access system has the capability of resisting the severe environment.
Therefore, the multi-source data access convergence all-in-one machine can encapsulate complex environment, reduce environmental influence and simplify the complexity of configuration and management of a data center.
S202: the data access convergence system is started automatically;
s203: a data access task is newly established, and various equipment data, file data and service data are automatically acquired;
in the embodiment of the present specification, the data access convergence system is a plug-and-play data access, and supports automatic data acquisition of a specific device and system, the data types may include internet of things data, service information data, file data, and the like, and the convergence mode includes library table scanning, file scanning, and the like.
S204: scanning the accessed multi-source data;
s205: judging whether the multi-source data is transmitted successfully or not to obtain a first judgment result;
s206: if the first judgment result is yes, writing the accessed multi-source data into a file record table;
s207: if the first judgment result is negative, returning to the step S204 to continue the multi-source data transmission;
s208: performing metadata management and data classification management on the accessed multi-source data;
in the embodiment of the specification, the data access system performs local self-organization management on accessed multi-source data, supports organization management of local data, and provides support functions such as metadata management and classification management for the data.
Furthermore, the data access system provides a classification management function for data, and may specifically adopt a data classification technology to define data according to the sensitivity of the data, and classify the data from the perspective of data protection, so as to perform security measures such as access control and data encryption. Meanwhile, data assets are planned by means of a data label system of the data access system, so that data can be read, are easy to understand and are convenient to use.
S209: providing local data services, which may include data access and data download;
s210: and providing topic data subscription service according to the constructed topic data subscription template.
The multisource data access system provided by the embodiment of the specification can automatically acquire the equipment data, the operation data and the service data of various devices, can provide customized data service for an information system by carrying out unified classification management on the access data, and adopts a self-organized data management mode in the data access system, so that the data access workload is greatly reduced, and the data organization circulation efficiency is effectively improved. Meanwhile, the data access system supports two modes of calculation, storage, separation and integrated deployment, supports various severe environment resistant designs, can effectively cover complicated and various use environments of frontier defense and marine defense, and has great application and popularization values.
Fig. 3 is a schematic structural diagram of a multi-source data access convergence device provided in an embodiment of the present disclosure based on the same inventive concept.
Wherein, the multi-source data access convergence device may include:
the configuration module 301 writes start-up configuration information of the data access system into the startup script;
a starting module 302, which is used for self-starting the data access system in the starting process;
the data access module 303 accesses multi-source data after the data access system is started;
the model establishing module 304 is used for establishing a metadata management model so as to convert the accessed multi-source data into a preset data format and obtain different types of subject data;
the data subscription module 305 constructs a topic data subscription template, so that the user selects topic data to be subscribed based on the topic data subscription template.
Further, accessing multi-source data, comprising:
a data access task is newly established;
configuring a data link;
and accessing multi-source data, wherein the multi-source data comprises internet of things data, business information data, equipment operation data and other file data.
Further, after accessing the multi-source data, the method further includes:
scanning the accessed multi-source data to obtain a scanning result;
judging whether the multi-source data is transmitted successfully or not according to the scanning result to obtain a first judgment result;
if the first judgment result is yes, writing the accessed multi-source data into a file record table;
and if the first judgment result is negative, continuing to perform multi-source data transmission.
Further, establishing a metadata management model, including:
according to a preset dimension, performing data classification on the accessed multi-source data;
and performing label processing on the multi-source data of different types to obtain the subject data of different label types.
Further, constructing a topic data subscription template, so that a user selects topic data to be subscribed based on the topic data subscription template, including:
selecting a subscription element by using the topic data subscription template;
and generating an offline subscription task according to the selected subscription element, wherein the offline subscription task carries a user identifier.
Further, the subscription element at least includes theme data to be subscribed, a data update period, and a trigger condition.
Further, the topic data subscription template is displayed in an interface of the data access system in a view mode.
The multi-source data access convergence device provided in the embodiment of the present specification writes start configuration information of a data access system into a start-up script, and in a start-up process, the data access system may be automatically started, and after the data access system is started, multi-source data is accessed, and a metadata management model is established to convert the accessed multi-source data into a preset data format, obtain subject data of different types, and construct a subject data subscription template, so that a user selects subject data to be subscribed based on the subject data subscription template.
In addition, the accessed multi-source data is managed in a unified format by establishing a metadata management model and a topic data subscription template, so that the operation configuration of the data access system can be greatly simplified, and a user can conveniently inquire and subscribe the multi-source data.
In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (e.g., improvements in method flows). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain a corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is an integrated circuit whose Logic functions are determined by a user programming the Device. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development, but the original code before compiling is also written in a specific Programming Language, which is called Hardware Description Language (HDL), and the HDL is not only one kind but many kinds, such as abll (Advanced boot Expression Language), AHDL (alternate hard Description Language), traffic, CUPL (computer universal Programming Language), HDCal (Java hard Description Language), lava, lola, HDL, PALASM, software, rhydl (Hardware Description Language), and vhul-Language (vhyg-Language), which is currently used in the field. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information and/or data which can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or apparatus comprising the element.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the system embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for relevant points.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art to which the present application pertains. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
Claims (10)
1. A multi-source data access aggregation method is characterized by comprising the following steps:
writing the starting configuration information of the data access system into a starting script;
in the starting process, the data access system is automatically started;
accessing multi-source data after starting the data access system;
establishing a metadata management model to convert the accessed multi-source data into a preset data format to obtain different types of subject data;
and constructing a topic data subscription template, so that the user selects topic data needing subscription based on the topic data subscription template.
2. The method of claim 1, wherein accessing multi-source data comprises:
a data access task is newly established;
configuring a data link;
and accessing multi-source data, wherein the multi-source data comprises data of an internet of things device, data of a business information system and data of other files.
3. The method of claim 2, after accessing multi-source data, further comprising:
scanning the accessed multi-source data to obtain a scanning result;
judging whether the multi-source data is transmitted successfully or not according to the scanning result to obtain a first judgment result;
if the first judgment result is yes, writing the accessed multi-source data into a file record table;
and if the first judgment result is negative, continuing to perform multi-source data transmission.
4. The method of claim 1, wherein building a metadata management model comprises:
according to a preset dimension, performing data classification on the accessed multi-source data;
and performing label processing on the multi-source data of different types to obtain the subject data of different label types.
5. The method of claim 1, wherein constructing a topic data subscription template such that a user selects topic data to be subscribed based on the topic data subscription template comprises:
selecting a subscription element using the topic data subscription template;
and generating an offline subscription task according to the selected subscription element, wherein the offline subscription task carries a user identifier.
6. The method of claim 5, wherein the subscription elements comprise at least topic data to subscribe, a data update period, and a trigger condition.
7. The method of claim 5, wherein the topic data subscription template is presented in a view within an interface of the data access system.
8. The method of claim 1, wherein the startup script is a startup script of a data access all-in-one machine, and the data access system is loaded in the data access all-in-one machine;
the method further comprises the following steps:
and if the data access all-in-one machine has equipment failure, restarting the data access all-in-one machine so as to automatically restart the data access system and automatically recover data access operation.
9. A multi-source data access convergence device, comprising:
the configuration module writes the starting configuration information of the data access system into the starting script;
the starting module is used for self-starting the data access system in the starting process;
the data access module accesses multi-source data after starting the data access system;
the model establishing module is used for establishing a metadata management model so as to convert the accessed multi-source data into a preset data format and obtain the subject data of different categories;
and the data subscription module is used for constructing a topic data subscription template so that a user selects topic data needing to be subscribed based on the topic data subscription template.
10. The apparatus of claim 9, after accessing multi-source data, further comprising:
scanning the accessed multi-source data to obtain a scanning result;
judging whether the multi-source data is transmitted successfully or not according to the scanning result to obtain a first judgment result;
if the first judgment result is yes, writing the accessed multi-source data into a file record table;
and if the first judgment result is negative, continuing to perform multi-source data transmission.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211320059.XA CN115391413A (en) | 2022-10-26 | 2022-10-26 | Multi-source data access aggregation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211320059.XA CN115391413A (en) | 2022-10-26 | 2022-10-26 | Multi-source data access aggregation method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115391413A true CN115391413A (en) | 2022-11-25 |
Family
ID=84128195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211320059.XA Pending CN115391413A (en) | 2022-10-26 | 2022-10-26 | Multi-source data access aggregation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115391413A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949850A (en) * | 2020-08-14 | 2020-11-17 | 北京锐安科技有限公司 | Multi-source data acquisition method, device, equipment and storage medium |
| CN114363096A (en) * | 2022-03-21 | 2022-04-15 | 中国电建集团华东勘测设计研究院有限公司 | Data access transmission system crossing safety partitions |
| CN115018642A (en) * | 2022-06-08 | 2022-09-06 | 国泰君安证券股份有限公司 | System for realizing high-availability receiving and processing aiming at multi-source real-time market data |
-
2022
- 2022-10-26 CN CN202211320059.XA patent/CN115391413A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949850A (en) * | 2020-08-14 | 2020-11-17 | 北京锐安科技有限公司 | Multi-source data acquisition method, device, equipment and storage medium |
| CN114363096A (en) * | 2022-03-21 | 2022-04-15 | 中国电建集团华东勘测设计研究院有限公司 | Data access transmission system crossing safety partitions |
| CN115018642A (en) * | 2022-06-08 | 2022-09-06 | 国泰君安证券股份有限公司 | System for realizing high-availability receiving and processing aiming at multi-source real-time market data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101788492B1 (en) | Mediated data exchange for sandboxed applications | |
| CN108628688B (en) | Message processing method, device and equipment | |
| CN108924029B (en) | Method and device for sending customer service data | |
| EP3025255A1 (en) | Virtual synchronization with on-demand data delivery | |
| CN111475142B (en) | Method, device and equipment for generating applet file package | |
| CN109947643B (en) | A/B test-based experimental scheme configuration method, device and equipment | |
| CN111400681B (en) | Data authority processing method, device and equipment | |
| CN110032409B (en) | Client screen adapting method and device and electronic equipment | |
| US9513762B1 (en) | Static content updates | |
| CN117075930B (en) | Computing framework management system | |
| CN111212227B (en) | Method, device and equipment for configuring video resolution of camera and storage medium | |
| US20150067701A1 (en) | Creating a custom series of commands | |
| CN111273965A (en) | Container application starting method, system and device and electronic equipment | |
| CN109144600B (en) | Application program running method and device and computer readable medium | |
| CN112559127B (en) | Virtual machine creation method, device, host and storage medium | |
| CN106156050B (en) | Data processing method and device | |
| CN112231384B (en) | Distributed application localization transformation method and equipment adapting to multiple data sources | |
| CN110633096A (en) | Node control method and device, version control method and device and distributed system | |
| CN110045997B (en) | Object initialization method, device, equipment and storage medium of basic function module | |
| CN115391413A (en) | Multi-source data access aggregation method and device | |
| CN111338655A (en) | Installation package distribution method and system | |
| CN110874322A (en) | Test method and test server for application program | |
| CN111835513B (en) | Method, device and equipment for updating certificate data | |
| CN114594944A (en) | Application construction system and device | |
| CN117519912B (en) | Mirror image warehouse deployment method, device, storage medium and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221125 |