WO2020168692A1 - Mass data sharing method, open sharing platform and electronic device - Google Patents

Mass data sharing method, open sharing platform and electronic device Download PDF

Info

Publication number
WO2020168692A1
WO2020168692A1 PCT/CN2019/100988 CN2019100988W WO2020168692A1 WO 2020168692 A1 WO2020168692 A1 WO 2020168692A1 CN 2019100988 W CN2019100988 W CN 2019100988W WO 2020168692 A1 WO2020168692 A1 WO 2020168692A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
user terminal
shared
gateway
storage
Prior art date
Application number
PCT/CN2019/100988
Other languages
French (fr)
Chinese (zh)
Inventor
潘森
朱力鹏
乔俊峰
胡斌
周爱华
饶玮
蒋静
杨佩
裘洪斌
黄进
Original Assignee
全球能源互联网研究院有限公司
国家电网有限公司
国网上海市电力公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 全球能源互联网研究院有限公司, 国家电网有限公司, 国网上海市电力公司 filed Critical 全球能源互联网研究院有限公司
Publication of WO2020168692A1 publication Critical patent/WO2020168692A1/en

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40Support for services or applications

Definitions

  • This application relates to the field of big data open sharing technology, for example, it relates to a massive data sharing method, an open sharing platform and electronic equipment.
  • Data sharing between different business application systems includes many forms. Among them, data retrieval, analysis, display, and export based on running components or tools on the big data storage platform are one of the common sharing methods.
  • the data requester can directly connect with the data provider and access the source data in the data provider. Therefore, there is a risk of the source data being modified, which reduces the security of the source data.
  • the embodiments of the present application provide a massive data sharing method, an open sharing platform, and electronic equipment to solve the risk that the source data in the data provider can be directly accessed by the data requester in the related technology. Reduce the security of the source data.
  • the embodiment of the present application provides a massive data sharing method, including: receiving a target shared directory access request sent by a user terminal; the target shared directory is constructed based on the storage metadata of the data requested by the user terminal, and the target The shared directory is associated with the configuration metadata of the storage platform that stores the data requested by the user terminal; the container component is created based on the configuration metadata, and the container component pulls the image file of the data requested by the user terminal to form a shared gateway; connects the user terminal with The shared gateway connects; the data requested by the user terminal is sent to the user terminal.
  • an embodiment of the present application provides an open sharing platform, including: a request receiving module configured to receive a target shared directory access request sent by a user terminal; the target shared directory is based on the storage element of the data requested to be accessed by the user terminal Data is constructed, the target shared directory is associated with the configuration metadata of the storage platform that stores the data requested by the user terminal; the container creation module is set to create a container component based on the configuration metadata, and the container component pulls the data that the user terminal requests to access
  • the mirror file forms a shared gateway; the terminal connection module is set to connect the user terminal with the shared gateway; the data sharing module is set to send the data requested by the user terminal to the user terminal.
  • an embodiment of the present application provides an electronic device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory is stored with computer instructions, and the processing The device executes the computer instructions to execute the massive data sharing method described in the foregoing embodiments.
  • an embodiment of the present application provides a computer-readable storage medium that stores computer instructions, and the computer instructions are used to make the computer execute the massive data described in the above-mentioned embodiments. Sharing method.
  • FIG. 1 is a method flowchart of a massive data sharing method provided by an embodiment of this application
  • Figure 2 is a flow chart of constructing a shared gateway provided by an embodiment of the application
  • FIG. 3 is a method flowchart of another massive data sharing method provided by an embodiment of the application.
  • Figure 4 is a schematic diagram of the configuration of access permissions for HDFS storage platform data
  • Figure 5 is a schematic diagram of the configuration of access permissions for HBase storage platform data and Hive storage platform data
  • FIG. 6 is a functional block diagram of an open sharing platform provided by an embodiment of the application.
  • FIG. 7 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the application.
  • This embodiment provides a massive data sharing method, which is suitable for a data exchange platform, which is an information exchange platform constructed based on a computer network.
  • the data exchange platform can be connected to multiple data storage platforms to receive and store data from multiple data platforms as the basis for data sharing; and can be connected to user terminals to send data requested by the user terminal to the user terminal to realize data sharing.
  • Fig. 1 shows a flowchart of a method for sharing massive data according to an embodiment of the present application. As shown in Fig. 1, the method may include the following steps:
  • the multiple data storage platforms include the Hadoop Distributed File System (HDFS) storage platform, the Hadoop Database (HBase) storage platform, and the data warehouse tool Hive storage platform.
  • the HDFS storage platform can store text files, semi-structured data, unstructured data, etc.
  • the HBase storage platform can store non-relational data
  • the Hive storage platform can store relational data.
  • the mirror warehouse includes multiple mirror files of data, where the mirror files are implemented by copying data, and the data stored in the mirror files are the same as the original data.
  • S1030 Obtain storage metadata and configuration metadata of multiple data storage platforms.
  • the storage metadata is used to describe multiple data.
  • the storage metadata of one piece of data may include multiple metadata items.
  • the metadata items of the storage metadata of the data may include: data name (dataName), data storage path (dataPath), data owner (dataOwner), data content description (dataDesc), one or more of data storage size (dataSize), data storage type (datatype), and data storage date (dataDate).
  • the metadata items of the data storage metadata can include: data table name (tableName), database to which the data table belongs (tableDatabase), data table description (tableDesc), data table One or more of the creation date (tableDate) and the data table creation statement (tableCreateSql).
  • the metadata items of the storage metadata of the data can include: data table name (tableName), database to which the data table belongs (tableDatabase), data table description (tableDesc), data table One or more of the creation date (tableDate) and the data table creation statement (tableCreateSql).
  • Metadata items can be adjusted according to the needs of actual application scenarios.
  • the names of the above-mentioned metadata items are only examples to facilitate those skilled in the art to understand the technical solutions of the embodiments of this application, and should not constitute the embodiments of this application.
  • the configuration metadata is used to describe multiple storage platforms, and defines the service name, address, port, user, security, etc. of the storage platform.
  • the configuration metadata can be Including core-site.xml, hdfs-site.xml, mapred-site.xml and yarn-site.xml configuration files in the HDFS storage platform;
  • the configuration metadata can include the HBase storage platform Configuration files such as hbase-site.xml in the Hive storage platform;
  • the configuration metadata can include configuration files such as hive-site.xml in the Hive storage platform.
  • the configuration metadata of a storage platform is updated, the configuration metadata of the storage platform is re-accessed and updated.
  • S1040 Construct a shared directory of multiple data according to the storage metadata to form a shared directory table, and associate the multiple shared directories in the shared directory table with configuration metadata corresponding to the multiple shared directories.
  • the shared directory table may be constructed by filtering and integrating the metadata items of the storage metadata of multiple data.
  • the target shared directory is a shared directory constructed based on the data requested to be accessed by the user terminal in the shared directory table.
  • the shared directory table is constructed based on the storage metadata of multiple data, including Store some or all of the metadata items in the metadata. Therefore, the user terminal can browse or search the shared directory table according to the information of the data (such as data name, data storage time or data content description, etc.) that it needs to access. In this way, the shared directory corresponding to the data to be accessed is determined, the shared directory is the target shared directory, and the user terminal requests to access the data corresponding to the target shared directory by sending a target shared directory access request.
  • S1060 Create a container component according to the configuration metadata, and the container component pulls the image file of the data requested by the user terminal to form a shared gateway.
  • the configuration metadata refers to the configuration metadata associated with the target shared directory.
  • the storage mode of the data requested by the user terminal (which type of storage platform is the data from Access data) to create a corresponding type of container component.
  • the construction process of the shared gateway is explained by taking the data requested by the user terminal as the data accessed by the HDFS storage platform as an example. As shown in FIG. 2, the construction process includes the following steps:
  • S1 The client submits the Pod specification file for creating HDFS to the Application Programming Interface (API) server.
  • API Application Programming Interface
  • S2 The API server writes the Pod object to the etcd data storage area (distributed consistent key-value storage system data storage area).
  • S3 The Scheduler automatically discovers the Pod objects newly created by the API server that are not bound to any node through the monitor.
  • S4 Scheduler allocates nodes for new unbound pods and updates them to the API server.
  • S6 The Kubelet container management service on each node automatically discovers the new Pod assigned to it through the monitor.
  • the Kubelet container management service calls the Docker interface, pulls the HDFS image file (the radial file of the data that the user terminal requests to access), and creates an HDFS container on its node.
  • the HDFS container is created and started, the HDFS configuration file (configuration metadata) is loaded, and the Pod is started.
  • the Kubelet container management service updates the pod status to the API server; the API server saves the pod status to etcd.
  • the API server writes the HDFS pod node status information to the etcd data storage area, and the shared gateway is created.
  • the client, API server, etcd data storage area, Scheduler, Kubelet container management service, and Docker interface in the above S1-S10 are all container components.
  • the construction process of the shared gateway can be understood with reference to the specific content of the above S1-S10, and will not be repeated here.
  • S1070 Connect the user terminal with the shared gateway.
  • a data sharing interface is generated on the sharing gateway, and when a data access request sent by the user terminal to the sharing gateway is received, the user terminal is connected to the data sharing interface on the sharing gateway.
  • the shared data requester when a data access request sent by a shared data request to the shared gateway is received, the shared data requester can be directly used as the user terminal (the user terminal that sent the target shared directory access request), and It connects with the shared gateway; it can also generate the login interface of the shared gateway.
  • the shared data requester After obtaining the login information of the shared data requester, it is judged whether it is a user terminal (the user terminal that sent the target shared directory access request), and only in this When the shared data requester is a user terminal, connect it to the shared gateway, so that the data in the shared gateway is finally obtained by the user terminal that requested the data, preventing the data in the shared gateway from being obtained by other illegal user terminals, and improving the Data sharing security of massive data sharing method.
  • S1080 Send the data requested to be accessed by the user terminal to the user terminal.
  • the container component after receiving the target shared directory access request sent by the user terminal, the container component is created according to the configuration metadata of the storage platform that stores the data requested to be accessed by the user terminal and is associated with the target shared directory, The container component pulls the image file of the data requested by the user terminal to form a shared gateway (that is, a virtual data provider) to connect to the user terminal and share the data requested by the user terminal to the user terminal.
  • the shared gateway can separate the data supply and the data operation environment of the user terminal, prevent the source data in the data provider from being modified, and ensure the security of the data provider itself.
  • the mass data sharing method may further include the following steps:
  • S1090 Determine whether the connection between the user terminal and the shared gateway is disconnected.
  • the shared data requestor is directly used as the user terminal, it can be determined that the connection between the user terminal and the shared gateway is disconnected when the step of sending the data requested by the user terminal to the user terminal (ie S1080) is completed;
  • the step of sending the data requested by the user terminal to the user terminal i.e. S1080
  • the step of sending the data requested by the user terminal to the user terminal i.e. S1080
  • the user terminal logs out it is judged that the connection between the user terminal and the shared gateway is disconnected.
  • S1100 Destroy the shared gateway when the connection between the user terminal and the shared gateway is disconnected.
  • the shared gateway used as a virtual data provider is dynamically created based on the user terminal’s target shared directory access request, only when the user terminal connects to the shared gateway and obtains data, the server resources occupied by it will be used.
  • the server resources occupied by the shared gateway will be idle, causing a waste of resources. Therefore, in this embodiment, when the connection between the user terminal and the shared gateway is disconnected , Destroying the shared gateway can recover the server resources occupied by the shared gateway in time, prevent the server resources from being idle, and further enable the server resources to be reused, reducing the server cost for the massive data sharing method.
  • Fig. 3 shows a flowchart of a method for analyzing a video code stream according to another embodiment of the present application. As shown in Fig. 3, the method may include the following steps:
  • S201 Access multiple data from multiple data storage platforms. For details, refer to S1010.
  • S202 Form a mirror warehouse of multiple data.
  • the mirror warehouse includes a plurality of mirror files of data.
  • S1020 For details, refer to S1020.
  • S203 Obtain storage metadata and configuration metadata of multiple data storage platforms. For details, refer to S1030.
  • S204 Construct a shared directory of multiple data according to the stored metadata to form a shared directory table, and associate the multiple shared directories in the shared directory table with configuration metadata corresponding to the multiple shared directories. For details, refer to S1040.
  • the access permissions for HDFS storage platform data, HBase storage platform data, and Hive storage platform data can all be defined and divided by their users.
  • every HDFS file or directory has an owner (owner) and a group (group).
  • a file or directory has different permissions to its owner, other user terminals in the same group, and all other user terminals.
  • you can set the access permission to the owner to have all permissions to read, write, and execute, and other user terminals in the same group only Have read permission, all user terminals outside the group have no permission.
  • HBase storage platform data and Hive storage platform data it is achieved by granting different read and write permissions to user terminals.
  • For administrator user terminals all permissions are granted to data.
  • each user who needs to request access to data (herein referred to as a legitimate user) is assigned a unique terminal user name as the basis for data access authority management.
  • the data access permission mapping is realized by mapping the terminal user name and the user group to which the HDFS storage platform data belongs.
  • the user terminal with the terminal user name datauser1 needs to request access to the data Files (File)1 and File2, and need to have all the permissions to read and write, you can add them to the user group "gp1" of File1 and set them as the owner “ow1" of the user group "gp1”.
  • the configuration of data access permissions can be performed by adding metadata items to the storage metadata of the data.
  • the metadata item data owner dataOwner
  • dataOwnerGroup The user group where the person belongs, and record the corresponding information under the corresponding metadata item to complete the configuration of the access rights to the HDFS storage platform data;
  • HBase storage platform data and Hive storage platform data you can add data table management
  • S206 Receive the target shared directory access request sent by the user terminal. For details, refer to S1050.
  • S207 Determine whether the user terminal has the authority to access the data corresponding to the target shared directory.
  • S208 When the user terminal has the permission to access the data corresponding to the target shared directory, create a container component according to the configuration metadata, and the container component pulls a mirror file of the data requested by the user terminal to form Shared gateway. For details, refer to S1060.
  • S209 Connect the user terminal with the shared gateway. For details, refer to S1070.
  • S210 Send the data requested to be accessed by the user terminal to the user terminal.
  • the access permissions for data corresponding to multiple shared directories in the shared directory table by configuring the access permissions for data corresponding to multiple shared directories in the shared directory table, only some user terminals (legitimate users) with access permissions are allowed to access the data corresponding to the shared directory, preventing the shared directory
  • the corresponding data is obtained and used by illegal user terminals, which improves the data sharing security of the massive data sharing method.
  • the mass data sharing method may further include the following steps:
  • S211 Determine whether the connection between the user terminal and the shared gateway is disconnected. For details, refer to S1090.
  • S212 When the connection between the user terminal and the shared gateway is disconnected, destroy the shared gateway. For details, refer to S1100.
  • FIG. 6 shows a principle block diagram of an open sharing platform according to an embodiment of the present application.
  • the device can be configured to implement the massive data sharing method described in Embodiment 1 or any optional implementation thereof.
  • the open sharing platform includes: a request receiving module 10, a container creation module 20, a terminal connection module 30, and a data sharing module 40.
  • the request receiving module 10 is configured to receive a target shared directory access request sent by a user terminal; the target shared directory is constructed based on the storage metadata of the data requested to be accessed by the user terminal, and the target shared directory is associated with storing the user The configuration metadata of the storage platform of the data that the terminal requests to access is associated; the container creation module 20 is configured to create a container component based on the configuration metadata, and the container component pulls a mirror file of the data requested by the user terminal to form Shared gateway; the terminal connection module 30 is configured to connect the user terminal with the shared gateway; the data sharing module 40 is configured to send the data requested by the user terminal to be accessed to the user terminal.
  • the container component after receiving the target shared directory access request sent by the user terminal, the container component is created according to the configuration metadata of the storage platform that stores the data requested to be accessed by the user terminal and is associated with the target shared directory, The container component pulls the image file of the data requested by the user terminal to form a shared gateway (that is, a virtual data provider) to connect to the user terminal and share the data requested by the user terminal to the user terminal.
  • the shared gateway can separate the data supply and the data operation environment of the user terminal, prevent the source data in the data provider from being modified, and ensure the security of the data provider itself.
  • the open sharing platform further includes: a connection judgment module 50 and a resource recovery module 60.
  • connection determination module 50 is configured to determine whether the connection between the user terminal and the sharing gateway is disconnected; the resource recovery module 60 is configured to determine whether the connection between the user terminal and the sharing gateway is disconnected Next, destroy the shared gateway.
  • the embodiment of the present application also provides an electronic device.
  • the electronic device may include a processor 71 and a memory 72, where the processor 71 and the memory 72 may be connected by a bus or other means. Take bus connection as an example.
  • the processor 71 may be a central processing unit (Central Processing Unit, CPU).
  • the processor 71 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), or Chips such as other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above-mentioned multiple types of chips.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • Chips such as other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above-mentioned multiple types of chips.
  • the memory 72 can be configured to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions corresponding to the massive data sharing method in the embodiment of the present application. Modules (such as the request receiving module 10, the container creation module 20, the terminal connection module 30 and the data sharing module 40 in FIG. 6).
  • the processor 71 executes multiple functional applications and data processing of the processor by running non-transitory software programs, instructions, and modules stored in the memory 72, that is, implements the method described in any of the foregoing embodiments.
  • the memory 72 may include a program storage area and a data storage area.
  • the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created by the processor 71 and the like.
  • the memory 72 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices.
  • the memory 72 may optionally include memories remotely provided with respect to the processor 71, and these remote memories may be connected to the processor 71 through a network. Examples of the aforementioned networks include the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the one or more modules are stored in the memory 72, and when executed by the processor 71, the mass data sharing method in the embodiment shown in FIGS. 1-5 is executed.
  • the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviation: HDD) or solid-state drive (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the foregoing types of memories.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Storage Device Security (AREA)

Abstract

Disclosed are a mass data sharing method, an open sharing platform and an electronic device. The mass data sharing method comprises: receiving a target shared directory access request sent by a user terminal, wherein a target shared directory is constructed based on storage metadata of data requested to be accessed by the user terminal, and the target shared directory is associated with configuration metadata of a storage platform storing the data requested to be accessed by the user terminal; creating a container assembly according to the configuration metadata, wherein the container assembly pulls a mirror image file of the data requested to be accessed by the user terminal to form a shared gateway; connecting the user terminal and the shared gateway; and sending to the user terminal the data requested to be accessed by the user terminal.

Description

海量数据共享方法、开放共享平台及电子设备Mass data sharing method, open sharing platform and electronic equipment
本申请要求在2019年02月22日提交中国专利局、申请号为201910134290.1的中国专利申请的优先权,该申请的全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office with an application number of 201910134290.1 on February 22, 2019. The entire content of this application is incorporated into this application by reference.
技术领域Technical field
本申请涉及大数据开放共享技术领域,例如涉及一种海量数据共享方法、开放共享平台及电子设备。This application relates to the field of big data open sharing technology, for example, it relates to a massive data sharing method, an open sharing platform and electronic equipment.
背景技术Background technique
随着电子信息化的快速发展,大型机构的多个职能部门甚至多种职能的内部机构面向各自领域业务需求,建设了多类电子业务应用系统,逐步积累了庞大的数据资源。这些数据资源广泛存储于多地、多级机构的不同单位、不同部门、不同系统甚至不同网络环境中,数据种类繁多。由于这些业务应用系统分散管理,采用的技术体制不统一、软硬件标准不一致、数据接口兼容性差,导致数据资源孤岛化和私有封闭问题严重,绝大部分数据资源无法融合共享,不同业务应用系统之间的数据共享利用的效率较低,给大型机构的高效精准治理带来严重挑战。With the rapid development of electronic informatization, multiple functional departments and even multi-functional internal organizations of large organizations have built multiple types of electronic business application systems to meet the business needs of their respective fields, and gradually accumulated huge data resources. These data resources are widely stored in different units, different departments, different systems and even different network environments in multiple locations and multi-level organizations, and there are many types of data. Due to the decentralized management of these business application systems, the inconsistent technical systems, inconsistent software and hardware standards, and poor data interface compatibility have led to serious problems of islanding and private closure of data resources. Most data resources cannot be integrated and shared. Different business application systems The low efficiency of data sharing and utilization between large-scale institutions poses serious challenges to the efficient and precise governance of large institutions.
不同业务应用系统之间的数据共享包括多种形式,其中,基于大数据存储平台上运行组件或工具直接进行数据检索、分析、展示、导出等是常见的共享方式之一,但是,相关技术中,数据请求方能够直接与数据提供方连接,访问数据提供方中的源数据,因此,存在源数据被修改的风险,降低了源数据的安全性。Data sharing between different business application systems includes many forms. Among them, data retrieval, analysis, display, and export based on running components or tools on the big data storage platform are one of the common sharing methods. However, in related technologies , The data requester can directly connect with the data provider and access the source data in the data provider. Therefore, there is a risk of the source data being modified, which reduces the security of the source data.
发明内容Summary of the invention
本申请实施例提供了一种海量数据共享方法、开放共享平台及电子设备,以解决相关技术中的数据请求方能够直接访问数据提供方中的源数据导致的,存在源数据被修改的风险,降低了源数据的安全性的问题。The embodiments of the present application provide a massive data sharing method, an open sharing platform, and electronic equipment to solve the risk that the source data in the data provider can be directly accessed by the data requester in the related technology. Reduce the security of the source data.
在一实施例中,本申请实施例提供了一种海量数据共享方法,包括:接收用户终端发送的目标共享目录访问请求;目标共享目录基于用户终端请求访问 的数据的存储元数据而构建,目标共享目录与存储用户终端请求访问的数据的存储平台的配置元数据相关联;根据配置元数据创建容器组件,容器组件拉取用户终端请求访问的数据的镜像文件,形成共享网关;将用户终端与共享网关进行连接;将用户终端请求访问的数据发送给用户终端。In one embodiment, the embodiment of the present application provides a massive data sharing method, including: receiving a target shared directory access request sent by a user terminal; the target shared directory is constructed based on the storage metadata of the data requested by the user terminal, and the target The shared directory is associated with the configuration metadata of the storage platform that stores the data requested by the user terminal; the container component is created based on the configuration metadata, and the container component pulls the image file of the data requested by the user terminal to form a shared gateway; connects the user terminal with The shared gateway connects; the data requested by the user terminal is sent to the user terminal.
在一实施例中,本申请实施例提供了一种开放共享平台,包括:请求接收模块,设置为接收用户终端发送的目标共享目录访问请求;目标共享目录基于用户终端请求访问的数据的存储元数据而构建,目标共享目录与存储用户终端请求访问的数据的存储平台的配置元数据相关联;容器创建模块,设置为根据配置元数据创建容器组件,容器组件拉取用户终端请求访问的数据的镜像文件,形成共享网关;终端连接模块,设置为将用户终端与共享网关进行连接;数据分享模块,设置为将用户终端请求访问的数据发送给用户终端。In one embodiment, an embodiment of the present application provides an open sharing platform, including: a request receiving module configured to receive a target shared directory access request sent by a user terminal; the target shared directory is based on the storage element of the data requested to be accessed by the user terminal Data is constructed, the target shared directory is associated with the configuration metadata of the storage platform that stores the data requested by the user terminal; the container creation module is set to create a container component based on the configuration metadata, and the container component pulls the data that the user terminal requests to access The mirror file forms a shared gateway; the terminal connection module is set to connect the user terminal with the shared gateway; the data sharing module is set to send the data requested by the user terminal to the user terminal.
在一实施例中,本申请实施例提供了一种电子设备,包括:存储器和处理器,所述存储器和所述处理器之间互相通信连接,所述存储器中存储有计算机指令,所述处理器通过执行所述计算机指令,从而执行上述实施例所述的海量数据共享方法。In an embodiment, an embodiment of the present application provides an electronic device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory is stored with computer instructions, and the processing The device executes the computer instructions to execute the massive data sharing method described in the foregoing embodiments.
在一实施例中,本申请实施例提供了一种计算机可读存储介质,所述计算机可读存储介质存储计算机指令,所述计算机指令用于使所述计算机执行上述实施例所述的海量数据共享方法。In an embodiment, an embodiment of the present application provides a computer-readable storage medium that stores computer instructions, and the computer instructions are used to make the computer execute the massive data described in the above-mentioned embodiments. Sharing method.
附图说明Description of the drawings
图1为本申请实施例提供的一种海量数据共享方法的方法流程图;FIG. 1 is a method flowchart of a massive data sharing method provided by an embodiment of this application;
图2为本申请实施例提供的共享网关的构建流程图;Figure 2 is a flow chart of constructing a shared gateway provided by an embodiment of the application;
图3为本申请实施例提供的另一种海量数据共享方法的方法流程图;FIG. 3 is a method flowchart of another massive data sharing method provided by an embodiment of the application;
图4为HDFS存储平台数据的访问权限配置示意图;Figure 4 is a schematic diagram of the configuration of access permissions for HDFS storage platform data;
图5为HBase存储平台数据和Hive存储平台数据的访问权限配置示意图;Figure 5 is a schematic diagram of the configuration of access permissions for HBase storage platform data and Hive storage platform data;
图6为本申请实施例提供的一种开放共享平台的原理框图;FIG. 6 is a functional block diagram of an open sharing platform provided by an embodiment of the application;
图7为本申请实施例提供的一种电子设备的硬件结构示意图。FIG. 7 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the application.
具体实施方式detailed description
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行描述,所描述的实施例是本申请一部分实施例,而不是全部的实施例。在本申请的描述中,术语“第一”、“第二”、“第三”仅用于描述目的,而不能理解为指示或暗示相对重要性。The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. The described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the description of this application, the terms "first", "second", and "third" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance.
实施例1Example 1
本实施例提供一种海量数据共享方法,适用于数据交换平台,该数据交换平台为基于计算机网络构建的信息交换平台。数据交换平台可与多个数据存储平台连接,接收并存储多个数据平台的数据,作为数据共享的基础;并可以与用户终端连接,将用户终端请求的数据发送给用户终端,实现数据共享。This embodiment provides a massive data sharing method, which is suitable for a data exchange platform, which is an information exchange platform constructed based on a computer network. The data exchange platform can be connected to multiple data storage platforms to receive and store data from multiple data platforms as the basis for data sharing; and can be connected to user terminals to send data requested by the user terminal to the user terminal to realize data sharing.
图1示出了本申请实施例的海量数据共享方法的流程图,如图1所示,该方法可以包括如下步骤:Fig. 1 shows a flowchart of a method for sharing massive data according to an embodiment of the present application. As shown in Fig. 1, the method may include the following steps:
S1010:接入来自多个数据存储平台的多个数据。在一实施例中,多个数据存储平台包括海杜普分布式文件系统(Hadoop Distributed File System,HDFS)存储平台、海杜普数据库(Hadoop Database,HBase)存储平台和数据仓库工具Hive存储平台中的一种或多种,其中,HDFS存储平台可以存储文本文件、半结构化数据以及非结构化数据等;HBase存储平台可以存储非关系数据,Hive存储平台可以存储关系数据。S1010: Access multiple data from multiple data storage platforms. In an embodiment, the multiple data storage platforms include the Hadoop Distributed File System (HDFS) storage platform, the Hadoop Database (HBase) storage platform, and the data warehouse tool Hive storage platform. One or more of, among them, the HDFS storage platform can store text files, semi-structured data, unstructured data, etc.; the HBase storage platform can store non-relational data, and the Hive storage platform can store relational data.
S1020:形成多个数据的镜像仓库。在一实施例中,镜像仓库包括多个数据的镜像文件,此处的镜像文件通过对数据的拷贝来实现,镜像文件中存储的数据与原始的数据相同。S1020: Form a mirror warehouse of multiple data. In one embodiment, the mirror warehouse includes multiple mirror files of data, where the mirror files are implemented by copying data, and the data stored in the mirror files are the same as the original data.
S1030:获取多个数据存储平台的存储元数据和配置元数据。在一实施例中,存储元数据用于描述多个数据,在一实施例中,一个数据的存储元数据可以包括多个元数据项。S1030: Obtain storage metadata and configuration metadata of multiple data storage platforms. In an embodiment, the storage metadata is used to describe multiple data. In an embodiment, the storage metadata of one piece of data may include multiple metadata items.
当接入的数据为HDFS存储平台的数据时,该数据的存储元数据的元数据项可以包括:数据名称(dataName),数据存储路径(dataPath),数据所有者(dataOwner),数据内容描述(dataDesc),数据的存储大小(dataSize),数据的存储类型(datatype)以及数据存储的日期(dataDate)中的一个或多个。When the accessed data is data of the HDFS storage platform, the metadata items of the storage metadata of the data may include: data name (dataName), data storage path (dataPath), data owner (dataOwner), data content description ( dataDesc), one or more of data storage size (dataSize), data storage type (datatype), and data storage date (dataDate).
当接入的数据为HBase存储平台的数据时,该数据的存储元数据的元数据项可以包括:数据表名称(tableName),数据表所属数据库(tableDatabase), 数据表描述(tableDesc),数据表建立日期(tableDate),数据表创建语句(tableCreateSql)中的一个或多个。When the accessed data is the data of the HBase storage platform, the metadata items of the data storage metadata can include: data table name (tableName), database to which the data table belongs (tableDatabase), data table description (tableDesc), data table One or more of the creation date (tableDate) and the data table creation statement (tableCreateSql).
当接入的数据为Hive存储平台的数据时,该数据的存储元数据的元数据项可以包括:数据表名称(tableName),数据表所属数据库(tableDatabase),数据表描述(tableDesc),数据表建立日期(tableDate),数据表创建语句(tableCreateSql)中的一个或多个。When the accessed data is the data of the Hive storage platform, the metadata items of the storage metadata of the data can include: data table name (tableName), database to which the data table belongs (tableDatabase), data table description (tableDesc), data table One or more of the creation date (tableDate) and the data table creation statement (tableCreateSql).
元数据项的类型和数量可以根据实际应用场景的需要进行调整,上述元数据项的名称仅为便于本领域技术人员理解本申请实施例的技术方案所举的示例,不应当对本申请实施例构成限制,此外,当一个数据产生了更新时,重新接入并更新该数据的存储元数据。The type and quantity of metadata items can be adjusted according to the needs of actual application scenarios. The names of the above-mentioned metadata items are only examples to facilitate those skilled in the art to understand the technical solutions of the embodiments of this application, and should not constitute the embodiments of this application. Limit, in addition, when a data is updated, re-access and update the storage metadata of the data.
在一实施例中,配置元数据用于描述多个存储平台,定义了存储平台的服务名称、地址、端口、用户、安全等内容,例如,当存储平台为HDFS存储平台时,配置元数据可以包括HDFS存储平台中的core-site.xml,hdfs-site.xml,mapred-site.xml以及yarn-site.xml等配置文件;当存储平台为HBase存储平台时,配置元数据可以包括HBase存储平台中的hbase-site.xml等配置文件;当存储平台为Hive存储平台时,配置元数据可以包括Hive存储平台中的hive-site.xml等配置文件。在一实施例中,当一存储平台的配置元数据产生更新时,重新接入并更新该存储平台的配置元数据。In one embodiment, the configuration metadata is used to describe multiple storage platforms, and defines the service name, address, port, user, security, etc. of the storage platform. For example, when the storage platform is an HDFS storage platform, the configuration metadata can be Including core-site.xml, hdfs-site.xml, mapred-site.xml and yarn-site.xml configuration files in the HDFS storage platform; when the storage platform is an HBase storage platform, the configuration metadata can include the HBase storage platform Configuration files such as hbase-site.xml in the Hive storage platform; when the storage platform is the Hive storage platform, the configuration metadata can include configuration files such as hive-site.xml in the Hive storage platform. In one embodiment, when the configuration metadata of a storage platform is updated, the configuration metadata of the storage platform is re-accessed and updated.
S1040:根据存储元数据构建多个数据的共享目录,形成共享目录表,并将共享目录表中的多个共享目录与多个共享目录分别对应的配置元数据相关联。在一实施例中,共享目录表可以通过对多个数据的存储元数据的元数据项进行筛选整合后构建。沿用上例,当多个数据存储平台包括HDFS存储平台、HBase存储平台和Hive存储平台三种时,可以将HDFS存储平台数据的数据名称(dataName),HBase存储平台数据的数据表名称(tableName),以及Hive存储平台数据的数据表名称(tableName)整合为一个表项“数据名称(dataName)”。S1040: Construct a shared directory of multiple data according to the storage metadata to form a shared directory table, and associate the multiple shared directories in the shared directory table with configuration metadata corresponding to the multiple shared directories. In an embodiment, the shared directory table may be constructed by filtering and integrating the metadata items of the storage metadata of multiple data. Following the above example, when multiple data storage platforms include HDFS storage platform, HBase storage platform and Hive storage platform, the data name of the HDFS storage platform data (dataName) and the data table name of the HBase storage platform data (tableName) can be used , And the data table name (tableName) of the Hive storage platform data are integrated into one table item "data name (dataName)".
S1050:接收用户终端发送的目标共享目录访问请求。在一实施例中,目标共享目录为共享目录表中的基于用户终端请求访问的数据构建的共享目录,在一实施例中,由于共享目录表基于多个数据的存储元数据而构建,包含了存储元数据中的部分或全部的元数据项,因此,用户终端可以根据自身需要访问的数据的信息(例如数据名称,数据的存储时间或者数据内容描述等),通过浏 览或者查找共享目录表的方式,确定自身需要访问的数据对应的共享目录,则该共享目录即为目标共享目录,用户终端通过发送目标共享目录访问请求,请求访问该目标共享目录对应的数据。S1050: Receive the target shared directory access request sent by the user terminal. In one embodiment, the target shared directory is a shared directory constructed based on the data requested to be accessed by the user terminal in the shared directory table. In one embodiment, the shared directory table is constructed based on the storage metadata of multiple data, including Store some or all of the metadata items in the metadata. Therefore, the user terminal can browse or search the shared directory table according to the information of the data (such as data name, data storage time or data content description, etc.) that it needs to access. In this way, the shared directory corresponding to the data to be accessed is determined, the shared directory is the target shared directory, and the user terminal requests to access the data corresponding to the target shared directory by sending a target shared directory access request.
S1060:根据配置元数据创建容器组件,容器组件拉取用户终端请求访问的数据的镜像文件,形成共享网关。在一实施例中,配置元数据是指与目标共享目录相关联的配置元数据,通过该配置元数据可以获知用户终端请求访问的数据的存储模式(该数据时从何种类型的存储平台中接入的数据),从而创建相应类型的容器组件。在一实施例中,以用户终端请求访问的数据为HDFS存储平台接入的数据为例来阐述共享网关的构建流程,如图2所示,则该构建流程包括如下步骤:S1060: Create a container component according to the configuration metadata, and the container component pulls the image file of the data requested by the user terminal to form a shared gateway. In one embodiment, the configuration metadata refers to the configuration metadata associated with the target shared directory. Through the configuration metadata, the storage mode of the data requested by the user terminal (which type of storage platform is the data from Access data) to create a corresponding type of container component. In an embodiment, the construction process of the shared gateway is explained by taking the data requested by the user terminal as the data accessed by the HDFS storage platform as an example. As shown in FIG. 2, the construction process includes the following steps:
S1:客户端将创建HDFS的Pod规范文件提交给应用程序编程接口(Application Programming Interface,API)服务器。S1: The client submits the Pod specification file for creating HDFS to the Application Programming Interface (API) server.
S2:API服务器将Pod对象写入到etcd数据存储区(分布式一致性键值存储系统数据存储区)。S2: The API server writes the Pod object to the etcd data storage area (distributed consistent key-value storage system data storage area).
S3:调度器(Scheduler)通过监视器自动发现API服务器新创建的未绑定到任何节点的Pod对象。S3: The Scheduler automatically discovers the Pod objects newly created by the API server that are not bound to any node through the monitor.
S4:Scheduler为未绑定的新pod分配节点并更新到API服务器中。S4: Scheduler allocates nodes for new unbound pods and updates them to the API server.
S5:API服务器将此pod节点分配信息写入到etcd数据存储区。S5: The API server writes this pod node allocation information to the etcd data storage area.
S6:每个节点上的Kubelet容器管理服务通过监视器自动发现分配给自己的新Pod。S6: The Kubelet container management service on each node automatically discovers the new Pod assigned to it through the monitor.
S7:Kubelet容器管理服务调用Docker接口,拉取HDFS镜像文件(用户终端请求访问的数据的径向文件),在其节点上创建HDFS的容器。S7: The Kubelet container management service calls the Docker interface, pulls the HDFS image file (the radial file of the data that the user terminal requests to access), and creates an HDFS container on its node.
S8:HDFS容器创建完成并启动,加载HDFS配置文件(配置元数据),完成Pod的启动。S8: The HDFS container is created and started, the HDFS configuration file (configuration metadata) is loaded, and the Pod is started.
S9:Kubelet容器管理服务将Pod状态更新到API服务器中;API服务器将pod状态保存到etcd中。S9: The Kubelet container management service updates the pod status to the API server; the API server saves the pod status to etcd.
S10:API服务器将HDFS的pod节点状态信息写入到etcd数据存储区,共享网关创建完成。S10: The API server writes the HDFS pod node status information to the etcd data storage area, and the shared gateway is created.
上述S1-S10中的客户端、API服务器、etcd数据存储区、Scheduler、Kubelet容器管理服务以及Docker接口均是容器组件。The client, API server, etcd data storage area, Scheduler, Kubelet container management service, and Docker interface in the above S1-S10 are all container components.
当用户终端请求访问的数据为HBase存储平台或hive存储平台接入的数据时,共享网关的构建流程可以参照上述S1-S10的具体内容进行理解,在此不再赘述。When the data that the user terminal requests to access is the data accessed by the HBase storage platform or the hive storage platform, the construction process of the shared gateway can be understood with reference to the specific content of the above S1-S10, and will not be repeated here.
S1070:将用户终端与共享网关进行连接。在一实施例中,共享网关上生成由数据共享接口,当接收到用户终端向共享网关发送的数据访问请求时,将用户终端与共享网关上的数据共享接口进行连接。在一实施例中,可以在接收到一共享数据请求方向共享网关发送的数据访问请求时,可以直接将该共享数据请求方作为用户终端(发送了目标共享目录访问请求的用户终端),并将其与共享网关进行连接;还可以生成共享网关的登录界面,在获取该共享数据请求方的登录信息后判断其是否为用户终端(发送了目标共享目录访问请求的用户终端),并仅在该共享数据请求方为用户终端时,将其与共享网关进行连接,从而使共享网关中的数据最终被请求了该数据的用户终端获取,防止共享网关中的数据被其他非法用户终端获取,提高该海量数据共享方法的数据共享安全性。S1070: Connect the user terminal with the shared gateway. In an embodiment, a data sharing interface is generated on the sharing gateway, and when a data access request sent by the user terminal to the sharing gateway is received, the user terminal is connected to the data sharing interface on the sharing gateway. In an embodiment, when a data access request sent by a shared data request to the shared gateway is received, the shared data requester can be directly used as the user terminal (the user terminal that sent the target shared directory access request), and It connects with the shared gateway; it can also generate the login interface of the shared gateway. After obtaining the login information of the shared data requester, it is judged whether it is a user terminal (the user terminal that sent the target shared directory access request), and only in this When the shared data requester is a user terminal, connect it to the shared gateway, so that the data in the shared gateway is finally obtained by the user terminal that requested the data, preventing the data in the shared gateway from being obtained by other illegal user terminals, and improving the Data sharing security of massive data sharing method.
S1080:将用户终端请求访问的数据发送给用户终端。S1080: Send the data requested to be accessed by the user terminal to the user terminal.
在本申请实施例中,通过在接收到用户终端发送的目标共享目录访问请求后,根据与目标共享目录相关联的、存储用户终端请求访问的数据的存储平台的配置元数据,创建容器组件,容器组件拉取用户终端请求访问的数据的镜像文件,形成共享网关(即为虚拟的数据提供方),从而使与用户终端相连接,以及将用户终端请求访问的数据共享给用户终端的均为该共享网关,从而能够将数据提供发与用户终端的数据操作环境相分离,防止数据提供方中的源数据被修改,保证了数据提供方本身的安全性。In the embodiment of the present application, after receiving the target shared directory access request sent by the user terminal, the container component is created according to the configuration metadata of the storage platform that stores the data requested to be accessed by the user terminal and is associated with the target shared directory, The container component pulls the image file of the data requested by the user terminal to form a shared gateway (that is, a virtual data provider) to connect to the user terminal and share the data requested by the user terminal to the user terminal. The shared gateway can separate the data supply and the data operation environment of the user terminal, prevent the source data in the data provider from being modified, and ensure the security of the data provider itself.
作为本申请实施例的一种可选实施方式,如图1所示,海量数据共享方法还可以包括如下步骤:As an optional implementation manner of the embodiments of the present application, as shown in FIG. 1, the mass data sharing method may further include the following steps:
S1090:判断用户终端与共享网关之间的连接是否断开。当直接将该共享数据请求方作为用户终端时,可以在将用户终端请求访问的数据发送给用户终端的步骤(即S1080)执行结束时,判断用户终端与共享网关之间的连接断开;当 需要通过生成生成共享网关的登录界面,获取共享数据请求方的登录信息并判断其是否为用户终端时,可以在将用户终端请求访问的数据发送给用户终端的步骤(即S1080)执行完成,或者用户终端退出登录时,判断判断用户终端与共享网关之间的连接断开。S1090: Determine whether the connection between the user terminal and the shared gateway is disconnected. When the shared data requestor is directly used as the user terminal, it can be determined that the connection between the user terminal and the shared gateway is disconnected when the step of sending the data requested by the user terminal to the user terminal (ie S1080) is completed; When it is necessary to generate the login interface of the shared gateway to obtain the login information of the shared data requester and determine whether it is a user terminal, the step of sending the data requested by the user terminal to the user terminal (i.e. S1080) can be completed, or When the user terminal logs out, it is judged that the connection between the user terminal and the shared gateway is disconnected.
S1100:当用户终端与共享网关之间的连接断开时,销毁共享网关。S1100: Destroy the shared gateway when the connection between the user terminal and the shared gateway is disconnected.
由于用于作为虚拟数据提供方的共享网关是根据用户终端的目标共享目录访问请求动态创建的,因此只有在用户终端与共享网关连接并获取数据时,其占用的服务器资源才会被使用,而当用户终端与共享网关之间的连接断开时,共享网关占用的服务器资源就会闲置,造成资源浪费,因此,在本实施例中,通过在用户终端与共享网关之间的连接断开时,销毁共享网关,能够及时回收被共享网关占用的服务器资源,防止服务器资源被闲置,并进一步使服务器资源能够复利用,降低了给海量数据共享方法的服务器成本。Since the shared gateway used as a virtual data provider is dynamically created based on the user terminal’s target shared directory access request, only when the user terminal connects to the shared gateway and obtains data, the server resources occupied by it will be used. When the connection between the user terminal and the shared gateway is disconnected, the server resources occupied by the shared gateway will be idle, causing a waste of resources. Therefore, in this embodiment, when the connection between the user terminal and the shared gateway is disconnected , Destroying the shared gateway can recover the server resources occupied by the shared gateway in time, prevent the server resources from being idle, and further enable the server resources to be reused, reducing the server cost for the massive data sharing method.
图3示出了根据本申请另一实施例的视频码流的分析方法的流程图,如图3所示,该方法可以包括如下步骤:Fig. 3 shows a flowchart of a method for analyzing a video code stream according to another embodiment of the present application. As shown in Fig. 3, the method may include the following steps:
S201:接入来自多个数据存储平台的多个数据。具体内容参考S1010所述。S201: Access multiple data from multiple data storage platforms. For details, refer to S1010.
S202:形成多个数据的镜像仓库。在一实施例中,镜像仓库包括多个数据的镜像文件。具体内容参考S1020所述。S202: Form a mirror warehouse of multiple data. In an embodiment, the mirror warehouse includes a plurality of mirror files of data. For details, refer to S1020.
S203:获取多个数据存储平台的存储元数据和配置元数据。具体内容参考S1030所述。S203: Obtain storage metadata and configuration metadata of multiple data storage platforms. For details, refer to S1030.
S204:根据存储元数据构建多个数据的共享目录,形成共享目录表,并将共享目录表中的多个共享目录与多个共享目录分别对应的配置元数据相关联。具体内容参考S1040所述。S204: Construct a shared directory of multiple data according to the stored metadata to form a shared directory table, and associate the multiple shared directories in the shared directory table with configuration metadata corresponding to the multiple shared directories. For details, refer to S1040.
S205:配置所述共享目录表中的各共享目录对应的数据的访问权限。在一实施例中,HDFS存储平台数据、HBase存储平台数据以及Hive存储平台数据的访问权限都可以通过所属用户进行定义和划分。对于HDFS存储平台数据,每一个HDFS文件或目录都有一个所有者(owner)和一个组(group)。文件或目录对其所有者、同组的其他用户终端以及所有其他用户终端分别有着不同的权限,例如,可以将访问权限设置为所有者拥有读写执行的所有权限,同组的其他用户终端只拥有读权限,组外的所有用户终端无权限。对于HBase存储平 台数据和Hive存储平台数据,则是通过为用户终端赋予不同的读写权限来实现的,对于管理员用户终端,赋予数据的所有权限,对于一般用户终端,则默认其无任何权限。S205: Configure access permissions for data corresponding to each shared directory in the shared directory table. In an embodiment, the access permissions for HDFS storage platform data, HBase storage platform data, and Hive storage platform data can all be defined and divided by their users. For HDFS storage platform data, every HDFS file or directory has an owner (owner) and a group (group). A file or directory has different permissions to its owner, other user terminals in the same group, and all other user terminals. For example, you can set the access permission to the owner to have all permissions to read, write, and execute, and other user terminals in the same group only Have read permission, all user terminals outside the group have no permission. For HBase storage platform data and Hive storage platform data, it is achieved by granting different read and write permissions to user terminals. For administrator user terminals, all permissions are granted to data. For general user terminals, there is no permission by default. .
在一实施例中,对于每一个需要请求访问数据的用户(此处是指合法用户),都赋予唯一的终端用户名,作为数据访问权限管理的基础。In one embodiment, each user who needs to request access to data (herein referred to as a legitimate user) is assigned a unique terminal user name as the basis for data access authority management.
例如,对于HDFS存储平台数据,通过终端用户名与HDFS存储平台数据所属的用户组进行映射,来实现数据访问权限的映射,如图4所示,终端用户名为datauser1的用户终端需要请求访问数据文件(File)1和File2,并且需要拥有读写执行的所有权限,则可以将其加入File1的用户组“gp1”,并将其设置为用户组“gp1”的所有者“ow1”,同时,将datauser1加入File2的用户组“gp2”,并将其设置为用户组“gp2”的所有者“ow2”。For example, for HDFS storage platform data, the data access permission mapping is realized by mapping the terminal user name and the user group to which the HDFS storage platform data belongs. As shown in Figure 4, the user terminal with the terminal user name datauser1 needs to request access to the data Files (File)1 and File2, and need to have all the permissions to read and write, you can add them to the user group "gp1" of File1 and set them as the owner "ow1" of the user group "gp1". At the same time, Add datauser1 to the user group "gp2" of File2 and set it as the owner "ow2" of the user group "gp2".
对于HBase存储平台数据和Hive存储平台数据(一般为均数据表),在确定需要请求访问数据的终端用户名后,同步将指定表的读权限赋予终端用户名,在共享结束后(S209执行结束后),取消已赋予的所有权限。沿用上例,如图5所示,当终端用户名为datauser1的用户终端还需要请求访问数据表(Table)1和Table2时,则直接将datauser1设置为Table1的数据表查询权限用户组“m1”中的一个用户终端,并将datauser1设置为Table2的数据表查询权限用户组“m2”中的一个用户终端,从而赋予datauser1对于Table1和Table2的读权项。For HBase storage platform data and Hive storage platform data (usually a data table), after determining the user name of the terminal that needs to request access to the data, the read permission of the specified table is synchronously assigned to the terminal user name. After the sharing is completed (S209, execution ends) After), cancel all permissions that have been granted. Following the above example, as shown in Figure 5, when the user terminal with the terminal user name datauser1 needs to request access to data tables (Table)1 and Table2, directly set datauser1 as the data table query permission user group "m1" of Table1 Set datauser1 as a user terminal in the data table query permission user group "m2" of Table2, so as to give datauser1 the right to read Table1 and Table2.
在一实施例中,可以在数据的存储元数据中增加元数据项的方式进行数据的访问权限的配置,例如,对于HDFS存储平台数据,可以增加元数据项数据所有者(dataOwner)和数据所有者所在的用户组(dataOwnerGroup),并在相应的元数据项下记录相应的信息,完成HDFS存储平台数据的访问权项配置;对于HBase存储平台数据和Hive存储平台数据,则可以增加数据表管理员(tableAdmin)和数据表查询权限用户(tableSelectUser),并在相应的元数据项下记录相应的信息,完成HBase存储平台数据和Hive存储平台数据的访问权项配置。在一实施例中,对于HBase存储平台数据和Hive存储平台数据的存储元数据中定义的数据管理员(tableAdmin)的用户名,由于其具有数据表的所有权限,因此,为了保证数据表的安全性,将这些用户名作为保留用户,禁止其在共享终端登录。In an embodiment, the configuration of data access permissions can be performed by adding metadata items to the storage metadata of the data. For example, for HDFS storage platform data, the metadata item data owner (dataOwner) and data owner can be added. The user group (dataOwnerGroup) where the person belongs, and record the corresponding information under the corresponding metadata item to complete the configuration of the access rights to the HDFS storage platform data; for HBase storage platform data and Hive storage platform data, you can add data table management The user (tableAdmin) and the data table query authority user (tableSelectUser), and record the corresponding information under the corresponding metadata items, complete the configuration of the access rights items for HBase storage platform data and Hive storage platform data. In an embodiment, for the user name of the data administrator (tableAdmin) defined in the storage metadata of the HBase storage platform data and the Hive storage platform data, since it has all the permissions of the data table, in order to ensure the security of the data table Therefore, these usernames are regarded as reserved users and they are forbidden to log in on the shared terminal.
上述终端用户名,数据名以及元数据项名等具体名称均为方便本领域技术 人员理解本申请实施例的技术方案所举的示例,不应当对本申请实施例构成限制。The specific names such as the terminal user name, data name, and metadata item name mentioned above are all examples to facilitate those skilled in the art to understand the technical solutions of the embodiments of the present application, and should not limit the embodiments of the present application.
S206:接收用户终端发送的目标共享目录访问请求。具体内容参考S1050所述。S206: Receive the target shared directory access request sent by the user terminal. For details, refer to S1050.
S207:判断所述用户终端是否具有访问所述目标共享目录对应的数据的权限。S207: Determine whether the user terminal has the authority to access the data corresponding to the target shared directory.
S208:当所述用户终端具有访问所述目标共享目录对应的数据的权限时,根据所述配置元数据创建容器组件,所述容器组件拉取所述用户终端请求访问的数据的镜像文件,形成共享网关。具体内容参考S1060所述。S208: When the user terminal has the permission to access the data corresponding to the target shared directory, create a container component according to the configuration metadata, and the container component pulls a mirror file of the data requested by the user terminal to form Shared gateway. For details, refer to S1060.
S209:将用户终端与共享网关进行连接。具体内容参考S1070所述。S209: Connect the user terminal with the shared gateway. For details, refer to S1070.
S210:将用户终端请求访问的数据发送给用户终端。S210: Send the data requested to be accessed by the user terminal to the user terminal.
在本申请实施例中,通过配置共享目录表中的多个共享目录对应的数据的访问权限,只允许部分被赋予了访问权限的用户终端(合法用户)访问共享目录对应的数据,防止共享目录对应的数据被非法用户终端获取使用,提高了该海量数据共享方法的数据共享安全性。In the embodiment of the present application, by configuring the access permissions for data corresponding to multiple shared directories in the shared directory table, only some user terminals (legitimate users) with access permissions are allowed to access the data corresponding to the shared directory, preventing the shared directory The corresponding data is obtained and used by illegal user terminals, which improves the data sharing security of the massive data sharing method.
作为本申请实施例的一种可选实施方式,海量数据共享方法还可以包括如下步骤:As an optional implementation manner of the embodiment of the present application, the mass data sharing method may further include the following steps:
S211:判断用户终端与共享网关之间的连接是否断开。具体内容参考S1090所述。S211: Determine whether the connection between the user terminal and the shared gateway is disconnected. For details, refer to S1090.
S212:当用户终端与共享网关之间的连接断开时,销毁共享网关。具体内容参考S1100所述。S212: When the connection between the user terminal and the shared gateway is disconnected, destroy the shared gateway. For details, refer to S1100.
实施例2Example 2
图6示出了本申请实施例的一种开放共享平台的原理框图,该装置可以设置为实现实施例1或者其任意可选实施方式所述的海量数据共享方法。如图6所示,该开放共享平台包括:请求接收模块10,容器创建模块20,终端连接模块30和数据分享模块40。FIG. 6 shows a principle block diagram of an open sharing platform according to an embodiment of the present application. The device can be configured to implement the massive data sharing method described in Embodiment 1 or any optional implementation thereof. As shown in FIG. 6, the open sharing platform includes: a request receiving module 10, a container creation module 20, a terminal connection module 30, and a data sharing module 40.
其中,请求接收模块10设置为接收用户终端发送的目标共享目录访问请求;所述目标共享目录基于所述用户终端请求访问的数据的存储元数据而构建,所 述目标共享目录与存储所述用户终端请求访问的数据的存储平台的配置元数据相关联;容器创建模块20设置为根据所述配置元数据创建容器组件,所述容器组件拉取所述用户终端请求访问的数据的镜像文件,形成共享网关;终端连接模块30设置为将所述用户终端与所述共享网关进行连接;数据分享模块40设置为将所述用户终端请求访问的数据发送给所述用户终端。Wherein, the request receiving module 10 is configured to receive a target shared directory access request sent by a user terminal; the target shared directory is constructed based on the storage metadata of the data requested to be accessed by the user terminal, and the target shared directory is associated with storing the user The configuration metadata of the storage platform of the data that the terminal requests to access is associated; the container creation module 20 is configured to create a container component based on the configuration metadata, and the container component pulls a mirror file of the data requested by the user terminal to form Shared gateway; the terminal connection module 30 is configured to connect the user terminal with the shared gateway; the data sharing module 40 is configured to send the data requested by the user terminal to be accessed to the user terminal.
在本申请实施例中,通过在接收到用户终端发送的目标共享目录访问请求后,根据与目标共享目录相关联的、存储用户终端请求访问的数据的存储平台的配置元数据,创建容器组件,容器组件拉取用户终端请求访问的数据的镜像文件,形成共享网关(即为虚拟的数据提供方),从而使与用户终端相连接,以及将用户终端请求访问的数据共享给用户终端的均为该共享网关,从而能够将数据提供发与用户终端的数据操作环境相分离,防止数据提供方中的源数据被修改,保证了数据提供方本身的安全性。In the embodiment of the present application, after receiving the target shared directory access request sent by the user terminal, the container component is created according to the configuration metadata of the storage platform that stores the data requested to be accessed by the user terminal and is associated with the target shared directory, The container component pulls the image file of the data requested by the user terminal to form a shared gateway (that is, a virtual data provider) to connect to the user terminal and share the data requested by the user terminal to the user terminal. The shared gateway can separate the data supply and the data operation environment of the user terminal, prevent the source data in the data provider from being modified, and ensure the security of the data provider itself.
作为本申请实施例的一种可选实施方式,如图6所示,开放共享平台还包括:连接判断模块50和资源回收模块60。As an optional implementation manner of the embodiment of the present application, as shown in FIG. 6, the open sharing platform further includes: a connection judgment module 50 and a resource recovery module 60.
其中,连接判断模块50设置为判断所述用户终端与所述共享网关之间的连接是否断开;资源回收模块60设置为在所述用户终端与所述共享网关之间的连接断开的情况下,销毁所述共享网关。Wherein, the connection determination module 50 is configured to determine whether the connection between the user terminal and the sharing gateway is disconnected; the resource recovery module 60 is configured to determine whether the connection between the user terminal and the sharing gateway is disconnected Next, destroy the shared gateway.
本申请实施例还提供了一种电子设备,如图7所示,该电子设备可以包括处理器71和存储器72,其中处理器71和存储器72可以通过总线或者其他方式连接,图7中以通过总线连接为例。The embodiment of the present application also provides an electronic device. As shown in FIG. 7, the electronic device may include a processor 71 and a memory 72, where the processor 71 and the memory 72 may be connected by a bus or other means. Take bus connection as an example.
处理器71可以为中央处理器(Central Processing Unit,CPU)。处理器71还可以为其他通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现场可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等芯片,或者上述多类芯片的组合。The processor 71 may be a central processing unit (Central Processing Unit, CPU). The processor 71 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), or Chips such as other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above-mentioned multiple types of chips.
存储器72作为一种非暂态计算机可读存储介质,可设置为存储非暂态软件程序、非暂态计算机可执行程序以及模块,如本申请实施例中的海量数据共享方法对应的程序指令/模块(如图6中的请求接收模块10,容器创建模块20,终端连接模块30和数据分享模块40)。处理器71通过运行存储在存储器72中的 非暂态软件程序、指令以及模块,从而执行处理器的多种功能应用以及数据处理,即实现上述任一实施例所述的方法。As a non-transitory computer-readable storage medium, the memory 72 can be configured to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions corresponding to the massive data sharing method in the embodiment of the present application. Modules (such as the request receiving module 10, the container creation module 20, the terminal connection module 30 and the data sharing module 40 in FIG. 6). The processor 71 executes multiple functional applications and data processing of the processor by running non-transitory software programs, instructions, and modules stored in the memory 72, that is, implements the method described in any of the foregoing embodiments.
存储器72可以包括存储程序区和存储数据区,其中,存储程序区可存储操作系统、至少一个功能所需要的应用程序;存储数据区可存储处理器71所创建的数据等。此外,存储器72可以包括高速随机存取存储器,还可以包括非暂态存储器,例如至少一个磁盘存储器件、闪存器件、或其他非暂态固态存储器件。在一些实施例中,存储器72可选包括相对于处理器71远程设置的存储器,这些远程存储器可以通过网络连接至处理器71。上述网络的实例包括互联网、企业内部网、局域网、移动通信网及其组合。The memory 72 may include a program storage area and a data storage area. The program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created by the processor 71 and the like. In addition, the memory 72 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 72 may optionally include memories remotely provided with respect to the processor 71, and these remote memories may be connected to the processor 71 through a network. Examples of the aforementioned networks include the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
所述一个或者多个模块存储在所述存储器72中,当被所述处理器71执行时,执行如图1-5所示实施例中的海量数据共享方法。The one or more modules are stored in the memory 72, and when executed by the processor 71, the mass data sharing method in the embodiment shown in FIGS. 1-5 is executed.
上述电子设备具体细节可以对应参阅图1至图5所示的实施例中对应的相关描述和效果进行理解,此处不再赘述。The specific details of the above electronic device can be understood by referring to the corresponding related description and effects in the embodiments shown in FIG. 1 to FIG. 5, and will not be repeated here.
本领域技术人员可以理解,实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,所述的程序可存储于一计算机可读取存储介质中,该程序在执行时,可包括上述任一实施例所述的方法的流程。其中,所述存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)、随机存储记忆体(Random Access Memory,RAM)、快闪存储器(Flash Memory)、硬盘(Hard Disk Drive,缩写:HDD)或固态硬盘(Solid-State Drive,SSD)等;所述存储介质还可以包括上述种类的存储器的组合。Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. During execution, it may include the process of the method described in any of the foregoing embodiments. Wherein, the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviation: HDD) or solid-state drive (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the foregoing types of memories.

Claims (11)

  1. 一种海量数据共享方法,包括:A massive data sharing method, including:
    接收用户终端发送的目标共享目录访问请求;所述目标共享目录基于所述用户终端请求访问的数据的存储元数据而构建,所述目标共享目录与存储所述用户终端请求访问的数据的存储平台的配置元数据相关联;Receive a target shared directory access request sent by the user terminal; the target shared directory is constructed based on the storage metadata of the data requested to be accessed by the user terminal, and the target shared directory is connected to a storage platform that stores the data requested to be accessed by the user terminal Is associated with the configuration metadata;
    根据所述配置元数据创建容器组件,所述容器组件拉取所述用户终端请求访问的数据的镜像文件,形成共享网关;Create a container component according to the configuration metadata, and the container component pulls the image file of the data requested by the user terminal to form a shared gateway;
    将所述用户终端与所述共享网关进行连接;Connecting the user terminal with the shared gateway;
    将所述用户终端请求访问的数据发送给所述用户终端。Sending the data requested by the user terminal to access to the user terminal.
  2. 根据权利要求1所述的方法,还包括:The method according to claim 1, further comprising:
    判断所述用户终端与所述共享网关之间的连接是否断开;Determine whether the connection between the user terminal and the shared gateway is disconnected;
    在所述用户终端与所述共享网关之间的连接断开的情况下,销毁所述共享网关。When the connection between the user terminal and the shared gateway is disconnected, destroy the shared gateway.
  3. 根据权利要求1所述的方法,还包括:The method according to claim 1, further comprising:
    接入来自多个数据存储平台的多个数据;Access multiple data from multiple data storage platforms;
    形成所述多个数据的镜像仓库;所述镜像仓库包括所述多个数据的镜像文件;Forming a mirror warehouse of the multiple data; the mirror warehouse includes mirror files of the multiple data;
    获取所述多个数据存储平台的存储元数据和配置元数据;所述存储元数据用于描述所述多个数据,所述配置元数据用于描述所述多个存储平台;Acquiring storage metadata and configuration metadata of the multiple data storage platforms; the storage metadata is used to describe the multiple data, and the configuration metadata is used to describe the multiple storage platforms;
    根据所述存储元数据构建所述多个数据的共享目录,形成共享目录表,并将所述共享目录表中的多个共享目录与所述多个共享目录分别对应的所述配置元数据相关联。Construct a shared directory of the multiple data according to the storage metadata to form a shared directory table, and correlate the multiple shared directories in the shared directory table with the configuration metadata corresponding to the multiple shared directories, respectively United.
  4. 根据权利要求3所述的方法,还包括:The method according to claim 3, further comprising:
    配置所述共享目录表中的多个共享目录分别对应的数据的访问权限。Configure access permissions for data corresponding to multiple shared directories in the shared directory table.
  5. 根据权利要求4所述的方法,在所述根据所述配置元数据创建容器组件,所述容器组件拉取对应的镜像文件,形成共享网关之前,还包括:The method according to claim 4, before said creating a container component according to said configuration metadata, said container component pulling a corresponding image file to form a shared gateway, further comprising:
    判断所述用户终端是否具有访问所述目标共享目录对应的数据的权限;在所述用户终端具有访问所述目标共享目录对应的数据的权限的情况下,根据所 述配置元数据创建容器组件,所述容器组件拉取所述用户终端请求访问的数据的镜像文件,形成共享网关。Determine whether the user terminal has the authority to access the data corresponding to the target shared directory; in the case that the user terminal has the authority to access the data corresponding to the target shared directory, create a container component according to the configuration metadata, The container component pulls the image file of the data requested by the user terminal to form a shared gateway.
  6. 根据权利要求1所述的方法,其中,所述将所述用户终端与所述共享网关进行连接,包括:The method according to claim 1, wherein said connecting said user terminal with said shared gateway comprises:
    生成所述共享网关的登录界面;Generating a login interface of the shared gateway;
    获取共享数据请求方的登录信息;Obtain the login information of the requesting party of the shared data;
    判断所述共享数据请求方是否为所述用户终端;Determine whether the shared data requestor is the user terminal;
    在所述共享数据请求方为所述用户终端的情况下,将所述用户终端与所述共享网关进行连接。In a case where the shared data requestor is the user terminal, connect the user terminal with the sharing gateway.
  7. 根据权利要求3所述的方法,其中,所述数据存储平台包括海杜普分布式文件系统HDFS存储平台、海杜普数据库HBase存储平台和数据仓库工具Hive存储平台中的一种或多种。The method according to claim 3, wherein the data storage platform comprises one or more of the Hidupu Distributed File System HDFS storage platform, the Hidupu database HBase storage platform, and the data warehouse tool Hive storage platform.
  8. 一种开放共享平台,包括:An open sharing platform, including:
    请求接收模块,设置为接收用户终端发送的目标共享目录访问请求;所述目标共享目录基于所述用户终端请求访问的数据的存储元数据而构建,所述目标共享目录与存储所述用户终端请求访问的数据的存储平台的配置元数据相关联;The request receiving module is configured to receive a target shared directory access request sent by the user terminal; the target shared directory is constructed based on the storage metadata of the data requested to be accessed by the user terminal, and the target shared directory is configured to store the user terminal request The configuration metadata of the storage platform of the accessed data is associated;
    容器创建模块,设置为根据所述配置元数据创建容器组件,所述容器组件拉取所述用户终端请求访问的数据的镜像文件,形成共享网关;A container creation module, configured to create a container component based on the configuration metadata, and the container component pulls a mirror file of the data requested by the user terminal to form a shared gateway;
    终端连接模块,设置为将所述用户终端与所述共享网关进行连接;A terminal connection module, configured to connect the user terminal with the shared gateway;
    数据分享模块,设置为将所述用户终端请求访问的数据发送给所述用户终端。The data sharing module is configured to send the data requested to be accessed by the user terminal to the user terminal.
  9. 根据权利要求8所述的开放共享平台,还包括:The open sharing platform according to claim 8, further comprising:
    连接判断模块,设置为判断所述用户终端与所述共享网关之间的连接是否断开;The connection judgment module is configured to judge whether the connection between the user terminal and the sharing gateway is disconnected;
    资源回收模块,设置为在所述用户终端与所述共享网关之间的连接断开的情况下,销毁所述共享网关。The resource recovery module is configured to destroy the shared gateway when the connection between the user terminal and the shared gateway is disconnected.
  10. 一种电子设备,包括:An electronic device including:
    存储器和处理器,所述存储器和所述处理器之间互相通信连接,所述存储器中存储有计算机指令,所述处理器通过执行所述计算机指令,从而执行权利要求1-7任一项所述的方法。A memory and a processor, the memory and the processor are in communication connection with each other, and computer instructions are stored in the memory, and the processor executes the computer instructions by executing any one of claims 1-7 The method described.
  11. 一种计算机可读存储介质,所述计算机可读存储介质存储有计算机指令,所述计算机指令用于使所述计算机执行权利要求1-7任一项所述的方法。A computer-readable storage medium storing computer instructions for causing the computer to execute the method according to any one of claims 1-7.
PCT/CN2019/100988 2019-02-22 2019-08-16 Mass data sharing method, open sharing platform and electronic device WO2020168692A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910134290.1A CN109936571B (en) 2019-02-22 2019-02-22 Mass data sharing method, open sharing platform and electronic equipment
CN201910134290.1 2019-02-22

Publications (1)

Publication Number Publication Date
WO2020168692A1 true WO2020168692A1 (en) 2020-08-27

Family

ID=66985741

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/100988 WO2020168692A1 (en) 2019-02-22 2019-08-16 Mass data sharing method, open sharing platform and electronic device

Country Status (2)

Country Link
CN (1) CN109936571B (en)
WO (1) WO2020168692A1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113766026A (en) * 2021-09-09 2021-12-07 国网电子商务有限公司 Data processing method and system applied to energy industrial network
CN113938890A (en) * 2021-09-15 2022-01-14 青岛海信移动通信技术股份有限公司 Data sharing method and terminal equipment
CN114064594A (en) * 2021-11-22 2022-02-18 马上消费金融股份有限公司 Data processing method and device
CN114969205A (en) * 2022-05-23 2022-08-30 国网安徽省电力有限公司信息通信分公司 Data isolation sharing service system based on data middling station
CN117370310A (en) * 2023-10-19 2024-01-09 中电云计算技术有限公司 Distributed file system cross-cluster data increment migration method
CN117390041A (en) * 2023-12-13 2024-01-12 浙江中控信息产业股份有限公司 Data sharing method and device, electronic equipment and storage medium
CN118035178A (en) * 2024-04-11 2024-05-14 恒生电子股份有限公司 Data investigation method, device, electronic equipment, computer readable storage medium and program product

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109936571B (en) * 2019-02-22 2020-05-29 全球能源互联网研究院有限公司 Mass data sharing method, open sharing platform and electronic equipment
US11436033B2 (en) * 2019-10-11 2022-09-06 International Business Machines Corporation Scalable virtual memory metadata management
CN112035420B (en) * 2020-09-03 2023-03-14 西北工业大学 Data sharing method, sharing device and system
CN112799740B (en) * 2021-02-08 2022-07-26 联想(北京)有限公司 Control method and device and electronic equipment
CN117834702A (en) * 2023-12-29 2024-04-05 慧之安信息技术股份有限公司 Gateway sharing connection control method and system based on containerized service configuration

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101799894A (en) * 2010-01-11 2010-08-11 中冶东方工程技术有限公司 EPC engineering data processing system and method in complex network environment
CN101945126A (en) * 2010-09-09 2011-01-12 中国林业科学研究院资源信息研究所 Forest resource heterogeneous data distributed management system
CN102567378A (en) * 2010-12-28 2012-07-11 上海杉达学院 Information retrieval system based on heterogeneous data
US20160164899A1 (en) * 2014-07-17 2016-06-09 Chengdu University Of Technology Data-oriented information technology system
CN105912950A (en) * 2016-04-13 2016-08-31 国信优易数据有限公司 Data sharing and publishing system and method
CN108446326A (en) * 2018-02-11 2018-08-24 江苏微锐超算科技有限公司 A kind of isomeric data management method and system based on container
CN109936571A (en) * 2019-02-22 2019-06-25 全球能源互联网研究院有限公司 A kind of mass data sharing method, opening and shares platform and electronic equipment

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012170565A2 (en) * 2011-06-06 2012-12-13 Stone Bond Technologies, L.P. Code generation and implementation method, system, and storage medium for delivering bidirectional data aggregation and updates
CN103607469B (en) * 2013-11-28 2017-05-17 东莞中国科学院云计算产业技术创新与育成中心 Data sharing method of cloud platform for achieving distributed isomerous data sharing
CN105245373B (en) * 2015-10-12 2017-08-04 天津市普迅电力信息技术有限公司 A kind of container cloud platform system is built and operation method
CN105740048B (en) * 2016-01-26 2019-03-08 华为技术有限公司 A kind of mirror image management method, apparatus and system
CN108134764B (en) * 2016-12-01 2021-01-29 中国电子科技集团公司第十五研究所 Distributed data sharing and exchanging method and system
CN107844539A (en) * 2017-10-19 2018-03-27 广州阿里巴巴文学信息技术有限公司 Isolated data management method, device, system, computing device and storage medium
CN107979493B (en) * 2017-11-21 2019-10-29 平安科技(深圳)有限公司 Platform is construction method, server and the storage medium for servicing PAAS container platform
CN108521458B (en) * 2018-03-30 2021-01-01 新华三云计算技术有限公司 Application platform access method and device
CN108958927B (en) * 2018-05-31 2023-04-18 康键信息技术(深圳)有限公司 Deployment method and device of container application, computer equipment and storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101799894A (en) * 2010-01-11 2010-08-11 中冶东方工程技术有限公司 EPC engineering data processing system and method in complex network environment
CN101945126A (en) * 2010-09-09 2011-01-12 中国林业科学研究院资源信息研究所 Forest resource heterogeneous data distributed management system
CN102567378A (en) * 2010-12-28 2012-07-11 上海杉达学院 Information retrieval system based on heterogeneous data
US20160164899A1 (en) * 2014-07-17 2016-06-09 Chengdu University Of Technology Data-oriented information technology system
CN105912950A (en) * 2016-04-13 2016-08-31 国信优易数据有限公司 Data sharing and publishing system and method
CN108446326A (en) * 2018-02-11 2018-08-24 江苏微锐超算科技有限公司 A kind of isomeric data management method and system based on container
CN109936571A (en) * 2019-02-22 2019-06-25 全球能源互联网研究院有限公司 A kind of mass data sharing method, opening and shares platform and electronic equipment

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113766026A (en) * 2021-09-09 2021-12-07 国网电子商务有限公司 Data processing method and system applied to energy industrial network
CN113766026B (en) * 2021-09-09 2023-10-03 国网数字科技控股有限公司 Data processing method and system applied to energy industrial network
CN113938890A (en) * 2021-09-15 2022-01-14 青岛海信移动通信技术股份有限公司 Data sharing method and terminal equipment
CN113938890B (en) * 2021-09-15 2024-04-30 青岛海信移动通信技术有限公司 Data sharing method and terminal equipment
CN114064594A (en) * 2021-11-22 2022-02-18 马上消费金融股份有限公司 Data processing method and device
CN114064594B (en) * 2021-11-22 2023-09-22 马上消费金融股份有限公司 Data processing method and device
CN114969205B (en) * 2022-05-23 2024-04-16 国网安徽省电力有限公司信息通信分公司 Data isolation sharing service system based on data center station
CN114969205A (en) * 2022-05-23 2022-08-30 国网安徽省电力有限公司信息通信分公司 Data isolation sharing service system based on data middling station
CN117370310A (en) * 2023-10-19 2024-01-09 中电云计算技术有限公司 Distributed file system cross-cluster data increment migration method
CN117370310B (en) * 2023-10-19 2024-05-28 中电云计算技术有限公司 Distributed file system cross-cluster data increment migration method
CN117390041A (en) * 2023-12-13 2024-01-12 浙江中控信息产业股份有限公司 Data sharing method and device, electronic equipment and storage medium
CN117390041B (en) * 2023-12-13 2024-03-22 浙江中控信息产业股份有限公司 Data sharing method and device, electronic equipment and storage medium
CN118035178A (en) * 2024-04-11 2024-05-14 恒生电子股份有限公司 Data investigation method, device, electronic equipment, computer readable storage medium and program product

Also Published As

Publication number Publication date
CN109936571A (en) 2019-06-25
CN109936571B (en) 2020-05-29

Similar Documents

Publication Publication Date Title
WO2020168692A1 (en) Mass data sharing method, open sharing platform and electronic device
US20210385087A1 (en) Zero-knowledge identity verification in a distributed computing system
US11082226B2 (en) Zero-knowledge identity verification in a distributed computing system
US11550763B2 (en) Versioning schemas for hierarchical data structures
US11574070B2 (en) Application specific schema extensions for a hierarchical data structure
US10423342B1 (en) Scaling events for hosting hierarchical data structures
US9311326B2 (en) Virtual file system for automated data replication and review
US7620630B2 (en) Directory system
KR100959473B1 (en) Systems and methods for interfacing application programs with an item-based storage platform
US20160156631A1 (en) Methods and systems for shared file storage
US10999370B1 (en) Syncing and sharing data across systems
KR20060080921A (en) Systems and methods for data modeling in an item-based storage platform
US20160179840A1 (en) Cloud bursting a database
US20200125453A1 (en) Systems and methods for cross-regional back up of distributed databases on a cloud service
WO2017174013A1 (en) Data storage management method and apparatus, and data storage system
US11100129B1 (en) Providing a consistent view of associations between independently replicated data objects
US9665732B2 (en) Secure Download from internet marketplace
US11425132B2 (en) Cross-domain authentication in a multi-entity database system
CN109063061B (en) Cross-distributed system data processing method, device, equipment and storage medium
US11500874B2 (en) Systems and methods for linking metric data to resources
RU2656739C1 (en) Data storage method and system
US11010361B1 (en) Executing code associated with objects in a hierarchial data structure
US10708253B2 (en) Identity information including a schemaless portion
CN113448775B (en) Multi-source heterogeneous data backup method and device
WO2023040504A1 (en) Data processing system, data processing method and related apparatus

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19916415

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19916415

Country of ref document: EP

Kind code of ref document: A1