CN109936571A - A kind of mass data sharing method, opening and shares platform and electronic equipment - Google Patents
A kind of mass data sharing method, opening and shares platform and electronic equipment Download PDFInfo
- Publication number
- CN109936571A CN109936571A CN201910134290.1A CN201910134290A CN109936571A CN 109936571 A CN109936571 A CN 109936571A CN 201910134290 A CN201910134290 A CN 201910134290A CN 109936571 A CN109936571 A CN 109936571A
- Authority
- CN
- China
- Prior art keywords
- data
- user terminal
- storage
- access
- share directory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
Abstract
The invention discloses a kind of mass data sharing method, opening and shares platform and electronic equipment, which includes: the target share directory access request for receiving user terminal and sending；Target share directory is the storage metadata building of the data accessed based on user terminal requests, and share directory is associated with the configuration metadata of storage platform of data of storage user terminal requests access；Container assemblies are created according to configuration metadata, container assemblies pull the image file of the data of user terminal requests access, form shared gateway；User terminal and shared gateway are attached；The data that user terminal requests access are sent to user terminal.By forming shared gateway (as virtual data providing), to make mutually to separate data providing with the data manipulation environment of user terminal, it prevents the source data in data providing from being modified, ensure that the safety of data providing sheet itself.
Description
Technical field
The present invention relates to big data opening and shares technical field more particularly to a kind of mass data sharing methods, opening
Shared platform and electronic equipment.
Background technique
With the fast development of electronic information, each functional department of the large corporation even internal mechanism of each function towards
Respective field business demand, has built all kinds of electron business application systems, has gradually had accumulated huge data resource.These data
Resource be stored in extensively various regions, the not commensurate of mechanism at different levels, different department, not in homologous ray even different network environments, number
According to many kinds of.Due to these business application system Decentralization, the technical system disunity of use, software and hardware standard are different
It causes, data-interface poor compatibility, causes data resource isolated island and privately owned closed question serious, most data resource can not
Fusion is shared, and the efficiency that the data sharing between different business application system utilizes is lower, and efficient to large corporation is precisely controlled
Reason brings serious challenge.
Currently, the data sharing between different business application system includes diversified forms, wherein flat based on big data storage
It is one of most common sharing mode that operating component or tool, which directly carry out data retrieval, analysis, displaying, export etc., on platform, but
It is that when due to using the current data sharing method based on big data storage platform, request of data side can directly and data
Provider's connection, accesses the source data in data providing, accordingly, there exist the risks that source data is modified, and reduce source data
Safety.
Summary of the invention
In view of this, the embodiment of the invention provides a kind of mass data sharing method, opening and shares platform and electronics to set
Standby, can directly access the source data in data providing with the request of data side solved in available data sharing method causes
, there are the risk that source data is modified, the problem of reducing the safety of source data.
According in a first aspect, the embodiment of the invention provides a kind of mass data sharing methods, comprising: receive user terminal
The target share directory access request of transmission；Target share directory is the storage member number of the data accessed based on user terminal requests
According to building, share directory is associated with the configuration metadata of storage platform of data of storage user terminal requests access；According to
It configures metadata and creates container assemblies, container assemblies pull the image file of the data of user terminal requests access, are formed shared
Gateway；User terminal and shared gateway are attached；The data that user terminal requests access are sent to user terminal.
By receive user terminal transmission target share directory access request after, according to target share directory phase
The configuration metadata of the storage platform of associated, storage user terminal requests access data, creates container assemblies, container assemblies
The image file for pulling the data of user terminal requests access, forms shared gateway (as virtual data providing), thus
Make to be connected with user terminal, and by the data sharing of user terminal requests access to user terminal is the shared net
It closes, so as to mutually separate data providing with the data manipulation environment of user terminal, prevents the source number in data providing
According to being modified, the safety of data providing itself ensure that.
With reference to first aspect, in first aspect first embodiment, mass data sharing method further includes following steps:
Judge whether the connection between user terminal and shared gateway disconnects；When the connection between user terminal and shared gateway disconnects
When, destroy shared gateway.
Due to for the shared gateway as virtual data provider being accessed according to the target share directory of user terminal
Request dynamic creation, therefore only when user terminal connect with shared gateway and obtains data, the server occupied provides
Source can just be used, and when the connection between user terminal and shared gateway disconnects, share the server resource that gateway occupies
It will leave unused, result in waste of resources, therefore, when being disconnected by the connection between user terminal and shared gateway, destroy shared
Gateway can recycle the server resource for being shared gateway occupancy in time, it is idle prevent server resource, and further make to take
Business device resource can reuse, and reduce the server cost of the mass data sharing method.
With reference to first aspect, in first aspect second embodiment, mass data sharing method further includes that building is shared
The method of catalogue listing, this method comprises the following steps: multiple data of the access from multiple storage platforms；Form multiple data
Mirror image warehouse；Mirror image warehouse includes the image file of each data；It obtains the storage metadata of multiple storage platforms and configures first number
According to；Storage metadata is for describing multiple data, and configuration metadata is for describing multiple storage platforms；According to storage metadata structure
It builds the share directory of multiple data, forms share directory table, and by each share directory and the corresponding configuration in share directory table
Metadata is associated.
Second embodiment with reference to first aspect, in first aspect third embodiment, mass data sharing method is also
Include the following steps: the access authority for configuring the corresponding data of each share directory in share directory table.
By configuring the access authority of the corresponding data of each share directory in share directory table, only part is allowed to be endowed
The corresponding data of the user terminal of access authority (legitimate user) access share directory, can prevent the corresponding number of share directory
It is used according to being obtained by illegal user's terminal, improves the data sharing safety of the mass data sharing method.
Third embodiment with reference to first aspect, in the 4th embodiment of first aspect, according to configuration metadata creation
Before the step of container assemblies, container assemblies pull the image file of the data of user terminal requests access, form shared gateway,
Further include: judge whether user terminal has the permission of the corresponding data of access target share directory；It is visited when user terminal has
When asking the permission of the corresponding data of target share directory, container assemblies are created according to configuration metadata, container assemblies pull user
The image file of the data of terminal request access forms shared gateway.
With reference to first aspect, in the 5th embodiment of first aspect, user terminal and shared gateway are attached
Step, comprising: generate the login interface of shared gateway；Obtain the log-on message of a shared request of data side；Judge shared data
Whether requesting party is user terminal；When shared data requesting party is user terminal, user terminal and shared gateway are connected
It connects.
By generating the login interface of shared gateway, and verified after the log-on message that request of data side is shared in acquisition one
Whether shared data requesting party is user terminal (user terminal for having sent target share directory access request), only when shared
When request of data side is user terminal, just user terminal and shared gateway are attached, thus, it is possible to make in shared gateway
The user terminal that data have finally been requested the data obtains, and further increases the data sharing peace of the mass data sharing method
Quan Xing.
Second embodiment with reference to first aspect, in first aspect sixth embodiment, multiple storage platforms include
One of HDFS storage platform, HBase storage platform and Hive storage platform are a variety of.
According to second aspect, the embodiment of the invention provides a kind of opening and shares platforms, comprising: request receiving module is used
In the target share directory access request for receiving user terminal transmission；Target share directory is to be accessed based on user terminal requests
The storage metadata of data constructs, the configuration member number of the storage platform of the data of share directory and storage user terminal requests access
According to associated；Container creation module, for creating container assemblies according to configuration metadata, container assemblies pull user terminal requests
The image file of the data of access forms shared gateway；Terminal connecting module, for connecting user terminal and shared gateway
It connects；Data sharing module, the data for accessing user terminal requests are sent to user terminal.
In conjunction with second aspect, in second aspect first embodiment, opening and shares platform further include: connection judgment mould
Whether block, the connection for judging between user terminal and shared gateway disconnect；Resource reclaim module, for when user terminal with
When connection between shared gateway disconnects, shared gateway is destroyed.
According to the third aspect, the embodiment of the invention provides a kind of electronic equipment, comprising: memory and processor, it is described
Connection is communicated with each other between memory and the processor, computer instruction is stored in the memory, and the processor is logical
It crosses and executes the computer instruction, thereby executing sea described in any one of first aspect or first aspect embodiment
Measure data sharing method.
It is described computer-readable the embodiment of the invention provides a kind of computer readable storage medium according to fourth aspect
Storage medium stores computer instruction, and the computer instruction is for making the computer execute first aspect or first aspect
Any one embodiment described in mass data sharing method.
Detailed description of the invention
It, below will be to specific in order to illustrate more clearly of the specific embodiment of the invention or technical solution in the prior art
Embodiment or attached drawing needed to be used in the description of the prior art be briefly described, it should be apparent that, it is described below
Attached drawing is some embodiments of the present invention, for those of ordinary skill in the art, before not making the creative labor
It puts, is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of a kind of method flow diagram of mass data sharing method provided in an embodiment of the present invention；
Fig. 2 is the building flow chart of shared gateway provided in an embodiment of the present invention；
Fig. 3 is a kind of another method flow diagram of mass data sharing method provided in an embodiment of the present invention；
Fig. 4 is the access authority configuration schematic diagram of HDFS storage platform data；
Fig. 5 is the access authority configuration schematic diagram of HBase storage platform data and Hive storage platform data；
Fig. 6 is a kind of functional block diagram of opening and shares platform provided in an embodiment of the present invention；
Fig. 7 is the hardware structural diagram of a kind of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those skilled in the art are not having
Every other embodiment obtained under the premise of creative work is made, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that term " first ", " second ", " third " are used for description purposes only,
It is not understood to indicate or imply relative importance.
Embodiment 1
The present embodiment provides a kind of mass data sharing methods, are suitable for data interchange platform, which is
Information exchange platform based on computer network building.Above-mentioned data interchange platform can be connect with multiple storage platforms, be received simultaneously
The data for storing multiple data storing platforms, the basis as data sharing；And can be connect with user terminal, by user terminal
The data of request are sent to user terminal, realize data sharing.
Fig. 1 shows the flow chart of the mass data sharing method of the embodiment of the present invention, as shown in Figure 1, this method can be with
Include the following steps:
S101: multiple data of the access from multiple storage platforms.Herein, multiple storage platforms include that HDFS storage is flat
One of platform, HBase storage platform and Hive storage platform are a variety of, wherein HDFS storage platform can store text text
Part, semi-structured data and unstructured data etc., HBase storage platform can store non-relation data, and Hive storage is flat
Platform can store relation data.
S102: the mirror image warehouse of multiple data is formed.Herein, mirror image warehouse includes the image file of each data, herein
Image file realize that the data stored in image file are identical as original data by the copy to data.
S103: the storage metadata and configuration metadata of multiple storage platforms are obtained.Herein, storage metadata is for retouching
Multiple data are stated, specifically, the storage metadata of a data may include multiple metadata items.
When the data of access are the data of HDFS storage platform, the metadata item of the storage metadata of the data be can wrap
Include: data name (dataName), data storage path (dataPath), data owner (dataOwner), data content is retouched
It states (dataDesc), the storage size (dataSize) of data, storage class (datatype) and the data storage of data
One or more of date (dataDate).
When the data of access are the data of HBase storage platform, the metadata item of the storage metadata of the data can be with
It include: data table name (tableName), the affiliated database of tables of data (tableDatabase), tables of data description
(tableDesc), tables of data building maths modec (tableDate), tables of data create one in sentence (tableCreateSql)
Or it is multiple.
When the data of access are the data of Hive storage platform, the metadata item of the storage metadata of the data be can wrap
It includes: data table name (tableName), the affiliated database of tables of data (tableDatabase), tables of data description
(tableDesc), tables of data building maths modec (tableDate), tables of data create one in sentence (tableCreateSql)
Or it is multiple.
It should be noted that the type and quantity of metadata item can be adjusted according to the needs of practical application scene,
And the title of above-mentioned metadata item is only to be convenient for those skilled in the art understand that showing of being lifted of the technical solution of the embodiment of the present invention
Example should not constitute any restrictions to the embodiment of the present invention, in addition, should access again when a data produce update
The data and the storage metadata for updating the data.
Herein, configuration metadata is for describing multiple storage platforms, the main definitions service name of storage platform,
The contents such as location, port, user, safety, specifically, when storage platform is HDFS storage platform, configuration metadata may include
Core-site.xml, hdfs-site.xml, mapred-site.xml and yarn-site.xml in HDFS storage platform
Equal configuration files；When storage platform is HBase storage platform, configuration metadata may include in HBase storage platform
The configuration files such as hbase-site.xml；When storage platform is Hive storage platform, configuration metadata may include that Hive is deposited
Store up the configuration files such as the hive-site.xml in platform.It should be noted that the configuration metadata when a storage platform generates more
When new, the configuration metadata of the storage platform should be accessed and updated again.
S104: constructing the share directories of multiple data according to storage metadata, forms share directory table, and by share directory
Each share directory in table is associated with corresponding configuration metadata.Herein, share directory table can be by multiple data
Storage metadata metadata item carry out screening integration after construct.It uses the example above, when multiple storage platforms include HDFS storage
It, can be by the data name of HDFS storage platform data when platform, HBase storage platform and three kinds of Hive storage platform
(dataName), the number of the data table name (tableName) of HBase storage platform data and Hive storage platform data
The list item " data name (dataName) " being integrated into according to table name (tableName) in share directory table.
S105: the target share directory access request that user terminal is sent is received.Herein, target share directory is shared
The share directory of the data building based on user terminal requests access in catalogue listing, specifically, since share directory table is base
In multiple data storage metadata construct, contain storage metadata in some or all of metadata item, therefore, user
Information (such as data name, the storage time or data content of data for the data that terminal can access according to their needs
Description etc.), by way of browsing or searching share directory table, determine the corresponding shared mesh of data for itself needing to access
Record, then the share directory is target share directory, and user terminal is requested access to by sending target share directory access request
The corresponding data of target share directory.
S106: container assemblies are created according to configuration metadata, container assemblies pull the data of user terminal requests access
Image file forms shared gateway.Herein, configuration metadata refers to configuration metadata associated with target share directory,
It can know that (data are from what type of for the memory module of data of user terminal requests access by the configuration metadata
The data accessed in storage platform, the data e.g. accessed from HDFS storage platform), to create the container of respective type
Component.Specifically, the data accessed with user terminal requests are the data instance of HDFS storage platform access to illustrate shared net
The building process of pass, as shown in Fig. 2, then the building process includes the following steps:
S1: the Pod authority file for creating HDFS is submitted to API server by client.
Pod object is written to the data storage area etcd by S2:API server.
S3:Scheduler has found the newly created Pod for being not bound to any node of API server by monitor automatically
Object.
S4:Scheduler is unbound new pod distribution node and updates into API server.
This pod node distribution information is written to the data storage area etcd by S5:API server.
S6: the Kubelet on each node has found to distribute to the new Pod of oneself automatically by monitor.
S7:Kubelet calls Docker interface, pulls the HDFS image file (mirror of the data of user terminal requests access
As file), the container of HDFS is created on its node.
The creation of S8:HDFS container is completed and is started, and load HDFS configuration file (configuration metadata) completes the starting of Pod.
S9:Kubelet updates Pod state into API server；Pod state is saved in etcd by API server.
The pod node status information of HDFS is written to the data storage area etcd by S10:API server, shares gateway creation
It completes.
It should be noted that client, API server, the data storage area etcd, Scheduler in above-mentioned S1-S10,
Kubelet and Docker interface is container assemblies.
When the data of user terminal requests access are the data that HBase storage platform or hive storage platform access, altogether
The building process for enjoying gateway is referred to the particular content of above-mentioned S1-S10 and is understood that details are not described herein.
S107: user terminal and shared gateway are attached.Herein, sharing to generate on gateway has data sharing to connect
Mouthful, when receiving the data access request that user terminal is sent to shared gateway, by the number on user terminal and shared gateway
It is attached according to shared interface.Herein, it is asked in the data access for receiving the shared gateway transmission in a shared request of data direction
When asking, (user of target share directory access request directly can be had sent for shared data requesting party as user terminal
Terminal), and it is attached with shared gateway；The login interface that shared gateway can also be generated is obtaining the shared data
Judge whether it is user terminal (user terminal for having sent target share directory access request) after the log-on message of requesting party,
And be only attached it with shared gateway when shared data requesting party is user terminal, to make in shared gateway
The user terminal that data have finally been requested the data obtains, and the data in shared gateway can be prevented by other illegal users end
End obtains, and improves the data sharing safety of the mass data sharing method.
S108: the data that user terminal requests access are sent to user terminal.
In embodiments of the present invention, by receive user terminal transmission target share directory access request after, root
According to the configuration metadata of the storage platform of associated with target share directory, storage user terminal requests access data, wound
Container assemblies are built, container assemblies pull the image file of the data of user terminal requests access, and it is (as virtual to form shared gateway
Data providing), to make to be connected with user terminal, and by the data sharing of user terminal requests access to user
Terminal is that the shared gateway prevents so as to mutually separate data providing with the data manipulation environment of user terminal
Source data in data providing is modified, and ensure that the safety of data providing itself.
As a kind of optional embodiment of the embodiment of the present invention, as shown in Figure 1, mass data sharing method can also wrap
Include following steps:
S109: judge whether the connection between user terminal and shared gateway disconnects.Herein, when directly by the shared number
It, can be the step of the data for accessing user terminal requests be sent to user terminal (i.e. when according to requesting party as user terminal
Step S108) execute at the end of, judge that connection between user terminal and shared gateway disconnects；Shared net is generated when needing to pass through
The login interface of pass can will used when obtaining the log-on message of shared data requesting party and judging whether it is user terminal
The step of data of family terminal request access are sent to user terminal (i.e. step S108) executes completion or user terminal exits
When login, judge that the connection between user terminal and shared gateway disconnects.
S110: when the connection between user terminal and shared gateway disconnects, shared gateway is destroyed.
Due to for the shared gateway as virtual data provider being accessed according to the target share directory of user terminal
Request dynamic creation, therefore only when user terminal connect with shared gateway and obtains data, the server occupied provides
Source can just be used, and when the connection between user terminal and shared gateway disconnects, share the server resource that gateway occupies
It will leave unused, result in waste of resources, it is therefore, in the present embodiment, disconnected by the connection between user terminal and shared gateway
When opening, shared gateway is destroyed, the server resource for being shared gateway occupancy can be recycled in time, prevent server resource not busy
It sets, and further server resource is enable to reuse, reduce the server cost of the mass data sharing method.
Fig. 3 shows the flow chart of the analysis method of video code flow according to another embodiment of the present invention, as shown in figure 3,
This method may include steps of:
S201: multiple data of the access from multiple storage platforms.Particular content is with reference to described in step S101.
S202: the mirror image warehouse of multiple data is formed.Herein, mirror image warehouse includes the image file of each data.Specifically
Content is with reference to described in step S102.
S203: the storage metadata and configuration metadata of multiple storage platforms are obtained.Particular content refers to step S103 institute
It states.
S204: constructing the share directories of multiple data according to storage metadata, forms share directory table, and by share directory
Each share directory in table is associated with corresponding configuration metadata.Particular content is with reference to described in step S104.
S205: the access authority of the corresponding data of each share directory in configuration share directory table.Herein, HDFS is stored
The access authority of platform data, HBase storage platform data and Hive storage platform data can by owning user into
Row defines and divides.For HDFS storage platform data, each HDFS file or catalogue have an owner (owner) and
One group (group), file or catalogue are whole to the every other user outside the other users terminal and group of its owner, same group
There is different permissions at end respectively, for example, can set access authority to the owner possesses all permissions that read-write executes, together
The other users terminal of group only possesses read right, all user terminal lacks of competence outside group.For HBase storage platform data and
Hive storage platform data are then to be realized by assigning different access limits for user terminal, for administrator
Terminal assigns all permissions of data, for general user's terminal, then defaults it without any permission.
Herein, it needs to request access to the user (referred to herein as legitimate user) of data for each, all assigns unique
Terminal user name, the basis as data access authority management.
Specifically, for HDFS storage platform data, pass through use belonging to terminal user name and HDFS storage platform data
Family group is mapped, the mapping of Lai Shixian data access authority, as shown in figure 4, the user of the entitled datauser1 of terminal user is whole
End needs to request access to data File1 and File2, and needs to possess all permissions that read-write executes, then can be added into
The user group " gp1 " of File1, and it is set to the owner " ow1 " of user group " gp1 ", meanwhile, datauser1 is added
The user group " gp2 " of File2, and it is set to the owner " ow2 " of user group " gp2 ".
For HBase storage platform data and Hive storage platform data (generally tables of data), need to request in determination
After the terminal user name for accessing the user terminal of data, the read right of the synchronous data for needing to request access to by the user terminal is assigned
The terminal user name is given, after shared (after S209 is executed), cancels all permissions assigned.It uses the example above, such as
Shown in Fig. 5, when the user terminal of the entitled datauser1 of terminal user also needs to request access to tables of data Table1 and Table2
When, then a user terminal in the tables of data search access right user group " m1 " of Table1 directly is set by datauser1, and
A user terminal in the tables of data search access right user group " m2 " of Table2 is set by datauser1, to assign
Read right of the datauser1 for Table1 and Table2.
Herein, the access right of data can be carried out by way of increasing metadata item in the storage metadata in data
The configuration of limit specifically for HDFS storage platform data, can increase metadata item " data owner (dataOwner) "
" user group (dataOwnerGroup) where data owner ", and record has been assigned power under corresponding metadata item
The mode of the terminal user name of limit completes the access authority configuration of HDFS storage platform data；For HBase storage platform data
With Hive storage platform data, then it can increase management data table person (tableAdmin) and tables of data search access right user
(tableSelectUser), record has been assigned the mode of the terminal user name of permission and under corresponding metadata item, completes
The configuration of the access authority of HBase storage platform data and Hive storage platform data.Herein, for HBase storage platform number
According to the user name of data administrator (tableAdmin) defined in the storage metadata with Hive storage platform data, due to it
Therefore all permissions with tables of data in order to guarantee the safety of tables of data, using these user names as user is retained, are prohibited
Only it is logged in shared gateway.
It should be noted that above-mentioned terminal user name, the specific names such as data name and metadata key name are for convenience of this
Field technical staff understands the example that the technical solution of the embodiment of the present invention is lifted, and should not constitute to the embodiment of the present invention any
Limitation.
S206: the target share directory access request that user terminal is sent is received.Particular content is with reference to described in step S105.
S207: judge whether user terminal has the permission of the corresponding data of access target share directory.
S208: when user terminal has the permission of the corresponding data of access target share directory, according to configuration metadata
Container assemblies are created, container assemblies pull the image file of the data of user terminal requests access, form shared gateway.In specific
Hold with reference to described in step S106.
S209: user terminal and shared gateway are attached.Particular content is with reference to described in step S107.
S210: the data that user terminal requests access are sent to user terminal.
In embodiments of the present invention, by configuring the access rights of the corresponding data of each share directory in share directory table
Limit, the user terminal (legitimate user) for only allowing part to have been assigned access authority access the corresponding data of share directory, can
It prevents from the corresponding data of share directory from being obtained by illegal user's terminal to use, the data for improving the mass data sharing method are total
Enjoy safety.
As a kind of optional embodiment of the embodiment of the present invention, mass data sharing method can also include following step
It is rapid:
S211: judge whether the connection between user terminal and shared gateway disconnects.Particular content refers to step S109 institute
It states.
S212: when the connection between user terminal and shared gateway disconnects, shared gateway is destroyed.Particular content is with reference to step
Described in rapid S110.
Embodiment 2
Fig. 6 shows a kind of functional block diagram of opening and shares platform of the embodiment of the present invention, which can be used to implement
Mass data sharing method described in embodiment 1 or its any optional embodiment.As shown in fig. 6, the opening and shares platform
It include: request receiving module 10, container creation module 20, terminal connecting module 30 and data sharing module 40.
Wherein, request receiving module 10 is used to receive the target share directory access request of user terminal transmission；Target is total
Enjoy the storage metadata building that catalogue is the data accessed based on user terminal requests, share directory and storage user terminal requests
The configuration metadata of the storage platform of the data of access is associated；Container creation module 20 is used to be held according to configuration metadata creation
Device assembly, container assemblies pull the image file of the data of user terminal requests access, form shared gateway；Terminal connecting module
30 for user terminal and shared gateway to be attached；Data sharing module 40 is used for the data for accessing user terminal requests
It is sent to user terminal.
In embodiments of the present invention, by receive user terminal transmission target share directory access request after, root
According to the configuration metadata of the storage platform of associated with target share directory, storage user terminal requests access data, wound
Container assemblies are built, container assemblies pull the image file of the data of user terminal requests access, and it is (as virtual to form shared gateway
Data providing), to make to be connected with user terminal, and by the data sharing of user terminal requests access to user's end
End is that the shared gateway prevents from counting so as to mutually separate data providing with the data manipulation environment of user terminal
It is modified according to the source data in provider, ensure that the safety of data providing itself.
As a kind of optional embodiment of the embodiment of the present invention, as shown in fig. 6, opening and shares platform further include: connection
Judgment module 50 and resource reclaim module 60.
Wherein, connection judgment module 50 is for judging whether the connection between user terminal and shared gateway disconnects；Resource
Recycling module 60 is used to destroy shared gateway when the connection between user terminal and shared gateway disconnects.
The embodiment of the invention also provides a kind of electronic equipment, as shown in fig. 7, the electronic equipment may include processor 71
With memory 72, wherein processor 71 can be connected with memory 72 by bus or other modes, to pass through bus in Fig. 7
For connection.
Processor 71 can be central processing unit (Central Processing Unit, CPU).Processor 71 can be with
For other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
The combination of the chips such as discrete hardware components or above-mentioned all kinds of chips.
Memory 72 is used as a kind of non-transient computer readable storage medium, can be used for storing non-transient software program, non-
Transient computer executable program and module, as the corresponding program of mass data sharing method in the embodiment of the present invention refers to
Order/module (request receiving module 10 in such as Fig. 6, container creation module 20, terminal connecting module 30 and data sharing module
40).Non-transient software program, instruction and the module that processor 71 is stored in memory 72 by operation, thereby executing place
Manage the various function application and data processing of device, i.e. mass data sharing method in realization above method embodiment.
Memory 72 may include storing program area and storage data area, wherein storing program area can storage program area,
Application program required at least one function；It storage data area can the data etc. that are created of storage processor 71.In addition, storage
Device 72 may include high-speed random access memory, can also include non-transient memory, for example, at least a magnetic disk storage
Part, flush memory device or other non-transient solid-state memories.In some embodiments, it includes relative to place that memory 72 is optional
The remotely located memory of device 71 is managed, these remote memories can pass through network connection to processor 71.The reality of above-mentioned network
Example includes but is not limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
One or more of modules are stored in the memory 72, when being executed by the processor 71, are executed
Mass data sharing method in embodiment as shown in Figs. 1-5.
Above-mentioned electronic equipment detail can correspond to corresponding associated description in embodiment referring to FIG. 1 to 5
Understood with effect, details are not described herein again.
It is that can lead to it will be understood by those skilled in the art that realizing all or part of the process in above-described embodiment method
Computer program is crossed to instruct relevant hardware and complete, the program can be stored in a computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can for magnetic disk,
CD, read-only memory (Read-Only Memory, ROM), random access memory (Random Access
Memory, RAM), flash memory (Flash Memory), hard disk (Hard Disk Drive, abbreviation: HDD) or solid state hard disk
(Solid-State Drive, SSD) etc.；The storage medium can also include the combination of the memory of mentioned kind.
Obviously, the above embodiments are merely examples for clarifying the description, and does not limit the embodiments.It is right
For those of ordinary skill in the art, can also make on the basis of the above description it is other it is various forms of variation or
It changes.There is no necessity and possibility to exhaust all the enbodiments.And it is extended from this it is obvious variation or
It changes still within the protection scope of the invention.
Claims (11)
1. a kind of mass data sharing method characterized by comprising
Receive the target share directory access request that user terminal is sent；The target share directory is based on the user terminal
The storage metadata of the data requested access to constructs, the data of the share directory and the storage user terminal requests access
The configuration metadata of storage platform is associated；
Container assemblies are created according to the configuration metadata, the container assemblies pull the data of the user terminal requests access
Image file, form shared gateway；
The user terminal and the shared gateway are attached；
The data that the user terminal requests access are sent to the user terminal.
2. mass data sharing method according to claim 1, which is characterized in that further include following steps:
Judge whether the connection between the user terminal and the shared gateway disconnects；
When the connection between the user terminal and the shared gateway disconnects, the shared gateway is destroyed.
3. mass data sharing method according to claim 1, which is characterized in that further include the side for constructing share directory table
Method, this method comprises the following steps:
Access multiple data from multiple storage platforms；
Form the mirror image warehouse of the multiple data；The mirror image warehouse includes the image file of each data；
Obtain the storage metadata and configuration metadata of the multiple storage platform；The storage metadata is described more for describing
A data, the configuration metadata is for describing the multiple storage platform；
The share directory that the multiple data are constructed according to the storage metadata forms share directory table, and will be described shared
Each share directory in catalogue listing is associated with the corresponding configuration metadata.
4. mass data sharing method according to claim 3, which is characterized in that further include following steps:
Configure the access authority of the corresponding data of each share directory in the share directory table.
5. mass data sharing method according to claim 4, which is characterized in that described to be created according to the configuration metadata
Container assemblies are built, the container assemblies pull the image file of the data of the user terminal requests access, form shared gateway
The step of before, further includes:
Judge whether the user terminal has the permission for accessing the corresponding data of the target share directory；When the user is whole
When holding the permission with the corresponding data of the access target share directory, container assemblies are created according to the configuration metadata,
The container assemblies pull the image file of the data of the user terminal requests access, form shared gateway.
6. mass data sharing method according to claim 1, which is characterized in that it is described by the user terminal with it is described
The step of shared gateway is attached, comprising:
Generate the login interface of the shared gateway；
Obtain the log-on message of a shared request of data side；
Judge whether the shared data requesting party is the user terminal；
When the shared data requesting party is the user terminal, the user terminal and the shared gateway are connected
It connects.
7. mass data sharing method according to claim 3, which is characterized in that the multiple storage platform includes HDFS
One of storage platform, HBase storage platform and Hive storage platform are a variety of.
8. a kind of opening and shares platform characterized by comprising
Request receiving module, for receiving the target share directory access request of user terminal transmission；The target share directory
For the storage metadata building of the data accessed based on the user terminal requests, the share directory and the storage user are whole
Hold the configuration metadata of the storage platform of the data requested access to associated；
Container creation module, for creating container assemblies according to the configuration metadata, the container assemblies pull the user
The image file of the data of terminal request access forms shared gateway；
Terminal connecting module, for the user terminal and the shared gateway to be attached；
Data sharing module, the data for accessing the user terminal requests are sent to the user terminal.
9. opening and shares platform according to claim 8, which is characterized in that further include:
Connection judgment module, for judging whether the connection between the user terminal and the shared gateway disconnects；
Resource reclaim module, for destroying described total when the connection between the user terminal and the shared gateway disconnects
Enjoy gateway.
10. a kind of electronic equipment characterized by comprising
Memory and processor communicate with each other connection, are stored in the memory between the memory and the processor
Computer instruction, the processor is by executing the computer instruction, thereby executing the described in any item seas claim 1-7
Measure data sharing method.
11. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer to refer to
It enables, the computer instruction is for making the computer perform claim require the described in any item mass data sharing methods of 1-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910134290.1A CN109936571B (en) | 2019-02-22 | 2019-02-22 | Mass data sharing method, open sharing platform and electronic equipment |
| PCT/CN2019/100988 WO2020168692A1 (en) | 2019-02-22 | 2019-08-16 | Mass data sharing method, open sharing platform and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910134290.1A CN109936571B (en) | 2019-02-22 | 2019-02-22 | Mass data sharing method, open sharing platform and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109936571A true CN109936571A (en) | 2019-06-25 |
| CN109936571B CN109936571B (en) | 2020-05-29 |
Family
ID=66985741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910134290.1A Active CN109936571B (en) | 2019-02-22 | 2019-02-22 | Mass data sharing method, open sharing platform and electronic equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109936571B (en) |
| WO (1) | WO2020168692A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020168692A1 (en) * | 2019-02-22 | 2020-08-27 | 全球能源互联网研究院有限公司 | Mass data sharing method, open sharing platform and electronic device |
| CN112035420A (en) * | 2020-09-03 | 2020-12-04 | 西北工业大学 | Data sharing method, sharing device and system |
| CN112799740A (en) * | 2021-02-08 | 2021-05-14 | 联想(北京)有限公司 | Control method and device and electronic equipment |
| CN114424172A (en) * | 2019-10-11 | 2022-04-29 | 国际商业机器公司 | Virtual memory metadata management |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113766026B (en) * | 2021-09-09 | 2023-10-03 | 国网数字科技控股有限公司 | Data processing method and system applied to energy industrial network |
| CN114064594B (en) * | 2021-11-22 | 2023-09-22 | 马上消费金融股份有限公司 | Data processing method and device |
| CN114969205B (en) * | 2022-05-23 | 2024-04-16 | 国网安徽省电力有限公司信息通信分公司 | Data isolation sharing service system based on data center station |
| CN117390041B (en) * | 2023-12-13 | 2024-03-22 | 浙江中控信息产业股份有限公司 | Data sharing method and device, electronic equipment and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012170565A2 (en) * | 2011-06-06 | 2012-12-13 | Stone Bond Technologies, L.P. | Code generation and implementation method, system, and storage medium for delivering bidirectional data aggregation and updates |
| CN103607469A (en) * | 2013-11-28 | 2014-02-26 | 东莞中国科学院云计算产业技术创新与育成中心 | Cloud platform for achieving distributed isomerous data sharing and data sharing method thereof |
| CN105245373A (en) * | 2015-10-12 | 2016-01-13 | 天津市普迅电力信息技术有限公司 | Construction and operation method of container cloud platform system |
| CN105740048A (en) * | 2016-01-26 | 2016-07-06 | 华为技术有限公司 | Image management method, device and system |
| CN107844539A (en) * | 2017-10-19 | 2018-03-27 | 广州阿里巴巴文学信息技术有限公司 | Isolated data management method, device, system, computing device and storage medium |
| CN107979493A (en) * | 2017-11-21 | 2018-05-01 | 平安科技（深圳）有限公司 | Platform is construction method, server and the storage medium for servicing PAAS container platforms |
| CN108134764A (en) * | 2016-12-01 | 2018-06-08 | 中国电子科技集团公司第十五研究所 | A kind of Distributed data share exchange method and system |
| CN108521458A (en) * | 2018-03-30 | 2018-09-11 | 新华三云计算技术有限公司 | Application platform access method and device |
| CN108958927A (en) * | 2018-05-31 | 2018-12-07 | 康键信息技术（深圳）有限公司 | Dispositions method, device, computer equipment and the storage medium of container application |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101799894A (en) * | 2010-01-11 | 2010-08-11 | 中冶东方工程技术有限公司 | EPC engineering data processing system and method in complex network environment |
| CN101945126A (en) * | 2010-09-09 | 2011-01-12 | 中国林业科学研究院资源信息研究所 | Forest resource heterogeneous data distributed management system |
| CN102567378A (en) * | 2010-12-28 | 2012-07-11 | 上海杉达学院 | Information retrieval system based on heterogeneous data |
| CN104076906B (en) * | 2014-07-17 | 2017-05-17 | 四川传承有序数据服务有限公司 | Information technology system oriented to data |
| CN105912950B (en) * | 2016-04-13 | 2019-03-26 | 国信优易数据有限公司 | A kind of data sharing delivery system and method |
| CN108446326B (en) * | 2018-02-11 | 2019-01-29 | 江苏微锐超算科技有限公司 | A kind of isomeric data management method and system based on container |
| CN109936571B (en) * | 2019-02-22 | 2020-05-29 | 全球能源互联网研究院有限公司 | Mass data sharing method, open sharing platform and electronic equipment |
-
2019
- 2019-02-22 CN CN201910134290.1A patent/CN109936571B/en active Active
- 2019-08-16 WO PCT/CN2019/100988 patent/WO2020168692A1/en active Application Filing
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012170565A2 (en) * | 2011-06-06 | 2012-12-13 | Stone Bond Technologies, L.P. | Code generation and implementation method, system, and storage medium for delivering bidirectional data aggregation and updates |
| CN103607469A (en) * | 2013-11-28 | 2014-02-26 | 东莞中国科学院云计算产业技术创新与育成中心 | Cloud platform for achieving distributed isomerous data sharing and data sharing method thereof |
| CN105245373A (en) * | 2015-10-12 | 2016-01-13 | 天津市普迅电力信息技术有限公司 | Construction and operation method of container cloud platform system |
| CN105740048A (en) * | 2016-01-26 | 2016-07-06 | 华为技术有限公司 | Image management method, device and system |
| CN108134764A (en) * | 2016-12-01 | 2018-06-08 | 中国电子科技集团公司第十五研究所 | A kind of Distributed data share exchange method and system |
| CN107844539A (en) * | 2017-10-19 | 2018-03-27 | 广州阿里巴巴文学信息技术有限公司 | Isolated data management method, device, system, computing device and storage medium |
| CN107979493A (en) * | 2017-11-21 | 2018-05-01 | 平安科技（深圳）有限公司 | Platform is construction method, server and the storage medium for servicing PAAS container platforms |
| CN108521458A (en) * | 2018-03-30 | 2018-09-11 | 新华三云计算技术有限公司 | Application platform access method and device |
| CN108958927A (en) * | 2018-05-31 | 2018-12-07 | 康键信息技术（深圳）有限公司 | Dispositions method, device, computer equipment and the storage medium of container application |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020168692A1 (en) * | 2019-02-22 | 2020-08-27 | 全球能源互联网研究院有限公司 | Mass data sharing method, open sharing platform and electronic device |
| CN114424172A (en) * | 2019-10-11 | 2022-04-29 | 国际商业机器公司 | Virtual memory metadata management |
| CN114424172B (en) * | 2019-10-11 | 2023-03-21 | 国际商业机器公司 | Virtual memory metadata management |
| CN112035420A (en) * | 2020-09-03 | 2020-12-04 | 西北工业大学 | Data sharing method, sharing device and system |
| CN112035420B (en) * | 2020-09-03 | 2023-03-14 | 西北工业大学 | Data sharing method, sharing device and system |
| CN112799740A (en) * | 2021-02-08 | 2021-05-14 | 联想(北京)有限公司 | Control method and device and electronic equipment |
| CN112799740B (en) * | 2021-02-08 | 2022-07-26 | 联想(北京)有限公司 | Control method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109936571B (en) | 2020-05-29 |
| WO2020168692A1 (en) | 2020-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109936571A (en) | A kind of mass data sharing method, opening and shares platform and electronic equipment | |
| US6553368B2 (en) | Network directory access mechanism | |
| US7035931B1 (en) | Volume location service for a distributed file system | |
| CA2538506C (en) | A directory system | |
| US20050108394A1 (en) | Grid-based computing to search a network | |
| CN1292965A (en) | Glient-server system for maintaining application preferences in hierarchical data structure | |
| JP2007514995A (en) | Computer system, method, and program for managing an enterprise storage system | |
| CN105868333A (en) | File processing method and device | |
| CN109587233A (en) | Cloudy Container Management method, equipment and computer readable storage medium | |
| CN109005433B (en) | A kind of video cloud service platform architecture and implementation method | |
| WO2016141783A1 (en) | Method for access control, policy acquisition, attribute acquisition and related apparatus | |
| KR20120106544A (en) | Method for accessing files of a file system according to metadata and device implementing the method | |
| CN100442711C (en) | File managing system and method in digital household network | |
| CN101789963A (en) | Data synchronization system | |
| CN108563957A (en) | A kind of Subscriber Management System | |
| US20230045914A1 (en) | Method and apparatus for controlling device in internet of things, and gateway device and storage medium | |
| CN114710350B (en) | Method and device for distributing callable resources, electronic equipment and storage medium | |
| TW200540652A (en) | File locking | |
| CN114172752B (en) | Group type interconnection method for nodes of Internet of things | |
| CN102439936B (en) | Controller and computer program for privacy management in an identity network | |
| KR101592860B1 (en) | Distributed storage system using Internet of Things Device and operating method thereof | |
| CN104980329B (en) | Book management method and device, Mobile Agent Server | |
| CN110825838A (en) | Object storage aggregation system and aggregation method thereof | |
| US20140280347A1 (en) | Managing Digital Files with Shared Locks | |
| CN114866609B (en) | Data interconnection and intercommunication method and device based on unified information model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |