CN107729394A - Data Mart management system and its application method based on Hadoop clusters - Google Patents
Data Mart management system and its application method based on Hadoop clusters Download PDFInfo
- Publication number
- CN107729394A CN107729394A CN201710854312.2A CN201710854312A CN107729394A CN 107729394 A CN107729394 A CN 107729394A CN 201710854312 A CN201710854312 A CN 201710854312A CN 107729394 A CN107729394 A CN 107729394A
- Authority
- CN
- China
- Prior art keywords
- information
- subsystem
- data
- metadata
- data mart
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The invention discloses the Data Mart management system and its application method based on Hadoop clusters, it is related to field of computer technology.Including：Cluster management subsystem is used for the configuration information for obtaining Data Mart, and configuration information is synchronized in rights management subsystem, is additionally operable to obtain the metadata information of Data Mart, metadata information is synchronized into data administration subsystem；Data message is synchronized in rights management subsystem by data administration subsystem again；Data administration subsystem is used for the modification information of the metadata in synchrodata fairground, then the modification information of metadata is synchronized in rights management subsystem；Application method includes：Access interface subsystem receives access request；From authority corresponding to the acquisition of rights management subsystem, and response is determined according to request and authority.Which overcomes the management of data nundinal to be short of, the cumbersome time-consuming technical problem in user accesses data fairground, and then reduces O＆M cost, improves the effect of data Use Limitation.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of Data Mart management system based on Hadoop clusters
And its application method, electronic equipment and computer-readable medium.
Background technology
With the extension of business event, substantial amounts of data can be produced in operation management and production process, and can be efficiently fast
Fast ground, which is analyzed and calculated to these data, directly influences big data value in the application and effect.Big data management
One of mode is using based on Hadoop Clusterings, (Hadoop is a kind of distributed system architecture, is divided available for realizing
Cloth file system, Hadoop clusters are exactly that will be handled in substantial amounts of data distribution to different machines) establish data set
City or data warehouse are managed to data, wherein, data fair refers generally to small-sized analytic type database, in order to from each
The analysis theme (such as user, cost, commodity) abstracted in the numerous and diverse business of kind is analyzed and established, and has very high collection
Become second nature.
In process of the present invention is realized, inventor has found that at least there are the following problems in the prior art：
Data Mart lacks effectively management, it is conducted interviews etc. operation when, it is necessary to go database bottom to obtain data set
The configuration information in city, process are cumbersome time-consuming；The metadata information weak management of Data Mart, poor in timeliness；User is to database
Courses of action are long when conducting interviews, and reduce the service efficiency of data, the data management of Data Mart shortcoming collection, data calculate and
Metadata management allows users to the number needed for inquiry from Data Mart rapidly and efficiently in the automated management system of one
According to.
The content of the invention
In view of this, the embodiment of the present invention provides a kind of Data Mart management system based on Hadoop clusters, can be right
Data Mart based on Hadoop clusters carries out data management, data calculate and metadata management is in the automatic management of one,
O＆M cost is reduced, allows users to the data needed for inquiry from Data Mart rapidly and efficiently.
To achieve the above object, one side according to embodiments of the present invention, there is provided a kind of based on Hadoop clusters
Data Mart management system, including：Cluster management subsystem；Data administration subsystem；Rights management subsystem；Access interface
System；Wherein, the cluster management subsystem is used to obtaining the configuration information of the Data Mart, and by the configuration information
It is synchronized to the rights management subsystem；The cluster management subsystem is additionally operable to obtain the metadata letter of the Data Mart
Breath, and the metadata information is synchronized to the data administration subsystem, the data administration subsystem is again by the member
Data message is synchronized to the rights management subsystem；The data administration subsystem is used for the first number for obtaining the Data Mart
According to modification information, the modification information of the metadata is synchronized to the rights management subsystem by the data administration subsystem again
System；The access interface subsystem is used to receive access request, from authority corresponding to rights management subsystem acquisition, and
The response to the access request is determined according to the request and the authority.
Alternatively, the Data Mart management system also includes：Messenger service subsystem, for by the Data Mart
The modification information of metadata is synchronized to the data administration subsystem；Wherein described messenger service subsystem includes：Memory cell,
For the information for the metadata for preserving the Data Mart；Log unit, for preserving data change in the memory cell
Information；Subscriber units, for obtaining and preserving the information in the log unit in real time；TU task unit, for by the subscription
The information preserved in unit is converted to the modification information of metadata, and by the synchronizing information to the data administration subsystem.
Alternatively, the cluster management subsystem obtains the Data Mart by the interface of HTTP type
Configuration information.
Alternatively, the subscriber units obtain information in the log unit in real time by configuring real-time acquisition tasks,
And preserve the information and be used for message subscribing.
Alternatively, the TU task unit is converted to the information preserved in the subscriber units by establishing stream process task
The modification information of metadata, and by the synchronizing information to the data administration subsystem.
To achieve the above object, other side according to embodiments of the present invention, there is provided one kind uses and is based on Hadoop
The method that the Data Mart management system of cluster accesses Data Mart, including：Cluster management subsystem obtains the Data Mart
Configuration information, and configuration information is synchronized in the rights management subsystem；The cluster management subsystem obtains institute
The metadata information of Data Mart is stated, and the metadata information is synchronized to data administration subsystem, the data management
The metadata information is synchronized to the rights management subsystem by subsystem again；The data administration subsystem obtains the number
According to the modification information of the metadata in fairground, the modification information of the metadata is synchronized to described by the data administration subsystem again
Rights management subsystem；Access interface subsystem receive access request, from the rights management subsystem obtain corresponding to authority,
And the response to the access request is determined according to the request and the authority.
Alternatively, methods described also includes：Messenger service subsystem in the Data Mart management system is by the number
The data administration subsystem is synchronized to according to the modification information of the metadata in fairground；Wherein, in the messenger service subsystem
Memory cell preserves the information of the metadata of the Data Mart；Described in log unit in the messenger service subsystem preserves
The information that data change in memory cell；Subscriber units in the messenger service subsystem obtain and preserve the daily record in real time
Information in unit；The information preserved in the subscriber units is converted to member by the TU task unit in the messenger service subsystem
The modification information of data, and by the synchronizing information to the data administration subsystem.
Alternatively, the cluster management subsystem obtains the Data Mart by the interface of HTTP type
Configuration information.
Alternatively, the subscriber units are by configuring the information in the real-time acquisition tasks acquisition log unit, and protect
Deposit the information and be used for message subscribing.
Alternatively, the TU task unit is converted to the information preserved in the subscriber units by establishing stream process task
The modification information of metadata, and by the synchronizing information to the data administration subsystem.
To achieve the above object, another aspect according to embodiments of the present invention, there is provided one kind uses and is based on Hadoop
The Data Mart management system of cluster accesses the electronic equipment of Data Mart, it is characterised in that including：
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are by one or more of computing devices so that one or more of processing
Device is realized accesses any described system in Data Mart method using the Data Mart management system based on Hadoop clusters.
To achieve the above object, another aspect according to embodiments of the present invention, there is provided one kind uses and is based on Hadoop
The Data Mart management system of cluster accesses the computer-readable medium of Data Mart, is stored thereon with computer program, and it is special
Sign is, is realized when described program is executed by processor and accesses data using the Data Mart management system based on Hadoop clusters
Any described method in the method for fairground.
One embodiment in foregoing invention has the following advantages that or beneficial effect：Because using introducing cluster management subsystem
System, data administration subsystem, rights management subsystem and interface access sub-system, data management, data are carried out to Data Mart
Calculate and metadata management is in the automatic management technological means of one, be short of so overcoming and data nundinal being managed, used
Family access the cumbersome time-consuming technical problem of Data Mart, and then reduce O＆M cost, allow users to rapidly and efficiently from
The technique effect of data needed for inquiry in Data Mart.
Further effect adds hereinafter in conjunction with embodiment possessed by above-mentioned non-usual optional mode
With explanation.
Brief description of the drawings
Accompanying drawing is used to more fully understand the present invention, does not form inappropriate limitation of the present invention.Wherein：
Fig. 1 is that the Data Mart management system of use according to embodiments of the present invention based on Hadoop clusters accesses data set
The schematic diagram of the key step of the method in city；
Fig. 2 is that the embodiment of the present invention can apply to exemplary system architecture figure therein；
Fig. 3 is adapted for the structural representation for realizing the terminal device of the embodiment of the present invention or the computer system of server
Figure.
Embodiment
The one exemplary embodiment of the present invention is explained below in conjunction with accompanying drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
Arrive, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, the description to known function and structure is eliminated in following description.
Fig. 1 is that the Data Mart management system of use according to embodiments of the present invention based on Hadoop clusters accesses data set
The schematic diagram of the key step of the method in city, as shown in figure 1, the timely management system 100 of the data based on Hadoop clusters
Including cluster management subsystem 101, data administration subsystem 102, rights management subsystem 103, the and of access interface subsystem 104
Messenger service subsystem 105.
Cluster management subsystem 101 is used for the configuration information for obtaining the Data Mart, and the configuration information is same
Walk to rights management subsystem 103；Described in cluster management subsystem 101 can be obtained by the interface of HTTP type
The configuration information of Data Mart.The configuration information of wherein Data Mart refer to Hadoop cluster names where Data Mart,
Bulk encoding, the RM addresses (Hadoop cluster resource managers Yarn management end address) of cluster, JH addresses (Hadoop resources
Manager Yarn historic task list address), NS addresses (full name NameSpace, be NameNode in Hadoop management text
Part system), user name, production account information, queuing message etc..
Cluster management subsystem 101 is additionally operable to obtain the metadata information of the Data Mart, and by the metadata
The metadata information is synchronized to rights management by synchronizing information to data administration subsystem 102, data administration subsystem 102 again
Subsystem 103；Wherein Data Mart metadata information includes：Library name and table name under fairground Name and Description information, fairground, table
The information such as structure, the description information of table, table director, in addition to library name, table name, field name, table structure, table description information, table
Director and creation time etc..
Data administration subsystem 102 is used for the modification information for obtaining the metadata of the Data Mart, the data management
The modification information of the metadata is synchronized to rights management subsystem 103 by subsystem again.
Messenger service subsystem 104, for the modification information of the metadata of the Data Mart to be synchronized into data management
Subsystem 102.Wherein messenger service subsystem 104 includes：Memory cell 1041, for preserving the metadata of the Data Mart
Information, conventional storage element has MySQL types database (a kind of Relational DBMS increased income)；Log unit
1042, it is MySQL numbers such as the Binlog files of MySQL database for preserving the information that data change in the memory cell
According to an attribute in storehouse, preserved, recorded to data generation or the potential SQL statement changed in the form of binary log；Order
Unit 1043 is read, for obtaining and preserving the information in log unit 1042 in real time；Subscriber units 1043 are adopted in real time by configuring
Set task, the data of production system are gathered, and data are reported into real time data bus, obtained in real time in log unit 1042
Information, and preserve the information and be used for message subscribing, such as kafka (a kind of distributed post of high-throughput subscribes to message system)；
TU task unit 1044, for the information preserved in subscriber units 1043 to be converted to the modification information of metadata, and by the information
The data administration subsystem 102 is synchronized to, TU task unit 1044 can be by establishing stream process task, as Storm tasks are (a kind of
The generic primitives that distribution calculates in real time handle message and update the data storehouse in real time) information preserved in subscriber units 1043 is turned
It is changed to the modification information of metadata, and by the synchronizing information to data administration subsystem 102.
Access interface subsystem 104 is used to receive access request, and corresponding authority is obtained from rights management subsystem 103,
And the response to the access request is determined according to the request and the authority.When have user need to Data Mart carry out
When accessing (1001), access interface subsystem 105 is used to receive access request；Access interface subsystem 105 is sub from rights management
Authority corresponding to being obtained in system 103, and according to the response of the request and authority determination to the access request
(1002).For example, user, when carrying out building table handling on access interface subsystem 105, access interface subsystem 105 can call power
Limit management subsystem 103 carries out automatic authorization, and authorized content includes the configuration of the metadata information and Data Mart of Data Mart
Information, after user carries out data query or data subscription on access interface subsystem 105, it can be held by task execution client
Row SQL query (MySQL database inquiry) task, after Query Result is back to client, can have two ways to be presented to
User, one kind are that data structure is uploaded into Dropbox, and Dropbox address then notified into user by mail, another way be by
Data query result is returned directly to be presented to user in access interface subsystem 105.
Fig. 2, which is shown, can apply Data Mart management system of the use of the embodiment of the present invention based on Hadoop clusters to visit
Ask the method for Data Mart or the exemplary system architecture 200 of the Data Mart administrative system apparatus based on Hadoop clusters.
As shown in Fig. 2 system architecture 200 can include terminal device 201,202,203, network 204 and server 205.
Network 204 between terminal device 201,202,203 and server 205 provide communication link medium.Network 204 can be with
Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be interacted with using terminal equipment 201,202,203 by network 204 with server 205, to receive or send out
Send message etc..Various telecommunication customer end applications, such as the application of shopping class, net can be installed on terminal device 201,202,203
(merely illustrative) such as the application of page browsing device, searching class application, JICQ, mailbox client, social platform softwares.
Terminal device 201,202,203 can have a display screen and a various electronic equipments that supported web page browses, bag
Include but be not limited to smart mobile phone, tablet personal computer, pocket computer on knee and desktop computer etc..
Server 205 can be to provide the server of various services, such as utilize terminal device 201,202,203 to user
The shopping class website browsed provides the back-stage management server (merely illustrative) supported.Back-stage management server can be to receiving
To the data such as information query request analyze etc. processing, and by result (such as target push information, product letter
Breath -- merely illustrative) feed back to terminal device.
It should be noted that Data Mart management system of the use based on Hadoop clusters that the embodiment of the present invention is provided
The method for accessing Data Mart is typically performed by server 205, correspondingly, the Data Mart management system based on Hadoop clusters
Device be generally positioned in server 205.
It should be understood that the number of the terminal device, network and server in Fig. 2 is only schematical.According to realizing need
Will, can have any number of terminal device, network and server.
Fig. 3 show the structural representation of the computer system 300 suitable for being used for the terminal device for realizing the embodiment of the present invention
Figure.Terminal device shown in Fig. 3 is only an example, the function and use range of the embodiment of the present invention should not be brought any
Limitation.
As shown in figure 3, computer system 300 includes CPU (CPU) 301, it can be read-only according to being stored in
Program in memory (ROM) 302 or be loaded into program in random access storage device (RAM) 303 from storage part 308 and
Perform various appropriate actions and processing.In RAM 303, also it is stored with system 300 and operates required various programs and data.
CPU 301, ROM 302 and RAM 303 are connected with each other by bus 304.Input/output (I/O) interface 305 is also connected to always
Line 304.
I/O interfaces 305 are connected to lower component：Importation 306 including keyboard, mouse etc.；Penetrated including such as negative electrode
The output par, c 307 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage part 308 including hard disk etc.；
And the communications portion 309 of the NIC including LAN card, modem etc..Communications portion 309 via such as because
The network of spy's net performs communication process.Driver 310 is also according to needing to be connected to I/O interfaces 305.Detachable media 311, such as
Disk, CD, magneto-optic disk, semiconductor memory etc., it is arranged on as needed on driver 310, in order to read from it
Computer program be mounted into as needed storage part 308.
Especially, according to embodiment disclosed by the invention, it is soft that the process that block diagram above describes may be implemented as computer
Part program.For example, embodiment disclosed by the invention includes a kind of computer program product, it includes being carried on computer-readable Jie
Computer program in matter, the computer program include the program code for being used for the method shown in execution block diagram.In such reality
To apply in example, the computer program can be downloaded and installed by communications portion 309 from network, and/or from detachable media
311 are mounted.When the computer program is performed by CPU (CPU) 301, perform what is limited in the system of the present invention
Above-mentioned function.
It should be noted that the computer-readable medium shown in the present invention includes computer-readable signal media or computer
Readable storage medium storing program for executing, or the two any combination.Computer-readable recording medium include but is not limited to electricity, magnetic, light,
Electromagnetism, infrared ray, the system of semiconductor, device or device, or any combination of the above.Computer-readable recording medium
It is specifically including but not limited to：Electrical connection, portable computer diskette with one or more wires, hard disk, random access are deposited
Reservoir (RAM), read-only storage (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, Portable, compact
Disk read-only storage (CD-ROM), light storage device, any combination of magnetic memory device or the above.In the present invention
In, computer-readable recording medium includes any include or the tangible medium of storage program, the program can be commanded and perform system
The either device use or in connection of system, device；Computer-readable signal media is included in a base band or conduct
The data-signal that a carrier wave part is propagated, wherein carrying computer-readable program code, the data-signal of this propagation can
To take various forms, the including but not limited to any combination of electromagnetic signal, optical signal or above-mentioned signal.Computer-readable letter
Number medium can also be any computer-readable medium beyond computer-readable recording medium, and the computer-readable medium can be with
Send, propagate and either transmit for by the use of instruction execution system, device or device or program in connection.Meter
The program code included on calculation machine computer-readable recording medium can be transmitted with any appropriate medium, be included but is not limited to：Wirelessly, electric wire,
Optical cable, RF (radiofrequency signal) etc., or any combination of above-mentioned medium.
Block diagram in accompanying drawing, it is illustrated that according to the system of various embodiments of the invention, method and computer program product
Architectural framework, function and operation in the cards, each square frame in block diagram can represent a module, program segment or code
A part, the part of above-mentioned module, program segment or code include it is one or more be used to realizing as defined in logic function
Executable instruction.It should be noted that at some as in the realization replaced, the function of being marked in square frame can also be with different from attached
The order marked in figure occurs.For example, two square frames succeedingly represented can essentially be performed in parallel, sometimes can also
Perform in the opposite order, its execution sequence is depending on involved function.It is also noted that each square frame in block diagram with
And combinations thereof, function or the special hardware based system of operation it can be realized as defined in execution, or can use special
Realized with the combination of hardware and computer instruction.
Being described in system or unit involved in the embodiment of the present invention can be realized by way of software, can also
Realized by way of hardware.Described system or unit can also be set within a processor, for example, can be described as：
A kind of processor, which includes cluster management subsystem, to be included：Data administration subsystem, rights management subsystem, access interface subsystem
System.Wherein, the title of these systems does not form the restriction to the system in itself under certain conditions, for example, access interface is sub
System be also described as " be used to receive access request, from the rights management subsystem obtain corresponding to authority, and root
The system for determining the response to the access request according to the request and the authority ".
On the other hand, the embodiment of the present invention additionally provides a kind of computer-readable medium, and the computer-readable medium can be with
It is included in the equipment described in above-described embodiment；Can also be individualism, and without be incorporated the equipment in.Above-mentioned meter
Calculation machine computer-readable recording medium carries one or more program, when said one or multiple programs are performed by the equipment,
So that the equipment includes：
Cluster management subsystem；
Data administration subsystem；
Rights management subsystem；
Access interface subsystem；
Wherein,
The cluster management subsystem is used for the configuration information for obtaining the Data Mart, and the configuration information is same
Step is into the rights management subsystem；
The cluster management subsystem is additionally operable to obtain the metadata information of the Data Mart, and by the metadata
The data message is synchronized to the authority by synchronizing information to the data administration subsystem, the data administration subsystem again
Manage in subsystem；
The data administration subsystem is used for the modification information of the metadata of the synchronous Data Mart, the data management
The modification information of the metadata is synchronized in the rights management subsystem by subsystem again；
The access interface subsystem is used to receive access request, and corresponding weigh is obtained from the rights management subsystem
Limit, and according to the response of the request and authority determination to the access request.
Technical scheme according to embodiments of the present invention because using introduce cluster management subsystem, data administration subsystem,
Rights management subsystem and interface access sub-system, to Data Mart carry out data management, data calculate and metadata management in
The automatic management technological means of one, it is short of so overcoming and data nundinal being managed, user accesses data fairground is cumbersome
Time-consuming technical problem, and then reduce O＆M cost, allow users to rapidly and efficiently from Data Mart inquiry needed for
The technique effect of data.
Above-mentioned embodiment, does not form limiting the scope of the invention.Those skilled in the art should be bright
It is white, depending on design requirement and other factors, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (12)
1. the Data Mart management system based on Hadoop clusters, it is characterised in that including：
Cluster management subsystem；
Data administration subsystem；
Rights management subsystem；
Access interface subsystem；
Wherein,
The cluster management subsystem is used for the configuration information for obtaining the Data Mart, and the configuration information is synchronized to
The rights management subsystem；
The cluster management subsystem is additionally operable to obtain the metadata information of the Data Mart, and by the metadata information
The data administration subsystem is synchronized to, the metadata information is synchronized to the authority pipe by the data administration subsystem again
Manage subsystem；
The data administration subsystem is used for the modification information for obtaining the metadata of the Data Mart, the data management subsystem
The modification information of the metadata is synchronized to the rights management subsystem by system again；
The access interface subsystem is used to receive access request, from authority corresponding to rights management subsystem acquisition, and
And the response to the access request is determined according to the request and the authority.
2. system according to claim 1, it is characterised in that the Data Mart management system also includes：
Messenger service subsystem, for the modification information of the metadata of the Data Mart to be synchronized into the data management subsystem
System；
Wherein described messenger service subsystem includes：
Memory cell, the information of the metadata for preserving the Data Mart；
Log unit, for preserving the information that data change in the memory cell；
Subscriber units, for obtaining and preserving the information in the log unit in real time；
TU task unit, for the information preserved in the subscriber units to be converted to the modification information of metadata, and by the information
It is synchronized to the data administration subsystem.
3. system according to claim 1, it is characterised in that the cluster management subsystem passes through HTTP
The interface of type obtains the configuration information of the Data Mart.
4. system according to claim 2, it is characterised in that the subscriber units are obtained by configuring real-time acquisition tasks
Information in the log unit, and preserve the information and be used for message subscribing.
5. system according to claim 2, it is characterised in that the TU task unit is by establishing stream process task by described in
The information preserved in subscriber units is converted to the modification information of metadata, and by the synchronizing information to the data management subsystem
System.
6. the method for Data Mart is accessed using the Data Mart management system based on Hadoop clusters, it is characterised in that including：
Cluster management subsystem obtains the configuration information of the Data Mart, and the configuration information is synchronized into rights management
Subsystem；
The cluster management subsystem obtains the metadata information of the Data Mart, and the metadata information is synchronized to
The metadata information is synchronized to the rights management subsystem by data administration subsystem, the data administration subsystem again；
The data administration subsystem obtains the modification information of the metadata of the Data Mart, and the data administration subsystem is again
The modification information of the metadata is synchronized to the rights management subsystem；
Access interface subsystem receives access request, from authority corresponding to rights management subsystem acquisition, and according to institute
State the response of request and authority determination to the access request.
7. according to the method for claim 6, it is characterised in that methods described also includes：The Data Mart management system
In messenger service subsystem the modification information of the metadata of the Data Mart is synchronized to the data administration subsystem；
Wherein, the memory cell in the messenger service subsystem preserves the information of the metadata of the Data Mart；
Log unit in the messenger service subsystem preserves the information that data change in the memory cell；
Subscriber units in the messenger service subsystem obtain and preserve the information in the log unit in real time；
TU task unit in the messenger service subsystem is converted to the information preserved in the subscriber units the change of metadata
More information, and by the synchronizing information to the data administration subsystem.
8. according to the method for claim 6, it is characterised in that the cluster management subsystem passes through HTTP
The interface of type obtains the configuration information of the Data Mart.
9. according to the method for claim 7, it is characterised in that the subscriber units are obtained by configuring real-time acquisition tasks
Information in the log unit, and preserve the information and be used for message subscribing.
10. according to the method for claim 7, it is characterised in that the TU task unit is by establishing stream process task by institute
State the modification information that the information preserved in subscriber units is converted to metadata, and by the synchronizing information to the data management subsystem
System.
11. a kind of Data Mart management system of use based on Hadoop clusters accesses the electronic equipment of Data Mart, its feature
It is, including：
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are by one or more of computing devices so that one or more of processors are real
The now method as described in any in claim 6-10.
12. a kind of computer-readable medium, is stored thereon with computer program, it is characterised in that described program is held by processor
The method as described in any in claim 6-10 is realized during row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710854312.2A CN107729394A (en) | 2017-09-20 | 2017-09-20 | Data Mart management system and its application method based on Hadoop clusters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710854312.2A CN107729394A (en) | 2017-09-20 | 2017-09-20 | Data Mart management system and its application method based on Hadoop clusters |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107729394A true CN107729394A (en) | 2018-02-23 |
Family
ID=61207641
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710854312.2A Pending CN107729394A (en) | 2017-09-20 | 2017-09-20 | Data Mart management system and its application method based on Hadoop clusters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107729394A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241358A (en) * | 2018-08-14 | 2019-01-18 | 中国平安财产保险股份有限公司 | Metadata management method, device, computer equipment and storage medium |
| CN109376161A (en) * | 2018-08-22 | 2019-02-22 | 中国平安人寿保险股份有限公司 | Label data update method, device, medium and electronic equipment based on big data |
| CN109857747A (en) * | 2018-12-18 | 2019-06-07 | 百度在线网络技术（北京）有限公司 | Data synchronization updating method, system and computer equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101566981A (en) * | 2008-04-24 | 2009-10-28 | 长沙创智天马财务软件有限公司 | Method for establishing dynamic virtual data base in analyzing and processing system |
| US20140006244A1 (en) * | 2011-12-19 | 2014-01-02 | Ften Inc. | Method and System for Aggregating and Managing Data from Disparate Sources in Consolidated Storage |
| CN103793204A (en) * | 2012-10-29 | 2014-05-14 | 顺软科技发展（大连）有限公司 | Data analysis system (SRC) based on cloud computing |
| CN106682096A (en) * | 2016-12-01 | 2017-05-17 | 北京奇虎科技有限公司 | Method and device for log data management |
-
2017
- 2017-09-20 CN CN201710854312.2A patent/CN107729394A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101566981A (en) * | 2008-04-24 | 2009-10-28 | 长沙创智天马财务软件有限公司 | Method for establishing dynamic virtual data base in analyzing and processing system |
| US20140006244A1 (en) * | 2011-12-19 | 2014-01-02 | Ften Inc. | Method and System for Aggregating and Managing Data from Disparate Sources in Consolidated Storage |
| CN103793204A (en) * | 2012-10-29 | 2014-05-14 | 顺软科技发展（大连）有限公司 | Data analysis system (SRC) based on cloud computing |
| CN106682096A (en) * | 2016-12-01 | 2017-05-17 | 北京奇虎科技有限公司 | Method and device for log data management |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241358A (en) * | 2018-08-14 | 2019-01-18 | 中国平安财产保险股份有限公司 | Metadata management method, device, computer equipment and storage medium |
| CN109376161A (en) * | 2018-08-22 | 2019-02-22 | 中国平安人寿保险股份有限公司 | Label data update method, device, medium and electronic equipment based on big data |
| CN109376161B (en) * | 2018-08-22 | 2023-07-18 | 中国平安人寿保险股份有限公司 | Tag data updating method and device based on big data, medium and electronic equipment |
| CN109857747A (en) * | 2018-12-18 | 2019-06-07 | 百度在线网络技术（北京）有限公司 | Data synchronization updating method, system and computer equipment |
| CN109857747B (en) * | 2018-12-18 | 2021-07-13 | 百度在线网络技术（北京）有限公司 | Data synchronous updating method, system and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11681651B1 (en) | Lineage data for data records | |
| US20170139952A1 (en) | System and method transforming source data into output data in big data environments | |
| CN109614402B (en) | Multidimensional data query method and device | |
| CN107451109A (en) | Report form generation method and system | |
| CN111666490A (en) | Information pushing method, device, equipment and storage medium based on kafka | |
| CN110472207A (en) | List generation method and device | |
| CN107609890A (en) | A kind of method and apparatus of order tracking | |
| CN109683998A (en) | Internationalize implementation method, device and system | |
| CN110019087A (en) | Data processing method and its system | |
| CN107491382B (en) | Log output method and device | |
| US10956438B2 (en) | Catalog with location of variables for data | |
| CN109002440A (en) | Method, apparatus and system for big data multidimensional analysis | |
| US20170140160A1 (en) | System and method for creating, tracking, and maintaining big data use cases | |
| WO2021023149A1 (en) | Method and apparatus for dynamically returning message | |
| CN110866040A (en) | User portrait generation method, device and system | |
| CN107729394A (en) | Data Mart management system and its application method based on Hadoop clusters | |
| CN113467775A (en) | Method and device for generating page | |
| CN108932640B (en) | Method and device for processing orders | |
| CN112818026A (en) | Data integration method and device | |
| CN109213824A (en) | Data grabber system, method and apparatus | |
| CN113190558A (en) | Data processing method and system | |
| CN107908662A (en) | The implementation method and realization device of search system | |
| CN110399397A (en) | A kind of data query method and system | |
| CN110347654A (en) | A kind of method and apparatus of online cluster features | |
| CN114357280A (en) | Information pushing method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180223 |