CN104750870A - Log storage system based on HBase and implementation method - Google Patents
Log storage system based on HBase and implementation method Download PDFInfo
- Publication number
- CN104750870A CN104750870A CN201510187185.6A CN201510187185A CN104750870A CN 104750870 A CN104750870 A CN 104750870A CN 201510187185 A CN201510187185 A CN 201510187185A CN 104750870 A CN104750870 A CN 104750870A
- Authority
- CN
- China
- Prior art keywords
- user
- log
- daily record
- service
- hbase
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a log storage system based on HBase and an implementation method, and belongs to the technical field of large data storage. The system comprises a user management module, a log management module, a remote service call module and a system inquiry module, the user management module is used for registering a user, checking information, allocating a storage space by an administrator and managing basic situations of the user, the log management module is used for applying for a system account, managing the system account, customizing and managing log types and inquiring and downloading logs, the remote service call module is used for calling Web Services issued by a server for a user system, the Web Services include an authentication service and a remote log storage service, and the system inquiry module is used for inquiring and displaying residual space amount, contacting the administrator through mails and providing chart display in various forms. A distributed Hadoop HBase is used as a log storage container, mass data storage requirements can be met, and the advantage of inexpensive and stable storage of a Hadoop cluster is sufficiently used.
Description
Technical field
The present invention relates to a kind of is PostgreSQL database based on Hadoop distributed storage based on HBase(HBase.Belong to the sub-project of Hadoop with HDFS, Mapreduce) daily record storage system and implementation method, belong to large technical field of data storage.
Background technology
Daily record is a kind of resource to be excavated of continuous accumulation, and As time goes on its quantity is explosive growth.Hydraulic department has multiple infosystem, and each system is all the time at generation log recording.How these log recordings are carried out the management of concentrating, thus to carry out further data mining be the problem demanding prompt solution that hydraulic department faces.
In order to solve massive logs problem of management, software developer has started and has constantly probed into.Some have been had to study in log management field at present: Hu Wei achieved in 2013 the centralized stores aimed at day in magnetic disk media, the file layout of daily record is file and database, this storage mode dumb in log query and analysis (Hu Wei. the research & design [D] of enterprise's daily record centralized management system. Beijing: Beijing University of Post & Telecommunication, 2013.); Wang Zhaoyong devised a key value database that can meet large data and store in 2011, daily record is stored into key value database by outside importing form, daily record real-time storage can not be met, simultaneously the performance of database also need to be considered (Wang Zhaoyong. towards the research [D] of extensive batch daily record data storage means. Xi'an: University of Electronic Science and Technology, 2011); Song in 2012 likes that green grass or young crops applies the Web type daily record storage system of Hadoop, the collection of daily record adopts CDN content distributing network to collect user journal nearby, the log collection of this mode is loaded down with trivial details, need more node overheads (Song Aiqing. based on the design and implimentation [D] of the Log Analysis System of Hadoop. Beijing: The Chinese Geology Univ. (Beijing), 2012).System designed by above-mentioned document adopts distinct methods to solve massive logs storage problem, but the demand in these systems dirigibility, real-time and high efficiency etc. not considering to store in daily record.
Summary of the invention
For problems of the prior art, the present invention proposes a kind of daily record storage system based on HBase and implementation method, is stored by distributed information log and introduces Water Resources Domain, to improve Water Resources Domain the centralized stores of daily record and data mining ability.
The present invention adopts following technical scheme for solving its technical matters:
Based on a daily record storage system of HBase, comprise user management module, manage for user's registration, registration examination & verification, keeper's memory allocated space and the basic condition to user; Log management module, for the management to system account, Log Types and daily record; Remote service calling module, the storage of daily record is carried out in the Web service of calling issue for the issue of Web service and custom system, comprises the long-range real-time storage service of authentication service and daily record; System queries module, for inquiry, displaying and the station interior letter contact management person of remaining space amount; Wherein user management module and distributed data base are bi-directionally connected, distributed data base and log management module are bi-directionally connected, log management module is connected with remote service calling module, and remote service calling module is by distributed data base and system queries model calling;
Described user management module, comprises user and audits unit, storage space allocation unit and subscriber information management unit;
Described log management module, comprises system account application unit, Log Types administrative unit and log query unit;
Described remote service calling module, comprises system configuration unit, service release unit and service call unit;
Described system queries module, comprises letter administrative unit in remaining space query unit, contact management person's unit and station.
Based on an implementation method for the daily record storage system of HBase, comprise following steps:
S-1: a concrete user of native system, hereinafter referred to as user, registers native system by registration interface; Keeper audits registered user, stores number to satisfactory user by power memory allocated space and maximum system;
S-2: the user's application system account succeeded in registration, system account is produced according to current hour of log-on, produce function according to random cipher and produce Unique System password, same registered user can apply for multiple system account simultaneously, system default maximum system application number is 10, and user can contact management person change as required;
S-3: the implication of each attribute section of User Defined and the indications of attribute section, system default ' # ', as the separator between attribute section, forms a new Log Types;
S-4: system starts backward Web server Web Services Publishing and calls for custom system;
S-5: custom system request called Web service, whether successfully the daily record that custom system produces is write in the log sheet of HBase in real time, return daily record simultaneously and store information.
Described step S-2 specifically comprises following implementation step:
Step S-2-1: judge whether system application number has reached the system account application upper limit, if reached the system account application upper limit, then prompting has been applied for unsuccessfully; Otherwise continue step S-2-2;
Step S-2-2: user applies for creating new system account, system converts present system time to time of 1970-00-00 00:00:00 to a millisecond number generation system account; System generates a string random train be made up of upper and lower case letter and numeral as system password according to self-defining key derivation function;
Step S-2-3: one system account is as the major key key of a custom system, and the information such as system password, user's major key are as value, and so information is stored in the system account table of HBase with the form of key-value key-value pair, system account application success; Otherwise, return step S-2-2;
Described step S-3 specifically comprises following implementation step:
Step S-3-1: the implication determining each attribute section of Log Types, and the indications of self-defined each attribute section and Log Types title;
Step S-3-2: attribute section is spliced into character string by the order of definition with ' # ', forms the content of Log Types;
Step S-3-3: using present system time to the millisecond number of 1970-00-00 00:00:00 as major key key, using user's major key and Log Types information as value;
Step S-3-4: by key-value pair key-value stored in the Log Types table of HBase, if store successfully, return success; Otherwise, store unsuccessfully, return step S-3-1.
Described step S-5 specifically comprises following implementation step:
Step S-5-1: system end adopts CXF mode to be configured, as the server end of Web service, for Web Services Publishing; Equally, use CXF mode custom system to be carried out to the client configuration of Web service, make custom system become the client of Web service;
Step S-5-2: after running native system, Web service is published by network; Run user system; When performing daily record and storing, perform step S-5-3;
Step S-5-3: call authentication service: custom system provides system account and system password, both are called the authentication service checking of " IsCorrect " to name as Parameter transfer; If authentication success, performs step S-5-4; Otherwise authentication failure, returns daily record and stores failure information;
Step S-5-4: call the service of daily record real-time storage, the new daily record string produced calls the daily record real-time storage service that name is called " AddLogs ";
Step S-5-5: utilize the function of time of Java to be converted to the major key key of the millisecond number from the current time in system to 1970-00-00 00:00:00 as a log information time, parameter in step S-5-4 is as value, stored in the log sheet of HBase, log information stores successfully; Otherwise daily record stores unsuccessfully, goes to step S-5-2.
Beneficial effect of the present invention is as follows:
Adopt the HBase database of distributed Hadoop as daily record storage container, can satisfying magnanimity data storage demand and make full use of that Hadoop cluster is cheap, metastable storage characteristics; The remote journal storage mode of sing on web service, will improve the inconvenience that brings to Web log mining of local text storage mode, is conducive to aiming at day the data mining under concentrated condition, reduces the space expense that custom system stores daily record simultaneously.Log Types is self-defined to be expanded Log Types, and the personalization of outstanding daily record stores, for the log analysis of follow-up personalization and data mining lay the first stone.
Accompanying drawing explanation
Fig. 1 is the daily record storage system functionality module map based on HBase of the embodiment of the present invention.
Fig. 2 is the Web service call flow chart of the embodiment of the present invention.
Fig. 3 is the daily record storage system process flow diagram based on HBase of the embodiment of the present invention.
Fig. 4 is the Hadoop aggregated pattern synoptic chart of the embodiment of the present invention.
Embodiment
Below in conjunction with accompanying drawing, illustrate the present invention further, these embodiments should be understood only be not used in for illustration of the present invention and limit the scope of the invention, after having read the present invention, the amendment of those skilled in the art to the various equivalent form of value of the present invention has all fallen within the application's claims limited range.
For the problem in daily record centralized management, the present invention propose and realize one based on B/S framework, can the daily record stores service platform of real-time storage massive logs.Daily record is stored into HBase distributed data base by called Web service by custom system; User, by browser access platform, enters log management interface and carries out log management.
Daily record storage system based on HBase is made up of user management module, log management module, remote service calling module and system queries module, as shown in Figure 1.User management module, manages for user's registration, registration examination & verification, keeper's memory allocated space and the basic condition to user; Log management module, for the management to system account, Log Types and daily record; Remote service calling module, calls the Web service of issue, comprises the long-range real-time storage service of authentication service and daily record for the issue of Web service and custom system; System queries module, for inquiry and displaying and the station interior letter contact management person of remaining space amount, provides multiple diagrammatic representation form.
User management module, mainly comprises user's examination & verification, by power memory allocated space, user basic information administrative unit.Wherein user audits unit and audits user's registration information for keeper, filters out effective registered user, avoids invasive data to cause the waste of storage space; Storage space allocation unit is used for keeper and carries out space size distribution to auditing the user passed through by power, and allocation of space has feature flexibly; Subscriber information management unit comprises user profile inquiry, user profile batch is downloaded, amendment maximum system stores number and storage space, details are shown and delete defective user operation; Under native system setting initial situation, the storage space under each user's original state is 0, and the system account number that each user applies for is 10, if exceed maximum system account number, user cannot the application of completion system account.Each user can only carry out daily record storage when storage space meets, and often stored in a daily record, whether system all calculates storage space and satisfy the demands.
Table 1 is several example information of subscriber's meter in HBase.In table 1, front two row represent two different editions of same data respectively.On the basis of the 1st article of data, the 2nd article of data are revised password, it is worth noting the disappearance allowing item in HBase table, " Realname " of such as the first row row, but can not Hadoop space outside occupying volume.Compared with traditional relational, HBase is utilized to store the waste greatly can avoiding unnecessary storage space in discrete data.
Table 1.HBase subscriber's meter example
The calculating of storage space, comprises two aspects: to the calculating to space when the calculating in space and keeper's allocation space when user journal stores.Because system account is very little with the storage space shared by system password and its growth rate is almost 0 compared with daily record, therefore when calculating user storage space, we only calculate the Hadoop space shared by user journal, and it is defined as:
Wherein, I represents user I,
represent the storage size shared by i-th system of I.
represent shared all storage size of user I.
Keeper's allocation space calculates and is defined as:
M represents total system user number, and n represents the system number of each user, and p represents the summation of system user and administrator; L represents Log Types sum; Q represents letter sum in station;
represent total daily record memory space of a jth system of i-th user;
represent the storage space that subscriber's meter takies;
represent the storage space that system table takies;
the storage space that Log Types table takies;
the storage space that in standing, letter takies.
Log management module, comprises system account application unit, Log Types administrative unit and log query unit; System account application unit is used for application system account and management system account, and system account and system password are two string character strings, for identify a custom system and when custom system request access data storehouse as authentication information.The corresponding Unique System password of each system account, improves the security of system access; Log Types administrative unit is used for expanding traditional Log Types and Log Types management, supports the self-defined of Log Types, improves the diversity that daily record stores; Log query unit is used for the multi-condition inquiry of daily record and daily record batch is downloaded, in order to avoid deleting the mistake of daily record, system do not support daily record by bar deletion action.Wherein Log Types administrative unit is mainly to the management of self-defined Log Types, and table 2 lists a customization type in system example.Wherein, typonym and attribute definition are all completed by User Defined, and type instance is that custom system is to this customization type example.
The self-defined example of table 2 Log Types
| Typonym | Operation Log |
| Attribute defines | Login name # password #IP address # operates the system account of custom system described in # |
| Type instance | User 1#11111#10.192.80.20# error in operation #14000294561 |
Log query unit uses the multi-condition inquiry based on HBase, adopts message-filter mechanism to compare full table scan and can significantly improve search efficiency.The key-value(key of Hadoop-be worth to) major key sorts according to dictionary in mechanism and HBase table design, make the search efficiency passed through in the design process of HBase table the design raising table of Rowkey.
Remote service calling module is the nucleus module of daily record storage system, comprises system configuration unit, service release unit and service call unit.System configuration unit comprises the configuration of two aspects, and first configure the service end that native system becomes Web service, secondly configure user system becomes the client of Web service; Service release unit is used for server end by Web Publishing Web service, comprises the real-time storage service of authentication service and daily record; Service call unit is used for the Web service that custom system far call is successfully issued.System account, system password are given the authentication service of " IsCorrect " by name by service call as Parameter transfer; Daily record string is given the Web service of " AddLogs " by name as a Parameter transfer, carry out final daily record remote storage by " AddLogs " service.Concrete flowchart is shown in Fig. 2.Same service can call by multiple custom system; Kind and the quantity of service can be modified by system engineer and add, and once after amendment, custom system can call up-to-date service, and this characteristic of Web service improves portability and the dirigibility of system greatly.
System queries module, comprises letter administrative unit in remaining space query unit, contact management person's unit and station.Remaining space query unit, the space inquiring about Hadoop cluster for user's search space surplus and use amount, keeper uses and surplus; Contact management person's unit is used for searching and managing person's information and contact management person, letter and mailbox two kinds of contact methods in contact management person's support station; Letter information in the station that letter administrative unit is used for account executive in standing, comprises collection of letters part and has sent deletion and the operation such as interval query by date of mail.
Based on the log storing method of HBase, implement as follows:
S-1: user registers native system such as by registration interface: user name is respectively two users of user 1 and user 2.Keeper audits the information of user 1 and user 2, if user 1 has obtained the license using this system before registration, and user 2 belongs to illegal registered user, so keeper can audit by operating user 1 and be user 1 memory allocated space, if license user 1 allows use maximum memory space be 100G, so keeper is the space that user 1 distributes 100G size, and keeper can carry out deletion action to the information of user 2 simultaneously;
S-2: after succeeding in registration, user 1 can application system account, system account is produced according to current hour of log-on, produce function according to random cipher and produce Unique System password, same registered user can apply for multiple system account simultaneously, the system default system account application upper limit is 10, and user can contact management person change as required;
Step S-2-1: judge whether the system application number of user 1 has reached the system account application upper limit 10.If reached the system account application upper limit, then prompting has been applied for unsuccessfully; Otherwise continue step S-2-2;
Step S-2-2: user 1 applies for creating new system account, system converts present system time to time of 1970-00-00 00:00:00 to millisecond number generation system account such as: 1400001294561; System generates a string random train be made up of upper and lower case letter and numeral as system password according to self-defining key derivation function, is 12, such as: a7TwM2fkYnQ0 in system testing stage definitions system password length;
Step S-2-3:1400001294561 is as the major key of a custom system of user 1, the information such as the major key 1339192226632 of system password: a7TwM2fkYnQ0, user 1 are as the value in key-value pair, information is stored in the system account Table A ppidTable of HBase database with the form of key-value key-value pair, so far, system account application success; Otherwise, return step S-2-2;
S-3: the self-defined Log Types of user 1.Define the implication of each attribute section and the indications of attribute section, system default ' # ', as the separator between attribute section, forms a new Log Types;
Step S-3-1: the implication determining each attribute section of Log Types, and the indications of self-defined each attribute section and Log Types title, as example in table 2, Log Types name: log in daily record, daily record is made up of 5 attribute sections, is login name, login password, login ip address, operation, system account major key respectively;
Step S-3-2: attribute section is spliced into character string by the order of definition with ' # ', forms the content of Log Types;
Step S-3-3: using present system time to the millisecond number of 1970-00-00 00:00:00 as major key key, key is in this example: 1400001327490, user's major key is 1339192226632: using user's major key and Log Types information as value value;
Step S-3-4: by key-value pair key-value stored in the Log Types table LogTypeTable of HBase, if store successfully, return success; Otherwise, store unsuccessfully, return step S-3-1.
S-4: system starts backward Web server Web Services Publishing and calls for custom system;
S-5:Web service is issued successfully, 1400001294561 system request of user 1 called Web service, and the daily record that custom system produces is write in the log sheet of HBase in real time, returns the whether successful information of daily record storage simultaneously.
Step S-5-1: the daily record storage system based on HBase adopts CXF(to be a Services framework of increasing income) mode is configured, as the server end of Web service, for Web Services Publishing; Equally, use CXF mode 1400001294561 systems to be carried out to the client configuration of Web service, make custom system become the client of Web service, the Web service of issue can be called;
Step S-5-2: after running the daily record storage system based on HBase, authentication service " IsCorrect " and daily record stores service " AddLogs " are published by network; 1400001294561 systems of run user 1; When performing daily record and storing, perform step S-5-3;
Step S-5-3: call authentication service " IsCorrect ".User 1 provides system account: 1400001294561 and system password: a7TwM2fkYnQ0, both as Parameter transfer to authentication service checking " IsCorrect ".If authentication success, performs step S-5-4; Otherwise authentication failure, returns daily record and stores failure information;
Step S-5-4: call daily record stores service AddLogs.Using the daily record string that newly produces as a parameter, call the daily record real-time storage service of AddLogs by name, be stored to HBase.HBase is the sub-project of Hadoop, and it runs based on Hadoop distributed storage architecture, and therefore utilizing before HBase carries out data storage, first want Install and configure Hadoop, the concrete configuration that Hadoop is configured in native system test phase is as shown in table 3.
The IP address assignment of table 3.Hadoop cluster
Step S-5-5: the function of time utilizing Java
system.currentTimeMill ()is converted to the millisecond number from the current time in system to 1970-00-00 00:00:00 the time, at this, millisecond number is 14000132474504: as the major key key of a log information, parameter in step S-5-4 is as value, stored in the log sheet LogsTable table of HBase database, log information stores successfully.Otherwise daily record stores unsuccessfully, goes to step S-5-2.
Claims (5)
1. based on a daily record storage system of HBase, it is characterized in that, comprise user management module, manage for user's registration, registration examination & verification, keeper's memory allocated space and the basic condition to user; Log management module, for the management to system account, Log Types and daily record; Remote service calling module, the storage of daily record is carried out in the Web service of calling issue for the issue of Web service and custom system, comprises the long-range real-time storage service of authentication service and daily record; System queries module, for inquiry, displaying and the station interior letter contact management person of remaining space amount; Wherein user management module and distributed data base are bi-directionally connected, distributed data base and log management module are bi-directionally connected, log management module is connected with remote service calling module, and remote service calling module is by distributed data base and system queries model calling;
Described user management module, comprises user and audits unit, storage space allocation unit and subscriber information management unit;
Described log management module, comprises system account application unit, Log Types administrative unit and log query unit;
Described remote service calling module, comprises system configuration unit, service release unit and service call unit;
Described system queries module, comprises letter administrative unit in remaining space query unit, contact management person's unit and station.
2. the implementation method of a kind of daily record storage system based on HBase according to claim 1, is characterized in that, comprise following steps:
S-1: a concrete user of native system, hereinafter referred to as user, registers native system by registration interface; Keeper audits registered user, stores number to satisfactory user by power memory allocated space and maximum system;
S-2: the user's application system account succeeded in registration, system account is produced according to current hour of log-on, produce function according to random cipher and produce Unique System password, same registered user can apply for multiple system account simultaneously, system default maximum system application number is 10, and user can contact management person change as required;
S-3: the implication of each attribute section of User Defined and the indications of attribute section, system default ' # ', as the separator between attribute section, forms a new Log Types;
S-4: system starts backward Web server Web Services Publishing and calls for custom system;
S-5: custom system request called Web service, whether successfully the daily record that custom system produces is write in the log sheet of HBase in real time, return daily record simultaneously and store information.
3. according to claims 2 a kind of based on HBase the implementation method of daily record storage system, it is characterized in that: described step S-2 specifically comprises following implementation step:
Step S-2-1: judge whether system application number has reached the system account application upper limit, if reached the system account application upper limit, then prompting has been applied for unsuccessfully; Otherwise continue step S-2-2;
Step S-2-2: user applies for creating new system account, system converts present system time to time of 1970-00-00 00:00:00 to a millisecond number generation system account; System generates a string random train be made up of upper and lower case letter and numeral as system password according to self-defining key derivation function;
Step S-2-3: one system account is as the major key key of a custom system, and the information such as system password, user's major key are as value, and so information is stored in the system account table of HBase with the form of key-value key-value pair, system account application success; Otherwise, return step S-2-2.
4. according to claims 2 a kind of based on HBase the implementation method of daily record storage system, it is characterized in that: described step S-3 specifically comprises following implementation step:
Step S-3-1: the implication determining each attribute section of Log Types, and the indications of self-defined each attribute section and Log Types title;
Step S-3-2: attribute section is spliced into character string by the order of definition with ' # ', forms the content of Log Types;
Step S-3-3: using present system time to the millisecond number of 1970-00-00 00:00:00 as major key key, using user's major key and Log Types information as value;
Step S-3-4: by key-value pair key-value stored in the Log Types table of HBase, if store successfully, return success; Otherwise, store unsuccessfully, return step S-3-1.
5. according to claims 2 a kind of based on HBase the implementation method of daily record storage system, it is characterized in that: described step S-5 specifically comprises following implementation step:
Step S-5-1: system end adopts CXF mode to be configured, as the server end of Web service, for Web Services Publishing; Equally, use CXF mode custom system to be carried out to the client configuration of Web service, make custom system become the client of Web service;
Step S-5-2: after running native system, Web service is published by network; Run user system; When performing daily record and storing, perform step S-5-3;
Step S-5-3: call authentication service: custom system provides system account and system password, both are called the authentication service checking of " IsCorrect " to name as Parameter transfer; If authentication success, performs step S-5-4; Otherwise authentication failure, returns daily record and stores failure information;
Step S-5-4: call the service of daily record real-time storage, the new daily record string produced calls the daily record real-time storage service that name is called " AddLogs ";
Step S-5-5: utilize the function of time of Java to be converted to the major key key of the millisecond number from the current time in system to 1970-00-00 00:00:00 as a log information time, parameter in step S-5-4 is as value, stored in the log sheet of HBase, log information stores successfully; Otherwise daily record stores unsuccessfully, goes to step S-5-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510187185.6A CN104750870B (en) | 2015-04-20 | 2015-04-20 | A kind of log storage system and implementation method based on HBase |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510187185.6A CN104750870B (en) | 2015-04-20 | 2015-04-20 | A kind of log storage system and implementation method based on HBase |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104750870A true CN104750870A (en) | 2015-07-01 |
| CN104750870B CN104750870B (en) | 2018-11-27 |
Family
ID=53590554
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510187185.6A Active CN104750870B (en) | 2015-04-20 | 2015-04-20 | A kind of log storage system and implementation method based on HBase |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104750870B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956099A (en) * | 2016-04-28 | 2016-09-21 | 安徽四创电子股份有限公司 | HBase high table-based primary key design method |
| CN108156230A (en) * | 2017-12-19 | 2018-06-12 | 杭州有赞科技有限公司 | Real-time data synchronization method, system and frame |
| CN108804606A (en) * | 2018-05-29 | 2018-11-13 | 上海欣能信息科技发展有限公司 | A kind of electric power measures class Data Migration to the method and system of HBase |
| CN109446196A (en) * | 2018-09-26 | 2019-03-08 | 努比亚技术有限公司 | Major key generates processing and determines that method, major key generates and application server |
| CN109447468A (en) * | 2018-10-30 | 2019-03-08 | 福建南威软件有限公司 | A kind of Log Administration System for supporting multisystem, polymorphic type log unified management |
| CN109933484A (en) * | 2017-12-15 | 2019-06-25 | 北京京东尚科信息技术有限公司 | Big data cluster quasi real time container resource allocation monitoring analysis method |
| CN109978291A (en) * | 2017-12-27 | 2019-07-05 | 广东电网有限责任公司电力调度控制中心 | A kind of Multifunctional power network dispatching management information system |
| CN110516434A (en) * | 2019-08-12 | 2019-11-29 | 广州海颐信息安全技术有限公司 | Franchise account scanning system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080270511A1 (en) * | 2005-04-30 | 2008-10-30 | Huawei Technologies Co., Ltd. | Method and system for managing home network |
| CN101635651A (en) * | 2009-08-31 | 2010-01-27 | 杭州华三通信技术有限公司 | Method, system and device for managing network log data |
| CN103391185A (en) * | 2013-08-12 | 2013-11-13 | 北京泰乐德信息技术有限公司 | Cloud security storage and processing method and system for rail transit monitoring data |
| CN103701851A (en) * | 2013-11-27 | 2014-04-02 | 中国农业科学院农业信息研究所 | Agricultural technology promotion cloud log management publishing system and method |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104252536A (en) * | 2014-09-16 | 2014-12-31 | 福建新大陆软件工程有限公司 | Hbase-based internet log data inquiring method and device |
-
2015
- 2015-04-20 CN CN201510187185.6A patent/CN104750870B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080270511A1 (en) * | 2005-04-30 | 2008-10-30 | Huawei Technologies Co., Ltd. | Method and system for managing home network |
| CN101635651A (en) * | 2009-08-31 | 2010-01-27 | 杭州华三通信技术有限公司 | Method, system and device for managing network log data |
| CN103391185A (en) * | 2013-08-12 | 2013-11-13 | 北京泰乐德信息技术有限公司 | Cloud security storage and processing method and system for rail transit monitoring data |
| CN103701851A (en) * | 2013-11-27 | 2014-04-02 | 中国农业科学院农业信息研究所 | Agricultural technology promotion cloud log management publishing system and method |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104252536A (en) * | 2014-09-16 | 2014-12-31 | 福建新大陆软件工程有限公司 | Hbase-based internet log data inquiring method and device |
Non-Patent Citations (1)
| Title |
|---|
| 陈耀兵 等: "基于Hadoop架构的大数据量日志存储和检索优化", 《信息网络安全》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956099A (en) * | 2016-04-28 | 2016-09-21 | 安徽四创电子股份有限公司 | HBase high table-based primary key design method |
| CN109933484A (en) * | 2017-12-15 | 2019-06-25 | 北京京东尚科信息技术有限公司 | Big data cluster quasi real time container resource allocation monitoring analysis method |
| CN108156230A (en) * | 2017-12-19 | 2018-06-12 | 杭州有赞科技有限公司 | Real-time data synchronization method, system and frame |
| CN108156230B (en) * | 2017-12-19 | 2020-09-04 | 杭州有赞科技有限公司 | Real-time data synchronization method, system and framework |
| CN109978291A (en) * | 2017-12-27 | 2019-07-05 | 广东电网有限责任公司电力调度控制中心 | A kind of Multifunctional power network dispatching management information system |
| CN108804606A (en) * | 2018-05-29 | 2018-11-13 | 上海欣能信息科技发展有限公司 | A kind of electric power measures class Data Migration to the method and system of HBase |
| CN109446196A (en) * | 2018-09-26 | 2019-03-08 | 努比亚技术有限公司 | Major key generates processing and determines that method, major key generates and application server |
| CN109447468A (en) * | 2018-10-30 | 2019-03-08 | 福建南威软件有限公司 | A kind of Log Administration System for supporting multisystem, polymorphic type log unified management |
| CN110516434A (en) * | 2019-08-12 | 2019-11-29 | 广州海颐信息安全技术有限公司 | Franchise account scanning system |
| CN110516434B (en) * | 2019-08-12 | 2021-12-10 | 广州海颐信息安全技术有限公司 | Privileged account scanning system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104750870B (en) | 2018-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104750870A (en) | Log storage system based on HBase and implementation method | |
| CN105069703B (en) | A kind of electrical network mass data management method | |
| CN108259622B (en) | Cross-region sharing method for electronic license data | |
| US9934233B2 (en) | Searchable data archive | |
| CN103020281B (en) | A kind of data storage and retrieval method based on spatial data numerical index | |
| CN104424229B (en) | A kind of calculation method and system that various dimensions are split | |
| CN102054000B (en) | Data querying method, device and system | |
| CN106886371B (en) | Caching data processing method and device | |
| CN109189782A (en) | A kind of indexing means in block chain commodity transaction inquiry | |
| CN102164186A (en) | Method and system for realizing cloud search service | |
| CN110958109B (en) | Light dynamic data integrity auditing method based on hierarchical merck hash tree | |
| CN103942287A (en) | Mass data processing method, database server and application server | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN103533023B (en) | Cloud service application cluster based on cloud service feature synchronizes system and synchronous method | |
| CN101442558A (en) | Method and system for providing index service for P2SP network | |
| CN110245145A (en) | Structure synchronization method and apparatus of the relevant database to Hadoop database | |
| CN105912601A (en) | Partition storage method for distributed real-time memory database of energy management system | |
| CN106886592A (en) | A kind of city space-time data collecting system and method based on data source service | |
| CN104486777A (en) | Method and device for processing data | |
| CN114677188B (en) | Full-amount acquisition method and device suitable for paperless certificate data | |
| CN105824892A (en) | Method for synchronizing and processing data by data pool | |
| CN102932443A (en) | HDFS (hadoop distributed file system) cluster based distributed cloud storage system | |
| CN103778223A (en) | Pervasive word-reciting system based on cloud platform and construction method thereof | |
| CN110992182A (en) | Transaction method and system of block chain wallet | |
| CN101901211A (en) | Processing method of report form |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |