CN112650779B - Cloud auditing method supporting deep page skipping query based on ElasticSearch - Google Patents
Cloud auditing method supporting deep page skipping query based on ElasticSearch Download PDFInfo
- Publication number
- CN112650779B CN112650779B CN202110038177.0A CN202110038177A CN112650779B CN 112650779 B CN112650779 B CN 112650779B CN 202110038177 A CN202110038177 A CN 202110038177A CN 112650779 B CN112650779 B CN 112650779B
- Authority
- CN
- China
- Prior art keywords
- data
- page
- query
- current
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a cloud auditing method supporting deep page skipping inquiry based on elastic search, belonging to the field, and the method provides operation record real-time retrieval within 180 days for a user, and supports the user to call OpenAPI, create, modify and delete operation records of a cloud service product for deep page skipping inquiry; the retrieval time range is expanded, and page skipping query can be performed.
Description
Technical Field
The invention relates to the technical field of cloud computing, in particular to a cloud auditing method based on ElasticSearch supporting deep page skipping inquiry.
Background
Cloud computing is a brand-new leading information technology which combines an IT technology and the Internet to realize super computing and high storage capacity, and the impetus for promoting the rise of cloud computing is the development of technologies such as high-speed Internet, virtualization technology, cheaper and powerful chips, hard disks and data centers. Cloud computing can be regarded as a product of development and fusion of traditional computer and network technologies, such as distributed computing, parallel computing, utility computing, network storage, virtualization, load balancing and the like. The rise of cloud computing generates huge data, and the related data is huge in scale, so that the data cannot be captured, managed, processed and organized into information which can help enterprises to make business decisions more positive in reasonable time through the current mainstream software tools. According to the definition of an authoritative NIST, the existing cloud computing is mainly divided into three service modes, namely IAAS (infrastructure as service) infrastructure, which mainly provides infrastructure services for users, wherein the services comprise a computer, a server, a firewall, a storage device, a network device and the like; a platform paas (platform as a service), which mainly provides an application program development, test and deployment platform for users, that is, a complete system platform including application design, application development, application test, application deployment and application hosting is provided for users as a service; saas (software as a service) software, which is mainly used for providing software such as application programs for users.
The auditing workload of the cloud platform big data is huge, and the operation data of the user is not audited comprehensively enough, so that the current auditing system cannot meet the requirement of cloud platform user operation auditing at all.
Disclosure of Invention
In order to solve the technical problem, the invention provides a cloud auditing method for supporting deep page skipping query based on elastic search, which provides a real-time retrieval service for recording within 180 days of cloud resource operation for a user and provides deep page skipping query data support.
The technical scheme of the invention is as follows:
a cloud auditing method based on ElasticSearch supporting depth skip page query uses ElasticSearch to build a data warehouse and an index of operation record data and provides a depth skip page query function service.
Furthermore, a self-research algorithm is used for providing a deep skip page query function, and the method mainly comprises the following steps:
the method comprises the following steps: reading a default retrieval step length configured in a configuration file;
step two: reading the minimum data number of the opening depth page skipping query configured in the configuration file, and then, weighing the minimum data;
step three: judging whether to need to open the depth query or not according to the comparison of the data size and the minimum data of the current page and each page of query;
step four: if the deep query does not need to be started, directly calling api of the ElasticSearch to query and returning a query result, otherwise, entering a fifth step;
step five: calling the api of the elastic search to inquire the long-stripe data of the retrieval step, judging whether the total data volume is smaller than the retrieval step, directly returning to the null state, or else, entering the sixth step;
step six: obtaining the subscript of the last piece of data, using a cursor mode to query, and obtaining the query data volume by using the following algorithm:
(a) judging whether the current subscript is larger than or equal to (current page-1) × the data size of each page, and using the data size of each page as the data volume;
(b) dividing the searching step size by a factor of (current page-1) data size per page-current subscript) to be larger than 0, and using the searching step size as the data volume, otherwise, using the result of (current page-1) data size per page-current subscript) as the data volume;
step seven: and step six, recursion calling is carried out for query until the data volume is found to be the size of each page of data, and the result is returned after the recursion is finished.
The invention has the advantages that
The method meets the requirement of a user on the real-time deep skip page retrieval of the operation records within a period of time (180 days), and compared with the traditional method of only providing 7-day log record retrieval, the time range is expanded by nearly 26 times. The traditional cloud auditing system uses the ElasticSearch as an index, does not support page jump inquiry, and only can carry out rolling inquiry, so that the cloud auditing method overcomes the defect.
Drawings
FIG. 1 is a timing diagram of call retrieval according to the present invention;
fig. 2 is a flowchart of the operation of the retrieval function.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, it is obvious that the described embodiments are some, but not all embodiments of the present invention, and based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
The cloud auditing system based on the ElasticSearch supporting deep page-skipping query uses the ElasticSearch to create an index, an operation log index is created monthly, and the operation log index before 6 months is deleted. As is known, the ElasticSearch does not provide a deep skip page query function, and the operation auditing system provides a deep skip page retrieval method capable of optimizing query speed online. The working flow of the method is as follows:
(1) reading a default retrieval step length configured in a configuration file;
(2) reading the minimum data number of the opening depth page skipping query configured in the configuration file, and then, weighing the minimum data;
(3) judging whether to start deep query or not according to the comparison of the current page, the data size of each page and the minimum data;
(4) if the deep query does not need to be started, directly calling the api of the ElasticSearch to perform query and returning a query result, otherwise, entering the fifth step;
(5) calling the api of the elastic search to inquire long-strip data of the retrieval step, judging whether the total data volume is smaller than the retrieval step, directly returning to the null state, or else, entering the sixth step;
(6) obtaining the subscript of the last piece of data, using a cursor mode to query, and obtaining the query data volume by using the following algorithm:
(a) judging whether the current subscript is larger than or equal to (current page-1) × the data size of each page, and using the data size of each page as the data volume;
(b) dividing the searching step size by a factor of (current page-1) data size per page-current subscript) to be larger than 0, and using the searching step size as the data volume, otherwise, using the result of (current page-1) data size per page-current subscript) as the data volume;
(7) and recursively calling a sixth step line for query until the data size is found to be the size of each page of data, and returning a result after the recurrence is finished.
The flow chart is shown in figure 2.
The main pseudo code is as follows:
meanwhile, the system provides a function of debugging the retrieval step length on line, and the retrieval step length can be dynamically adjusted to adapt to the scale of the data size of the current cluster index.
The invention can provide indexing and display services for the operations of creating, modifying, deleting cloud resources and the like of the tenant.
The above description is only a preferred embodiment of the present invention, and is only used to illustrate the technical solutions of the present invention, and not to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (3)
1. A cloud auditing method based on ElasticSearch supporting deep page jump query is characterized in that,
providing an operation log data storage index service; deep page skipping retrieval of all operation records within 180 days is supported, and debugging and query efficiency is optimized on line;
creating an index, creating an operation log index according to months, and deleting the operation log index before 6 months;
the method comprises the following steps:
the method comprises the following steps: reading a default retrieval step length configured in a configuration file;
step two: reading the minimum data number of the opening depth page skipping query configured in the configuration file, and then, weighing the minimum data;
step three: judging whether to start deep query or not according to the comparison of the current page, the data size of each page and the minimum data;
step four: if the deep query does not need to be started, directly calling the api of the ElasticSearch to perform query and returning a query result, otherwise, entering the step five;
step five: calling the api of the elastic search to inquire the long-stripe data of the retrieval step, judging whether the total data volume is smaller than the retrieval step, directly returning to the null state, or else, entering the sixth step;
step six: obtaining the subscript of the last piece of data, and querying in a cursor mode;
step seven: the recursion calling step six is used for inquiring until the data volume is found to be the size of each page of data, and the result is returned after the recursion is finished;
the query data volume in the step six is obtained by using the following method:
judging whether the current subscript is larger than or equal to (current page-1) × the data size of each page, and using the data size of each page as the data volume;
the query data volume in the step six is obtained by using the following method:
dividing the search step size by (current page-1) × data size per page-current index) by more than 0, the search step size is used as the data amount, otherwise, the result ((current page-1) × data size per page-current index) is used as the data amount.
2. The method of claim 1,
and the working mode of data storage and index providing deep page skipping inquiry and operating link tracking analysis service is performed.
3. The method of claim 1,
and a function of online debugging the retrieval step length is provided, and the retrieval step length is dynamically adjusted to adapt to the scale of the data size of the current cluster index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110038177.0A CN112650779B (en) | 2021-01-12 | 2021-01-12 | Cloud auditing method supporting deep page skipping query based on ElasticSearch |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110038177.0A CN112650779B (en) | 2021-01-12 | 2021-01-12 | Cloud auditing method supporting deep page skipping query based on ElasticSearch |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112650779A CN112650779A (en) | 2021-04-13 |
| CN112650779B true CN112650779B (en) | 2022-09-06 |
Family
ID=75367973
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110038177.0A Active CN112650779B (en) | 2021-01-12 | 2021-01-12 | Cloud auditing method supporting deep page skipping query based on ElasticSearch |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650779B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117793652A (en) * | 2023-12-12 | 2024-03-29 | 沈阳林科信息技术有限公司 | New attention user guiding method based on 5GRCSchatbot |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100370459C (en) * | 2005-12-08 | 2008-02-20 | 华为技术有限公司 | Method and apparatus for reducing paging data retrieve time |
| CN103338260B (en) * | 2013-07-04 | 2016-05-25 | 武汉世纪金桥安全技术有限公司 | The distributed analysis system of URL daily record and analytical method in network audit |
| CN103886022B (en) * | 2014-02-24 | 2019-01-18 | 上海上讯信息技术股份有限公司 | A kind of query facility and its method carrying out paging query based on major key field |
| CN107783978A (en) * | 2016-08-24 | 2018-03-09 | 中安消物联传感（深圳）有限公司 | A kind of big data inquiry return method and device |
| CN107748766B (en) * | 2017-09-28 | 2021-08-24 | 南威软件股份有限公司 | Big data fast query method based on Presto and elastic search |
| CN109492148B (en) * | 2018-11-22 | 2020-07-24 | 北京明朝万达科技股份有限公司 | ElasticSearch paging query method and equipment based on Redis |
| CN109783513B (en) * | 2018-12-20 | 2021-03-16 | 北京大米科技有限公司 | Data processing method, device, server and computer readable storage medium |
| US10423662B1 (en) * | 2019-05-24 | 2019-09-24 | Hydrolix Inc. | Efficient and scalable time-series data storage and retrieval over a network |
| CN110928901B (en) * | 2019-10-28 | 2022-05-31 | 武大吉奥信息技术有限公司 | Map layer joint query method, device and storage device based on MapServer service protocol |
| CN111625551A (en) * | 2020-05-15 | 2020-09-04 | 贵州易鲸捷信息技术有限公司 | Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof |
| CN111680063B (en) * | 2020-05-25 | 2023-08-18 | 泰康保险集团股份有限公司 | Method and device for paging query data by elastic search |
| CN111708740A (en) * | 2020-06-16 | 2020-09-25 | 荆门汇易佳信息科技有限公司 | Mass search query log calculation analysis system based on cloud platform |
-
2021
- 2021-01-12 CN CN202110038177.0A patent/CN112650779B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112650779A (en) | 2021-04-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9052938B1 (en) | Correlation and associated display of virtual machine data and storage performance data | |
| CN107491515B (en) | Intelligent power distribution and utilization data conversion method based on big data platform | |
| JP2017512338A (en) | Implementation of semi-structured data as first class database elements | |
| US20210256023A1 (en) | Subquery predicate generation to reduce processing in a multi-table join | |
| CN103620601A (en) | Joining tables in a mapreduce procedure | |
| CN102779138B (en) | The hard disk access method of real time data | |
| KR20130049111A (en) | Forensic index method and apparatus by distributed processing | |
| CN111782640B (en) | Data processing method and system of cloud platform, electronic equipment and storage medium | |
| CN112015741A (en) | Method and device for storing massive data in different databases and tables | |
| US11537613B1 (en) | Merge small file consolidation | |
| WO2017092444A1 (en) | Log data mining method and system based on hadoop | |
| CN102314491A (en) | Method for identifying similar behavior mode users in multicore environment based on massive logs | |
| CN104834650A (en) | Method and system for generating effective query tasks | |
| Zhi et al. | Research of Hadoop-based data flow management system | |
| CN112650779B (en) | Cloud auditing method supporting deep page skipping query based on ElasticSearch | |
| CN114416670A (en) | Index creating method and device suitable for network disk document, network disk and storage medium | |
| WO2023244972A1 (en) | Unstructured file replication staged between database deployments | |
| Guo et al. | Research on improved A Priori algorithm based on coding and MapReduce | |
| US9229969B2 (en) | Management of searches in a database system | |
| CN112241396B (en) | Spark-based method and system for merging small files of Delta | |
| US11055262B1 (en) | Extensible streams on data sources | |
| US11593310B2 (en) | Providing writable streams for external data sources | |
| CN116126901A (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| Jiadi et al. | Research on Data Center Operation and Maintenance Management Based on Big Data | |
| JP7387116B2 (en) | Data archiving methods and systems to minimize costs when transmitting and querying data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |