CN112131215A - Bottom-up database information acquisition method and device - Google Patents
Bottom-up database information acquisition method and device Download PDFInfo
- Publication number
- CN112131215A CN112131215A CN201910556105.8A CN201910556105A CN112131215A CN 112131215 A CN112131215 A CN 112131215A CN 201910556105 A CN201910556105 A CN 201910556105A CN 112131215 A CN112131215 A CN 112131215A
- Authority
- CN
- China
- Prior art keywords
- data table
- database
- information
- data
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
Abstract
The invention discloses a bottom-up database information acquisition method and a bottom-up database information acquisition device, wherein the method comprises the following steps: collecting and analyzing database session information to obtain database access information; analyzing the access information of the database, and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables; and iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information. The invention can be used for conveniently and continuously updating and acquiring the database information, and is simple, convenient and quick to operate. And the whole acquisition process application system is non-invasive and non-reconstruction, and can be suitable for various IT systems. Furthermore, the acquired database information also comprises a data table type, so that the subsequent data cold and hot separation, data life cycle management and the like can be realized conveniently according to the data table type.
Description
Technical Field
The invention relates to the technical field of data services, in particular to a bottom-up database information acquisition method and device.
Background
Data is the most important asset in an IT company or an informatization department. With the development of big data technology, the data capacity stored in the IT system is increased explosively, for example, in the case of Chongqing move, the data capacity stored in the sub data warehouse exceeds 300TB, the data capacity of the big data platform exceeds 3PB, and the data increase exceeds 35% every year. It is a technical problem to extract information such as association between data tables, data access frequency, data cold and hot distribution from such large data to perform data asset management.
In the current large IT system, no matter NOSQL data stored in a large data platform or relational data stored in a transaction system, when the number of data tables exceeds ten thousand, the association between the data tables and the data access frequency are very difficult to find. When acquiring data association, the method mainly depends on the combing of upper-layer application, generally adopts a top-down mode, and acquires the use information of the data table, the association relation of the data table and the like by a mode of searching the whole code.
The prior art adopts top-down application layer full carding and has the following technical problems:
1. application layer full carding can only check the use condition of the data table from the program code, but because the program code runs dynamically, the code carding cannot cover all business scenes, and the incidence relation of the data table obtained by carding is incomplete. Meanwhile, a new data object is introduced when a new program is on line, and the data assets which are only combed by the full code once are inaccurate, so that the association relation of the data table cannot be dynamically updated.
2. The application layer carding can only analyze the association condition of the data table, and cannot analyze information such as data distribution, data access frequency, cold and hot degrees and the like.
3. The application carding is a one-time action, the output data assets cannot be dynamically updated, and the original data assets are inaccurate after the application version and the bottom data are changed.
Disclosure of Invention
In view of the above, the present invention has been made to provide a bottom-up database information acquisition method and apparatus that overcomes or at least partially solves the above-mentioned problems.
According to an aspect of the present invention, there is provided a bottom-up database information acquisition method, including:
collecting and analyzing database session information to obtain database access information;
analyzing the access information of the database, and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables;
and iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information.
According to another aspect of the present invention, there is provided a bottom-up database information acquisition apparatus including:
the collection module is suitable for collecting and analyzing database session information to obtain database access information;
the analysis module is suitable for analyzing the access information of the database and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables;
and the association module is suitable for iteratively acquiring a database full-scale map containing the association relation and the type of the data table based on a splitting algorithm according to the information of the data table.
According to still another aspect of the present invention, there is provided an electronic apparatus including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the bottom-up database information acquisition method.
According to still another aspect of the present invention, a computer storage medium is provided, where at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to perform operations corresponding to the bottom-up database information obtaining method.
According to the bottom-up database information acquisition method and device, the database session information is collected and analyzed to obtain the database access information; analyzing the access information of the database, and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables; and iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information. The invention can be used for conveniently and continuously updating and acquiring the database information, and is simple, convenient and quick to operate. And the whole acquisition process application system is non-invasive and non-reconstruction, and can be suitable for various IT systems. Furthermore, the acquired database information also comprises a data table type, so that the subsequent data cold and hot separation, data life cycle management and the like can be realized conveniently according to the data table type.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 illustrates a flow diagram of a bottom-up database information acquisition method according to one embodiment of the invention;
FIG. 2 illustrates a flow diagram of a bottom-up database information acquisition method according to another embodiment of the present invention;
FIG. 3 shows a functional block diagram of a bottom-up database information retrieval apparatus according to one embodiment of the present invention;
fig. 4 shows a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIG. 1 shows a flow diagram of a bottom-up database information acquisition method according to one embodiment of the invention. As shown in fig. 1, the bottom-up database information acquisition method specifically includes the following steps:
step S101, collecting and analyzing database session information to obtain database access information.
In the prior art, when database information is acquired from top to bottom, operations such as embedding points in an application system are adopted to acquire access information of the database. Therefore, the application system needs to be modified, and when the application system performs operations such as code modification, the operations such as embedding points also need to be correspondingly updated to acquire the access information of the database. In view of the above, the present embodiment employs a bottom-up approach to obtain database information. And starting from the bottom to the top, namely from the database side, and acquiring the database information. Specifically, when collecting the database session information, the database session information may be collected in a manner of monitoring the database at the database side without paying attention to a specific code at the system side. After the database session information is collected, analyzing the database session information, classifying and sorting the database session information, and generating corresponding database access information for each piece of session information. Namely, the data information normalization processing is carried out on the database session information. The obtained database access information includes, for example, a host name, an access IP address, an application name, an SQL statement, and the like. The database access information may be stored in an XML file. That is, a piece of database session information generates a corresponding XML file, wherein the XML file records the database access information. The database access information recorded in the XML file may be in the following format:
wherein, the < num > tag records XML file number and distinguishes each XML file; the < IP > tag records the IP address of the access database; the < machine > tag records the host name of the access database; the < application > tag records the name of the application accessing the database; the < SQL1> tag records the SQL statements to be executed to access the database. Other database access information, such as a schema for accessing the database, may also be recorded in the XML file, which is not limited herein.
And step S102, analyzing the access information of the database and determining the information of each data table in the database.
Analyzing the obtained database access information in the XML file, specifically, analyzing an SQL statement to be executed to access the database, which is recorded by the < SQL1> tag in the database access information, and obtaining related data table operation information in the SQL statement. The data table operation information comprises a data table name, an SQL statement operation type, a data table incidence relation and the like. The data table name is a data table related in the SQL statement, the operation type of the SQL statement is related to the SQL statement, and the data table incidence relation is obtained by analyzing the data table related in the SQL statement and the query condition.
Determining each data table information according to the data table operation information. The data table information includes the access amount of the data table, the association degree information between the data tables, the data variation amount of the data table, and the like. The access amount of the data table is determined according to the number of times of occurrence of the name of the data table. After the SQL statement is analyzed, the more times the name of the data table appears, the larger the access amount of the obtained data table is. And determining the association degree information among the data tables according to the association relation among two or more data tables. After the SQL statement is analyzed, the incidence relation between two or more data tables is obtained for many times, and the incidence degree between the data tables is high. And the data variation of the data table needs to be monitored, and the data variation of the data table is counted according to the data variation of the data table. The data variation may be counted in units of days.
And S103, iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information.
And accumulating the association degree information of each data table according to the data table information and aiming at each data table to obtain an association degree accumulated value of each data table. And sorting according to the relevance accumulated value from high to low, sequentially selecting a data table with the relevance accumulated value sorted in the front as an iteration starting point of the splitting algorithm, iteratively obtaining a data table with direct or indirect relevance with the data table, and generating a database full-scale map containing the relevance of the data table. If the data table A with the highest accumulated value of the association degrees is determined, retrieval is carried out through the association relation among the data tables, the data tables A1, A2, A3 and the like which have the association relation with the data table A are retrieved, then iteration is continued to search other data tables such as A11, A12 and the like which have the association relation with the data table A1 by taking the data table A1 as a starting point in sequence until all the data tables which have direct or indirect association relation with the data table A are found, and until the data tables which have the association relation cannot be found in iteration, the maps of the associated data tables related to the data table A are obtained through carding according to the association relation. And according to the sequence of the accumulated value of the degree of association, such as a data table A, a data table B, a data table C and a data table D … …, after the atlas of the data table A is obtained through iterative search, continuously iteratively searching the data table which has direct or indirect association relation with the data table B and the data table C … … based on a splitting algorithm, and thus obtaining the full-amount atlas of the database. Further, if the data table D has an association relationship with the data table B, after the map of the data table B is searched, the association map of the data table D does not need to be searched.
Furthermore, the type of the data table in the database full-scale map can be marked according to the data variation of the data table and the access amount of the data table. The data table types include a state table, a log table, a configuration table, a dictionary table, a temporary table, a no access table, and the like. According to the difference between the data variation of the data table and the access amount of the data table, for example, when the data variation of the data table is large and the access amount of the data table is large, such frequently-accessed data tables are state tables, log tables and the like; for the data tables with small data variation and small data table access, the data tables are generally configuration tables, dictionary tables and the like; the data tables include a temporary table, a no access table, and the like, in general, in which the data change amount of the data table exists only for a certain period of time and the data table has no access amount. According to the type of the data table, the data table can be distinguished by different colors in the database full-scale map.
According to the bottom-up database information acquisition method provided by the invention, database session information is collected and analyzed to obtain database access information; analyzing the access information of the database, and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables; and iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information. The invention can be used for conveniently and continuously updating and acquiring the database information, and is simple, convenient and quick to operate. And the whole acquisition process application system is non-invasive and non-reconstruction, and can be suitable for various IT systems. Furthermore, the acquired database information also comprises a data table type, so that the subsequent data cold and hot separation, data life cycle management and the like can be realized conveniently according to the data table type.
Fig. 2 shows a flow diagram of a bottom-up database information acquisition method according to another embodiment of the invention. As shown in fig. 2, the bottom-up database information obtaining method specifically includes the following steps:
step S201, collecting and analyzing database session information to obtain database access information.
When the database session information is collected from bottom to top, the monitoring is carried out on the database side, so that the session information generated by directly calling the database through manual operation can be collected besides the session information of the application system to the database. Analyzing the session information collected by manual operation, and obtaining the following database access information:
the application system can be distinguished from the application system accessing the database daily according to the host name of the access database recorded by the < machine > tag and the application name of the access database recorded by the < application > tag. Host names are inconsistent, application names are database tools, and the like.
Step S202, removal processing is performed on the database access information.
The manually operated database access information has a great randomness, and unlike the database access information of the application system, it may affect the database access information of the application system. Therefore, the removal process is also required for the obtained database access information. The removal process includes removing database access information that conforms to the type of manual operation.
Step S203, analyzing the access information of the database and determining the information of each data table in the database.
Specifically, for example, the names of the data tables involved in parsing SQL statement selection b.right tid from OPERATOR _ work a, work _ RIGHT B where a.status ═ 1and b.status ═ 1and a.role ═ b.itemid and a.operator ═ OPERATOR and b.right tid: RIGHT tid can be determined as OPERATOR _ work and work _ RIGHT; the SQL statement operation type is Data Query Language (DQL); the data table association relationship, namely, the relationship between OPERATOR _ working _ RIGHT and working _ RIGHT exists. The above database operation information may be stored in the format shown in table 1 below:
| numbering | Data sheet | Degree of association |
| 1 | Subscribe | 2 |
| 1 | busscustsubs | 2 |
| 2 | OPERATOR_WORKGROUP | 1 |
| 2 | WORKGROUP_right | 1 |
TABLE 1
After SQL sentences in database access information are analyzed, the data table can be known
There is an association between OPERATOR _ working _ RIGHT and working _ RIGHT, and two data numbered 2 in table 1 record that the association between data tables OPERATOR _ working _ RIGHT and working _ RIGHT is 1. Namely, the data tables with the same number have an association relationship. The degree of association represents the number of associated accesses between two or more data tables. That is, when analyzing the SQL statement of the multiple pieces of database access information, if the associated data table is found in the SQL statement, if the associated data table is not recorded in table 1, the associated data table with the association degree of 1 is newly added, and if the associated data table is recorded in table 1, the association degree is directly added by 1 on the basis of the already recorded associated data table. The two related data tables numbered 1 in table 1 have a degree of association of 2.
In addition to the above table 1, which records the information of the degree of association between data tables, other tables may be used to record the access amount of the data tables. Wherein, the access amount of the data table is determined according to the occurrence number of the name of the data table. After the SQL statement is analyzed, the more times the name of the data table appears, the larger the access amount of the obtained data table is.
And further, determining a data table to be monitored according to the SQL statement operation type. The SQL statement operation types include, for example, a Data Manipulation Language (DML), a Data Query Language (DQL), a database schema Definition Language (DDL), and the like. The SQL statements of the DML operation type comprise SQL statements such as insert, update, delete and the like, and the SQL statements of the DDL operation type comprise SQL statements such as create, alter, drop and the like. The SQL sentences of the DML and DDL operation types can influence the data change sent by the data table, the data table related to the SQL sentences of the DML and DDL operation types is monitored, and the data change quantity of the data table is counted. The data variation may be counted in units of days. The information of the statistical data variation can be added on the basis of table 1, which is specifically shown in table 2 below:
| numbering | Watch (A) | Degree of association | Data change | Data variance |
| 1 | Subscribe | 2 | Y | 50M/day |
| 1 | busscustsubs | 2 | Y | 0.2M/day |
| 2 | OPERATOR_WORKGROUP | 1 | N | 0 |
| 2 | WORKGROUP_right | 1 | N | 0 |
TABLE 2
Table 2 adds data change (related to SQL statement operation type) and data change amount to the data table on the basis of table 1.
And step S204, iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information.
According to the data table information recorded in table 2, for each data table, the association degree information of each data table (i.e., the association degree in table 2) is accumulated to obtain an association degree accumulated value of each data table. And sorting according to the relevance accumulated value from high to low, sequentially selecting a data table with the relevance accumulated value sorted in the front as an iteration starting point of the splitting algorithm, iteratively obtaining a data table with direct or indirect relevance with the data table, and generating a database full-scale map containing the relevance of the data table. If the data table A with the highest accumulated value of the association degree is determined, the data tables A1, A2, A3 and the like which have the association relation with the data table A are searched in the table 2, then other data tables such as A11, A12 and the like which have the association relation with the data table A1 are continuously and iteratively searched from the data table A1 in sequence by starting points until all data tables which have direct or indirect association relation with the data table A are found until the data tables which have the association relation cannot be found in iteration, and the related association data table maps of the data table A are obtained through the association relation carding. And according to the sequence of the accumulated value of the degree of association, such as a data table A, a data table B, a data table C and a data table D … …, after the atlas of the data table A is obtained through iterative search, continuously iteratively searching the data table which has direct or indirect association relation with the data table B and the data table C … … based on a splitting algorithm, and thus obtaining the full-amount atlas of the database. Further, if the data table D has an association relationship with the data table B, after the map of the data table B is searched, the association map of the data table D does not need to be searched.
Furthermore, the type of the data table in the database full-scale map can be marked according to the data variation of the data table and the access amount of the data table. The data table types include a state table, a log table, a configuration table, a dictionary table, a temporary table, a no access table, and the like. According to the difference between the data variation of the data table and the access amount of the data table, for example, when the data variation of the data table is large and the access amount of the data table is large, such frequently-accessed data tables are state tables, log tables and the like; for the data tables with small data variation and small data table access, the data tables are generally configuration tables, dictionary tables and the like; the data tables include a temporary table, a no access table, and the like, in general, in which the data change amount of the data table exists only for a certain period of time and the data table has no access amount. According to the type of the data table, the data table can be distinguished by different colors in the database full-scale map.
According to the bottom-up database information acquisition method provided by the invention, noise influence generated by manual operation is removed, the database information can be conveniently and continuously acquired in an updating manner on the basis of the session information of the bottom-up collection application system on the database, and the operation is simple, convenient and quick. And the whole acquisition process application system is non-invasive and non-reconstruction, and can be suitable for various IT systems. Furthermore, the acquired database information also comprises a data table type, so that the subsequent data cold and hot separation, data life cycle management and the like can be realized conveniently according to the data table type.
Fig. 3 shows a functional block diagram of a bottom-up database information acquisition apparatus according to an embodiment of the present invention. As shown in fig. 3, the bottom-up database information acquisition apparatus includes the following modules:
the collection module 310 is adapted to: and collecting and analyzing database session information to obtain database access information.
The analysis module 320 is adapted to: analyzing the access information of the database, and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables.
The association module 330 is adapted to: and iteratively acquiring a database full-scale map containing the data table association relation and the data table type based on a splitting algorithm according to the data table information.
Optionally, the collecting module 310 is further adapted to: monitoring the database and collecting database session information; analyzing the database session information, classifying and sorting the database session information, and generating corresponding database access information for each piece of session information.
Optionally, the apparatus further comprises: the module 340 is removed. The removal module 340 is adapted to: removing the database access information; wherein the removing process comprises removing the database access information conforming to the manual operation type.
Optionally, the analysis module 320 is further adapted to: analyzing SQL sentences in the database access information to obtain related data table operation information in the SQL sentences; the data table operation information comprises a data table name, an SQL statement operation type and a data table association relation; determining the access amount of each data table in the database and the association degree information among the data tables according to the data table operation information; the access amount of the data table is determined according to the number of times of name occurrence of the data table; determining the association degree information among the data tables according to the association relation of the data tables; and determining a data table to be monitored according to the SQL statement operation type, and counting the data variation of the data table.
Optionally, the association module 330 is further adapted to: accumulating the association degree information of each data table aiming at each data table to obtain an association degree accumulated value of each data table; sorting according to the relevance accumulated value from high to low, sequentially selecting a data table with the relevance accumulated value sorted in front as an iteration starting point of a splitting algorithm, iteratively obtaining a data table with direct or indirect relevance with the data table, and generating a database full-scale map containing the relevance of the data table; and marking the type of the data table in the database full-scale map according to the data variation of the data table and/or the access amount of the data table.
The descriptions of the modules refer to the corresponding descriptions in the method embodiments, and are not repeated herein.
The present application further provides a non-volatile computer storage medium, where at least one executable instruction is stored in the computer storage medium, and the computer executable instruction can execute the bottom-up database information obtaining method in any of the above method embodiments.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the electronic device.
As shown in fig. 4, the electronic device may include: a processor (processor)402, a Communications Interface 404, a memory 406, and a Communications bus 408.
Wherein:
the processor 402, communication interface 404, and memory 406 communicate with each other via a communication bus 408.
A communication interface 404 for communicating with network elements of other devices, such as clients or other servers.
The processor 402 is configured to execute the program 410, and may specifically perform relevant steps in the bottom-up database information obtaining method embodiment described above.
In particular, program 410 may include program code comprising computer operating instructions.
The processor 402 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The electronic device comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 406 for storing a program 410. Memory 406 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 410 may be specifically configured to cause the processor 402 to execute a bottom-up database information acquisition method in any of the above-described method embodiments. For specific implementation of each step in the program 410, reference may be made to the corresponding steps and corresponding descriptions in the units in the bottom-up database information acquisition embodiment, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a bottom-up database information retrieval apparatus according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (10)
1. A bottom-up database information acquisition method is characterized by comprising the following steps:
collecting and analyzing database session information to obtain database access information;
analyzing the database access information to determine the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables;
and iteratively obtaining a database full-scale map containing the data table incidence relation and the data table type based on a splitting algorithm according to the data table information.
2. The method of claim 1, wherein the database access information comprises a host name, an access IP address, an application name, and/or an SQL statement; the database access information is of an XML file type.
3. The method of claim 2, wherein collecting and parsing database session information to obtain database access information further comprises:
monitoring the database and collecting database session information;
analyzing the database session information, classifying and sorting the database session information, and generating corresponding database access information for each piece of session information.
4. The method of claim 2, wherein prior to said analyzing said database access information to determine each data table information in the database, said method further comprises:
removing the database access information; wherein the removing process comprises removing database access information conforming to a manual operation type.
5. The method of claim 2, wherein analyzing the database access information to determine the information of each data table in the database further comprises:
analyzing the SQL sentences in the database access information to obtain related data table operation information in the SQL sentences; the data table operation information comprises a data table name, an SQL statement operation type and a data table association relation;
determining the access amount of each data table in the database and the association degree information among the data tables according to the data table operation information; the access amount of the data table is determined according to the number of times of occurrence of the name of the data table; determining the association degree information among the data tables according to the association relation of the data tables;
and determining a data table to be monitored according to the SQL statement operation type, and counting the data variation of the data table.
6. The method of claim 5, wherein iteratively obtaining a database full-scale map containing data table associations and data table types based on a splitting algorithm according to the data table information further comprises:
accumulating the association degree information of each data table aiming at each data table to obtain an association degree accumulated value of each data table;
sorting according to the relevance accumulated value from high to low, sequentially selecting a data table with the relevance accumulated value sorted in front as an iteration starting point of a splitting algorithm, iteratively obtaining a data table with direct or indirect relevance with the data table, and generating a database full-scale map containing the relevance of the data table;
and marking the type of the data table in the database full-scale map according to the data variation of the data table and/or the access amount of the data table.
7. The method of claim 1, wherein the data table types comprise a state table, a log table, a configuration table, a dictionary table, a temporary table, and/or a no access table.
8. A bottom-up database information retrieval apparatus, comprising:
the collection module is suitable for collecting and analyzing database session information to obtain database access information;
the analysis module is suitable for analyzing the database access information and determining the information of each data table in the database; the data table information comprises the access amount of the data tables, the association degree information among the data tables and/or the data variation of the data tables;
and the association module is suitable for iteratively acquiring a database full-scale map containing the association relation and the type of the data table based on a splitting algorithm according to the information of the data table.
9. An electronic device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the bottom-up database information acquisition method as claimed in any one of claims 1 to 7.
10. A computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the bottom-up database information retrieval method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910556105.8A CN112131215B (en) | 2019-06-25 | 2019-06-25 | Bottom-up database information acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910556105.8A CN112131215B (en) | 2019-06-25 | 2019-06-25 | Bottom-up database information acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112131215A true CN112131215A (en) | 2020-12-25 |
| CN112131215B CN112131215B (en) | 2023-09-19 |
Family
ID=73850095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910556105.8A Active CN112131215B (en) | 2019-06-25 | 2019-06-25 | Bottom-up database information acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131215B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117453229A (en) * | 2023-12-26 | 2024-01-26 | 天津戎行集团有限公司 | Output code analysis method for automatic code generation database |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101425060A (en) * | 2007-11-02 | 2009-05-06 | 英业达股份有限公司 | Database data table generating method |
| US20090182779A1 (en) * | 2008-01-15 | 2009-07-16 | At&T Services, Inc. | Complex dependencies for efficient data warehouse updates |
| US20100262631A1 (en) * | 2009-04-14 | 2010-10-14 | Sun Microsystems, Inc. | Mapping Information Stored In a LDAP Tree Structure to a Relational Database Structure |
| CN105512200A (en) * | 2015-11-26 | 2016-04-20 | 华为技术有限公司 | Distributed database processing method and device |
| CN106227892A (en) * | 2016-08-24 | 2016-12-14 | 深圳市卓讯信息技术有限公司 | A kind of intellectual analysis database table relation generates the method and device of E R figure |
| CN106649378A (en) * | 2015-11-02 | 2017-05-10 | 北大方正集团有限公司 | Data synchronizing method and device |
| CN107145574A (en) * | 2017-05-05 | 2017-09-08 | 恒生电子股份有限公司 | database data processing method, device and storage medium and electronic equipment |
| CN107239458A (en) * | 2016-03-28 | 2017-10-10 | 阿里巴巴集团控股有限公司 | The method and device of development object relation is calculated based on big data |
| CN107766431A (en) * | 2017-09-18 | 2018-03-06 | 杭州安恒信息技术有限公司 | It is a kind of that Parameter Function Unit method and system are gone based on syntax parsing |
| CN109101632A (en) * | 2018-08-15 | 2018-12-28 | 中国人民解放军海军航空大学 | Product quality abnormal data retrospective analysis method based on manufacture big data |
| CN109213765A (en) * | 2018-08-20 | 2019-01-15 | 中国平安人寿保险股份有限公司 | The associated method, apparatus of tables of data, computer equipment and storage medium |
| CN109325038A (en) * | 2018-09-05 | 2019-02-12 | 天津航旭科技发展有限公司 | Knowledge mapping extended model, structural knowledge storage method and equipment |
| CN109344152A (en) * | 2018-08-22 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN109697066A (en) * | 2018-12-28 | 2019-04-30 | 第四范式（北京）技术有限公司 | Realize the method and system of tables of data splicing and automatic training machine learning model |
-
2019
- 2019-06-25 CN CN201910556105.8A patent/CN112131215B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101425060A (en) * | 2007-11-02 | 2009-05-06 | 英业达股份有限公司 | Database data table generating method |
| US20090182779A1 (en) * | 2008-01-15 | 2009-07-16 | At&T Services, Inc. | Complex dependencies for efficient data warehouse updates |
| US20100262631A1 (en) * | 2009-04-14 | 2010-10-14 | Sun Microsystems, Inc. | Mapping Information Stored In a LDAP Tree Structure to a Relational Database Structure |
| CN106649378A (en) * | 2015-11-02 | 2017-05-10 | 北大方正集团有限公司 | Data synchronizing method and device |
| CN105512200A (en) * | 2015-11-26 | 2016-04-20 | 华为技术有限公司 | Distributed database processing method and device |
| CN107239458A (en) * | 2016-03-28 | 2017-10-10 | 阿里巴巴集团控股有限公司 | The method and device of development object relation is calculated based on big data |
| CN106227892A (en) * | 2016-08-24 | 2016-12-14 | 深圳市卓讯信息技术有限公司 | A kind of intellectual analysis database table relation generates the method and device of E R figure |
| CN107145574A (en) * | 2017-05-05 | 2017-09-08 | 恒生电子股份有限公司 | database data processing method, device and storage medium and electronic equipment |
| CN107766431A (en) * | 2017-09-18 | 2018-03-06 | 杭州安恒信息技术有限公司 | It is a kind of that Parameter Function Unit method and system are gone based on syntax parsing |
| CN109101632A (en) * | 2018-08-15 | 2018-12-28 | 中国人民解放军海军航空大学 | Product quality abnormal data retrospective analysis method based on manufacture big data |
| CN109213765A (en) * | 2018-08-20 | 2019-01-15 | 中国平安人寿保险股份有限公司 | The associated method, apparatus of tables of data, computer equipment and storage medium |
| CN109344152A (en) * | 2018-08-22 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN109325038A (en) * | 2018-09-05 | 2019-02-12 | 天津航旭科技发展有限公司 | Knowledge mapping extended model, structural knowledge storage method and equipment |
| CN109697066A (en) * | 2018-12-28 | 2019-04-30 | 第四范式（北京）技术有限公司 | Realize the method and system of tables of data splicing and automatic training machine learning model |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117453229A (en) * | 2023-12-26 | 2024-01-26 | 天津戎行集团有限公司 | Output code analysis method for automatic code generation database |
| CN117453229B (en) * | 2023-12-26 | 2024-02-23 | 天津戎行集团有限公司 | Output code analysis method for automatic code generation database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112131215B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019214245A1 (en) | Information pushing method and apparatus, and terminal device and storage medium | |
| JP5575902B2 (en) | Information retrieval based on query semantic patterns | |
| CN111459985B (en) | Identification information processing method and device | |
| US9390176B2 (en) | System and method for recursively traversing the internet and other sources to identify, gather, curate, adjudicate, and qualify business identity and related data | |
| WO2017097231A1 (en) | Topic processing method and device | |
| CN110109910A (en) | Data processing method and system, electronic equipment and computer readable storage medium | |
| CN111125086B (en) | Method, device, storage medium and processor for acquiring data resources | |
| CN115145871A (en) | File query method and device and electronic equipment | |
| JP6375029B2 (en) | A metadata-based online analytical processing system that analyzes the importance of reports | |
| US10628421B2 (en) | Managing a single database management system | |
| CN116450890A (en) | Graph data processing method, device and system, electronic equipment and storage medium | |
| CN111913860A (en) | Operation behavior analysis method and device | |
| CN103226601A (en) | Method and device for image search | |
| CN112131215B (en) | Bottom-up database information acquisition method and device | |
| US20180341709A1 (en) | Unstructured search query generation from a set of structured data terms | |
| CN111159213A (en) | Data query method, device, system and storage medium | |
| CN108170693B (en) | Hot word pushing method and device | |
| CN114416848A (en) | Data blood relationship processing method and device based on data warehouse | |
| CN109948015A (en) | A kind of Meta Search Engine tabulating result abstracting method and system | |
| CN109408727B (en) | Intelligent user attention information recommendation method and system based on multidimensional perception data | |
| JP5944867B2 (en) | Search intention dictionary creation device, search intention dictionary creation method, and search intention dictionary creation program | |
| CN110633430B (en) | Event discovery method, apparatus, device, and computer-readable storage medium | |
| CN112241262A (en) | Software-defined satellite-oriented reusable code extracting, analyzing and retrieving method and device | |
| CN111581220A (en) | Storage and retrieval method, device, equipment and storage medium for time series data | |
| JP2020154381A (en) | Information processing system, information processing device, information processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |