CN115185774B - Automatic database based on open source technology - Google Patents

Abstract

The invention relates to the technical field of data processing and discloses an automatic database based on an open source technology, which comprises a central control module, an automatic monitoring module, a database index judging module, an automatic calculation capacity expansion module, an application program module and a resource recovery module, wherein the central control module is used for controlling the core of the database, not only can send out instructions to the automatic calculation capacity expansion module, but also can regulate and control other modules to control, the automatic monitoring module receives the instructions on the central control module to complete tasks, the automatic monitoring module detects indexes such as CPU utilization rate in real time, and the like, when the pressure of a system is found to be increased, resources are automatically recovered when the access amount is reduced through the database index judging module, so that unnecessary resource waste is avoided, the labor cost and the time cost are saved, and the stability of the system is enhanced.

Description

Automatic database based on open source technology

Technical Field

The invention relates to the technical field of data processing, in particular to an automatic database based on an open source technology.

Background

The access amount of the website refers to the index of the number of users accessing a website and the number of web pages browsed by the users, with the rapid development of the network age, the explosion type growth of big data is generated, so that the number of users and the number of web pages browsed by the website are gradually increased, in order to cope with the sudden increase of the access amount and the pressure on a database caused by the high access amount in some scenes, the common solution is to manually expand resources, and when the access amount is reduced, the resource recovery also needs to be manually adjusted, but the manual adjustment wastes huge manpower and time cost, the timeliness of the manual adjustment is poor, and the capacity expansion cannot be timely carried out when the access amount is rapidly increased.

There are mainly two cases that lead to an increase in the amount of access:

1. one is irregular traffic fluctuations: when a certain service application makes an activity method to make the access quantity suddenly rise, the access quantity gradually decreases after the activity is finished, and the time for the free rise and fall of the access quantity is difficult to predict due to irregular fluctuation of the business quantity caused by the activity, so timeliness is difficult to achieve by manually adjusting a database, and the adjustment quantity is also uncertain;

2. another is regular traffic fluctuations: in financial software, the end of the month or the beginning of the month is the period with highest use frequency, the access amount is increased rapidly when the financial software is used at high frequency, and the access amount at other times is not very large, so that the manual adjustment of the database at the end of the month or the beginning of the month is labor-consuming and time-consuming.

Disclosure of Invention

In order to overcome the defects in the prior art, the invention provides an automatic database based on an open source technology, which detects indexes such as CPU utilization rate and the like in real time through an automatic monitoring module, and once the system pressure is found to be increased, the number of the expansion required by the database index judging module can be timely determined, and resources can be automatically recovered when the access amount is reduced, so that unnecessary resource waste is avoided, the labor cost and the time cost are saved, and the stability of the system is enhanced.

The technical scheme is as follows:

in order to achieve the above purpose, the present invention provides the following technical solutions: the automatic capacity-expanding system comprises a central control module, an automatic monitoring module, a database index judging module, an automatic calculation capacity-expanding module, an application program module and a resource recycling module, wherein the central control module is used for controlling the core of a database, not only can the automatic calculation capacity-expanding module send out instructions, but also can regulate and control other modules to control, and the automatic monitoring module receives the instructions on the central control module to complete tasks and comprises a data acquisition unit, a data processing unit and a data monitoring display unit.

Preferably, the database index judging module judges the performance index of the database through a probability model to determine whether capacity expansion is needed, when the preset probability is exceeded, the instruction is transmitted to the automatic capacity expansion calculating module, when the preset probability is exceeded, the instruction is transmitted to the resource recycling module, and the automatic capacity expansion calculating module recalculates and distributes the existing resources and the existing data through a dock creation plan to select an optimal capacity expansion scheme.

Preferably, the application program module dynamically changes the route based on the scheme allocated by the automatic calculation capacity expansion module, and the resource recovery module recovers the initial state of the application program based on the condition that the index on the database adaptation index judgment module is lower than a preset value.

Preferably, the central control module includes data instruction processing, information storage, I/O interface control, and a power supply unit, where the data instruction processing refers to an operation instruction for data in a database, the data processing instruction includes a data transmission instruction, an arithmetic logic operation instruction, and a comparison instruction, and is mainly used for bidirectional transmission of data between a storage and a register, the information storage refers to calculation data, update data, insertion data, deletion data, or other functions of the database, and for the information storage, there are many methods including non-relational database storage, where for the data storage of a crawler, there is a case where some fields of one piece of data are lost due to failure in extraction, the data are adjusted at any time, and there is a nested relation between the data, and by the storage of the non-relational database, the database is simple and efficient, the I/O interface control refers to input and output between an internal database and an external database or other peripheral devices, and the power supply unit is used for providing uninterrupted continuous power supply to the central control module, the automation monitoring module, and other modules, so as to ensure normal operation of each device in the whole system.

Preferably, the automatic monitoring module comprises a data acquisition unit, a data processing unit and a data processing unit, wherein the data acquisition unit is used for acquiring as many data dimensions as possible while guaranteeing the quality of data through multi-source data acquisition, the multi-source data mainly comprises data acquisition modes of an open data source, crawling, log acquisition and sensor acquisition, and automatic early warning and intelligent error reporting are performed on the data, wherein the key description is that crawling is aimed at specific websites or capturing data of websites in an APP, and three processes of a Python crawler are as follows: firstly, crawling content by using Requests; secondly, analyzing the content by using XPath; XPath analyzes and indexes the position through the element and attribute; and thirdly, using Pandas to store data, wherein the data monitoring display unit is used for monitoring the database in real time through a mobile phone, a PC and other monitoring devices by using the man-machine interaction unit.

Preferably, the database index judging module judges whether the capacity expansion is needed or not according to a probability model, when the probability exceeds a preset probability, the command is transmitted to the automatic calculation capacity expansion module, when the probability exceeds the preset probability, the command is transmitted to the resource recycling module, the database index module comprises a transaction unit, a query performance unit, a user and query conflict unit, a capacity unit and other configuration units, the transaction unit is used for observing the behavior of a real user, capturing real-time performance when the application program module interacts, and comparing the response time of the whole transaction with the response time of each part of the composition transaction to confirm whether the current transaction is in a normal state or not; the query performance unit is used for checking whether the redundant data are more or not, whether the tables are connected in an inefficient manner or not, and the query problem caused by too much or too little index is selected in the query; the performance index of the capacity unit includes whether the CPU shares the load of the server, whether the IOPS is insufficient and whether the obtained disk is configured.

Preferably, the automatic calculation capacity expansion module is formed by creating a database mirror image through a dock creation plan, and the specific steps are as follows:

s1, firstly, pulling a mysql database from a certain website and installing the mysql database;

s2, creating a mysql database by using root users;

s3, after the database is established, the state can be checked through commands.

Preferably, the application program module refers to moving data in or out by a program when using the database, and the application program is connected to the database and makes a request: "acquire this data and store it in the specified location", another application makes a request: "find specified data and provide it to me" applications that interact with the database run when the user interacts with the Web page, such as when the user clicks the submit button after filling in the Web form, the program processes the information in the form and stores it in the database.

In a preferred embodiment, the resource recycling module automatically recycles the database under the condition of sufficient resources, and cleaning files in the resource recycling module is carried out by periodically scanning and dynamic tasks, namely cleaning according to a threshold mode while periodically scanning, for example, starting the automatic cleaning task when the threshold exceeds 75 percent, and comprehensively calculating according to the space size and the change time in the resource recycling module to obtain a corresponding processing mode.

The beneficial effects are that:

the invention provides an automatic database based on an open source technology, which has the following beneficial effects:

this an automatic database based on open source technique detects indexes such as CPU utilization ratio in real time through automatic monitoring module, in case when finding that system pressure increases, can be timely through the quantity of the required dilatation of database index judgement module, still can retrieve the resource automatically when the access volume reduces to avoid unnecessary wasting of resources, practiced thrift human cost and time cost also, strengthened the stability of system simultaneously.

Drawings

Fig. 1 is an overall system diagram of the present invention.

Fig. 2 is an overall flow chart of the present invention.

Fig. 3 is a block diagram of a central control module of the present invention.

Fig. 4 is a diagram of an automated monitoring module of the present invention.

FIG. 5 is a block diagram of a database index determination module according to the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Meanwhile, it should be understood that the sizes of the respective parts shown in the drawings are not drawn in actual scale for convenience of description.

The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the application, its application, or uses.

Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.

Embodiments of the present application may be applied to computer systems/servers that are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the computer system/server include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments that include any of the foregoing, and the like.

A computer system/server may be described in the general context of computer-system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.

Examples

The invention provides a technical scheme that: an automatic database based on an open source technology comprises a central control module, an automatic monitoring module, a database index judging module, an automatic calculation capacity expansion module, an application program module and a resource recovery module, wherein the central control module is used for controlling the core of the database, not only can the automatic calculation capacity expansion module send out instructions, but also can regulate and control and connect other modules to control, the automatic monitoring module receives the instructions on the central control module to complete tasks, the automatic database comprises a data acquisition unit, a data processing unit and a data monitoring display unit, the database index judging module judges the performance index of the database through a probability model to determine whether capacity expansion is needed, when the preset probability is exceeded, the instructions are transmitted to the automatic calculation capacity expansion module, when the preset probability is exceeded, the instructions are transmitted to the resource recovery module, the automatic calculation capacity expansion module recalculates and distributes existing resources and existing data through a dock creation plan, an optimal capacity expansion scheme is selected, the application program module dynamically changes routes based on the scheme distributed by the automatic calculation capacity expansion module, and the resource recovery module is adapted to the corresponding to the performance index on the database after the preset index is lower than the initial preset value.

In this embodiment, it needs to be specifically described that the central control module includes a data instruction processing unit, an information storage unit, an I/O interface control unit, and a power supply unit, where the data instruction processing unit refers to an operation instruction for data in the database, the data processing instruction includes a data transmission instruction, an arithmetic logic operation instruction, and a comparison instruction, and is mainly used for bidirectional transmission of data between the storage unit and the register, the information storage unit refers to a function of calculating data, updating data, inserting data, deleting data, or operating data in the database, and for many information storage methods, including non-relational database storage, where for the data storage of a crawler, there is a case where some fields of data are extracted and lost, the data is adjusted at any time, and there is a nesting relationship between the data, and by the storage of the non-relational database, the database is simple and efficient, the I/O interface control refers to input and output between the internal database and the external database or other peripheral devices, and the power supply unit is used for providing uninterrupted continuous power supply for the central control module, the automation monitoring module, and other modules, so as to ensure that each device in the whole system operates normally.

In this embodiment, it needs to be specifically described that the automatic monitoring module includes a data acquisition unit, a data processing unit, and a processing unit, where the data acquisition unit collects as many data dimensions as possible while guaranteeing the quality of data through multi-source data acquisition, and the multi-source data mainly includes data source opening, crawler capturing, log acquisition, and sensor acquisition, where it is mainly described that the crawler capturing is capturing data of a website in a specific website or APP, and three processes of Python crawler: firstly, crawling content by using Requests; secondly, analyzing the content by using XPath; XPath analyzes and indexes the position through the element and attribute; thirdly, the Pandas is used for storing data, the data monitoring display unit is used for monitoring the database in real time through a mobile phone, a PC and other monitoring equipment by using the man-machine interaction unit by a user, and the specific mode is as follows:

s1, a database configuration file my.ini is opened in a database installation directory;

s2, adding log=log.txt codes in the last row of the database;

s3, restarting the mysql database;

s4, removing a database data catalog;

s5, operating the database;

s5, checking the log.txt file content.

The specific description is that the change of the mysql database is monitored when the change description is found in the database, and the query, deletion, update and insertion of the database can be found, and the embodiment is not limited in detail.

In this embodiment, it needs to be specifically described that the database index judging module judges the performance index of the database through a probability model to determine whether capacity expansion is needed, when the preset probability is exceeded, an instruction is transmitted to the automatic calculation capacity expansion module, when the preset probability is exceeded, the instruction is transmitted to the resource recycling module, the database index module includes a transaction unit, a query performance unit, a user and query conflict unit, a capacity unit and other configuration units, the transaction unit is to observe the behavior of a real user, capture real-time performance when the application program module interacts, and obtain the response time of the whole transaction and the response time of each part of the composition transaction to confirm whether the current transaction is in a normal state; the query performance unit is used for checking whether the redundant data are more or not, whether the tables are connected in an inefficient manner or not, and the query problem caused by too much or too little index is selected in the query; the performance index of the capacity unit includes whether the CPU shares the load of the server, whether the IOPS is insufficient, and whether the configuration is obtained from the disk.

In this embodiment, it should be specifically described that the automatic computing capacity expansion module creates the database mirror image through a dock creation plan, and the specific steps are as follows:

s1, firstly, pulling a mysql database from a certain website and installing the mysql database;

s2, creating a mysql database by using root users;

s3, after the database is established, the state can be checked through commands: dock ps-looks at which processes dock has started; docker stop fusion-stop mysql database; docker start fusion-initiate mysql database.

In this embodiment, it should be specifically described that the application program module refers to moving data in or out by a program when using the database, and the application program connects to the database and makes a request: "acquire this data and store it in the specified location", another application makes a request: "find specified data and provide it to me" applications that interact with the database run when the user interacts with the Web page, such as when the user clicks the submit button after filling in the Web form, the program processes the information in the form and stores it in the database.

In this embodiment, it should be specifically described that the resource recovery module realizes automatic recovery of the database under the condition of sufficient resources, and cleaning files in the resource recovery module is a combination of periodic scanning and dynamic tasks, that is, cleaning is performed according to a threshold mode while periodic scanning, for example, when the threshold exceeds 75%, an automatic cleaning task is started, and comprehensive calculation is performed in the resource recovery module according to the space size and the change time to obtain a corresponding processing mode.

An automatic database management method based on open source technology comprises the following steps:

101. firstly, a central control module controls a database to send out instructions to an automatic calculation capacity expansion module and can regulate and control other modules to control, and then an automatic monitoring module receives the instructions on the central control module to complete tasks through a data acquisition unit, a data processing unit and a data monitoring display unit;

102. the database index judging module judges the performance index of the database through a probability model to determine whether capacity expansion is needed or not, when the probability exceeds the preset probability, the instruction is transmitted to the automatic capacity expansion calculating module, and when the probability is lower than the preset probability, the instruction is transmitted to the resource recycling module;

103. then the automatic calculation capacity expansion module utilizes a dock to create a plan for the database to recalculate and allocate the existing resources and the existing data, an optimal capacity expansion scheme is selected, and then the application program module dynamically changes the route based on the scheme allocated by the automatic calculation capacity expansion module;

104. and finally, the resource recovery module recovers the initial state of the application program after the index on the database adaptation index based judgment module is lower than a preset value.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (1)

1. An automated database based on open source technology, which is characterized in that: the system comprises a central control module, an automatic monitoring module, a database index judging module, an automatic calculation capacity expansion module, an application program module and a resource recycling module;

the central control module is used for sending out instructions to the automatic calculation capacity expansion module and controlling other modules connected with the automatic database through regulation and control;

the automatic monitoring module is used for completing tasks by receiving instructions of the central control module;

the database index judging module judges and processes the performance index of the database through the probability model to judge whether the database needs to be expanded, when the probability exceeds the preset probability, the central control module transmits the instruction to the automatic calculation expansion module, and when the probability is lower than the preset probability, the central control module transmits the instruction to the resource recycling module;

the automatic calculation capacity expansion module is used for carrying out recalculation and distribution on the existing resources and the existing data through a dock creation plan, and selecting an optimal capacity expansion scheme;

the application program module dynamically changes the route according to the scheme distributed by the automatic calculation capacity expansion module, wherein when a user interacts with a Web page, the application program moves data in or out of the database;

the resource recovery module is used for automatically recovering the database under the condition of sufficient resources according to the condition that the index judged by the database index judging module is lower than a preset value, cleaning files in the resource recovery module in a mode of periodically scanning and simultaneously cleaning the files according to a threshold value, and comprehensively calculating in the resource recovery module according to the space size and the change time to obtain a corresponding processing mode;

the central control module comprises a data instruction processing unit, an information storage unit, an I/O interface control unit and a power supply unit;

the data instruction processing means an operation instruction for data in a database, including a data transmission instruction, an arithmetic logic operation instruction and a comparison instruction, and is used for carrying out bidirectional transmission on the data between a storage and a register;

the information storage is non-relational database storage;

the I/O interface controls data input and output between the internal database and the external database or other peripheral equipment;

the power supply unit supplies power for the central control module, the automatic monitoring module and other modules of the automatic database;

the automatic monitoring module comprises a data acquisition unit, a data processing unit and a data monitoring display unit;

the data acquisition unit collects multiple dimensional data through multi-source data acquisition while guaranteeing the quality of the data, and automatically early warning and intelligently reporting errors are carried out on the data in a mode of opening a data source, crawling, log acquisition and sensor acquisition, wherein crawling is aimed at specific websites or capturing data of websites in APP, and three processes of Python crawlers are carried out: firstly, crawling content by using Requests; secondly, analyzing the content by using XPath, and carrying out position index by the XPath analysis through elements and attributes; thirdly, using Pandas to store data;

the user utilizes the man-machine interaction unit to monitor the database in real time through the mobile phone, the PC and the monitoring equipment and the data monitoring display unit, and the concrete mode is as follows:

s1, a database configuration file my.ini is opened in a database installation directory;

s2, adding log=log.txt codes in the last row of the database;

s3, restarting the mysql database;

s4, removing a database data catalog;

s5, operating the database, and checking log.txt file contents;

the database index judging module comprises a transaction unit, a query performance unit, a user and query conflict unit, a capacity unit and other configuration units;

the transaction unit is used for acquiring the behavior of a real user, capturing real-time performance when the application program module interacts, acquiring the response time of the whole transaction and the response time of each part forming the transaction, and comparing to confirm whether the current transaction is in a normal state or not;

the query performance unit is used for checking whether multiple redundant data exist in selection, whether the tables are connected in an inefficient manner, and query problems caused by excessive or insufficient indexes;

acquiring performance indexes of the capacity unit by judging whether a CPU of a server is loaded, whether an IOPS is insufficient or not and whether disk configuration is correct or not;

the automatic calculation capacity expansion module is used for creating a database mirror image through a dock creation plan, and specifically comprises the following steps:

s1, firstly, pulling a mysql database from a certain website and installing the mysql database;

s2, creating a mysql database by using root users;

s3, checking a state through a command after the database is established, wherein the command comprises: dock ps-view dock process, docker stop fusion-stop mysql database, docker start fusion-start mysql database.

Images