CN110362564B - Data source switching processing method, device and system - Google Patents
Data source switching processing method, device and system Download PDFInfo
- Publication number
- CN110362564B CN110362564B CN201810306525.6A CN201810306525A CN110362564B CN 110362564 B CN110362564 B CN 110362564B CN 201810306525 A CN201810306525 A CN 201810306525A CN 110362564 B CN110362564 B CN 110362564B
- Authority
- CN
- China
- Prior art keywords
- database
- data
- unit
- mysql
- unit time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 7
- 238000000034 method Methods 0.000 claims abstract description 88
- 230000008569 process Effects 0.000 claims abstract description 41
- 238000012544 monitoring process Methods 0.000 claims description 42
- 238000011161 development Methods 0.000 claims description 33
- 230000008439 repair process Effects 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 17
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 238000013500 data storage Methods 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 4
- 230000005856 abnormality Effects 0.000 claims 5
- 238000010586 diagram Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application shows a processing method, a device and a system for switching data sources, wherein the method comprises the following steps: acquiring a query request; traversing a database based on the query request, determining display content, the database comprising: ES database, and MYSQL database; and sending the display content to the user side. According to the method, the system defaults to traverse the ES database, the ES database integrates a plurality of data tables in MYSQL into one database, only one data table is required to be traversed in the traversing process, the conditions of time consumption and the like of multiple associated query in the query process are avoided, the waiting time of an application platform server is shortened, and the utilization rate of resources such as system bandwidth, the database and the like is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data source switching processing method, device, and system.
Background
Along with the development of science and technology and the transition of age, the twenty-first century China has been in the industrial age, vehicles become common daily riding tools, and people work and life are closely related to automobiles at all times. The coming of the vehicle era brings great social progress, promotes the rapid development of national economy, and draws in examples among people, places and places. With the rapid development of the vehicle industry, the variety of vehicles is gradually increased, and the difficulty of searching for a target vehicle suitable for itself in a plurality of vehicles is gradually increased, and in this case, a vehicle search method is developed.
The prior art provides various vehicle retrieval methods for finding a target vehicle among a wide variety of vehicles. The vehicle searching method in the prior art comprises the following specific processes: firstly, a user inputs a limiting condition of search at a user side, then, an application platform server searches display data matched with the limiting condition according to the limiting condition and sends the display data to the user side (the display data comprises related information of a target vehicle), and then the user side displays the display data.
The prior art provides various vehicle retrieval methods, generally, a platform server is applied to store collected vehicle data in a MYSQL database, and the data tables in the MYSQL database are traversed in sequence based on limiting conditions in the query process, so that display data are finally determined. For example: MYSQL databases typically include a vehicle table, a city table, a financial schema table, and a financial schema intermediate table; when the user inputs: when the value-added tax of the Shenyang Audi vehicle 2015 is met, the application platform server sequentially traverses a plurality of related tables to finally obtain display data, and in the process that the application platform server traverses the related tables, the application platform server is in a waiting state for a long time, so that the utilization rate of resources such as system bandwidth, a database and the like is reduced.
Disclosure of Invention
The invention aims to provide a processing method, a device and a system for switching data sources in order to solve the technical problems that in the prior art, a vehicle retrieval method is in a waiting state for a long time by an application platform server in the searching process, and the utilization rate of resources such as system bandwidth, a database and the like is reduced.
A first aspect of an embodiment of the present application shows a method for processing data source switching, where the method includes:
acquiring a query request;
traversing a database based on the query request, determining display content, the database comprising: ES database, and MYSQL database;
and sending the display content to the user side.
Optionally, the step of traversing the database includes:
enabling an ES database;
judging whether the ES database runs normally or not;
if the ES database normally operates, traversing the ES database;
and if the ES database runs abnormally, traversing the MYSQL database.
Optionally, the step of traversing the database includes:
monitoring service is started, and the running condition of the ES database is monitored in real time;
judging whether the running condition of the ES database accords with a monitoring rule;
if the monitoring rule is met, traversing the ES database;
if the monitoring rule is not met, traversing the MYSQL database and repairing the ES database.
Optionally, the step of repairing the ES database includes:
sending a repair request for repairing the ES database to a development server;
and receiving a repair completion instruction sent by the development server, and starting the ES database.
Optionally, the construction process of the ES database specifically includes:
traversing a data table in the MYSQL database;
combining records in different data tables to obtain an ES record;
and storing the ES record to generate an ES database.
Optionally, the data table comprises one or a combination of several of a city table, a vehicle source table, a time table, a financial scheme intermediate table and a financial scheme table.
A second aspect of an embodiment of the present application shows a processing apparatus for switching a data source, where the apparatus includes:
the acquisition unit is used for acquiring the query request;
a determining unit, configured to traverse a database based on the query request, and determine display content, where the database includes: ES database, and MYSQL database;
and the sending unit is used for sending the display content to the user side.
Optionally, the determining unit includes:
a first enabling unit for enabling the ES database;
a first judging unit for judging whether the ES database operates normally
The first ES traversing unit is used for traversing the ES database if the ES database runs normally;
the first MYSQL traversing unit is used for traversing the MYSQL database if the ES database runs abnormally.
Optionally, the determining unit includes:
a monitoring unit for enabling a monitoring service to monitor the operation status of the ES database in real time
The second judging unit is used for judging whether the running condition of the ES database accords with the monitoring rule;
the second ES traversing unit is used for traversing the ES database if the monitoring rule is met;
and the second MYSQL traversing unit is used for traversing the MYSQL database and repairing the ES database if the monitoring rule is not met.
Optionally, the second MYSQL traversal unit includes:
the first sending unit is used for sending a repair request for repairing the ES database to the development server;
and the first receiving unit is used for receiving the repair completion instruction sent by the development server and starting the ES database.
Optionally, the apparatus further comprises:
the data table traversing unit is used for traversing the data table in the MYSQL database;
a combining unit for combining records in different data tables to obtain an ES record;
and the storage unit is used for storing the ES records and generating an ES database.
A third aspect of embodiments of the present application illustrates a processing system for data source switching, the system including:
the system comprises an application platform server, a data storage server and a development server, wherein the data storage server is connected with the application platform server, and is arranged in the application platform server or independently, and the development server is connected with the application platform server and is connected with a terminal through the Internet;
the terminal is used for displaying information;
the application platform server is used for realizing the method shown in the embodiment of the application;
the data storage server is used for storing related data.
As can be seen from the above technical solutions, the embodiments of the present application show a processing method, apparatus and system for switching data sources, where the method includes: acquiring a query request; traversing a database based on the query request, determining display content, the database comprising: ES database, and MYSQL database; and sending the display content to the user side. According to the method, the system defaults to traverse the ES database, the ES database integrates a plurality of data tables in MYSQL into one database, only one data table is required to be traversed in the traversing process, the conditions of time consumption and the like of multiple associated query in the query process are avoided, the waiting time of an application platform server is shortened, and the utilization rate of resources such as system bandwidth, the database and the like is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1-1 is a block diagram illustrating a processing system for data source switching in accordance with a preferred embodiment;
FIGS. 1-2 are block diagrams illustrating a processing system for data source switching in accordance with yet another preferred embodiment;
FIG. 2 is a block diagram of a processing system for data source switching in accordance with yet another preferred embodiment;
FIG. 3 is a flow chart illustrating a method of handling data source switching in accordance with a preferred embodiment;
FIG. 4 is a detailed flow chart of step S102 shown in accordance with a preferred embodiment;
fig. 5 is a detailed flowchart of step S102 shown according to yet another preferred embodiment;
FIG. 6 is a detailed flow chart of step S10224 shown in accordance with a preferred embodiment;
FIG. 7 is a flow chart illustrating a method of building an ES database according to a preferred embodiment;
FIG. 8 is a block diagram illustrating a data source switching processing device in accordance with a preferred embodiment;
fig. 9 is a block diagram showing the structure of a determination unit according to a preferred embodiment;
fig. 10 is a block diagram showing the structure of a determination unit according to still another preferred embodiment;
FIG. 11 is a block diagram illustrating the structure of a second MYSQL traversal unit according to a preferred embodiment;
FIG. 12 is a block diagram showing a construction apparatus of an ES database according to a preferred embodiment;
fig. 13 is a block diagram of a server according to a preferred embodiment.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims of this application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be capable of implementation in sequences other than those illustrated or described herein, for example. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1:
the prior art provides various vehicle retrieval methods, generally, a platform server is applied to store collected vehicle data in a MYSQL database, and the data tables in the MYSQL database are traversed in sequence based on limiting conditions in the query process, so that display data are finally determined. For example: MYSQL databases typically include a vehicle table, a city table, a financial schema table, and a financial schema intermediate table; when the user inputs: when the value-added tax of the Shenyang Audi vehicle 2015 is met, the application platform server sequentially traverses a plurality of related tables to finally obtain display data, and in the process that the application platform server traverses the related tables, the application platform server is in a waiting state for a long time, so that the utilization rate of resources such as system bandwidth, a database and the like is reduced.
In order to solve the above technical problems, a first aspect of an embodiment of the present application shows a processing system for switching data sources;
referring to fig. 1-1, and to fig. 1-2, the system includes:
an application platform server 31, a data storage server 32 connected to the application platform server 31, wherein the data storage server 32 is disposed inside the application platform server 31 or independently disposed, a development server 34 connected to the application platform server 31, and the application platform server 31 is connected to a terminal 33 through the internet;
the terminal 33 is used for displaying information;
the terminal 33 shown in the embodiment of the present application is a device located at the outermost periphery of the network in the computer network, and is mainly used for inputting user information, outputting processing results, and the like. The prior art shows mobile terminals such as: the mobile phone and the PAD are both in the protection scope of the embodiment of the application.
The application platform server 31 is configured to implement the method shown in the embodiment of the present application;
the application platform server 31 shown in the embodiments of the present application provides a simple and manageable access mechanism to system resources for Web applications. The application platform server 31 also provides low-level services such as implementation of the TCP/IC protocol and database connection management.
The data storage server is used for storing related data.
Please refer to fig. 2:
the application platform server is internally provided with a data switch, and the data switch defaults to directly traverse an ES database based on a query request sent by a receiving terminal;
the system shown in the embodiment of the application applies an ES database in a platform server based on GRCP protocol;
the application platform server shown in the embodiments of the present application requests a connection to a GRPC based on the TCP/IP protocol that spans the transport and application layers in the OSI network communication model. Compared with the traditional application platform server traversing MYSQL database based on the HTTP protocol, the method has faster data transmission speed and improves the operation efficiency of the system.
When the data source of the ES database fails, the data switch is directly switched to the MYSQL database, and the application platform server directly traverses the MYSQL database to search out the display content matched with the query request.
The system shown in the embodiment of the application effectively avoids the problem of failure in acquiring data when the data source fails, and meanwhile, the MYSQL database and the ES database are mutually backed up, so that the performance availability is enhanced.
Meanwhile, in the running process of the MYSQL database, the error rate of the MYSQL database in unit time is counted in real time. If the error rate of the ES database in unit time exceeds the monitoring rule, the direct data switch is switched to the MYSQL database. And directly traversing the MYSQL database, and simultaneously, sending a mail to a development server, wherein the development server repairs the ES database or checks the data in the ES database. When the repair of the ES database or the inspection of the data in the ES database is completed, the development server sends a repair completion instruction to the application platform server, and at this time, the application platform server switches the data switch to the ES database, and the application platform server continues to traverse the ES database.
For example:
the user inquires 'Audi A6, 2015 value-added tax', the system defaults to traverse the ES database, when the 5 th data is found, the application platform server feeds back the display interface 'no record', the data switch is switched to the MYSQL database, and the application platform server continues to traverse the MYSQL database based on 'Audi A6, 2015 value-added tax'. When the 6 th data is detected, the data switch is switched to the ES database, the application platform server continuously traverses the ES database based on the 'Audi A6, 2015 value added tax', meanwhile, the application platform server monitors the error rate of the ES database in real time, for example, when the 203 th data is searched, the error rate of the ES database in 1 minute is 100 times, at the moment, the error rate of the ES database in unit time exceeds the monitoring rule, the application platform database sends mails to the development server, and the development server repairs the ES database or verifies the data in the ES database. When the repair of the ES database is finished or the data in the ES database is checked, the development server sends a repair finishing instruction to the application platform server, and at the moment, the application platform server switches the data switch to the ES database and the application platform server continuously traverses the ES database.
It is noted that, in the process of repairing the ES database by the development service or verifying the data in the ES database, the data switch of the application platform database is directly switched to the MYSQL database, and when the application platform server receives the repair completion instruction sent by the development server, the data switch is continuously switched to the ES database.
In summary, in the processing system for switching data sources shown in the embodiments of the present application, the system traverses the ES database by default, the ES database integrates a plurality of data tables in MYSQL into one database, and only one data table needs to be traversed in the traversal process, so that the situations of time consuming and the like of multiple associated queries in the query process are avoided, the waiting time of the application platform server is shortened, and the utilization rate of resources such as system bandwidth, database and the like is improved.
Example 2:
a second aspect of the embodiments of the present application shows a method for processing data source switching, referring to fig. 3, where the method includes:
s101, acquiring a query request;
the query request may be a query for a class of vehicle quantity, such as the quantity of brown odi vehicles;
the vehicle information of a certain vehicle may be queried, for example: the VIN code is: inquiring vehicle information of 123456789123 vehicles;
a service record of a certain vehicle may be queried.
Or may be a query for additional information about a vehicle or class of vehicles, including additional fees paid by the user during the purchase of the vehicle, e.g. value-added tax, insurance … …
It should be noted that, in the present embodiment of the present application, only the query of several vehicle related information is recorded by way of example, and in practical application, all the related items of the query of the vehicle are within the protection scope of the embodiment of the present application, and therefore, the present application is not limited by way of example.
S102, traversing a database based on the query request, and determining display content, wherein the database comprises: ES database, and MYSQL database;
in the actual traversing process, the system traverses the ES database by default, when the data source of the ES database fails, the data switch is directly switched to the MYSQL database, and the application platform server directly traverses the MYSQL database to search out the display content matched with the query request.
The system defaults to traverse the ES database, the ES database integrates a plurality of data tables in MYSQL into one database, only one data table needs to be traversed in the traversing process, the conditions of multiple associated inquiry time consumption and the like in the inquiring process are avoided, the waiting time of an application platform server is shortened, and the utilization rate of resources such as system bandwidth, databases and the like is improved.
Meanwhile, the ES database is based on the TCP/IP protocol, which spans the transport layer and the application layer in the OSI network communication model. Compared with the traditional application platform server traversing MYSQL database based on the HTTP protocol, the method has faster data transmission speed and improves the operation efficiency of the system.
Files in the ES database are stored in a binary form, so that compared with a complex MYSQL storage mode, the storage speed is faster; meanwhile, the complexity of data processing logic is low, the complex MYSQL statement is stored in a binary form, the problem that the MYSQL statement needs to cache data for multiple times when data change is effectively avoided, memory is saved, the running load of the system is not increased, and the running efficiency of the system is improved.
S103, sending the display content to the user side.
According to the method, the system defaults to traverse the ES database, the ES database integrates a plurality of data tables in MYSQL into one database, only one data table is required to be traversed in the traversing process, the conditions of time consumption and the like of multiple associated query in the query process are avoided, the waiting time of an application platform server is shortened, and the utilization rate of resources such as system bandwidth, the database and the like is improved.
Meanwhile, when the data source of the ES database fails, the data switch is directly switched to the MYSQL database, and the application platform server directly traverses the MYSQL database to search out the display content matched with the query request.
The method effectively avoids the problem of failure in acquiring data when the data source fails, and meanwhile, the MYSQL database and the ES database are mutually backed up, so that the performance availability is enhanced.
Example 3:
in order to further shorten the method shown in the embodiment of the present application, in the data searching process, the waiting time of the application platform server is shortened, and the situation that the application platform server is in a waiting state for a long time because the application platform server traverses the wrong database in the searching process is avoided, in which the embodiment of the present application shows a switching method of an ES database and a MYSQL database, specifically, please refer to fig. 4;
embodiment 3 has similar steps to the technical solution shown in embodiment 2, the only difference being that the step of traversing the database in step S102 in the technical solution shown in embodiment 2 comprises the following steps:
s10211, enabling an ES database;
enabling the ES database by default by the system;
s10212, judging whether the ES database runs normally or not;
judging whether the ES database runs normally or not, wherein the ES database comprises the following steps;
judging whether the ES database gives out a search result within a specified time; for example, if the predetermined time is 1s, if the search result is not given within 1s, the ES database is determined to be abnormal, and the data switch is switched to the MYSQL database
If the system returns a status code in the process of traversing the ES database, proving that the ES database operates abnormally;
if the system returns a prompt in the process of traversing the ES database, proving that the ES database operates abnormally;
in the actual traversal process, the abnormal operation of the ES database can be judged even if the traversal result does not return the correct display result, and the abnormal states are not listed one by one because of limited space.
S10213, traversing the ES database if the ES database operates normally;
s10214, traversing the MYSQL database if the ES database runs abnormally.
The embodiment of the application shows a switching process of an ES database and a MYSQL database. The system directly traverses the ES data by default, when the data source of the ES database fails, the data switch is directly switched to the MYSQL database, and the application platform server directly traverses the MYSQL database to search out the display content matched with the query request.
The system disclosed by the embodiment of the application effectively avoids the problem of data acquisition failure caused by the occurrence of faults of the data source, shortens the process of waiting for the ES database to be repaired by the application platform server when the ES database is abnormal, avoids repeated requests for invalid data, and improves the utilization rate of resources such as system bandwidth, databases and the like.
Meanwhile, the MYSQL database and the ES database are mutually backed up, so that the performance availability is enhanced.
Example 4:
if the ES database is frequently abnormal in operation, the ES database is preferably traversed, and when the ES database is abnormal in operation, the MYSQL database is traversed again, which clearly increases the selection time of the system, in order to solve the above problems, the embodiment of the present application shows a monitoring scheme of the ES database, and specifically, please refer to fig. 5:
embodiment 4 has similar steps to the technical solution shown in embodiment 2, and the only difference is that the step of traversing the database in step S102 in the technical solution shown in embodiment 2 includes the following steps, specifically, please refer to fig. 5:
s10221, starting a monitoring service, and monitoring the running condition of the ES database in real time;
in the inquiring process, a monitoring service is started, and the monitoring service is used for monitoring the running condition of the ES database in real time, wherein the running condition comprises the detection of the error rate of the ES database in unit time.
S10222, judging whether the running condition of the ES database accords with a monitoring rule;
the detection rule is to allow the ES database to be online with error rate in unit time;
for example, a calculation rule of a specific monitoring rule, for example, ES database 1s may provide 200 search results, MYSQL database 1s may provide 150 search results;
if 20 running anomalies occur in the ES database in the next 1s, the search results provided by traversing MYSQL directly in 1s are more than those provided by traversing the ES database first, such as the search results provided by traversing the MYSQL database after the running anomalies of the ES database are found.
At this time, 20 times/s are used as monitoring rules;
it should be noted that, in the embodiment of the present application, only an exemplary calculation scheme describing a monitoring rule is described, in the practical application process, where it can be determined that the search result provided by directly traversing MYSQL in a unit time is more than that provided by traversing the ES database first, for example, the monitoring rule of finding that the ES database is abnormal in operation and then traversing the search result provided by traversing the MYSQL database is within the protection range of the method shown in the embodiment of the present application, again, because of limited space, the method is not listed one by one.
S10223, traversing the ES database if the monitoring rule is met;
s10224, if the monitoring rule is not met, traversing the MYSQL database and repairing the ES database.
In the repair process of the ES database, the data switch is directly switched to the MYSQL database, during which the application platform server directly traverses the MYSQL database, and when the ES database is repaired, the data switch is switched to the ES database.
For example, if 1000 data are searched, the error rate of the ES database is found to have exceeded the monitoring rule during the 500 th data search, at this time, the data switch is directly switched to the MYSQL database, the ES database is repaired during the 700 th data search, and the system defaults to search the MYSQL database during both the 500 th data search and the 700 th data search.
According to the scheme shown in the embodiment of the application, the system defaults to traverse the MYSQL database in the abnormal operation process of the ES database, so that the database with frequent traversal errors of the application platform server is effectively avoided, the time waste in the data switch switching process is shortened, the waiting time of the application platform server is further shortened, and the utilization rate of resources such as system bandwidth, the database and the like is improved.
Example 5:
in order to enhance the connection between the application platform server and the development server, the embodiment of the application shows a connection method of an application platform, and specifically, please refer to fig. 6:
embodiment 5 has similar steps to the technical solution shown in embodiment 4, with the only difference that the step of repairing the ES database in step S10224 in the technical solution shown in embodiment 2 includes the following steps:
s102241, sending a repair request for repairing the ES database to a development server;
s102242 receives a repair completion instruction sent by the development server, and enables the ES database.
For example, the user queries "Audi A6, 2015 value-added tax" in real time through a processing system of data source switching, the system traverses the ES database by default, when the 5 th data is found, the application platform server feeds back a display interface "no record of the record", the data switch switches to the MYSQL database, and the application platform server continues to traverse the MYSQL database based on "Audi A6, 2015 value-added tax". When the 6 th data is detected, the data switch is switched to the ES database, the application platform server continuously traverses the ES database based on the 'Audi A6, 2015 value added tax', meanwhile, the application platform server monitors the error rate of the ES database in real time, for example, when the 203 th data is searched, the error rate of the ES database in 1 minute is 100 times, at the moment, the error rate of the ES database in unit time exceeds the monitoring rule, the application platform database sends mails to the development server, and the development server repairs the ES database or verifies the data in the ES database. When the repair of the ES database or the inspection of the data in the ES database is completed, the development server sends a repair completion instruction to the application platform server, and at this time, the application platform server switches the data switch to the ES database, and the application platform server continues to traverse the ES database.
According to the method, when the MYSQL runs abnormally, a repair request is directly sent to the development server, when the repair of the ES database is completed, a repair completion instruction is directly sent to the application platform server, the data switch is directly regulated to the ES database, the direct work close connection of each server is effectively ensured, the coordination time of personnel in each middle department is shortened, the waiting time of the application platform server is further shortened, and the utilization rate of resources such as system bandwidth, databases and the like is improved.
Example 6:
referring to fig. 7, the embodiment 6 has similar steps as the technical solutions shown in embodiments 2-5, and the only difference is that, in the technical solutions shown in embodiments 2-5, the construction process of the ES database specifically includes:
s104, traversing a data table in the MYSQL database;
s105, combining records in different data tables to obtain an ES record;
s106, storing the ES record, and generating an ES database.
Optionally, the data table comprises one or a combination of several of a city table, a vehicle source table, a time table, a financial scheme intermediate table and a financial scheme table.
Typically the data tables in a MYSQL database include: one or a combination of a plurality of vehicle source tables, time tables, financial scheme intermediate tables and financial scheme tables;
taking a vehicle source table, a city table, a financial scheme intermediate table and a financial scheme table as an example to introduce a combination process of the data table;
the traversal process using MYSWQL database is:
the vehicle source meter comprises a vehicle source 1, a vehicle source 2 and a vehicle source 3 … …;
firstly, traversing the original list of the vehicle, and determining the vehicle information corresponding to the query vehicle source 1.
City table 1 corresponding to vehicle source 1 includes city 1, city 2, city 3, city 4 … …;
traversing the city table 1, and determining vehicle information corresponding to the query city 2;
the financial scheme table 3 corresponding to city 2 includes: value-added tax, and, rate of value;
the calculation formulas corresponding to the intermediate tables of the financial schemes corresponding to the value-added tax of the traversing financial scheme table 3 are y respectively 1 ＝a 1 x+b 1 y 2 ＝a 2 x+b 2 ；
Then calculate the display content according to the inquiry requestThe volume "value-added tax of vehicle source 1 in city 2" is y 1 ＝a 1 x+b 1
The method shown in the embodiment of the application;
directly converging the vehicle source list, the city list, the financial scheme intermediate list and the financial scheme list into one list; the table includes:
value added tax of vehicle source 1 in city 2: y is 1 ＝a 1 x+b 1 ；
Value added tax of vehicle source 2 in city 2: y is 1 ＝a 1 x+b 1 ；
The value-added tax of the vehicle source 2 in the city 3: y is 2 ＝a 2 x+b 2 … …, etc. corresponds to one display content per series of query requests.
The records in the different data tables can be the combination of all the data tables or the combination of partial data tables;
the value-added tax of the vehicle source 2 in the city 3 corresponds to the combination of a vehicle source list, a city list, a financial scheme intermediate list and a financial scheme;
the number of vehicles in city 2 by vehicle source 1, corresponding vehicle source list and city list.
According to the method, the system defaults to traverse the ES database, the ES database integrates a plurality of data tables in MYSQL into one database, only one data table is required to be traversed in the traversing process, the conditions of time consumption and the like of multiple associated query in the query process are avoided, the waiting time of an application platform server is shortened, and the utilization rate of resources such as system bandwidth, the database and the like is improved.
Example 7:
referring to fig. 8, a third aspect of the embodiment of the present application shows a processing apparatus for switching a data source, where the apparatus includes:
an acquisition unit 21 for acquiring a query request;
a determining unit 22, configured to traverse a database based on the query request, and determine display content, where the database includes: ES database, and MYSQL database;
a sending unit 23, configured to send the display content to the user terminal.
Example 8:
referring to fig. 9, the embodiment 8 has a similar structure to the technical solution shown in embodiment 7, and the only difference is that, in the technical solution shown in embodiment 7, the determining unit 22 includes:
a first enabling unit 2211 for enabling the ES database;
a first judging unit 2212 for judging whether the ES database is operating normally
A first ES traversing unit 2213 configured to traverse the ES database if the ES database is operating normally;
the first MYSQL traversing unit 2214 is configured to traverse a MYSQL database if the ES database runs abnormally.
Example 9:
referring to fig. 10, embodiment 9 has a similar structure to the technical solution shown in embodiment 7, and the only difference is that, in the technical solution shown in embodiment 7, the determining unit 22 includes:
a monitoring unit 2221, configured to enable a monitoring service to monitor an operation condition of the ES database in real time;
a second judging unit 2222, configured to judge whether the operation status of the ES database meets a monitoring rule;
a second ES traversal unit 2223 configured to traverse the ES database if the monitoring rule is met;
and the second MYSQL traversing unit 2224 is configured to traverse the MYSQL database and repair the ES database if the monitoring rule is not met.
Example 10:
referring to fig. 11, the embodiment 10 has a similar structure to the technical solution shown in embodiment 9, and the only difference is that, in the technical solution shown in embodiment 9, the second MYSQL traversal unit 2224 includes:
a first transmitting unit 22241 configured to transmit a repair request for repairing the ES database to the development server;
the first receiving unit 22242 is configured to receive a repair completion instruction sent by the development server, and enable the ES database.
Example 11:
referring to fig. 12, the embodiment 11 has a similar structure to the technical solution shown in embodiment 7, and the only difference is that, in the technical solution shown in embodiment 7, the apparatus further includes:
a data table traversing unit 24, configured to traverse the data table in the MYSQL database;
a combining unit 25 for combining the records in the different data tables to obtain an ES record;
and a storage unit 26, configured to store the ES record and generate an ES database.
Example 12:
a fourth aspect of the embodiments of the present application shows a server, please refer to fig. 13, which includes:
one or more processors 41;
a memory 42 for storing one or more programs;
the one or more programs, when executed by the one or more processors 41, cause the one or more processors 41 to implement methods in accordance with embodiments of the present application.
As can be seen from the above technical solutions, the embodiments of the present application show a processing method, apparatus and system for switching data sources, where the method includes: acquiring a query request; traversing a database based on the query request, determining display content, the database comprising: ES database, and MYSQL database; and sending the display content to the user side. According to the method, the system defaults to traverse the ES database, the ES database integrates a plurality of data tables in MYSQL into one database, only one data table is required to be traversed in the traversing process, the conditions of time consumption and the like of multiple associated query in the query process are avoided, the waiting time of an application platform server is shortened, and the utilization rate of resources such as system bandwidth, the database and the like is improved.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.
It should be noted that, in a specific implementation, the present application further provides a computer storage medium, where the computer storage medium may store a program, where the program may include some or all of the steps in embodiments of the service providing method or the user registering method for user identity provided in the present application when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.
It will be apparent to those skilled in the art that the techniques in the embodiments of the present application may be implemented in software plus the necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application may be embodied in essence or what contributes to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments of the present application.
The same or similar parts between the various embodiments in this specification are referred to each other. In particular, for embodiments of the service providing device or the user registering device for user identities, the comparison is simple as it is substantially similar to the method embodiments, and the description in the method embodiments is sufficient for the relevant points.
The above-described embodiments of the present application are not intended to limit the scope of the present application.
It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present application is to be limited only by the appended claims, and other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
Claims (9)
1. A method for processing data source switching, the method comprising:
acquiring a query request;
based on the request for a query in question,
starting an ES database, and counting the error rate of the ES database in unit time in real time;
if the error rate of the ES database in unit time does not exceed the monitoring rule, traversing the ES database; the monitoring rule calculates according to the preset times of abnormality of the ES database in unit time, and when the ES database is abnormal in unit time, the number of search results provided by the ES database in unit time is smaller than that provided by the MYSQL database in unit time;
if the error rate of the ES database in unit time exceeds a monitoring rule, namely when the abnormal times of the ES database in unit time are larger than the preset times, switching the data switch to the MYSQL database, and repairing the ES database until the repair is completed, switching the data switch to the ES database;
determining display content;
and sending the display content to the user side.
2. The method of claim 1, wherein the step of repairing the ES database comprises:
sending a repair request for repairing the ES database to a development server;
and receiving a repair completion instruction sent by the development server, and starting the ES database.
3. The method according to claim 1 or 2, wherein the construction process of the ES database specifically comprises:
traversing a data table in the MYSQL database;
combining records in different data tables to obtain an ES record;
and storing the ES record to generate an ES database.
4. The method of claim 3, wherein the data table comprises one or a combination of a city table, a vehicle source table, a schedule, a financial scheme intermediate table, and a financial scheme table.
5. A processing apparatus for data source switching, the apparatus comprising:
the acquisition unit is used for acquiring the query request;
a determining unit, configured to enable an ES database based on the query request; counting the error rate of the ES database in unit time in real time; if the error rate of the ES database in unit time does not exceed the monitoring rule, namely when the number of times of abnormality occurrence of the ES database in unit time is smaller than the preset number of times, traversing the ES database; if the error rate of the ES database in unit time exceeds a monitoring rule, namely when the number of times of abnormality occurrence of the ES database in unit time is larger than the preset number of times, switching a data switch to a MYSQL database, and repairing the ES database until the repair is completed, switching the data switch to the ES database; determining display content; the monitoring rule calculates according to the preset times of abnormality of the ES database in unit time, and when the ES database is abnormal in unit time, the number of search results provided by the ES database in unit time is smaller than that provided by the MYSQL database in unit time;
and the sending unit is used for sending the display content to the user side.
6. The apparatus according to claim 5, wherein the determining unit includes:
a monitoring unit for enabling a monitoring service to monitor the operation status of the ES database in real time
The second judging unit is used for judging whether the running condition of the ES database accords with the monitoring rule;
the second ES traversing unit is used for traversing the ES database if the error rate of the ES database in unit time does not exceed the monitoring rule, namely when the number of times of abnormality occurrence of the ES database in unit time is smaller than the preset number of times;
and the second MYSQL traversing unit is used for traversing the MYSQL database to repair the ES database if the error rate of the ES database in unit time exceeds the monitoring rule, namely when the abnormal times of the ES database in unit time are larger than the preset times, and switching the data switch to the ES database until the repair is completed.
7. The apparatus of claim 6, wherein the second MYSQL traversal unit comprises:
the first sending unit is used for sending a repair request for repairing the ES database to the development server;
and the first receiving unit is used for receiving the repair completion instruction sent by the development server and starting the ES database.
8. The apparatus according to any one of claims 5-7, further comprising:
the data table traversing unit is used for traversing the data table in the MYSQL database;
a combining unit for combining records in different data tables to obtain an ES record;
and the storage unit is used for storing the ES records and generating an ES database.
9. A processing system for data source switching, the system comprising:
the system comprises an application platform server, a data storage server and a development server, wherein the data storage server is connected with the application platform server, and is arranged in the application platform server or independently, and the development server is connected with the application platform server and is connected with a terminal through the Internet;
the terminal is used for displaying information;
the application platform server being configured to implement the method of any one of claims 1-4;
the data storage server is used for storing related data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810306525.6A CN110362564B (en) | 2018-04-08 | 2018-04-08 | Data source switching processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810306525.6A CN110362564B (en) | 2018-04-08 | 2018-04-08 | Data source switching processing method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362564A CN110362564A (en) | 2019-10-22 |
| CN110362564B true CN110362564B (en) | 2024-01-23 |
Family
ID=68213392
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810306525.6A Active CN110362564B (en) | 2018-04-08 | 2018-04-08 | Data source switching processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362564B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888774B (en) * | 2019-11-07 | 2023-06-23 | 中盈优创资讯科技有限公司 | HBASE-based big data report processing method and device |
| CN111954038B (en) * | 2020-08-21 | 2021-12-21 | 湖南快乐阳光互动娱乐传媒有限公司 | Configurable search method and system based on ES efficient automatic synchronization |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106970943A (en) * | 2017-02-21 | 2017-07-21 | 南京中新赛克科技有限责任公司 | Fusion query method based on heterogeneous data source and distributed file system |
| CN107169083A (en) * | 2017-05-11 | 2017-09-15 | 聚龙融创科技有限公司 | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment |
| CN107729404A (en) * | 2017-09-25 | 2018-02-23 | 安徽畅通行交通信息服务有限公司 | A kind of region car statistics analysis and management system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11709833B2 (en) * | 2016-06-24 | 2023-07-25 | Dremio Corporation | Self-service data platform |
-
2018
- 2018-04-08 CN CN201810306525.6A patent/CN110362564B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106970943A (en) * | 2017-02-21 | 2017-07-21 | 南京中新赛克科技有限责任公司 | Fusion query method based on heterogeneous data source and distributed file system |
| CN107169083A (en) * | 2017-05-11 | 2017-09-15 | 聚龙融创科技有限公司 | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment |
| CN107729404A (en) * | 2017-09-25 | 2018-02-23 | 安徽畅通行交通信息服务有限公司 | A kind of region car statistics analysis and management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362564A (en) | 2019-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2635567C (en) | Method and system for providing enhanced matching from customer driven queries | |
| US7467102B2 (en) | Request type grid computing | |
| RU2557761C2 (en) | Bottom-up optimised search system and method | |
| US20110219054A1 (en) | Method and System for Processing Online Joint Guarantee | |
| US10503613B1 (en) | Efficient serving of resources during server unavailability | |
| CN110362564B (en) | Data source switching processing method, device and system | |
| US20130018917A1 (en) | Selective Storing of Mining Models for Enabling Interactive Data Mining | |
| CN116090808A (en) | RPA breakpoint reconstruction method and device, electronic equipment and medium | |
| CN111428095B (en) | Graph data quality verification method and graph data quality verification device | |
| CN111865673A (en) | Automatic fault management method, device and system | |
| CN108809673A (en) | A kind of method and device of fault alarm | |
| CN112329978A (en) | Intelligent public resource transaction subject performance monitoring and credit evaluation method | |
| CN109377391B (en) | Information tracking method, storage medium and server | |
| CN111144987A (en) | Abnormal shopping behavior limiting method, limiting assembly and shopping system | |
| TWI760216B (en) | Computer-implemented system and method for managing highly available distributed hybrid database | |
| CN112035760B (en) | Task allocation method and computer equipment | |
| JP2002352026A (en) | Method, system, and program for product repair | |
| CN107885780A (en) | A kind of performance data collection method performed for distributed query | |
| CN112597119A (en) | Method and device for generating processing log and storage medium | |
| CN112131100A (en) | Metadata-based front-end and back-end verification method and device | |
| CN116701410B (en) | Method and system for storing memory state data for data language of digital networking | |
| CN117313856B (en) | Reliability test planning system and method | |
| TWI760982B (en) | Computer-implemented systems and methods for generating demand forecasting data by performing wavelet transform for generating accurate purchase orders | |
| CN117829759A (en) | Innovative business model saas service system combining S2B2C line with offline | |
| TWI578173B (en) | Statistical e-commerce transaction data, e-commerce transaction data statistics system and application server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |