CN111352707A - Method, device and equipment for distinguishing splitting process based on OGG technology - Google Patents
Method, device and equipment for distinguishing splitting process based on OGG technology Download PDFInfo
- Publication number
- CN111352707A CN111352707A CN201811563104.8A CN201811563104A CN111352707A CN 111352707 A CN111352707 A CN 111352707A CN 201811563104 A CN201811563104 A CN 201811563104A CN 111352707 A CN111352707 A CN 111352707A
- Authority
- CN
- China
- Prior art keywords
- throughput
- ogg
- splitting
- target file
- preset condition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Probability & Statistics with Applications (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application provides a method, a device and equipment for judging a splitting process based on an OGG technology, which comprises the following steps: acquiring statistical information of a process, and determining the actual throughput of the process, wherein the statistical information comprises a target file; determining the pre-throughput according to the transfer log quantity and the conversion coefficient of the target file; judging whether the relation between the pre-throughput and the actual throughput meets a first preset condition or not; and splitting the process under the condition of meeting the preset condition. According to the method and the device, the statistical information is acquired, the execution and the process splitting are optimized, so that the process based on the OGG technology can timely cope with the data change caused by the change of the service system, the data timeliness and consistency are prevented from being influenced by the data synchronization delay caused by the sudden service change, and the labor cost is reduced while the data extraction and copying efficiency of the OGG process is improved.
Description
Technical Field
The invention belongs to the technical field of data transmission, and particularly relates to a method, a device, equipment and a computer storage medium for distinguishing a splitting process based on an OGG (one glass solution) technology.
Background
When using Oracle golden gate (hereinafter, OGG) as a database replication solution, extracting (extract) and replicating (replication) process components are often the first to appear performance bottlenecks as traffic load increases. To accommodate the increasing traffic load, process splitting is required to achieve parallel processing.
At present, an extract process component and a replay process component in an OGG process are independent from each other, and whether to perform process splitting or not mainly depends on maintenance experience and technical level accumulated by a database administrator (DBA) for a long time. However, with the change of the service, the DBA cannot timely determine the data change caused by the change of the service, which often means that a great processing delay is generated between an extract process component and a replicat process component, at this time, the configuration of the OGG process needs to be adjusted by the DBA intervention, and generally, the time from the change of the service to the final DBA intervention is often several hours or even several days, so that the problem cannot be solved in time, and the efficiency is low.
Therefore, a more reasonable process splitting method is needed to be provided, which can achieve the purpose of reducing the labor cost while improving the efficiency of OGG process data extraction and replication.
Disclosure of Invention
The embodiment of the application provides a method, a device and equipment for splitting an OGG (open log graph) process and a computer storage medium, and solves the problems that the process is not split timely due to service change, the consistency and the real-time performance of OGG process data extraction and copying are affected, and manual intervention is reduced.
In a first aspect, an embodiment of the present application provides a method for discriminating a split process based on an OGG technique, which may specifically include:
acquiring statistical information of a process, and determining the actual throughput of the process, wherein the statistical information comprises a target file;
determining the pre-throughput according to the transfer log quantity and the conversion coefficient of the target file;
judging whether the relation between the pre-throughput and the actual throughput meets a first preset condition or not;
and splitting the process under the condition of meeting the preset condition.
According to the method and the device, the statistical information is acquired, the execution and the process splitting are optimized, so that the process based on the OGG technology can timely cope with the data change caused by the change of the service system, the data timeliness and consistency are prevented from being influenced by the data synchronization delay caused by the sudden service change, and the labor cost is reduced while the data extraction and copying efficiency of the OGG process is improved.
In a possible implementation manner, the step of obtaining statistical information of the process and determining the actual throughput of the process may specifically include: counting the time t used by the process for processing the target file; determining relative byte addresses of two adjacent processing target files; and determining the actual throughput of the process according to the relative byte addresses and t of the two adjacent processing target files.
In another possible implementation, the step of determining the pre-throughput according to the transfer coefficient of the target file and the transfer log amount may specifically include: calling the log quantity in unit time; determining a conversion coefficient of a target file according to the ratio of the amount of the target file generated by the log in unit time to the amount of the log in unit time; and determining the pre-throughput according to the log amount in unit time and the conversion coefficient of the target file.
In another possible implementation, the step of determining whether the relationship between the pre-throughput and the actual throughput satisfies the first preset condition may specifically include: judging whether the pre-throughput is smaller than the actual throughput; if the pre-throughput is less than or equal to the actual throughput, the first preset condition is not met; if the pre-throughput is greater than the actual throughput, a first preset condition is met.
In another possible implementation, before the step of "splitting the process", the method may further include: suspending the running of the process; and judging whether the target file can be transmitted to the target side or not according to the time and the capacity of generating the target file.
In yet another possible implementation, the "process" is an extraction process based on the OGG technique.
In a second aspect, an embodiment of the present application provides a method for discriminating a split process based on an OGG technique, which may specifically include:
acquiring statistical information of a process;
judging whether the time for copying the statistical information by the process meets a second preset condition or not;
and splitting the process corresponding to the statistical information under the condition of meeting a second preset condition.
According to the method and the device, through the steps of obtaining statistical information, optimizing execution, splitting process and the like, the process based on the OGG technology can timely cope with data change caused by service system change, the data timeliness and consistency can be prevented from being influenced by data synchronization delay caused by sudden service change, and the labor cost is reduced while the data extraction and copying efficiency of the OGG process is improved.
In one possible embodiment, the "process" is a replication process based on OGG technology.
In yet another possible implementation, the "second preset condition" may include: at least one of disk IO information, index information, or latency.
In a third aspect, an embodiment of the present application provides an apparatus for determining a splitting process based on an OGG technique, which may specifically include:
the acquiring module is used for acquiring statistical information of the process and determining the actual throughput of the process, wherein the statistical information comprises a target file;
the processing module is used for determining the pre-throughput according to the transfer log quantity and the conversion coefficient of the target file;
the judging module is used for judging whether the relation between the pre-throughput and the actual throughput meets a first preset condition or not;
and the splitting module is used for splitting the extraction process under the condition of meeting the preset condition.
According to the method and the device, through the steps of obtaining statistical information, optimizing execution, splitting process and the like, the process based on the OGG technology can timely cope with data change caused by service system change, the data timeliness and consistency can be prevented from being influenced by data synchronization delay caused by sudden service change, and the labor cost is reduced while the data extraction and copying efficiency of the OGG process is improved.
In a possible implementation manner, the "obtaining module" may be specifically configured to count a time t taken by the process to process the target file; determining relative byte addresses of two adjacent processing target files; and determining the actual throughput of the process according to the relative byte addresses and t of the two adjacent processing target files.
In another possible implementation, the "processing module" may be specifically configured to retrieve the log amount per unit time; determining a conversion coefficient of a target file according to the ratio of the amount of the target file generated by the log in unit time to the amount of the log in unit time; and determining the pre-throughput according to the log amount in unit time and the conversion coefficient of the target file.
In another possible implementation, the "determining module" may be specifically configured to determine whether the pre-throughput is smaller than the actual throughput; if the pre-throughput is less than or equal to the actual throughput, the first preset condition is not met; if the pre-throughput is greater than the actual throughput, a first preset condition is met.
In yet another possible implementation, the "processing module" may be further configured to suspend the running of the process; and judging whether the target file can be transmitted to the target side or not according to the time and the capacity of generating the target file.
In yet another possible implementation, the "process" is an extraction process based on the OGG technique.
In a fourth aspect, an embodiment of the present application provides an apparatus for discriminating a split process based on an OGG technique, which may specifically include:
the acquisition module is used for acquiring the statistical information of the process;
the judging module is used for judging whether the time for copying the statistical information by the process meets a second preset condition or not;
and the splitting module is used for splitting the process corresponding to the statistical information under the condition of meeting a second preset condition.
According to the method and the device, through the steps of obtaining statistical information, optimizing execution, splitting process and the like, the process based on the OGG technology can timely cope with data change caused by service system change, the data timeliness and consistency can be prevented from being influenced by data synchronization delay caused by sudden service change, and the labor cost is reduced while the data extraction and copying efficiency of the OGG process is improved.
In one possible embodiment, the "process" is a replication process based on OGG technology.
In yet another possible implementation, the "second preset condition" may include: at least one of disk IO information, index information, or latency.
In a fifth aspect, an embodiment of the present application provides an apparatus for discriminating a split process based on an OGG technique, where the apparatus includes a processor and a memory storing computer program instructions;
a processor, when executing computer program instructions, implements a method of privacy protection as in any one of the first or second aspects.
In a sixth aspect, the present application provides a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the method for discriminating a split process based on the OGG technique as in any one of the first aspect or the second aspect.
In a seventh aspect, the present application provides a computer program product containing instructions, which when executed on a computer, causes the computer to execute the method for discriminating a splitting process based on the OGG technology as in any one of the first aspect or the second aspect.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for discriminating a split process based on an OGG technique according to an embodiment of the present application;
fig. 2 is a flowchart of another method for discriminating a split process based on the OGG technique according to an embodiment of the present application;
fig. 3 is a flowchart of a method for discriminating between a split extraction process and a copy process based on the OGG technique according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an apparatus for discriminating a split process based on an OGG technique according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of another apparatus for discriminating a split process based on the OGG technique according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a device for discriminating a split process based on the OGG technique according to an embodiment of the present application.
Detailed Description
Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
To facilitate understanding of what is referred to in this application, first, the OGG technique is explained, which may include: an extraction process and a copy process. The two processes respectively determine whether the process needs to be split or not through different judgment modes. The splitting mode can generally adopt a common splitting mode to split, wherein the splitting mode mainly comprises two modes: one is splitting a list in data by using a specific function (such as @ RANGE function) provided by an OGG technology, and uniformly distributing changes on the list to a plurality of processes by carrying out a hash (hash) algorithm on the data on the list so as to reduce the load of one process; another is to divide the list into several groups, and use multiple processes to process one of the groups of lists separately.
Next, a method of discriminating, splitting, extracting, and copying processes in the OGG-based technology is combined with fig. 1 and 2.
Fig. 1 is a flowchart of a method for discriminating a split process based on an OGG technique according to an embodiment of the present application.
As shown in fig. 1, the method may include S110-S140, and the specific steps are as follows:
s110: and acquiring statistical information of the process, and determining the actual throughput of the process, wherein the statistical information comprises a target file.
Specifically, the time t used by the process for processing the target file is counted; determining relative byte addresses of two adjacent processing target files; and determining the actual throughput of the process according to the relative byte addresses and t of the two adjacent processing target files.
For example, by counting the time (X minutes (minute), X is a positive number greater than 0) taken by a process to process a certain trail file, the detail information of throughput is calculated by subtracting Relative Byte Address (RBA) obtained by two info detail commands from the beginning, as shown in formula 1: and E is (B-a)/X, where a is the RBA of the first execution of the command, B is the RBA obtained by the second execution, and E is the finally calculated throughput (i.e., actual throughput).
S120: and determining the pre-throughput according to the transfer log amount and the conversion coefficient of the target file.
Specifically, the log amount in a unit time is called; determining a conversion coefficient of a target file according to the ratio of the amount of the target file generated by the log in unit time to the amount of the log in unit time; and determining the pre-throughput according to the log amount in unit time and the conversion coefficient of the target file.
Based on the example in S110, S120 is illustrated: firstly, the log quantity (total log quantity) and the redo switching frequency (time) are searched from a database through sql, and the redo log quantity per unit time is calculated, wherein T is the total log quantity/time;
the Trail file conversion coefficient k is the ratio of the amount of Trail files generated by analyzing the unit redo log and the amount of the unit redo log.
After the program deployment starts, sampling is periodically performed, and a conversion coefficient mean value k is calculated, as shown in formula 2:
wherein St is trail file quantity, Sr is redo log quantity, i is the number of times of each acquisition, and n is the total number of times.
Second, the throughput required for the process is calculated:
and calculating the required program throughput W-T k according to the log quantity and the trail file conversion coefficient k.
S130: and judging whether the relation between the pre-throughput and the actual throughput meets a first preset condition or not.
Specifically, whether the pre-throughput is smaller than the actual throughput is judged; if the pre-throughput is less than or equal to the actual throughput, the first preset condition is not met; if the pre-throughput is greater than the actual throughput, a first preset condition is met.
Based on S110-S120, S130 is illustrated, when the throughput of the process is smaller than the current actual throughput, that is, W < E, it can be determined that the process does not have delay and does not need to be split;
when the pre-throughput of the process is greater than the current actual throughput, namely W > E, it can be judged that the process is delayed and needs to be split.
In the case where the preset condition is satisfied, S140 is performed: and splitting the process.
Specifically, before S140, the method may further include: suspending the running of the process; and judging whether the target file can be transmitted to the target side or not according to the time and the capacity of generating the target file.
Based on S110-S130, steps that may also be included before S140 are illustrated. Firstly, before splitting, all processes are stopped, and whether the processes are completely transmitted to a target side can be judged according to the generation time of the trail file and whether the file size is increased or not on the target side, so that the automatic splitting device can split according to the following rules in order to avoid other faults caused after splitting. The program acquires the parameter configuration of the existing process, the corresponding trail file information and the checkpoint information of the process, automatically copies the key information to the new process, distinguishes the key information through the schema according to the statistical information collected in the step S110, and specifies the new process to be responsible for the corresponding schema or split a part of a table under a certain schema; putting the related tables in DML (datamanagement language)/DDL (data definition language) in the same process to ensure the integrity of data; the data in the same table is split into multiple processes using the @ RANGE function.
Secondly, before the process is started, the program automatically detects the checkpoint information of the new process and the old process, and the processes are started again after being confirmed to be consistent.
It should be noted that the above processes all refer to an extraction process based on the OGG technology.
Next, another method for identifying the splitting process based on the OGG technique will be described in detail with reference to fig. 2.
Fig. 2 is a flowchart of another method for discriminating a split process based on the OGG technique according to an embodiment of the present application.
As shown in fig. 2, the method may include S110-S130, and the specific steps are as follows:
it should be noted that the following processes all refer to the copy process in the OGG-based technology.
S110: and acquiring the statistical information of the process.
Specifically, for example, the program collects relevant statistical information through system and database instructions: first, statistics may be performed using disk IO. Second, the OGG target-side database typically disables automatic tasks (including statistics collection), and if there are a large number of operations in the table and the statistics are not updated in a timely manner, an inefficient execution plan may be used when the OGG replication process performs data reproduction because the statistics are too long. And inquiring the LAST _ ANALYZED field in the dba _ tables/user _ tables to obtain the statistical information collected for the LAST time. The copy process is connected to the database for data replay, so that SQL executed by the OGG process can be acquired by querying database sessions, whether low execution efficiency caused by index loss is judged by analyzing the SQL execution plan or not is further acquired by acquiring the SQL and the SQL execution plan, index information query is realized through SQL statements, and SQL Tuning Advisor or custom rules can be called by analyzing the execution plan.
S120: and judging whether the time for copying the statistical information by the process meets a second preset condition.
Specifically, the second preset condition may include: at least one of disk IO information, index information, or latency.
Based on S110, S120 is illustrated, and if the replication process delay time (latency) Checkpoint Lag is greater than a set threshold (e.g., 30 minutes), the following steps are continuously performed, otherwise, the replication process is not split.
Analyzing the reason of the abnormal performance of the copy process through the data collected in the S110 to decide whether to split the process: firstly, the method comprises the following steps: if the bottleneck is disk IO, the splitting process can increase IO overhead, thereby increasing delay and not splitting; otherwise, the reverse is carried out. Secondly, the method comprises the following steps: the fact that the statistical information is too old can cause a poor execution plan when the replicat process is copied, and the statistical information should be collected to generate a correct execution plan without splitting; otherwise, the reverse is carried out. Thirdly, index deletion often causes low replication efficiency of a replicat process, large delay is generated, a reasonable index is required to be created, and splitting is not carried out; otherwise, splitting the copy process.
And S130, splitting the process corresponding to the statistical information under the condition of meeting a second preset condition.
Specifically, when the preset condition in S120 is satisfied, the following steps may be performed: the replication process is automatically split, and the automatic script ensures that the same trail is not read by 3 replication processes simultaneously, so that disk contention is prevented.
Counting hot spot table information through a stats command, and dividing the hot spot tables into two groups; stopping the process needing to be split and recording check point information; adding a new replicat process, and respectively placing the hot spot list into the two replicat processes; modifying the checkpoint information of the newly added process to keep the checkpoint information consistent with the checkpoint information of the process before splitting; the replicat process is initiated.
It should be noted that, the contents shown in fig. 1 and fig. 2 are both methods for determining a split process based on the OGG technology, and specifically, as shown in fig. 3, the flowcharts of the methods for determining whether a process in the OGG technology is split or not are combined together and presented by using the examples shown in fig. 1 and fig. 2.
In conclusion, the scheme optimizes the execution and the process splitting by acquiring the statistical information, so that the process based on the OGG technology can timely cope with the data change caused by the change of the business system, the consistency and the real-time performance of the data extraction and the data copying of the OGG process are influenced by avoiding the dependence on DBA experience and untimely process splitting caused by the change of business, and the timeliness and the consistency of the data are prevented from being influenced by the data synchronization delay caused by sudden business change under the condition of reducing manual intervention, and the labor cost is reduced while the data extraction and copying efficiency of the OGG process is improved.
Fig. 4 is a schematic structural diagram of an apparatus for discriminating a splitting process based on an OGG technique according to an embodiment of the present application.
As shown in fig. 4, the apparatus 40 may specifically include:
the obtaining module 401 is configured to obtain statistical information of a process, and determine actual throughput of the process, where the statistical information includes a target file.
And the processing module 402 is configured to determine the pre-throughput according to the transfer log amount and the conversion coefficient of the target file.
A judging module 403, configured to judge whether a relationship between the pre-throughput and the actual throughput satisfies a first preset condition.
A splitting module 404, configured to split the extraction process when a preset condition is met.
The obtaining module 401 may be specifically configured to count time t used by the process to process the target file; determining relative byte addresses of two adjacent processing target files; and determining the actual throughput of the process according to the relative byte addresses and t of the two adjacent processing target files.
The processing module 402 may be specifically configured to retrieve the log amount in a unit time; determining a conversion coefficient of a target file according to the ratio of the amount of the target file generated by the log in unit time to the amount of the log in unit time; and determining the pre-throughput according to the log amount in unit time and the conversion coefficient of the target file.
The determining module 403 may specifically be configured to determine whether the pre-throughput is smaller than the actual throughput; if the pre-throughput is less than or equal to the actual throughput, the first preset condition is not met; if the pre-throughput is greater than the actual throughput, a first preset condition is met.
The processing module 402 may also be configured to suspend the execution of the process; and judging whether the target file can be transmitted to the target side or not according to the time and the capacity of generating the target file.
It should be noted that the above processes all refer to an extraction process based on the OGG technology.
Fig. 5 is a schematic structural diagram of another apparatus for discriminating a split process based on the OGG technique according to an embodiment of the present application
As shown in fig. 5, the apparatus 50 may comprise
An obtaining module 501, configured to obtain statistical information of a process.
The determining module 502 is configured to determine whether the time for copying the statistical information by the process meets a second preset condition.
The splitting module 503 is configured to split the process corresponding to the statistical information when a second preset condition is met.
Wherein, the above processes are all referred to as copy processes in the OGG technology. The second preset condition may include: at least one of disk IO information, index information, or latency.
It should be noted that, since the methods corresponding to fig. 1 and 2 are based on the OGG technology, only the provided methods are applicable to different processes, the apparatuses corresponding to fig. 1 and 2, fig. 3 and 4, in one possible embodiment, may be the same apparatuses, for example: the obtaining module 401 and the obtaining module 501 are the same module, the judging module 403 and the judging module 502 are the same module, and the splitting module 404 and the splitting module 503 are the same module. Of course, in another possible embodiment, the modules may be different modules.
Fig. 6 is a schematic structural diagram of a device for discriminating a split process based on the OGG technique according to an embodiment of the present application.
As shown in fig. 6, the apparatus may include a processor 601 and a memory 602 storing computer program instructions.
Specifically, the processor 601 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
The processor 601 reads and executes the computer program instructions stored in the memory 602 to implement any one of the above-described embodiments of the method for discriminating a split process based on the OGG technique.
The transceiver 603 is mainly used for implementing communication between at least two of the modules, devices, units, clients, or servers in the embodiment of the present invention.
In one example, the device may also include a bus 604. As shown in fig. 6, the processor 601, the memory 602, and the transceiver 603 are connected via a bus 604 and communicate with each other.
The bus 604 comprises hardware, software, or both that couple the above-described components to one another. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 603 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application. This device is applicable to all methods referred to in this application in fig. 1-4.
In addition, in combination with the method for determining the splitting process based on the OGG technology in the foregoing embodiments, the embodiments of the present application may provide a computer storage medium to implement. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any of the above-described embodiments of a method for discriminating a split process based on OGG techniques.
It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions or change the order between the steps after comprehending the spirit of the present application.
The functional blocks shown in the above structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.
It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.
As will be apparent to those skilled in the art, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.
Claims (13)
1. A method for distinguishing a splitting process based on an OGG technology is characterized by comprising the following steps:
acquiring statistical information of a process, and determining the actual throughput of the process, wherein the statistical information comprises a target file;
determining the pre-throughput according to the transfer log quantity and the conversion coefficient of the target file;
judging whether the relation between the pre-throughput and the actual throughput meets a first preset condition or not;
and splitting the process under the condition of meeting a preset condition.
2. The method of claim 1, wherein obtaining statistics of a process and determining an actual throughput of the process comprises:
counting the time t used by the process for processing the target file;
determining the relative byte addresses of the target file processed twice adjacently;
and determining the actual throughput of the process according to the relative byte addresses of the target file processed twice and the t.
3. The method of claim 1 or 2, wherein determining the pre-throughput based on the call log amount and the conversion factor of the target file comprises:
calling the log quantity in unit time;
determining a conversion coefficient of the target file according to the ratio of the amount of the target file generated by the log in unit time to the amount of the log in unit time;
and determining the pre-throughput according to the log amount in the unit time and the conversion coefficient of the target file.
4. The method of claim 1 or 3, wherein the determining whether the relation between the pre-throughput and the actual throughput satisfies a first preset condition comprises:
judging whether the pre-throughput is smaller than the actual throughput;
if the pre-throughput is less than or equal to the actual throughput, the first preset condition is not met;
and if the pre-throughput is greater than the actual throughput, the first preset condition is met.
5. The method of claim 1 or 4, further comprising, before the step of splitting the process:
suspending the running of the process;
and judging whether the target file can be transmitted to the target side or not according to the time and the capacity of the target file generation.
6. The method of claim 1, wherein the process is an extraction process in the OGG-based technique.
7. A method for distinguishing a splitting process based on an OGG technology is characterized by comprising the following steps:
acquiring statistical information of a process;
judging whether the time for copying the statistical information by the process meets a second preset condition or not;
and splitting the process corresponding to the statistical information under the condition of meeting the second preset condition.
8. The method of claim 7, wherein the process is a replication process in OGG-based technology.
9. The method according to claim 7 or 8, wherein the second preset condition comprises:
at least one of disk IO information, index information, or latency.
10. An apparatus for discriminating a split process based on an OGG technique, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring statistical information of a process and determining the actual throughput of the process, and the statistical information comprises a target file;
the processing module is used for determining the pre-throughput according to the transfer log quantity and the conversion coefficient of the target file;
the judging module is used for judging whether the relation between the pre-throughput and the actual throughput meets a first preset condition or not;
and the splitting module is used for splitting the extraction process under the condition of meeting a preset condition.
11. An apparatus for discriminating a split process based on an OGG technique, comprising:
the acquisition module is used for acquiring the statistical information of the process;
the judging module is used for judging whether the time for copying the statistical information by the process meets a second preset condition or not;
and the splitting module is used for splitting the process corresponding to the statistical information under the condition of meeting the second preset condition.
12. An apparatus for discriminating a split process based on an OGG technique, the apparatus comprising: a processor and a memory storing computer program instructions;
the processor, when executing the computer program instructions, implements a method of discriminating a split process based on OGG techniques as claimed in any of claims 1-9.
13. A computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method of discriminating a split procedure based on OGG techniques as claimed in any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811563104.8A CN111352707A (en) | 2018-12-20 | 2018-12-20 | Method, device and equipment for distinguishing splitting process based on OGG technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811563104.8A CN111352707A (en) | 2018-12-20 | 2018-12-20 | Method, device and equipment for distinguishing splitting process based on OGG technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111352707A true CN111352707A (en) | 2020-06-30 |
Family
ID=71193777
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811563104.8A Pending CN111352707A (en) | 2018-12-20 | 2018-12-20 | Method, device and equipment for distinguishing splitting process based on OGG technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111352707A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902666A (en) * | 2014-03-11 | 2014-07-02 | 国家电网公司 | Configuration file collecting and monitoring method based on OGG database replication |
| CN107231435A (en) * | 2017-07-13 | 2017-10-03 | 中体彩科技发展有限公司 | Data syn-chronization monitoring method and system |
-
2018
- 2018-12-20 CN CN201811563104.8A patent/CN111352707A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902666A (en) * | 2014-03-11 | 2014-07-02 | 国家电网公司 | Configuration file collecting and monitoring method based on OGG database replication |
| CN107231435A (en) * | 2017-07-13 | 2017-10-03 | 中体彩科技发展有限公司 | Data syn-chronization monitoring method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107818150B (en) | Log auditing method and device | |
| US10171620B2 (en) | Non-transitory computer-readable recording medium having stored therein control program, control apparatus and control method | |
| CN109522316B (en) | Log processing method, device, equipment and storage medium | |
| CN104714984A (en) | Database optimization method and device | |
| CN111008192B (en) | Data management method, device, equipment and medium | |
| CN106878038B (en) | Fault positioning method and device in communication network | |
| CN115865378B (en) | Streaming media real-time certification and verification method based on blockchain | |
| CN111224807B (en) | Distributed log processing method, device, equipment and computer storage medium | |
| CN108632078B (en) | Configuration acquisition method | |
| CN111352707A (en) | Method, device and equipment for distinguishing splitting process based on OGG technology | |
| CN110704223B (en) | Recovery system and method for single-node abnormity of database | |
| CN106599326B (en) | Recorded data duplication eliminating processing method and system under cloud architecture | |
| JP7025098B2 (en) | Abnormal traffic analyzer, abnormal traffic analysis method and abnormal traffic analysis program | |
| CN113792340B (en) | Method and device for auditing logical logs of database | |
| CN112994924B (en) | Online merging method for network service logs | |
| CN112269879B (en) | Method and equipment for analyzing middle station log based on k-means algorithm | |
| US8037077B2 (en) | Computer-readable recording medium, method, and apparatus for creating message patterns | |
| CN114238359A (en) | Report data management method, device, equipment, medium and computer program product | |
| CN114579662A (en) | Memory database synchronization method, system, equipment and storage medium | |
| CN111726249B (en) | Configuration file processing method and device of network equipment | |
| CN109992381A (en) | Process splits method of discrimination, device, equipment and medium | |
| CN111917826A (en) | PBFT consensus algorithm based on block chain intellectual property protection | |
| CN117689349B (en) | Office personnel-oriented enterprise data rapid splitting and sharing method | |
| CN110858206B (en) | Terminal feature extraction method based on hostname data | |
| CN104424447A (en) | Method and device for creating database firewalls |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |