CN105205144B - Method and system for data diagnosis optimization - Google Patents
Method and system for data diagnosis optimization Download PDFInfo
- Publication number
- CN105205144B CN105205144B CN201510599661.5A CN201510599661A CN105205144B CN 105205144 B CN105205144 B CN 105205144B CN 201510599661 A CN201510599661 A CN 201510599661A CN 105205144 B CN105205144 B CN 105205144B
- Authority
- CN
- China
- Prior art keywords
- information
- rule
- diagnostic
- analysis
- optimization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
This application discloses the method and systems optimized for data diagnosis.One specific embodiment of the method includes: building rule database, and Rule Information is stored, and the Rule Information includes diagnostic rule information and principle of optimality information；Gained actual log when being run according to the Rule Information analysis task, and generate actual log information；Judged according to the actual log information；Judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis；The diagnostic analysis result in conjunction with described in the diagnostic rule information acquisition or optimization analysis result.The embodiment, which realizes, reduces task execution cost, improves ease for use and validity.
Description
Technical field
This application involves field of computer technology, and in particular to Internet technical field more particularly to data diagnosis optimization
Method and system.
Background technique
Along with current internet fast development, data scale is increasing, requires in resource utilization higher and higher
In the case of, big data processing is carried out how to efficiently use existing resource, obtains mesh under the premise of shortest time and most resource-saving
Mark result has become the problem that each large enterprises face.In above-mentioned background, it is necessary first to accomplish to operation operation failure cause
It is quick positioning i.e. diagnose, allow developer can be developed with most fast speed processing data procedure operation, save exploitation
Personnel carry out the time cost of big data exploitation, improve the time scale that machine is effectively calculated.Secondly it was run in program
It needing to carry out machine resources in journey maximumlly using optimizing, this is not only the requirement quickly handled data,
It is effective control to cost is calculated.To give user ineffective for can completely for the method for existing big data operation diagnosis optimization
Feedback, user can reasonably be modified for existing handling situations.
Summary of the invention
The purpose of the application is to propose a kind of method and system for data diagnosis optimization, to solve background above skill
The technical issues of art part is mentioned.
In a first aspect, the above method includes: building rule this application provides a kind of method for data diagnosis optimization
Database stores Rule Information, and above-mentioned Rule Information includes diagnostic rule information and principle of optimality information；According to above-mentioned rule
Gained actual log when information analysis task run, and generate actual log information；Sentenced according to above-mentioned actual log information
It is disconnected；Judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis；In conjunction with above-mentioned diagnostic rule
Diagnostic analysis result described in information acquisition；Result is analyzed in conjunction with the above-mentioned optimization of above-mentioned principle of optimality information acquisition.
In some embodiments, above-mentioned by Rule Information storage includes: create-rule file, and above-mentioned rule file includes examining
Disconnected file and optimization file.
In some embodiments, above-mentioned diagnostic file includes several diagnostic rules, and above-mentioned diagnostic rule includes following several
Information: rule name, matched text, task type, Diagnosis of Primary because and advisory information.
In some embodiments, above-mentioned optimization file includes several principles of optimality, and the above-mentioned principle of optimality includes following several
Item information: rule name, task type, rule description, regular importance, calculation formula, threshold value and advisory information.
In some embodiments, above-mentioned actual log information includes Mission Success information or mission failure information.
In some embodiments, above-mentioned optimization analysis includes: extraction environment variable, analyzes log information and statistical information,
Extract independent variable from regular calculation formula according to above-mentioned environmental variance and above-mentioned log information, will calculate resulting value and threshold value into
Can row compares, be passed through according to comparison result judgement.
In some embodiments, above-mentioned diagnostic analysis includes: analysis task journal file, and judge incorrectly reason, judges to tie
Fruit is that exception then extracts exception information from journal file, and judging result is that mistake then believe from other journal files by extraction mistake
Breath.
Second aspect, this application provides a kind of systems for data diagnosis optimization, and above system includes: database list
Member is configured to building rule database, Rule Information is stored, and above-mentioned Rule Information includes that diagnostic rule information and optimization are advised
Then information；Log generation unit, gained actual log when being configured to be run according to above-mentioned Rule Information analysis task, and generate
Actual log information；Judging unit is configured to be judged according to above-mentioned actual log information；Analytical unit is configured to
Judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis；Feedback unit is configured to tie
It closes and states the above-mentioned diagnostic analysis of diagnostic rule information acquisition as a result, in conjunction with the above-mentioned optimization analysis knot of above-mentioned principle of optimality information acquisition
Fruit.
In some embodiments, above-mentioned by Rule Information storage includes: create-rule file, and above-mentioned rule file includes examining
Disconnected file and optimization file.
In some embodiments, above-mentioned diagnostic file includes several diagnostic rules, and above-mentioned diagnostic rule includes following several
Information: rule name, matched text, task type, Diagnosis of Primary because and advisory information.
In some embodiments, above-mentioned optimization file includes several principles of optimality, and the above-mentioned principle of optimality includes following several
Item information: rule name, task type, rule description, regular importance, calculation formula, threshold value and advisory information.
In some embodiments, above-mentioned actual log information includes Mission Success information or mission failure information.
In some embodiments, above-mentioned optimization analysis includes: extraction environment variable, analyzes log information and statistical information,
Extract independent variable from regular calculation formula according to above-mentioned environmental variance and above-mentioned log information, will calculate resulting value and threshold value into
Can row compares, be passed through according to comparison result judgement.
In some embodiments, above-mentioned diagnostic analysis includes: analysis task journal file, and judge incorrectly reason, judges to tie
Fruit is that exception then extracts exception information from journal file, and judging result is that mistake then believe from other journal files by extraction mistake
Breath.
Method and system provided by the present application for data diagnosis optimization, first building rule database, rule is believed
Breath storage, above-mentioned Rule Information include diagnostic rule information and principle of optimality information；Further according to above-mentioned Rule Information analysis task
Gained actual log when operation, and generate actual log information；Then judged according to above-mentioned actual log information；Judgement knot
Fruit is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis；Finally combine above-mentioned diagnostic rule information
Obtain above-mentioned diagnostic analysis result or above-mentioned optimization analysis result.To effectively reduce task execution cost, improve easily
With property and validity.
Detailed description of the invention
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon:
Fig. 1 is that this application can be applied to exemplary system architecture figures therein；
Fig. 2 is the flow chart according to one embodiment of the method for data diagnosis optimization of the application；
Fig. 3 is the flow chart according to another embodiment of the method for data diagnosis optimization of the application；
Fig. 4 is the flow chart according to another embodiment of the method for data diagnosis optimization of the application；
Fig. 5 is the structural schematic diagram according to one embodiment for data diagnosis optimization system of the application；
Fig. 6 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present application
Figure.
Specific embodiment
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to
Convenient for description, part relevant to related invention is illustrated only in attached drawing.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 is shown can be using the implementation of the system for the method and data diagnosis optimization that the data diagnosis of the application optimizes
The exemplary system architecture 100 of example.
As shown in Figure 1, system architecture 100 may include terminal device 101,102,103, network 104 and server 105.
Network 104 between terminal device 101,102,103 and server 105 to provide the medium of communication link.Network 104 can be with
Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be used terminal device 101,102,103 and be interacted by network 104 with server 105, to receive or send out
Send message etc..Various telecommunication customer end applications can be installed, such as web browser is answered on terminal device 101,102,103
With, shopping class application, searching class application, instant messaging tools, mailbox client, social platform software etc..
Terminal device 101,102,103 can be the various electronic equipments with display screen and supported web page browsing, packet
Include but be not limited to smart phone, tablet computer, E-book reader, MP3 player (Moving Picture Experts
Group Audio Layer III, dynamic image expert's compression standard audio level 3), MP4 (Moving Picture
Experts Group Audio Layer IV, dynamic image expert's compression standard audio level 4) it is player, on knee portable
Computer and desktop computer etc..
Server 105 can be to provide the server of various services, such as to data on terminal device 101,102,103 point
Analysis processing provides the processing server supported.Processing server can carry out the data received the processing such as analyzing, and will place
Reason result (such as feedback information) is sent to terminal device.
It should be noted that the method for the optimization of data diagnosis provided by the embodiment of the present application is generally held by server 105
Row, correspondingly, data diagnosis optimization device are generally positioned in server 105.
It should be understood that the number of terminal device, network and server in Fig. 1 is only schematical.According to realization need
It wants, can have any number of terminal device, network and server.
With continued reference to Fig. 2, the process of one embodiment of the method according to the optimization of the data diagnosis of the application is shown
200.The data diagnosis optimization method, comprising the following steps:
Step 201, rule database is constructed, Rule Information is stored, Rule Information includes diagnostic rule information and optimization
Rule Information.
In the present embodiment, electronic equipment (such as the service shown in FIG. 1 of data diagnosis optimization method operation thereon
Device).
In general, user browses webpage using the web browser installed in terminal, at this moment, user can be by directly defeated
Enter the chain in the webpage presented in network address or webpage clicking browser to fetch to web page server initiation web page browsing request.?
In the present embodiment, above-mentioned webpage may include html format, xhtml format, asp format, php format, jsp format, shtml lattice
Formula, nsp format, the webpage of xml format or other futures are by the webpage of the format of exploitation (as long as the web page files of this format
Can be opened with browser and browse it includes the contents such as picture, animation, text).
Step 202, gained actual log when being run according to Rule Information analysis task, and generate actual log information.
In the present embodiment, above two rule is loaded into memory when diagnosing optimization system starting, with the side of tree
Formula is organized, convenient tracking and lookup to rule.After job run, diagnosis optimization system starts to collect each of operation
Kind log and environmental variance.The success or not information of the available operation in the log of operation holds the operation of failure
Row diagnostic analysis, and analysis is optimized for successful job execution.
It should be noted that the various methods of above-mentioned actual log analysis mode are to study and apply known extensively at present
Technology, details are not described herein.
Step 203, judged according to actual log information.
In the present embodiment, the suggestion for diagnosing and optimizing is all based on various aspects situation when job run, analyzes user
The log information of generation retrieves the text in diagnostic rule for the operation of failure in log, and scan abnormalities stack directly positions
Feedback information is constructed according to diagnostic rule and job logging after to the code of user.
Step 204, judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis.
In the present embodiment, the exception and mistake when can feed back job run out effective and reasonablely are diagnosed, use is navigated to
Erroneous point in the code of family, user can directly modify program code according to diagnostic feedback, submit correct program again；Optimization
It is then for successful operation, feedback information gives user's suggestion of rationalization from system parameter to job parameter, makees user
Industry can be more reasonably utilized machine resources when next time runs again.
Step 205, it is analyzed in conjunction with diagnostic rule information acquisition diagnostic analysis as a result, optimizing in conjunction with principle of optimality information acquisition
As a result.
In embodiment, all feedback informations give user after summarizing, and give the suggestion of user's diagnosis or optimization.
With continued reference to Fig. 3, the process of another embodiment of the method according to the optimization of the data diagnosis of the application is shown
300.The data diagnosis optimization method, comprising the following steps:
Step 301, judge homework type.
In the present embodiment, the type of operation has been divided into hadoop operation and spark operation.The mesh that operation is diagnosed
Be in order to obtain operation failure the reason of, diagnostic mode can be directly targeted to the code level of user, be fed directly to use
Family.
Step 302, the job history log jhist file of Hadoop is analyzed.
Step 303, judge operation error the reason is that due to exception or mistake.
Step 304, judging result is the abnormal abnormal stack that operation is then directly obtained from jhist file.
Step 305, judging result is that mistake is then obtained from other log informations.
In the present embodiment, very detailed there is no showing in jhist file in the case of judging result is mistake
Error message, need to obtain from other log informations.Such as: scanning stderr, syslog and stdout file, line by line
Matching rule and sentence therein obtain the wrong stack of operation.It is filtered after obtaining wrong stack, filters out frame and language
It says the calling of itself, retains personal code work information, can thus be directly targeted at the code error of user.The diagnosis of Spark
It is substantially identical as the diagnosis of hadoop, but it does not generate jhist file, thus directly from stderr, syslog and
Stdout file is started with, and analysis is scanned.
Step 306, wrong abnormal stack is filtered.
In the present embodiment, it is filtered after obtaining wrong abnormal stack, filters out the calling of frame and language itself,
Retain personal code work information, can thus be directly targeted at the code error of user.
It should be noted that the diagnosis of Spark is substantially identical as the diagnosis of hadoop, but it does not generate jhist
File is scanned analysis so we directly start with from stderr, syslog and stdout file.
With further reference to Fig. 4, it illustrates the flow charts of another embodiment of the method optimized for data diagnosis
400.The process 400 of the data diagnosis optimization method, comprising the following steps:
Step 401, Experience norms design rule, the configuration of evaluating operation and the reasonability of parameter are based on.
In the present embodiment, optimization is analyzed, as hadoop operation with the process of spark operation is, it is base
In Experience norms design rule, the configuration of evaluating operation and the reasonability of parameter, in order to give the suggestion of user optimization.
Step 402, independent variable is analyzed.
In the present embodiment, operation is tied by analysis hadoop or spark job logging in conjunction with predefined rule
Diagnosis and Optimizing Suggestions are given in the operation of beam.When the job logging content of Hadoop and spark usually all includes job execution
Various information reacts the execution details of operation, in conjunction with operation feelings of the available operation of runtime environment in system of operation
Condition and all kinds of metric datas.Can be analyzed according to predefined rule provide operation procedure need the place modified or
The configuration optimization direction of hadoop or spark system parameter.
Step 403, the independent variable of system is obtained.
In the present embodiment, the principle of optimality is traversed, according to the formula in independent variable computation rule, determines acquisition system oneself
Variable.
Step 404, log-file information is obtained.
In the present embodiment, it prints the Various types of data for being able to reflect job run situation as far as possible in log, wraps
Include the frequency of the Memory recycle of Java Virtual Machine, time that Memory recycle expends etc..
Step 405, environmental variance is obtained.
In the present embodiment, in optimization process, it is necessary first to obtain the environmental variance of system, such as CPU number, memory
The Various types of data statistic of size etc. and program.
Step 406, all kinds of log information and other statistics, the independent variable that will occur in regular calculation formula are analyzed
All extract.
In the present embodiment, during job run, need maximumlly to utilize the performance of machine.And work how is turned up
Machine utilization rate common practice when industry is run is that job parameter is adjusted according to priori knowledge coarseness, such reliability
It is not high.
Step 407, according to the formula in independent variable computation rule obtained in the previous step, by obtained value with it is preset
Threshold value compares, and judges whether the rule passes through.
In the present embodiment, the case where being run on a specific machine according to operation itself, reasonable calculating are every excellent
The reasonability for changing criterion, gives user and timely optimizes feedback.
Step 408, to fail by rule, program return fail by the reason of and suggestion later.
As can be seen that diagnosis optimisation technique mainly includes two parts content from Fig. 3 and Fig. 4: building rule base will diagnose
It is saved hereof with the rule of optimization, is loaded into Installed System Memory in analysis；The actual log that analysis job run generates obtains
To actual log information.Diagnosis or the optimum results of operation are obtained by comparing with calculating log information and Rule Information later.
It should be noted that Fig. 3 and Fig. 4 respectively illustrate the side of rule-based hadoop, spark operation diagnosis optimization
Method from the environment of job execution and log the case where analysis job run, and then gives the suggestion of user optimization or diagnosis.
The case where various aspects, show that diagnosis can be fed back effective and reasonablely out when the suggestion of diagnosis and optimization is all based on job run
Exception and mistake when job run, navigate to the erroneous point in personal code work, and user can directly modify according to diagnostic feedback
Program code submits correct program again；Optimization is then for successful operation, and feedback information is joined from system parameter to operation
Number gives user's suggestion of rationalization, and user job is enable to be more reasonably utilized machine resources when next time runs again.
It further illustrates, it is necessary first to formulate the rule of operation diagnosis and optimization, the rule of diagnosis is based on text matches
, error reason and suggestion are defined in rule, and distinguish different homework types；The rule of optimization is based on threshold value ratio
Compared with, definition calculation formula and threshold value, and the suggestion of optimization.In diagnosis optimization process, the ring during job execution is collected
The log information that border variable and hadoop, spark are generated, and certain filtering is done, exclude the operation analyzed and other
Invalid job information (such as successful oozie launcher operation).The log information that user generates is analyzed, for the work of failure
Industry retrieves the text in diagnostic rule in log, and scan abnormalities stack is directly targeted to after the code of user according to diagnostic rule
Feedback information is constructed with job logging；For successful operation, the data statistics amount in environmental variance and log is parsed, according to excellent
Change the calculation formula in rule, calculate the score of the rule, with its threshold value comparison, returns to feedback information.All feedback letters
Breath gives user after summarizing, and gives the suggestion of user's diagnosis or optimization.
With further reference to Fig. 5, as the realization to method shown in above-mentioned each figure, this application provides excellent for data diagnosis
The structural schematic diagram of one embodiment of change system；The system embodiment is corresponding with embodiment of the method shown in Fig. 2, the system
It specifically can be applied in various electronic equipments.
As shown in figure 5, data diagnosis optimization system 500 described in the present embodiment includes: log crawler 501, log point
Parser 502, operation diagnostor 503, optimization of job device 504, rule parsing device 505 and ultramagnifier 506.It is log crawler first
501 crawl the relevant log of operation of target job cluster and environmental variance, these logs and variable are by log crawler 501
After crawling in deposit diagnosis optimization system.Then log and the variable of group operation are analyzed by log analyzer 502 again, simultaneously
Carry out the statistics of the Various types of data statistic of user job program.Before these logs and statistic are loaded into core system,
System can be loaded into the rule of diagnosis and optimization, and the rule of tree construction is formed after the parsing of rule parsing device 505, in order to
Accelerate rule analysis and matched speed.Judge that operation is success or failure in log analyzer 502, failure operation enters
Operation diagnostor 503 is handled, and is inputted as diagnostic rule and various log variables；Success operation enters optimization of job device
504 processing, input as the principle of optimality and various log variables.After the diagnosis described by front and Optimizing Flow, by feeding back
506 synthetic job of device diagnosis and optimization as a result, feeding back to user.
It will be understood by those skilled in the art that above-mentioned diagnosis optimization system further includes some other known features, such as locate
Device, memory etc. are managed, in order to unnecessarily obscure embodiment of the disclosure, these well known structures are not shown in Figure 5.
Below with reference to Fig. 6, it illustrates the calculating of the terminal device or server that are suitable for being used to realize the embodiment of the present application
The structural schematic diagram of machine system 600.
As shown in fig. 6, computer system 600 includes central processing unit (CPU) 601, it can be read-only according to being stored in
Program in memory (ROM) 602 or be loaded into the program in random access storage device (RAM) 603 from storage section 608 and
Execute various movements appropriate and processing.In RAM 603, also it is stored with system 600 and operates required various programs and data.
CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always
Line 604.
I/O interface 605 is connected to lower component: the importation 606 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 608 including hard disk etc.；
And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net executes communication process.Driver 610 is also connected to I/O interface 605 as needed.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 610, in order to read from thereon
Computer program be mounted into storage section 608 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be tangibly embodied in machine readable
Computer program on medium, the computer program include the program code for method shown in execution flow chart.At this
In the embodiment of sample, which can be downloaded and installed from network by communications portion 609, and/or from removable
Medium 611 is unloaded to be mounted.
Flow chart and block diagram in attached drawing are illustrated according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of the module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart, Ke Yiyong
The dedicated hardware based system of defined functions or operations is executed to realize, or can be referred to specialized hardware and computer
The combination of order is realized.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described unit also can be set in the processor, for example, can be described as: a kind of processor packet
Include receiving unit, resolution unit, information extracting unit and generation unit.Wherein, the title of these units is under certain conditions simultaneously
The restriction to the unit itself is not constituted, for example, receiving unit is also described as " receiving the web page browsing request of user
Unit ".
As on the other hand, present invention also provides a kind of nonvolatile computer storage media, the non-volatile calculating
Machine storage medium can be nonvolatile computer storage media included in device described in above-described embodiment；It is also possible to
Individualism, without the nonvolatile computer storage media in supplying terminal.Above-mentioned nonvolatile computer storage media is deposited
One or more program is contained, when one or more of programs are executed by an equipment, so that the equipment: building
Rule database stores Rule Information, and the Rule Information includes diagnostic rule information and principle of optimality information；According to described
Gained actual log when Rule Information analysis task is run, and generate actual log information；According to the actual log information into
Row judgement；Judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis；In conjunction with the diagnosis
Rule Information obtains the diagnostic analysis result or result is analyzed in the optimization.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.Those skilled in the art
Member is it should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic
Scheme, while should also cover in the case where not departing from the inventive concept, it is carried out by above-mentioned technical characteristic or its equivalent feature
Any combination and the other technical solutions formed.Such as features described above has similar function with (but being not limited to) disclosed herein
Can technical characteristic replaced mutually and the technical solution that is formed.
Claims (8)
1. a kind of method for data diagnosis optimization characterized by comprising
Rule database is constructed, Rule Information is stored, the Rule Information includes diagnostic rule information and principle of optimality information,
Wherein, described by Rule Information storage includes: create-rule file, and the rule file includes diagnostic file and optimization file；
Gained actual log when being run according to the Rule Information analysis task, and generate actual log information；
Judged according to the actual log information；
Judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnostic analysis；Wherein, the optimization point
Analysis includes: extraction environment variable, analyzes log information and statistical information, according to the environmental variance and the log information from rule
Independent variable then is extracted in calculation formula, resulting value will be calculated and be compared with threshold value, can be passed through according to comparison result judgement；Institute
Stating diagnostic analysis includes: analysis task journal file, and judge incorrectly reason, and judging result is that exception is then extracted from journal file
Exception information, judging result are that mistake then extracts error message from other journal files；
The diagnostic analysis result in conjunction with described in the diagnostic rule information acquisition；
The optimization analysis result in conjunction with described in the principle of optimality information acquisition.
2. described the method according to claim 1, wherein the diagnostic file includes several diagnostic rules
Diagnostic rule includes following items information: rule name, matched text, task type, Diagnosis of Primary because and advisory information.
3. described the method according to claim 1, wherein the optimization file includes several principles of optimality
The principle of optimality includes following items information: rule name, task type, rule description, regular importance, calculation formula, threshold value
And advisory information.
4. method described in one of -3 according to claim 1, which is characterized in that the actual log information includes Mission Success letter
Breath or mission failure information.
5. a kind of system for data diagnosis optimization characterized by comprising
Database Unit is configured to building rule database, Rule Information is stored, and the Rule Information includes diagnostic rule
Information and principle of optimality information, wherein described by Rule Information storage includes: create-rule file, and the rule file includes
Diagnostic file and optimization file；
Log generation unit, gained actual log when being configured to be run according to the Rule Information analysis task, and generate reality
Border log information；
Judging unit is configured to be judged according to the actual log information；
Analytical unit, being configured to judging result is successfully then to optimize analysis, and judging result is that failure then carries out diagnosis point
Analysis；Wherein, the optimization analysis includes: extraction environment variable, log information and statistical information is analyzed, according to the environmental variance
Independent variable is extracted from regular calculation formula with the log information, resulting value will be calculated and be compared with threshold value, according to comparing
As a result can judgement pass through；The diagnostic analysis includes: analysis task journal file, and judge incorrectly reason, and judging result is different
Exception information is extracted in Chang Zecong journal file, judging result is that mistake then extracts error message from other journal files；
Feedback unit is configured to the diagnostic analysis in conjunction with described in the diagnostic rule information acquisition as a result, advising in conjunction with the optimization
Then optimization analysis result described in information acquisition.
6. system according to claim 5, which is characterized in that the diagnostic file includes several diagnostic rules, described
Diagnostic rule includes following items information: rule name, matched text, task type, Diagnosis of Primary because and advisory information.
7. system according to claim 5, which is characterized in that the optimization file includes several principles of optimality, described
The principle of optimality includes following items information: rule name, task type, rule description, regular importance, calculation formula, threshold value
And advisory information.
8. the system according to one of claim 5-7, which is characterized in that the actual log information includes Mission Success letter
Breath or mission failure information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510599661.5A CN105205144B (en) | 2015-09-18 | 2015-09-18 | Method and system for data diagnosis optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510599661.5A CN105205144B (en) | 2015-09-18 | 2015-09-18 | Method and system for data diagnosis optimization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105205144A CN105205144A (en) | 2015-12-30 |
| CN105205144B true CN105205144B (en) | 2019-03-26 |
Family
ID=54952827
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510599661.5A Active CN105205144B (en) | 2015-09-18 | 2015-09-18 | Method and system for data diagnosis optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105205144B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106227727A (en) * | 2016-06-30 | 2016-12-14 | 乐视控股（北京）有限公司 | Daily record update method, device and the system of a kind of distributed system |
| CN106446525B (en) * | 2016-08-31 | 2019-05-10 | 杭州逸曜信息技术有限公司 | The processing method of medication Rule Information similarity |
| KR20180076058A (en) * | 2016-12-27 | 2018-07-05 | 에이치피프린팅코리아 주식회사 | Image forming apparatus and method for optimizing operating environment of virtual machine executed in image forming apparatus |
| CN108804215B (en) * | 2018-06-12 | 2021-06-22 | 北京奇艺世纪科技有限公司 | Task processing method and device and electronic equipment |
| CN108962372A (en) * | 2018-06-28 | 2018-12-07 | 上海联影医疗科技有限公司 | A method of processing image reconstruction failure |
| CN109491880A (en) * | 2018-11-01 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of method and apparatus optimizing log analysis |
| CN110765237B (en) * | 2019-10-10 | 2023-09-26 | 腾讯科技（深圳）有限公司 | Document processing method and device, storage medium and electronic equipment |
| CN111078665B (en) * | 2019-12-09 | 2023-04-11 | 苏宁金融科技(南京)有限公司 | Method, apparatus, computer device, and medium for determining optimization information for HIVE task |
| CN111259318A (en) * | 2020-01-19 | 2020-06-09 | 平安科技（深圳）有限公司 | Intelligent data optimization method and device and computer readable storage medium |
| CN113064807A (en) * | 2021-04-22 | 2021-07-02 | 中国工商银行股份有限公司 | Log diagnosis method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101765084A (en) * | 2008-12-24 | 2010-06-30 | 中国移动通信集团天津有限公司 | Processing method for core network data and device thereof |
| CN104077328A (en) * | 2013-03-29 | 2014-10-01 | 百度在线网络技术（北京）有限公司 | Operation diagnosis method and device for MapReduce distributed system |
| CN104503914A (en) * | 2014-12-29 | 2015-04-08 | 浪潮(北京)电子信息产业有限公司 | Information diagnosis method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040117387A1 (en) * | 2000-02-25 | 2004-06-17 | Vincent Civetta | Database sizing and diagnostic utility |
-
2015
- 2015-09-18 CN CN201510599661.5A patent/CN105205144B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101765084A (en) * | 2008-12-24 | 2010-06-30 | 中国移动通信集团天津有限公司 | Processing method for core network data and device thereof |
| CN104077328A (en) * | 2013-03-29 | 2014-10-01 | 百度在线网络技术（北京）有限公司 | Operation diagnosis method and device for MapReduce distributed system |
| CN104503914A (en) * | 2014-12-29 | 2015-04-08 | 浪潮(北京)电子信息产业有限公司 | Information diagnosis method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105205144A (en) | 2015-12-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105205144B (en) | Method and system for data diagnosis optimization | |
| US20210158711A1 (en) | Guiding creation of an electronic survey | |
| US10628129B2 (en) | Intellectual automated security, performance and code generation framework | |
| US10873596B1 (en) | Cybersecurity alert, assessment, and remediation engine | |
| US20170161061A1 (en) | Custom code migration suggestion system based on actual change references | |
| US8712990B2 (en) | Methods and systems for providing a business repository | |
| CN108764808A (en) | Data Analysis Services system and its on-time model dispositions method | |
| US20080262860A1 (en) | System and Method for Supporting Software | |
| US10169037B2 (en) | Identifying equivalent JavaScript events | |
| CN110659349A (en) | Log query method, device, equipment and computer readable storage medium | |
| US11809810B1 (en) | Dynamically displaying portions of a hierarchically organized data file in a web application | |
| US7257735B2 (en) | Identifying solutions to computer problems in client/server system | |
| US11836202B2 (en) | Methods and systems for dynamic search listing ranking of software components | |
| US7418628B2 (en) | Systems and methods for identifying solutions to computer problems using contexts and versions | |
| US8341212B2 (en) | Service description refinement based on actual service use | |
| WO2013054248A1 (en) | Generating a predictive data structure | |
| US20040153881A1 (en) | Identifying solutions to computer problems in main system by service system in distributed system landscape | |
| US7403933B2 (en) | Identifying solutions to computer problems in main system by service system | |
| US11960492B2 (en) | Methods and systems for display of search item scores and related information for easier search result selection | |
| US20220365812A1 (en) | Method and system for sustainability measurement | |
| US20220269744A1 (en) | Methods and systems for enabling dynamic filters for software search optimization | |
| US9542171B2 (en) | Managing an application modification process | |
| US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
| WO2018147954A1 (en) | Automatic generation of interactive web page content | |
| US20180067837A1 (en) | Framework for detecting source code anomalies |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |