CN109460432A - A kind of data processing method and system - Google Patents
A kind of data processing method and system Download PDFInfo
- Publication number
- CN109460432A CN109460432A CN201811354476.XA CN201811354476A CN109460432A CN 109460432 A CN109460432 A CN 109460432A CN 201811354476 A CN201811354476 A CN 201811354476A CN 109460432 A CN109460432 A CN 109460432A
- Authority
- CN
- China
- Prior art keywords
- target object
- monitored
- abnormal
- business datum
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the present application provides a kind of data processing method and system, it is related to Internet technical field, this method comprises: acquiring the business datum of multiple objects to be monitored according to preset condition, the business datum of target object is filtered out from multiple objects to be monitored according to the business datum of multiple objects to be monitored.Whether the business datum for then analyzing each target object matches with preset rules；If so, target object is determined as normal target object, target object is otherwise determined as abnormal target object, receives the processing information of abnormal target object later, and verifies the processing status of abnormal target object.Due to first filtering out target object, then analyze whether each target object with preset condition matching determines whether target object is abnormal, it is more efficient and more targetedly for manual analysis.In the processing information for receiving abnormal target object, the processing status of abnormal target object is verified, realizes the supervision to abnormality processing.
Description
Technical field
The present embodiments relate to Internet technical field more particularly to a kind of data processing methods and system.
Background technique
With the development of internet technology, business datum amount gradually increases.In order to guarantee the normal development of business, advise in advance
Business risk is kept away, needs to be monitored business.In the prior art, main to pass through the manually extraction section business from business datum
Data are analyzed, and determine abnormal traffic.And the business datum amount of internet development bring big data era is huge, relies on people
The efficiency that work point analyses business datum is lower.
Summary of the invention
Manually a large amount of business datum is analyzed due to relying in the prior art, causes data-handling efficiency is low to ask
Topic, the embodiment of the present application provides a kind of data processing method and system, the system include:
On the one hand, the embodiment of the present application provides a kind of data processing method, this method comprises:
The business datum of multiple objects to be monitored is acquired according to preset condition；
Target object is filtered out from the multiple object to be monitored according to the business datum of the multiple object to be monitored
Business datum；
Whether the business datum for analyzing each target object matches with preset rules；
If so, the target object is determined as normal target object；
Otherwise the target object is determined as abnormal target object, receives the processing information of the abnormal target object,
And verify the processing status of the abnormal target object.
On the other hand, the embodiment of the present application provides a kind of data processing system, which includes:
Collector, monitoring module, anomaly analysis module, exception processing module；
The collector, for acquiring the business datum of multiple objects to be monitored according to preset condition, by it is the multiple to
The business datum of monitored object is sent to the monitoring module；
The monitoring module, for according to the business datum of the multiple object to be monitored from the multiple object to be monitored
In filter out the business datum of target object；
The anomaly analysis module for obtaining the business datum of target object, and analyzes the industry of each target object
Whether business data match with preset rules；If so, the target object is determined as normal target object；Otherwise by the mesh
Mark object is determined as abnormal target object；
The exception processing module for receiving the processing information of the abnormal target object, and verifies the abnormal mesh
Mark the processing status of object.
In another aspect, the embodiment of the present application provides a kind of data processing equipment, including at least one processor, Yi Jizhi
A few memory, wherein the memory is stored with computer program, when described program is executed by the processor, makes
Obtain the step of processor executes above-mentioned data processing method.
Another aspect, the embodiment of the present application provide a kind of computer readable storage medium, and being stored with can be by data
The computer program that equipment executes is managed, when described program is run on data processing equipment, so that the data processing equipment
The step of executing above-mentioned data processing method.
In the embodiment of the present application, according to preset condition acquire multiple objects to be monitored business datum and will be multiple to be monitored
Then the business datum of object filters out target object from each object to be monitored according to the business datum of each object to be monitored
Business datum.Whether the business datum for analyzing each target object again later is abnormal.When occurring abnormal object in target object
When object, the processing information of abnormal target object is received, and verifies the processing status of abnormal target object.Due to according to respectively wait supervise
The business datum of control object filters out target object from each object to be monitored, and the target object filtered out is more targeted,
The efficiency that can improve anomaly analysis is analyzed the target object filtered out, secondly, presetting condition, is then analyzed each
Whether a target object with preset condition matching determines whether target object is abnormal, for manual analysis, efficiency
It is higher.In the case where business datum amount is huge, all business datums of target object can be analyzed, determine whether target object is different
Often, rather than extraction section data are analyzed, to ensure that the accuracy of data analysis result.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this
For the those of ordinary skill in field, without any creative labor, it can also be obtained according to these attached drawings
His attached drawing.
Fig. 1 a is a kind of schematic diagram of data handling procedure provided by the embodiments of the present application；
Fig. 1 b is a kind of application scenarios schematic diagram that the application is applicable in；
Fig. 2 is a kind of flow diagram of data processing method provided by the present application；
Fig. 3 is a kind of schematic diagram of resource-niche provided by the present application；
Fig. 4 is a kind of schematic diagram for fluctuating situation provided by the present application；
Fig. 5 is a kind of schematic diagram for fluctuating situation provided by the present application；
Fig. 6 is a kind of stability bandwidth distribution schematic diagram provided by the present application；
Fig. 7 is a kind of schematic diagram for analyzing Status view provided by the present application；
Fig. 8 is a kind of schematic diagram of processing status view provided by the present application；
Fig. 9 is a kind of structural schematic diagram of data processing system provided by the present application；
Figure 10 is a kind of structural schematic diagram of data processing system provided by the present application；
Figure 11 is a kind of structural schematic diagram of data processing system provided by the present application；
Figure 12 is a kind of structural schematic diagram of data processing system provided by the present application；
Figure 13 is a kind of application scenario diagram that the application is applicable in；
Figure 14 is a kind of structural schematic diagram of data processing system provided by the present application；
Figure 15 is a kind of structural schematic diagram of data processing equipment provided by the present application.
Specific embodiment
In order to which the purpose of the present invention, technical solution and beneficial effect is more clearly understood, below in conjunction with attached drawing and implementation
Example, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only used to explain this hair
It is bright, it is not intended to limit the present invention.
In order to facilitate understanding, noun involved in the embodiment of the present invention is explained below.
TDW: Tencent's Distributed Data Warehouse (Tencent distributed Data Warehouse, abbreviation TDW), base
It is constructed in open source software Hadoop and Hive, has broken traditional data warehouse and be unable to linear expansion, the limitation of poor controllability.
During concrete practice, it was found by the inventors of the present invention that when through analysis business datum monitoring business state,
The method for mostly using artificial sampling sheet now, i.e., artificial rule of thumb extraction section data are divided from the business datum of acquisition
Analysis, then judges whether business is normal according to the analysis result of partial service data.But with the arrival of big data era,
Data volume is increased sharply, and the data that can manually analyze are limited, still using the manually extraction section from huge business datum amount
When data are analyzed, the virtual condition of identified analysis result and business be may differ by farther out, lead to data-handling efficiency
Lower, data analysis result deviation is larger.
For this purpose, the present inventor it is considered that analyzed using data processing system business datum, is automatically determined
Whether object to be monitored is abnormal, and specific logic is as shown in Figure 1a, first carries out data monitoring to the business datum of monitored object, determines
Then the overall trend and target object of business datum carry out anomaly analysis to target object.Determining target object appearance
When abnormal, target object is determined as abnormal target object, then related personnel is notified to handle abnormal target object.Together
When the business datum of abnormal target object is monitored, followed up according to the business datum monitored results of abnormal target object abnormal
The processing status of object.
Using collector, monitoring module, anomaly analysis module and different in data processing system in the embodiment of the present application
The above-mentioned logic of normal processing modules implement, specifically: collector acquires the business datum of multiple objects to be monitored according to preset condition
And the business datum of multiple objects to be monitored is sent to monitoring module, then by monitoring module according to the industry of each object to be monitored
Business data filter out the business datum of target object from each object to be monitored.Each is analyzed by anomaly analysis module again later
Whether the business datum of target object is abnormal, if abnormal, abnormal target object is sent to exception processing module, abnormality processing mould
Block receives the processing information of abnormal target object, and according to the business datum school of the abnormal target object obtained from monitoring module
Test the processing status of abnormal target object.Due to monitoring module according to the business datum of each object to be monitored from each object to be monitored
In filter out target object, the target object filtered out is more targeted, to target object carry out analysis can improve extremely divide
The efficiency of analysis, secondly, preset condition, then by anomaly analysis module analyze each target object whether with preset condition
It matches to determine whether target object is abnormal, it is more efficient for manual analysis.In the feelings that business datum amount is huge
Under condition, anomaly analysis module analyzes all business datums of target object, determines whether target object is abnormal, rather than extracting part
Divided data is analyzed, to ensure that the accuracy of data analysis result.Exception processing module receives abnormal target object
Processing information after, the processing status of abnormal target object is verified, to realize the supervision of abnormality processing.
The method dependence of artificial monitoring business manually analyzes business datum, therefore the period monitored is longer, still
Business is fast-developing now, the rapid iteration of product, and the business for resulting in the need for monitoring is more and more, the real-time manually monitored compared with
Difference will lead to material risk identification lag.For this purpose, collector acquires the business of object to be monitored in real time in the embodiment of the present application
Monitoring module is sent to after data, monitoring module periodically filters out the business datum of target object, then by anomaly analysis module
Whether the business datum for analyzing target object is abnormal, therefore the target object that can be noted abnormalities in time, avoids material risk.
Data processing system in the embodiment of the present application can be applied to application scenarios as shown in Figure 1 b, in the applied field
It include business side database 101, data processing system 102 and user terminal 103 in scape.
The business datum that object to be monitored generates is saved in business side database 101, business side database 101 can be
Offline database, such as TDW, the database being also possible in online database, such as service server.Data processing system
102 pull business datum from business side data library 101, and specifically, for TDW, configuration querying task pulls business in TDW
Data.For the database in service server, sql script is executed by timing and pulls business datum.Data processing system 102
After analyzing business datum, trend data, abnormal data of business etc. are obtained.Data processing system 102 is a service
The server cluster or cloud computing center of device or several servers composition.In order to facilitate human-computer interaction and operation, data
It need to show in visual form.Data processing system 102 is by means of Yii Framework realization, Yii Framework
It is one based on component, the high-performance PHP frame for developing large Web application, realizes model-view-controller (MVC)
Design pattern.When user needs to check the analysis result of data processing system 102, user terminal 103 passes through browser to data
Processing system 102 sends request of data, and data processing system 102 returns to business datum and shows view, report in a browser
Deng.User checks business monitoring trend, abnormal conditions by view, report etc., while carrying out problem supplement in a browser and retouching
The operation such as typing is stated, correspondingly data in data processing system 102 are changed.User terminal 103 can be smart phone,
Tablet computer or portable personal computer etc..
Based on application scenario diagram shown in Fig. 1 b, the embodiment of the invention provides a kind of data processing method, this method can
To be executed by data processing system, as shown in Figure 2, comprising:
Step S201 acquires the business datum of multiple objects to be monitored according to preset condition.
Object to be monitored can be the resource-niche in flow platform, for example, advertisement position in communication platform, under application program
The publication position etc. of application program in carrying platform.Fig. 3 is illustrated in a kind of flow platform provided by the embodiments of the present application
The schematic diagram of resource-niche, in Fig. 3, the publication position of wechat is a resource-niche of the application program download platform.
Business datum includes light exposure, click volume, download, transfer amount, amount of collection, comment amount of resource-niche etc., is also wrapped
Include the attributes such as time, the place of user's operation.
It optionally, can be big according to the production cycle of business datum, data volume when acquiring the business datum of object to be monitored
Small and monitoring demand is pulled necessary data, and is cleaned, classified, rejected to data by daily/weekly/frequency monthly
Invalid data.
Step S202 filters out target object from multiple objects to be monitored according to the business datum of multiple objects to be monitored
Business datum.
Step S203, whether the business datum for analyzing each target object matches with preset rules, if so, executing step
Rapid S204, it is no to then follow the steps S205.
Target object is determined as normal target object by step S204.
Target object is determined as abnormal target object by step S205.
Specifically, the attribute that preset rules can according to need the target object of analysis is preset, for example needs to analyze
When the stability bandwidth of target object, preset rules can be set as to stability bandwidth in the first preset range.According to target object
Business datum determines the stability bandwidth of target object, then analyzes the stability bandwidth of target object whether in the first preset range, if
It is that target object is then determined as normal target object, target object is otherwise determined as abnormal target object.For another example, it needs
When analyzing the stability of the business datum of target object, the variance that preset rules can be set as to business datum, which is less than, to be preset
Threshold value.Then determine target object business datum variance, analyze the business datum of target object variance whether be less than it is pre-
If threshold value, if so, target object is determined as normal target object, target object is otherwise determined as abnormal target object.
Step S206, receives the processing information of abnormal target object, and verifies the processing status of abnormal target object.
The processing information of abnormal target object indicates the processing mode to abnormal target object, different abnormal target objects
Different processing modes may be corresponded to.Processing mode includes but is not limited to delete target object, modification target object.It is receiving
After the processing information of abnormal target object, the processing mode to abnormal target object can be determined according to processing information, further
It ground can be using the processing status of different verification mode verification abnormal target objects for different processing modes.Abnormal mesh
The processing status of object is marked including at least untreated, processed.
Due to filtering out target object from each object to be monitored according to the business datum of each object to be monitored, filter out
Target object is more targeted, analyzes target object the efficiency that can improve anomaly analysis, secondly, presetting item
Part, analyzes whether each target object with preset condition matching determines whether target object is abnormal, compared to manual analysis
For, it is more efficient.In the case where business datum amount is huge, all business datums of analysis target object determine target
Whether object is abnormal, rather than extraction section data are analyzed, to ensure that the accuracy of data analysis result.It receives different
The processing status that abnormal target object is verified after the processing information of normal target object, realizes the supervision to abnormality processing.
Optionally, it in above-mentioned steps S202, in the business datum for filtering out target object, can be waited for for each
Monitored object obtains the current service data of object to be monitored and the history service data of object to be monitored, then according to wait supervise
The history service data of the current service data and object to be monitored of controlling object determine the movement of the business datum of object to be monitored
Average value is ranked up each object to be monitored according to the moving average of the business datum of each object to be monitored, will come preceding N
The object to be monitored of position is determined as target object, and N is preset threshold.
Illustratively, setting needs to be monitored advertisement position in wechat platform in September light exposure on the 3rd, in wechat platform
Including three advertisement positions, respectively advertisement position A, advertisement position B and advertisement position C.Then the exposure of these three advertisement positions is calculated separately
The moving average of light quantity is specifically described by taking advertisement position A as an example below.Wechat platform is obtained from monitoring data library 2021
Middle advertisement position A is 1000 in the light exposure on the 3rd of September in 2018, while obtaining advertisement position A in the light exposure on the 1st of September in 2018 and being
800, advertisement position A are 1200 in the light exposure on the 2nd of September in 2018.Then advertisement position A is calculated from September 1 day to September exposure on the 3rd
The average value of amount is 1000, it is hereby achieved that advertisement position A is 1000 in the moving average of September light exposure on the 3rd.Using same
It is 200 that the method for sample, which calculates advertisement position B in the moving average of September light exposure on the 3rd, and the light exposure of advertisement position C was at September 3rd
Moving average be 800.A, C, B are obtained after sorting from large to small according to the moving average of the light exposure of advertisement position, set N
It is 2, therefore advertisement position A and advertisement position C are determined as target object.Due to the movement according to the business datum of object to be monitored
Average value determines target object from object to be monitored, therefore can go out selective analysis object with Effective selection, to improve data point
The efficiency of analysis.
Optionally, before above-mentioned steps S203, total bulk wave can be determined according to the business datum of multiple objects to be monitored
Dynamic rate, the stability bandwidth of each target object is determined according to the business datum of each target object.
Illustratively, setting in platform A includes 3 resource-niches, as unit of day, from September 1 day to September 5th, is calculated every
Light exposure of the light exposure average value of one day 3 resource-niche as platform A, then according on the day of platform A light exposure with it is previous
Compared to the total ripple rate of the previous day on the day of it light exposure computing platform A.For each resource-niche in 3 resource-niches,
Compared to the stability bandwidth of the previous day on the day of according to the light exposure computing resource position of the light exposure and the previous day on the day of resource-niche.
It is possible to further generate total ripple view according to total ripple rate and show, according to each target object
Stability bandwidth generate the fluctuation view of each target object and show.
Illustratively, from September 2 days to September 5th, the total ripple rate of platform A was 10%, 20%, 10%, 12% for setting,
Then the total ripple view of platform A is as shown in Figure 4.
Illustratively, setting was from September 2 days to September 5th, and in 3 resource-niches of platform A, resource-niche 1 was from September 2 days to September
Stability bandwidth on the 4th is 10%, 30%, 15%, 12%, then the fluctuation view of resource-niche 1 is as shown in Figure 5.
It optionally, can be using total ripple rate to each target object in above-mentioned steps S203 into step S205
Stability bandwidth be modified, then whether the stability bandwidth of each target object after analysis corrections is located at the first preset range,
If so, determining that target object fluctuation is normal；Otherwise determine that target object fluctuation is abnormal.
Specifically, the stability bandwidth of target object meets normal distribution, illustratively, as shown in fig. 6, the fluctuation of target object
Probability of the rate between [μ-σ, μ+σ] is 68.2%, and probability of the stability bandwidth of target object between [+2 σ of μ -2 σ, μ] is
95.4%, wherein μ indicates that average value, σ indicate standard deviation.It can be seen that the stability bandwidth maximum probability of target object is located at [μ -2 σ, μ
+ 2 σ] between, when the stability bandwidth of target object is greater than+2 σ of μ or less than μ -2 σ, then illustrates that the stability bandwidth of target object has very probably
There is exception in rate, therefore [+2 σ of μ -2 σ, μ] can be set as the first preset range.
Due to the influence of festivals or holidays, flow platform is it is possible that the case where light exposure is increased sharply, and flow platform is total at this time
Body stability bandwidth will will increase, and be modified by stability bandwidth of the total ripple rate to each target object, and it is false can to exclude section
Influence of the day to the stability bandwidth of target object.Illustratively, platform A is set in the stability bandwidth at weekend as 30%, and it is workaday
Normal fluctuation rate is 10%, because weekend is the reason of movable user on platform A increases when 20% stability bandwidth being higher by is possible.
Resource-niche 1 on platform A is set in the stability bandwidth at weekend as 40%, the first preset range is [- 15%, 15%], in analysis resource
The stability bandwidth of resource-niche 1 is first subtracted 20%, obtains revised stability bandwidth 20% by position 1 in the stability bandwidth at weekend.Due to repairing
The stability bandwidth of resource-niche 1 after just is not in the first preset range, it is determined that resource-niche 1 fluctuates exception.It is flat by analysis flow
Then the total ripple rate of platform is modified the stability bandwidth of target object using total ripple rate, to avoid festivals or holidays etc.
Influence of the normal phenomenon to the stability bandwidth of target object, to improve the accuracy of the stability bandwidth of analysis target object.
Optionally, after determining total ripple rate according to the business datum of multiple objects to be monitored, it can analyze total ripple
Whether rate is located at the second preset range, if so, determining that total ripple rate is normal, otherwise determines that total ripple rate is abnormal.Setting
The method of second preset range is identical as the method for first threshold range is determined, details are not described herein again.By analyzing totality in real time
Stability bandwidth can find the fluctuation exception of platform entirety and alert, in time to avoid material risk.
Optionally, for the ease of being interacted with user, analysis target object business datum and preset rules whether
When matching, analysis Status view can be generated and show, illustratively, setting filters out the business datum of 8 target objects, mesh
Mark object analysis state include it is to be analyzed, analyzing and analyzing, analyze this 8 target objects business datum when
Analysis Status view is generated as shown in fig. 7, as shown in Figure 7, in 8 target objects, target object to be analyzed there are 4, and Zhan is total
Target object 50%, the target object in analysis has 1, the 12% of Zhan total target object, the target object analyzed
There are 3, the 38% of Zhan total target object.It is real-time convenient for monitoring personnel by the analysis state of each target object of real-time exhibition
The analysis progress of master goal object.
Optionally, in above-mentioned steps S206, the different processing information of abnormal target object corresponds to different processing statuses
Verification mode.
In a kind of possible embodiment, when receiving the processing information of suppressing exception target object, verification is default
Whether the business datum of abnormal target object is collected in period, if so, determining that the processing status of abnormal target object is
It is untreated, otherwise determine that the processing status of abnormal target object is processed.
Illustratively, there is exception in the light exposure for setting the resource-niche A of platform 1, and improvement project is undercarriage resource-niche A.It obtains
After knowing the processing information, the subsequent light exposure of resource-niche A is monitored if still monitoring the light exposure of resource-niche A and illustrates resource
The exception of position A is untreated.If not monitoring the light exposure of resource-niche A, illustrate that the exception of resource-niche A is processed.
In a kind of possible embodiment, when receiving the processing information of modification abnormal target object, abnormal mesh is verified
Whether mark object matches with preset rules；If so, determine abnormal target object processing status be it is processed, otherwise determine different
The processing status of normal target object is untreated.
Illustratively, there is exception in the light exposure for setting the resource-niche M of platform 1, and improvement project is the hair of adjustresources position M
Cloth position.After knowing the processing information, the subsequent analysis of monitoring resource-niche M is as a result, if analysis result is still resource-niche M
Light exposure is abnormal, illustrates that the exception of resource-niche M is untreated.If analysis illustrates resource the result is that the light exposure of resource-niche M is normal
The exception of position M is processed.Due to passing through processing information checking exception mesh after related personnel handles abnormal target object
The processing status of object is marked, realizes the supervision to abnormality processing.
Further, when determining that target object is determined as abnormal target object, alarm view and processing shape can be generated
State view, and shown to user.The processing information of abnormal target object is received, and verifies the processing status of abnormal target object
Afterwards, processing status view is updated according to the result of verification.
Specifically, the attribute information of each abnormal target object is shown in alarm view, alarm view, which can be, to be received
It is just shown when the inquiry instruction of user, is also possible to show in a manner of automatic pop-up in newly-increased abnormal target object.Processing
Status view can both show the processing status of each abnormal target object, can also show the institute found in set period of time
The processing status of some abnormal target objects, processing status include untreated, processed.Processing status view is specifically described below
Figure, is set in August and has found 5 abnormal target objects, have found 8 abnormal target objects in September, have found 6 in October
Abnormal target object, processing status include untreated, processed.According to August, September and received abnormal target object in October
Processing information and check results update processing status view, it is specific as shown in figure 8, August discovery 5 abnormal target objects
In, the processing statuses of 4 abnormal target objects be it is processed, the processing status of 1 abnormal target object is untreated.September hair
In 8 existing abnormal target objects, the processing status of 6 abnormal target objects is processed, the processing of 2 abnormal target objects
State is untreated.In 6 abnormal target objects of discovery in October, the processing statuses of 4 abnormal target objects be it is processed, 2
The processing status of a abnormal target object is untreated.
Conceived based on same technique, the embodiment of the present application provides a kind of data processing system, which can execute number
According to processing method, as shown in Figure 9, comprising:
Collector 901, monitoring module 902, anomaly analysis module 903, exception processing module 904.
Collector 901, for acquired according to preset condition multiple objects to be monitored business datum and will be multiple to be monitored
The business datum of object is sent to monitoring module 902.
Monitoring module 902, for being filtered out from multiple objects to be monitored according to the business datum of multiple objects to be monitored
The business datum of target object.
Anomaly analysis module 903 for obtaining the business datum of target object, and analyzes the business of each target object
Whether data match with preset rules, if so, determining that target object is normal, otherwise determine that target object is abnormal.
Exception processing module 904 for receiving the processing information of abnormal target object, and verifies the place of abnormal target object
Reason state.
Optionally, as shown in Figure 10, above-mentioned monitoring module 902 further includes monitoring data library 9021, and monitoring module 902 receives
The business datum for multiple objects to be monitored that collector 901 is sent simultaneously is stored in monitoring data library 9021.
Monitoring module 902 is specifically used for: being directed to each object to be monitored, obtains from monitoring data library 9021 to be monitored
The history service data of the current service data of object and object to be monitored, then according to the current service data of object to be monitored
The moving average that the business datum of object to be monitored is determined with the history service data of object to be monitored, according to each to be monitored right
The moving average of the business datum of elephant is ranked up each object to be monitored, and the object to be monitored for coming top N is determined as
Target object, N are preset threshold.
Optionally, monitoring module 902 is also used to: total ripple rate is determined according to the business datum of multiple objects to be monitored,
The stability bandwidth that each target object is determined according to the business datum of each target object, by total ripple rate and each mesh
The stability bandwidth of mark object is sent to anomaly analysis module 903.
Optionally, as shown in figure 11, anomaly analysis module 903 further includes anomaly analysis database 9031.Anomaly analysis mould
The total ripple that block 903 receives the business datum for multiple objects to be monitored that collector 901 acquires and monitoring module 902 is sent
The stability bandwidth of rate and target object, and be stored in anomaly analysis database 9031.
Optionally, anomaly analysis module 903 is specifically used for: using total ripple rate to the stability bandwidth of each target object
It is modified, whether the stability bandwidth of each target object after analysis corrections is located at the first preset range, if so, determining mesh
It is normal to mark object fluctuation, otherwise determines that target object fluctuation is abnormal.
Optionally, anomaly analysis module 903 is also used to: whether analysis total ripple rate is located at the second preset range, if so,
It then determines that total ripple rate is normal, otherwise determines that total ripple rate is abnormal.
Optionally, as shown in figure 12, above-mentioned exception processing module 904 further includes issue database 2041.Abnormality processing mould
Block 904 obtains the attribute information of 903 each abnormal target objects from anomaly analysis module, then by the attribute of each abnormal target object
Information preservation is to issue database 9041.Specifically, the attribute information of abnormal target object include abnormal target object mark,
Exception Type etc..Exception processing module 904 classifies to abnormal target object according to the Exception Type of abnormal target object, so
It is stored in issue database according to classification afterwards.
Optionally, exception processing module 904 is specifically used for:
The processing information of suppressing exception target object is received, whether verification collects abnormal object pair within a preset period of time
The business datum of elephant；
If so, determining that the processing status of abnormal target object is untreated；
Otherwise determine that the processing status of abnormal target object is processed.
Optionally, exception processing module 904 is specifically used for:
The processing information of modification abnormal target object is received, whether verification abnormal target object matches with preset rules；
If so, determining that the processing status of abnormal target object is processed；
Otherwise determine that the processing status of abnormal target object is untreated.
Embodiment in order to preferably explain the present invention describes the embodiment of the present invention below with reference to specific implement scene and provides
A kind of data processing system, set object to be monitored as the resource-niche in flow platform, the business datum of object to be monitored is
The light exposure of resource-niche, as shown in figure 13, the business datum of resource-niche are stored in business side database, and business side database includes
TDW and Linux server.Data processing system configuration querying task in TDW pulls the light exposure of resource-niche in flow platform,
By timing perform script from the light exposure for pulling resource-niche in flow platform in the database of Linux server.For in TDW
The data pulled, data processing system execute the exposure of analysis resource-niche by uploading python script, Allocation Analysis calculating task
Analysis result is stored in monitoring data library, anomaly analysis database and issue database by light quantity.For in Linux server
The data pulled, according to analysis needs, using a variety of analysis scripts such as sql/python/php, configuring timing tasks are executed, will
Analysis result is stored in monitoring data library, anomaly analysis database and issue database.Above-mentioned analysis result include trend data,
Abnormal data etc..When user wants to understand the state of data analysis, it can be sent out by browser to data processing system entrance script
Request is sent, entrance script loads an application configuration, creates a controller, and controller creates an application example and processing is gone to ask
It asks, the operational instances in controller can load the data from database, view, report etc. be generated, then by view and report
Table is back to browser, shows user.
Further, as shown in figure 14, data processing system by collector, monitoring module, anomaly analysis module and
Exception processing module, which is realized, to be pulled and analyzing to the light exposure of resource-niche in flow platform.Collector drawing is gone every on flow platform
The daily light exposure of a resource-niche.Since repetition, invalid data are easy to interfere subsequent analysis model, first adopt
It is rejected with analysis script.By monitoring data library of the synchronizing traffic data after cleaning into monitoring module.It simultaneously will be each
The light exposure of resource-niche individually summarizes, and the anomaly analysis database being synchronized in anomaly analysis module.
Monitoring module analyzes the light exposure of flow platform entirety, calculate the daily light exposure of flow platform compared to
The total ripple rate of the previous day, and total ripple rate is transferred to anomaly analysis module.In addition monitoring module calculates single resource
The moving average of the light exposure of position filters out header resource position according to the moving average of light exposure, such as by light exposure
The resource-niche that moving average comes preceding 100 is determined as header resource position, and the mark of header resource position is then transferred to exception
Analysis module.
Anomaly analysis module analyzes the light exposure of specific resource-niche, and in specific implementation, anomaly analysis module can
It is analyzed, the light exposure for the header resource position that monitoring module transmits can also be carried out with the light exposure to each resource-niche
Analysis.Whether the light exposure of the specific resource-niche of anomaly analysis module Main Analysis meets preset rules, the fluctuation situation of light exposure
Deng.For example, first counting the exposure of each header resource position respectively when whether the light exposure for analyzing header resource position meets preset rules
Then light quantity is ranked up each header resource position according to light exposure, determines the practical sequence of each header resource position, by each head
The practical sequence of resource-niche is compared with the sequence of each header resource position determined according to preset rules, when same header resource
The difference of the arrangement serial number of position is greater than preset threshold, illustrates that the light exposure of the resource-niche is abnormal.For example, in analysis header resource position
Light exposure fluctuation situation it is whether abnormal when, the light exposure of each header resource position is first counted respectively, in festivals or holidays, using stream
The stability bandwidth of amount platform totality is modified the light exposure of head resource-niche, excludes festivals or holidays to the stability bandwidth of head resource-niche
Influence.Then in the stability bandwidth for judging each header resource position whether in the first preset range, if so, illustrating head
Resource-niche fluctuation is normal, otherwise illustrates that the fluctuation of header resource position is abnormal.Anomaly analysis module believes the attribute of abnormal resource-niche
Breath is transferred to the problems in exception processing module database.
Exception processing module is Visualization Platform, by the exception of the monitored results of monitoring module and anomaly analysis module point
Analysis result is visualized.For example, exception processing module obtains the attribute information of abnormal resource position from issue database,
Work order is generated in Visualization Platform to be managed, according to the Exception Type of the subsidiary resource-niche of work order, generate alarm view with
And it processing status view and is shown.Exception processing module receives the data of monitoring module feedback, true according to the data of feedback
Determine the processing status of abnormal resource position.Exception processing module receives the analysis of anomaly analysis module as a result, based on the analysis results really
Determine the processing status of abnormal resource position.When the analysis result for data or the anomaly analysis module transmitting fed back according to monitoring module is true
Determine abnormal resource position to be processed, then updates alarm view and processing status view.
Since monitoring module filters out target object from each object to be monitored according to the business datum of each object to be monitored,
The target object filtered out is more targeted, analyzes target object the efficiency that can improve anomaly analysis, secondly, in advance
Then setting condition analyzes whether each target object determines target object with preset condition matching by anomaly analysis module
It is whether abnormal, it is more efficient for manual analysis.In the case where business datum amount is huge, anomaly analysis module
All business datums of target object are analyzed, whether abnormal determine target object, rather than extraction section data are analyzed, from
And it ensure that the accuracy of data analysis result.In addition, exception processing module by obtain monitoring module monitored results and
Anomaly analysis module analyzes the processing status as a result, determining abnormal target object, and the processing status view that timely updates, and is convenient for
It exercises supervision to the subsequent processing of abnormal target object.
Based on the same technical idea, the embodiment of the present application provides a kind of data processing equipment, as shown in figure 15, including
At least one processor 1501, and the memory 1502 connecting at least one processor do not limit in the embodiment of the present application
Specific connection medium between processor 1501 and memory 1502 passes through between processor 1501 and memory 1502 in Figure 15
For bus connection.Bus can be divided into address bus, data/address bus, control bus etc..
In the embodiment of the present application, memory 1502 is stored with the instruction that can be executed by least one processor 1501, until
The instruction that a few processor 1501 is stored by executing memory 1502 can execute included in aforementioned data processing method
The step of.
Wherein, processor 1501 is the control centre of data processing, can use various interfaces and connection tracking vehicle
The various pieces of the equipment of position, are stored in by running or executing the instruction being stored in memory 1502 and calling
Data in reservoir 1502, to handle data.Optionally, processor 1501 may include one or more processing units, processing
Device 1501 can integrate application processor and modem processor, wherein the main processing operation system of application processor, Yong Hujie
Face and application program etc., modem processor mainly handle wireless communication.It is understood that above-mentioned modem processor
It can not also be integrated into processor 1501.In some embodiments, processor 1501 and memory 1502 can be in same cores
On piece realizes that in some embodiments, they can also be realized respectively on independent chip.
Processor 1501 can be general processor, such as central processing unit (CPU), digital signal processor, dedicated collection
At circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array or other
Perhaps transistor logic, discrete hardware components may be implemented or execute the application reality for programmable logic device, discrete gate
Apply each method, step disclosed in example and logic diagram.General processor can be microprocessor or any conventional processing
Device etc..The step of method in conjunction with disclosed in the embodiment of the present application, can be embodied directly in hardware processor and execute completion, or
With in processor hardware and software module combination execute completion.
Memory 1502 is used as a kind of non-volatile computer readable storage medium storing program for executing, can be used for storing non-volatile software journey
Sequence, non-volatile computer executable program and module.Memory 1502 may include the storage medium of at least one type,
It such as may include flash memory, hard disk, multimedia card, card-type memory, random access storage device (Random Access
Memory, RAM), static random-access memory (Static Random Access Memory, SRAM), may be programmed read-only deposit
Reservoir (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), band
Electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory,
EEPROM), magnetic storage, disk, CD etc..Memory 1502 can be used for carrying or storing have instruction or data
The desired program code of structure type and can by any other medium of computer access, but not limited to this.The application is real
Applying the memory 1502 in example can also be circuit or other devices that arbitrarily can be realized store function, for storing program
Instruction and/or data.
Based on the same inventive concept, the embodiment of the present application provides a kind of computer readable storage medium, and being stored with can
The computer program executed by data processing equipment, when described program is run on data processing equipment, so that data processing
The step of equipment configuration for executing data processing.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (13)
1. a kind of data processing method characterized by comprising
The business datum of multiple objects to be monitored is acquired according to preset condition；
The industry of target object is filtered out from the multiple object to be monitored according to the business datum of the multiple object to be monitored
Business data；
Whether the business datum for analyzing each target object matches with preset rules；
If so, the target object is determined as normal target object；
Otherwise the target object is determined as abnormal target object, receives the processing information of the abnormal target object, and school
Test the processing status of the abnormal target object.
2. the method as described in claim 1, which is characterized in that the business datum according to the multiple object to be monitored from
The business datum of target object is filtered out in the multiple object to be monitored, comprising:
For each object to be monitored, the current service data of the object to be monitored and going through for the object to be monitored are obtained
History business datum；
According to the history service data of the current service data of the object to be monitored and the object to be monitored determine it is described to
The moving average of the business datum of monitored object；
The multiple object to be monitored is ranked up according to the moving average of the business datum of the multiple object to be monitored；
The object to be monitored for coming top N is determined as target object, N is preset threshold.
3. the method as described in claim 1, which is characterized in that the business datum for analyzing each target object and default
Before whether rule matches, further includes:
Total ripple rate is determined according to the business datum of the multiple object to be monitored；
The stability bandwidth of each target object is determined according to the business datum of each target object.
4. method as claimed in claim 3, which is characterized in that the business datum for analyzing each target object and default
Whether rule matches, comprising:
The stability bandwidth of each target object is modified using the total ripple rate, each target after analysis corrections
Whether the stability bandwidth of object is located at the first preset range.
5. the method as described in Claims 1-4 is any, which is characterized in that the processing for receiving the abnormal target object
Information, and verify the processing status of the abnormal target object, comprising:
The processing information for deleting the abnormal target object is received, whether verification collects the abnormal mesh within a preset period of time
Mark the business datum of object；
If so, determining that the processing status of the abnormal target object is untreated；
Otherwise determine that the processing status of the abnormal target object is processed.
6. the method as described in Claims 1-4 is any, which is characterized in that the processing for receiving the abnormal target object
Information, and verify the processing status of the abnormal target object, comprising:
Receive the processing information for modifying the abnormal target object；
Verify whether the abnormal target object matches with the preset rules；
If so, determining that the processing status of the abnormal target object is processed；
Otherwise determine that the processing status of the abnormal target object is untreated.
7. a kind of data processing system characterized by comprising
Collector, monitoring module, anomaly analysis module, exception processing module；
The collector will be the multiple to be monitored for acquiring the business datum of multiple objects to be monitored according to preset condition
The business datum of object is sent to the monitoring module；
The monitoring module, for being sieved from the multiple object to be monitored according to the business datum of the multiple object to be monitored
Select the business datum of target object；
The anomaly analysis module for obtaining the business datum of target object, and analyzes the business number of each target object
Whether matched according to preset rules；If so, the target object is determined as normal target object；Otherwise by the target pair
As being determined as abnormal target object；
The exception processing module for receiving the processing information of the abnormal target object, and verifies the abnormal object pair
The processing status of elephant.
8. system as claimed in claim 7, which is characterized in that the monitoring module is specifically used for:
For each object to be monitored, obtained from the monitoring data library object to be monitored current service data and
The history service data of the object to be monitored；
According to the history service data of the current service data of the object to be monitored and the object to be monitored determine it is described to
The moving average of the business datum of monitored object；
The multiple object to be monitored is ranked up according to the moving average of the business datum of the multiple object to be monitored；
The object to be monitored for coming top N is determined as target object, N is preset threshold.
9. system as claimed in claim 7, which is characterized in that the monitoring module is also used to:
Total ripple rate is determined according to the business datum of the multiple object to be monitored；
The stability bandwidth of each target object is determined according to the business datum of each target object；
The stability bandwidth of the total ripple rate and each target object is sent to the anomaly analysis module.
10. system as claimed in claim 8, which is characterized in that the anomaly analysis module is specifically used for:
The stability bandwidth of each target object is modified using the total ripple rate；
Whether the stability bandwidth of each target object after analysis corrections is located at the first preset range；
If so, determining that the target object fluctuation is normal；Otherwise determine that the target object fluctuation is abnormal.
11. system the method according to any one of claims 7 to 10, which is characterized in that the exception processing module is specifically used for:
The processing information for deleting the abnormal target object is received, whether verification collects the abnormal mesh within a preset period of time
Mark the business datum of object；
If so, determining that the processing status of the abnormal target object is untreated；
Otherwise determine that the processing status of the abnormal target object is processed.
12. system the method according to any one of claims 7 to 10, which is characterized in that the exception processing module is specifically used for:
Receive the processing information for modifying the abnormal target object；
Verify whether the abnormal target object matches with the preset rules；
If so, determining that the processing status of the abnormal target object is processed；
Otherwise determine that the processing status of the abnormal target object is untreated.
13. a kind of data processing equipment, which is characterized in that including at least one processor and at least one processor,
In, the memory is stored with computer program, when described program is executed by the processor, so that the processor executes
The step of claim 1~6 any claim the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811354476.XA CN109460432B (en) | 2018-11-14 | 2018-11-14 | Data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811354476.XA CN109460432B (en) | 2018-11-14 | 2018-11-14 | Data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109460432A true CN109460432A (en) | 2019-03-12 |
| CN109460432B CN109460432B (en) | 2020-06-26 |
Family
ID=65610535
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811354476.XA Active CN109460432B (en) | 2018-11-14 | 2018-11-14 | Data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109460432B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110297828A (en) * | 2019-06-13 | 2019-10-01 | 北京字节跳动网络技术有限公司 | Manage method, apparatus, storage medium and the electronic equipment of data warehouse table |
| CN110830445A (en) * | 2019-10-14 | 2020-02-21 | 中国平安财产保险股份有限公司 | Method and device for identifying abnormal access object |
| CN111143163A (en) * | 2019-12-13 | 2020-05-12 | 上海硬通网络科技有限公司 | Data monitoring method and device, computer equipment and storage medium |
| CN111866924A (en) * | 2019-04-25 | 2020-10-30 | 中国移动通信集团安徽有限公司 | Performance index monitoring method and device, computing equipment and computer storage medium |
| CN111966915A (en) * | 2019-05-20 | 2020-11-20 | 腾讯科技（深圳）有限公司 | Information inspection method, computer equipment and storage medium |
| CN112463807A (en) * | 2020-11-20 | 2021-03-09 | 广东电网有限责任公司 | Data processing method, device, server and storage medium |
| CN112785446A (en) * | 2021-01-26 | 2021-05-11 | 中国人寿保险股份有限公司上海数据中心 | Premium data self-correction real-time display method, system and storage medium |
| CN116911914A (en) * | 2023-09-08 | 2023-10-20 | 杭州联海网络科技有限公司 | Marketing management method and system based on visual terminal |
| CN117591530A (en) * | 2024-01-17 | 2024-02-23 | 杭银消费金融股份有限公司 | Data cross section processing method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731816A (en) * | 2013-12-23 | 2015-06-24 | 阿里巴巴集团控股有限公司 | Method and device for processing abnormal business data |
| CN105468701A (en) * | 2015-11-18 | 2016-04-06 | 车智互联（北京）科技有限公司 | Method and device for computing influence degree of external sources on website traffic fluctuation |
| CN106557655A (en) * | 2016-11-17 | 2017-04-05 | 包磊 | Method and apparatus is recommended in a kind of analysis based on User Status |
| US20170132063A1 (en) * | 2015-11-10 | 2017-05-11 | China Construction Bank Corporation | Information system fault scenario information collecting method and system |
-
2018
- 2018-11-14 CN CN201811354476.XA patent/CN109460432B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731816A (en) * | 2013-12-23 | 2015-06-24 | 阿里巴巴集团控股有限公司 | Method and device for processing abnormal business data |
| US20170132063A1 (en) * | 2015-11-10 | 2017-05-11 | China Construction Bank Corporation | Information system fault scenario information collecting method and system |
| CN105468701A (en) * | 2015-11-18 | 2016-04-06 | 车智互联（北京）科技有限公司 | Method and device for computing influence degree of external sources on website traffic fluctuation |
| CN106557655A (en) * | 2016-11-17 | 2017-04-05 | 包磊 | Method and apparatus is recommended in a kind of analysis based on User Status |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111866924A (en) * | 2019-04-25 | 2020-10-30 | 中国移动通信集团安徽有限公司 | Performance index monitoring method and device, computing equipment and computer storage medium |
| CN111966915A (en) * | 2019-05-20 | 2020-11-20 | 腾讯科技（深圳）有限公司 | Information inspection method, computer equipment and storage medium |
| CN110297828A (en) * | 2019-06-13 | 2019-10-01 | 北京字节跳动网络技术有限公司 | Manage method, apparatus, storage medium and the electronic equipment of data warehouse table |
| CN110830445A (en) * | 2019-10-14 | 2020-02-21 | 中国平安财产保险股份有限公司 | Method and device for identifying abnormal access object |
| CN110830445B (en) * | 2019-10-14 | 2023-02-03 | 中国平安财产保险股份有限公司 | Method and device for identifying abnormal access object |
| CN111143163A (en) * | 2019-12-13 | 2020-05-12 | 上海硬通网络科技有限公司 | Data monitoring method and device, computer equipment and storage medium |
| CN111143163B (en) * | 2019-12-13 | 2024-04-16 | 上海硬通网络科技有限公司 | Data monitoring method, device, computer equipment and storage medium |
| CN112463807A (en) * | 2020-11-20 | 2021-03-09 | 广东电网有限责任公司 | Data processing method, device, server and storage medium |
| CN112785446A (en) * | 2021-01-26 | 2021-05-11 | 中国人寿保险股份有限公司上海数据中心 | Premium data self-correction real-time display method, system and storage medium |
| CN116911914A (en) * | 2023-09-08 | 2023-10-20 | 杭州联海网络科技有限公司 | Marketing management method and system based on visual terminal |
| CN117591530A (en) * | 2024-01-17 | 2024-02-23 | 杭银消费金融股份有限公司 | Data cross section processing method and system |
| CN117591530B (en) * | 2024-01-17 | 2024-04-19 | 杭银消费金融股份有限公司 | Data cross section processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109460432B (en) | 2020-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109460432A (en) | A kind of data processing method and system | |
| US10565442B2 (en) | Picture recognition method and apparatus, computer device and computer- readable medium | |
| US20180240062A1 (en) | Collaborative algorithm development, deployment, and tuning platform | |
| CN110197280A (en) | A kind of knowledge mapping construction method, apparatus and system | |
| CN111740884B (en) | Log processing method, electronic equipment, server and storage medium | |
| US20180276508A1 (en) | Automated visual information context and meaning comprehension system | |
| CN110442737A (en) | The twin method and system of number based on chart database | |
| EP3118784A1 (en) | Method and system for enabling dynamic capacity planning | |
| CN110349013A (en) | Risk control method and device | |
| CN107729138A (en) | A kind of analysis method and device of high-performance distributed Vector spatial data | |
| CN109873790A (en) | Network security detection method, device and computer readable storage medium | |
| CN114722091A (en) | Data processing method, data processing device, storage medium and processor | |
| CN110300002A (en) | A kind of visual distributed O&M method and device | |
| CN109376981B (en) | Data processing mode determining method and device, server and data processing method | |
| Shankar et al. | Janus: Benchmarking commercial and open-source cloud and edge platforms for object and anomaly detection workloads | |
| CN111598700A (en) | Financial wind control system and method | |
| CN113569162A (en) | Data processing method, device, equipment and storage medium | |
| CN106228029B (en) | Quantification problem method for solving and device based on crowdsourcing | |
| Xhafa et al. | A software chain approach to big data stream processing and analytics | |
| WO2021168346A1 (en) | Systems and methods for anonymizing sensitive data and simulating accelerated schedule parameters using the anonymized data | |
| CN115576973B (en) | Service deployment method, device, computer equipment and readable storage medium | |
| CN110910241A (en) | Cash flow evaluation method, apparatus, server device and storage medium | |
| CN114095390B (en) | Method, device, equipment and storage medium for predicting flow of objects in area | |
| CN114661571A (en) | Model evaluation method, model evaluation device, electronic equipment and storage medium | |
| CN107872352A (en) | A kind of performance test methods of network management system, apparatus and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |