CN108536812A - Sweep-out method, device, equipment and the computer-readable medium of invalid data resource - Google Patents
Sweep-out method, device, equipment and the computer-readable medium of invalid data resource Download PDFInfo
- Publication number
- CN108536812A CN108536812A CN201810302134.7A CN201810302134A CN108536812A CN 108536812 A CN108536812 A CN 108536812A CN 201810302134 A CN201810302134 A CN 201810302134A CN 108536812 A CN108536812 A CN 108536812A
- Authority
- CN
- China
- Prior art keywords
- invalid
- data
- invalid data
- data resource
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention proposes a kind of sweep-out method of invalid data resource, it is searched in various types of data resources according to invalid mining rule, it determines and whether there is invalid data resource in various types of data resources, if, invalid data resource then is removed according to the corresponding reset mode of invalid data resource, invalid data resource includes at least one of invalid inquiry job, invalid computation operation, invalid data storage and invalid data calling.Invalid data resource is effectively removed, wasting of resources caused by computing repeatedly operation, invalid computation operation, data storage, data call occupancy resource in data analysis process is avoided.The present invention also provides a kind of remove device of invalid data resource, a kind of removing equipment of invalid data resource and a kind of computer readable storage mediums, have above-mentioned technique effect.
Description
Technical field
The present invention relates to computer realms, are related to a kind of sweep-out method of invalid data resource, further relate to a kind of invalid number
According to the remove device of resource, a kind of removing equipment of invalid data resource and a kind of computer readable storage medium.
Background technology
In internet and other all trades and professions, data analysis is used more and more, various particularly with type of business
Major company, each department, each business in major company not only have data to calculate, storage, the demand field for the applications such as calling
Scape, and will appear data cross analysis application.The application of calculating, storage, calling for these data etc., which once lacks, to be had
The management of effect just will appear the waste of resource, such as：Invalid computation computes repeatedly, invalid storage, repeats storage, invalid calling
Deng.
Currently, conventional management method is artificial combing after reporting, or carried out by the way of internal bill mechanism
Processing.Conventional management method main problem is：(1) efficiency is low, due to needing manually to report and arrange, judge, timeliness
It is poor with efficiency；(2) nonautonomy, the application such as the calculating operation being related to, data storage, data call lean on service line from Your Majesty
Report, reasonability can not strictly be ensured by relying on consciousness；(3) limitation, report, internal bill mechanism all has certain subjectivity,
Period of supervision can not also accomplish flexibly, there is certain limitation.
Therefore, how to avoid causing resource unrestrained due to calculating invalid during data analysis application, storage, calling etc.
Take, is the technical issues of being badly in need of solving.
Invention content
The embodiment of the present invention provides a kind of sweep-out method of invalid data resource, further relates to a kind of the clear of invalid data resource
Except device, a kind of removing equipment of invalid data resource and a kind of computer readable storage medium, to solve in the prior art
At least one of the above technical problem.
In a first aspect, an embodiment of the present invention provides a kind of sweep-out methods of invalid data resource, including：
It is searched, is determined in various types of data resources in various types of data resources according to invalid mining rule
With the presence or absence of invalid data resource；
If so, removing invalid data resource, invalid data resource packet according to the corresponding reset mode of invalid data resource
Include at least one of invalid inquiry job, invalid computation operation, invalid data storage and invalid data calling.
With reference to first aspect, the present invention is in the first embodiment of first aspect, according to invalid mining rule various
The data resource of type is searched, and is determined and be whether there is invalid data resource in various types of data resources, including：
When data resource is inquiry job, the similarity between each inquiry job is calculated；
An inquiry job is selected from each inquiry job that the similarity is more than threshold value, non-selected inquiry is made
Industry is determined as invalid inquiry job.
First embodiment with reference to first aspect, the present invention is in the second embodiment of first aspect, according to invalid
The corresponding reset mode of data resource removes invalid data resource, including：
The inquiry job selected is executed, query result is obtained；
Common data fairground table is established according to the query result；
Delete invalid inquiry job.
With reference to first aspect, the present invention is in the third embodiment of first aspect, according to invalid mining rule various
The data resource of type is searched, and is determined and be whether there is invalid data resource in various types of data resources, including：
When data resource is that data store, each ephemeral data storage table is scanned；
Judge whether the ephemeral data storage table scanned repeats, is selected from the ephemeral data storage table repeated
Non-selected ephemeral data storage table is determined as invalid data storage by one ephemeral data storage table.
Third embodiment with reference to first aspect, the present invention is in the 4th embodiment of first aspect, according to invalid
The corresponding reset mode of data resource removes invalid data resource, including：
Offline invalid data storage.
With reference to first aspect, the present invention is in the 5th embodiment of first aspect, according to invalid mining rule various
The data resource of type is searched, and is determined and be whether there is invalid data resource in various types of data resources, including：
When data resource is to calculate operation, then judge that the calculating operation is according to the output journal for calculating operation
No abnormal interrupt；If so, the calculating operation of abnormal interrupt is determined as invalid computation operation.
With reference to first aspect, the present invention is in the sixth embodiment of first aspect, according to invalid mining rule various
The data resource of type is searched, and is determined and be whether there is invalid data resource in various types of data resources, including：
According to the binding relationship of various types of data resources and human resource system, it is determined whether there are ex-employee's
Invalid data resource, wherein the invalid data resource of ex-employee includes that the invalid computation operation of ex-employee, invalid data are deposited
At least one of storage and invalid data calling.
With reference to first aspect, the present invention is in the 7th embodiment of first aspect, according to invalid mining rule various
The data resource of type is searched, and is determined and be whether there is invalid data resource in various types of data resources, including：
According to the binding relationship of various types of data resources and business management system, it is determined whether there is the business of being turned off
Invalid data resource, wherein the invalid data resource for being turned off business includes that the invalid of the business that is turned off can be regarded as industry, invalid number
At least one of called according to storage and invalid data.
One kind in the 5th embodiment with reference to first aspect, sixth embodiment and the 7th embodiment, this hair
It is bright to remove invalid data resource according to the corresponding reset mode of invalid data resource in the 8th embodiment of first aspect,
Including：
By the offline closing of invalid computation operation, and clear up the rubbish that the invalid computation operation generates in history calculating；
Or
Invalid data is stored offline；Or
Invalid data is called offline.
Second aspect, an embodiment of the present invention provides a kind of remove devices of invalid data resource, including：
Invalid data resource lookup module, for being looked into various types of data resources according to invalid mining rule
It looks for；
Invalid data resource confirmation module is provided for determining in various types of data resources with the presence or absence of invalid data
Source；
Invalid data resource removes module, if in various types of data resources there are invalid data resource,
Remove invalid data resource according to the corresponding reset mode of invalid data resource, invalid data resource include invalid inquiry job,
At least one of invalid computation operation, invalid data storage and invalid data calling.
In conjunction with second aspect, for the present invention in the first embodiment of second aspect, the invalid data resource confirms mould
Block includes：
Similarity calculated is used for when data resource is inquiry job, between each inquiry job of calculating
Similarity；
Acknowledgment of your inquiry unit is repeated, for selecting an inquiry from each inquiry job that the similarity is more than threshold value
Non-selected inquiry job is determined as invalid inquiry job by operation.
In conjunction with the first embodiment of second aspect, the present invention is described invalid in the second embodiment of second aspect
Data resource removes module：
Invalid inquiry job clearing cell obtains query result, is looked into according to described for executing the inquiry job selected
It askes result and establishes common data fairground table, delete invalid inquiry job.
In conjunction with second aspect, for the present invention in the third embodiment of second aspect, the invalid data resource confirms mould
Block further includes：
Storage table scanning element, for when data resource is that data store, scanning each ephemeral data storage table；
Invalid data stores confirmation unit, is deposited for selecting an ephemeral data from the ephemeral data storage table repeated
Table is stored up, non-selected ephemeral data storage table is determined as invalid data storage.
In conjunction with the third embodiment of second aspect, the present invention is described invalid in the 4th embodiment of second aspect
Data resource removes module：
Invalid data stores offline unit, is stored for offline invalid data.
In conjunction with second aspect, for the present invention in the 5th embodiment of second aspect, the invalid data resource confirms mould
Block further includes：
Invalid computation confirmation unit is used for when data resource is to calculate operation, then according to the output for calculating operation
Daily record judge the calculating operation whether abnormal interrupt；If so, the calculating operation of abnormal interrupt is determined as invalid computation operation.
In conjunction with second aspect, for the present invention in the sixth embodiment of second aspect, the invalid data resource confirms mould
Block further includes：
Ex-employee's data resource confirmation unit, for tying up according to various types of data resources and human resource system
Determine relationship, it is determined whether there are the invalid data resources of ex-employee, wherein the invalid data resource of ex-employee includes leaving office
At least one of invalid computation operation, invalid data storage and invalid data calling of employee.
In conjunction with second aspect, for the present invention in the 7th embodiment of second aspect, the invalid data resource confirms mould
Block further includes：
Business datum resource confirmation unit is closed, for tying up according to various types of data resources and business management system
Determine relationship, it is determined whether there is the invalid data resource for the business of being turned off, wherein the invalid data resource for being turned off business includes
It is turned off the invalid of business and can be regarded as at least one of industry, invalid data storage and invalid data calling.
In conjunction with second aspect the 5th embodiment, sixth embodiment and the 7th embodiment in one kind, the present invention
In the 8th embodiment of second aspect, the invalid data resource removes module and further includes：
The offline unit of invalid data resource is used for the offline closing of invalid computation operation, and is cleared up the invalid computation and made
Industry rubbish for generating in history calculating, or invalid data stored offline, or invalid data called offline.
The third aspect, an embodiment of the present invention provides a kind of removing equipment of invalid data resource, the functions of the equipment
Corresponding software realization can also be executed by hardware by hardware realization.The hardware or software include one or more
A module corresponding with above-mentioned function.
In a possible design, the structure of the removing equipment of invalid data resource includes that processor and storage fill
It sets, the storage device, which is used to store, supports the remove device of invalid data resource to execute method described in above-mentioned first aspect
Program, the processor are configurable for executing the program stored in the storage device.The invalid data resource it is clear
Except equipment can also include communication interface, removing equipment and other equipment or communication for invalid data resource.
Fourth aspect, an embodiment of the present invention provides a kind of computer readable storage mediums, for storing invalid data money
Computer software instructions used in the removing equipment in source comprising for executing the clear of invalid data resource in above-mentioned first aspect
Except the program involved by the remove device that method is invalid data resource.
A technical solution in above-mentioned technical proposal has the following advantages that or advantageous effect：In the present solution, according to invalid
Mining rule is searched in various types of data resources, is determined and be whether there is invalid data in various types of data resources
Resource, if so, removing invalid data resource according to the corresponding reset mode of invalid data resource, invalid data resource includes nothing
Imitate at least one of inquiry job, invalid computation operation, invalid data storage and invalid data calling.This programme avoids
Since calculating, invalid calculating, storage, the calling etc. that are repeated during data analysis application occupy money in data analysis process
Source, caused by the wasting of resources.
Above-mentioned general introduction is merely to illustrate that the purpose of book, it is not intended to be limited in any way.Except foregoing description
Schematical aspect, except embodiment and feature, by reference to attached drawing and the following detailed description, the present invention is further
Aspect, embodiment and feature, which will be, to be readily apparent that.
Description of the drawings
In the accompanying drawings, unless specified otherwise herein, otherwise run through the identical reference numeral of multiple attached drawings and indicate same or analogous
Component or element.What these attached drawings were not necessarily to scale.It should be understood that these attached drawings are depicted only according to the present invention
Some disclosed embodiments, and should not serve to limit the scope of the present invention.
Fig. 1 is a kind of flow chart of the sweep-out method of invalid data resource provided in an embodiment of the present invention；
Fig. 2 is a kind of schematic diagram of the sweep-out method of invalid data resource provided in an embodiment of the present invention；
Fig. 3 is a kind of remove device structure chart of invalid data resource provided in an embodiment of the present invention；
Fig. 4 is a kind of removing equipment structure chart of invalid data resource provided in an embodiment of the present invention.
Specific implementation mode
Hereinafter, certain exemplary embodiments are simply just described.As one skilled in the art will recognize that
Like that, without departing from the spirit or scope of the present invention, described embodiment can be changed by various different modes.
Therefore, attached drawing and description are considered essentially illustrative rather than restrictive.
In existing data analysis, different business departments selects same data to make for common data analysis requirements
Industry, the phenomenon that causing the repetition and the wasting of resources of data operation.In order to avoid the wasting of resources, this application provides be implemented as follows
Mode, to solve existing issue.
Embodiment one
In a kind of specific embodiment, as shown in Figure 1, providing a kind of sweep-out method of invalid data resource, wrap
It includes：
Step S100：It is searched, is determined various types of in various types of data resources according to invalid mining rule
It whether there is invalid data resource in data resource.
Include various types of departments in the tissues such as company, school, government, market, such as Finance Department, public relation department
Door, Human Resource Department and business department etc., different types of department is corresponding with data resource, constitutes different types of
Data resource includes some invalid data resources in various types of data resources, is provided for various types of data
Source is excavated in vain respectively, for example, by the scanning to data resource to obtain total data, then it is directed to different types of wait for
The data resource of excavation is searched according to different invalid mining rules respectively, determines in data resource to be excavated whether deposit
In invalid data resource.
Step S200：If so, removing invalid data resource, invalid number according to the corresponding reset mode of invalid data resource
Include at least one of invalid inquiry job, invalid computation operation, invalid data storage and invalid data calling according to resource.
As shown in Figure 2, on the one hand, the data resource for repeating operation or storage occupies resource, the inquiry job such as repeated, weight
Multiple data storage etc. carries out integration to the data resource repeated and unitizes, and the data resource repeated is offline, can prevent wave
Take.On the other hand, detect that some export the data resource that frequently interrupts, this kind of data operation be set as invalid, and by its
It removes.The third aspect, by with the employee to have left office or and close the relevant data resource of business be set as invalid, such as nothing
Calculating operation, data storage and the data call of effect, and removed.
In the present embodiment, the sweep-out method of the invalid data resource provided a kind of avoid in data analysis process due to
Calculating, invalid calculating, storage, the calling etc. repeated during data analysis application occupies resource, caused by resource wave
Take.
In one implementation, it is searched, is determined each in various types of data resources according to invalid mining rule
It whether there is invalid data resource in the data resource of type, including：
When data resource is inquiry job, the similarity between each inquiry job is calculated；
An inquiry job is selected from each inquiry job that the similarity is more than threshold value, non-selected inquiry is made
Industry is determined as invalid inquiry job.
In the present embodiment, referring to Fig. 2, inquiry job such as HQL (the Hibernate Query of foundation
Language, the query language of object-oriented) it may include according in query type, inquiry table, inquiry field and queries etc.
One or more parameters are inquired, and are repeated inquiry job (such as HQL) to find out in next step and are prepared.Inquiry is established to make
After industry, whether the inquiry job of more each department is consistent.The mode compared can calculate the corresponding inquiry of each department
Similarity between operation.If similarity be more than threshold value, as threshold value be 1 when, illustrate that inquiry job is closer to, for example, if
The inquiry result of calculation of two HQL is consistent, it is believed that the two HQL are similar, and one of inquiry job may be selected later
It shares, non-selected inquiry job is determined as invalid inquiry job.
In one implementation, invalid data resource is removed according to the corresponding reset mode of invalid data resource, including：
The inquiry job selected is executed, query result is obtained, common data fairground table is established according to query result, is deleted
Invalid inquiry job retains the strongest common data fairground table of timeliness.
For example, being looked into respectively the data resource of each department according to the highest inquiry job of the similarity extracted
It askes.It can be shared due to inquiring obtained result, the shared data inquired are formed into the strongest common data sets of timeliness
City's table (datamart, can abbreviation MART tables) when carrying out inquiry job to wherein a certain department so as to next time, omits inquiry and makees
The execution of industry, but directly data are extracted from this common data fairground table.For example, it is more than 1 HQL sentences for similarity,
Establish public MART tables.The HQL repeated is offline, retain the strongest production MART tables of a timeliness.Each department is no longer
It needs to compute repeatedly same part data, evidence of fetching in the strongest MART tables of timeliness.In addition, the repeat number that inquiry obtains
According to can remove.
In one implementation, it is searched, is determined each in various types of data resources according to invalid mining rule
It whether there is invalid data resource in the data resource of type, including：
When data resource is that data store, each ephemeral data storage table is scanned；
Judge whether the ephemeral data storage table scanned repeats, is selected from the ephemeral data storage table repeated
Non-selected ephemeral data storage table is determined as invalid data storage by one ephemeral data storage table.
In one implementation, invalid data resource is removed according to the corresponding reset mode of invalid data resource, including：
Offline invalid data storage.Meanwhile retaining the strongest ephemeral data storage table of timeliness.
Referring to Fig. 2, often there are same part data in different business line or department similar analysis dimension, this occurs as soon as difference
Service line causes Data duplication to store, storage resource waste by storing job storage with the possibility of a ephemeral data.
In present embodiment, by scanning the corresponding ephemeral data storage table of each business department, find that duplicate data storage is asked in time
It inscribes, selecting one from each ephemeral data storage table repeated is used in conjunction with, and retains the best ephemeral data of timeliness and deposits
Table is stored up, offline non-selected ephemeral data storage table solves such problem of resource waste.
In one implementation, it is searched, is determined each in various types of data resources according to invalid mining rule
It whether there is invalid data resource in the data resource of type, including：
When data resource is to calculate operation, then judge calculating operation whether in exception according to the output journal for calculating operation
It is disconnected；If so, the calculating operation of abnormal interrupt is determined as invalid computation operation.
In general, invalid computation operation is no longer valuable calculating operation, type includes a variety of, such as the meter frequently to fail
Can be regarded as industry, ex-employee calculating operation, the calculating operation etc. that is turned off business, referring to Fig. 2.Wherein, the meter frequently to fail
It can be regarded as industry, the calculating operation etc. of unmanned concern does not only take up computing resource, also without any value.Therefore, invalid computation can be made
Industry is excavated, will no longer valuable calculating operation it is offline in time, can be to avoid the wasting of resources.Wherein, specific excavation side
Method is to determine whether invalid computation operation according to the output journal obtained after execution for calculating operation, such as can pass through setting
Whether the threshold value of the number frequently to fail is abnormal to judge this calculating operation, if the frequency of failure of the output recorded in output journal
More than this threshold value, then judge that this calculates operation as invalid computation operation.Invalid computation operation is removed in time.For example, right
These invalid operations carry out offline closing, and history calculates the rubbish (garbage) generated also cleaning in time.
In one implementation, it is searched, is determined each in various types of data resources according to invalid mining rule
It whether there is invalid data resource in the data resource of type, including：
According to the binding relationship of various types of data resources and human resource system, it is determined whether there are ex-employee's
Invalid data resource, wherein the invalid data resource of ex-employee includes that the invalid computation operation of ex-employee, invalid data are deposited
At least one of storage and invalid data calling.
Since many business departments all have labor turnover, there are related operations such as to calculate operation, data storage and number
According to unsharp situation is called, causing after causing labor turnover, ephemeral data storage is deleted not in time, still occupies storage resource,
And invalid data operation continues to operation, expends computing resource.In the present embodiment, setting data store and manpower
The mechanism of the mechanism and data operation and human resource system's binding validatation of resource system binding validatation, it is ensured that leaving office person
The ephemeral data of work stores and data call task, and anergy is perceived, is offline in time after failure in time.
In one implementation, it is searched, is determined each in various types of data resources according to invalid mining rule
It whether there is invalid data resource in the data resource of type, including：
According to the binding relationship of various types of data resources and business management system, it is determined whether there is the business of being turned off
Invalid data resource, wherein the invalid data resource for being turned off business includes that the invalid of the business that is turned off can be regarded as industry, invalid number
At least one of called according to storage and invalid data.
Since each business department has the business being turned off, and the storage of data and the operation of data operation are in public affairs
Occur in big data platform altogether, if business is closed, but can not with the relevant data storage of the business of closing and data operation
It is offline in time.In the present embodiment, it can be arranged and calculate operation, data storage and data call and business management system binding validatation
Mechanism, it is ensured that be turned off the data call task of business, anergy is perceived in time, fail after it is offline in time.
Referring to Fig. 2, when labor turnover or business closing, it is relevant invalid with labor turnover and business closing to excavate
Calculating operation, data storage and data call, and the invalid data and data operation excavated are purged, are kept away
Exempt from waste of resource.
Invalid data resource is removed according to the corresponding reset mode of invalid data resource, including：
By the offline closing of invalid computation operation, and the rubbish that invalid computation operation generates in history calculating is cleared up, or will
Invalid data storage is offline, or invalid data is called offline.
Embodiment two
In another specific embodiment, as shown in figure 3, a kind of remove device of invalid data resource is provided,
Including：
Invalid data resource lookup module 10, for being looked into various types of data resources according to invalid mining rule
It looks for；
Invalid data resource confirmation module 20 is provided for determining in various types of data resources with the presence or absence of invalid data
Source；
Invalid data resource removes module 30, if in various types of data resources there are invalid data resource,
Invalid data resource then is removed according to the corresponding reset mode of invalid data resource, invalid data resource includes that invalid inquiry is made
At least one of industry, invalid computation operation, invalid data storage and invalid data calling.
Further, in above-mentioned apparatus, invalid data resource confirmation module includes：
Similarity calculated, for when data resource is inquiry job, calculating similar between each inquiry job
Degree；
Acknowledgment of your inquiry unit is repeated, the inquiry job for similarity to be more than to threshold value is determined as the inquiry job repeated,
The inquiry job repeated is determined as invalid inquiry job.
Further, in above-mentioned apparatus, invalid data resource removes module and includes：
Data Mart table establishes unit, for executing inquiry job, obtains query result, is established according to query result public
Data Mart table deletes the inquiry job of repetition, retains the strongest common data fairground table of timeliness.
Further, in above-mentioned apparatus, invalid data resource confirmation module further includes：
Storage table scanning element, for when data resource is that data store, scanning each ephemeral data storage table；
Invalid data stores confirmation unit, and whether the ephemeral data storage table for judging to have scanned repeats, by what is repeated
Ephemeral data storage table is determined as invalid data storage.
Further, in above-mentioned apparatus, invalid data resource removes module and further includes：
The offline unit of invalid storage table is used for the ephemeral data storage table of offline repetition, and it is strongest interim to retain timeliness
Table data store.
Further, in above-mentioned apparatus, invalid data resource confirmation module further includes：
Invalid computation confirmation unit is used for when data resource is to calculate operation, then according to the output journal for calculating operation
Judge calculate operation whether abnormal interrupt；If so, the calculating operation of abnormal interrupt is determined as invalid computation operation.
Further, in above-mentioned apparatus, invalid data resource confirmation module further includes：
Ex-employee's data resource confirmation unit, for tying up according to various types of data resources and human resource system
Determine relationship, it is determined whether there are the invalid data resources of ex-employee, wherein the invalid data resource of ex-employee includes leaving office
At least one of invalid computation operation, invalid data storage and invalid data calling of employee.
Further, in above-mentioned apparatus, invalid data resource confirmation module further includes：
Business datum resource confirmation unit is closed, for tying up according to various types of data resources and business management system
Determine relationship, it is determined whether there is the invalid data resource for the business of being turned off, wherein the invalid data resource for being turned off business includes
It is turned off the invalid of business and can be regarded as at least one of industry, invalid data storage and invalid data calling.
Further, in above-mentioned apparatus, invalid data resource removes module and further includes：
The offline unit of invalid data resource is used for the offline closing of invalid computation operation, and is cleared up the invalid computation and made
Industry rubbish for generating in history calculating, or invalid data stored offline, or invalid data called offline.
Embodiment three
An embodiment of the present invention provides a kind of removing equipment of invalid data resource, as shown in figure 4, including：
Memory 400 and processor 500 are stored with the computer journey that can be run on processor 500 in memory 400
Sequence.Processor 500 realizes the sound control intelligent household equipment in above-described embodiment method when executing the computer program.Storage
The quantity of device 400 and processor 500 can be one or more.
Communication interface 600 is communicated for memory 400 and processor 500 with outside.
Memory 400 may include high-speed RAM memory, it is also possible to further include nonvolatile memory (non-
Volatile memory), a for example, at least magnetic disk storage.
If memory 400, processor 500 and the independent realization of communication interface 600, memory 400, processor 500
And communication interface 600 can be connected with each other by bus and complete mutual communication.The bus can be industrial standard
Architecture (ISA, Industry Standard Architecture) bus, external equipment interconnection (PCI, Peripheral
Component) bus or extended industry-standard architecture (EISA, Extended Industry Standard
Component) bus etc..The bus can be divided into address bus, data/address bus, controlling bus etc..For ease of indicating, Fig. 3
In only indicated with a thick line, it is not intended that an only bus or a type of bus.
Optionally, in specific implementation, if memory 400, processor 500 and communication interface 600 are integrated in one piece
On chip, then memory 400, processor 500 and communication interface 600 can complete mutual communication by internal interface.
Example IV
A kind of computer readable storage medium is stored with computer program, is realized when described program is executed by processor
Embodiment one include it is any as described in invalid data resource sweep-out method.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.Moreover, particular features, structures, materials, or characteristics described
It may be combined in any suitable manner in any one or more of the embodiments or examples.In addition, without conflicting with each other, this
The technical staff in field can be by the spy of different embodiments or examples described in this specification and different embodiments or examples
Sign is combined.
In addition, term " first ", " second " are used for description purposes only, it is not understood to indicate or imply relative importance
Or implicitly indicate the quantity of indicated technical characteristic." first " is defined as a result, the feature of " second " can be expressed or hidden
Include at least one this feature containing ground.In the description of the present invention, the meaning of " plurality " is two or more, unless otherwise
Clear specific restriction.
Any process described otherwise above or method description are construed as in flow chart or herein, and expression includes
It is one or more for realizing specific logical function or process the step of executable instruction code module, segment or portion
Point, and the range of the preferred embodiment of the present invention includes other realization, wherein can not press shown or discuss suitable
Sequence, include according to involved function by it is basic simultaneously in the way of or in the opposite order, to execute function, this should be of the invention
Embodiment person of ordinary skill in the field understood.
Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered use
In the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable medium, for
Instruction execution system, device or equipment (system of such as computer based system including processor or other can be held from instruction
The instruction fetch of row system, device or equipment and the system executed instruction) it uses, or combine these instruction execution systems, device or set
It is standby and use.For the purpose of this specification, " computer-readable medium " can any can be included, store, communicating, propagating or passing
Defeated program is for instruction execution system, device or equipment or the dress used in conjunction with these instruction execution systems, device or equipment
It sets.The more specific example (non-exhaustive list) of computer-readable medium includes following：Electricity with one or more wiring
Interconnecting piece (electronic device), portable computer diskette box (magnetic device), random access memory (RAM), read-only memory
(ROM), erasable edit read-only storage (EPROM or flash memory), fiber device and portable read-only memory
(CDROM).In addition, computer-readable medium can even is that the paper that can print described program on it or other suitable Jie
Matter, because can be for example by carrying out optical scanner to paper or other media, then into edlin, interpretation or when necessary with other
Suitable method is handled electronically to obtain described program, is then stored in computer storage.
It should be appreciated that each section of the present invention can be realized with hardware, software, firmware or combination thereof.Above-mentioned
In embodiment, software that multiple steps or method can in memory and by suitable instruction execution system be executed with storage
Or firmware is realized.It, and in another embodiment, can be under well known in the art for example, if realized with hardware
Any one of row technology or their combination are realized：With the logic gates for realizing logic function to data-signal
Discrete logic, with suitable combinational logic gate circuit application-specific integrated circuit, programmable gate array (PGA), scene
Programmable gate array (FPGA) etc..
Those skilled in the art are appreciated that realize all or part of step that above-described embodiment method carries
Suddenly it is that relevant hardware can be instructed to complete by program, the program can be stored in a kind of computer-readable storage medium
In matter, which includes the steps that one or a combination set of embodiment of the method when being executed.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, it can also
That each unit physically exists alone, can also two or more units be integrated in a module.Above-mentioned integrated mould
The form that hardware had both may be used in block is realized, can also be realized in the form of software function module.The integrated module is such as
Fruit is realized in the form of software function module and when sold or used as an independent product, can also be stored in a computer
In readable storage medium storing program for executing.The storage medium can be read-only memory, disk or CD etc..
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can readily occur in its various change or replacement,
These should be covered by the protection scope of the present invention.Therefore, protection scope of the present invention should be with the guarantor of the claim
It protects subject to range.
Claims (20)
1. a kind of sweep-out method of invalid data resource, which is characterized in that including：
Searched in various types of data resources according to invalid mining rule, determine in various types of data resources whether
There are invalid data resources；
If so, removing invalid data resource according to the corresponding reset mode of invalid data resource, invalid data resource includes nothing
Imitate at least one of inquiry job, invalid computation operation, invalid data storage and invalid data calling.
2. according to the method described in claim 1, it is characterized in that, according to invalid mining rule in various types of data resources
It is searched, determines and whether there is invalid data resource in various types of data resources, including：
When data resource is inquiry job, the similarity between each inquiry job is calculated；
An inquiry job is selected from each inquiry job that the similarity is more than threshold value, non-selected inquiry job is true
It is set to invalid inquiry job.
3. according to the method described in claim 2, it is characterized in that, removing nothing according to the corresponding reset mode of invalid data resource
Data resource is imitated, including：
The inquiry job selected is executed, query result is obtained；
Common data fairground table is established according to the query result；
Delete invalid inquiry job.
4. according to the method described in claim 1, it is characterized in that, according to invalid mining rule in various types of data resources
It is searched, determines and whether there is invalid data resource in various types of data resources, including：
When data resource is that data store, each ephemeral data storage table is scanned；
Judge whether the ephemeral data storage table scanned repeats, one is selected from the ephemeral data storage table repeated
Non-selected ephemeral data storage table is determined as invalid data storage by ephemeral data storage table.
5. according to the method described in claim 1, it is characterized in that, removing nothing according to the corresponding reset mode of invalid data resource
Data resource is imitated, including：
Offline invalid data storage.
6. according to the method described in claim 1, it is characterized in that, according to invalid mining rule in various types of data resources
It is searched, determines and whether there is invalid data resource in various types of data resources, including：
When data resource is to calculate operation, then judge whether the calculating operation is different according to the output journal for calculating operation
Often interrupt；If so, the calculating operation of abnormal interrupt is determined as invalid computation operation.
7. according to the method described in claim 1, it is characterized in that, according to invalid mining rule in various types of data resources
It is searched, determines and whether there is invalid data resource in various types of data resources, including：
According to the binding relationship of various types of data resources and human resource system, it is determined whether there are the invalid of ex-employee
Data resource, wherein the invalid data resource of ex-employee include the invalid computation operation of ex-employee, invalid data storage and
At least one of invalid data calling.
8. according to the method described in claim 1, it is characterized in that, according to invalid mining rule in various types of data resources
It is searched, determines and whether there is invalid data resource in various types of data resources, including：
According to the binding relationship of various types of data resources and business management system, it is determined whether there is the nothing for the business of being turned off
Imitate data resource, wherein the invalid data resource for being turned off business include the business that is turned off it is invalid can be regarded as industry, invalid data is deposited
At least one of storage and invalid data calling.
9. the method according to any one of claim 6 to 8, which is characterized in that corresponding clear according to invalid data resource
Except mode removes invalid data resource, including：
By the offline closing of invalid computation operation, and clear up the rubbish that the invalid computation operation generates in history calculating；Or
Invalid data is stored offline；Or
Invalid data is called offline.
10. a kind of remove device of invalid data resource, which is characterized in that including：
Invalid data resource lookup module, for being searched in various types of data resources according to invalid mining rule；
Invalid data resource confirmation module whether there is invalid data resource for determining in various types of data resources；
Invalid data resource removes module, if in various types of data resources there are invalid data resource, according to
The corresponding reset mode of invalid data resource removes invalid data resource, and invalid data resource includes invalid inquiry job, invalid
Calculate at least one of operation, invalid data storage and invalid data calling.
11. device as claimed in claim 10, which is characterized in that the invalid data resource confirmation module includes：
Similarity calculated is used for when data resource is inquiry job, similar between each inquiry job of calculating
Degree；
Acknowledgment of your inquiry unit is repeated, is made for selecting an inquiry from each inquiry job that the similarity is more than threshold value
Non-selected inquiry job is determined as invalid inquiry job by industry.
12. device as claimed in claim 11, which is characterized in that the invalid data resource removes module and includes：
Invalid inquiry job clearing cell obtains query result, is tied according to the inquiry for executing the inquiry job selected
Fruit establishes common data fairground table, deletes invalid inquiry job.
13. device as claimed in claim 10, which is characterized in that the invalid data resource confirmation module further includes：
Storage table scanning element, for when data resource is that data store, scanning each ephemeral data storage table；
Invalid data stores confirmation unit, for selecting an ephemeral data storage from the ephemeral data storage table repeated
Non-selected ephemeral data storage table is determined as invalid data storage by table.
14. device as claimed in claim 13, which is characterized in that the invalid data resource removes module and further includes：
Invalid data stores offline unit, is stored for offline invalid data.
15. device as claimed in claim 10, which is characterized in that the invalid data resource confirmation module further includes：
Invalid computation confirmation unit is used for when data resource is to calculate operation, then according to the output journal for calculating operation
Judge the calculating operation whether abnormal interrupt；If so, the calculating operation of abnormal interrupt is determined as invalid computation operation.
16. device as claimed in claim 10, which is characterized in that the invalid data resource confirmation module further includes：
Ex-employee's data resource confirmation unit, for being closed according to the binding of various types of data resources and human resource system
System, it is determined whether there are the invalid data resources of ex-employee, wherein the invalid data resource of ex-employee includes ex-employee
Invalid computation operation, invalid data storage and invalid data call at least one of.
17. device as claimed in claim 10, which is characterized in that the invalid data resource confirmation module further includes：
Business datum resource confirmation unit is closed, for being closed according to the binding of various types of data resources and business management system
System, it is determined whether there is the invalid data resource for the business of being turned off, wherein the invalid data resource for being turned off business includes having closed
The invalid of business that close can be regarded as at least one of industry, invalid data storage and invalid data calling.
18. such as claim 15 to 17 any one of them device, which is characterized in that the invalid data resource removes module also
Including：
The offline unit of invalid data resource is used for the offline closing of invalid computation operation, and clears up the invalid computation operation and exist
The rubbish that generates during history calculates, or invalid data stored offline, or invalid data called offline.
19. a kind of removing equipment of invalid data resource, which is characterized in that including：
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors so that one or more of processors
Realize the method as described in any in claim 1-9.
20. a kind of computer readable storage medium, is stored with computer program, which is characterized in that the computer program quilt
The method as described in any in claim 1-9 is realized when processor executes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810302134.7A CN108536812B (en) | 2018-04-04 | 2018-04-04 | Method, device and equipment for clearing invalid data resources and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810302134.7A CN108536812B (en) | 2018-04-04 | 2018-04-04 | Method, device and equipment for clearing invalid data resources and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108536812A true CN108536812A (en) | 2018-09-14 |
| CN108536812B CN108536812B (en) | 2020-05-08 |
Family
ID=63483234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810302134.7A Active CN108536812B (en) | 2018-04-04 | 2018-04-04 | Method, device and equipment for clearing invalid data resources and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108536812B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112116303A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Operation ticket early warning system |
| CN112116173A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Invalid operation reduction system |
| CN112116118A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Operation reduction early warning system based on data mining |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6185552B1 (en) * | 1998-03-19 | 2001-02-06 | 3Com Corporation | Method and apparatus using a binary search engine for searching and maintaining a distributed data structure |
| CN104317627A (en) * | 2014-11-13 | 2015-01-28 | 北京奇虎科技有限公司 | Clear data one click scan method and device of storage space |
| CN104317628A (en) * | 2014-11-13 | 2015-01-28 | 北京奇虎科技有限公司 | Mobile terminal and storage space cleaning method thereof |
| CN104317534A (en) * | 2014-10-29 | 2015-01-28 | 小米科技有限责任公司 | Method and device for cleaning data storage space |
| US20160004446A1 (en) * | 2010-06-28 | 2016-01-07 | Western Digital Technologies, Inc. | System and method for performing data retention in solid-state memory |
| US20170199903A1 (en) * | 2014-04-25 | 2017-07-13 | Bank Of America Corporation | System for backing out data |
-
2018
- 2018-04-04 CN CN201810302134.7A patent/CN108536812B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6185552B1 (en) * | 1998-03-19 | 2001-02-06 | 3Com Corporation | Method and apparatus using a binary search engine for searching and maintaining a distributed data structure |
| US20160004446A1 (en) * | 2010-06-28 | 2016-01-07 | Western Digital Technologies, Inc. | System and method for performing data retention in solid-state memory |
| US20170199903A1 (en) * | 2014-04-25 | 2017-07-13 | Bank Of America Corporation | System for backing out data |
| CN104317534A (en) * | 2014-10-29 | 2015-01-28 | 小米科技有限责任公司 | Method and device for cleaning data storage space |
| CN104317627A (en) * | 2014-11-13 | 2015-01-28 | 北京奇虎科技有限公司 | Clear data one click scan method and device of storage space |
| CN104317628A (en) * | 2014-11-13 | 2015-01-28 | 北京奇虎科技有限公司 | Mobile terminal and storage space cleaning method thereof |
Non-Patent Citations (2)
| Title |
|---|
| LI TIAN等: "A New Filtering Method of Laser Scanner Data for Automated Vehicle Obstacle Recognition in Unstructured Environment", 《IEEE》 * |
| 周芝芬: "基于数据仓库的数据清洗方法研究", 《中国优秀博硕士学位论文全文数据库 (硕士) 信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112116303A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Operation ticket early warning system |
| CN112116173A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Invalid operation reduction system |
| CN112116118A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Operation reduction early warning system based on data mining |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108536812B (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Mesiar et al. | Characterization of invariant aggregation operators | |
| CN108536812A (en) | Sweep-out method, device, equipment and the computer-readable medium of invalid data resource | |
| CN109344153A (en) | The processing method and terminal device of business datum | |
| CN111652732A (en) | Bit currency abnormal transaction entity identification method based on transaction graph matching | |
| CN112633842B (en) | Task pushing method, device and system | |
| CN111291229A (en) | Method and system for detecting dense multi-part graphs | |
| Michalak et al. | A logic-based representation for coalitional games with externalities | |
| Cole | (Infra) structural discontinuity: capital, labour, and technological change | |
| CN112598510B (en) | Resource data processing method and device | |
| CN111784488B (en) | Enterprise fund risk prediction method and device | |
| CN106681803A (en) | Task scheduling method and server | |
| CN108256816A (en) | A kind of commission data processing method, device and equipment | |
| Lamson | Planning for resilient coastal communities: Lessons from ecological systems theory | |
| CN114416819A (en) | Mining method and device for public clients | |
| Wan et al. | A new algorithm for enumerating all maximal cliques in complex network | |
| Kim | Population monotonic rules for fair allocation problems | |
| CN108921490A (en) | A kind of sending method of notification information, storage medium and server | |
| Modani et al. | Discovery and analysis of tightly knit communities in telecom social networks | |
| Shi et al. | The Interval‐Valued Intuitionistic Fuzzy Optimized Weighted Bonferroni Means and Their Application | |
| Li et al. | A conceptual approach to modeling the procurement process of construction using petri-nets | |
| CN110826055B (en) | Tenant safety automatic intelligent detection method and system based on service load | |
| Shiu et al. | Modelling Hybrid Rule/Frame-Based Expert Systems Using Coloured Petri Nets. | |
| Wu | Constraint programming applications in designing electronic agents: An experimental study | |
| Goldman | What is a Number? | |
| Harrison | Why Your Operation Should Have an Agriculture Environmental Management System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |