CN110489407A - Data filling mining method, apparatus, computer equipment and storage medium - Google Patents
Data filling mining method, apparatus, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110489407A CN110489407A CN201910690813.0A CN201910690813A CN110489407A CN 110489407 A CN110489407 A CN 110489407A CN 201910690813 A CN201910690813 A CN 201910690813A CN 110489407 A CN110489407 A CN 110489407A
- Authority
- CN
- China
- Prior art keywords
- data
- terminal
- filling mining
- task
- state bitmap
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Abstract
The present invention relates to data filling mining method, apparatus, computer equipment and storage mediums, belong to network technique field.The described method includes: determining to supplementary data；It is determined to supplementary data according to first state bitmap, first state bitmap is used for the existence of characterize data；Obtain terminal identification information corresponding with first state bitmap；According to supplementary data and terminal identification information, filling mining task is generated；According to filling mining task, filling mining control instruction is sent to the first collection terminal；Filling mining control instruction obtains corresponding filling mining data according to filling mining task for controlling the first collection terminal.Above-mentioned technical proposal solves the problems, such as that data filling mining efficiency is lower.Determine that the access for not needing to carry out storing data one by one can quickly generate filling mining task, effectively improve the efficiency of data filling mining to supplementary data according to state bitmap.
Description
Technical field
The present invention relates to network technique fields, are situated between more particularly to data filling mining method, apparatus, computer equipment and storage
Matter.
Background technique
Collection terminal generally requires to processing terminal, processing end etc. to carry out data transmission collected at processing end data
Reason.But the transmit process of data is easy to be influenced by environmental disturbances, meanwhile, part collection terminal, which has operation task queue, leads
The problem of causing Missing data, it is not 100% reliable that this, which allows for data acquired in processing end,.This is just to need to not upper
The data sent or acquired are acquired again, i.e. progress filling mining.
In realizing process of the present invention, inventor has found that at least there are the following problems: traditional data filling mining in traditional approach
It is simple rely on database one by one statistical data the shortcomings that, establish the mapping relations with collection terminal, form filling mining task, and then realize
Data filling mining.Not only process is complicated for such filling mining mode, but also accesses a large amount of collection terminals for processing end or need to receive
The case where mass data, the filling mining task for establishing substantial amounts require a great deal of time, this allows for data filling mining efficiency
It is lower.
Summary of the invention
Based on this, the embodiment of the invention provides data filling mining method, apparatus, computer equipment and storage mediums, can have
Effect improves data filling mining efficiency.
The content of the embodiment of the present invention is as follows:
In a first aspect, the embodiment of the present invention provides a kind of data filling mining method, comprising the following steps: determine number to be supplemented
According to；Described to be determined to supplementary data according to first state bitmap, the first state bitmap is used for the existence of characterize data；
Obtain terminal identification information corresponding with the first state bitmap；Believed according to described to supplementary data and the terminal iidentification
Breath generates filling mining task；According to the filling mining task, filling mining control instruction is sent to the first collection terminal；The filling mining control refers to
It enables and obtains corresponding filling mining data according to the filling mining task for controlling first collection terminal.
In one embodiment, before the step of determination is to supplementary data, further includes: refer to receiving filling mining triggering
When enabling, by the first memory library file in scanning memory bank, the first state bitmap is obtained；Wherein, the memory bank is
The database of data storage is carried out according to the storage mode of memory.
In one embodiment, before the step of acquisition first state bitmap, further includes: receive the second collection terminal hair
The terminal task data sent；The terminal class of temporal information and second collection terminal is carried in the terminal task data
Type；According to the terminal type and the temporal information, the second memory library file is established.
In one embodiment, the terminal address of second collection terminal is also carried in the terminal task data；Institute
After the step of stating according to the terminal type and the temporal information, establishing the second memory library file, further includes: to described
The terminal address of two collection terminals runs hash algorithm, obtains record identification；It judges whether there is and the terminal task data pair
The acquisition data answered；According to judging result, corresponding second state bitmap is determined；According to the record identification in described second
The second state bitmap is recorded in warehousing file.
In one embodiment, described according to the filling mining task, the step of filling mining control instruction is sent to the first collection terminal
After rapid, further includes: receive the filling mining data that the first collection terminal is sent；Terminal address is carried in the filling mining data；According to
The terminal address carried in filling mining data positions and modifies corresponding state bitmap in the memory bank.
In one embodiment, described the step of receiving the terminal task data that the second collection terminal is sent, comprising: receive the
The terminal task data that two collection terminals are sent by message queue.
In one embodiment, described the step of sending filling mining control instruction to the first collection terminal, comprising: pass through message team
It arranges to the first collection terminal and sends filling mining control instruction.
In one embodiment, the terminal identification information includes terminal type and terminal address；It is described according to
To supplementary data and the terminal identification information, the step of generating filling mining task, comprising: according to described to supplementary data, institute
Terminal type and the terminal address are stated, the filling mining task is generated.
Second aspect, the embodiment of the present invention provide a kind of data filling mining device, comprising: data determining module, for determining
To supplementary data；Described to be determined to supplementary data according to first state bitmap, the first state bitmap is for characterize data
Existence；Data obtaining module, for obtaining terminal identification information corresponding with the first state bitmap；Task generates mould
Block, for, to supplementary data and the terminal identification information, generating filling mining task according to described；Instruction sending module is used for
According to the filling mining task, filling mining control instruction is sent to the first collection terminal；The filling mining control instruction is for controlling described the
One collection terminal acquires corresponding data according to the filling mining task.
The third aspect, the embodiment of the present invention provide a kind of computer equipment, including memory and processor, the memory
It is stored with computer program, the processor performs the steps of when executing the computer program to be determined to supplementary data；Institute
It states and is determined to supplementary data according to first state bitmap, the first state bitmap is used for the existence of characterize data；It obtains
Terminal identification information corresponding with the first state bitmap；According to described to supplementary data and the terminal identification information,
Generate filling mining task；According to the filling mining task, filling mining control instruction is sent to the first collection terminal；The filling mining control instruction is used
Corresponding filling mining data are obtained according to the filling mining task in controlling first collection terminal.
Fourth aspect, the embodiment of the present invention provide a kind of computer readable storage medium, are stored thereon with computer program,
The computer program performs the steps of when being executed by processor to be determined to supplementary data；It is described to supplementary data according to
One state bitmap determines that the first state bitmap is used for the existence of characterize data；It obtains and the first state bitmap
Corresponding terminal identification information；According to described to supplementary data and the terminal identification information, filling mining task is generated；According to institute
Filling mining task is stated, sends filling mining control instruction to the first collection terminal；The filling mining control instruction is for controlling first acquisition
End obtains corresponding filling mining data according to the filling mining task.
A technical solution in above-mentioned technical proposal has the following advantages that or the utility model has the advantages that true according to first state bitmap
The state for determining not needing to access data to supplementary data one by one can quickly generate filling mining task, effectively improve data benefit
The efficiency adopted.
Detailed description of the invention
Fig. 1 is the applied environment figure of data filling mining method in one embodiment；
Fig. 2 is the flow diagram of data filling mining method in one embodiment；
Fig. 3 is the schematic diagram of memory library file in one embodiment；
Fig. 4 is the schematic diagram that data filling mining method is realized in another embodiment；
Fig. 5 is the structural block diagram of data filling mining device in one embodiment；
Fig. 6 is the internal structure of computer equipment in one embodiment.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
Referenced herein " embodiment " is it is meant that a particular feature, structure, or characteristic described can wrap in conjunction with the embodiments
It is contained at least one embodiment of the application.Each position in the description occur the phrase might not each mean it is identical
Embodiment, nor the independent or alternative embodiment with other embodiments mutual exclusion.Those skilled in the art explicitly and
Implicitly understand, embodiment described herein can be combined with other embodiments.
Data filling mining method provided by the present application can be applied in application environment as shown in Figure 1.The application environment packet
Processing end 101 and collection terminal 102 are included, the two can carry out network communication.Wherein, processing end 101 generates according to state bitmap and mends
Task is adopted, and then controls the filling mining that collection terminal 102 carries out data.In practical application scene, processing end 101 can be service
Device, specifically, can be realized with the server cluster of the either multiple server compositions of independent server.Certainly, it handles
End 101 or terminal, can be, but not limited to be various personal computers, laptop, smart phone, tablet computer and
Portable wearable device.Collection terminal 102 can be the various equipment with data acquisition function, device etc., such as: sensing
Device, data sink, ammeter etc..
The embodiment of the present invention provides a kind of data filling mining method, apparatus, computer equipment and storage medium.Individually below into
Row is described in detail.
In one embodiment, as shown in Fig. 2, providing a kind of data filling mining method.It is applied in Fig. 1 in this way
It is illustrated for processing end 101, comprising the following steps:
S201, it determines to supplementary data；It is determined to supplementary data according to first state bitmap, first state bitmap is used for table
Levy the existence of data.
Wherein, collection terminal can be power information acquisition terminal etc..
Whether the existence that first state bitmap can store the acquired data of each collection terminal (can be understood as being written
The corresponding database in processing end), corresponding file can be established for each collection terminal, include first state bitmap in file,
And first state bitmap can acquire data be written in memory when become 1, without acquisition data or acquisition data not at
Become 0 when in function write-in memory.Further, collection terminal data collected may be more huge, therefore can be according to acquisition
Type, address and the task number at end etc. carry out the differentiation of file record.
Specifically, the determination process of first state bitmap can be with are as follows: processing end inquiry acquisition data whether there is, according to number
According to existence determine the value of first state bitmap, will be A pairs with terminal for example, the data for inquiring a certain terminal A are not present
The value for the first state bitmap answered is determined as 0, and the data for inquiring a certain terminal B exist, will first state corresponding with terminal B
The value of bitmap is determined as 1.It should be noted that the initial value of state bitmap can be 0, it, can be in the presence of having acquisition data
Its value is revised as 1.
Based on this, processing end can determine data incomplete therein according to first state bitmap, and then obtain number to be supplemented
According to.
S202, acquisition terminal identification information corresponding with first state bitmap.
This step determines terminal identification information corresponding with first state bitmap.
In some embodiments, this terminal identification information can refer to including terminal type and terminal address.Wherein, eventually
End type can be understood as the type of collection terminal, model etc..Terminal address can be understood as the MAC Address of collection terminal, IP address
Deng, it is understood that it is the number of collection terminal.
S203, basis generate filling mining task to supplementary data and terminal identification information.
Wherein, filling mining task is referred to mission bit stream generated when needing to carry out data filling mining.
In one embodiment, terminal identification information includes terminal type and terminal address；According to supplementary data with
And terminal identification information, the step of generating filling mining task, comprising: raw according to supplementary data, terminal type and terminal address
At filling mining task.
Be exemplified below to filling mining task generated: 1, collection terminal A does not upload a1 data in time；2, collection terminal B will not
B2 data upload in time.
S204, according to filling mining task, send filling mining control instruction to the first collection terminal；Filling mining control instruction is for controlling the
One collection terminal obtains corresponding filling mining data according to filling mining task.
Wherein, the first collection terminal, which can be directed toward in the corresponding data of processing end, provides the original terminal of data, can also refer to specially
Door carries out the terminal of data filling mining.For second situation, the first collection terminal can be obtained by way of accessing original terminal
Filling mining data can also directly acquire corresponding data and obtain filling mining data.
With the development of big data analysis technology, the requirement of real-time of data acquisition is higher and higher.Acquire real time data not
But it is required that data the acquisition granularity very little (second grade), and it is very high to the integrity demands of data, it can just meet big data point in this way
The business demand of analysis technology.But traditional data filling mining method has the following problems: 1, filling mining task needs to spend a large amount of
Time, 2, processing end need to establish mapping relations with collection terminal, 3, data bank service table often temporally put storing data, these
Traffic table saves even several annual datas some months, and data volume is very big, and filling mining is usually per diem filling mining, this just needs database
The every day statistics deletion condition of traffic table, calculation amount are very big (scanning one by one).Problems above results in data filling mining
Inefficiency.And data filling mining method provided in this embodiment, according to first state bitmap determine to supplementary data, do not need by
A access for carrying out storing data, can quickly generate filling mining task, effectively improve the efficiency of data filling mining.
In one embodiment, before the step of determining to supplementary data, further includes: receiving filling mining triggering command
When, by the first memory library file in scanning memory bank, obtain first state bitmap；Wherein, memory bank is according to memory
The database of storage mode progress data storage.
Wherein, filling mining triggering command can be needed to send by some terminal according to business；Timer setting can also be passed through
Timed task is sent by timer when reaching the time, is able to achieve the regular filling mining of data in this way.
First state bitmap in the embodiment of the present invention is stored in the memory library file of memory bank, and collection terminal is collected
Data can realize data persistence with separate storage in memory, will not influence underlying programs upper procedure failure
Data: processing end break down cause internal storage data to be lost when, reboot process end, processing end is according in memory library file
First state bitmap can know the relevant information for the data that should be stored in memory, thus can carry out filling mining, prevent memory
The loss of data.In addition, the embodiment of the present invention uses mask data library business, using the memory bank of file-type, task search and
Scanning does not influence the service operation of database completely.
In some embodiments, it can according to need and establish multiple memory library files in memory bank, these memory library texts
Part can be established according to the terminal type of collection terminal, terminal address, temporal information, such as: it is directed to different terminal classes respectively
Type, terminal address, time establish different memory library files.It the case where for establishing memory library file according to the time, can be with every
As soon as it establishes a memory library file, the data such as state bitmap that the same day obtains are stored in corresponding memory library file.
Wherein, temporal information can refer to the data that the time correlation of data is acquired with collection terminal, can refer to that collection terminal carries out
Acquisition time when data acquire is also possible to the sending time that collection terminal sends acquisition data to processing end, can also be place
Manage the receiving time that end receives the acquisition data that collection terminal is sent.Meanwhile temporal information is when may include the date and is specific
Every minute and second etc. also may include the information such as frequency acquisition, history acquisition time, history times of collection.
Further, before the step of obtaining first state bitmap, further includes: receive the terminal that the second collection terminal is sent and appoint
Business data；The terminal type of temporal information and the second collection terminal is carried in terminal task data；According to terminal type and when
Between information, establish the second memory library file.
Wherein, terminal task data can refer to the data of evaluation the acquired data of collection terminal, may include time letter
The information such as breath, terminal type, terminal address, task number.Further, different acquisition end can send multiple terminal task numbers
According to the task number can be understood as the serial number for each terminal task data that a certain collection terminal is sent, it is understood that be each
The overall serial number (collection terminal not being distinguished) for each terminal task data that collection terminal is sent.
Further, while receiving the terminal task data that the second collection terminal is sent, the second acquisition can also be received
Data collected are held, processing end can be by the second collection terminal data deposit memory collected (i.e., it is possible to terminal task number
According to being stored separately) in.
In addition, it is necessary to explanation, the first collection terminal and the second collection terminal can refer to same terminal, or different whole
End.Further, the data of the first collection terminal and the second collection terminal all can be one, two even more than.
Above-described embodiment, by memory library file come recording status bitmap, when carrying out the inquiry of state bitmap, positioning is corresponded to
Memory library file, can quick obtaining to corresponding state bitmap, and then according to state bitmap determination to supplementary data, energy
Effectively improve the efficiency of data filling mining.
In one embodiment, the terminal address of the second collection terminal is also carried in terminal task data；According to terminal class
Type and temporal information, after the step of establishing the second memory library file, further includes: the terminal address of the second collection terminal is run
Hash algorithm obtains record identification；Judge whether there is acquisition data corresponding with terminal task data；According to judging result,
Determine corresponding second state bitmap；The second state bitmap is recorded in the second memory library file according to record identification.
Wherein, whole operation Hash to terminal address be can be to the terminal address operation hash algorithm of the second collection terminal
Algorithm is also possible to a part operation hash algorithm therein.The terminal address of structure this for area code+serial number, can
To carry out hash algorithm to serial number therein, when carrying out the inquiry of state bitmap using serial number as key assignments, such processing
Mode can effectively improve the location efficiency of state bitmap, and then improve the efficiency of data filling mining, lifting system processing speed.
In some embodiments, it is also possible to run other algorithms to terminal address.It is, of course, also possible to directly by terminal
The all or part of location is as record identification, without running hash algorithm.
The realization process for judging whether there is acquisition data corresponding with terminal task data can be with are as follows: audit memory is empty
Between, it determines whether to be written with acquisition data corresponding with the terminal task data.Further, it judges whether there is and whole
Hold the corresponding acquisition data of task data；According to judging result, determine that the realization process of corresponding second state bitmap can be with are as follows:
If it is determined that being written with corresponding acquisition data in memory, corresponding second state bitmap is determined as 1；If it is determined that not having in memory
Corresponding acquisition data are written, corresponding second state bitmap is determined as 0.
In some embodiments, state bitmap with a position (minimum unit of data storage) in addition to being indicated；It can be with
It is indicated with multiple positions, these positions can also characterize acquisition other than it can characterize in memory and whether have corresponding acquisition data
The integrality of data.Such as: 00 indicates without acquisition data, and 01 indicates to lack partial data, and 10 indicate partial data damage, 11 tables
Registration is according to complete and correct.
In some embodiments, in memory library file other than recording status bitmap, with can recorde terminal type, terminal
Location, temporal information, task number etc., to be distinguished to each record.Memory library file can be in as shown in figure 3, Fig. 3 shows
Partial memory library file 302 in warehousing 301.The memory bank establishes memory library file according to the date in temporal information (i.e.
One day correspondence, one memory library file, certainly, memory library file can also distinguish in other manners).Below with
For Article 3 record (00000101) on July 20th, 2019 corresponding memory library file, to what is recorded in memory bank file
Information explains:
This record can be divided into following four part: 00 00 010 1, wherein 6-7 (since left several 1-2
Position) indicate terminal type, 4-5 indicated to the record identification after terminal address operation hash algorithm, 1-3 expression times
Information, the 0th expression state bitmap.
Wherein, the temporal information recorded in memory library file can be to collection terminal send temporal information handle after
Information.Such as: it was divided into 8 equal portions for one day, each equal portions are indicated with three positions, such as: 00:00 indicates with 000,03:00
It is indicated with 001,06:00 is indicated with 010 ... ..., and 21:00 is indicated with 111.What such mode can not only send each time
Terminal task data are recorded, and can also occupy memory space as few as possible (can be recorded completely by a byte
Terminal task data), effectively improve the utilization rate of memory space.
Certainly, the record in memory library file may be other forms (information of each expression can be different),
Every records occupied byte number and can also be different, such as can be 2 bytes, 3 bytes etc..
In addition, state bitmap can also only indicate temporal information and data existence, and terminal type, terminal
Location, task number etc. can store by other means.
Above-described embodiment stores terminal task data by several bytes (or even only needing a byte), occupied
Memory space is small；Meanwhile when carrying out terminal task data query, does not need to access a large amount of data, terminal can be effectively improved
The search efficiency of task data, and then improve the efficiency of data filling mining.
In one embodiment, according to filling mining task, after the step of sending filling mining control instruction to the first collection terminal, also
It include: the filling mining data for receiving the first collection terminal and sending；Terminal address is carried in filling mining data；It is carried according in filling mining data
Terminal address, position and modify corresponding state bitmap in memory bank.
In some embodiments, the positioning of state bitmap can combine terminal address and terminal type, temporal information etc.,
It is of course also possible to carry out the positioning of state bitmap separately through terminal type, temporal information etc..
Above-described embodiment is when receiving filling mining data, positioning states bitmap, in time to the state bitmap positioned into
Row modification can prevent when progress data filling mining there is no need to carry out filling mining to data corresponding with the state bitmap next time
The case where repeating filling mining.
In one embodiment, the step of receiving the terminal task data that the second collection terminal is sent, comprising: receive second and adopt
The terminal task data that collection end is sent by message queue.
Further, the step of sending filling mining control instruction to the first collection terminal, comprising: adopted by message queue to first
Collect end and sends filling mining control instruction.
Above-described embodiment realizes that the data between processing end and the first collection terminal, the second collection terminal are handed over by message queue
Mutually, the pressure of memory can be effectively relieved when data volume is big.
In one embodiment, processing end can be automatic metering system.Automatic metering system is measured as energy source interconnection
The important component of net is able to satisfy distributed energy access and management by technologies such as intelligent meter application, big data analysis
Demand, adapt to electric car business development need, effectively support expense control system, the service ability that develops skill and competitiveness.
The above method in order to better understand, as shown in figure 4, by taking processing end is automatic metering system as an example, in detailed below
Illustrate the application example of a data filling mining method of the present invention.401 in Fig. 4 indicate automatic metering system.
1, automatic metering system receives the terminal task data that collection terminal is sent by message queue；
2, automatic metering system carries out task analysis to terminal task data, extract terminal type therein, terminal address,
Task number；
3, automatic metering system positions corresponding state bitmap in memory bank according to terminal address, task number etc., to state
Bitmap modifies (have acquisition data is changed to 1)；
4, automatic metering system scans each state bitmap in memory bank, by it when receiving filling mining triggering command
The relevant information (terminal type, terminal address, task number, temporal information etc.) that middle state bitmap is 0 extracts, and gives birth to accordingly
At filling mining task；
5, automatic metering system carries out filling mining task by message queue controlling terminal filling mining module.
In electrical network field, with the increment of power consumer, terminal access amount will be increased significantly, and new situations are to electric flux
Data requirements is more and more, and the data item and density of user power utilization information collection will all increase therewith, and data volume will substantially increase
Add, existing software architecture and hardware and software platform have been unsatisfactory for following access demand, cannot support the acquisition of big data quantity
Demand.Current network system be on the whole from acquisition freezing data to acquisition near-realtime data transition stage.By acquiring
Near-realtime data will be following trend to acquisition real time data transition.Quasi real time collection and real-time data acquisition not only require
Data the acquisition granularity very little (second grade), and it is very high to the integrity demands of data, it can just meet the industry of big data analysis in this way
Business demand.The network mainstream that metering main website is given in the acquisition of metering automation terminal is wireless public network (2G, 4G, 5G etc.), is limited
The unstable characteristic and part terminal operating task queue problem of wireless network being easy by environmental disturbances, it cannot be guaranteed that number
It is reliable according to acquisition 100%, then in time on send the data (claiming task data) of (acquisition) need to rely on main website on demand into
Row filling mining guarantees the integrality of business as far as possible.Data filling mining method provided in this embodiment is determined according to state bitmap wait supplement
Data, and then filling mining task is generated, this makes the generation of filling mining task not need the state for checking data one by one, can quickly generate
Filling mining task, effectively improves the efficiency of data filling mining, while can guarantee the integrality of business datum.
It should be noted that for the various method embodiments described above, describing for simplicity, it is all expressed as a series of
Combination of actions, but those skilled in the art should understand that, the present invention is not limited by the sequence of acts described, because according to
According to the present invention, certain steps can use other sequences or carry out simultaneously.
Based on thought identical with the data filling mining method in above-described embodiment, the present invention also provides data filling mining device,
The device can be used for executing above-mentioned data filling mining method.For ease of description, the structural schematic diagram of data filling mining Installation practice
In, it illustrate only part related to the embodiment of the present invention, it will be understood by those skilled in the art that schematic structure is not constituted
Restriction to device may include perhaps combining certain components or different component cloth than illustrating more or fewer components
It sets.
As shown in figure 5, data filling mining device includes data determining module 501, data obtaining module 502, task generation mould
Block 503 and instruction sending module 504, detailed description are as follows:
Data determining module 501, for determining to supplementary data；It is determined to supplementary data according to first state bitmap, the
One state bitmap is used for the existence of characterize data.
Data obtaining module 502, for obtaining terminal identification information corresponding with first state bitmap.
Task generation module 503, for generating filling mining task according to supplementary data and terminal identification information.
Instruction sending module 504, for sending filling mining control instruction to the first collection terminal according to filling mining task；Filling mining control
System instruction acquires corresponding data according to filling mining task for controlling the first collection terminal.
The present embodiment determines to supplementary data according to first state bitmap, does not need the access for carrying out storing data one by one,
Filling mining task can be quickly generated, the efficiency of data filling mining is effectively improved.
In one embodiment, further includes: file scan determining module, for leading to when receiving filling mining triggering command
The first memory library file in memory bank is over-scanned, first state bitmap is obtained；Wherein, memory bank is the storage side according to memory
The database of formula progress data storage.
In one embodiment, further includes: data reception module, for receiving the terminal task number of the second collection terminal transmission
According to；The terminal type of temporal information and the second collection terminal is carried in terminal task data；File establishes module, is used for basis
Terminal type and temporal information establish the second memory library file.
In one embodiment, the terminal address of the second collection terminal is also carried in terminal task data；Further include: algorithm
Computing module runs hash algorithm for the terminal address to the second collection terminal, obtains record identification；Judgment module, for sentencing
It is disconnected to whether there is acquisition data corresponding with terminal task data；State bitmap determining module, for determining according to judging result
Corresponding second state bitmap；Mode bit figure logging modle, for recording in the second memory library file according to record identification
Two-state bitmap.
In one embodiment, further includes: filling mining data reception module, for receiving the filling mining number of the first collection terminal transmission
According to；Terminal address is carried in filling mining data；State bitmap modified module, for according to the terminal carried in filling mining data
Location positions and modifies corresponding state bitmap in memory bank.
In one embodiment, data reception module is also used to receive the end that the second collection terminal is sent by message queue
Hold task data.
In one embodiment, instruction sending module 504 are also used to send filling mining to the first collection terminal by message queue
Control instruction.
In one embodiment, terminal identification information includes terminal type and terminal address；Task generation module 503,
It is also used to generate filling mining task according to supplementary data, terminal type and terminal address.
It should be noted that data filling mining device of the invention and data filling mining method of the invention correspond, upper
The technical characteristic and its advantages of stating the embodiment elaboration of data filling mining method are suitable for the embodiment of data filling mining device
In, for details, please refer to the description in the embodiment of the method for the present invention, and details are not described herein again, hereby give notice that.
In addition, the logical partitioning of each program module is only citing in the embodiment of the data filling mining device of above-mentioned example
Illustrate, can according to need in practical application, such as is examined for the convenience of the realization of the configuration requirement or software of corresponding hardware
Consider, above-mentioned function distribution is completed by different program modules, i.e., is divided into the internal structure of data filling mining device different
Program module, to complete all or part of the functions described above.
Data filling mining method provided by the present application can be applied in computer equipment as shown in FIG. 6.The computer is set
It is standby to can be processing end, it is also possible to terminal device, internal structure chart can be as shown in Figure 6.The computer equipment includes logical
Cross processor, memory, network interface and the database of system bus connection.Wherein, processor is calculated and is controlled for providing
Ability；Memory includes non-volatile memory medium, built-in storage, which is stored with operating system, meter
Calculation machine program (computer program realizes a kind of data filling mining method when being executed by processor) and database, the built-in storage are
The operation of operating system and computer program in non-volatile memory medium provides environment；Database filling mining for storing data
The various data that method is related to during realizing；Network interface is used to communicate with external terminal by network connection, such as:
It is connect with collection terminal.
It will be understood by those skilled in the art that structure shown in Fig. 6, only part relevant to application scheme is tied
The block diagram of structure does not constitute the restriction for the computer equipment being applied thereon to application scheme, specific computer equipment
It may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.
In one embodiment, a kind of computer equipment, including memory and processor are provided, memory is stored with meter
Calculation machine program, processor performs the steps of when executing computer program to be determined to supplementary data；To supplementary data according to first
State bitmap determines that first state bitmap is used for the existence of characterize data；Obtain terminal corresponding with first state bitmap
Identification information；According to supplementary data and terminal identification information, filling mining task is generated；According to filling mining task, to the first acquisition
End sends filling mining control instruction；Filling mining control instruction obtains corresponding filling mining number according to filling mining task for controlling the first collection terminal
According to.
In one embodiment, it is also performed the steps of when processor executes computer program and is receiving filling mining triggering
When instruction, by the first memory library file in scanning memory bank, first state bitmap is obtained；Wherein, memory bank is according to interior
The storage mode deposited carries out the database of data storage.
In one embodiment, it is also performed the steps of when processor executes computer program and receives the second collection terminal hair
The terminal task data sent；The terminal type of temporal information and the second collection terminal is carried in terminal task data；According to end
Type and temporal information are held, the second memory library file is established.
In one embodiment, the terminal address of the second collection terminal is also carried in terminal task data；Processor executes
It is also performed the steps of when computer program and hash algorithm is run to the terminal address of the second collection terminal, obtain record identification；Sentence
It is disconnected to whether there is acquisition data corresponding with terminal task data；According to judging result, corresponding second state bitmap is determined；Root
The second state bitmap is recorded in the second memory library file according to record identification.
In one embodiment, it is also performed the steps of when processor executes computer program and receives the first collection terminal hair
The filling mining data sent；Terminal address is carried in filling mining data；According to the terminal address carried in filling mining data, positions and modify
Corresponding state bitmap in memory bank.
In one embodiment, it is logical that the second collection terminal of reception is also performed the steps of when processor executes computer program
Cross the terminal task data of message queue transmission.
In one embodiment, it is also performed the steps of when processor executes computer program through message queue to the
One collection terminal sends filling mining control instruction.
In one embodiment, terminal identification information includes terminal type and terminal address；Processor executes computer
It is also performed the steps of when program according to supplementary data, terminal type and terminal address, generates filling mining task.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program performs the steps of when being executed by processor to be determined to supplementary data；It is true according to first state bitmap to supplementary data
Fixed, first state bitmap is used for the existence of characterize data；Obtain terminal identification information corresponding with first state bitmap；Root
According to supplementary data and terminal identification information, filling mining task is generated；According to filling mining task, filling mining control is sent to the first collection terminal
System instruction；Filling mining control instruction obtains corresponding filling mining data according to filling mining task for controlling the first collection terminal.
In one embodiment, it is also performed the steps of when computer program is executed by processor and is receiving filling mining touching
When sending instructions, by the first memory library file in scanning memory bank, first state bitmap is obtained；Wherein, memory bank be according to
The storage mode of memory carries out the database of data storage.
In one embodiment, it is also performed the steps of when processor executes computer program and receives the second collection terminal hair
The terminal task data sent；The terminal type of temporal information and the second collection terminal is carried in terminal task data；According to end
Type and temporal information are held, the second memory library file is established.
In one embodiment, the terminal address of the second collection terminal is also carried in terminal task data；Processor executes
It is also performed the steps of when computer program and hash algorithm is run to the terminal address of the second collection terminal, obtain record identification；Sentence
It is disconnected to whether there is acquisition data corresponding with terminal task data；According to judging result, corresponding second state bitmap is determined；Root
The second state bitmap is recorded in the second memory library file according to record identification.
In one embodiment, it is also performed the steps of when processor executes computer program and receives the first collection terminal hair
The filling mining data sent；Terminal address is carried in filling mining data；According to the terminal address carried in filling mining data, positions and modify
Corresponding state bitmap in memory bank.
In one embodiment, it is logical that the second collection terminal of reception is also performed the steps of when processor executes computer program
Cross the terminal task data of message queue transmission.
In one embodiment, it is also performed the steps of when processor executes computer program through message queue to the
One collection terminal sends filling mining control instruction.
In one embodiment, terminal identification information includes terminal type and terminal address；Processor executes computer
It is also performed the steps of when program according to supplementary data, terminal type and terminal address, generates filling mining task.
It will appreciated by the skilled person that realizing all or part of the process in above-described embodiment method, being can
It is completed with instructing relevant hardware by computer program, the program can be stored in a computer-readable storage and be situated between
In matter, sells or use as independent product.The more specific example (non-exhaustive list) of computer-readable medium includes
Below: there is the electrical connection section (electronic device) of one or more wirings, portable computer diskette box (magnetic device), arbitrary access
Memory (RAM), read-only memory (ROM), erasable edit read-only storage (EPROM or flash memory), optical fiber dress
It sets and portable optic disk read-only storage (CDROM).In addition, computer-readable medium, which can even is that, to be printed on it
The paper of described program or other suitable media, because can be for example by carrying out optical scanner to paper or other media, then
It edited, interpreted or is handled when necessary with other suitable methods electronically to obtain described program, then by it
Storage is in computer storage.
It should be appreciated that each section of the invention can be realized with hardware, software, firmware or their combination.Above-mentioned
In embodiment, software that multiple steps or method can be executed in memory and by suitable instruction execution system with storage
Or firmware is realized.It, and in another embodiment, can be under well known in the art for example, if realized with hardware
Any one of column technology or their combination are realized: having a logic gates for realizing logic function to data-signal
Discrete logic, with suitable combinational logic gate circuit specific integrated circuit, programmable gate array (PGA), scene
Programmable gate array (FPGA) etc..
The term " includes " of the embodiment of the present invention and " having " and their any deformations, it is intended that cover non-exclusive
Include.Such as contain series of steps or the process, method, system, product or equipment of (module) unit are not limited to
The step of listing or unit, but optionally further comprising the step of not listing or unit, or optionally further comprising for these
The intrinsic other step or units of process, method, product or equipment.
Each technical characteristic of embodiment described above can be combined arbitrarily, for simplicity of description, not to above-mentioned reality
It applies all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited
In contradiction, all should be considered as described in this specification.
The embodiments described above only express several embodiments of the present invention, should not be understood as to the invention patent range
Limitation.It should be pointed out that for those of ordinary skill in the art, without departing from the inventive concept of the premise,
Various modifications and improvements can be made, and these are all within the scope of protection of the present invention.Therefore, the scope of protection of the patent of the present invention
It should be determined by the appended claims.
Claims (10)
1. a kind of data filling mining method, which comprises the following steps:
It determines to supplementary data；Described to be determined to supplementary data according to first state bitmap, the first state bitmap is used for table
Levy the existence of data；
Obtain terminal identification information corresponding with the first state bitmap；
According to described to supplementary data and the terminal identification information, filling mining task is generated；
According to the filling mining task, filling mining control instruction is sent to the first collection terminal；The filling mining control instruction is for controlling institute
It states the first collection terminal and obtains corresponding filling mining data according to the filling mining task.
2. the method according to claim 1, wherein before the step of determination is to supplementary data, further includes:
When receiving filling mining triggering command, by the first memory library file in scanning memory bank, the first state is obtained
Bitmap；Wherein, the memory bank is the database that data storage is carried out according to the storage mode of memory.
3. according to the method described in claim 2, it is characterized in that, also being wrapped before the step of the acquisition first state bitmap
It includes:
Receive the terminal task data of the second collection terminal transmission；Temporal information and described is carried in the terminal task data
The terminal type of second collection terminal；
According to the terminal type and the temporal information, the second memory library file is established.
4. according to the method described in claim 3, being adopted it is characterized in that, also carrying described second in the terminal task data
Collect the terminal address at end；
It is described according to the terminal type and the temporal information, after the step of establishing the second memory library file, further includes:
Hash algorithm is run to the terminal address of second collection terminal, obtains record identification；
Judge whether there is acquisition data corresponding with the terminal task data；
According to judging result, corresponding second state bitmap is determined；
The second state bitmap is recorded in the second memory library file according to the record identification.
5. according to the method described in claim 3, being sent out to the first collection terminal it is characterized in that, described according to the filling mining task
After the step of sending filling mining control instruction, further includes:
Receive the filling mining data of the first collection terminal transmission；Terminal address is carried in the filling mining data；
According to the terminal address carried in filling mining data, positions and modify corresponding state bitmap in the memory bank.
6. according to the method described in claim 3, it is characterized in that,
Described the step of receiving the terminal task data that the second collection terminal is sent, comprising: receive the second collection terminal and pass through message team
Arrange the terminal task data sent；
And/or
Described the step of sending filling mining control instruction to the first collection terminal, comprising: sent by message queue to the first collection terminal
Filling mining control instruction.
7. method according to any one of claims 1 to 6, which is characterized in that the terminal identification information includes terminal class
Type and terminal address；
It is described according to supplementary data and the terminal identification information, the step of generating filling mining task, comprising:
According to described to supplementary data, the terminal type and the terminal address, the filling mining task is generated.
8. a kind of data filling mining device characterized by comprising
Data determining module, for determining to supplementary data；It is described to be determined to supplementary data according to first state bitmap, described the
One state bitmap is used for the existence of characterize data；
Data obtaining module, for obtaining terminal identification information corresponding with the first state bitmap；
Task generation module, for, to supplementary data and the terminal identification information, generating filling mining task according to described；
Instruction sending module, for sending filling mining control instruction to the first collection terminal according to the filling mining task；The filling mining control
System instruction acquires corresponding data according to the filling mining task for controlling first collection terminal.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists
In, the processor realizes claim 1 to 7 described in any item methods when executing computer program the step of.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of claim 1 to 7 described in any item methods are realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910690813.0A CN110489407A (en) | 2019-07-29 | 2019-07-29 | Data filling mining method, apparatus, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910690813.0A CN110489407A (en) | 2019-07-29 | 2019-07-29 | Data filling mining method, apparatus, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110489407A true CN110489407A (en) | 2019-11-22 |
Family
ID=68548576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910690813.0A Pending CN110489407A (en) | 2019-07-29 | 2019-07-29 | Data filling mining method, apparatus, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110489407A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111209271A (en) * | 2019-12-25 | 2020-05-29 | 深圳供电局有限公司 | Electric power data complementary acquisition method and device, computer equipment and storage medium |
| CN111798655A (en) * | 2020-05-29 | 2020-10-20 | 国网江苏省电力有限公司信息通信分公司 | Operation data minute-level acquisition method suitable for power Internet of things platform area |
| CN112008766A (en) * | 2020-09-03 | 2020-12-01 | 国网江苏省电力有限公司南通供电分公司 | RPA robot-based data recall automation method |
| CN112559505A (en) * | 2020-12-21 | 2021-03-26 | 深圳市航天泰瑞捷电子有限公司 | Real-time data complementary acquisition device and method, computer equipment and storage medium |
| CN113064793A (en) * | 2021-03-30 | 2021-07-02 | 山东英信计算机技术有限公司 | Bus equipment position information polling collection method, system, terminal and storage medium |
| CN113704341A (en) * | 2021-08-30 | 2021-11-26 | 远景智能国际私人投资有限公司 | Data acquisition system and method |
| CN114661513A (en) * | 2022-04-18 | 2022-06-24 | 广州菩润信息科技有限公司 | Distributed multi-source data acquisition method, system, equipment and storage medium |
| CN115665209A (en) * | 2022-12-09 | 2023-01-31 | 施维智能计量系统服务(长沙)有限公司 | Ammeter data acquisition method and device, electronic equipment and readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070143531A1 (en) * | 2005-12-15 | 2007-06-21 | Atri Sunil R | Power loss recovery for bit alterable memory |
| US20100268705A1 (en) * | 2009-03-13 | 2010-10-21 | David Douglas | Database and data access layer |
| CN107945048A (en) * | 2017-11-15 | 2018-04-20 | 广东电网有限责任公司电力科学研究院 | A kind of data additional collecting system based on metering automation system |
| CN109859611A (en) * | 2019-01-16 | 2019-06-07 | 北京百度网讯科技有限公司 | Acquisition method, device, equipment and the storage medium of map datum |
-
2019
- 2019-07-29 CN CN201910690813.0A patent/CN110489407A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070143531A1 (en) * | 2005-12-15 | 2007-06-21 | Atri Sunil R | Power loss recovery for bit alterable memory |
| US20100268705A1 (en) * | 2009-03-13 | 2010-10-21 | David Douglas | Database and data access layer |
| CN107945048A (en) * | 2017-11-15 | 2018-04-20 | 广东电网有限责任公司电力科学研究院 | A kind of data additional collecting system based on metering automation system |
| CN109859611A (en) * | 2019-01-16 | 2019-06-07 | 北京百度网讯科技有限公司 | Acquisition method, device, equipment and the storage medium of map datum |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111209271A (en) * | 2019-12-25 | 2020-05-29 | 深圳供电局有限公司 | Electric power data complementary acquisition method and device, computer equipment and storage medium |
| CN111798655A (en) * | 2020-05-29 | 2020-10-20 | 国网江苏省电力有限公司信息通信分公司 | Operation data minute-level acquisition method suitable for power Internet of things platform area |
| CN111798655B (en) * | 2020-05-29 | 2021-12-10 | 国网江苏省电力有限公司信息通信分公司 | Operation data minute-level acquisition method suitable for power Internet of things platform area |
| CN112008766A (en) * | 2020-09-03 | 2020-12-01 | 国网江苏省电力有限公司南通供电分公司 | RPA robot-based data recall automation method |
| CN112559505A (en) * | 2020-12-21 | 2021-03-26 | 深圳市航天泰瑞捷电子有限公司 | Real-time data complementary acquisition device and method, computer equipment and storage medium |
| CN113064793A (en) * | 2021-03-30 | 2021-07-02 | 山东英信计算机技术有限公司 | Bus equipment position information polling collection method, system, terminal and storage medium |
| CN113704341A (en) * | 2021-08-30 | 2021-11-26 | 远景智能国际私人投资有限公司 | Data acquisition system and method |
| CN114661513A (en) * | 2022-04-18 | 2022-06-24 | 广州菩润信息科技有限公司 | Distributed multi-source data acquisition method, system, equipment and storage medium |
| CN114661513B (en) * | 2022-04-18 | 2024-01-23 | 广州菩润信息科技有限公司 | Distributed multi-source data acquisition method, system, equipment and storage medium |
| CN115665209A (en) * | 2022-12-09 | 2023-01-31 | 施维智能计量系统服务(长沙)有限公司 | Ammeter data acquisition method and device, electronic equipment and readable storage medium |
| CN115665209B (en) * | 2022-12-09 | 2023-05-05 | 云南电网有限责任公司 | Ammeter data acquisition method and device, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489407A (en) | Data filling mining method, apparatus, computer equipment and storage medium | |
| CN102542071B (en) | Distributed data processing system and method | |
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| CN110019125A (en) | The method and apparatus of data base administration | |
| CN110765489A (en) | Multi-tenant database isolation method and system, electronic device and computer storage medium | |
| CN110399383A (en) | Applied to the data processing method of server, device, calculate equipment, medium | |
| CN111400246B (en) | Asynchronous file import method, device, computer equipment and storage medium | |
| CN108052569A (en) | Data bank access method, device, computer readable storage medium and computing device | |
| CN108710644A (en) | One kind is about government affairs big data processing method | |
| WO2023098374A1 (en) | Network resource deployment method and apparatus, and electronic device and storage medium | |
| CN111898007A (en) | Three-dimensional scene model construction system and method for transformer substation | |
| CN106503008A (en) | File memory method and device and file polling method and apparatus | |
| CN106021566A (en) | Method, device and system for improving concurrent processing capacity of single database | |
| CN111680900A (en) | Work order issuing method and device, electronic equipment and storage medium | |
| CN110019263A (en) | Information storage means and device | |
| CN111198918B (en) | Data processing system based on big data platform and link optimization method | |
| CN106326082A (en) | Method and device for recording log in network system | |
| CN114218318B (en) | Data processing system and method for electric power big data | |
| CN106776372B (en) | Emulation data access method and device based on FPGA | |
| CN108111578A (en) | The method of distribution terminal data acquisition platform access terminal equipment based on NIO | |
| CN107958345A (en) | A kind of SCD file collocation method and device based on database and file fragment | |
| CN110110153A (en) | A kind of method and apparatus of node searching | |
| CN106776255A (en) | The log extracting method and device of intelligent television system | |
| CN105677478A (en) | Method and apparatus for resources management | |
| CN110347654A (en) | A kind of method and apparatus of online cluster features |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191122 |
|
| RJ01 | Rejection of invention patent application after publication |