CN110347661A - Method, apparatus, server and the storage medium that data source is quasi real time put in storage - Google Patents
Method, apparatus, server and the storage medium that data source is quasi real time put in storage Download PDFInfo
- Publication number
- CN110347661A CN110347661A CN201910602946.8A CN201910602946A CN110347661A CN 110347661 A CN110347661 A CN 110347661A CN 201910602946 A CN201910602946 A CN 201910602946A CN 110347661 A CN110347661 A CN 110347661A
- Authority
- CN
- China
- Prior art keywords
- storage
- data
- time
- filename
- period
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method, apparatus that data source is quasi real time put in storage, server and storage mediums.This method comprises: obtaining the first data to be put in storage based on period first time；The document time of the described first data to be put in storage is obtained, the document time is distributed with second time period；First data to be put in storage are stored according to the second time period.By obtaining the first data to be put in storage within period first time, and the document time of the first data to be put in storage is stored according to second time period, reaches and quasi real time data are handled, improve the effect of the timeliness of data processing.
Description
Technical field
The present embodiments relate to big data processing field more particularly to a kind of methods that data source is quasi real time put in storage, dress
It sets, server and storage medium.
Background technique
With the rapid development of computer technology and Internet technology, the data of transmission are also more and more, therefore for number
According to timeliness require it is also higher and higher.
Currently, using the data processing policy of T+1, the i.e. same day to the original data processing of manufacturer in big data business
The data that the previous day manufacturer can only be pushed are put in storage.However, this storage mode timeliness is very poor, it is not able to satisfy big data
Requirement of the epoch for data processing high-timeliness.
Also, it is put in storage in real time if carrying out data with the real-time frame increased income on the market now, not only due to the scale of construction is huge
Cause exploitation debugging work load huge and high to server hardware performance requirement, it can not be on the server of low configuration
Operation.
Summary of the invention
The embodiment of the present invention provides a kind of method, apparatus that data source is quasi real time put in storage, server and storage medium, with reality
Now data source can be quasi real time put in storage.
In a first aspect, the embodiment of the invention provides a kind of methods that data source is quasi real time put in storage, comprising:
The first data to be put in storage are obtained based on period first time；
The document time of the described first data to be put in storage is obtained, the document time is distributed with second time period；
First data to be put in storage are stored according to the second time period.
Optionally, described to include: based on the first data to be put in storage of period first time acquisition
The first filename to be put in storage, the default scanning rule are obtained according to default scanning rule based on period first time
Including filename rule, scan path and sweep time window；
First data to be put in storage are obtained according to the described first filename to be put in storage.
It is optionally, described that first filename to be put in storage is obtained according to default scanning rule based on period first time, comprising:
It is obtained based on period first time in the scan path and meets the filename rule in the sweep time window
The described first filename to be put in storage then.
It is optionally, described that first data to be put in storage are obtained according to the described first filename to be put in storage, comprising:
It matches the described first filename to be put in storage and has pulled log；
It has pulled filename wait be put in storage in filename to reject described first and obtains the second filename to be put in storage；
First data to be put in storage are obtained according to the described second filename to be put in storage.
Optionally, in first based on period first time in scan path acquisition sweep time window wait be put in storage
After filename, comprising:
Judge whether the described first filename quantity to be put in storage is less than preset quantity threshold value；
If the first filename quantity to be put in storage is less than the preset quantity threshold value, the sweep time window is adjusted
Mouthful.
Optionally, described to include: according to second time period storage first data to be put in storage
It matches the described first data to be put in storage and described be put in storage log and obtain the second data to be put in storage；
Second data to be put in storage are stored according to the second time period.
Optionally, it is described according to the second time period storage described first wait be put in storage data after, comprising:
Judgement has been put in storage whether data reach life cycle；
If the data that have been put in storage reach the life cycle, deletion is described to be put in storage data.
Second aspect, the embodiment of the invention provides a kind of devices that data source is quasi real time put in storage, comprising:
Module is obtained, for obtaining the first data to be put in storage based on period first time；Obtain the described first number to be put in storage
According to document time, the document time is distributed with second time period；
Memory module, for storing first data to be put in storage according to the second time period.
The third aspect, the embodiment of the invention provides a kind of servers, comprising:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing
Device realizes the method that the data source as described in any embodiment of that present invention is quasi real time put in storage.
Fourth aspect, the embodiment of the invention provides a kind of computer readable storage mediums, are stored thereon with computer journey
Sequence realizes the method that the data source as described in any embodiment of that present invention is quasi real time put in storage when the program is executed by processor.
The embodiment of the present invention within period first time by obtaining the first data to be put in storage, and by the first data to be put in storage
Document time stored according to second time period, solve the problems, such as cannot in real time by data loading, realize can in real time it is right
Data are handled, and the effect of the timeliness of data processing is improved.
Detailed description of the invention
Fig. 1 is the flow chart for the method that a kind of data source that the embodiment of the present invention one provides quasi real time is put in storage；
Fig. 2 is the flow chart for the method that a kind of data source provided by Embodiment 2 of the present invention is quasi real time put in storage；
Fig. 3 is the flow chart for the method that a kind of data source provided by Embodiment 2 of the present invention is quasi real time put in storage；
Fig. 4 is the flow chart for the method that a kind of data source provided by Embodiment 2 of the present invention is quasi real time put in storage；
Fig. 5 is the flow chart for the method that a kind of data source that the embodiment of the present invention three provides quasi real time is put in storage；
Fig. 6 is the flow chart for the method that a kind of data source that the embodiment of the present invention three provides quasi real time is put in storage；
Fig. 7 is the structural schematic diagram for the device that a kind of data source that the embodiment of the present invention four provides quasi real time is put in storage；
Fig. 8 is a kind of structural schematic diagram for server that the embodiment of the present invention five provides.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limiting the invention.It also should be noted that in order to just
Only the parts related to the present invention are shown in description, attached drawing rather than entire infrastructure.
It should be mentioned that some exemplary embodiments are described as before exemplary embodiment is discussed in greater detail
The processing or method described as flow chart.Although each step is described as the processing of sequence by flow chart, many of these
Step can be implemented concurrently, concomitantly or simultaneously.In addition, the sequence of each step can be rearranged.When its operation
Processing can be terminated when completion, it is also possible to have the additional step being not included in attached drawing.Handle the side of can correspond to
Method, function, regulation, subroutine, subprogram etc..
In addition, term " first ", " second " etc. can be used to describe herein various directions, movement, step or element etc.,
But these directions, movement, step or element should not be limited by these terms.These terms are only used to by first direction, movement, step
Rapid or element and another direction, movement, step or element are distinguished.For example, the case where not departing from scope of the present application
Under, it can be second speed difference by First Speed difference, and similarly, it is poor second speed difference can be known as First Speed
Value.First Speed difference and second speed difference both speed difference, but it is not same speed difference.Term " the
One ", " second " etc. is not understood to indicate or imply relative importance or implicitly indicates the number of indicated technical characteristic
Amount." first " is defined as a result, the feature of " second " can explicitly or implicitly include one or more of the features.
In the description of the present invention, " multiple ", " batch " are meant that at least two, such as two, three etc., unless otherwise clearly having
The restriction of body.
Embodiment one
Fig. 1 is a kind of flow chart for data source quasi real time storage method that the embodiment of the present invention one provides, and the present embodiment can
Suitable for the scene for being put in storage manufacturer's initial data in real time, this method can be executed by server, be specifically included as follows
Step:
S110, the first data to be put in storage are obtained based on period first time.
Wherein, period first time refers to that scan period, scan period refer at regular intervals to manufacturer's initial data
Carry out single pass.Illustratively, when the scan period is 5 minutes, then manufacturer's initial data is scanned every 5 minutes.The
The setting of a period of time is related with the source period of manufacturer's initial data.Period first time longest must not exceed coming for manufacturer
The source period.Illustratively, the source period of manufacturer's initial data is 15 minutes, i.e., manufacturer carries out one to initial data in every 15 minutes
Secondary push, then period first time longest can only be 15 minutes.Period first time is longer, and timeliness is higher, but performance is opened
It sells also bigger.Preferably, when the source period of manufacturer is 15 minutes, period first time is 5 minutes.First data to be put in storage
Refer to the initial data from manufacturer being put in storage.In this step, it can be obtained in real time according to period first time
First data to be put in storage.
S120, the document time for obtaining first data to be put in storage, the document time are distributed with second time period.
Wherein, document time refers to that manufacturer generates the time of data.Specifically, manufacturer can generate one when generating data
A filename includes the time for generating the data in filename.Second time period refers to for by the original number of manufacturer
According to the time cycle for carrying out subregion distribution according to a time interval.Second time period can according to need setting, herein not
It is limited.Preferably, the setting of second time period is consistent with the source period of manufacturer data.The vendor source period is 15 minutes
When, second time period is also 15 minutes, can be with the source of manufacturer when the initial data of manufacturer being carried out subregion distribution to realize
Period to correspondence, to be more convenient later period trace back data.Illustratively, second time period is set as 15 minutes, then can be
9:00-9:15 is a subregion, and 9:15-9:30 is a subregion, every one subregion of setting in 15 minutes.
S130, first data to be put in storage are stored according to the second time period.
Wherein, it since the first data to be put in storage have the document time for generating the data, and is set according to second time period
Multiple period subregions are set.First data to be put in storage are stored according to second time period, can initial data to manufacturer into
Row presses subregion distribution, so that the later period traces data.In this step, storage refers to the first number to be put in storage that will have been obtained
According to being pushed under HDFS distributed file system specified directory.Likewise, this timing storage method can also be used when push.
The technical solution of the embodiment of the present invention can be in real time by obtaining the first data to be put in storage within period first time
Obtain the initial data of manufacturer.The document time of first data to be put in storage is stored according to second time period, thus by manufacturer
Initial data press subregion distribution, solve the problems, such as cannot in real time by data loading, reach can quasi real time to data at
Reason, improves the technical effect of the timeliness of data processing.The technical solution of the embodiment of the present invention is accomplished in performance and real-time
Optimal compatibility, not only ensure that real-time, but also can implement operation on lower performance configuration server.
Embodiment two
Fig. 2 is the flow chart that a kind of data source provided by Embodiment 2 of the present invention is quasi real time put in storage.The present embodiment is upper
The further refinement for stating technical solution, suitable for the scene for being put in storage manufacturer's initial data in real time.This method can be by taking
Business device executes, comprising:
S210, the first filename to be put in storage is obtained according to default scanning rule based on period first time, described preset is swept
Retouching rule includes filename rule, scan path and sweep time window.
Wherein, default scanning rule includes filename rule, scan path and sweep time window.Since original is stored by manufacturer
There is different filenames in the data source of beginning data, filename rule is for judging which data is to be obtained and be put in storage
's.Filename rule can be one, be also possible to multiple, be not particularly limited herein, according to the file for the data to be pulled
It determines.The mode that filename rule can be named or be scanned according to the initial data of manufacturer is configured, herein for filename
Specific rules form with no restrictions.Scan path refers to the path of storage manufacturer's initial data, just can be in scan path
The data needed are obtained, scan path can be one, be also possible to multiple, be not particularly limited herein.Scan path can be with
It is determined according to the path of the initial data of manufacturer storage, is not particularly limited herein.Sweep time window refers to that initial data arrives
Certain time is elapsed forward using current time as section terminal up to the time in scan path.Since the initial data of manufacturer is raw
At with certain retardance, sweep time window must not be lower than period first time, otherwise will lead to the data of partial time period
File can not be scanned.
By taking period first time is 5 minutes, sweep time window is 60 minutes as an example, if current time is 9:00, and
Period first time is reached exactly to, at this time then in scan path, obtains in 8:00-9:00 time period and reaches scan path simultaneously
And meet the data of filename rule.Since period first time is 5 minutes, then 9:05 timesharing will do it single pass again,
What is obtained at this time is the data that scan path is reached in 8:05-9:05 time period.First filename to be put in storage refers to all symbols
Close the set for the data that period first time acquires according to default scanning rule.Therefore, period first time can be based on
The described first filename to be put in storage for meeting the filename rule in the sweep time window is obtained in the scan path.
S220, first data to be put in storage are obtained according to the described first filename to be put in storage.
It in this step, then can be according to the first filename sum number to be put in storage due to having obtained the first filename to be put in storage
It is corresponded according to the raw data file name in source and obtains the first data to be put in storage.
S230, the document time for obtaining first data to be put in storage, the document time are distributed with second time period.
Wherein, document time refers to that manufacturer generates the time of data.Specifically, manufacturer can generate one when generating data
A filename includes the time for generating the data in filename.Second time period refers to for by the original number of manufacturer
According to the time cycle for carrying out subregion distribution according to a time interval.Second time period can according to need setting, herein not
It is limited.Preferably, the setting of second time period is consistent with the source period of manufacturer data.The vendor source period is 15 minutes
When, second time period is also 15 minutes, can be with the source of manufacturer when the initial data of manufacturer being carried out subregion distribution to realize
Period to correspondence, to be more convenient later period trace back data.
S240, first data to be put in storage are stored according to the second time period.
Wherein, it since the first data to be put in storage have the document time for generating the data, and is set according to second time period
Multiple subregions are set.First data to be put in storage are stored according to second time period, can the initial data to manufacturer carry out by
Subregion distribution, so that the later period traces data.In this step, storage refers to that the first data to be put in storage that will have been obtained push away
It is sent under HDFS distributed file system specified directory.Likewise, this timing storage method can also be used when push.
In the present embodiment, by the way that sweep time window is arranged, it can generate and postpone to avoid manufacturer data, and lead to part
Data can not be scanned.
With reference to Fig. 3, optionally, step S220 may include:
S221, it matches first filename to be put in storage and has pulled log.
Wherein, scanning acquisition first can generate when being put in storage filename and pull log every time, to facilitate confirmation and chase after
Trace back the filename to be put in storage that scanning obtains each time.Log has been pulled by needing the collection for being put in storage filename before present scan
It closes.Matching, which refers to, to be compared the first filename to be put in storage and has pulled the filename in log, to avoid according to first wait be put in storage
Filename obtains when being put in storage data, by duplicate data wasted storage server resource to be put in storage.
S222, it has pulled filename wait be put in storage in filename to reject described first and obtains the second filename to be put in storage.
Wherein, it has pulled filename and has referred to and pulled the filename in log.When sweep time window is greater than scanning week
When the phase, partial data can be repetitively scanned.Therefore, remaining first after being put in storage in filename and rejecting and pulled filename
Filename is the filename for the new data that present scan obtains.Second filename to be put in storage refers to the filename of all new datas
Set.
S223, first data to be put in storage are obtained according to the described second filename to be put in storage.
Wherein, the second filename to be put in storage all is the filename of new data, therefore is obtained according to the second filename to be put in storage
First data to be put in storage can obtain same data to avoid repetition.
In the present embodiment, file has been pulled by rejecting, same data can have been obtained to avoid repetition, improves server
Resource utilization.
With reference to Fig. 4, optional, include: after step S210
S211, judge whether the described first filename quantity to be put in storage is less than preset quantity threshold value.
Wherein, if data generate delay it is serious, data in sweep time window may it is considerably less very
To being not have.By judging the quantity of the first filename to be put in storage and the size of preset quantity threshold value, to judge current time
The severity of data delay.Preferably, preset quantity threshold value is 1.
If S212, the first filename quantity to be put in storage are less than the preset quantity threshold value, the scanning is adjusted
Time window.
Wherein, adjustment sweep time window refers to that by sweep time window, the range of passage is by window sliding toward left translation
Distance determines.By taking period first time is 5 minutes, sweep time window is 60 minutes, window sliding distance is 10 minutes as an example,
When current time is 9:00, period first time is reached exactly to, 8:00-9:00 time period is then scanned at this time and reaches scan path
In data.When being less than preset quantity threshold value due to the first quantity wait be put in storage filename, window sliding distance is adjusted.Work as the time
When reaching 9:05, since window sliding distance is 10 minutes, scans reach scanning road in 7:50-8:50 time period at this time
Data in diameter.Window sliding distance is remoter, and timeliness is poorer.
It in the present embodiment, can be when data generate significant delays, moreover it is possible to completely obtain by adjusting sweep time window
Obtain the initial data of manufacturer.
The technical solution of the embodiment of the present invention can be in real time by obtaining the first data to be put in storage within period first time
Obtain the initial data of manufacturer.The document time of first data to be put in storage is stored according to second time period, thus by manufacturer
Initial data press subregion distribution, solve the problems, such as cannot in real time by data loading, reach can quasi real time to data at
Reason, improves the technical effect of the timeliness of data processing.In addition, can be determined that the serious of data delay by sweep time window
Property, and achieved the effect that accurate acquisition data by adjusting sweep time window.The technical solution of the embodiment of the present invention is in property
Optimal compatibility is accomplished in energy and real-time, not only ensure that real-time, but also fortune can be implemented on lower performance configuration server
Row.
Embodiment three
Fig. 5 is the flow chart for the method that a kind of data source that the embodiment of the present invention three provides quasi real time is put in storage, the present embodiment
It advanced optimizes on the basis of the above embodiments, suitable for the scene for being put in storage manufacturer's initial data in real time.Such as Fig. 5 institute
To show, this method can be executed by server, it specifically includes:
S310, the first data to be put in storage are obtained based on period first time.
Wherein, period first time refers to that scan period, scan period refer at regular intervals to manufacturer's initial data
Carry out single pass.
S320, the document time for obtaining first data to be put in storage, the document time are distributed with second time period.
Wherein, document time refers to that manufacturer generates the time of data.Specifically, manufacturer can generate one when generating data
A filename includes the time for generating the data in filename.Second time period refers to for by the original number of manufacturer
According to the time cycle for carrying out subregion distribution according to a time interval.Second time period can according to need setting, herein not
It is limited.
S330, first data to be put in storage are stored according to the second time period.
Wherein, it since the first data to be put in storage have the document time for generating the data, and is set according to second time period
Multiple subregions are set.First data to be put in storage are stored according to second time period, can the initial data to manufacturer carry out by
Subregion distribution, so that the later period traces data.In this step, storage refers to that the first data to be put in storage that will have been obtained push away
It is sent under HDFS distributed file system specified directory.Likewise, this timing storage method can also be used when push.
S340, judgement have been put in storage whether data reach life cycle.
Wherein, life cycle refers to the time that data may exist, and obtains first wait be put in storage from based on period first time
Data start to be calculated entry time.Life cycle can be seven days, is also possible to one month, does not do specific limit herein
System.
If S350, the data that have been put in storage reach the life cycle, deletion is described to be put in storage data.
Wherein, when entry time reaches life cycle, the data that arrived life cycle are deleted.By to number
According to a life cycle, the data are deleted after reaching life cycle, release the memory space of memory.
It replaces that step S340 and S350 can be removed in embodiment one.
With reference to Fig. 6, optionally, step S330 may include:
S331, matching first data to be put in storage and described log has been put in storage it.
Wherein, it can be generated when data being stored and are put in storage every time and be put in storage log, to facilitate confirmation and retrospect to enter each time
The data in library.It has been put in storage all set for being put in storage data before log is this storage.Matching refers to comparison first wait be put in storage
Data and the data being put in storage in log, to avoid the first data to be put in storage, storage is being stored according to the second time period
Duplicate data.
S332, it data has been put in storage obtains the second data to be put in storage wait be put in storage in data to reject described first.
Wherein, first after being put in storage in data and rejecting and be put in storage data, remaining data are that this needs is put in storage
The filename of new data.Second data to be put in storage refer to the set of all new datas.
S333, second data to be put in storage are stored according to the second time period.
It in the present embodiment, can be by manufacturer by the way that the second data to be put in storage are pressed subregion distribution with second time period
Initial data according to generation time store, in order to the later period trace confirm.
The technical solution of the embodiment of the present invention can be in real time by obtaining the first data to be put in storage within period first time
Obtain the initial data of manufacturer.The document time of first data to be put in storage is stored according to second time period, thus by manufacturer
Initial data press subregion distribution, solve the problems, such as cannot in real time by data loading, reach can quasi real time to data at
Reason, improves the technical effect of the timeliness of data processing.In addition, being deleted by the data that will reach life cycle, releases and deposit
The memory space of reservoir.The technical solution of the embodiment of the present invention has accomplished optimal compatibility in performance and real-time, both ensure that
Real-time, and operation can be implemented on lower performance configuration server.
Example IV
Fig. 7 is the structural schematic diagram for the device that a kind of data source that the embodiment of the present invention four provides quasi real time is put in storage, this reality
It applies example and is applicable to the scene for being put in storage manufacturer's initial data in real time, which can use the side of software and/or hardware
Formula is realized, and can be integrated on the server.
As shown in fig. 7, quasi real time loading device may include obtaining module 410 and storage to data source provided in this embodiment
Module 420, in which:
Module 410 is obtained, for obtaining the first data to be put in storage based on period first time；Described first is obtained wait be put in storage
The document time of data, the document time are distributed with second time period；
Memory module 420, for storing first data to be put in storage according to the second time period.
Optionally, module is obtained to be specifically used for:
The first filename to be put in storage, the default scanning rule are obtained according to default scanning rule based on period first time
Including filename rule, scan path and sweep time window；
First data to be put in storage are obtained according to the described first filename to be put in storage.
Optionally, module is obtained to be also used to:
It is obtained based on period first time in the scan path and meets the filename rule in the sweep time window
The described first filename to be put in storage then.
Optionally, obtaining module includes:
Matching unit, for matching the described first filename to be put in storage and having pulled log；
Culling unit, for having pulled filename wait be put in storage in filename to reject described first and obtaining the second library text to be entered
Part name；
Acquiring unit, for obtaining first data to be put in storage according to the described second filename to be put in storage.
Optionally, the device further include:
Judgment module, for judging whether the described first filename quantity to be put in storage is less than preset quantity threshold value；
If the first filename quantity to be put in storage is less than the preset quantity threshold value, the sweep time window is adjusted
Mouthful.
Optionally, matching unit is also used to match the described first data to be put in storage and described has been put in storage log；
Culling unit is also used to be put in storage data wait be put in storage in data to reject described first and obtain the second data to be put in storage；
Storage unit is also used to store second data to be put in storage according to the second time period.
Optionally, judgment module is also used to judge to be put in storage whether data reach life cycle；
If the data that have been put in storage reach the life cycle, deletion is described to be put in storage data.
Quasi real time the executable any embodiment of that present invention of loading device is provided data source provided by the embodiment of the present invention
Data source quasi real time storage method, have the corresponding functional module of execution method and beneficial effect.
Embodiment five
Fig. 8 is a kind of structural schematic diagram for server that the embodiment of the present invention five provides.Fig. 8, which is shown, to be suitable for being used to realizing
The block diagram of the exemplary servers 612 of embodiment of the present invention.The server 612 that Fig. 8 is shown is only an example, should not be right
The function and use scope of the embodiment of the present invention bring any restrictions.
As shown in figure 8, server 612 is showed in the form of generic server.The component of server 612 may include but not
Be limited to: one or more processor 616, storage device 628 connect different system components (including storage device 628 and processing
Device 616) bus 618.
Bus 618 indicates one of a few class bus structures or a variety of, including storage device bus or storage device control
Device processed, peripheral bus, graphics acceleration port, processor or total using the local of any bus structures in a variety of bus structures
Line.For example, these architectures include but is not limited to industry standard architecture (Industry Subversive
Alliance, ISA) bus, microchannel architecture (Micro Channel Architecture, MAC) bus is enhanced
Isa bus, Video Electronics Standards Association (Video Electronics Standards Association, VESA) local are total
Line and peripheral component interconnection (Peripheral Component Interconnect, PCI) bus.
Server 612 typically comprises a variety of computer system readable media.These media can be it is any being capable of bedding and clothing
The usable medium that business device 612 accesses, including volatile and non-volatile media, moveable and immovable medium.
Storage device 628 may include the computer system readable media of form of volatile memory, such as arbitrary access
Memory (Random Access Memory, RAM) 630 and/or cache memory 632.Terminal 612 can be wrapped further
Include other removable/nonremovable, volatile/non-volatile computer system storage mediums.Only as an example, storage system
634 can be used for reading and writing immovable, non-volatile magnetic media (Fig. 8 do not show, commonly referred to as " hard disk drive ").Although
It is not shown in Fig. 8, the disc driver for reading and writing to removable non-volatile magnetic disk (such as " floppy disk ") can be provided, and
To removable anonvolatile optical disk, such as CD-ROM (Compact Disc Read-Only Memory, CD-ROM), number
Optic disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical mediums) read-write CD drive
Dynamic device.In these cases, each driver can be connected by one or more data media interfaces with bus 618.It deposits
Storage device 628 may include at least one program product, which has one group of (for example, at least one) program module, this
A little program modules are configured to perform the function of various embodiments of the present invention.
Program/utility 640 with one group of (at least one) program module 642 can store in such as storage dress
It sets in 628, such program module 642 includes but is not limited to operating system, one or more application program, other program moulds
It may include the realization of network environment in block and program data, each of these examples or certain combination.Program module
642 usually execute function and/or method in embodiment described in the invention.
Server 612 can also be with one or more external equipments 614 (such as keyboard, direction terminal, display 624 etc.)
Communication, can also be enabled a user to one or more terminal interact with the server 612 communicate, and/or with make the clothes
Any terminal (such as network interface card, modem etc.) that business device 612 can be communicated with one or more of the other computing terminal
Communication.This communication can be carried out by input/output (I/O) interface 622.Also, server 612 can also be suitable by network
Orchestration 620 and one or more network (such as local area network (Local Area Network, LAN), wide area network (Wide Area
Network, WAN) and/or public network, such as internet) communication.As shown in figure 8, network adapter 620 passes through bus 618
It is communicated with other modules of server 612.It should be understood that although not shown in the drawings, can be used in conjunction with server 612 other hard
Part and/or software module, including but not limited to: microcode, terminal driver, redundant processor, external disk drive array, magnetic
Disk array (Redundant Arrays of Independent Disks, RAID) system, tape drive and data backup
Storage system etc..
The program that processor 616 is stored in storage device 628 by operation, thereby executing various function application and number
According to processing, such as realize the method that a kind of data source provided by any embodiment of the invention is quasi real time put in storage, this method can be with
Include:
The first data to be put in storage are obtained based on period first time；
The document time of the described first data to be put in storage is obtained, the document time is distributed with second time period；
First data to be put in storage are stored according to the second time period.
The technical solution of the embodiment of the present invention can be in real time by obtaining the first data to be put in storage within period first time
Obtain the initial data of manufacturer.The document time of first data to be put in storage is stored according to second time period, thus by manufacturer
Initial data press subregion distribution, solve the problems, such as cannot in real time by data loading, reach can quasi real time to data at
Reason, improves the technical effect of the timeliness of data processing.The technical solution of the embodiment of the present invention is accomplished in performance and real-time
Optimal compatibility, not only ensure that real-time, but also can implement operation on lower performance configuration server.
Embodiment six
The embodiment of the present invention six additionally provides a kind of computer readable storage medium, is stored thereon with computer program, should
A kind of data source such as provided by any embodiment of the invention quasi real time storage method, the party are realized when program is executed by processor
Method may include:
The first data to be put in storage are obtained based on period first time；
The document time of the described first data to be put in storage is obtained, the document time is distributed with second time period；
First data to be put in storage are stored according to the second time period.
The computer readable storage medium of the embodiment of the present invention, can be using one or more computer-readable media
Any combination.Computer-readable medium can be computer-readable signal media or computer readable storage medium.Computer
Readable storage medium storing program for executing for example may be-but not limited to-the system of electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, dress
It sets or device, or any above combination.The more specific example (non exhaustive list) of computer readable storage medium wraps
It includes: there is the electrical connection of one or more conducting wires, portable computer diskette, hard disk, random access memory (RAM), read-only
Memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory
(CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.In this document, computer-readable
Storage medium can be it is any include or storage program tangible medium, the program can be commanded execution system, device or
Device use or in connection.
Computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.
The program code for including on storage medium can transmit with any suitable medium, including --- but it is not limited to nothing
Line, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
The computer for executing operation of the present invention can be write with one or more programming languages or combinations thereof
Program code, described program design language include object oriented program language-such as Java, Smalltalk, C++,
It further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
It fully executes, partly execute on the user computer on the user computer, being executed as an independent software package, portion
Divide and partially executes or executed on remote computer or terminal completely on the remote computer on the user computer.It is relating to
And in the situation of remote computer, remote computer can pass through the network of any kind --- including local area network (LAN) or extensively
Domain net (WAN)-be connected to subscriber computer, or, it may be connected to outer computer (such as provided using Internet service
Quotient is connected by internet).
The technical solution of the embodiment of the present invention can be in real time by obtaining the first data to be put in storage within period first time
Obtain the initial data of manufacturer.The document time of first data to be put in storage is stored according to second time period, thus by manufacturer
Initial data press subregion distribution, solve the problems, such as cannot in real time by data loading, reach can quasi real time to data at
Reason, improves the technical effect of the timeliness of data processing.The technical solution of the embodiment of the present invention is accomplished in performance and real-time
Optimal compatibility, not only ensure that real-time, but also can implement operation on lower performance configuration server.
Note that the above is only a better embodiment of the present invention and the applied technical principle.It will be appreciated by those skilled in the art that
The invention is not limited to the specific embodiments described herein, be able to carry out for a person skilled in the art it is various it is apparent variation,
It readjusts and substitutes without departing from protection scope of the present invention.Therefore, although being carried out by above embodiments to the present invention
It is described in further detail, but the present invention is not limited to the above embodiments only, without departing from the inventive concept, also
It may include more other equivalent embodiments, and the scope of the invention is determined by the scope of the appended claims.
Claims (10)
1. a kind of method that data source is quasi real time put in storage characterized by comprising
The first data to be put in storage are obtained based on period first time；
The document time of the described first data to be put in storage is obtained, the document time is distributed with second time period；
First data to be put in storage are stored according to the second time period.
2. a kind of method that data source is quasi real time put in storage as described in claim 1, which is characterized in that described based at the first time
Period obtains the first data to be put in storage
The first filename to be put in storage is obtained according to default scanning rule based on period first time, the default scanning rule includes
Filename rule, scan path and sweep time window；
First data to be put in storage are obtained according to the described first filename to be put in storage.
3. the method that data source as claimed in claim 2 is quasi real time put in storage, which is characterized in that described to be based on period first time
The first filename to be put in storage is obtained according to default scanning rule, comprising:
It is obtained based on period first time in the scan path and meets the filename rule in the sweep time window
First filename to be put in storage.
4. the method that data source as claimed in claim 2 is quasi real time put in storage, which is characterized in that it is described according to described first wait enter
Library file name obtains first data to be put in storage, comprising:
It matches the described first filename to be put in storage and has pulled log；
It has pulled filename wait be put in storage in filename to reject described first and obtains the second filename to be put in storage；
First data to be put in storage are obtained according to the described second filename to be put in storage.
5. the method that data source as claimed in claim 2 is quasi real time put in storage, which is characterized in that described based on week at the first time
Phase obtains first wait be put in storage after filename according to default scanning rule, comprising:
Judge whether the described first filename quantity to be put in storage is less than preset quantity threshold value；
If the first filename quantity to be put in storage is less than the preset quantity threshold value, the sweep time window is adjusted.
6. the method that data source as described in claim 1 is quasi real time put in storage, which is characterized in that described according to second time
Period stores first data to be put in storage
It matches the described first data to be put in storage and described has been put in storage log；
It has been put in storage data wait be put in storage in data to reject described first and obtains the second data to be put in storage；
Second data to be put in storage are stored according to the second time period.
7. the method that data source as described in claim 1 is quasi real time put in storage, which is characterized in that it is described according to described second when
Between period storage described first wait be put in storage after data, comprising:
Judgement has been put in storage whether data reach life cycle；
If the data that have been put in storage reach the life cycle, deletion is described to be put in storage data.
8. a kind of device that data source is quasi real time put in storage characterized by comprising
Module is obtained, for obtaining the first data to be put in storage based on period first time；Obtain the described first data to be put in storage
Document time, the document time are distributed with second time period；
Memory module, for storing first data to be put in storage according to the second time period.
9. a kind of server characterized by comprising
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
The now method that such as data source of any of claims 1-7 is quasi real time put in storage.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The method being quasi real time put in storage such as data source of any of claims 1-7 is realized when execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910602946.8A CN110347661A (en) | 2019-07-05 | 2019-07-05 | Method, apparatus, server and the storage medium that data source is quasi real time put in storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910602946.8A CN110347661A (en) | 2019-07-05 | 2019-07-05 | Method, apparatus, server and the storage medium that data source is quasi real time put in storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110347661A true CN110347661A (en) | 2019-10-18 |
Family
ID=68178347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910602946.8A Pending CN110347661A (en) | 2019-07-05 | 2019-07-05 | Method, apparatus, server and the storage medium that data source is quasi real time put in storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110347661A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081947A (en) * | 2010-08-05 | 2011-06-01 | 浙江工业大学 | Management control method of video and audio file on embedded multimedia system |
| CN102111453A (en) * | 2011-03-04 | 2011-06-29 | 创博亚太科技（山东）有限公司 | Method and system for extracting Internet user network behaviors |
| CN103092840A (en) * | 2011-10-28 | 2013-05-08 | 上海邮电设计咨询研究院有限公司 | Method for acquiring self-increment mass data files from multiple sources |
| CN104133822A (en) * | 2013-07-15 | 2014-11-05 | 腾讯科技（深圳）有限公司 | Method and device for scanning files in memory |
| CN104166650A (en) * | 2013-05-16 | 2014-11-26 | 方正宽带网络服务股份有限公司 | Data storage device and method |
| CN104484782A (en) * | 2014-11-25 | 2015-04-01 | 苏州迪云信息科技有限公司 | Processing method and device for scanned data |
| CN105162622A (en) * | 2015-08-03 | 2015-12-16 | 瑞斯康达科技发展股份有限公司 | Storage method and system |
| US9384227B1 (en) * | 2013-06-04 | 2016-07-05 | Amazon Technologies, Inc. | Database system providing skew metrics across a key space |
| CN106302751A (en) * | 2016-08-17 | 2017-01-04 | 中国联合网络通信集团有限公司 | The archiving method of electronic record, interface server and filing system |
| CN107861989A (en) * | 2017-10-17 | 2018-03-30 | 平安科技（深圳）有限公司 | Partitioned storage method, apparatus, computer equipment and the storage medium of data |
-
2019
- 2019-07-05 CN CN201910602946.8A patent/CN110347661A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081947A (en) * | 2010-08-05 | 2011-06-01 | 浙江工业大学 | Management control method of video and audio file on embedded multimedia system |
| CN102111453A (en) * | 2011-03-04 | 2011-06-29 | 创博亚太科技（山东）有限公司 | Method and system for extracting Internet user network behaviors |
| CN103092840A (en) * | 2011-10-28 | 2013-05-08 | 上海邮电设计咨询研究院有限公司 | Method for acquiring self-increment mass data files from multiple sources |
| CN104166650A (en) * | 2013-05-16 | 2014-11-26 | 方正宽带网络服务股份有限公司 | Data storage device and method |
| US9384227B1 (en) * | 2013-06-04 | 2016-07-05 | Amazon Technologies, Inc. | Database system providing skew metrics across a key space |
| CN104133822A (en) * | 2013-07-15 | 2014-11-05 | 腾讯科技（深圳）有限公司 | Method and device for scanning files in memory |
| CN104484782A (en) * | 2014-11-25 | 2015-04-01 | 苏州迪云信息科技有限公司 | Processing method and device for scanned data |
| CN105162622A (en) * | 2015-08-03 | 2015-12-16 | 瑞斯康达科技发展股份有限公司 | Storage method and system |
| CN106302751A (en) * | 2016-08-17 | 2017-01-04 | 中国联合网络通信集团有限公司 | The archiving method of electronic record, interface server and filing system |
| CN107861989A (en) * | 2017-10-17 | 2018-03-30 | 平安科技（深圳）有限公司 | Partitioned storage method, apparatus, computer equipment and the storage medium of data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108897691A (en) | Data processing method, device, server and medium based on interface analog service | |
| CN102508840B (en) | Concurrent editing of online drawings | |
| CN104462534B (en) | network information sharing method and device | |
| CN110417879A (en) | A kind of message treatment method, device, equipment and storage medium | |
| CN109471851A (en) | Data processing method, device, server and storage medium | |
| CN110457038B (en) | Card display method, device and equipment of auxiliary application terminal and storage medium | |
| CN110377429A (en) | A kind of control method, device, server and storage medium that real-time task calculates | |
| CN109618225A (en) | Video takes out frame method, device, equipment and medium | |
| CN110472178A (en) | A kind of probe method for implanting, device, computer equipment and storage medium | |
| CN111695876A (en) | Approval process processing method, device, equipment and storage medium | |
| CN108337301A (en) | Network request processing method, device, server and the storage medium of application program | |
| CN109448823A (en) | Data processing method, device, server and storage medium | |
| CN111385294B (en) | Data processing method, system, computer device and storage medium | |
| CN110443690A (en) | A kind of method, apparatus, server and the storage medium of variance data reconciliation | |
| CN109284108A (en) | Date storage method, device, electronic equipment and storage medium | |
| CN110413971A (en) | Information processing method, device, terminal and the server of online document | |
| CN109558331A (en) | Defect inspection method, device, equipment and the storage medium of application program | |
| CN116738901A (en) | Chip verification method and device | |
| CN110413675A (en) | A kind of control method, device, server and storage medium that real-time task calculates | |
| CN110162476A (en) | Test method and device | |
| CN112749072B (en) | Testing method and device for cloud storage system for storing training data | |
| CN109561212A (en) | A kind of merging method of release information, device, equipment and storage medium | |
| CN110347661A (en) | Method, apparatus, server and the storage medium that data source is quasi real time put in storage | |
| CN109408539A (en) | Data manipulation method, device, server and storage medium | |
| CN110515749A (en) | Method, apparatus, server and the storage medium for the queue scheduling that information is sent |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |