CN106844411A - A kind of big data random access system and method based on reducing subspaces - Google Patents
A kind of big data random access system and method based on reducing subspaces Download PDFInfo
- Publication number
- CN106844411A CN106844411A CN201610907792.XA CN201610907792A CN106844411A CN 106844411 A CN106844411 A CN 106844411A CN 201610907792 A CN201610907792 A CN 201610907792A CN 106844411 A CN106844411 A CN 106844411A
- Authority
- CN
- China
- Prior art keywords
- data
- random
- queue
- parameter
- usbkey
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
- G06F21/34—User authentication involving the use of external additional devices, e.g. dongles or smart cards
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2107—File encryption
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Storage Device Security (AREA)
Abstract
The invention discloses a kind of big data random access system based on reducing subspaces and method, the described method comprises the following steps：Under the support of UsbKey keys, generate random number and data slicer using MT algorithms and encrypt storage to database；Parameter or acquiescence start-up parameter were closed according to last time, the random number and data slicer stored in loading of databases are with combination producing random sequence and are distributed to n random queue, and the random sequence for needing to load is decrypted and is loaded into random queue using reducing subspaces algorithm from data slicer；Whether the data slicer information monitoring according to loading needs the new data slicer information of loading；According to request of data is obtained, corresponding Joseph's parameter is obtained in Usbkey, data are obtained from random queue.The method of the present invention has taken into full account the safety and randomness of random data, is realized using computer technology and obtains random data at high speed, meets the data skew rate problem in big data access.
Description
Technical field
The present invention relates to a kind of big data random access system and method, more particularly to a kind of big number based on reducing subspaces
According to random access system and method.
Background technology
Random number algorithm has a very wide range of applications in the world today, for example finance, machine-building, IT networks etc..
This also promotes the people to carry out more in-depth study to random number algorithm, and the true random number of random number service is even provided with present
Website, random number is produced using atmospheric noise or certain uncertain large-scale stochastic source.
In actual applications, it should need to require to be selected according to the different of system using which type of random number algorithm
Select.For fairly simple demand, such as the random verification code of website, the pseudo-random algorithm carried using operating system just can be with
Meet and require.For bank cipher or data encryption, the requirement of random number is very strict even very harsh, once ging wrong, have can
Very big loss can be caused.
Meanwhile, requirement of the big data subregion to random number algorithm is very high, and it directly affects the dispersion of data storage
It is whether consistent.One good random algorithm can reduce the deviation of data, so as to improve inquiry velocity.
With advances in technology, using randomness pseudo-random algorithm very high and the internal logic mechanism of a series of complex
The method being combined produces random number, can meet requirement.So, although increased the complexity of program, but need not increase
Plus extra hardware supported, and can ensure the not repeated of random number within years thousands of or even up to ten thousand.
Therefore, this area needs to develop system and the side that a kind of use pseudo-random algorithm is combined with internal logic mechanism
Method, to produce data to deviate less and safe and secret degree random number high.
The content of the invention
To solve above-mentioned the problems of the prior art, deposited at random the invention provides a kind of big data based on reducing subspaces
Take system and method.
To achieve the above object, the particular technique of a kind of big data random access regime based on reducing subspaces of the invention
Scheme is as follows：
A kind of big data random access regime based on reducing subspaces, comprises the following steps：Step one, in UsbKey keys
Support under, using Mersenne Prime rotation random algorithm generation random number and data slicer, and random number and data slicer are added
Database is arrived in close storage；Step 2, under the support of UsbKey keys, parameter or acquiescence start-up parameter was closed according to last time,
The random sequence of generation is distributed to n by the random number and data slicer stored in loading of databases with combination producing random sequence
Individual random queue, decrypts the random sequence for needing to load and be loaded into fleet from data slicer using reducing subspaces algorithm
In row, to increase loading degree of randomness；Step 3, the data whether the data slicer information monitoring according to loading needs loading new are cut
Piece information, when the data deficiencies in random queue, triggering continues to load data and be distributed to random queue；Step 4, according to
The reducing subspaces parameter reference in request of data is obtained, corresponding Joseph's parameter is obtained in Usbkey, obtained from random queue
Access evidence.
According to another aspect of the present invention, a kind of big data random access system based on reducing subspaces, institute are additionally provided
Stating system includes following functions module：Random number generation unit, can be rotated under the support of UsbKey keys using Mersenne Prime
Random algorithm generates random number and data slicer, and random number and data slicer encryption are stored in database；Random sequence
Dispatching Unit is loaded, parameter or acquiescence start-up parameter can be closed according to last time under the support of UsbKey keys, load data
The random number and data slicer stored in storehouse with combination producing random sequence, then by the random sequence of generation be distributed to n with
Fleet is arranged, and the random sequence for needing to load is decrypted and is loaded into random queue using reducing subspaces algorithm from data slicer
In, to increase loading degree of randomness；Monitoring unit, the data slicer information according to loading decides whether the new data slicer letter of loading
Breath, when the data deficiencies in random queue, triggering continues to load data and is simultaneously distributed to random queue, with monitoring random queue
Data；Data capture unit, according to the reducing subspaces parameter reference obtained in request of data, obtains corresponding in Usbkey
Joseph's parameter, data are obtained from random queue.
In big data random access system and method based on reducing subspaces of the invention, rotated by by Mersenne Prime
Random algorithm (MT pseudo-random algorithms) is combined with the internal logic of reducing subspaces, random number needed for producing, and is ensureing random number
While randomness, the safe and secret of data is met, even if system is produced during system designer cannot also obtain practical application
Random number information.
Brief description of the drawings
Fig. 1 is the structural representation of the big data random access system based on reducing subspaces of the invention；
Fig. 2 is the workflow schematic diagram of the big data random access system based on reducing subspaces of the invention；
Fig. 3 shows the random sequence generation stream in the big data random access regime based on reducing subspaces of the invention
Journey；
Fig. 4 shows that the random sequence in the big data random access regime based on reducing subspaces of the invention loads stream
Journey；
Fig. 5 shows the monitoring flow in the big data random access regime based on reducing subspaces of the invention；
Fig. 6 shows the acquisition data flow in the big data random access regime based on reducing subspaces of the invention.
Specific embodiment
In order to be best understood from the purpose of the present invention, structure and function, below in conjunction with the accompanying drawings, to of the invention based on about plucked instrument
The big data random access system and method for husband's ring are described in further detail.
As shown in Figure 1-2, the big data random access system based on reducing subspaces of the invention includes following functions module：
Random number generation unit, can be random using Mersenne Prime rotation under the support of UsbKey (authentication) key
Algorithm (MT algorithms) generates random number and data slicer, and random number and data slicer encryption storage are arrived file system (in full
According to storehouse) in.
Random sequence loads Dispatching Unit, can close parameter or acquiescence according to last time under the support of UsbKey keys
Start-up parameter, the random number stored in loading of databases and data slicer with combination producing random sequence, then will generate with
Machine sequence is distributed to n random queue, using reducing subspaces algorithm the random sequence decryption for needing to load from data slicer
And be loaded into random queue, to increase loading degree of randomness.
Monitoring unit, the data slicer information according to loading decides whether the new data slicer information of loading, when with fleet
During data deficiencies in row, triggering continues to load data and is simultaneously distributed to random queue, to monitor the data in random queue.
Data capture unit, according to the reducing subspaces parameter reference obtained in request of data, obtains corresponding in Usbkey
Joseph's parameter, from random queue obtain data.
Random sequence loads Dispatching Unit includes random sequence loader, queue distributor and random queue.Wherein, with
Machine sequence loader can be in the case where the key of Usbkey be supported, comprehensive random number, data slicer and system last time close parameter, from
The random sequence for needing to load is decrypted and is loaded into database in data slicer.Queue distributor is responsible for stochastic ordering to equip
The random sequence for carrying device loading is distributed in corresponding n random queue, and largest random is reached as far as possible.Random queue is deposited
The data of random sequence are stored up, this can improve and take data rate equivalent to the effect of caching.Once in random queue data compared with
It is few, can be initiated to load the operation of random queue to random sequence loader from monitoring unit or data capture unit.
Describe each functional unit of composition described above system in detail separately below.
Further, Mersenne Twister random number Algorithm are used in random number generation unit
(Mersenne Prime rotates random algorithm, MT19937).MT algorithms are that a pseudo random number occurs algorithm, based on limited binary word
Matrix linear recurrence in section, can quickly produce high-quality pseudo random number, have modified classic random number generation algorithm very
Many defects.Usually using two close variants, variant difference is to have used different Mersenne Primes, one to this algorithm
Individual renewal and more usually MT19937,32 word lengths, also one variant is 64 MT19937-64 of version.For one
Individual k length, Mason's Rotation Algorithm can generate the random number of highway network design between the interval of [0,2k-1].
A general variant MT19937 using Mason's Rotation Algorithm is calculated at random, can produce 32 integer sequences, is had
Have the advantages that following：
1) there is the cycle very long of 219937-1, our 500,000,000,000 unduplicated requirements are ensured enough；
2) can equal distribution between the dimension of 1≤k≤623；
3) in addition to the incorrect random number generator on statistical significance, in all PRNG methods
In be most fast.
Random number can record the quantity of the random number generated using the algorithm, generation next time random data while generation
When, the data volume for having generated is skipped first, the not repeated of pseudo random number is so ensured that, and random seed is constant
's.
In order to ensure the safety of data, the risk in data generation, storing process is reduced, we use data slicer skill
Art stored record information, MT algorithms produce partial data storage in internal memory, and anyone cannot know specific content, but one
After denier storage, just there is the risk of leakage, so we are in addition to data break up section, also the file consolidation cut into slices is added
It is close.
Wherein, the generation of record data, it with a data is one group to be, that is, data slicer file is also with a as unit,
Every data is cut into n parts at random first, n file is constituted, then this n file is encrypted, then this n file
Be respectively written into file system, write-in file system before the step of be all to be carried out in internal memory, it is ensured that data genaration person
Have no chance to view file content.While a record is broken up as n file, corresponding slice information can be generated,
After write-in file system, these information are saved in database.To ensure that usbkey effectively, can normally get during operation
Encryption key.Period, there is any problem in any link, this recorded at random failed regeneration, and meeting rollback is operated after failure,
Ensure the integrality of data.
Thus, random number generation unit is responsible for using MT algorithms to generate random number, and data slicer storage to file system
System, related information and slice information are preserved as in database.It by manually performing not is automatic that the operation for generating random number is
, and need the key of UsbKey to support.
Further, run when random sequence loader starts in every subsystem, random queuing data is not enough.Start
When can load last time and close the record information (data deficiencies without loading, because loading) for preserving, according to these
Information judges need which batch data section file loaded, and judges which this batch data section file has been sold to
, then from the beginning is originally opened, random sequence is loaded into random queue at random using queue distributor, until random queue dress
Untill being loaded with.
The data slicer file composition random sequence of random sequence loader loading is memory-resident, unless this sequence
Acquisition is finished.Once monitoring unit finds that random queuing data is not enough, random sequence loader will be triggered and perform loading operation.
Random sequence loader can check whether the random sequence of current maintenance can meet the data demand of random queue first, if not
Can meet will load follow-up data slicer file supplement random sequence, and inspection is finished, and calls queue distributor with fleet
Row are full of.
Further, queue distributor is used to supplement random queuing data, starts in system or monitoring unit finds at random
Can be performed in the case of queue deficiency.Random queue is just generated when system starts, also without data, so queue distributor handle
The data got from random sequence loader are put into some random queue at random, and random algorithm herein can be using being
The random function that system is carried can also use MT algorithms, untill all of n random queue is full of.
Further, random queue is created to ensure to take the speed of record information, equivalent to a fast cache,
Another purpose is that, in order to increase the randomness of access evidence, random queue has n, every time access according to when, according to about plucked instrument
Husband's ring algorithm skips k evidence to obtain the record of request from some random queue, increases the degree of randomness of access evidence.At random
The length of queue is needed to be adjusted according to effective sale situation, and sales volume maximum queue length is accordingly tuned up, and sales volume is small can be appropriate
Reduce.
Further, when monitoring unit finds certain random queuing data less than b% (10%-20%), will touch
Hair random sequence loader, random sequence loader is checked out after its data, and queue distributor will be called to distribute data
To random queue, untill random queue is full of.Monitoring unit will ceaselessly circulate execution after system start-up, be one
Timed task.It can monitor the data volume in random queue, once less than the value of regulation, random sequence loader will be called to fill
Carry data.
Monitoring unit monitors the data in random queue, once the data deficiencies in random queue, can request data loading
Device loads data, and so as to be distributed to random queue, data loader decides whether to add according to the data slicer information of current loading
Carry new data slicer information.
In addition, monitoring unit can also check the quantity of the sale daily record of data capture unit, once data volume reaches regulation
Log recording will be put in storage preservation by value more than the time limit most long of regulation, it is to avoid the hydraulic performance decline caused by frequent updating.
Monitoring unit ensure that fail caused by data because of that will not be lacked, while also because being improved without real-time update database
The speed of access evidence.
Further, data capture unit obtains data using reducing subspaces, and reducing subspaces are an application problems for mathematics,
Its mathematical derivation formula is：F [1]=1, f [n]=(f [n-1]+k) %n (n>1).
Data capture unit can use reducing subspaces when data are obtained, and we specify n, k, m value of series at random
Be stored in usbkey, obtain data request with using which n, k, m value index, this index value be it is random,
According to this group of n, k, m value that this index goes from usbkey the insides, so as to obtain the record number to be obtained in random queue.About
Plucked instrument husband ring can make the data randomness of acquirement more preferably, due to there is usbkey encrypting storing parameters, it is ensured that even if being aware of
Data in random queue, can not calculate which record this request can return on earth.
Here acquisition data, refer to just to obtain a random sequence number, obtain data and are performed by data capture unit, only
Receiving request can just obtain data.Obtaining data needs the index with reducing subspaces parameter, this index be request again with
Machine produce a positive integer, from usbkey obtain reducing subspaces parameter when, according to definition parameter group number
Remainder, then obtains reducing subspaces parameter, and calculated according to formula should be random from that for data capture unit after obtaining parameter
Which data queue obtains the data for returning.After obtaining data, data capture unit record obtains record, and then data are returned
Back to request.
Data capture unit is after respective request is returned data to, it should check the data feelings of each random queue
Condition, once finding to have the random queuing data not enough (less than the 10%-20% of queue length) should notify monitoring unit, monitoring is single
Unit can call random sequence loader loading data, will not cause to obtain the problem for failing due to data deficiencies.
Further, in systems, Usbkey is safety-critical, for storing the key and data capture unit of slice of data
The reducing subspaces parameter for using, has serious problem once occurring leaking.System uses usbkey more advanced at present
Technology carrys out encrypting storing our data, and strictly controls the access right of usbkey, to ensure the peace of data to greatest extent
Entirely.
Thus, the system of the random acquisition data based on reducing subspaces of the invention, is not merely to produce random sequence
Process, it increased the randomness and security for obtaining data in the flow for obtaining random sequence.Use usbkey encryption sides
Method, it is ensured that the acquirement of the uncontrollable random number of operator, even if usbkey have leaked, because each ID for obtaining also has phase
When randomness, also can guarantee that cannot arbitrarily obtain desired data.
In addition, with database be combined for data slicer technology, encryption technology by the present invention, it is ensured that the safety of data, its
Generation process is related to personnel seldom, and security performance is very good, and whole process producers cannot obtain any record
Information.
Although it is very thoughtful that system safety of the invention and random performance consider, the complexity of system, gesture are the increase in
The speed of access evidence must be influenceed, and due to bulk data terminate-and-stay-resident, the also situation of redundancy, so to the expense of system
Also it is larger.However, because current server performance has been greatly improved, expense aspect is without there is excessive consideration.As for
The speed of evidence of fetching, can improve written in code quality to control by adjusting the parameter of each several part.
On the other hand, the invention also discloses using the above-mentioned big data random access system based on reducing subspaces carry out with
The method of machine access, it is specific as follows.
Big data random access regime based on reducing subspaces of the invention is comprised the following steps：
Step one, under the support of UsbKey (authentication) key, using Mersenne Prime rotation random algorithm, (MT is calculated
Method) generation random number and data slicer, and random number and data slicer encryption storage are arrived in file system (such as database).
Step 2, under the support of UsbKey keys, parameter or acquiescence start-up parameter was closed according to last time, loaded data
The random number and data slicer stored in storehouse with combination producing random sequence, then by the random sequence of generation be distributed to n with
Fleet is arranged, and the random sequence for needing to load is decrypted and is loaded into random queue using reducing subspaces algorithm from data slicer
In, to increase loading degree of randomness.
Step 3, the data slicer information according to loading decides whether the new data slicer information of loading, when random queue
In data deficiencies when, triggering continue load data simultaneously be distributed to random queue, to monitor the data in random queue.
Step 4, according to the reducing subspaces parameter reference obtained in request of data, obtains corresponding about plucked instrument in Usbkey
Husband's parameter, data are obtained from random queue.
Further, as shown in figure 3, in step 2, the product process of random sequence is as follows：
1) log-on data generation client, checks whether usbkey is normal, the ginseng such as batch, quantity of input generation data
Number；
2) generation record random sequence；
3) the records series section of generation, it is divided into n parts, is put into n file, and use the key handle obtained from usbkey
This n file encryption is completed；
4) data slicer is stored in specific file, and the information related to data slicer is stored in database.
Random data product process is the flow for generating record data ID, and data genaration flow is born by random number generator
Duty, it is except being responsible for the generation of random number, while also being responsible for the section encryption storage of data, the in-stockroom operation of data message.
These operations need the support (whether key is normal) of Usbkey, and key cannot be obtained if system is not inserted into Usbkey.
Further, as shown in figure 4, in step 2, the flow that random sequence is loaded is as follows：
1) check whether necessary usbkey is normal, is directly unsuccessfully exited Ru abnormal；
2) parameter when upper subsystem is exited is obtained, if starting for the first time, the configuration parameter of acquiescence is used；
3) the secret key decryption section file obtained in usbkey is used；
4) information in section file and database generates ID sequences；
5) data are sequentially obtained from ID sequences, any one random queue is distributed at random.
Random sequence loads flow and can be performed when restarting every time, and main function is the parameter according to last time
Loading data slicer file, is combined into random sequence, is then distributed to random queue to be obtained.The module being related to has stochastic ordering
Row loader, queue distributor, random queue, usbkey.
If starting for the first time, then last time start-up parameter should be the first time start-up parameter of system default.In addition
When decryption section file, may scanning file whether there is, whether database information is complete etc..
Further, as shown in figure 5, in step 3, the monitoring flow of random queue is as follows：
1) judge in the random queue of M with the presence or absence of data deficiencies, maximum data 20% queue；
2) if there is the 20% of random queuing data deficiency maximum data, then the number of slices loaded in random sequence is judged
According to whether the 20% of not enough maximum load slice of data；
If 20% of data deficiencies maximum load slice of data 3) in random sequence, would remove one group of slice of data text
Part, takes decruption key to be combined into new random sequence, and continued access is in current random sequence；
4) from random sequence order extracted data radom insertion in the random queue of data deficiencies, until all random
Untill queue is full of；
5) judge whether the sale daily record for obtaining data exceedes maximum, if it exceeds maximum, sale log recording
Storage；
6) judge acquisition data does not write whether logging time exceedes setting, if it exceeds setting is log recording
Storage；
7) judge to complete, dormancy for a period of time, continues return to step 1) cycle criterion.
Monitoring process is a cyclic process, can all be performed one time at regular intervals since system starts, until being
System is closed.
Further, as shown in fig. 6, in step 4, the flow for obtaining data is as follows：
1) whether checking request parameter and usbkey are normal, if abnormal, directly return to error；
2) the random number parameter according to required parameter, gets the reducing subspaces parameter for using from usbkey；
3) according to reducing subspaces parameter, it is determined that from which the queue access evidence in m random queue；
4) according to reducing subspaces parameter, the information to be asked is obtained in the random queue of access evidence；
5) record information daily record；
6) judge whether random queuing data is sufficient, if sufficient, directly return to request data, if random number of queues
According to inadequate, call load queue sub-process load data (load queue sub-process is asynchronous procedure, calls rear function to return,
It is then back to request data).
Data flow design data acquiring unit, random queue, random sequence loader, queue distributor are obtained, wherein
Data capture unit and random queue are bound to perform, and random sequence loader and queue distributor are only obtaining number
Can just be performed when data deficiencies according in rear judgement queue.
Obtaining data needs request to carry random reducing subspaces index data, is found in usbkey according to this index data
Corresponding reducing subspaces parameter, should be from that random queue access evidence so as to be calculated according to these parameters, it should which takes
Individual data, increase obtains the randomness of data.
The method of the present invention has taken into full account the safety and randomness of random data, using modern computer technology
The acquisition random data of high speed, meets the data skew rate problem in big data access.
Embodiment one：
Using MT random number algorithms, rule produces random sequence to system of the invention according to the rules, using microtomy every
Bar random number Record ID is cut into n (5) piece, is put at random in the individual random section files of n (5), and the order and rule of storage are by data
Library storage, section file is recorded as one group with m (10,000,000), and each section file encrypts storage, encryption using AES
Key storage is in usbkey softdogs.
System starts decrypted random section file combination database stores information composition complete documentation sequence first every time,
Then parameter is preserved according to last time shutdown, generates k (20) individual random queue, each random queue is full of the individual record numbers of a (50000)
According to it is client return ID data to be ready to.
During client request data, extracted from 20 random queues according to reducing subspaces algorithm is random according to required parameter
One random ID is returned.When 20% (10000) of the data deficiencies full queue in queue, system is according to rule from random
Extracted data is full of queue in sequence.
In encryption usbkey in addition to storing key, the parameter array that reducing subspaces need, client request are also stored
During random sequence, the random parameter of random number (or MT algorithms) is carried using system, system is according to this parameter from usbkey
Middle acquirement reducing subspaces parameter, and then obtain data in random queue.One group of slice of data surplus is less than 20% (200w
) when, system loads another set slice of data, in order to avoid data deficiencies influence sale.
Therefore, system of the invention has reached following effect：Random sequence ensures 500,000,000,000 (5,000,000,000/year x1000)
Within random data do not repeat；Support that at least 2000/second take data rate；Guarantee data security, be not stolen, distorting can
Energy；The section file loading records series time is no more than 1 minute.
It is appreciated that the present invention is described by some embodiments, and what those skilled in the art knew, do not taking off
In the case of the spirit and scope of the present invention, various changes or equivalence replacement can be carried out to these features and embodiment.Separately
Outward, under the teachings of the present invention, these features and embodiment can be modified with adapt to particular situation and material without
The spirit and scope of the present invention can be departed from.Therefore, the present invention is not limited to the particular embodiment disclosed, and is fallen with
Embodiment in the range of claims hereof is belonged in the range of of the invention protection.
Claims (7)
1. a kind of big data random access regime based on reducing subspaces, it is characterised in that comprise the following steps：
Step one, under the support of UsbKey keys, using Mersenne Prime rotation random algorithm generation random number and data slicer,
And random number and data slicer encryption storage are arrived database；
Step 2, under the support of UsbKey keys, closed parameter or acquiescence start-up parameter, in loading of databases according to last time
The random sequence of generation is distributed to n random queue by the random number and data slicer of storage with combination producing random sequence, from
The random sequence for needing to load is decrypted and is loaded into random queue using reducing subspaces algorithm in data slicer, to increase dress
Carry degree of randomness；
Whether step 3, the data slicer information monitoring according to loading needs the new data slicer information of loading, when random queue
In data deficiencies when, triggering continue load data simultaneously be distributed to random queue；
Step 4, according to the reducing subspaces parameter reference obtained in request of data, obtains corresponding Joseph's ginseng in Usbkey
Number, data are obtained from random queue.
2. the big data random access regime based on reducing subspaces according to claim 1, it is characterised in that in step 2
In, the product process of random sequence is comprised the following steps：Log-on data generates client, checks whether usbkey is normal, input
Generate the parameters such as batch, the quantity of data；Generation record random sequence；The records series section of generation, it is divided into n parts, is put into n
Individual file, and this n file encryption is completed using the key obtained from usbkey；Data slicer is stored in specific file, with number
It is stored in database according to the related information of section.
3. the big data random access regime based on reducing subspaces according to claim 2, it is characterised in that in step 2
In, the flow that random sequence is loaded is comprised the following steps：Check whether necessary usbkey is normal, is directly unsuccessfully moved back Ru abnormal
Go out；Parameter when subsystem is exited in acquisition, if starting for the first time, uses the configuration parameter of acquiescence；Using in usbkey
The secret key decryption section file of acquisition；Information in section file and database generates ID sequences；The order from ID sequences
Data are obtained, any one random queue is distributed at random.
4. the big data random access regime based on reducing subspaces according to claim 1, it is characterised in that in step 3
In, the monitoring flow of random queue comprises the following steps：Judge to whether there is data deficiencies maximum data in M random queue
20% queue；If there is the 20% of random queuing data deficiency maximum data, then the section loaded in random sequence is judged
Whether data are not enough the 20% of maximum load slice of data；If data deficiencies maximum load slice of data in random sequence
20%, one group of slice of data file is removed, decruption key is taken to be combined into new random sequence, and continued access is to current stochastic ordering
In row；Order extracted data radom insertion is in the random queue of data deficiencies from random sequence, until all random queues
Untill being full of；Judge whether the sale daily record for obtaining data exceedes maximum, if it exceeds maximum, enters sale log recording
Storehouse；Judge acquisition data does not write whether logging time exceedes setting, if it exceeds setting is put in storage log recording；Sentence
Disconnected to complete, dormancy for a period of time, continues to come back for cycle criterion.
5. the big data random access regime based on reducing subspaces according to claim 1, it is characterised in that in step 4
In, the flow for obtaining data comprises the following steps：Whether checking request parameter and usbkey are normal, if abnormal, directly return
Return back out mistake；Random number parameter according to required parameter, gets the reducing subspaces parameter for using from usbkey；According to Joseph
Ring parameter, it is determined that from which the queue access evidence in m random queue；According to reducing subspaces parameter, access evidence with
The information to be asked is obtained in fleet row；Record information daily record；Judge whether random queuing data is sufficient, if sufficient, directly
Request data is returned, if random queuing data is inadequate, calls load queue sub-process to load data.
6. the big data random access regime based on reducing subspaces any one of a kind of implementation claim 1-5 is
System, it is characterised in that including following functions module：
Random number generation unit, can be under the support of UsbKey keys, using Mersenne Prime rotation random algorithm generation random number
And data slicer, and random number and data slicer encryption are stored in database；
Random sequence loads Dispatching Unit, can close parameter according to last time or acquiescence starts under the support of UsbKey keys
Parameter, the random number stored in loading of databases and data slicer are with combination producing random sequence, the stochastic ordering that then will be generated
Row are distributed to n random queue, and the random sequence for needing to load is decrypted and filled using reducing subspaces algorithm from data slicer
It is downloaded in random queue, to increase loading degree of randomness；
Monitoring unit, the data slicer information according to loading decides whether the new data slicer information of loading, when in random queue
Data deficiencies when, triggering continue load data simultaneously be distributed to random queue, to monitor the data in random queue；
Data capture unit, according to the reducing subspaces parameter reference obtained in request of data, obtains accordingly about in Usbkey
Plucked instrument husband's parameter, data are obtained from random queue.
7. the system that implementation according to claim 6 is based on the big data random access regime of reducing subspaces, its feature exists
In random sequence loads Dispatching Unit includes random sequence loader, queue distributor and random queue, wherein random sequence
Loader can be in the case where the key of Usbkey be supported, comprehensive random number, data slicer and system last time close parameter, are cut from data
The random sequence for needing to load is decrypted and is loaded into database in piece；Queue distributor is responsible for random sequence loader to fill
The random sequence of load is distributed in corresponding n random queue, and largest random is reached as far as possible；Random queue storage is random
The data of sequence, data rate is taken to improve.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610907792.XA CN106844411B (en) | 2016-10-19 | 2016-10-19 | Joseph ring-based big data random access system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610907792.XA CN106844411B (en) | 2016-10-19 | 2016-10-19 | Joseph ring-based big data random access system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106844411A true CN106844411A (en) | 2017-06-13 |
| CN106844411B CN106844411B (en) | 2020-03-17 |
Family
ID=59145980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610907792.XA Active CN106844411B (en) | 2016-10-19 | 2016-10-19 | Joseph ring-based big data random access system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106844411B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108566262A (en) * | 2018-02-12 | 2018-09-21 | 上海航天电子有限公司 | A kind of satellite test data comparison method for Cortex data formats |
| CN110190947A (en) * | 2019-04-08 | 2019-08-30 | 努比亚技术有限公司 | Information encryption and decryption method, terminal and computer readable storage medium |
| TWI707246B (en) * | 2018-11-05 | 2020-10-11 | 開曼群島商現代財富控股有限公司 | Key management system based on distributed multi-layered recursive and method thereof |
| CN112000942B (en) * | 2020-10-30 | 2021-01-22 | 成都掌控者网络科技有限公司 | Authority list matching method, device, equipment and medium based on authorization behavior |
| CN113467974A (en) * | 2021-07-21 | 2021-10-01 | 上海云海万邦数据科技有限公司 | Identification information generation method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101815993A (en) * | 2007-09-14 | 2010-08-25 | 万年知力有限公司 | Data security apparatus |
| CN103929563A (en) * | 2014-04-11 | 2014-07-16 | 陕西师范大学 | Image encryption and decryption method based on improved Joseph traversal and generalized Henon mapping |
| CN105045561A (en) * | 2015-08-12 | 2015-11-11 | 中国电子科技集团公司第四十一研究所 | Pseudo-random number generating method |
-
2016
- 2016-10-19 CN CN201610907792.XA patent/CN106844411B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101815993A (en) * | 2007-09-14 | 2010-08-25 | 万年知力有限公司 | Data security apparatus |
| CN103929563A (en) * | 2014-04-11 | 2014-07-16 | 陕西师范大学 | Image encryption and decryption method based on improved Joseph traversal and generalized Henon mapping |
| CN105045561A (en) * | 2015-08-12 | 2015-11-11 | 中国电子科技集团公司第四十一研究所 | Pseudo-random number generating method |
Non-Patent Citations (1)
| Title |
|---|
| 王洋 等: "约瑟夫环在加密算法中应用", 《电子制作》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108566262A (en) * | 2018-02-12 | 2018-09-21 | 上海航天电子有限公司 | A kind of satellite test data comparison method for Cortex data formats |
| TWI707246B (en) * | 2018-11-05 | 2020-10-11 | 開曼群島商現代財富控股有限公司 | Key management system based on distributed multi-layered recursive and method thereof |
| CN110190947A (en) * | 2019-04-08 | 2019-08-30 | 努比亚技术有限公司 | Information encryption and decryption method, terminal and computer readable storage medium |
| CN110190947B (en) * | 2019-04-08 | 2022-09-13 | 重庆葵海数字科技有限公司 | Information encryption and decryption method, terminal and computer readable storage medium |
| CN112000942B (en) * | 2020-10-30 | 2021-01-22 | 成都掌控者网络科技有限公司 | Authority list matching method, device, equipment and medium based on authorization behavior |
| CN113467974A (en) * | 2021-07-21 | 2021-10-01 | 上海云海万邦数据科技有限公司 | Identification information generation method |
| CN113467974B (en) * | 2021-07-21 | 2024-04-05 | 上海南洋万邦软件技术有限公司 | Identification information generation method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106844411B (en) | 2020-03-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106844411A (en) | A kind of big data random access system and method based on reducing subspaces | |
| CN108595989B (en) | Mobile APP safety protection system and method under iOS | |
| Wong et al. | Security in outsourcing of association rule mining | |
| Ohrimenko et al. | The Melbourne shuffle: Improving oblivious storage in the cloud | |
| CN109474423A (en) | Data encryption/decryption method, server and storage medium | |
| CN105426775B (en) | A kind of method and system for protecting smart mobile phone information security | |
| CN116383857B (en) | Cross-platform authentication-based storage method, device and storage medium | |
| CN104396182A (en) | Method of encrypting data | |
| US10572635B2 (en) | Automatic correction of cryptographic application program interfaces | |
| CN115442163A (en) | Big data one-way encryption transmission method based on asymmetric algorithm | |
| EP3697021B1 (en) | Secure and encrypted logging systems and methods with data recovery | |
| CN114499875A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN111737747B (en) | Database confidentiality method, device, equipment and computer storage medium | |
| CN116361774A (en) | Password cracking method and device | |
| RU2538913C2 (en) | Method for depersonalisation of personal data | |
| US20190305934A1 (en) | System and methods for recording codes in a distributed environment | |
| CN115314228A (en) | Unmanned aerial vehicle identity authentication method, device and system | |
| CN112153046B (en) | Data encryption and data decryption method, related equipment and storage medium | |
| CN114693410A (en) | Verification method and system based on block chain | |
| CN112989321A (en) | Secret sharing algorithm-based key management method and system | |
| CN114448650B (en) | Encryption authentication method, device, equipment and storage medium | |
| CN106997299A (en) | Encryption and hot update method, device | |
| CN112738108B (en) | Multi-data encryption system and method for cloud computing system | |
| CN117390684B (en) | Data encryption processing method and system based on slice level association | |
| CN117633849B (en) | Intelligent control method and system for whole flow of coal mine operation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |