CN106844411B - Joseph ring-based big data random access system and method - Google Patents
Joseph ring-based big data random access system and method Download PDFInfo
- Publication number
- CN106844411B CN106844411B CN201610907792.XA CN201610907792A CN106844411B CN 106844411 B CN106844411 B CN 106844411B CN 201610907792 A CN201610907792 A CN 201610907792A CN 106844411 B CN106844411 B CN 106844411B
- Authority
- CN
- China
- Prior art keywords
- random
- data
- queue
- usbkey
- sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
- G06F21/34—User authentication involving the use of external additional devices, e.g. dongles or smart cards
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2107—File encryption
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Storage Device Security (AREA)
Abstract
The invention discloses a Joseph ring-based big data random access system and a method, wherein the method comprises the following steps: under the support of a UsbKey key, random numbers and data slices are generated by using an MT algorithm and are encrypted and stored in a database; loading random numbers and data slices stored in a database according to the last closing parameter or the default starting parameter to generate random sequences in a combined mode and distribute the random sequences to n random queues, and decrypting the random sequences needing to be loaded from the data slices by using a Joseph ring algorithm and loading the random sequences into the random queues; monitoring whether new data slice information needs to be loaded or not according to the loaded data slice information; and acquiring corresponding Josephson parameters in the Usbkey according to the data acquisition request, and acquiring data from the random queue. The method of the invention fully considers the safety and the randomness of the random data, realizes the high-speed acquisition of the random data by utilizing the computer technology, and meets the problem of data skewness in the large data access.
Description
Technical Field
The present invention relates to a big data random access system and method, and more particularly, to a big data random access system and method based on joseph ring.
Background
Random number algorithms are used in a very wide variety of applications in today's world, such as finance, machine manufacturing, IT networks, and so on. This has also led to more intensive research into random number arithmetic, and even true random number websites that provide random number services currently use atmospheric noise or some unpredictable wide range of random sources to generate random numbers.
In practical applications, what random number algorithm should be used needs to be selected according to different requirements of the system. For simpler requirements, such as random verification codes of websites, the requirements can be met by using a pseudo-random algorithm carried by an operating system. For bank passwords or data encryption, the requirements of random numbers are very strict or even very strict, and once a problem occurs, a great loss can be caused.
Meanwhile, the requirement of large data partition on random number algorithm is very high, which directly influences whether the data storage is scattered uniformly or not. A good random algorithm can reduce the deviation of data, thereby improving the query speed.
With the progress of the technology, the method of generating random numbers by combining a pseudo-random algorithm with very high randomness and a series of complex internal logic mechanisms can meet the requirements. Thus, although the complexity of the program is increased, additional hardware support is not required to be added, and the non-repeatability of the random number can be ensured within thousands or even tens of thousands of years.
Therefore, there is a need in the art to develop a system and method that employs a pseudo-random algorithm in combination with an internal logic mechanism to generate random numbers with low data skew and high security.
Disclosure of Invention
To solve the above-mentioned problems in the prior art, the present invention provides a josephson ring-based big data random access system and method.
In order to achieve the above object, a specific technical solution of a joseph ring-based big data random access method of the present invention is as follows:
a josephson ring based big data random access method, comprising the steps of: under the support of a Usbkey key, generating a random number and a data slice by utilizing a Messen prime number rotation random algorithm, and encrypting and storing the random number and the data slice into a database; secondly, under the support of a Usbkey key, loading random numbers and data slices stored in a database according to a last closing parameter or a default starting parameter to generate random sequences in a combined mode, distributing the generated random sequences to n random queues, decrypting the random sequences to be loaded from the data slices by using a Joseph-ring algorithm and loading the random sequences into the random queues to increase the loading randomness; monitoring whether new data slice information needs to be loaded or not according to the loaded data slice information, and triggering to continue loading data and distributing the data to a random queue when the data in the random queue is insufficient; and step four, acquiring corresponding Joseph parameters from Usbkey according to the Joseph ring parameter index in the data acquisition request, and acquiring data from the random queue.
According to another aspect of the present invention, there is also provided a joseph ring-based big data random access system, the system including the following functional modules: the random number generating unit can generate random numbers and data slices by utilizing a Messen prime number rotation random algorithm under the support of a Usbkey key, and encrypts and stores the random numbers and the data slices into a database; the random sequence loading and distributing unit can load random numbers and data slices stored in a database to generate random sequences in a combined mode according to a last closing parameter or a default starting parameter under the support of a Usbkey key, then distribute the generated random sequences to n random queues, decrypt the random sequences needing to be loaded from the data slices by using a Joseph ring algorithm and load the random sequences into the random queues to increase the loading randomness; the monitoring unit determines whether to load new data slice information according to the loaded data slice information, and when the data in the random queue is insufficient, the monitoring unit triggers to continuously load the data and distributes the data to the random queue so as to monitor the data in the random queue; and the data acquisition unit acquires corresponding Joseph parameters from the Usbkey according to the Joseph ring parameter index in the data acquisition request, and acquires data from the random queue.
In the Joseph ring-based big data random access system and method, the required random number is generated by combining the Messen prime number rotation random algorithm (MT pseudo-random algorithm) with the internal logic of the Joseph ring, the randomness of the random number is ensured, the security and confidentiality of data are met, and even a system designer cannot obtain the information of the random number generated by the system in practical application.
Drawings
FIG. 1 is a schematic diagram of a Joseph ring-based big data random access system according to the present invention;
FIG. 2 is a schematic diagram of the operation of the Joseph ring-based big data random access system of the present invention;
fig. 3 illustrates a random sequence generation flow in the josephson ring-based big data random access method of the present invention;
FIG. 4 illustrates a random sequence loading flow in the Joseph-Ring based big data random access method of the present invention;
FIG. 5 illustrates a monitoring flow in the Joseph ring-based big data random access method of the present invention;
fig. 6 shows an acquisition data flow in the josephson ring-based big data random access method of the present invention.
Detailed Description
For a better understanding of the objects, structure and function of the present invention, the josephson ring-based big data random access system and method of the present invention will be described in further detail with reference to the accompanying drawings.
As shown in fig. 1-2, the joseph ring-based big data random access system of the present invention comprises the following functional blocks:
and the random number generating unit can generate random numbers and data slices by utilizing a Messen prime number rotation random algorithm (MT algorithm) under the support of a Usbkey (identity authentication) key, and encrypt and store the random numbers and the data slices into a file system (such as a database).
The random sequence loading and distributing unit can load random numbers and data slices stored in a database to generate random sequences in a combined mode according to last closing parameters or default starting parameters under the support of a Usbkey key, then distribute the generated random sequences to n random queues, decrypt the random sequences needing to be loaded from the data slices by using a Joseph ring algorithm and load the random sequences into the random queues to increase the loading randomness.
And the monitoring unit determines whether to load new data slice information according to the loaded data slice information, and when the data in the random queue is insufficient, the monitoring unit triggers to continuously load the data and distributes the data to the random queue so as to monitor the data in the random queue.
And the data acquisition unit acquires corresponding Joseph parameters from the Usbkey according to the Joseph ring parameter index in the data acquisition request, and acquires data from the random queue.
The random sequence loading and distributing unit comprises a random sequence loader, a queue distributor and a random queue. The random sequence loader can synthesize random numbers, data slices and last-time system shutdown parameters under the support of a Usbkey key, decrypt random sequences to be loaded from the data slices and load the random sequences into a database. The queue distributor is responsible for distributing the random sequence loaded by the random sequence loader to the corresponding n random queues to achieve the maximum randomness as far as possible. The random queue stores data of a random sequence, which is equivalent to the function of buffering, and can improve the data fetching rate. Once the data in the random queue is less, the monitoring unit or the data acquisition unit initiates an operation of loading the random queue to the random sequence loader.
The respective functional units constituting the above system are described in detail below.
Further, Mersenne Twister random number Algorithm (Messen prime number rotation random Algorithm, MT19937) is used in the random number generation unit. The MT algorithm is a pseudo-random number generation algorithm, can quickly generate high-quality pseudo-random numbers based on matrix linear recursion on a finite binary field, and corrects a plurality of defects of the classical random number generation algorithm. This algorithm usually uses two similar variants, the variants differing in that different meisenna prime numbers are used, one more recent and more common being MT19937, 32 bit word length, and one variant being the 64 bit version of MT 19937-64. For a length of k bits, the Meisen rotation algorithm generates discrete uniformly distributed random numbers between intervals of [0, 2k-1 ].
Random calculations, typically performed using a variant of the metson rotation algorithm MT19937, can produce a 32-bit integer sequence with the following advantages:
1) the cycle is 219937-1, which is long enough to ensure the requirement of not repeating in 5000 hundred million;
2) the dimensions of k is more than or equal to 1 and less than or equal to 623 can be distributed equally;
3) except for statistically incorrect random number generators, is the fastest of all pseudo-random number generator methods.
The random number is generated while the number of the random numbers generated by the algorithm is recorded, and the generated data amount is skipped first when the random data is generated next time, so that the non-repeatability of the pseudo random number can be ensured, and the random seed is unchanged.
In order to ensure the safety of data and reduce the risk in the data generation and storage processes, the data slicing technology is used for storing record information, an MT algorithm generates complete data to be stored in a memory, anyone cannot know specific content, but once the data is stored, the risk of leakage exists, and therefore the file of the slices is required to be uniformly encrypted in addition to the data scattering slices.
The recorded data is generated by taking a pieces of data as a group, namely, a is also taken as a unit, firstly, each piece of data is randomly cut into n parts to form n files, then the n files are encrypted, then the n files are respectively written into a file system, and the steps before the n files are written into the file system are all carried out in a memory, so that a data generator can be ensured not to have a chance to check the content of the files. When the a records are scattered into n files, corresponding slice information is generated and is stored in a database after being written into a file system. During operation, Usbkey is guaranteed to be effective, and the encryption key can be normally obtained. Any problem occurs in any link in the period, the random record generation fails, and the operation is rolled back after the random record generation fails, so that the integrity of data is ensured.
Thus, the random number generating unit is responsible for generating random numbers using the MT algorithm and storing the data slices in a file system, with the relevant information and slice information stored as a database. The operation of generating random numbers is performed manually and is not automatic and requires key support by Usbkey.
Further, the random sequence loader operates each time the system is started and the random queue data is insufficient. The method comprises the steps of loading record information which is closed and stored last time when starting (the data is not enough and is not loaded because the record information is loaded), judging which batch of data slice files needs to be loaded according to the information, judging which batch of data slice files are sold, and randomly loading a random sequence into a random queue by using a queue distributor from the current batch of data slice files until the random queue is full of the record information.
The random sequence formed by the data slice files loaded by the random sequence loader is resident in the memory unless the sequence is acquired. Once the monitoring unit finds that the random queue data is insufficient, the random sequence loader is triggered to execute the loading operation. The random sequence loader firstly checks whether the currently maintained random sequence can meet the data requirement of the random queue, if not, the random sequence loader loads the subsequent data slice file to supplement the random sequence, and after the check, the queue distributor is called to fill the random queue.
Further, the queue distributor is used for supplementing random queue data, and the random queue data is executed in the case that the system is started or the monitoring unit finds that the random queue is insufficient. When the system is started, the random queue is just generated, and no data exists, so the queue distributor randomly places the data acquired from the random sequence loader into a certain random queue, and the random algorithm can use a random function carried by the system or an MT algorithm until all n random queues are full.
Furthermore, the random queues are created for ensuring the speed of fetching the record information, which is equivalent to a fast cache, and the other purpose is to increase the randomness of the data fetching, wherein n random queues are provided, and when the data is fetched each time, k data are skipped from one random queue according to the Joseph ring algorithm to obtain the requested record, so that the randomness of the data fetching is increased. The length of the random queue needs to be adjusted according to the actual sales condition, the length of the queue with large sales volume is correspondingly increased, and the sales volume can be properly reduced when the sales volume is small.
Further, when the monitoring unit finds that the data of a certain random queue is less than b% (10% -20%), the random sequence loader is triggered, and after the random sequence loader checks the data of the random sequence loader, the queue distributor is called to distribute the data to the random queue until the random queue is full. The monitoring unit can be continuously and circularly executed after the system is started, and is a timing task. It monitors the amount of data in the random queue and, once below a specified value, invokes the random sequence loader to load the data.
The monitoring unit monitors data in the random queue, and once the data in the random queue is insufficient, the data loader is requested to load the data so as to distribute the data to the random queue, and the data loader determines whether to load new data slice information according to the currently loaded data slice information.
In addition, the monitoring unit also checks the number of the sales logs of the data acquisition unit, and records the logs into a database for storage once the data volume reaches a specified value or exceeds a specified maximum time limit, so that the performance reduction caused by frequent updating is avoided. The monitoring unit ensures that failure caused by lack of data can not occur, and simultaneously, the data fetching speed is improved because the database is not updated in real time.
Further, the data acquisition unit acquires data by adopting a Joseph ring, wherein the Joseph ring is a mathematical application problem, and the mathematical derivation formula is as follows: f 1 ═ 1, f n ═ f n-1 + k)% n (n > 1).
The data acquisition unit uses Joseph ring when acquiring data, a random specified series of n, k and m values are stored in Usbkey, the request for acquiring data has an index using the n, k and m values, the index value is random, and the group of n, k and m values in the Usbkey is acquired according to the index, thereby acquiring the record number to be acquired in a random queue. The Joseph ring can make the acquired data more random, and due to the Usbkey encryption storage parameters, the record which the request can return to can not be calculated even if the data in the random queue is known.
The acquiring of data here means acquiring a random number, the acquiring of data is performed by the data acquiring unit, and the data is acquired only when a request is received. When acquiring Joseph ring parameters from Usbkey, the data acquisition unit acquires the parameters and calculates the data to be returned from the data in the random queue according to a formula. After the data is acquired, the data acquisition unit records the acquisition record and then returns the data to the request.
After the data acquisition unit returns data to the corresponding request, the data acquisition unit should check the data condition of each random queue, and once the monitoring unit is found that the data in the random queue is insufficient (less than 10% -20% of the queue length), the monitoring unit should notify the monitoring unit, the monitoring unit will call the random sequence loader to load the data, and the problem of acquisition failure due to insufficient data will not occur.
Further, in the system, Usbkey is a safety key, and a key for storing slice data and josephson loop parameters used by the data acquisition unit have a serious problem once leakage occurs. The system uses the current advanced Usbkey technology to encrypt and store our data, and strictly controls the use authority of the Usbkey to ensure the safety of the data to the maximum extent.
Therefore, the system for randomly acquiring data based on the Joseph ring not only is a process for generating a random sequence, but also increases the randomness and the safety of acquiring data in the process of acquiring the random sequence. By using the Usbkey encryption method, the situation that an operator cannot control the acquisition of random numbers can be ensured, and even if the Usbkey leaks, because the ID acquired each time has considerable randomness, the situation that the desired data cannot be acquired at will can be ensured.
In addition, the invention combines the data slicing technology, the encryption technology and the database, thereby ensuring the safety of data, having few personnel involved in the production process and very good confidentiality, and ensuring that production personnel can not obtain any recorded information in the whole process.
Although the system of the invention has a good consideration on safety and random performance, the complexity of the system is increased, the data fetching speed is influenced, and the system overhead is larger because a large amount of data resides in a memory and has a redundant condition. However, as current server performance has advanced significantly, overhead considerations are not excessive. As for the speed of data fetching, the data fetching speed can be controlled by adjusting the parameters of each part and improving the code writing quality.
On the other hand, the invention also discloses a method for random access by using the Joseph ring-based big data random access system, which is concretely as follows.
The inventive large data random access method based on Joseph ring comprises the following steps:
the method comprises the steps of firstly, generating random numbers and data slices by utilizing a Messen prime number rotation random algorithm (MT algorithm) under the support of a Usbkey (identity authentication) key, and encrypting and storing the random numbers and the data slices into a file system (such as a database).
And secondly, loading random numbers and data slices stored in a database to generate random sequences in a combined manner according to the last closing parameter or the default starting parameter under the support of the Usbkey key, then distributing the generated random sequences to n random queues, decrypting the random sequences to be loaded from the data slices by using a Joseph-ring algorithm, and loading the random sequences to be loaded into the random queues to increase the loading randomness.
And step three, determining whether to load new data slice information according to the loaded data slice information, and triggering to continuously load data and distributing the data to the random queue when the data in the random queue is insufficient so as to monitor the data in the random queue.
And step four, acquiring corresponding Joseph parameters from Usbkey according to the Joseph ring parameter index in the data acquisition request, and acquiring data from the random queue.
Further, as shown in fig. 3, in step two, the generation flow of the random sequence is as follows:
1) starting a data generation client, checking whether the Usbkey is normal or not, and inputting parameters such as batch, quantity and the like of generated data;
2) generating a recording random sequence;
3) slicing the generated recording sequence into n parts, putting the n parts of files into the n parts of files, and encrypting the n parts of files by using a key obtained from Usbkey;
4) the data slice is stored in a concrete file, and information related to the data slice is stored in a database.
The random data generation flow is a flow for generating the recorded data ID, the data generation flow is responsible for the random number generator, and the random number generator is responsible for the generation of the random number, the slice encryption storage of the data and the warehousing operation of the data information. These operations require support from the Usbkey (whether the key is normal or not), and the key cannot be obtained if the system is not inserted into the Usbkey.
Further, as shown in fig. 4, in step two, the flow of random sequence loading is as follows:
1) checking whether the necessary Usbkey is normal, and if not, directly failing to quit;
2) acquiring parameters of the last time when the system quits, and using default configuration parameters if the system quits for the first time;
3) decrypting the slice file by using the key acquired from the Usbkey;
4) generating an ID sequence according to the slice file and information in the database;
5) and sequentially acquiring data from the ID sequence, and randomly distributing the data to any random queue.
The random sequence loading process is executed during each restart, and the main function is to load data slice files according to the last parameters, combine the data slice files into a random sequence and distribute the random sequence to a random queue to be acquired. The related modules comprise a random sequence loader, a queue distributor, a random queue and a Usbkey.
If it is the first boot, then the last boot parameters should be the default first boot parameters for the system. In addition, when decrypting a slice file, it is possible to scan whether the file exists, whether the database information is complete, and the like.
Further, as shown in fig. 5, in step three, the monitoring process of the random queue is as follows:
1) judging whether a queue with insufficient data and 20% of maximum data exists in the m random queues;
2) if the random queue data is less than 20% of the maximum data, judging whether the slice data loaded in the random sequence is less than 20% of the maximum loaded slice data;
3) if the data in the random sequence is less than 20% of the maximum loaded slice data, taking down a group of slice data files, taking the decryption key to combine into a new random sequence, and continuing to the current random sequence;
4) sequentially extracting data from the random sequence and randomly inserting the data into random queues with insufficient data until all the random queues are full;
5) judging whether the sales log of the acquired data exceeds the maximum value or not, and if so, recording the sales log into a warehouse;
6) judging whether the time of the unwritten log of the acquired data exceeds a specified value or not, and if so, recording the log into a warehouse;
7) and (5) after the judgment is finished, sleeping for a period of time, and continuously returning to the step 1) for circular judgment.
The monitoring process is a loop process, which is performed at regular intervals from the start of the system until the system is shut down.
Further, as shown in fig. 6, in step four, the process of acquiring data is as follows:
1) verifying whether the request parameters and the Usbkey are normal or not, and if not, directly returning an error;
2) acquiring a used Joseph ring parameter from Usbkey according to a random number parameter of the request parameter;
3) determining from which of the m random queues data should be taken according to josephson ring parameters;
4) acquiring information to be requested in a random queue of the data acquisition according to Joseph ring parameters;
5) recording an information log;
6) and judging whether the random queue data is sufficient, if so, directly returning the request data, and if so, calling a loading queue subprocess to load the data (the loading queue subprocess is an asynchronous process, and the function returns after calling and then returns the request data).
The data acquisition unit and the random queue are executed certainly, and the random sequence loader and the queue distributor are executed only when judging that the data in the queue is insufficient after the data are acquired.
The data acquisition needs to request to carry random Joseph ring index data, corresponding Joseph ring parameters are found in Usbkey according to the index data, and therefore the random queue data to be acquired and the data to be acquired are calculated according to the parameters, and the randomness of the acquired data is increased.
The method of the invention fully considers the safety and the randomness of the random data, realizes the high-speed acquisition of the random data by utilizing the modern computer technology, and meets the problem of data skewness in the large data access.
The first embodiment is as follows:
the system of the invention adopts MT random number algorithm to generate random sequence according to the prescribed rule, uses slicing technique to cut each random number record ID into n (5) pieces, randomly puts into n (5) random slice files, the storage sequence and rule are stored by database, the slice files are recorded into a group by m (1000 ten thousand) pieces, each slice file is stored by encryption algorithm, the encryption key is stored in Usbkey dongle.
The system firstly decrypts the random slice file and combines with the database storage information to form a complete record number sequence at each start, then k (20) random queues are generated according to the last shutdown storage parameters, and each random queue is filled with a (50000) record data and is ready to return ID data to the client.
When the client requests data, a random ID is randomly extracted from the 20 random queues according to request parameters and a Joseph ring algorithm and returned. Once the data in the queue is less than 20% (10000) of the full queue, the system extracts the data from the random sequence according to the rule to fill the queue.
The encryption Usbkey stores a key and a parameter array required by the Joseph ring, when a client requests a random sequence, a random number (or MT algorithm) carried by the system is utilized to randomly obtain a parameter, the system obtains the Joseph ring parameter from the Usbkey according to the parameter, and then obtains data from a random queue. When the sale of one set of slice data is less than 20% (200 w), the system loads another set of slice data to avoid the sale from being influenced by the shortage of data.
Therefore, the system of the invention achieves the following effects: random sequences ensure that random data is not repeated within 5000 hundred million (50 hundred million/year x1000 years); support a data rate of at least 2000 frames/second; the data security is ensured, and the possibility of stealing and tampering is avoided; the slice file loading recording sequence time does not exceed 1 minute.
It is to be understood that the present invention has been described with reference to certain embodiments, and that various changes in the features and embodiments, or equivalent substitutions may be made therein by those skilled in the art without departing from the spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (7)
1. A big data random access method based on Joseph ring, comprising the steps of:
under the support of a Usbkey key, generating a random number and a data slice by utilizing a Messen prime number rotation random algorithm, and encrypting and storing the random number and the data slice into a database;
secondly, under the support of a Usbkey key, loading random numbers and data slices stored in a database according to a last closing parameter or a default starting parameter to generate random sequences in a combined mode, distributing the generated random sequences to n random queues, decrypting the random sequences to be loaded from the data slices by using a Joseph-ring algorithm and loading the random sequences into the random queues to increase the loading randomness;
monitoring whether new data slice information needs to be loaded or not according to the loaded data slice information, and triggering to continue loading data and distributing the data to a random queue when the data in the random queue is insufficient;
and step four, acquiring corresponding Joseph parameters from Usbkey according to the Joseph ring parameter index in the data acquisition request, and acquiring data from the random queue.
2. The josephson ring-based big data random access method of claim 1, wherein in the step two, the generation flow of the random sequence comprises the following steps: starting a data generation client, checking whether the Usbkey is normal or not, and inputting batch and quantity parameters of generated data; generating a recording random sequence; slicing the generated recording sequence into n parts, putting the n parts of files into the n parts of files, and encrypting the n parts of files by using a key obtained from Usbkey; the data slice is stored in a concrete file, and information related to the data slice is stored in a database.
3. The josephson ring-based big data random access method of claim 2, wherein in step two, the flow of random sequence loading comprises the steps of: checking whether the necessary Usbkey is normal, and if not, directly failing to quit; acquiring parameters of the last time when the system quits, and using default configuration parameters if the system quits for the first time; decrypting the slice file by using the key acquired from the Usbkey; generating an ID sequence according to the slice file and information in the database; and sequentially acquiring data from the ID sequence, and randomly distributing the data to any random queue.
4. The josephson ring-based big data random access method of claim 1, wherein in step three, the monitoring procedure of the random queue comprises the following steps: judging whether a queue with less than 20% of maximum data exists in the m random queues or not; if the random queue data is less than 20% of the maximum data, judging whether the slice data loaded in the random sequence is less than 20% of the maximum loaded slice data; if the data in the random sequence is less than 20% of the maximum loaded slice data, taking down a group of slice data files, taking the decryption key to combine into a new random sequence, and continuing to the current random sequence; sequentially extracting data from the random sequence and randomly inserting the data into random queues with insufficient data until all the random queues are full; judging whether the sales log of the acquired data exceeds the maximum value or not, and if so, recording the sales log into a warehouse; judging whether the time of the unwritten log of the acquired data exceeds a specified value or not, and if so, recording the log into a warehouse; and after the judgment is finished, sleeping for a period of time, and continuously returning to carry out the circulating judgment.
5. The josephson ring-based big data random access method of claim 1, wherein in step four, the process of acquiring data comprises the steps of: verifying whether the request parameters and the Usbkey are normal or not, and if not, directly returning an error; acquiring a used Joseph ring parameter from Usbkey according to a random number parameter of the request parameter; determining from which of the m random queues data should be taken according to josephson ring parameters; acquiring information to be requested in a random queue of the data acquisition according to Joseph ring parameters; recording an information log; and judging whether the random queue data is sufficient, if so, directly returning the request data, and if not, calling a loading queue sub-process to load the data.
6. A system implementing the Joseph ring-based big data random access method of any one of claims 1-5, comprising the following functional blocks:
the random number generating unit can generate random numbers and data slices by utilizing a Messen prime number rotation random algorithm under the support of a Usbkey key, and encrypts and stores the random numbers and the data slices into a database;
the random sequence loading and distributing unit can load random numbers and data slices stored in a database to generate random sequences in a combined mode according to a last closing parameter or a default starting parameter under the support of a Usbkey key, then distribute the generated random sequences to n random queues, decrypt the random sequences needing to be loaded from the data slices by using a Joseph ring algorithm and load the random sequences into the random queues to increase the loading randomness;
the monitoring unit determines whether to load new data slice information according to the loaded data slice information, and when the data in the random queue is insufficient, the monitoring unit triggers to continuously load the data and distributes the data to the random queue so as to monitor the data in the random queue;
and the data acquisition unit acquires corresponding Joseph parameters from the Usbkey according to the Joseph ring parameter index in the data acquisition request, and acquires data from the random queue.
7. The system for implementing josephson ring-based big data random access method according to claim 6, wherein the random sequence loading and distributing unit comprises a random sequence loader, a queue distributor and a random queue, wherein the random sequence loader can integrate random numbers, data slices and system last-time shutdown parameters under the key support of Usbkey, decrypt the random sequences to be loaded from the data slices and load the random sequences into the database; the queue distributor is responsible for distributing the random sequence loaded by the random sequence loader to the corresponding n random queues; the random queue stores a random sequence of data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610907792.XA CN106844411B (en) | 2016-10-19 | 2016-10-19 | Joseph ring-based big data random access system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610907792.XA CN106844411B (en) | 2016-10-19 | 2016-10-19 | Joseph ring-based big data random access system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106844411A CN106844411A (en) | 2017-06-13 |
| CN106844411B true CN106844411B (en) | 2020-03-17 |
Family
ID=59145980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610907792.XA Active CN106844411B (en) | 2016-10-19 | 2016-10-19 | Joseph ring-based big data random access system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106844411B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108566262A (en) * | 2018-02-12 | 2018-09-21 | 上海航天电子有限公司 | A kind of satellite test data comparison method for Cortex data formats |
| TWI707246B (en) * | 2018-11-05 | 2020-10-11 | 開曼群島商現代財富控股有限公司 | Key management system based on distributed multi-layered recursive and method thereof |
| CN110190947B (en) * | 2019-04-08 | 2022-09-13 | 重庆葵海数字科技有限公司 | Information encryption and decryption method, terminal and computer readable storage medium |
| CN112000942B (en) * | 2020-10-30 | 2021-01-22 | 成都掌控者网络科技有限公司 | Authority list matching method, device, equipment and medium based on authorization behavior |
| CN113467974B (en) * | 2021-07-21 | 2024-04-05 | 上海南洋万邦软件技术有限公司 | Identification information generation method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101815993A (en) * | 2007-09-14 | 2010-08-25 | 万年知力有限公司 | Data security apparatus |
| CN103929563A (en) * | 2014-04-11 | 2014-07-16 | 陕西师范大学 | Image encryption and decryption method based on improved Joseph traversal and generalized Henon mapping |
| CN105045561A (en) * | 2015-08-12 | 2015-11-11 | 中国电子科技集团公司第四十一研究所 | Pseudo-random number generating method |
-
2016
- 2016-10-19 CN CN201610907792.XA patent/CN106844411B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101815993A (en) * | 2007-09-14 | 2010-08-25 | 万年知力有限公司 | Data security apparatus |
| CN103929563A (en) * | 2014-04-11 | 2014-07-16 | 陕西师范大学 | Image encryption and decryption method based on improved Joseph traversal and generalized Henon mapping |
| CN105045561A (en) * | 2015-08-12 | 2015-11-11 | 中国电子科技集团公司第四十一研究所 | Pseudo-random number generating method |
Non-Patent Citations (1)
| Title |
|---|
| 约瑟夫环在加密算法中应用;王洋 等;《电子制作》;20150630(第11期);第41页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106844411A (en) | 2017-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106844411B (en) | Joseph ring-based big data random access system and method | |
| Ohrimenko et al. | The Melbourne shuffle: Improving oblivious storage in the cloud | |
| US10721062B2 (en) | Utilizing error correction for secure secret sharing | |
| JP6285536B2 (en) | System and method for encrypting data | |
| US10536266B2 (en) | Cryptographically securing entropy for later use | |
| Storer et al. | POTSHARDS: secure long-term storage without encryption | |
| US10650169B2 (en) | Secure memory systems | |
| US11216577B1 (en) | Systems, methods, and computer-readable media for utilizing anonymous sharding techniques to protect distributed data | |
| US10846373B2 (en) | Method and system for securing a client's access to a DRM agent's services for a video player | |
| CN109586898B (en) | Dual-system communication key generation method and computer-readable storage medium | |
| Guan et al. | Hadoop-based secure storage solution for big data in cloud computing environment | |
| CN107980135B (en) | Method and system for generating random number of terminal | |
| EP3207727A1 (en) | Method to manage subscriptions in a provisioning server | |
| CN115361198A (en) | Decryption method, encryption method, device, computer equipment and storage medium | |
| Du et al. | Secure and verifiable keyword search in multiple clouds | |
| FR3010598A1 (en) | METHOD FOR MANAGING COHERENCE COACHES | |
| CN201364576Y (en) | Network software account password and computer system safety device | |
| CN111914280B (en) | File self-encryption and decryption method | |
| Choudary et al. | Security analysis and decryption of filevault 2 | |
| US11809323B1 (en) | Maintaining real-time cache coherency during distributed computational functions | |
| CN115589289B (en) | Service processing method and system for server cipher machine | |
| Karnakanti | Reduction of spatial overhead in decentralized cloud storage using IDA | |
| CN106997299A (en) | Encryption and hot update method, device | |
| US11943338B2 (en) | Object-level encryption | |
| KR102441641B1 (en) | Apparatus and method for managing security information based on random slicing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |