CN111177782A - Method and device for extracting distributed data based on big data and storage medium - Google Patents
Method and device for extracting distributed data based on big data and storage medium Download PDFInfo
- Publication number
- CN111177782A CN111177782A CN201911405058.3A CN201911405058A CN111177782A CN 111177782 A CN111177782 A CN 111177782A CN 201911405058 A CN201911405058 A CN 201911405058A CN 111177782 A CN111177782 A CN 111177782A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- storage database
- distributed
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Storage Device Security (AREA)
Abstract
The application provides a method, a device, a storage medium and a processor for extracting distributed data based on big data, wherein the extraction method comprises the following steps: establishing a basic storage database by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data; limiting access rights of data in the underlying storage database using at least the authorization key; data are extracted according to the access authority, a basic storage database is established by adopting a distributed column storage database and a distributed retrieval engine, the rapid retrieval of the data and the larger data throughput are realized, the access authority of the data in the basic storage database is limited through an authorization key, the access of the data is further limited, more important clients are preferentially allowed to access the database to extract the data, and the rapid extraction of the data is further realized.
Description
Technical Field
The application relates to the field of big data, in particular to a method and a device for extracting distributed data based on big data, a storage medium and a processor.
Background
The current big data platform is mainly used for data storage service of big data, and as a bottom layer support server, the server often crashes due to sudden flow increase, and normal business process is affected.
Meanwhile, because the application mechanism is continuously established and perfected, a service decoupling mechanism is indispensable, and according to the importance degree of application, under the condition of limited resources, reasonable scheduling and control are particularly important.
The existing big data technology of big data has no support system for responding to external services of data, so that calling among application systems cannot be limited, and resources of a big data platform are unreasonably utilized.
Due to the defects of the big data technology in the prior art, the extraction efficiency of the distributed data based on the big data is low.
The above information disclosed in this background section is only for enhancement of understanding of the background of the technology described herein and, therefore, certain information may be included in the background that does not form the prior art that is already known in this country to a person of ordinary skill in the art.
Disclosure of Invention
The present application mainly aims to provide a method, an apparatus, a storage medium, and a processor for extracting distributed data based on big data, so as to solve the problem in the prior art that the extraction efficiency of distributed data based on big data is low.
In order to achieve the above object, according to an aspect of the present application, there is provided a big data-based distributed data extraction method, including: establishing a basic storage database by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data; restricting access rights to the data in the base storage database using at least an authorization key; and extracting the data according to the access right.
Further, the authorization key includes an authorization code, and at least the authorization key is used to limit the access right of the data in the basic storage database, including: and limiting the access authority of the data in the basic storage database by adopting the authorization code, wherein the authorization code is used for identifying the importance degree of a user, a data type list, an ip address of a server which is authorized to access, the maximum single derivable data volume and the number of tasks which can be submitted in unit time.
Further, restricting access to the data in the underlying storage database using at least an authorization key, comprising: and limiting the access right of the data in the basic storage database by adopting the authorization key and a priority algorithm, wherein the priority algorithm is used for identifying the priority of the task.
Further, extracting the data according to the access right includes: storing the task into a task cache queue according to the priority of the task; generating a unique task id according to the authorization code, the task submission time and the task submission sequence, and feeding the task id back to the user; acquiring the task with the highest priority from a task export queue; dividing the task with the highest priority into a plurality of subtasks according to the time range, the data type and the node number of the task with the highest priority, wherein the task export queue is obtained according to the task cache queue; the subtasks are evenly distributed to all the export nodes according to the subtask id; extracting the data from the export node.
Further, the index information is a primary key of the data, the primary key is a unique identifier of the data, and the primary key is composed of a generation date of the data, a code value of the data, and a hash value of the data.
Further, the extraction method further comprises: shielding all external ports through a firewall mechanism, and opening a unique port for providing data extraction service; the user submits the authorization code and the task id to inquire the task state and returns the execution state of the task; and the data extraction program feeds the execution state back to the user.
According to another aspect of the present application, there is provided a big data-based distributed data extraction apparatus, including: the system comprises an establishing unit, a searching unit and a searching unit, wherein the establishing unit is used for establishing a basic storage database by adopting a distributed column storage database and a distributed searching engine, the distributed column storage database is used for storing data, and the distributed searching engine is used for storing index information of the data; a restriction unit for restricting access rights of the data in the base storage database using at least an authorization key; and the extraction unit is used for extracting the data according to the access right.
Further, the authorization key includes an authorization code, and the restriction unit includes: and the limiting module is used for limiting the access authority of the data in the basic storage database by adopting the authorization code, and the authorization code is used for identifying the importance degree of a user, a data type list, an ip address of a server which is authorized to access, the maximum single derivable data volume and the number of tasks which can be submitted in unit time.
According to still another aspect of the present application, there is provided a storage medium including a stored program, wherein the program executes any one of the extraction methods.
According to yet another aspect of the present application, there is provided a processor for executing a program, wherein the program executes to perform any one of the extraction methods.
According to the technical scheme, firstly, a basic storage database is established by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data; then, at least adopting an authorization key to limit the access authority of the data in the basic storage database; and finally, extracting data according to the access authority. According to the method, the basic storage database is established by adopting the distributed column storage database and the distributed retrieval engine, so that the rapid retrieval of data and the high data throughput are realized, the access authority of the data in the basic storage database is subsequently limited through the authorization key, the access of the data is further limited, more important clients are preferentially allowed to access the database for data extraction, the rapid extraction of the data is further realized, the extraction efficiency of the data is high, and the problem that the extraction efficiency of the distributed data based on the big data in the prior art is low is solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 is a schematic diagram illustrating a big data-based distributed data extraction method according to an embodiment of the present application; and
fig. 2 shows a schematic diagram of an extraction apparatus for big data-based distributed data according to an embodiment of the present application.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Also, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element or "connected" to the other element through a third element.
For convenience of description, some terms or expressions referred to in the embodiments of the present application are explained below:
hbase: the distributed column storage database is a distributed and column-oriented open source database, and the Hbase is different from a general database and is a database suitable for unstructured data storage.
Elastic search: a distributed data retrieval engine is a search server based on Lucene, provides a full-text search engine with distributed multi-user capability, and is a popular enterprise-level search engine.
Database primary key: refers to a combination of one or more columns whose value uniquely identifies each row in the table by which the physical integrity of the table is enforced, the primary key being used primarily in association with the foreign keys of other tables, as well as modification and deletion of the record.
According to the embodiment of the application, a method for extracting distributed data based on big data is provided.
Fig. 1 is a flowchart of a big data-based distributed data extraction method according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
step S101, a basic storage database is established by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data;
step S102, at least adopting an authorization key, limiting the access authority of the data in the basic storage database;
and step S103, extracting the data according to the access authority.
In the scheme, the basic storage database is established by adopting the distributed column storage database and the distributed retrieval engine, so that the rapid retrieval of data and the high data throughput are realized, the access authority of the data in the basic storage database is subsequently limited through the authorization key, the access of the data is further limited, more important clients are preferentially allowed to access the database for data extraction, the rapid extraction of the data is further realized, the extraction efficiency of the data is high, and the problem that the extraction efficiency of the distributed data based on the big data in the prior art is low is solved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
In an embodiment of the application, the authorization key includes an authorization code, and at least an authorization key is used to limit the access right of the data in the basic storage database, including: and the authorization code is used for identifying the importance degree of the user, a data type list, an ip address of the server with access right, the maximum single derivable data volume and the number of tasks which can be submitted in unit time, namely the authorization code is used for limiting the access authority of the user, so that more important clients are preferentially allowed to access the database to extract the data, and the rapid extraction of the data is further ensured.
In an embodiment of the present application, limiting access rights of the data in the basic storage database by using at least an authorization key includes: and limiting the access permission of the data in the basic storage database by adopting the authorization key and the priority algorithm, wherein the priority algorithm is used for identifying the priority of the task, namely reasonably scheduling the data use process through the priority algorithm, controlling the resource utilization rate and further realizing the rapid extraction of the data.
In another embodiment of the present application, extracting the data according to the access right includes: storing the task into a task buffer queue according to the priority of the task; generating a unique task id according to the authorization code, the task submission time and the task submission sequence, and feeding the task id back to the user; acquiring the task with the highest priority from a task export queue; dividing the task with the highest priority into a plurality of subtasks according to the time range, the data type and the node number of the task with the highest priority, wherein the task export queue is acquired according to the task cache queue; the subtasks are evenly distributed to all the export nodes according to the subtask id; and extracting the data according to the export nodes, namely caching and exporting the tasks according to the priorities of the tasks, further acquiring the task with the highest priority from the task export queue, further dividing the selected task with the highest priority into a plurality of subtasks, further distributing the subtasks to each export node according to the subtask id, and then extracting the data so as to realize the rapid extraction of the data.
In an embodiment of the present application, the index information is a primary key of the data, the primary key is a unique identifier of the data, and the primary key is composed of a generation date of the data, a code value of the data, and a hash value of the data.
In a specific embodiment of the present application, a method for generating a unique primary key is provided, which specifically includes the following steps:
step A: generating MD5 values for the data;
the MD5 values for the data are unique, the MD5 values for different data are different, and the MD5 value is a 32-bit alpha-plus-numeric combination such as: e10adc3949ba59abbe56e057f20f883 e;
and B: generating a hash value of the MD5 value of the data, and obtaining an absolute value of the hash value;
the hash value is a random integer value, and the hash value generated by the same data is unique. For example, the absolute value of the hash value of e10adc3949ba59abbe56e057f20f883e is 60;
and C: defining an array list, wherein the array is a-z plus 0-9 to form a 36-bit fixed array;
step D: taking the remainder of the generated hash value 36 to obtain a unique subscript of the data corresponding to the hash value, for example, taking the remainder of 60 pairs of 36, wherein the obtained subscript is 24, and the characters in the corresponding array are x;
step E: determining the generation time of each piece of data, wherein the data generation time must exist in each piece of data, and if not, the default is the current time of the system, for example, the time corresponding to the data is 2019-11-1218: 55: 55;
step F: determining a unique primary key corresponding to the data according to the characters + # +, the + # + of the date and the MD5 encoding value of the data in the rule array, for example, the unique primary key (key) corresponding to the data is as follows: x #20191112# e10adc3949ba59abbe56e057f20f883 e.
The key generation algorithm can reversely calculate the key value corresponding to the data according to the data, meanwhile, the data can be found according to the key value, the key is randomly generated, the data can be rapidly retrieved according to the time while the data are uniformly distributed, and the data throughput within the range time is improved.
In an embodiment of the present application, the extracting method further includes: shielding all external ports through a firewall mechanism, opening a unique port for providing data extraction service to realize quick access of data, submitting the authorization code and the task id by the user for inquiring the task state, and returning the execution state of the task; and the data extraction program feeds the execution state back to the user so as to realize the monitoring of the task execution state and ensure the accuracy of data extraction.
The embodiment of the present application further provides an extraction device for distributed data based on big data, and it should be noted that the extraction device for distributed data based on big data according to the embodiment of the present application may be used to execute the extraction method for distributed data based on big data according to the embodiment of the present application. The following describes an extraction apparatus for big data based distributed data according to an embodiment of the present application.
Fig. 2 is a schematic diagram of an apparatus for extracting big data-based distributed data according to an embodiment of the present application. As shown in fig. 2, the apparatus includes:
the system comprises an establishing unit 10, a searching unit and a searching unit, wherein the establishing unit is used for establishing a basic storage database by adopting a distributed column storage database and a distributed searching engine, the distributed column storage database is used for storing data, and the distributed searching engine is used for storing index information of the data;
a limiting unit 20, configured to limit an access right of the data in the basic storage database by using at least an authorization key;
an extracting unit 30, configured to extract the data according to the access right.
In the scheme, the establishing unit establishes the basic storage database by adopting the distributed column storage database and the distributed retrieval engine, so that the rapid retrieval of data and the larger data throughput are realized, the subsequent limiting unit limits the access authority of the data in the basic storage database through the authorization key so as to limit the access of the data, more important clients are preferentially allowed to access the database to extract the data, the extracting unit realizes the rapid extraction of the data, the extraction efficiency of the data is higher, and the problem that the extraction efficiency of the distributed data based on the big data in the prior art is lower is solved.
In an embodiment of the application, the authorization key includes an authorization code, the limiting unit is further configured to limit the access permission of the data in the basic storage database by using the authorization code, where the authorization code is used to identify an importance degree of a user, a data type list, an ip address of a server authorized to access, a maximum single derivable data amount, and a number of tasks that can be submitted in unit time, that is, the access permission of the user is limited by the authorization code, so that a more important client is preferentially allowed to access the database to extract the data, and rapid extraction of the data is further ensured.
In an embodiment of the application, at least an authorization key is used, the restriction unit is further configured to restrict the access right of the data in the basic storage database by using the authorization key and a priority algorithm, and the priority algorithm is used to identify a priority of a task, that is, a data use process is reasonably scheduled by the priority algorithm, so that a resource utilization rate is controlled, and thus, fast extraction of the data is achieved.
In another embodiment of the present application, the extraction unit includes: the system comprises a storage module, a generation module, an acquisition module, a division module, a distribution module and an extraction module, wherein the storage module is used for storing the tasks into a task cache queue according to the priorities of the tasks; the generating module is used for generating a unique task id according to the authorization code, the task submitting time and the task submitting sequence and feeding the task id back to the user; the acquisition module is used for acquiring the task with the highest priority from a task export queue; the dividing module is used for dividing the task with the highest priority into a plurality of subtasks according to the time range, the data type and the node number of the task with the highest priority, and the task export queue is acquired according to the task cache queue; the distribution module is used for distributing the subtasks to each export node according to the subtask id on average; the extraction module is used for extracting the data according to the export nodes, namely caching and exporting the tasks according to the priorities of the tasks, further acquiring the tasks with the highest priorities from the task export queues, further dividing the selected tasks with the highest priorities into a plurality of subtasks, further distributing the subtasks to the export nodes according to the subtask ids, and then extracting the data so as to realize the rapid extraction of the data.
In an embodiment of the present application, the index information is a primary key of the data, the primary key is a unique identifier of the data, and the primary key is composed of a generation date of the data, a code value of the data, and a hash value of the data.
In a specific embodiment of the present application, a method for generating a unique primary key is provided, which specifically includes the following steps:
step A: generating MD5 values for the data;
the MD5 values for the data are unique, the MD5 values for different data are different, and the MD5 value is a 32-bit alpha-plus-numeric combination such as: e10adc3949ba59abbe56e057f20f883 e;
and B: generating a hash value of the MD5 value of the data, and obtaining an absolute value of the hash value;
the hash value is a random integer value, and the hash value generated by the same data is unique. For example, the absolute value of the hash value of e10adc3949ba59abbe56e057f20f883e is 60;
and C: defining an array list, wherein the array is a-z plus 0-9 to form a 36-bit fixed array;
step D: taking the remainder of the generated hash value 36 to obtain a unique subscript of the data corresponding to the hash value, for example, taking the remainder of 60 pairs of 36, wherein the obtained subscript is 24, and the characters in the corresponding array are x;
step E: determining the generation time of each piece of data, wherein the data generation time must exist in each piece of data, and if not, the default is the current time of the system, for example, the time corresponding to the data is 2019-11-1218: 55: 55;
step F: determining a unique primary key corresponding to the data according to the characters + # +, the + # + of the date and the MD5 encoding value of the data in the rule array, for example, the unique primary key (key) corresponding to the data is as follows: x #20191112# e10adc3949ba59abbe56e057f20f883 e.
The key generation algorithm can reversely calculate the key value corresponding to the data according to the data, meanwhile, the data can be found according to the key value, the key is randomly generated, the data can be rapidly retrieved according to the time while the data are uniformly distributed, and the data throughput within the range time is improved.
In an embodiment of the application, the extraction device further includes a shielding unit, an inquiry unit, and a feedback unit, where the shielding unit is configured to shield all external ports by a firewall mechanism, and open a unique port for providing a data extraction service to achieve fast access of data, and the inquiry unit is configured to submit the authorization code and the task id by the user to perform inquiry on a task state, and return an execution state of a task; the feedback unit is used for feeding the execution state back to the user by the data extraction program so as to realize the monitoring of the task execution state and ensure the accuracy of data extraction.
The extraction device of the distributed data based on the big data comprises a processor and a memory, wherein the establishing unit, the limiting unit, the extraction unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more, and the efficiency of data extraction is improved by adjusting kernel parameters. The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a storage medium, on which a program is stored, and when the program is executed by a processor, the method for extracting distributed data based on big data is implemented.
The embodiment of the invention provides a processor, wherein the processor is used for running a program, and the method for extracting the distributed data based on the big data is executed when the program runs.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein when the processor executes the program, at least the following steps are realized:
step S101, a basic storage database is established by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data;
step S102, at least adopting an authorization key, limiting the access authority of the data in the basic storage database;
and step S103, extracting the data according to the access authority.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program of initializing at least the following method steps when executed on a data processing device:
step S101, a basic storage database is established by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data;
step S102, at least adopting an authorization key, limiting the access authority of the data in the basic storage database;
and step S103, extracting the data according to the access authority.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
From the above description, it can be seen that the above-described embodiments of the present application achieve the following technical effects:
1) according to the method for extracting the distributed data based on the big data, the basic storage database is established by adopting the distributed column storage database and the distributed retrieval engine, so that the rapid retrieval of the data and the large data throughput are realized, the access authority of the data in the basic storage database is limited through the authorization key, the access of the data is limited, more important clients are preferentially allowed to access the database to extract the data, and the rapid extraction of the data is realized.
2) According to the device for extracting the distributed data based on the big data, the establishing unit establishes the basic storage database by adopting the distributed column storage database and the distributed retrieval engine, so that the rapid retrieval of the data and the large data throughput are realized, the limiting unit limits the access authority of the data in the basic storage database through the authorization key, the access of the data is further limited, more important clients are preferentially allowed to access the database to extract the data, and the extracting unit realizes the rapid extraction of the data.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for extracting distributed data based on big data is characterized by comprising the following steps:
establishing a basic storage database by adopting a distributed column storage database and a distributed retrieval engine, wherein the distributed column storage database is used for storing data, and the distributed retrieval engine is used for storing index information of the data;
restricting access rights to the data in the base storage database using at least an authorization key;
and extracting the data according to the access right.
2. The extraction method according to claim 1, wherein the authorization key includes an authorization code, and limiting access rights of the data in the base storage database using at least an authorization key comprises:
and limiting the access authority of the data in the basic storage database by adopting the authorization code, wherein the authorization code is used for identifying the importance degree of a user, a data type list, an ip address of a server which is authorized to access, the maximum single derivable data volume and the number of tasks which can be submitted in unit time.
3. The extraction method according to claim 2, wherein limiting access rights to the data in the base storage database using at least an authorization key comprises:
and limiting the access right of the data in the basic storage database by adopting the authorization key and a priority algorithm, wherein the priority algorithm is used for identifying the priority of the task.
4. The extraction method according to claim 3, wherein extracting the data according to the access right comprises:
storing the task into a task cache queue according to the priority of the task;
generating a unique task id according to the authorization code, the task submission time and the task submission sequence, and feeding the task id back to the user;
acquiring the task with the highest priority from a task export queue;
dividing the task with the highest priority into a plurality of subtasks according to the time range, the data type and the node number of the task with the highest priority, wherein the task export queue is obtained according to the task cache queue;
the subtasks are evenly distributed to all the export nodes according to the subtask id;
extracting the data from the export node.
5. The extraction method according to claim 1, wherein the index information is a primary key of the data, the primary key being a unique identifier of the data, the primary key being composed of a generation date of the data, an encoded value of the data, and a hash value of the data.
6. The extraction method according to claim 4, further comprising:
shielding all external ports through a firewall mechanism, and opening a unique port for providing data extraction service;
the user submits the authorization code and the task id to inquire the task state and returns the execution state of the task;
and the data extraction program feeds the execution state back to the user.
7. An extraction apparatus for big data-based distributed data, comprising:
the system comprises an establishing unit, a searching unit and a searching unit, wherein the establishing unit is used for establishing a basic storage database by adopting a distributed column storage database and a distributed searching engine, the distributed column storage database is used for storing data, and the distributed searching engine is used for storing index information of the data;
a restriction unit for restricting access rights of the data in the base storage database using at least an authorization key;
and the extraction unit is used for extracting the data according to the access right.
8. The extraction apparatus according to claim 7, wherein the authorization key includes an authorization code, and the restriction unit includes:
and the limiting module is used for limiting the access authority of the data in the basic storage database by adopting the authorization code, and the authorization code is used for identifying the importance degree of a user, a data type list, an ip address of a server which is authorized to access, the maximum single derivable data volume and the number of tasks which can be submitted in unit time.
9. A storage medium characterized by comprising a stored program, wherein the program executes the extraction method of any one of claims 1 to 6.
10. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the extraction method according to any one of claims 1 to 6 when running.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911405058.3A CN111177782A (en) | 2019-12-30 | 2019-12-30 | Method and device for extracting distributed data based on big data and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911405058.3A CN111177782A (en) | 2019-12-30 | 2019-12-30 | Method and device for extracting distributed data based on big data and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111177782A true CN111177782A (en) | 2020-05-19 |
Family
ID=70646561
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911405058.3A Pending CN111177782A (en) | 2019-12-30 | 2019-12-30 | Method and device for extracting distributed data based on big data and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177782A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609168A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113672479A (en) * | 2021-04-27 | 2021-11-19 | 全球能源互联网研究院有限公司 | Data sharing method and device and computer equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102546946A (en) * | 2012-01-05 | 2012-07-04 | 中国联合网络通信集团有限公司 | Method and device for processing task on mobile terminal |
| CN102707993A (en) * | 2012-04-11 | 2012-10-03 | 招商银行股份有限公司 | System and method for centralized processing of service |
| CN103384206A (en) * | 2012-05-02 | 2013-11-06 | 中国科学院计算机网络信息中心 | Concurrent processing method and system for mass data |
| CN108334549A (en) * | 2017-12-28 | 2018-07-27 | 海尔优家智能科技（北京）有限公司 | A kind of device data storage method, extracting method, storage platform and extraction platform |
| WO2018170276A2 (en) * | 2017-03-15 | 2018-09-20 | Fauna, Inc. | Methods and systems for a database |
-
2019
- 2019-12-30 CN CN201911405058.3A patent/CN111177782A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102546946A (en) * | 2012-01-05 | 2012-07-04 | 中国联合网络通信集团有限公司 | Method and device for processing task on mobile terminal |
| CN102707993A (en) * | 2012-04-11 | 2012-10-03 | 招商银行股份有限公司 | System and method for centralized processing of service |
| CN103384206A (en) * | 2012-05-02 | 2013-11-06 | 中国科学院计算机网络信息中心 | Concurrent processing method and system for mass data |
| WO2018170276A2 (en) * | 2017-03-15 | 2018-09-20 | Fauna, Inc. | Methods and systems for a database |
| CN108334549A (en) * | 2017-12-28 | 2018-07-27 | 海尔优家智能科技（北京）有限公司 | A kind of device data storage method, extracting method, storage platform and extraction platform |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672479A (en) * | 2021-04-27 | 2021-11-19 | 全球能源互联网研究院有限公司 | Data sharing method and device and computer equipment |
| CN113609168A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113609168B (en) * | 2021-07-14 | 2024-01-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10693951B2 (en) | Decentralized, resource aware load distribution in a distributed system | |
| US8782762B2 (en) | Building data security in a networked computing environment | |
| US9229997B1 (en) | Embeddable cloud analytics | |
| US8615528B2 (en) | Cloud database sharing | |
| US8468120B2 (en) | Systems and methods for tracking and reporting provenance of data used in a massively distributed analytics cloud | |
| Shakarami et al. | Data replication schemes in cloud computing: a survey | |
| CN106547784B (en) | Data splitting and storing method and device | |
| US10754954B2 (en) | Securely exchanging information during application startup | |
| CN113079200A (en) | Data processing method, device and system | |
| US9591018B1 (en) | Aggregation of network traffic source behavior data across network-based endpoints | |
| Huang et al. | Elastic resource allocation against imbalanced transaction assignments in sharding-based permissioned blockchains | |
| US10659531B2 (en) | Initiator aware data migration | |
| US9641522B1 (en) | Token management in a managed directory service | |
| US11297066B2 (en) | Constrained roles for access management | |
| KR101621385B1 (en) | System and method for searching file in cloud storage service, and method for controlling file therein | |
| CN111177782A (en) | Method and device for extracting distributed data based on big data and storage medium | |
| US10162876B1 (en) | Embeddable cloud analytics | |
| CN109145621B (en) | Document management method and device | |
| US11836269B2 (en) | Protection of data of database clients from persistent adversaries | |
| Castelino et al. | Integration of big data and cloud computing | |
| WO2023098433A1 (en) | Secure policy distribution in a cloud environment | |
| CN109542401B (en) | Web development method and device, storage medium and processor | |
| US20170279911A1 (en) | Resource usage anonymization | |
| CN111159192B (en) | Big data based data warehousing method and device, storage medium and processor | |
| Gupta et al. | Role and access based data segregator for security of big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200731 Address after: 1608, 14 / F, No. 65, Beisihuan West Road, Haidian District, Beijing 100080 Applicant after: BEIJING INTERNETWARE Ltd. Address before: No. 603, floor 6, No. 9, Shangdi 9th Street, Haidian District, Beijing 100085 Applicant before: Smart Shenzhou (Beijing) Technology Co.,Ltd. |
|
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200519 |