CN111782636A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN111782636A CN111782636A CN202010617752.8A CN202010617752A CN111782636A CN 111782636 A CN111782636 A CN 111782636A CN 202010617752 A CN202010617752 A CN 202010617752A CN 111782636 A CN111782636 A CN 111782636A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- storage unit
- preset
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method and device, which call a preset database abstract interface to access a preset K-V type database. And calling a preset database index interface to obtain an operation command sent by the external application. And calling a preset file system interface, triggering a preset Linux file system to arrange the storage unit in the K-V type database and executing an operation command. The storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit. Based on the data processing method provided by the application, the K-V type database can completely replace a time sequence database, and the requirements of data acquisition equipment of embedded Linux on a local database are met.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
In practical application, a data acquisition device of the embedded Linux has a requirement for storing acquired data to a local place, for example, under the condition that a long-distance public network transmission or a wireless transmission is common in the oil and gas gathering and transportation industry, but a network is unstable or is interrupted accidentally, the data acquisition device cannot transmit the acquired data to a data center temporarily, and the acquired data needs to be stored locally. Accordingly, local databases that provide a storage environment for data collection devices have evolved. The local database storing the collected data is typically a time-series database, since the collected data is basically a digital or analog quantity that is continuous in time, usually the time generated by time stamping the data.
Currently, timing databases such as infiuxdb, OpenTSDB, and dalmallnerdb are mostly used in existing local databases. However, in practical applications, for an embedded Linux application scenario (i.e. an embedded Linux data acquisition device performs data addition, deletion, and modification on a local database), the existing time sequence database has the following problems:
1. the CS architecture is mainly used, and in practical application, a database based on the CS architecture needs to communicate with data acquisition equipment through a local loopback network. However, the efficiency of reading and writing data to the database is very low due to network communication resources.
2. The embedded Linux system provides too many complex functions (e.g., a data redundancy function and a data rollback function), however, the functions required by the embedded Linux application scenario are simpler (mainly adding, deleting, modifying or checking data), and therefore, the overall design of the database is too bloated, which results in more resource waste.
3. High-level languages such as JAVA, GO, C #, C + + and the like are adopted for writing, so that the database can depend on a large number of operation libraries in order to be adaptive to the data server. However, the CPU, RAM, and FLASH within the data acquisition device have limited computational resources and do not have the conditions for developing or adapting these runtime libraries.
In summary, the existing time sequence database cannot meet the requirement of the data acquisition equipment of the embedded Linux on the local database.
Disclosure of Invention
The application provides a data processing method and a data processing device, and aims to provide an effective local database for data acquisition equipment of embedded Linux.
In order to achieve the above object, the present application provides the following technical solutions:
a data processing apparatus comprising:
the interface layer module is used for providing a database index interface for the core layer module and external application;
a data layer module for providing a database abstraction interface to the core layer module;
the system layer module is used for providing a file system interface for the core layer module;
the core layer module is used for calling the database abstract interface and accessing a preset K-V type database; calling the database index interface to acquire an operation command sent by the external application; calling the file system interface, triggering a preset Linux file system to arrange the storage unit in the K-V type database and executing the operation command;
the storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit.
Optionally, the database index interface includes:
a first operation interface, configured to receive a first operation command sent by the external application, where the first operation command indicates to create the storage unit in the K-V type database;
the second operation interface is used for receiving a second operation command sent by the external application, and the second operation command indicates that data entries are written in the K-V type database;
a third operation interface, configured to receive a third operation command sent by the external application, where the third operation command indicates to read the data entry stored in the K-V type database within a preset time period;
a fourth operation interface, configured to receive a fourth operation command sent by the external application, where the fourth operation command indicates to read a target data entry in the K-V type database, where the target data entry is a preset number of data entries recorded in the K-V type database starting at a preset time;
a fifth operation interface, configured to receive a fifth operation command sent by the external application, where the fifth operation command indicates to delete the storage unit in the K-V type database.
Optionally, the core layer module calls the file system interface to trigger a preset Linux file system to program the storage unit in the K-V type database, including:
the core layer module is specifically used for calling the file system interface, triggering a preset Linux file system, and creating an attribute file of a storage unit by taking the creation date of the storage unit in the K-V type database as a file name; and triggering the Linux file system, creating a folder by using a preset folder name, and storing the attribute file in the folder.
Optionally, the core layer module is further configured to:
determining a target storage unit in the K-V type database according to a preset index requirement, wherein the index requirement comprises the name of a target file and the storage time of a data item to be queried, the target storage unit is the storage unit to which the data item to be queried belongs, and the target file is the attribute file of the target storage unit; acquiring a time stamp of each data unit in the target storage unit; sequencing the timestamps by adopting a preset sequencing algorithm to obtain a timestamp sequence; searching a target timestamp from the timestamp sequence by adopting a preset searching algorithm, wherein the target timestamp indicates the storage time of the data item to be inquired; and acquiring the data entry corresponding to the target timestamp.
Optionally, the sorting algorithm includes:
a comparison ranking algorithm, or a technical ranking algorithm.
Optionally, the search algorithm includes:
and (4) an ordered search algorithm.
Optionally, the different types of K-V databases correspond to different abstract interfaces of the database.
A method of data processing, comprising:
calling a preset database abstract interface, and accessing a preset K-V type database;
calling a preset database index interface to obtain an operation command sent by an external application;
calling a preset file system interface, triggering a preset Linux file system to arrange the storage units in the K-V type database and executing the operation command;
the storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit.
A storage medium comprising a stored program, wherein the program executes the above-described data processing method.
A data processing apparatus comprising: a processor, a memory, and a bus; the processor and the memory are connected through the bus;
the memory is used for storing programs, and the processor is used for running the programs, wherein the programs execute the data processing method when running.
The technical scheme provided by the application provides a data processing device which comprises an interface layer module, a data layer module, a system layer module and a core layer module. The interface layer module provides a database index interface for the data processing device and an external application, the data layer module provides a database abstract interface for the data processing device and a K-V type database, the system layer module provides a file system interface for the data processing device and a Linux file system, and the core layer module calls the three interfaces, receives an operation command sent by the external application and triggers the Linux file system to execute the operation command, so that data processing in the K-V type database is realized. The data storage structure is characterized in that the storage format of data in the K-V type database is a key value pair, values of the key value pair are used for storing data entries, keys are used for recording timestamps of the data entries, the timestamps indicate the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit, and based on the data storage structure, the K-V type database can completely replace the existing time sequence database.
In addition, the storage units in the K-V type database are arranged through a Linux file system, and data stored in any date can be quickly searched from the K-V type database based on the folder name and the file name. And moreover, a database index mechanism is designed for the K-V type database, so that the recorded data in the K-V type database table can be quickly searched or sequenced, and the data processing speed of the K-V type database is improved.
Compared with the existing time sequence database, based on the characteristics of the K-V type database, the efficiency of data reading and writing by external application is higher, and excessive communication resources cannot be occupied. And because the data are stored in the K-V type database, the waste of excessive computing resources can not be caused based on the characteristic of simple functional design of the K-V type database.
Obviously, the database abstract interface is called to establish a connection with the K-V type database, the interface is called to trigger the file system interface, the Linux file system is triggered to perform data processing on the K-V type database, additional development or adaptation to other operation libraries is not needed, the used operation resources are simple, the applicability is wide, and therefore the requirements of the embedded Linux data acquisition equipment on the local database can be met.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1a is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 1b is a schematic flowchart illustrating a data processing performed by a data processing apparatus according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of a data processing method according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
As shown in fig. 1a, an architecture diagram of a data processing apparatus provided in the embodiment of the present application includes:
In this embodiment, the interface layer module 100 is packaged with a preset database index interface, the data layer module 300 is packaged with a preset database abstraction interface, and the system layer module 400 is packaged with a preset file system interface.
As shown in fig. 1b, the specific implementation process of the data processing performed by the data processing apparatus includes the following steps:
s101: and the core layer module calls a database abstract interface in the data layer module to access a preset K-V type database.
The preset K-V type database includes, but is not limited to, bsdbd, leveldb, and uniqlite types. In addition, different types of K-V type databases correspond to different database abstraction interfaces, namely, the data layer module comprises a plurality of pre-packaged database abstraction interfaces, and each database abstraction interface corresponds to one type of K-V type databases.
In the K-V type database, data is stored in a key-value pair (key-value) format, the key-value pair is used for storing the data and recording the storage time of the data, specifically, the key is used for indicating a timestamp (i.e., indicating the storage time of the data, which is usually recorded by using a system clock), and the value is used for indicating the content of the data. In addition, in this embodiment, the data structure of the K-V database is specifically: each data item is packaged in a preset storage unit in the form of a data unit, and the storage date of each data item in the storage unit is consistent with the creation date of the storage unit, that is, the data items recorded in the same date are stored in the same storage unit.
Note that the data format of the key-value pair includes a JSON format (an existing data format) and a C format (an existing data format), and specifically, see table 1 below.
TABLE 1
| Data format | key | value |
| JSON format | 1582185869(unixtimestamp) | {“name”:“foo”,“age”:32} |
| C format | 1582185869(unixtimestamp) | 0x120x340x560x78 |
It should be noted that the unix timestamp shown in table 1 above specifically refers to a timestamp type (i.e., unix timestamp), and "0 x 120 x 340 x 560 x 78" specifically refers to an ASCLL code (a conventional encoding standard) of data.
S102: and the core layer module calls a file system interface in the system layer module to trigger a preset Linux file system to arrange the storage units in the K-V type database.
The specific implementation process of arranging the storage unit by the Linux file system comprises the following steps: the creation date of the storage unit is used as the file name, and the property file of the storage unit is created, for example, if the creation date of the storage unit is 5, 13 and 2020, the file name of the property file of the storage unit may be "20200513". In order to manage each attribute file conveniently, a folder is created by using a preset folder name, and the attribute files are stored in the folder. Specifically, the Linux file system may refer to fig. 2 below as a result of the arrangement of the storage unit.
It should be noted that, by means of the folder name and the file name, data stored in any date can be quickly searched from the K-V type database. In addition, since the data entry is packaged in the storage unit in the form of a data unit, and the data format of the data unit is a key-value pair with a time stamp as key and data content as value, the data stored in any time period can be quickly searched from the storage unit by the time stamp.
It is emphasized that the Linux file system provides a series of functional logics for data reading, data writing, file arrangement, file creation, and file deletion. In addition, the functional logic provided by the Linux file system is common knowledge familiar to those skilled in the art, and is not described herein again.
S103: and the core layer module calls a database index interface in the interface layer module to acquire an operation command sent by the external application.
The operation command specifically refers to: and at least one of adding, deleting, modifying and searching the data in the K-V type database. In this embodiment, the external application may specifically be: the application runs in the data acquisition equipment of the embedded Linux.
It should be noted that the type of the database index interface may be different for different types of operation commands. In this embodiment, the database index interface includes a first operation interface, a second operation interface, a third operation interface, a fourth operation interface, and a fifth operation interface.
In particular, the method comprises the following steps of,
the first operation interface is used for receiving a first operation command, and the first operation command instructs to create a storage unit in the K-V type database.
The second operation interface is used for receiving a second operation command, and the second operation command indicates that data entries are written in the K-V type database.
The third operation interface is used for receiving a third operation command, and the third operation command indicates to read data entries stored in the K-V type database within a preset time period.
The fourth operation interface is configured to receive a fourth operation command, where the fourth operation command indicates to read a target data entry in the K-V database, where the target data entry specifically refers to: starting with a preset time, a preset number of data entries are recorded (in this embodiment, one data unit is recorded as one data entry).
And the fifth operation interface is used for receiving a fifth operation command, and the fifth operation command indicates that the storage unit in the K-V type database is deleted.
It should be noted that specific names, specific parameter settings, and specific descriptions of the first operation interface, the second operation interface, the third operation interface, the fourth operation interface, and the fifth operation interface mentioned above are shown in table 2 below.
TABLE 2
S104: and the core layer module calls a file system interface in the system layer module to trigger the Linux file system to execute the operation command.
The specific implementation process of executing the operation command by the Linux file system, that is, at least one of adding, deleting, modifying and searching data in the K-V database, is common general knowledge familiar to those skilled in the art, and is not described herein again.
It should be emphasized that, in order to improve the data query efficiency of the K-V type database, the core layer module also constructs a database indexing mechanism for the K-V type database, so-called indexing is: the method has a structure for sequencing one or more columns of values in the database table, thereby realizing the quick search or sequencing of the record data in the database table.
Specifically, the database indexing mechanism comprises the following steps:
s105: and the core layer module determines a target storage unit in the K-V type database according to a preset index requirement.
The preset index requirement includes the name of the target file (i.e. the storage date of the data entry to be queried), the storage time of the data entry to be queried, and the name of the target folder. The target storage unit is a storage unit to which the data item to be inquired belongs, the target file is an attribute file of the target storage unit, and the target file is stored in the target folder.
S106: the core layer module obtains the time stamp of each data unit in the target storage unit.
The time stamp of each data unit in the target storage unit is obtained, which is substantially the key of each data unit (the data format of the data unit is a key value pair) in the target storage unit.
S107: and the core layer module sorts the timestamps by adopting a preset sorting algorithm to obtain a timestamp sequence.
The preset sorting algorithm may be a comparison sorting algorithm or a technical sorting algorithm. Comparison sorting algorithms include, but are not limited to: algorithms for insert ordering, hill ordering, select ordering, heap ordering, bubble ordering, quick ordering, and merge ordering. Computational ranking algorithms include, but are not limited to: counting sorting, bucket sorting, cardinal sorting, and the like.
It should be noted that, sorting each timestamp is actually sorting the key of each data unit, and the timestamp sequence is substantially the key sequence.
S108: and the core layer module searches the target timestamp from the timestamp sequence by adopting a preset search algorithm.
The preset search algorithm may specifically be an ordered search algorithm, and the ordered search algorithm includes, but is not limited to: sequential lookup, binary lookup, interpolation lookup, and fibonacci lookup. The target timestamp indicates the storage time of the data entry to be queried.
It should be noted that, the target timestamp is found from the timestamp sequence, and in essence, the target key is found from the key sequence, for example, the key sequence includes 202005271230, 202005271231, 202005271232, and 202005271233, and the target key is 202005271231.
S109: and the core layer module acquires a data entry corresponding to the target timestamp.
The data entry corresponding to the target timestamp is obtained, namely, a target value corresponding to the target key is obtained, and the target value indicates specific data content of the data entry to be queried.
In summary, the present embodiment provides a data processing apparatus, which includes an interface layer module, a core layer module, a data layer module, and a system layer module. The interface layer module provides a database index interface for the data processing device and an external application, the data layer module provides a database abstract interface for the data processing device and a K-V type database, the system layer module provides a file system interface for the data processing device and a Linux file system, and the core layer module calls the three interfaces, receives an operation command sent by the external application and triggers the Linux file system to execute the operation command, so that data processing in the K-V type database is realized. The data storage structure is characterized in that the storage format of data in the K-V type database is a key value pair, values of the key value pair are used for storing data entries, keys are used for recording timestamps of the data entries, the timestamps indicate the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit, and based on the data storage structure, the K-V type database can completely replace the existing time sequence database.
In addition, the storage units in the K-V type database are arranged through a Linux file system, and data stored in any date can be quickly searched from the K-V type database based on the folder name and the file name. And moreover, a database index mechanism is designed for the K-V type database, so that the recorded data in the K-V type database table can be quickly searched or sequenced, and the data processing speed of the K-V type database is improved.
Compared with the existing time sequence database, based on the characteristics of the K-V type database, the efficiency of data reading and writing by external application is higher, and excessive communication resources cannot be occupied. And because the data are stored in the K-V type database, the waste of excessive computing resources can not be caused based on the characteristic of simple functional design of the K-V type database. In addition, the abstract interface of the calling database is linked with the K-V type database, the interface of the calling database is used for triggering the interface of the file system, the Linux file system is triggered to perform data processing on the K-V type database, additional development or adaptation to other operation libraries is not needed, the used operation resources are simple, the applicability is wide, and therefore the requirements of the data acquisition equipment of the embedded Linux on the local database can be met.
Corresponding to the data processing device provided by the embodiment of the application, the application also provides a data processing method.
As shown in fig. 2, a schematic diagram of a data processing method provided in an embodiment of the present application includes the following steps:
s201: and calling a preset data base abstract interface to access a preset K-V type data base.
The storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit.
It should be noted that the specific implementation process and implementation principle of S201 are consistent with the specific implementation process and implementation principle of S101, and are not described herein again.
S202: and calling a preset database index interface to obtain an operation command sent by the external application.
The specific implementation process and implementation principle of S202 are consistent with the specific implementation process and implementation principle of S103, and are not described herein again.
Optionally, the preset database index interface includes: the first operation interface is used for receiving a first operation command sent by an external application, and the first operation command indicates that a storage unit is created in the K-V type database; the second operation interface is used for receiving a second operation command sent by the external application, and the second operation command indicates that data entries are written in the K-V type database; the third operation interface is used for receiving a third operation command sent by the external application, and the third operation command indicates to read data items stored in the K-V type database within a preset time period; the fourth operation interface is used for receiving a fourth operation command sent by the external application, wherein the fourth operation command indicates to read target data entries in the K-V type database, and the target data entries are a preset number of data entries recorded in the K-V type database by taking a preset moment as a start; and the fifth operation interface is used for receiving a fifth operation command sent by the external application, and the fifth operation command indicates that the storage unit in the K-V type database is deleted.
S203: and calling a preset file system interface, triggering a preset Linux file system to arrange the storage unit in the K-V type database and executing an operation command.
Optionally, the specific implementation process of calling a preset file system interface and triggering a preset Linux file system to arrange the storage unit in the K-V type database includes: and calling a file system interface, triggering a preset Linux file system, and establishing an attribute file of a storage unit by taking the creation date of the storage unit in the K-V type database as a file name. And triggering the Linux file system, creating a folder by using a preset folder name, and storing the attribute file in the folder.
Optionally, in this embodiment, a target storage unit in the K-V database may also be determined according to a preset index requirement, where the index requirement includes a name of a target file and a storage time of a data entry to be queried, the target storage unit is a storage unit to which the data entry to be queried belongs, and the target file is an attribute file of the target storage unit. Acquiring a time stamp of each data unit in a target storage unit; and sequencing the timestamps by adopting a preset sequencing algorithm to obtain a timestamp sequence. And searching a target timestamp from the timestamp sequence by adopting a preset searching algorithm, wherein the target timestamp indicates the storage time of the data entry to be inquired, and acquiring the data entry corresponding to the target timestamp.
In conclusion, the preset database abstraction interface is called to access the preset K-V type database. And calling a preset database index interface to obtain an operation command sent by the external application. And calling a preset file system interface, triggering a preset Linux file system to arrange the storage unit in the K-V type database and executing an operation command. The storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit. Based on the data processing method provided by the embodiment, the K-V type database can completely replace a time sequence database, and the requirements of the data acquisition equipment of the embedded Linux on the local database are met.
The application also provides a storage medium, which comprises a stored program, wherein the program executes the data processing method provided by the application.
The present application also provides a data processing apparatus, including: a processor, a memory, and a bus. The processor is connected with the memory through a bus, the memory is used for storing programs, and the processor is used for running the programs, wherein the programs execute the data processing method provided by the application when running.
The functions described in the method of the embodiment of the present application, if implemented in the form of software functional units and sold or used as independent products, may be stored in a storage medium readable by a computing device. Based on such understanding, part of the contribution to the prior art of the embodiments of the present application or part of the technical solution may be embodied in the form of a software product stored in a storage medium and including several instructions for causing a computing device (which may be a personal computer, a server, a mobile computing device or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data processing apparatus, comprising:
the interface layer module is used for providing a database index interface for the core layer module and external application;
a data layer module for providing a database abstraction interface to the core layer module;
the system layer module is used for providing a file system interface for the core layer module;
the core layer module is used for calling the database abstract interface and accessing a preset K-V type database; calling the database index interface to acquire an operation command sent by the external application; calling the file system interface, triggering a preset Linux file system to arrange the storage unit in the K-V type database and executing the operation command;
the storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit.
2. The apparatus of claim 1, wherein the database index interface comprises:
a first operation interface, configured to receive a first operation command sent by the external application, where the first operation command indicates to create the storage unit in the K-V type database;
the second operation interface is used for receiving a second operation command sent by the external application, and the second operation command indicates that data entries are written in the K-V type database;
a third operation interface, configured to receive a third operation command sent by the external application, where the third operation command indicates to read the data entry stored in the K-V type database within a preset time period;
a fourth operation interface, configured to receive a fourth operation command sent by the external application, where the fourth operation command indicates to read a target data entry in the K-V type database, where the target data entry is a preset number of data entries recorded in the K-V type database starting at a preset time;
a fifth operation interface, configured to receive a fifth operation command sent by the external application, where the fifth operation command indicates to delete the storage unit in the K-V type database.
3. The apparatus according to claim 1, wherein the kernel layer module calls the file system interface to trigger a predetermined Linux file system to program the storage unit in the K-V database, and comprises:
the core layer module is specifically used for calling the file system interface, triggering a preset Linux file system, and creating an attribute file of a storage unit by taking the creation date of the storage unit in the K-V type database as a file name; and triggering the Linux file system, creating a folder by using a preset folder name, and storing the attribute file in the folder.
4. The apparatus of claim 3, wherein the core layer module is further configured to:
determining a target storage unit in the K-V type database according to a preset index requirement, wherein the index requirement comprises the name of a target file and the storage time of a data item to be queried, the target storage unit is the storage unit to which the data item to be queried belongs, and the target file is the attribute file of the target storage unit; acquiring a time stamp of each data unit in the target storage unit; sequencing the timestamps by adopting a preset sequencing algorithm to obtain a timestamp sequence; searching a target timestamp from the timestamp sequence by adopting a preset searching algorithm, wherein the target timestamp indicates the storage time of the data item to be inquired; and acquiring the data entry corresponding to the target timestamp.
5. The apparatus of claim 4, wherein the ranking algorithm comprises:
a comparison ranking algorithm, or a technical ranking algorithm.
6. The apparatus of claim 4, wherein the lookup algorithm comprises:
and (4) an ordered search algorithm.
7. The apparatus of any of claims 1-6, wherein different types of the K-V databases correspond to different ones of the database abstraction interfaces.
8. A data processing method, comprising:
calling a preset database abstract interface, and accessing a preset K-V type database;
calling a preset database index interface to obtain an operation command sent by an external application;
calling a preset file system interface, triggering a preset Linux file system to arrange the storage units in the K-V type database and executing the operation command;
the storage format of the data in the K-V type database is a key value pair, the value of the key value pair is used for storing data entries, the key is used for recording the time stamp of the data entries, the time stamp indicates the storage time of the data entries, each data entry is packaged in the storage unit in the form of a data unit, and the storage date of each data entry in the storage unit is consistent with the creation date of the storage unit.
9. A storage medium characterized by comprising a stored program, wherein the program executes the data processing method of claim 8.
10. A data processing apparatus, characterized by comprising: a processor, a memory, and a bus; the processor and the memory are connected through the bus;
the memory is used for storing a program and the processor is used for running the program, wherein the program executes the data processing method of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010617752.8A CN111782636A (en) | 2020-06-30 | 2020-06-30 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010617752.8A CN111782636A (en) | 2020-06-30 | 2020-06-30 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111782636A true CN111782636A (en) | 2020-10-16 |
Family
ID=72760541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010617752.8A Pending CN111782636A (en) | 2020-06-30 | 2020-06-30 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111782636A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104504105A (en) * | 2014-12-30 | 2015-04-08 | 青岛海信网络科技股份有限公司 | Storage method of real-time database |
| US20170193041A1 (en) * | 2016-01-05 | 2017-07-06 | Sqrrl Data, Inc. | Document-partitioned secondary indexes in a sorted, distributed key/value data store |
| CN110019085A (en) * | 2017-10-31 | 2019-07-16 | 上海宝信软件股份有限公司 | A kind of distributed time series database based on HBase |
| CN110109866A (en) * | 2017-12-28 | 2019-08-09 | 中移（杭州）信息技术有限公司 | A kind of management method and equipment of file system directories |
| CN110795428A (en) * | 2019-10-10 | 2020-02-14 | 中盈优创资讯科技有限公司 | Time sequence data storage method and time sequence database applied to industrial Internet of things |
| CN111221894A (en) * | 2020-01-10 | 2020-06-02 | 山西西山煤电股份有限公司 | Configuration-based time sequence database storage method and device and server |
-
2020
- 2020-06-30 CN CN202010617752.8A patent/CN111782636A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104504105A (en) * | 2014-12-30 | 2015-04-08 | 青岛海信网络科技股份有限公司 | Storage method of real-time database |
| US20170193041A1 (en) * | 2016-01-05 | 2017-07-06 | Sqrrl Data, Inc. | Document-partitioned secondary indexes in a sorted, distributed key/value data store |
| CN110019085A (en) * | 2017-10-31 | 2019-07-16 | 上海宝信软件股份有限公司 | A kind of distributed time series database based on HBase |
| CN110109866A (en) * | 2017-12-28 | 2019-08-09 | 中移（杭州）信息技术有限公司 | A kind of management method and equipment of file system directories |
| CN110795428A (en) * | 2019-10-10 | 2020-02-14 | 中盈优创资讯科技有限公司 | Time sequence data storage method and time sequence database applied to industrial Internet of things |
| CN111221894A (en) * | 2020-01-10 | 2020-06-02 | 山西西山煤电股份有限公司 | Configuration-based time sequence database storage method and device and server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR100971863B1 (en) | System and method for batched indexing of network documents | |
| JP5661104B2 (en) | Method and system for search using search engine indexing and index | |
| CN106980665B (en) | Data dictionary implementation method and device and data dictionary management system | |
| CN110888842A (en) | File storage method, file query method, file storage device, file query device and file query equipment | |
| CN110134681B (en) | Data storage and query method and device, computer equipment and storage medium | |
| CN111400334B (en) | Data processing method, data processing device, storage medium and electronic device | |
| CN112306957A (en) | Method and device for acquiring index node number, computing equipment and storage medium | |
| US20190147047A1 (en) | Object-level image query and retrieval | |
| CN116756253B (en) | Data storage and query methods, devices, equipment and media of relational database | |
| CN107133334B (en) | Data synchronization method based on high-bandwidth storage system | |
| CN112835638A (en) | Configuration information management method and device based on embedded application program | |
| CN113326288A (en) | Data processing method and device and electronic equipment | |
| US7441252B2 (en) | Cache control device, and method and computer program for the same | |
| CN112579633A (en) | Data retrieval method, device, equipment and storage medium | |
| CN111782636A (en) | Data processing method and device | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN115543918A (en) | File snapshot method, system, electronic equipment and storage medium | |
| US20230138113A1 (en) | System for retrieval of large datasets in cloud environments | |
| CN112632266B (en) | Data writing method and device, computer equipment and readable storage medium | |
| CN112699129A (en) | Data processing system, method and device | |
| CN107918654B (en) | File decompression method and device and electronic equipment | |
| CN101930451B (en) | For storing the method and apparatus of at least one inquiry data element of efficiently searching | |
| CN112015791A (en) | Data processing method and device, electronic equipment and computer storage medium | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| CN112402955B (en) | Game log recording method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |