CN111061721A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN111061721A CN111061721A CN201811200505.7A CN201811200505A CN111061721A CN 111061721 A CN111061721 A CN 111061721A CN 201811200505 A CN201811200505 A CN 201811200505A CN 111061721 A CN111061721 A CN 111061721A
- Authority
- CN
- China
- Prior art keywords
- attribute information
- database
- data
- small table
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention provides a data processing method and a data processing device, wherein the method comprises the following steps: acquiring attribute information of first target data to be stored in a database; determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises a large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in at least one small table and the attribute information of the data stored in at least one small table; and storing the first target data into a small table corresponding to the attribute information of the first target data. The data processing method and the data processing device improve the data processing efficiency in the database.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a data processing method and apparatus.
Background
A log file is a log-type data in a database used to record database update operations. The log file is used for tracking and recording all modifications to the database data, and the database can be protected through the log file of the database so as to perform data recovery or rollback after the database fails or assist a user in analyzing and solving problems in the database.
In the prior art, a database generates a new log file as long as data is updated in daily use, so that the number of log files stored in a table for storing the log files in the database is increased continuously in the use process of the database. More and more log files are not beneficial to searching and processing data such as database log files, and the efficiency of database data processing is low. Therefore, how to improve the efficiency of the database in processing data is a technical problem to be solved urgently at present.
Disclosure of Invention
The invention provides a data processing method and device, which are used for improving the data processing efficiency when a database processes data.
A first aspect of the present invention provides a data processing method, including:
acquiring attribute information of first target data to be stored in a database;
determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises the large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in the at least one small table and the attribute information of the data stored in the at least one small table;
and storing the first target data into a small table corresponding to the attribute information of the first target data.
In an embodiment of the first aspect of the present invention, after storing the first target data in the small table corresponding to the attribute information, the method further includes:
acquiring attribute information of second target data to be processed in the database;
determining a small table corresponding to the attribute information of the second target data according to the large table of the database;
and acquiring the second target data from a small table corresponding to the attribute information of the second target data.
In an embodiment of the first aspect of the present invention, after acquiring the second target data from the small table corresponding to the attribute information of the second target data, the method further includes:
and performing addition, deletion, modification and/or query operations on the second target data.
In an embodiment of the first aspect of the present invention, the method further includes:
acquiring attribute information of all data stored in the database;
determining the number of the at least one small table in the database according to the classification result of the attribute information of all the data;
and establishing the at least one small table in the database, and storing the corresponding relation between the at least one small table and the attribute information of the data stored in the at least one small table in the large table.
In an embodiment of the first aspect of the present invention, the determining, according to the large table of the database, the small table corresponding to the attribute information of the first target data includes:
determining whether a small table corresponding to the attribute information of the first target data exists in the database;
and if the first target data does not exist, establishing a small table corresponding to the attribute information of the first target data in the database.
In an embodiment of the first aspect of the present invention, the database is configured to store log data; the attribute information is the generation time of the log data.
A second aspect of the present invention provides a data processing apparatus comprising:
the acquisition module is used for acquiring attribute information of first target data to be stored in a database;
the determining module is used for determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises the large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in the at least one small table and the attribute information of the data stored in the at least one small table;
and the processing module is used for storing the first target data into a small table corresponding to the attribute information of the first target data.
In an embodiment of the second aspect of the present invention, the obtaining module is further configured to obtain attribute information of second target data to be processed in the database;
the determining module is further configured to determine a small table corresponding to the attribute information of the second target data according to the large table of the database;
the processing module is further configured to obtain the second target data from a small table corresponding to the attribute information of the second target data.
In an embodiment of the second aspect of the present invention, the processing module is further configured to perform operations of adding, deleting, modifying and/or querying the second target data.
In an embodiment of the second aspect of the present invention, the obtaining module is further configured to obtain attribute information of all data stored in the database;
the determining module is further configured to determine the number of the at least one small table in the database according to the classification result of the attribute information of all the data;
the determining module is further configured to establish the at least one small table in the database, and store a corresponding relationship between the at least one small table and attribute information of data stored in the at least one small table in the large table.
In an embodiment of the second aspect of the present invention, the determining module is specifically configured to determine whether a small table corresponding to the attribute information of the first target data exists in the database;
if the attribute information does not exist, the determining module is further used for establishing a small table corresponding to the attribute information of the first target data in the database.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores program code, and when the program code is executed, the computer-readable storage medium executes the data processing method according to any one of the first aspect of the present invention.
In a fourth aspect, an embodiment of the present invention provides a passenger information system PIS data transmission device, including: a processor and a memory; the memory is used for storing programs; the processor is configured to call a program stored in the memory to execute the data processing method according to any one of the first aspect of the present invention.
In summary, the present invention provides a data processing method and apparatus, wherein the method includes: acquiring attribute information of first target data to be stored in a database; determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises a large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in at least one small table and the attribute information of the data stored in at least one small table; and storing the first target data into a small table corresponding to the attribute information of the first target data. The data processing method and the data processing device provided by the invention store data of different attribute information through the small table in the database, and the large table comprises the corresponding relation between the small table and the attribute information of the stored data. Therefore, data are stored in different small tables, query and subsequent processing of the data stored in the small tables can be realized through the large table, and the efficiency of data processing in the database is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram illustrating a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a data processing method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention.
With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments. The technical solution of the present invention will be described in detail below with specific examples. The following embodiments may be combined with each other and may not be described in detail in some embodiments for the same or similar concepts or processes.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical solution of the present invention will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention. As shown in fig. 1, the data processing method provided in this embodiment includes:
s101: acquiring attribute information of first target data to be stored in a database.
The execution main body of this embodiment is an electronic device, such as a computer, a server, or a mobile phone, which has a function of processing data in a database, or a chip in the electronic device, so as to implement the processing of data in the database in this embodiment.
Specifically, in S101, when the electronic device acquires first target data that needs to be stored in the database, first attribute information of the first target data to be stored in the database is acquired.
Before acquiring the attribute information, the electronic device may receive first target data by receiving a user instruction, for example, an instruction for a user to send the first target data and store the first target data in a specified position of a database; or, the data automatically generated in the database is used as the first target data and is stored in the position where the data is stored by default.
For example: the first target data is a log file of the database, and the log data is generated after the database performs operations such as data addition, modification and deletion every time. The log data generated by each operation can be used as the first target data in this embodiment, and when the log data is used as the first target data, before the log data is stored in the database, the first target data, that is, the attribute information of the log data, needs to be acquired.
Optionally, the attribute information of the first target data may include one or more of the following: generation time, modification time, data size, operating user, specifying storage location, etc. For example, in the above example, when the first target data is a log file of the database, the attribute information of the log file may be a generation time of the log file.
S102: determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises a large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in at least one small table and the attribute information of the data stored in at least one small table.
S103: and storing the first target data into a small table corresponding to the attribute information of the first target data.
Specifically, in S102, according to the attribute information of the first target data determined in S101, a small table corresponding to the attribute information of the first target data is determined from the large table of the database, and in S103, the first target data is stored into the small table determined in S102. The database at least comprises the large table and at least one small table, and each small table corresponds to different attribute information of the data. The large table is used as an index of the small table, and stores the corresponding relation between all the small tables and the attribute information of the data, and the corresponding small table can be determined through the attribute information of the data through the corresponding relation in the large table.
It should be noted that the large table and the small table are abstract table concepts in the database, and substantially correspond to different storage spaces in the database, the large table and the small table are part of the storage space of the database, and the sum of the storage spaces of all the large table and the small table should be smaller than or equal to the size of the total storage space of the database.
Also taking the first target data as a log file and the attribute information as the generation time in the foregoing example, different small tables may be allocated in the database according to the generation time of different log files, for example, 12 small tables are set in the database, and each small table is used for storing a log file generated in one month. In S102, all the log data generated by the database are determined according to the generated time, and the corresponding small table of the 12 small tables is determined, and the first target data, i.e. the log file, is stored in the corresponding small table.
Optionally, all the small tables and attribute information of the data stored in the small tables may be stored in the large table, for example, the correspondence stored in the large table includes 12 small tables corresponding to 12 months of "1 month-small table 1", "2 month-small table 2" … …, where the correspondence may be a name of the small table, such as small table 1, or the correspondence may also be a specific storage location of the small table, so as to store the data generated in the corresponding month into the corresponding small table according to the storage location of the small table.
For example: when the first target data acquired in S101 is a log file, and the attribute information of the first target data is acquired such that the generation time of the log file is 1 month and 1 day, the small table corresponding to 1 month and 1 day determined from the large table of the database in S102 is "small table 1", and the log file is stored in the small table 1 of the database in S103.
Alternatively, in the above-described embodiment, the small tables generated in the database are identical, that is, have the same structure of definition and table, and differ only in the attribute information of the stored data. Alternatively, the small tables in the database may also have different definitions and table structures according to the stored attribute information.
Optionally, in the above embodiment, when the attribute information is modification time, data size, operation user, storage location designation, and the like of the first target data, the attribute information may also be divided into small tables of different 12 months according to months of different modification time, 2 small tables of small data of large data according to data size, or small tables of the same number as the number of users of the database according to operation of each user, or the attribute information carried by the first target data may designate that the data exists in a fixed area of the database, then the small table corresponding to the fixed area is determined.
Optionally, the database described in this embodiment is a MySQL database, wherein the flexible property of the MySQL database pluggable engine is used to implement the above embodiment based on the MySQL engine and the combination of the programs executing the above embodiment.
In summary, in the data processing method provided in this embodiment, attribute information of first target data to be stored in a database is obtained; determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises a large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in at least one small table and the attribute information of the data stored in at least one small table; and storing the first target data into a small table corresponding to the attribute information of the first target data. Therefore, the data processing method provided by this embodiment can store data of different attribute information through the small table in the database, and include the corresponding relationship between the small table and the attribute information of the data stored in the large table. Therefore, data are stored in different small tables, query and subsequent processing of the data stored in the small tables can be realized through the large table, and the efficiency of data processing in the database is improved.
Fig. 2 is a flowchart illustrating a data processing method according to an embodiment of the present invention. The embodiment shown in fig. 2 is based on the embodiment shown in fig. 1, and is a step after S103. It is understood that fig. 1 is an operation of storing data in a database, and the embodiment shown in fig. 2 provides a flow of searching for data in the database, the definitions of the large table and the small table of the database are the same as those in fig. 1, and the searched second target data may be the same as or different from the first target data. Specifically, the data processing method shown in fig. 2 includes:
s201: and acquiring attribute information of the second target data to be processed in the database.
Specifically, the method for acquiring the attribute information of the second target data in S201 in this embodiment is the same as the method for acquiring the attribute information of the first target data in S101, and is not described again.
S202: and determining a small table corresponding to the attribute information of the second target data according to the large table of the database.
Specifically, the method for determining the attribute information of the second target data from the large table of the database in S202 in this embodiment is the same as the method for determining the attribute information of the first target data from the large table of the database in S102, and is not described again.
S203: and acquiring the second target data from the small table corresponding to the attribute information of the second target data.
S204: and adding, deleting, modifying and/or querying the second target data.
Specifically, in S203 and S204 of the present embodiment, according to the small table determined in S202, the second target data in the small table may be searched again and finally determined, and the second target data is extracted to perform operations of adding, deleting, modifying and/or querying the second target data.
Therefore, with the embodiment as shown in fig. 2, it is possible to store data of different attribute information by a small table in a database, and include a correspondence relationship between the small table and the attribute information of the data stored therein in the large table. Therefore, data are stored in different small tables, and query and subsequent processing of the data stored in the small tables can be realized through the large table. Compared with the prior art that all log files of a database are stored in one large table and the retrieval needs to be carried out in the huge large table during the retrieval, the small table where the data are located can be determined directly through the index of the large table, the retrieval is carried out in the small table with small data quantity, and the efficiency of data processing in the database, particularly the efficiency of data query, can be greatly improved.
Alternatively, in the embodiments shown in fig. 1 and fig. 2, before S101, a manner of establishing a large table and a small table in advance in a database may be further included as follows. The method specifically comprises the following steps: acquiring attribute information of all data stored in a database; determining the number of at least one small table in the database according to the classification result of the attribute information of all the data; and establishing at least one small table in the database, and storing the corresponding relation between the at least one small table and the attribute information of the data stored in the at least one small table in the large table.
Specifically, in the present embodiment, the small table to be created is determined according to the attribute information of all data stored in the database. All data here may be all data currently stored in the database, or may be all data that may be stored in the database. For example, all the log information of a year may be stored in the database, and when the attribute information is the generation time of the log file, it may be determined that the attribute information of all the data in the database is 12 months, and thus it is determined that 12 small tables are established in the database, each small table being used for the log file generated in a different month. And establishes correspondence relationships such as "1 month-small table 1", "2 month-small table 2" … … in the large table.
Optionally, in the embodiment shown in fig. 1 and fig. 2, in the step of determining the small table corresponding to the first target attribute information in S102, the step may specifically include: determining whether a small table corresponding to the attribute information of the first target data exists in the database; and if the attribute information does not exist, establishing a small table corresponding to the attribute information of the first target data in the database.
Specifically, in this embodiment, if the first target data is stored in the embodiments shown in fig. 1 and 2, there is no small table corresponding to the attribute information. A small table may be newly created when the first target data needs to be stored in the database. For example: all log information of a certain year may be stored in the database, and when the attribute information is the generation time of the log file, if only a large table and a small table 1 exist in the database at this time, the corresponding relation of "1 month-small table 1" is recorded in the large table. If the attribute information of the first target data acquired at this time is: if the generation time is 2 months and 1 day, but the database does not have a small table for storing the data, the small table 2 can be newly built in the database for storing the log file with the generation time of 2 months, and the corresponding relation of '2 months-small table 2' is added in the large table.
More specifically, when the database is MySQL, the mapping relationship of the small table may be stored as the large table by the database file suffixed with ". MRG" in the present embodiment, and the small table may be the database file suffixed with ". MYD". When looking up the data in the ". MYD" of the small table, the small table needs to be determined first through the mapping relation stored in the ". MRG". Therefore, the development of complex path functions of front-end service codes in the database is reduced, and the query speed is improved under the condition of large data volume. The test shows that the query speed can be controlled within millisecond level under the condition of hundreds of GB data volume.
FIG. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 3, the data processing apparatus provided in the present embodiment includes: an acquisition module 301, a determination module 302 and a processing module 303.
The obtaining module 301 is configured to obtain attribute information of first target data to be stored in a database; the determining module 302 is configured to determine a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises a large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in at least one small table and the attribute information of the data stored in at least one small table; the processing module 303 is configured to store the first target data into a small table corresponding to the attribute information of the first target data.
The data processing apparatus provided in this embodiment can be used to execute the data processing method shown in fig. 1, and the implementation manner and principle thereof are the same, and are not described herein again.
Optionally, in the foregoing embodiment, the obtaining module 301 is further configured to obtain attribute information of second target data to be processed in the database; the determining module 302 is further configured to determine a small table corresponding to the attribute information of the second target data according to the large table of the database; the processing module 303 is further configured to obtain second target data from a small table corresponding to the attribute information of the second target data; the processing module 303 is further configured to perform operations of adding, deleting, modifying and/or querying the second target data.
The data processing apparatus provided in this embodiment can be used to execute the data processing method shown in fig. 2, and the implementation manner and principle thereof are the same, and are not described herein again.
Optionally, in the above embodiment, the obtaining module 301 is further configured to obtain attribute information of all data stored in the database; the determining module 302 is further configured to determine the number of at least one small table in the database according to the classification result of the attribute information of all the data; the determining module 302 is further configured to establish at least one small table in the database, and store a corresponding relationship between the at least one small table and the attribute information of the data stored in the at least one small table in the large table.
Optionally, in the foregoing embodiment, the determining module 302 is specifically configured to determine whether a small table corresponding to the attribute information of the first target data exists in the database; if the attribute information does not exist, the determining module is further used for establishing a small table corresponding to the attribute information of the first target data in the database.
Optionally, in the above embodiment, the database is used to store log data; the attribute information is the generation time of the log data.
The data processing apparatus provided in the foregoing embodiments can be used to execute the data processing method shown in the foregoing corresponding embodiments, and the implementation manner and principle thereof are the same, and are not described herein again.
It should be noted that the division of the modules in the embodiments of the present application is schematic, and is only one division of logic functions, and there may be another division manner in actual implementation. Each functional module in the embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The present application also provides a computer-readable storage medium having stored therein program code that, when executed, performs the data processing method as in any one of the above embodiments.
The present application also provides a computer program product comprising program code that, when executed by a processor, implements the data processing method as in any of the above embodiments.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and any simple modification, equivalent change and modification made to the above embodiment according to the technical spirit of the present invention are still within the scope of the technical solution of the present invention.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring attribute information of first target data to be stored in a database;
determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises the large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in the at least one small table and the attribute information of the data stored in the at least one small table;
and storing the first target data into a small table corresponding to the attribute information of the first target data.
2. The method according to claim 1, wherein after storing the first target data into the small table corresponding to the attribute information, further comprising:
acquiring attribute information of second target data to be processed in the database;
determining a small table corresponding to the attribute information of the second target data according to the large table of the database;
and acquiring the second target data from a small table corresponding to the attribute information of the second target data.
3. The method according to claim 2, wherein after the obtaining the second target data from the small table corresponding to the attribute information of the second target data, further comprising:
and performing addition, deletion, modification and/or query operations on the second target data.
4. The method according to any one of claims 1-3, further comprising:
acquiring attribute information of all data stored in the database;
determining the number of the at least one small table in the database according to the classification result of the attribute information of all the data;
and establishing the at least one small table in the database, and storing the corresponding relation between the at least one small table and the attribute information of the data stored in the at least one small table in the large table.
5. The method according to any one of claims 1 to 3, wherein the determining, according to the large table of the database, the small table corresponding to the attribute information of the first target data includes:
determining whether a small table corresponding to the attribute information of the first target data exists in the database;
and if the first target data does not exist, establishing a small table corresponding to the attribute information of the first target data in the database.
6. The method of any of claims 1-3, wherein the database is configured to store log data; the attribute information is the generation time of the log data.
7. A data processing apparatus characterized by comprising:
the acquisition module is used for acquiring attribute information of first target data to be stored in a database;
the determining module is used for determining a small table corresponding to the attribute information of the first target data according to the large table of the database; the database comprises the large table and at least one small table, the attribute information of the data stored in each small table is different, and the large table comprises the corresponding relation between the attribute information of the data stored in the at least one small table and the attribute information of the data stored in the at least one small table;
and the processing module is used for storing the first target data into a small table corresponding to the attribute information of the first target data.
8. The apparatus of claim 7,
the acquisition module is further used for acquiring attribute information of second target data to be processed in the database;
the determining module is further configured to determine a small table corresponding to the attribute information of the second target data according to the large table of the database;
the processing module is further configured to obtain the second target data from a small table corresponding to the attribute information of the second target data;
the processing module is further configured to perform operations of adding, deleting, modifying, and/or querying the second target data.
9. The apparatus according to claim 7 or 8,
the acquisition module is further used for acquiring attribute information of all data stored in the database;
the determining module is further configured to determine the number of the at least one small table in the database according to the classification result of the attribute information of all the data;
the determining module is further configured to establish the at least one small table in the database, and store a corresponding relationship between the at least one small table and attribute information of data stored in the at least one small table in the large table.
10. The apparatus according to claim 7 or 8,
the determining module is specifically configured to determine whether a small table corresponding to the attribute information of the first target data exists in the database;
if the attribute information does not exist, the determining module is further used for establishing a small table corresponding to the attribute information of the first target data in the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811200505.7A CN111061721B (en) | 2018-10-16 | 2018-10-16 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811200505.7A CN111061721B (en) | 2018-10-16 | 2018-10-16 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111061721A true CN111061721A (en) | 2020-04-24 |
| CN111061721B CN111061721B (en) | 2023-07-25 |
Family
ID=70296281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811200505.7A Active CN111061721B (en) | 2018-10-16 | 2018-10-16 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111061721B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6542893B1 (en) * | 2000-02-29 | 2003-04-01 | Unisys Corporation | Database sizer for preemptive multitasking operating system |
| US20070233648A1 (en) * | 2006-03-30 | 2007-10-04 | International Business Machines Corporation | Execution cost reduction of sampled queries in a database |
| CN105260399A (en) * | 2015-09-17 | 2016-01-20 | 西安未来国际信息股份有限公司 | Method for acquiring and retrieving distributed log |
| CN105989140A (en) * | 2015-02-27 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data block processing method and equipment |
| CN106021395A (en) * | 2016-05-12 | 2016-10-12 | 成都鼎智汇科技有限公司 | Drug information storage method in internet-of-things environment |
| CN106407172A (en) * | 2016-11-21 | 2017-02-15 | 成都鼎桥通信技术有限公司 | XML (Extensible Markup Language)-based report generating method |
| KR20170081557A (en) * | 2016-01-04 | 2017-07-12 | 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. | Method and device for updating object data in object storage system |
| CN107239382A (en) * | 2017-06-23 | 2017-10-10 | 深圳市冬泉谷信息技术有限公司 | The log processing method and system of a kind of container application |
| CN107451034A (en) * | 2017-08-17 | 2017-12-08 | 浪潮软件股份有限公司 | A kind of big data cluster log management apparatus, method and system |
| US20180004737A1 (en) * | 2016-06-30 | 2018-01-04 | Canon Kabushiki Kaisha | Information processing apparatus, control method, and storage medium |
| US9898614B1 (en) * | 2015-09-18 | 2018-02-20 | Amazon Technologies, Inc. | Implicit prioritization to rate-limit secondary index creation for an online table |
-
2018
- 2018-10-16 CN CN201811200505.7A patent/CN111061721B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6542893B1 (en) * | 2000-02-29 | 2003-04-01 | Unisys Corporation | Database sizer for preemptive multitasking operating system |
| US20070233648A1 (en) * | 2006-03-30 | 2007-10-04 | International Business Machines Corporation | Execution cost reduction of sampled queries in a database |
| CN105989140A (en) * | 2015-02-27 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data block processing method and equipment |
| CN105260399A (en) * | 2015-09-17 | 2016-01-20 | 西安未来国际信息股份有限公司 | Method for acquiring and retrieving distributed log |
| US9898614B1 (en) * | 2015-09-18 | 2018-02-20 | Amazon Technologies, Inc. | Implicit prioritization to rate-limit secondary index creation for an online table |
| KR20170081557A (en) * | 2016-01-04 | 2017-07-12 | 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. | Method and device for updating object data in object storage system |
| CN106021395A (en) * | 2016-05-12 | 2016-10-12 | 成都鼎智汇科技有限公司 | Drug information storage method in internet-of-things environment |
| US20180004737A1 (en) * | 2016-06-30 | 2018-01-04 | Canon Kabushiki Kaisha | Information processing apparatus, control method, and storage medium |
| CN106407172A (en) * | 2016-11-21 | 2017-02-15 | 成都鼎桥通信技术有限公司 | XML (Extensible Markup Language)-based report generating method |
| CN107239382A (en) * | 2017-06-23 | 2017-10-10 | 深圳市冬泉谷信息技术有限公司 | The log processing method and system of a kind of container application |
| CN107451034A (en) * | 2017-08-17 | 2017-12-08 | 浪潮软件股份有限公司 | A kind of big data cluster log management apparatus, method and system |
Non-Patent Citations (1)
| Title |
|---|
| 田一泓: "Cloud Foundry的云资源管理系统的设计与实现" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111061721B (en) | 2023-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2018006721A1 (en) | Method and apparatus for storing log file | |
| CN111367976A (en) | Method and device for exporting EXCEL file data based on JAVA reflection mechanism | |
| US9104567B2 (en) | Memory-leak identification | |
| CN111124872A (en) | Branch detection method and device based on difference code analysis and storage medium | |
| CN112840334A (en) | Method and device for managing data of partition table, management node and storage medium | |
| CN114328601A (en) | Data down-sampling and data query method, system and storage medium | |
| CN113326288A (en) | Data processing method and device and electronic equipment | |
| CN110298178B (en) | Trusted policy learning method and device and trusted security management platform | |
| CN112306957A (en) | Method and device for acquiring index node number, computing equipment and storage medium | |
| CN104090924A (en) | Method and device for cleaning up privacy data | |
| CN116414935A (en) | Method for distributed Search space vector data based on Elastic Search | |
| CN110019783B (en) | Attribute word clustering method and device | |
| CN111061721A (en) | Data processing method and device | |
| CN115544050A (en) | Operation log recording method, device, equipment and storage medium | |
| CN112632266B (en) | Data writing method and device, computer equipment and readable storage medium | |
| CN113282586A (en) | Information processing method, device, equipment and readable storage medium | |
| CN114860362A (en) | Interface updating method and device | |
| CN110221952B (en) | Service data processing method and device and service data processing system | |
| CN114297236A (en) | Data blood relationship analysis method, terminal equipment and storage medium | |
| CN112699129A (en) | Data processing system, method and device | |
| CN108132971B (en) | Analysis method and device for database fragment files | |
| JP2016133946A (en) | Source code reviewing method and system therefor | |
| CN110597828A (en) | Database changing method, device, equipment and storage medium | |
| CN112749189A (en) | Data query method and device | |
| CN117093597B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |