CN110990334A - File processing method, system, device and storage medium for HDFS - Google Patents
File processing method, system, device and storage medium for HDFS Download PDFInfo
- Publication number
- CN110990334A CN110990334A CN201911140253.8A CN201911140253A CN110990334A CN 110990334 A CN110990334 A CN 110990334A CN 201911140253 A CN201911140253 A CN 201911140253A CN 110990334 A CN110990334 A CN 110990334A
- Authority
- CN
- China
- Prior art keywords
- directory
- hdfs
- merging
- file
- files
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The application discloses a file processing method, a file processing system, file processing equipment and a file processing storage medium for an HDFS. The method comprises the steps of configuring a first directory where small files of the HDFS to be combined are located and a second directory which is output after combination; merging the small files of the HDFS to be merged based on a MapReduce program; judging whether the number of data lines before and after the merging of the small files is the same or not; if the files are the same, deleting the files under the first directory, and then moving the files under the second directory to the first directory; if not, the merge fails. The method and the device solve the technical problem that the processing effect is not good when a large number of small files exist in the HDFS. By adopting the MapReduce program to carry out merging, the merging speed is high, and the result is verified, so that the merging correctness is ensured.
Description
Technical Field
The present application relates to the field of distributed file processing, and in particular, to a file processing method, system, device, and storage medium for HDFS.
Background
Hadoop is a Distributed System infrastructure developed by the Apache foundation, and Hadoop realizes a Distributed File System (Hadoop Distributed File System), HDFS for short. The invention focuses on a method for combining small files of an HDFS (Hadoop distributed File System), so that the storage space and the performance influence of data are reduced.
The inventor finds that if a large number of small files exist on the HDFS, serious problems are caused to system performance, and the existing mode cannot fully utilize cluster resources, so that data is lost.
Aiming at the problem that the HDFS in the related art is poor in processing effect when a large number of small files exist, an effective solution is not provided at present.
Disclosure of Invention
The present application mainly aims to provide a file processing method, a file processing system, a file processing device, and a file processing storage medium for an HDFS, so as to solve the problem of poor processing effect when a large number of small files exist in the HDFS.
In order to achieve the above object, according to an aspect of the present application, there is provided a file processing method for an HDFS, for merging small files of the HDFS.
The file processing method for the HDFS comprises the following steps: configuring a first directory where small files of the HDFS to be merged are located and a second directory output after merging; merging the small files of the HDFS to be merged based on a MapReduce program; judging whether the number of data lines before and after the merging of the small files is the same or not; if the files are the same, deleting the files under the first directory, and then moving the files under the second directory to the first directory; if not, the merge fails.
Preferably, when merging the small files of the HDFS to be merged based on the MapReduce program, the method further includes:
and when the small files of the HDFS to be merged are merged based on a MapReduce program, compressing the data and then outputting the compressed data to the second directory.
Preferably, if the files are the same, deleting the file under the first directory, and then moving the file under the second directory to the first directory includes:
respectively creating two Hive temporary tables, wherein an input table points to an inputpath directory, and an output table points to an outputpath directory;
and comparing the number of data lines before combination with the number of data lines after combination according to the query result, deleting the file under the inputpath directory if the data lines are the same, and then moving the file under the outputpath directory to the inputpath directory.
Preferably, merging the small files of the HDFS to be merged based on a MapReduce program includes:
and adopting a MapReduce program based on CombineFileInputFormat to merge the small files of the HDFS to be merged, and enabling a data processing function Math.
In order to achieve the above object, according to another aspect of the present application, there is provided a file processing system for an HDFS for merging small files of the HDFS.
The file processing system for the HDFS according to the present application includes: the configuration module is used for configuring a first directory where the small files of the HDFS to be combined are located and a second directory output after combination; the merging module is used for merging the small files of the HDFS to be merged based on a MapReduce program; the judging module is used for judging whether the number of the data lines before and after the merging of the small files is the same or not; the first processing module is used for deleting the file under the first directory and then moving the file under the second directory to the first directory when the number of data lines is the same; and the second processing module is used for failing to merge when the data rows are different.
Preferably, the compression module is configured to compress data and output the compressed data to the second directory when merging the small files of the HDFS to be merged based on a MapReduce program.
Preferably, the first processing module is configured to create two Hive temporary tables, where an input table points to an inputpath directory and an output table points to an outputpath directory;
and comparing the number of data lines before combination with the number of data lines after combination according to the query result, deleting the file under the inputpath directory if the data lines are the same, and then moving the file under the outputpath directory to the inputpath directory.
Preferably, the merging module is configured to merge the small files of the HDFS to be merged by using a MapReduce program based on combinaefileinputformat, and enable a data processing function math.ceil to process the small files therein.
In the file processing method, the file processing system, the file processing equipment and the file processing medium for the HDFS, the small files of the HDFS to be merged are merged based on a MapReduce program in a mode of configuring a first directory where the small files of the HDFS to be merged are located and a second directory output after merging, and whether the number of data lines before merging and after merging of the small files is the same is judged; if the files are the same, the files under the first directory are deleted, and then the files under the second directory are moved to the first directory, so that the aims of merging the small files of the HDFS and verifying the merging result are fulfilled, the technical effects of reducing the number and the size of the files are achieved, and the technical problem of poor processing effect when a large number of small files exist in the HDFS is solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, serve to provide a further understanding of the application and to enable other features, objects, and advantages of the application to be more apparent. The drawings and their description illustrate the embodiments of the invention and do not limit it. In the drawings:
FIG. 1 is a schematic flow chart of a file processing method for an HDFS according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a file processing apparatus for HDFS according to an embodiment of the present application;
fig. 3 is a schematic diagram of an implementation principle according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In this application, the terms "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings. These terms are used primarily to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to a particular orientation or to be constructed and operated in a particular orientation.
Moreover, some of the above terms may be used to indicate other meanings besides the orientation or positional relationship, for example, the term "on" may also be used to indicate some kind of attachment or connection relationship in some cases. The specific meaning of these terms in this application will be understood by those of ordinary skill in the art as appropriate.
Furthermore, the terms "mounted," "disposed," "provided," "connected," and "sleeved" are to be construed broadly. For example, it may be a fixed connection, a removable connection, or a unitary construction; can be a mechanical connection, or an electrical connection; may be directly connected, or indirectly connected through intervening media, or may be in internal communication between two devices, elements or components. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1, the method includes steps S101 to S104 as follows:
step S101, configuring a first directory where small files of the HDFS to be merged are located and a second directory output after merging;
and configuring the first directory where the small files of the HDFS to be merged are located and configuring the second directory which is output after merging.
Specifically, a directory inputpath where the HDFS small files to be merged are located and a directory outputpath output after merging are configured.
Step S102, merging the small files of the HDFS to be merged based on a MapReduce program;
and carrying out merging operation on the small files of the HDFS to be merged based on a MapReduce program.
Specifically, a MapReduce program based on CombineFileInputFormat is adopted for merging, assuming that the total size of small files under an inputpath directory is allsize, a fragment size split _ size is configured, CombineFileInputFormat is used for merging a plurality of small files under the inputpath directory on an HDFS into Math.ceil (allsize/split _ size) InputSplits, and then Math.ceil (allsize/split _ size) mappers are enabled to process the files inside, so that the number of Maps is reduced.
Step S103, judging whether the number of rows of the data before merging and the number of rows of the data after merging of the small files are the same or not;
and checking the integrity by judging whether the number of the data lines before merging the small files is the same as that after merging the small files.
Specifically, two hive temporary tables are created respectively: the input table points to the input path and the output table points to the output path directory.
create table if not exists input(data string)location'$inputpath'
create table if not exists output(data string)location'outputpath'
Using select count (0) from input and select count (0) from output path query result to compare the number of data lines before merging and after merging
Step S104, if the files are the same, deleting the files under the first directory, and then moving the files under the second directory to the first directory;
specifically, if the verification results are the same, the file under the first directory is deleted, and then the file under the second directory is moved to the first directory.
And step S105, if the two are not the same, the combination fails.
And if the check results are not the same, the combination result fails.
From the above description, it can be seen that the following technical effects are achieved by the present application:
in the embodiment of the application, small files of the HDFS to be merged are merged based on a MapReduce program by adopting a mode of configuring a first directory where the small files of the HDFS to be merged are located and a second directory output after merging, and whether the number of data lines before merging and after merging of the small files is the same is judged; if the files are the same, the files under the first directory are deleted, and then the files under the second directory are moved to the first directory, so that the aims of merging the small files of the HDFS and verifying the merging result are fulfilled, the technical effects of reducing the number and the size of the files are achieved, and the technical problem of poor processing effect when a large number of small files exist in the HDFS is solved.
According to the embodiment of the present application, as an optimization in the embodiment, when the MapReduce program merges the small files of the HDFS to be merged, the method further includes: and when the small files of the HDFS to be merged are merged based on a MapReduce program, compressing the data and then outputting the compressed data to the second directory.
Specifically, gzip is adopted to compress and output the final data to the outputpath directory by setting OutputCompressorClass to gzipodec.
According to the embodiment of the present application, as a preferable preference in the embodiment, if the files are the same, deleting the file under the first directory, and then moving the file under the second directory to the first directory includes: respectively creating two Hive temporary tables, wherein an input table points to an inputpath directory, and an output table points to an outputpath directory; and comparing the number of data lines before combination with the number of data lines after combination according to the query result, deleting the file under the inputpath directory if the data lines are the same, and then moving the file under the outputpath directory to the inputpath directory.
Specifically, the MapReduce-based program divides the input file into pieces (Split), each piece corresponds to a map task, and at least one piece is defaulted for a file, and one piece only belongs to one file. Such a large number of small files results in a large number of map tasks, resulting in excessive consumption of resources and inefficiency. In the embodiment of the application, the small files are merged, the number of data lines before merging and after merging is compared according to the query result, if the number of data lines is the same, the files in the inputpath directory are deleted firstly, and then the files in the outputpath directory are moved to the inputpath directory.
According to the embodiment of the present application, as an optimization in the embodiment, merging the small files of the HDFS to be merged based on a MapReduce program includes: and adopting a MapReduce program based on CombineFileInputFormat to merge the small files of the HDFS to be merged, and enabling a data processing function Math.
Specifically, a MapReduce program based on CombineFileInputFormat is adopted for merging, assuming that the total size of small files under an inputpath directory is allsize, a fragment size split _ size is configured, CombineFileInputFormat is used for merging a plurality of small files under the inputpath directory on an HDFS into Math.ceil (allsize/split _ size) InputSplits, and then Math.ceil (allsize/split _ size) mappers are enabled to process the files inside, so that the number of Maps is reduced.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
According to an embodiment of the present application, there is also provided a file processing system for HDFS, for implementing the above method, as shown in fig. 2, the system including: the configuration module 10 is configured to configure a first directory where the small files of the HDFS to be merged are located and a second directory output after merging; a merging module 11, configured to merge the small files of the HDFS to be merged based on a MapReduce program; a judging module 12, configured to judge whether the number of data lines before merging and the number of data lines after merging of the small files are the same; the first processing module 13 is configured to delete the file in the first directory and then move the file in the second directory to the first directory when the number of data lines is the same; and the second processing module 14 is configured to fail merging when the number of data lines is different.
The configuration module 10 in the embodiment of the present application configures the first directory where the small files of the HDFS to be merged are located and configures the second directory output after merging.
Specifically, a directory inputpath where the HDFS small files to be merged are located and a directory outputpath output after merging are configured.
The merging module 11 of the embodiment of the present application performs merging operations on the small files of the HDFS to be merged based on a MapReduce program.
Specifically, a MapReduce program based on CombineFileInputFormat is adopted for merging, assuming that the total size of small files under an inputpath directory is allsize, a fragment size split _ size is configured, CombineFileInputFormat is used for merging a plurality of small files under the inputpath directory on an HDFS into Math.ceil (allsize/split _ size) InputSplits, and then Math.ceil (allsize/split _ size) mappers are enabled to process the files inside, so that the number of Maps is reduced.
In the determining module 12 of the embodiment of the present application, integrity is checked by determining whether the number of rows of data before merging of the small file is the same as that after merging of the small file.
Specifically, two hive temporary tables are created respectively: the input table points to the input path and the output table points to the output path directory.
create table if not exists input(data string)location'$inputpath'
create table if not exists output(data string)location'outputpath'
Using select count (0) from input and select count (0) from output path query result to compare the number of data lines before merging and after merging
Specifically, in the first processing module 13 of the embodiment of the present application, if the verification results are the same, the file under the first directory is deleted first, and then the file under the second directory is moved to the first directory.
According to the embodiment of the present application, as a preference in the embodiment, the system further includes: and a compression module (not shown) configured to compress data and output the compressed data to the second directory when merging the small files of the HDFS to be merged based on a MapReduce program.
Specifically, the compression module compresses the final data by gzip and outputs the compressed final data to the outputpath directory by setting outputcompressed class to gzipodec.
According to the embodiment of the present application, as a preferred embodiment in the present embodiment, the first processing module 13 is configured to create two Hive temporary tables, where an input table points to an inputpath directory and an output table points to an outputpath directory; and comparing the number of data lines before combination with the number of data lines after combination according to the query result, deleting the file under the inputpath directory if the data lines are the same, and then moving the file under the outputpath directory to the inputpath directory.
Specifically, the first processing module 13 divides (Split) the input file based on the MapReduce program, where each of the divided files corresponds to a map task, and a file is defaulted to have at least one divided file, and a divided file only belongs to one file. Such a large number of small files results in a large number of map tasks, resulting in excessive consumption of resources and inefficiency. In the embodiment of the application, the small files are merged, the number of data lines before merging and after merging is compared according to the query result, if the number of data lines is the same, the files in the inputpath directory are deleted firstly, and then the files in the outputpath directory are moved to the inputpath directory.
According to the embodiment of the present application, as a preferred choice in the embodiment, the merging module 11 is configured to merge the small files of the HDFS to be merged by using a MapReduce program based on combinaefileinputformat, and enable a data processing function math.
Specifically, the merging module 11 merges by using a MapReduce program based on CombineFileInputFormat, assuming that the total size of the small files under the inputpath directory is allsize, configures the fragmentation size split _ size, merges the multiple small files under the inputpath directory on the HDFS into the math.ceil (allsize/split _ size) InputSplit with the CombineFileInputFormat, and then enables the math.ceil (allsize/split _ size) mappers to process the files therein, so as to reduce the number of maps.
It will be apparent to those skilled in the art that the modules or steps of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present application is not limited to any specific combination of hardware and software.
Please refer to fig. 3, which is a schematic diagram of an implementation principle of the present application, and includes the following steps:
step S1, configuring a small file directory inputpath to be merged;
step S2, configuring the merged file output directory outputpath;
step S3, merging the compressed small files based on MapReduce;
the method comprises the steps of adopting a MapReduce program based on CombineFileInputFormat to carry out merging, assuming that the total size of small files under an inputpath directory is allsize, configuring a fragmentation size split _ size, merging a plurality of small files under the inputpath directory on the HDFS into Math.ceil (allsize/split _ size) InputSplits by using the CombineFileInputFormat, and then enabling Math.ceil (allsize/split _ size) mappers to process the files inside so as to reduce the number of maps. And setting OutputCompressorClass as GzipCodec.
Step S4, determine whether the number of lines is equal?
Two hive temporary tables are created respectively: input table points to input path, output table points to output path directory
create table if not exists input(data string)location'$inputpath'
create table if not exists output(data string)location'outputpath'
Using select count (0) from input and select count (0) from output path query results to compare whether the number of rows of data before and after merging is equal?
If so, the process proceeds to step S5, where the output path file is moved to the input path.
And (4) successfully merging the small files, deleting the files in the inputpath directory, and then moving the files in the outputpath directory to the inputpath directory.
If not, the step S6 is entered, the merging fails, and an alarm mail is sent;
and (4) the number of data lines before and after the small files are merged is inconsistent, and the warning mail is sent after the merging fails.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A file processing method for an HDFS (Hadoop distributed File System), which is used for merging small files of the HDFS, comprises the following steps:
configuring a first directory where small files of the HDFS to be merged are located and a second directory output after merging;
merging the small files of the HDFS to be merged based on a MapReduce program;
judging whether the number of data lines before and after the merging of the small files is the same or not;
if the files are the same, deleting the files under the first directory, and then moving the files under the second directory to the first directory;
if not, the merge fails.
2. The file processing method for the HDFS according to claim 1, wherein when merging the small files of the HDFS to be merged based on a MapReduce program, the method further includes:
and when the small files of the HDFS to be merged are merged based on a MapReduce program, compressing the data and then outputting the compressed data to the second directory.
3. The file processing method for the HDFS according to claim 1, wherein if the files are the same, deleting the file under the first directory and then moving the file under the second directory to the first directory comprises:
respectively creating two Hive temporary tables, wherein an input table points to an inputpath directory, and an output table points to an outputpath directory;
and comparing the number of data lines before combination with the number of data lines after combination according to the query result, deleting the file under the inputpath directory if the data lines are the same, and then moving the file under the outputpath directory to the inputpath directory.
4. The file processing method for the HDFS according to claim 1, wherein merging the small files of the HDFS to be merged based on a MapReduce program includes:
and adopting a MapReduce program based on CombineFileInputFormat to merge the small files of the HDFS to be merged, and enabling a data processing function Math.
5. A file processing system for HDFS, for merging small files of HDFS, comprising:
the configuration module is used for configuring a first directory where the small files of the HDFS to be combined are located and a second directory output after combination;
the merging module is used for merging the small files of the HDFS to be merged based on a MapReduce program;
the judging module is used for judging whether the number of the data lines before and after the merging of the small files is the same or not;
the first processing module is used for deleting the file under the first directory and then moving the file under the second directory to the first directory when the number of data lines is the same;
and the second processing module is used for failing to merge when the data rows are different.
6. The file processing system for HDFS according to claim 5, further comprising: and the compression module is used for compressing the data and then outputting the data to the second directory when the small files of the HDFS to be merged are merged based on the MapReduce program.
7. The file processing system for the HDFS according to claim 5, wherein the first processing module is configured to create two Hive temporary tables, respectively, an input table pointing to an inputpath directory and an output table pointing to an outputpath directory;
and comparing the number of data lines before combination with the number of data lines after combination according to the query result, deleting the file under the inputpath directory if the data lines are the same, and then moving the file under the outputpath directory to the inputpath directory.
8. The file processing system of claim 5, wherein the merging module is configured to perform the small file merging of the HDFS to be merged by using a MapReduce program based on combinarefileinputformat, and enable a data processing function math.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the file processing method for HDFS according to any one of claims 1 to 4 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the file processing method for HDFS according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911140253.8A CN110990334A (en) | 2019-11-19 | 2019-11-19 | File processing method, system, device and storage medium for HDFS |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911140253.8A CN110990334A (en) | 2019-11-19 | 2019-11-19 | File processing method, system, device and storage medium for HDFS |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110990334A true CN110990334A (en) | 2020-04-10 |
Family
ID=70085470
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911140253.8A Withdrawn CN110990334A (en) | 2019-11-19 | 2019-11-19 | File processing method, system, device and storage medium for HDFS |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990334A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231293A (en) * | 2020-09-14 | 2021-01-15 | 杭州数梦工场科技有限公司 | File reading method and device, electronic equipment and storage medium |
-
2019
- 2019-11-19 CN CN201911140253.8A patent/CN110990334A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231293A (en) * | 2020-09-14 | 2021-01-15 | 杭州数梦工场科技有限公司 | File reading method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20150347305A1 (en) | Method and apparatus for outputting log information | |
| CN112434000B (en) | Small file merging method, device and equipment based on HDFS | |
| CN105653425A (en) | Complicated event processing engine based monitoring system | |
| CN111355816B (en) | Server selection method, device, equipment and distributed service system | |
| CN105302920A (en) | Optimal management method and system for cloud storage data | |
| CN111930770A (en) | Data query method and device and electronic equipment | |
| CN105528454A (en) | Log treatment method and distributed cluster computing device | |
| CN105868218B (en) | A kind of data processing method and electronic equipment | |
| CN103455346A (en) | Application program deployment method, deployment main control computer, deployment client side and cluster | |
| CN110990334A (en) | File processing method, system, device and storage medium for HDFS | |
| CN110825706B (en) | Data compression method and related equipment | |
| US11544260B2 (en) | Transaction processing method and system, and server | |
| CN104853061A (en) | Hadoop-based video processing method and system | |
| CN117056303B (en) | Data storage method and device suitable for military operation big data | |
| CN114448972A (en) | Distributed storage log compression downloading method, system, terminal and storage medium | |
| CN105721179A (en) | Log collection system and data transmission method and local server therein | |
| CN113468182B (en) | Data storage method and system | |
| CN110704404A (en) | Data quality checking method, device and system | |
| CN110998524A (en) | Method for processing configuration file, processing unit, touch chip, device and medium | |
| CN114466000A (en) | CDN gateway source returning method and device | |
| CN112861188A (en) | Data aggregation system and method for multiple clusters | |
| CN113157475A (en) | Log processing method and device, storage medium and electronic equipment | |
| CN112738157A (en) | File uploading method, system and equipment | |
| CN109818767B (en) | Method and device for adjusting Redis cluster capacity and storage medium | |
| WO2020019314A1 (en) | Graph data storage method and system and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200410 |
|
| WW01 | Invention patent application withdrawn after publication |