CN110515894B - Data format conversion method, device, equipment and readable storage medium - Google Patents
Data format conversion method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN110515894B CN110515894B CN201910712544.3A CN201910712544A CN110515894B CN 110515894 B CN110515894 B CN 110515894B CN 201910712544 A CN201910712544 A CN 201910712544A CN 110515894 B CN110515894 B CN 110515894B
- Authority
- CN
- China
- Prior art keywords
- source
- data
- format
- destination
- path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a data format conversion method applied to an HDFS distributed file system, comprising the following steps: acquiring a source path of source data input by a user through a man-machine interaction interface, and a destination path and a destination format of destination data; reading source data according to a source path and determining a source format of the source data; calling a format conversion program matched with a source format and a destination format from a preset format converter to convert the source data from the source format into the destination format, and obtaining the destination data; the format converter stores a plurality of format conversion programs; and storing the destination data according to the destination path. The method and the device can automatically complete data format conversion according to parameters set by a user on a human-computer interaction interface aiming at any source data to be converted, so that conversion efficiency is improved, and good universality is achieved. Accordingly, the data format conversion device, the data format conversion equipment and the readable storage medium disclosed by the application also have the technical effects.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data format conversion method, apparatus, device, and readable storage medium.
Background
In an HDFS distributed file system, in order for different data to have the same format, the data needs to be format converted. The method comprises the following steps: the developer writes the conversion program according to the format of the source data and the destination format to be converted, and then the source data is converted into the destination format by using the program. However, since the amount of data in the HDFS distributed file system is relatively large, if the data format is converted according to the above conversion method, the workload of the developer is increased, and the data format conversion efficiency is also reduced.
Therefore, how to improve the data format conversion efficiency is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the foregoing, it is an object of the present application to provide a data format conversion method, apparatus, device and readable storage medium, so as to improve the data format conversion efficiency. The specific scheme is as follows:
in a first aspect, the present application provides a data format conversion method applied to an HDFS distributed file system, including:
acquiring a source path of source data input by a user through a man-machine interaction interface, and a destination path and a destination format of destination data;
reading source data according to a source path and determining a source format of the source data;
calling a format conversion program matched with a source format and a destination format from a preset format converter to convert the source data from the source format into the destination format, and obtaining the destination data; the format converter stores a plurality of format conversion programs;
and storing the destination data according to the destination path.
Preferably, acquiring a source path of source data input by a user through a man-machine interaction interface includes:
and acquiring source paths of a plurality of source data input by a user through a man-machine interaction interface.
Preferably, acquiring a source path of a plurality of source data input by a user through a man-machine interaction interface includes:
acquiring an incomplete source path of source data input by a user through a man-machine interaction interface;
and searching all complete source paths corresponding to the incomplete source paths by using a regular matching mode, and determining all complete source paths as source paths.
Preferably, after storing the destination data according to the destination path, the method further comprises:
and acquiring a deleting instruction input by a user, and deleting the source data according to the deleting instruction.
Preferably, after storing the destination data according to the destination path, the method further comprises:
and storing the source path, the destination path and the destination format into a history parameter record table.
Preferably, after storing the destination data according to the destination path, the method further comprises:
and calling a data reading program corresponding to the destination format to read the destination data.
Preferably, the method further comprises:
recording a format conversion log of the source data.
In a second aspect, the present application provides a data format conversion device, applied to an HDFS distributed file system, including:
the acquisition module is used for acquiring a source path of source data, a destination path of destination data and a destination format of the destination data which are input by a user through a man-machine interaction interface;
the reading module is used for reading the source data according to the source path and determining the source format of the source data;
the conversion module is used for calling a format conversion program matched with the source format and the destination format from a preset format converter to convert the source data from the source format into the destination format, so as to obtain the destination data; the format converter stores a plurality of format conversion programs;
and the storage module is used for storing the destination data according to the destination path.
In a third aspect, the present application provides a data format conversion apparatus, including:
a memory for storing a computer program;
a processor for executing a computer program to implement the data format conversion method disclosed previously.
In a fourth aspect, the present application provides a readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data format conversion method disclosed previously.
According to the scheme, the application provides a data format conversion method applied to an HDFS distributed file system, comprising the following steps: acquiring a source path of source data input by a user through a man-machine interaction interface, and a destination path and a destination format of destination data; reading source data according to a source path and determining a source format of the source data; calling a format conversion program matched with a source format and a destination format from a preset format converter to convert the source data from the source format into the destination format, and obtaining the destination data; the format converter stores a plurality of format conversion programs; and storing the destination data according to the destination path.
Therefore, the method and the device can provide a man-machine interaction interface for the user, and the user can input the source path of the source data, the destination path of the destination data and the destination format through the man-machine interaction interface; that is, before the data conversion, the user has set the source data to be converted, the destination format to be converted, and where the converted data is stored after the conversion is completed through the human-computer interface. Therefore, corresponding source data can be read in the HDFS distributed file system according to a source path input by a user, so that the source format of the source data can be determined; further, a format conversion program matched with the current source format and the destination format can be called from a preset format converter stored with a plurality of format conversion programs to convert the source data, so that destination data in the destination format can be obtained, and the destination data in the destination format is stored in the HDFS distributed file system. Aiming at any source data to be converted, a user can set parameters related to data format conversion on a human-computer interaction interface, so that the data format conversion can be automatically completed according to the parameters, and the data format conversion efficiency is improved; and the format converter stores a plurality of format conversion programs, so that the application has good universality.
Correspondingly, the data format conversion device, the data format conversion equipment and the readable storage medium have the technical effects.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings may be obtained according to the provided drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of a first data format conversion method disclosed in the present application;
FIG. 2 is a detailed flow chart of step S101 in FIG. 1;
FIG. 3 is a flow chart of a second data format conversion method disclosed in the present application;
FIG. 4 is a schematic diagram of a data format conversion device disclosed in the present application;
FIG. 5 is a schematic diagram of a data format conversion device disclosed in the present application;
FIG. 6 is a schematic diagram of a data format conversion system disclosed herein;
fig. 7 is a schematic diagram of a visual operation page disclosed in the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
At present, the data volume in the HDFS distributed file system is relatively large, and if the data format is converted according to the existing conversion mode, the workload of a developer is increased, and the data format conversion efficiency is also reduced. Therefore, the data format conversion scheme provided by the application can improve the data format conversion efficiency and has good universality.
Referring to fig. 1, an embodiment of the present application discloses a first data format conversion method, which is applied to an HDFS distributed file system, and includes:
s101, acquiring a source path of source data, a destination path of destination data and a destination format of the destination data, which are input by a user through a human-computer interaction interface;
the user can write the source path in the man-machine interaction interface through the keyboard, and can select the source path by utilizing the mouse. The destination path and destination format of the destination data input by the user through the man-machine interaction interface can be the destination path and destination format set by the man-machine interaction interface and used for the user to select, or can be the destination path and destination format freely input by the user.
S102, reading source data according to a source path, and determining a source format of the source data;
the reading program of the data with different formats can be preset, so that the data with any format can be read row by using the reading program.
S103, calling a format conversion program matched with the source format and the destination format from a preset format converter to convert the source data from the source format into the destination format, and obtaining the destination data;
wherein, a plurality of format conversion programs are stored in the format converter. The format conversion programs stored in the format converter are numerous, such as: sequence-to-Text format conversion program, text-to-Sequence format conversion program. Sequence is a binary data format; text is a Text data format. For ease of distinction, different format conversion programs may be identified, for example: "1, 2, 3 … … N" is used as the identification of each format conversion program. It is of course also possible to identify the individual format conversion programs "Sequence-Text", "Text-Sequence" in order to find from the format converter the format conversion program that matches the current source format and the destination format.
S104, storing destination data according to the destination path.
After the destination data in the destination format is obtained through conversion, the destination data can be stored in the HDFS distributed file system according to the destination path of the destination data input by the user. Thus, the data reading program corresponding to the destination format can be called to read the destination data.
Referring to fig. 2, fig. 2 is a detailed flowchart of step S101 in fig. 1. The specific implementation steps of S101 in fig. 1 include:
s201, acquiring an incomplete source path of source data input by a user through a man-machine interaction interface, and a destination path and a destination format of destination data;
s202, searching all complete source paths corresponding to the incomplete source paths by using a regular matching mode, and determining all complete source paths as source paths.
In this embodiment, when a plurality of source data are to be converted at the same time and have similar source paths, a user may input incomplete source paths in the man-machine interface.
For example: it is intended to convert 3 source data simultaneously, the source paths of the 3 source data are respectively:
hdfs:192.168.1.0:9000/tmp/sequence/test1，
hdfs:192.168.1.0:9000/tmp/sequence/test2，
hdfs:192.168.1.0:9000/tmp/sequence/test3。
then the user can input hdfs:192.168.1.0:9000/tmp/sequence/test on the man-machine interface, and further can find out the real path corresponding to hdfs:192.168.1.0:9000/tmp/sequence/test in a regular matching manner, and the found path is the complete source path, namely:
hdfs:192.168.1.0:9000/tmp/sequence/test1，
hdfs:192.168.1.0:9000/tmp/sequence/test2，
hdfs:192.168.1.0:9000/tmp/sequence/test3。
thus, the operation of the user can be facilitated, and the user experience is improved.
Of course, the historical source path, the historical destination path and the historical destination format input by the user can also be stored, so that when the user inputs the parameters again, the historical parameters can be called for the user to select. That is, the source path, the destination path, and the destination format are stored in the history parameter table.
As can be seen, the embodiment of the application discloses a first data format conversion method, which aims at any source data to be converted, and can automatically complete data format conversion according to parameters set by a user on a man-machine interaction interface, so that the efficiency of data format conversion is improved; and the format converter in the embodiment stores a plurality of format conversion programs, so that the embodiment has good universality.
Referring to fig. 3, the embodiment of the application discloses a second data format conversion method, which is applied to an HDFS distributed file system and includes:
s301, acquiring source paths of a plurality of source data input by a user through a man-machine interaction interface, and a destination path and a destination format of destination data;
specifically, the user can input the source paths of a plurality of source data at the same time in the man-machine interaction interface, so that the plurality of source data can be converted into the target format at the same time. For example: the user may input the following source paths at the human-machine interface:
hdfs:192.168.1.0:9000/tmp/sequence/test1，
hdfs:192.168.1.0:9000/tmp/sequence/test2，
hdfs:192.168.1.0:9000/tmp/sequence/test3。
in this way, the source data corresponding to the three source paths can be converted into the destination format at the same time. After the source data corresponding to the three source paths are converted, destination data corresponding to the three source data can be obtained, and the three destination data can be stored as different files under the same destination path or as the same file under the same destination path.
S302, reading source data according to a source path and determining a source format of the source data;
s303, calling a format conversion program matched with a source format and a destination format from a preset format converter to convert the source data from the source format to the destination format, and obtaining destination data;
wherein, a plurality of format conversion programs are stored in the format converter;
s304, storing destination data according to the destination path;
s305, acquiring a deleting instruction input by a user, and deleting the source data according to the deleting instruction.
In this embodiment, the process of performing data format conversion may be recorded in a log system in the HDFS distributed file system, so as to obtain a format conversion log of the source data, where the format conversion log may be recorded: name of source data, source path, source format; name of destination data, destination path, destination format; converting identification information of a format conversion program called by the procedure, and the like.
It should be noted that other implementation steps in the present embodiment are the same as or similar to those in the above embodiment, so that the description of the present embodiment is omitted here.
As can be seen from the above, the present embodiment discloses a second data format conversion method, which aims at any source data to be converted, and can automatically complete data format conversion according to parameters set by a user on a man-machine interaction interface, so that the efficiency of data format conversion is improved; and the format converter in the embodiment stores a plurality of format conversion programs, so that the embodiment has good universality.
A data format conversion device provided in the embodiments of the present application is described below, and a data format conversion device described below and a data format conversion method described above may be referred to with each other.
Referring to fig. 4, an embodiment of the present application discloses a data format conversion device, which is applied to an HDFS distributed file system, and includes:
the acquiring module 401 is configured to acquire a source path of source data, a destination path of destination data, and a destination format of the destination data, which are input by a user through a human-computer interaction interface;
a reading module 402, configured to read the source data according to the source path and determine a source format of the source data;
a conversion module 403, configured to call a format conversion program matched with the source format and the destination format from a preset format converter, and convert the source data from the source format to the destination format, so as to obtain destination data; the format converter stores a plurality of format conversion programs;
the storage module 404 is configured to store destination data according to the destination path.
In one embodiment, the obtaining module is specifically configured to:
and acquiring source paths of a plurality of source data input by a user through a man-machine interaction interface.
In one embodiment, the obtaining module includes:
the acquisition unit is used for acquiring an incomplete source path of source data input by a user through a man-machine interaction interface;
and the searching unit is used for searching all the complete source paths corresponding to the incomplete source paths in a regular matching mode and determining all the complete source paths as source paths.
In one specific embodiment, the method further comprises:
and the deleting module is used for acquiring a deleting instruction input by a user and deleting the source data according to the deleting instruction.
In one specific embodiment, the method further comprises:
and the parameter storage module is used for storing the source path, the destination path and the destination format into the history parameter record table.
In one specific embodiment, the method further comprises:
and the calling module is used for calling a data reading program corresponding to the destination format to read the destination data.
In one specific embodiment, the method further comprises:
and the recording module is used for recording the format conversion log of the source data.
The more specific working process of each module and unit in this embodiment may refer to the corresponding content disclosed in the foregoing embodiment, and will not be described herein.
It can be seen that the present embodiment provides a data format conversion device, including: the device comprises an acquisition module, a reading module, a conversion module and a storage module. Firstly, an acquisition module acquires a source path of source data, a destination path of destination data and a destination format of the destination data, wherein the source path of the source data, the destination path of the destination data and the destination format are input by a user through a human-computer interaction interface; then the reading module reads the source data according to the source path and determines the source format of the source data; the conversion module further calls a format conversion program matched with the source format and the destination format from a preset format converter to convert the source data from the source format into the destination format, and destination data are obtained; the format converter stores a plurality of format conversion programs; and finally, the storage module stores the destination data according to the destination path. Therefore, the modules work separately and cooperate, and the conversion efficiency is improved, and the scheme has good universality.
The following describes a data format conversion device provided in an embodiment of the present application, and the data format conversion device described below and the data format conversion method and apparatus described above may be referred to each other.
Referring to fig. 5, an embodiment of the present application discloses a data format conversion device, including:
a memory 501 for storing a computer program;
a processor 502 for executing the computer program to implement the method disclosed in any of the embodiments above.
The following describes a readable storage medium provided in the embodiments of the present application, and the readable storage medium described below and a data format conversion method, apparatus and device described above may be referred to with each other.
A readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the data format conversion method disclosed in the foregoing embodiments. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
Referring to fig. 6, fig. 6 is a schematic diagram of a data format conversion system disclosed in the present application. As shown in fig. 6, the system includes: the visual operation page, the data reader, the format converter, the data writer and the data remover are arranged on the client.
The visual operation page may be a web page, and the user may fill in a data path (the path is a path on the HDFS distributed file system) to be converted in the visual operation page, and store the data after format conversion (the storage path is a path on the HDFS distributed file system), so as to determine whether to clear the source data after completing the format conversion. While also providing a "submit" button, the visual operation page can be seen in fig. 7. The visual operation page is the man-machine interaction interface.
The path form of the source data may be a single path or a multi-path. When filling out multiple paths, different paths are "separated".
Specifically, the single path may be filled in as follows:
hdfs:192.168.1.0:9000/tmp/sequence/test-r-00000。
multipath may fill out the complete multiple paths directly, for example:
hdfs:192.168.1.0:9000/tmp/sequence/test1，
hdfs:192.168.1.0:9000/tmp/sequence/test2，
hdfs:192.168.1.0:9000/tmp/sequence/test3。
the multipath may also fill in only incomplete paths, so that the system searches for corresponding paths by using a regular matching method, and the incomplete paths may be filled in as follows: hdfs 192.168.1.0:9000/tmp/sequence/test. Where "×" denotes an arbitrary path suffix, such as "1, 2, 3" and the like listed above. When a user fills in hdfs:192.168.1.0:9000/tmp/sequence/test in a visual operation page, the system can automatically query the following paths:
hdfs:192.168.1.0:9000/tmp/sequence/test1，
hdfs:192.168.1.0:9000/tmp/sequence/test2，
hdfs:192.168.1.0:9000/tmp/sequence/test3，
……
hdfs:192.168.1.0:9000/tmp/sequence/testN。
then, all the queried paths are the source paths of the source data.
The user can see whether the source data path, the destination data format and the source data filled in the visual operation page are deleted or not, and the source data path, the destination data format and the source data are all regarded as transfer parameters.
For example: the transfer parameters include:
the source data path is: hdfs 192.168.1.0:9000/tmp/sequence/test-r-00000; the destination data path is: hdfs 192.168.1.0:9000/tmp/text/test-text-r-00000; the destination data format is: text; the deletion of source data is selected.
The data reader is used for reading the data file under the path according to the data file path to be converted, identifying the data format in the data file, calling a corresponding format reading program according to the data format, and then reading the data row by row.
Wherein, the data reader is specifically used for: acquiring a file under a source data path, and identifying the format of data in the file; and calling a reading program in a corresponding format according to the format to read the data. For example: the test-r-0000 file is identified as a Sequence format binary file, and then a program for reading the Sequence binary file is called to read data.
The format converter is used for calling a format conversion program matched with the source format and the destination format so as to complete format conversion by using the format conversion program.
The format converter is specifically configured to: acquiring data to be converted, and reading the data row by row; and converting the read data line by line into a target format. For example: the method comprises the steps of obtaining a file test-r-0000 in a Sequence format, reading data in the file test-r-0000 row by row, converting the data read row by row into a String type by binary byte, and obtaining a Text object so as to achieve the purpose of converting the data from the Sequence into the Text.
The data writer is used for completely writing the converted data into the HDFS distributed file system according to the storage path of the converted data file.
The data writer is specifically used for: and calling a write operation function to store the converted data to a destination data path filled in by the user in the visual operation page.
The data cleaner is used for executing cleaning operation on the source data after writing the converted data.
If the user selects 'yes' in the visual operation page, the data cleaner deletes the file in the source data path after the data is written; if the user selects no in the visual operation page, the data cleaner will not be started after the data is written, and the source data will be reserved.
Therefore, in the system, through the visual operation page, a user can easily fill in a source data path, a destination data format and whether source data are deleted or not, and the parameters filled in by the user can be transmitted to a data reader, a format converter, a data writer and a data remover, so that the data format conversion process can be completed, and finally, the data in the destination format is output, and the aim of converting the data in different formats into a unified format is fulfilled.
Reference to "first," "second," "third," "fourth," etc. (if present) herein is used to distinguish similar objects from each other and does not necessarily describe a particular order or sequence. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, or apparatus.
It should be noted that the description herein of "first," "second," etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be regarded as not exist and not within the protection scope of the present application.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of readable storage medium known in the art.
Specific examples are used herein to describe the origin and embodiments of the present application, the description of the examples above being merely for aiding in the understanding of the method of the present application and its core ideas; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (6)
1. A data format conversion method, applied to an HDFS distributed file system, comprising:
acquiring a source path of source data input by a user through a man-machine interaction interface, and a destination path and a destination format of destination data; the source path and the destination path are paths in an HDFS distributed file system;
reading the source data according to the source path and determining a source format of the source data;
calling a format conversion program matched with the source format and the destination format from a preset format converter to convert the source data from the source format to the destination format, and obtaining the destination data; the format converter stores therein a plurality of format conversion programs including: a format conversion program converting a binary data format into a text data format, a format conversion program converting a text data format into a binary data format;
storing the destination data according to the destination path;
the method for acquiring the source path of the source data input by the user through the man-machine interaction interface comprises the following steps:
acquiring source paths of a plurality of source data input by a user through a man-machine interaction interface, so as to simultaneously convert the source data corresponding to the source paths into a destination format, and storing the destination data obtained by conversion into different files under the same destination path or into the same file under the same destination path;
the method for acquiring the source paths of the plurality of source data input by the user through the man-machine interaction interface comprises the following steps:
acquiring an incomplete source path of source data input by a user through a man-machine interaction interface;
searching all complete source paths corresponding to the incomplete source paths by using a regular matching mode, and determining all complete source paths as the source paths;
wherein after the destination data is stored according to the destination path, the method further comprises:
acquiring a deleting instruction input by a user through a man-machine interaction interface, and deleting the source data according to the deleting instruction;
wherein, still include:
recording a format conversion log of the source data; record in the format conversion log: name of source data, source path, source format; name of destination data, destination path, destination format; converting the identification information of the format conversion program called by the procedure.
2. The data format conversion method according to claim 1, further comprising, after storing the destination data according to the destination path:
and storing the source path, the destination path and the destination format into a history parameter record table.
3. The data format conversion method according to claim 1, further comprising, after storing the destination data according to the destination path:
and calling a data reading program corresponding to the destination format to read the destination data.
4. A data format conversion device, applied to an HDFS distributed file system, comprising:
the acquisition module is used for acquiring a source path of source data, a destination path of destination data and a destination format of the destination data which are input by a user through a man-machine interaction interface; the source path and the destination path are paths in an HDFS distributed file system;
the reading module is used for reading the source data according to the source path and determining a source format of the source data;
the conversion module is used for calling a format conversion program matched with the source format and the destination format from a preset format converter to convert the source data from the source format into the destination format, so as to obtain the destination data; the format converter stores therein a plurality of format conversion programs including: a format conversion program converting a binary data format into a text data format, a format conversion program converting a text data format into a binary data format;
the storage module is used for storing the destination data according to the destination path;
the acquisition module is specifically configured to: acquiring source paths of a plurality of source data input by a user through a man-machine interaction interface, so as to simultaneously convert the source data corresponding to the source paths into a destination format, and storing the destination data obtained by conversion into different files under the same destination path or into the same file under the same destination path;
wherein, the acquisition module includes:
the acquisition unit is used for acquiring an incomplete source path of source data input by a user through a man-machine interaction interface;
the searching unit is used for searching all complete source paths corresponding to the incomplete source paths in a regular matching mode and determining all the complete source paths as source paths;
wherein, still include:
the deleting module is used for acquiring a deleting instruction input by a user through a man-machine interaction interface and deleting the source data according to the deleting instruction;
wherein, still include:
the recording module is used for recording the format conversion log of the source data; record in the format conversion log: name of source data, source path, source format; name of destination data, destination path, destination format; converting the identification information of the format conversion program called by the procedure.
5. A data format conversion apparatus, characterized by comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the data format conversion method as claimed in any one of claims 1 to 3.
6. A readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the data format conversion method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910712544.3A CN110515894B (en) | 2019-08-02 | 2019-08-02 | Data format conversion method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910712544.3A CN110515894B (en) | 2019-08-02 | 2019-08-02 | Data format conversion method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110515894A CN110515894A (en) | 2019-11-29 |
| CN110515894B true CN110515894B (en) | 2023-07-21 |
Family
ID=68624872
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910712544.3A Active CN110515894B (en) | 2019-08-02 | 2019-08-02 | Data format conversion method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110515894B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112185494B (en) * | 2020-09-03 | 2023-04-25 | 武汉联影医疗科技有限公司 | Data storage method, device, computer equipment and storage medium |
| CN112422565B (en) * | 2020-11-18 | 2023-02-03 | 青岛海尔科技有限公司 | Message processing method and device, storage medium and electronic device |
| CN113127226A (en) * | 2021-03-12 | 2021-07-16 | 创业慧康科技股份有限公司 | Method for generating data conversion model, data conversion method and device |
| CN113326681A (en) * | 2021-05-25 | 2021-08-31 | 上海微盟企业发展有限公司 | Data processing method, device, equipment and computer readable storage medium |
| CN114840597B (en) * | 2022-07-04 | 2023-03-14 | 杭州安恒信息技术股份有限公司 | Component parameter format conversion method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104793567A (en) * | 2014-01-16 | 2015-07-22 | 中国南方航空工业(集团)有限公司 | Numerical control machining program conversion method and numerical control machining program conversion device |
| CN105824718A (en) * | 2016-04-01 | 2016-08-03 | 北京大学 | Automatic repairing method and automatic repairing system for software configuration fault based on question and answer website knowledge |
| CN106663224A (en) * | 2014-06-30 | 2017-05-10 | 亚马逊科技公司 | Interactive interfaces for machine learning model evaluations |
| CN106933206A (en) * | 2015-10-09 | 2017-07-07 | 费希尔-罗斯蒙特系统公司 | The inquiry independently of source in distributed industrial systems |
| CN109413154A (en) * | 2018-09-26 | 2019-03-01 | 平安普惠企业管理有限公司 | Conversion method of data format, device, computer equipment and storage medium |
| CN109582551A (en) * | 2018-10-11 | 2019-04-05 | 平安科技（深圳）有限公司 | Daily record data analytic method, device, computer equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101169711A (en) * | 2006-10-27 | 2008-04-30 | 鸿富锦精密工业（深圳）有限公司 | Data conversion system and method |
| JP5693159B2 (en) * | 2010-11-05 | 2015-04-01 | キヤノン株式会社 | Image processing apparatus and control method thereof |
| US10386827B2 (en) * | 2013-03-04 | 2019-08-20 | Fisher-Rosemount Systems, Inc. | Distributed industrial performance monitoring and analytics platform |
| CN105468702B (en) * | 2015-11-18 | 2019-03-22 | 中国科学院计算机网络信息中心 | A kind of extensive RDF data associated path discovery method |
| CN106354843A (en) * | 2016-08-31 | 2017-01-25 | 虎扑（上海）文化传播股份有限公司 | Web crawler system and method |
| CN108021661A (en) * | 2017-12-04 | 2018-05-11 | 北京锐安科技有限公司 | A kind of conversion method of data format and system |
| CN108363737B (en) * | 2018-01-19 | 2022-04-12 | 深圳市宏电技术股份有限公司 | Data format conversion method, device and equipment |
-
2019
- 2019-08-02 CN CN201910712544.3A patent/CN110515894B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104793567A (en) * | 2014-01-16 | 2015-07-22 | 中国南方航空工业(集团)有限公司 | Numerical control machining program conversion method and numerical control machining program conversion device |
| CN106663224A (en) * | 2014-06-30 | 2017-05-10 | 亚马逊科技公司 | Interactive interfaces for machine learning model evaluations |
| CN106933206A (en) * | 2015-10-09 | 2017-07-07 | 费希尔-罗斯蒙特系统公司 | The inquiry independently of source in distributed industrial systems |
| CN105824718A (en) * | 2016-04-01 | 2016-08-03 | 北京大学 | Automatic repairing method and automatic repairing system for software configuration fault based on question and answer website knowledge |
| CN109413154A (en) * | 2018-09-26 | 2019-03-01 | 平安普惠企业管理有限公司 | Conversion method of data format, device, computer equipment and storage medium |
| CN109582551A (en) * | 2018-10-11 | 2019-04-05 | 平安科技（深圳）有限公司 | Daily record data analytic method, device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| An Optimal Solution of Storing and Processing Small Image Files on Hadoop;Lu Lu 等;《Procedia Computer Science》;第154卷;581-587 * |
| 图像去模糊系统研究;庄宇;《中国优秀硕士学位论文全文数据库 信息科技辑》(第05期);I138-1632 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110515894A (en) | 2019-11-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110515894B (en) | Data format conversion method, device, equipment and readable storage medium | |
| CN109739855B (en) | Method and system for realizing data sheet splicing and automatically training machine learning model | |
| CN110287696B (en) | Detection method, device and equipment for rebound shell process | |
| CN107967122B (en) | Data writing method, device and medium for block device | |
| JP6020291B2 (en) | Communication monitoring method, communication monitoring apparatus, communication monitoring program | |
| KR101935346B1 (en) | Conservative garbage collecting and tagged integers for memory management | |
| CN110659257B (en) | Metadata object repairing method, device, equipment and readable storage medium | |
| JP2021518021A (en) | Data processing methods, equipment and computer readable storage media | |
| CN104375934A (en) | Method for testing reliability of Android mobile phone software | |
| CN110008197B (en) | Data processing method and system, electronic equipment and storage medium | |
| CN115080398A (en) | Automatic interface test system and method | |
| CN108874441B (en) | Board card configuration method, device, server and storage medium | |
| CN111125008B (en) | Dynamic repairing method and system of abnormal inode and related components | |
| CN109101412B (en) | Test file generation method, test file generation device, test file testing method, test file testing device, storage medium and computer equipment | |
| CN110968296B (en) | Data acquisition method, device, equipment and readable storage medium | |
| CN108829342B (en) | Log storage method, system and storage device | |
| CN112000971B (en) | File permission recording method, system and related device | |
| US9965491B2 (en) | Method and device for recording system log | |
| CN110941605B (en) | Method and device for deleting repeated data on line and readable storage medium | |
| CN105205062A (en) | Data storage method and data reading method and device | |
| WO2022253251A1 (en) | Method and apparatus for evaluating interaction performance of interaction system | |
| CN110750496A (en) | File copying method, system, equipment and computer readable storage medium | |
| CN109918346A (en) | A kind of creation method and device of file | |
| CN109039782A (en) | A kind of cluster log recording method and relevant apparatus | |
| CN111400173B (en) | VTS test method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |