CN114281876A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114281876A CN114281876A CN202111598329.9A CN202111598329A CN114281876A CN 114281876 A CN114281876 A CN 114281876A CN 202111598329 A CN202111598329 A CN 202111598329A CN 114281876 A CN114281876 A CN 114281876A
- Authority
- CN
- China
- Prior art keywords
- data
- standard
- information
- platform
- format
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring original data generated by a source data platform and standard data specifications of a target data platform; determining storage structure information of the original data according to the identification information of the source data platform; according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform; through the technical scheme, the problem that the multi-source heterogeneous data access platform cannot process is solved, and the workload during data access is reduced.
Description
Technical Field
The embodiment of the application relates to the technical field of data processing, in particular to a data processing method, a data processing device, data processing equipment and a storage medium.
Background
Because each data platform may adopt different data storage formats for data storage, the existence of multi-source heterogeneous data is caused. Due to the difference of data sources, data structures or data standards, the multi-source heterogeneous data has great differences in types, properties, forms, contents and the like, which easily causes data information sharing and interconnection obstacles.
In the prior art, the traditional processing method of multi-source heterogeneous data is to modify data adapted and converged by codes at an accessed data platform, the codes need to be rewritten each time the structure of data of other platforms is changed, and each code needs to be coded by investing special manpower to do repeated and complicated development work, so that the workload is overlarge.
Therefore, there is a need for improvement in view of the problems in the prior art.
Disclosure of Invention
The application provides a data processing method, a data processing device, data processing equipment and a storage medium, which are used for solving the problem that a multi-source heterogeneous data access platform cannot process data and reducing the workload during data access.
In a first aspect, an embodiment of the present application provides a data processing method, where the method includes:
acquiring original data generated by a source data platform and standard data specifications of a target data platform;
determining storage structure information of the original data according to the identification information of the source data platform;
and according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform.
In a second aspect, an embodiment of the present application further provides a data processing apparatus, where the apparatus includes:
the data acquisition module is used for acquiring original data generated by the source data platform and standard data specifications of the target data platform;
the structure information determining module is used for determining the storage structure information of the original data according to the identification information of the source data platform;
and the data conversion module is used for carrying out format conversion on the original data according to the standard data specification and the storage structure information to obtain standard data required by the target data platform.
In a third aspect, an embodiment of the present application further provides an electronic device, where the device includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data processing methods provided in the embodiments of the first aspect.
In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements any one of the data processing methods provided in the embodiments of the first aspect.
The method comprises the steps of acquiring original data generated by a source data platform and standard data specifications of a target data platform; determining storage structure information of the original data according to the identification information of the source data platform; and according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform. By the technical scheme, format conversion is carried out on original data according to standard data specifications and storage structure information, the problem that multi-source heterogeneous data can not be processed when accessing a target data platform is solved, meanwhile, due to the adoption of a general configuration data processing scheme, when the data structure of a source data platform or the standard data specifications of the target data platform are changed, only configuration files need to be modified, the data format is configured in advance, the workload during data access is reduced, the working efficiency is improved, in addition, because customized development is not needed to be carried out on each data type, and the expansibility is stronger at a code level.
Drawings
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a flowchart of a data processing method according to a second embodiment of the present application;
fig. 3 is a flowchart of a data processing method according to a third embodiment of the present application;
fig. 4 is a schematic diagram of a data processing apparatus according to a fourth embodiment of the present application;
fig. 5 is a schematic view of an electronic device provided in this application embodiment five.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application. The embodiment of the application can be suitable for the case of carrying out lattice conversion on the source data according to the standard format specification. The method can be executed by a data processing device, which can be implemented by software and/or hardware, and is specifically configured in an electronic device, which can be a mobile terminal or a fixed terminal.
Referring to fig. 1, a data processing method provided in an embodiment of the present application includes:
s110, acquiring original data generated by a source data platform and standard data specifications of a target data platform.
The source data platform can transmit original data to the target data platform, and the target data platform can perform subsequent logic service processing. In this embodiment, the number of source data platforms may be at least one or at least two. Typically, the number of source data platforms may be multiple, such as 10.
Alternatively, the raw data generated by the source data platform may include raw data of text type, picture type, video type, and sound type, among others. The raw data generated by different source data platforms is different for the target data platform.
For example, it is assumed that input data of a target data platform is derived from raw data aggregated by 10 source data platforms, and the 10 source data platforms aggregate the raw data to the target data platform according to respective templates, so that the target data platform needs to perform access processing on the raw data, and a data format aggregated by each source data platform is inconsistent with a data format that can be processed by the target data platform. However, the above method is too heavy to implement and difficult for later expansion.
Therefore, by adopting the data processing method provided by the embodiment of the application, the standard data specification of the access data of the target data platform is formulated, and the unified data format conversion is performed on the original data according to the processing format requirement of the target data platform, so that the target data platform can process the converged original data according to the standard during processing, even if the original data of other source data platforms are converged at the later stage, the customized development work is not needed, only the configuration file needs to be modified, and the workload is greatly reduced.
In this embodiment, the standard data specification is used to describe a data format specification of the target data platform for processing data, and when the data format specification of the target data platform for processing data changes, the standard data specification may be modified correspondingly.
And S120, determining the storage structure information of the original data according to the identification information of the source data platform.
The identification information may be a platform name, a platform website address, other platform identifications, and the like, and is used for distinguishing the source data platform.
Specifically, the storage structure information of the original data generated by each source data platform may be predetermined, and the identification information of each source data platform and the corresponding storage structure information may be stored in association, so that when the source data platform generates the original data and sends the original data to the target data platform, the identification information of the source data platform may be obtained, and the storage structure information of the original data may be determined according to the identification information of the source data platform.
The storage structure information is used for explaining information stored in the original data, for example, the original data can store information in a key-value pair manner, and different positions can correspondingly store different key-value pair information.
For example, assuming that there are two source data platforms a and B, a and B may provide data to the target data platform at the same time, the raw data generated by a and B are D1 and D2, respectively, where D1 and D2 store data in the form of key-value pairs, the first key-value-pair location of D1 stores critical data information, the second key-value-pair location stores additional data information, and the first key-value-pair location of D2 stores additional data information, the second key-value-pair location stores critical data information. Obviously, the storage structure information of D1 and D2 is different.
And S130, converting the format of the original data according to the standard data specification and the storage structure information to obtain standard data required by the target data platform.
In this embodiment, the original data may be subjected to lattice transformation based on a data lattice transformation program developed in a componentization manner, and specifically, the data lattice transformation program performs the format transformation on the original data according to the standard data specification and the storage structure information to obtain standard data, and forwards the standard data to the target data platform.
Typically, the original data is text-type file data, and the standard data is picture-type file data. For example, the original data may be txt (a text format) type file data provided by a manufacturer, and the standard data may be jpg (a picture format) type file data.
It can be understood that, in different service scenarios, corresponding original data and standard data are different, and may be specifically set according to actual requirements and use conditions.
The method comprises the steps of acquiring original data generated by a source data platform and standard data specifications of a target data platform; determining storage structure information of the original data according to the identification information of the source data platform; and according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform. By the technical scheme, format conversion is carried out on original data according to standard data specifications and storage structure information, the problem that multi-source heterogeneous data can not be processed when accessing a target data platform is solved, meanwhile, due to the adoption of a general configuration data processing scheme, when the data structure of a source data platform or the standard data specifications of the target data platform are changed, only configuration files need to be modified, the data format is configured in advance, the workload during data access is reduced, the working efficiency is improved, in addition, because customized development is not needed to be carried out on each data type, and the expansibility is stronger at a code level.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present application, where the present embodiment is an optimization of the foregoing scheme based on the foregoing embodiment.
Further, the standard data specification comprises a standard format specification; correspondingly, the operation of converting the format of the original data according to the standard data specification and the storage structure information to obtain the standard data required by the target data platform is refined into the operation of determining the standard format of data storage according to the standard format specification; extracting key data information from the original data according to the storage structure information; and according to the key data information and the standard format, carrying out format conversion on the key data information according to the standard format to obtain standard data required by the target data platform so as to complete the determination process of the standard data.
Wherein explanations of the same or corresponding terms as those of the above-described embodiments are omitted.
Referring to fig. 2, the data processing method provided in this embodiment includes:
s210, acquiring original data generated by a source data platform and standard data specifications of a target data platform.
S220, determining the storage structure information of the original data according to the identification information of the source data platform.
And S230, determining a standard format of data storage according to a standard format specification in the standard data specification.
The standard format specification is used for explaining the format specification of standard data.
In this embodiment, the standard format of the data storage may be a picture format such as jpg or png.
And S240, extracting key data information from the original data according to the storage structure information.
In this embodiment, the key data information may correspond to a special unique identifier, and after determining the storage structure information of the original data, the original data may be analyzed according to the unique identifier, and the key data information is extracted from the original data.
For example, the key data information may be portrait data information, which may be stored in a string or binary form.
And S250, converting the format of the key data information according to the standard format according to the key data information and the standard format to obtain standard data required by the target data platform.
Continuing with the above example, the key data information is portrait data information, and taking the standard format as jpg format as an example, the portrait data information can be converted into a corresponding face picture in a format conversion manner.
Optionally, when determining the storage structure information of the original data according to the identification information of the source data platform, the method further includes: determining storage additional information of the original data, wherein the storage additional information comprises a preset encryption mode of the key data information; correspondingly, the performing format conversion on the key data information according to the key data information and the standard format to obtain the standard data required by the target data platform includes: decrypting the key data information according to the preset encryption mode to obtain decrypted key data information; and according to the decrypted key data information and the standard format, carrying out format conversion on the decrypted key data information according to the standard format to obtain standard data required by the target data platform.
The storage additional information can be generally defaulted, and data processing is carried out according to the defaulted storage additional information; when the data storage mode of the source data platform is changed, the source data platform can send the storage additional information to the target data platform in an active sending mode so as to update the storage additional information.
In this embodiment, the preset encryption mode may be set according to actual requirements and use conditions. For example, the predetermined encryption scheme may be the Base64 encryption scheme, in which the Base64 is a method for converting data of non-ASCII (American Standard Code for Information exchange) characters into ASCII characters.
It can be understood that, in order to accurately perform format conversion, the key data information may be decoded based on a preset encryption manner of the key data information, and the conversion may be performed based on the decoded key data information, so as to ensure the accuracy of data processing.
On the basis of the embodiment, the embodiment of the application perfects the determination process of standard data, and the standard data specification comprises a standard format specification; correspondingly, the converting the format of the original data according to the standard data specification and the storage structure information to obtain the standard data required by the target data platform includes: determining a standard format of data storage according to the standard format specification; extracting key data information from the original data according to the storage structure information; according to the key data information and the standard format, format conversion is carried out on the key data information according to the standard format to obtain standard data required by the target data platform.
EXAMPLE III
Fig. 3 is a flowchart of a data processing method provided in the third embodiment of the present application, and this embodiment is an optimization of the above-mentioned scheme based on the above-mentioned embodiment.
Further, the standard data specification further comprises a standard naming specification; correspondingly, after the operation of performing format conversion on the key data information according to the key data information and the standard format, adding the operation of determining a naming rule and at least one standard field required by naming according to the standard naming specification; according to the storage structure information, obtaining a standard field value corresponding to each standard field from the original data; combining the standard field values according to the naming rule to obtain a new data name; and obtaining standard data required by the target data platform according to the new data name and the key data information after format conversion so as to further perfect the determination process of the standard data.
Wherein explanations of the same or corresponding terms as those of the above-described embodiments are omitted.
Referring to fig. 3, the data processing method provided in this embodiment includes:
s310, acquiring original data generated by a source data platform and standard data specifications of a target data platform.
And S320, determining the storage structure information of the original data according to the identification information of the source data platform.
S330, determining a standard format of data storage according to a standard format specification in the standard data specification.
And S340, extracting key data information from the original data according to the storage structure information.
And S350, converting the format of the key data information according to the standard format according to the key data information and the standard format.
S360, determining a naming rule and at least one standard field required by naming according to a standard naming specification in the standard data specification.
The standard naming specification is used for explaining the naming specification of the standard data.
In this embodiment, the naming rule refers to a rule for naming standard data. For example, the naming of the standard data may be composed of at least one standard field and a preset default field, which are connected by a special character.
Optionally, the criteria field includes at least one of a timestamp, location information, and data source identification.
It is understood that, in order to describe the content of the standard data, the standard data may be named by using a specific standard field, so that the naming of the standard data is more meaningful and the readability of the standard data can be improved.
And S370, acquiring standard field values corresponding to the standard fields from the original data according to the storage structure information.
Specifically, the original data may be analyzed according to the storage structure information, and the standard field value corresponding to each standard field may be obtained from the original data, for example, the standard field value corresponding to each standard field may be determined in a program traversal manner. For example, the timestamp may be a standard field, the corresponding standard field value may be 1664563653, and the correspondence between the standard field and the standard field value may be represented by "timestamp (1664563653)".
And S380, combining the standard field values according to the naming rule to obtain a new data name.
Take the new DATA name DATA _ W4101020023_1664563653_113.435675_34.235463_ F00250023 as an example, wherein "DATA" belongs to a preset default field, fixed; "_" belongs to a special character; "W4101020023" is a standard field value corresponding to the standard field of the device coding, "F00250023" is a standard field value corresponding to the standard field of the unique identification corresponding to the device; "1664563653" is the standard field value corresponding to the standard field of decimal time stamp; "113.435675" is the standard field value corresponding to the longitude standard field, and "34.235463" is the standard field value corresponding to the latitude standard field.
And S390, obtaining standard data required by the target data platform according to the new data name and the key data information after format conversion.
For example, with the DATA processing scheme provided by the present embodiment, the original DATA a.txt can be converted into the standard DATA _ W4101020023_1664563653_113.435675_34.235463_ f00250023. jpg.
Optionally, when determining the storage structure information of the original data according to the identification information of the source data platform, the method further includes: determining storage additional information of the original data, wherein the storage additional information comprises a preset memory storage mode of the standard field value; correspondingly, after obtaining the standard field value corresponding to the standard field from the original data, the method further includes: and updating the standard field value based on a standard storage specification in the standard data specifications according to the preset memory storage mode and the standard field value.
In this embodiment, when naming the standard data, a default memory storage manner is adopted, for example, a decimal memory storage manner is used, however, when extracting the standard field value corresponding to the standard field from the original data, the extracted standard field value may not be decimal, but may be binary or octal, and therefore, in order to ensure the accuracy of obtaining the standard field value, the standard field value needs to be updated based on the preset memory storage manner corresponding to the standard field value.
It should be noted that the memory storage modes of the standard field values corresponding to different standard fields may be the same or different, and may be specifically determined according to the actual storage situation.
On the basis of the embodiment, the embodiment of the application further perfects the determination process of the standard data, and the standard data specification also comprises a standard naming specification; correspondingly, after the format conversion is performed on the key data information according to the key data information and the standard format, the method further includes: determining a naming rule and at least one standard field required for naming according to the standard naming specification; according to the storage structure information, obtaining a standard field value corresponding to each standard field from the original data; combining the standard field values according to the naming rule to obtain a new data name; according to the new data name and the key data information after format conversion, standard data required by the target data platform are obtained, and through the technical scheme, the original data are subjected to data format conversion according to the processing format requirement of the target data platform, so that the format and file name of the data can be unified, and the target data platform can perform subsequent data processing conveniently.
Example four
Fig. 4 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present application. Referring to fig. 4, a data processing apparatus provided in an embodiment of the present application includes: a data acquisition module 410, a structure information determination module 420, and a data conversion module 430.
A data obtaining module 410, configured to obtain original data generated by a source data platform and a standard data specification of a target data platform;
a structure information determining module 420, configured to determine storage structure information of the original data according to the identification information of the source data platform;
and the data conversion module 430 is configured to perform format conversion on the original data according to the standard data specification and the storage structure information to obtain standard data required by the target data platform.
The method comprises the steps of acquiring original data generated by a source data platform and standard data specifications of a target data platform; determining storage structure information of the original data according to the identification information of the source data platform; and according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform. By the technical scheme, format conversion is carried out on original data according to standard data specifications and storage structure information, the problem that multi-source heterogeneous data can not be processed when accessing a target data platform is solved, meanwhile, due to the adoption of a general configuration data processing scheme, when the data structure of a source data platform or the standard data specifications of the target data platform are changed, only configuration files need to be modified, the data format is configured in advance, the workload during data access is reduced, the working efficiency is improved, in addition, because customized development is not needed to be carried out on each data type, and the expansibility is stronger at a code level.
Further, the standard data specification comprises a standard format specification;
accordingly, the data conversion module 430 includes:
the standard format determining unit is used for determining the standard format of data storage according to the standard format specification;
the key data extraction unit is used for extracting key data information from the original data according to the storage structure information;
and the format conversion unit is used for carrying out format conversion on the key data information according to the standard format according to the key data information and the standard format to obtain standard data required by the target data platform.
Further, the standard data specification further comprises a standard naming specification;
accordingly, the data conversion module 430 includes:
the standard field determining unit is used for determining a naming rule and at least one standard field required by naming according to the standard naming specification;
a standard field value extracting unit, configured to obtain a standard field value corresponding to each standard field from the original data according to the storage structure information;
the data name determining unit is used for combining the standard field values according to the naming rule to obtain a new data name;
and the standard data determining unit is used for obtaining the standard data required by the target data platform according to the new data name and the key data information after format conversion.
Further, the criteria field includes at least one of a timestamp, location information, and a data source identification.
Further, the apparatus further comprises:
an encryption mode determining unit, configured to determine storage additional information of the original data when determining storage structure information of the original data according to the identification information of the source data platform, where the storage additional information includes a preset encryption mode of the key data information;
correspondingly, the format conversion unit comprises:
the decryption subunit is configured to decrypt the key data information according to the preset encryption manner to obtain decrypted key data information;
and the format conversion unit is used for performing format conversion on the decrypted key data information according to the standard format according to the decrypted key data information and the standard format to obtain standard data required by the target data platform.
Further, the apparatus further comprises:
a storage mode determining unit, configured to determine storage additional information of the original data when determining storage structure information of the original data according to the identification information of the source data platform, where the storage additional information includes a preset memory storage mode of the standard field value;
correspondingly, the standard data determination unit includes:
and the updating subunit is used for updating the standard field value based on a standard storage specification in the standard data specification according to the preset memory storage mode and the standard field value.
Further, the original data is file data of a text type, and the standard data is file data of a picture type.
The data processing device provided by the embodiment of the application can execute the data processing method provided by any embodiment of the application, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 5 is a structural diagram of an electronic device according to a fifth embodiment of the present application. FIG. 5 illustrates a block diagram of an exemplary electronic device 512 suitable for use in implementing embodiments of the present application. The electronic device 512 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 5, electronic device 512 is in the form of a general purpose computing device. Components of the electronic device 512 may include, but are not limited to: one or more processors or processing units 516, a system memory 528, and a bus 518 that couples the various system components including the system memory 528 and the processing unit 516.
The system memory 528 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)530 and/or cache memory 532. The electronic device 512 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 534 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 518 through one or more data media interfaces. System memory 528 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.
A program/utility 540 having a set (at least one) of program modules 542 may be stored, for example, in system memory 528, such program modules 542 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 542 generally perform the functions and/or methods of the embodiments described herein.
The electronic device 512 may also communicate with one or more external devices 514 (e.g., keyboard, pointing device, display 524, etc.), with one or more devices that enable a user to interact with the electronic device 512, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 512 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 522. Also, the electronic device 512 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 520. As shown, the network adapter 520 communicates with the other modules of the electronic device 512 via the bus 518. It should be appreciated that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with the electronic device 512, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 516 executes various functional applications and data processing by executing at least one of other programs stored in the system memory 528, for example, to implement any one of the data processing methods provided in the embodiments of the present application.
EXAMPLE six
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data processing method provided in any embodiment of the present application, and the method includes: acquiring original data generated by a source data platform and standard data specifications of a target data platform; determining storage structure information of the original data according to the identification information of the source data platform; and according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform.
From the above description of the embodiments, it is obvious for those skilled in the art that the present application can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods described in the embodiments of the present application.
It should be noted that, in the embodiment of the data processing apparatus, the included units and modules are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (10)
1. A data processing method, comprising:
acquiring original data generated by a source data platform and standard data specifications of a target data platform;
determining storage structure information of the original data according to the identification information of the source data platform;
and according to the standard data specification and the storage structure information, carrying out format conversion on the original data to obtain standard data required by the target data platform.
2. The method of claim 1, wherein the standard data specification comprises a standard format specification;
correspondingly, the converting the format of the original data according to the standard data specification and the storage structure information to obtain the standard data required by the target data platform includes:
determining a standard format of data storage according to the standard format specification;
extracting key data information from the original data according to the storage structure information;
and according to the key data information and the standard format, carrying out format conversion on the key data information according to the standard format to obtain standard data required by the target data platform.
3. The method of claim 2, wherein the standard data specification further comprises a standard naming specification;
correspondingly, after the format conversion is performed on the key data information according to the key data information and the standard format, the method further includes:
determining a naming rule and at least one standard field required for naming according to the standard naming specification;
according to the storage structure information, obtaining a standard field value corresponding to each standard field from the original data;
combining the standard field values according to the naming rule to obtain a new data name;
and obtaining standard data required by the target data platform according to the new data name and the key data information after format conversion.
4. The method of claim 3, wherein the criteria field comprises at least one of a timestamp, location information, and a data source identification.
5. The method of claim 2, wherein when determining the storage structure information of the original data according to the identification information of the source data platform, the method further comprises:
determining storage additional information of the original data, wherein the storage additional information comprises a preset encryption mode of the key data information;
correspondingly, the performing format conversion on the key data information according to the key data information and the standard format to obtain the standard data required by the target data platform includes:
decrypting the key data information according to the preset encryption mode to obtain decrypted key data information;
and according to the decrypted key data information and the standard format, carrying out format conversion on the decrypted key data information according to the standard format to obtain standard data required by the target data platform.
6. The method of claim 3, wherein when determining the storage structure information of the original data according to the identification information of the source data platform, the method further comprises:
determining storage additional information of the original data, wherein the storage additional information comprises a preset memory storage mode of the standard field value;
correspondingly, after obtaining the standard field value corresponding to the standard field from the original data, the method further includes:
and updating the standard field value based on a standard storage specification in the standard data specifications according to the preset memory storage mode and the standard field value.
7. The method according to any one of claims 1 to 6, wherein the original data is a text-type file data and the standard data is a picture-type file data.
8. A data processing apparatus, characterized in that the apparatus comprises:
the data acquisition module is used for acquiring original data generated by the source data platform and standard data specifications of the target data platform;
the structure information determining module is used for determining the storage structure information of the original data according to the identification information of the source data platform;
and the data conversion module is used for carrying out format conversion on the original data according to the standard data specification and the storage structure information to obtain standard data required by the target data platform.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111598329.9A CN114281876A (en) | 2021-12-24 | 2021-12-24 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111598329.9A CN114281876A (en) | 2021-12-24 | 2021-12-24 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114281876A true CN114281876A (en) | 2022-04-05 |
Family
ID=80874886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111598329.9A Pending CN114281876A (en) | 2021-12-24 | 2021-12-24 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114281876A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115314566A (en) * | 2022-07-26 | 2022-11-08 | 北京金山云网络技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN115952229A (en) * | 2023-03-10 | 2023-04-11 | 北京全路通信信号研究设计院集团有限公司 | Method, device and equipment for generating track configuration data and storage medium |
-
2021
- 2021-12-24 CN CN202111598329.9A patent/CN114281876A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115314566A (en) * | 2022-07-26 | 2022-11-08 | 北京金山云网络技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN115952229A (en) * | 2023-03-10 | 2023-04-11 | 北京全路通信信号研究设计院集团有限公司 | Method, device and equipment for generating track configuration data and storage medium |
| CN115952229B (en) * | 2023-03-10 | 2023-08-11 | 北京全路通信信号研究设计院集团有限公司 | Track configuration data generation method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111090628B (en) | Data processing method and device, storage medium and electronic equipment | |
| CN114281876A (en) | Data processing method, device, equipment and storage medium | |
| WO2017028514A1 (en) | Method and device for storing and reading data | |
| CN110263277B (en) | Page data display method, page data updating device, page data equipment and storage medium | |
| WO2019134340A1 (en) | Salary calculation method, application server, and computer readable storage medium | |
| CN108921552B (en) | Evidence verification method and device | |
| CN111614757A (en) | Message interaction method, device, equipment and medium for client and server | |
| US11449352B2 (en) | Systems and methods for converting record formats | |
| WO2017097159A1 (en) | Method and apparatus for generating random character string | |
| CN111694866A (en) | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium | |
| CN111782614B (en) | Data access method, device, equipment and storage medium | |
| CN111046085B (en) | Data tracing processing method and device, medium and equipment | |
| CN107992457B (en) | Information conversion method, device, terminal equipment and storage medium | |
| CN113919310A (en) | Short message content determination method and device, electronic equipment and storage medium | |
| CN113806556A (en) | Method, device, equipment and medium for constructing knowledge graph based on power grid data | |
| CN113569289A (en) | Data desensitization method and device | |
| CN114265966A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112364268A (en) | Resource acquisition method and device, electronic equipment and storage medium | |
| CN114189331B (en) | Key storage and reading method, device, equipment and storage medium | |
| CN111324587A (en) | Compression method, analysis method and system for data of OBJ file | |
| CN111435452A (en) | Model training method, device, equipment and medium | |
| CN113806416B (en) | Method and device for realizing real-time data service and electronic equipment | |
| CN113672880B (en) | File leakage prevention method and device, electronic equipment and storage medium | |
| CN113627938B (en) | Data deleting method, device and equipment of block chain and storage medium | |
| CN111831620B (en) | Method, apparatus and computer program product for storage management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |