Summary of the invention
Fundamental purpose of the present invention is to provide a kind of data transformations information processing method and device, searches the low problem of data transformations information efficiency that is successful and data transformations failure to solve in correlation technique.
To achieve these goals, according to an aspect of the present invention, a kind of data transformations information processing method is provided.
Data transformations information processing method according to the present invention comprises: obtain data to be transformed; Treat conversion data and perform conversion processing, obtain conversion results; Conversion successful data in data to be transformed and data relay miss data to be transformed is obtained according to conversion results, wherein, transform successful data and refer to that in data to be transformed, successful conversion is the data of target data, transform miss data and refer to the unsuccessful data being converted into target data in data to be transformed; And the conversion successful data in data to be transformed is stored to first object memory location, and the conversion miss data in data to be transformed is stored to the second target storage position.
Further, obtain data to be transformed to comprise: from presetting database, obtain source data; Source data is stored to the 3rd target storage position; And obtain data to be transformed from the 3rd target storage position.
Further, treat conversion data execution conversion processing to comprise: determine default conversion types; Be target data according to default conversion types by data transformations to be transformed, wherein, the type of target data is for presetting conversion types; Set up the corresponding relation of target data and data to be transformed; And export the corresponding relation of target data and data to be transformed.
Further, the conversion successful data in data to be transformed is stored to first object memory location, and the conversion miss data in data to be transformed is stored to the second target storage position comprises: obtain preset data file layout; And be stored to first object memory location according to the conversion successful data in preset data file layout data just to be transformed, and according to preset data file layout, the conversion miss data in data to be transformed is stored to the second target storage position.
Further, after the conversion successful data obtained according to conversion results in data to be transformed and data relay miss data to be transformed, the method also comprises: transform failed type according to transforming successful data and transforming miss data parsing conversion miss data; And export the failed type of conversion miss data conversion.
To achieve these goals, according to a further aspect in the invention, a kind of data transformations signal conditioning package is provided.
Data transformations signal conditioning package according to the present invention comprises: the first acquiring unit, for obtaining data to be transformed; Conversion unit, performing conversion processing for treating conversion data, obtaining conversion results; Second acquisition unit, for obtaining conversion successful data in data to be transformed and data relay miss data to be transformed according to conversion results, wherein, transform successful data and refer to that in data to be transformed, successful conversion is the data of target data, transform miss data and refer to the unsuccessful data being converted into target data in data to be transformed; And storage unit, for the conversion successful data in data to be transformed is stored to first object memory location, and the conversion miss data in data to be transformed is stored to the second target storage position.
Further, the first acquiring unit comprises: the first acquisition module, for obtaining source data from presetting database; First memory module, for being stored to the 3rd target storage position by source data; And second acquisition module, for obtaining data to be transformed from the 3rd target storage position.
Further, conversion unit comprises: determination module, for determining default conversion types; Conversion module, for being target data according to presetting conversion types by data transformations to be transformed, wherein, the type of target data is for presetting conversion types; Set up module, for setting up the corresponding relation of target data and data to be transformed; And output module, for exporting the corresponding relation of target data and data to be transformed.
Further, storage unit comprises: the 3rd acquisition module, for obtaining preset data file layout; And second memory module, for being stored to first object memory location according to the conversion successful data in preset data file layout data just to be transformed, and according to preset data file layout, the conversion miss data in data to be transformed is stored to the second target storage position.
Further, this device also comprises: resolution unit, for transforming failed type according to transforming successful data and transforming miss data parsing conversion miss data; And output unit, transform the failed type of miss data conversion for exporting.
Pass through the present invention, adopt and automatically store conversion successful data in data to be transformed and data relay miss data to be transformed respectively, solve in correlation technique and search the low problem of data transformations information efficiency that is successful and data transformations failure, and then reach the effect improving the information efficiency searching data transformations success and data transformations failure.
Embodiment
It should be noted that, when not conflicting, the embodiment in the application and the feature in embodiment can combine mutually.Below with reference to the accompanying drawings and describe the present invention in detail in conjunction with the embodiments.
The application's scheme is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present application, technical scheme in the embodiment of the present application is clearly and completely described, obviously, described embodiment is only the embodiment of the application's part, instead of whole embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not making the every other embodiment obtained under creative work prerequisite, all should belong to the scope of the application's protection.
It should be noted that, term " first ", " second " etc. in the instructions of the application and claims and above-mentioned accompanying drawing are for distinguishing similar object, and need not be used for describing specific order or precedence.Should be appreciated that the data used like this can be exchanged, in the appropriate case so that the embodiment of the application described herein.In addition, term " comprises " and " having " and their any distortion, intention is to cover not exclusive comprising, such as, contain those steps or unit that the process of series of steps or unit, method, system, product or equipment is not necessarily limited to clearly list, but can comprise clearly do not list or for intrinsic other step of these processes, method, product or equipment or unit.
According to the embodiment of the present invention, provide a kind of data transformations information processing method.Fig. 1 is the process flow diagram of the data transformations information processing method according to the embodiment of the present invention.
As shown in Figure 1, this data transformations information processing method comprises following step S102 to step S108:
Step S102: obtain data to be transformed.
Data transformations refers to and raw data is converted into target data according to preset rules, and such as, data type conversion, Data Integration, data mining etc., the embodiment of the present invention is described to carry out data transformations based on SSIS platform.Data to be transformed refer to the data needing to perform conversion processing, and can be the data obtained from database, also can be the data obtained from other electrical forms, such as, and the data obtained from Excel form.Preferably, obtain data to be transformed to comprise: from presetting database, obtain source data; Source data is stored to the 3rd target storage position; And obtain data to be transformed from the 3rd target storage position.
Presetting database is data source data storehouse, particularly, can filter out the data needing to transform, i.e. source data by SQL statement from presetting database.In data transformations process, whether loss of data is there is in order to detect, the source data acquired can be carried out a dump, such as, be stored in the 3rd target storage position, thus, after completing data transformations, data after conversion and this source data can be compared to judge whether data transformations process exists loss of data, such as, before data transformations, source data amount is 100, transforming successful data volume after data transformations is 85, and transforming miss data amount is 10, then can obtain transforming obliterated data amount is 5.In dump procedure, occur that loss of data causes the above results inaccurate, the embodiment of the present invention obtains data to be transformed from the 3rd target storage position.
Step S104: treat conversion data and perform conversion processing, obtain conversion results.
SSIS platform can realize data transformations easily, and the embodiment of the present invention is described to adopt SSIS platform to carry out data transformations.Particularly, treat conversion data execution conversion processing to comprise: determine default conversion types; Be target data according to default conversion types by data transformations to be transformed, wherein, the type of target data is for presetting conversion types; Set up the corresponding relation of target data and data to be transformed; And export the corresponding relation of target data and data to be transformed.
Data transformations comprises polytype, and such as, conversion in type, Data Integration etc., default conversion types can be any one data transformations type.When adopting SSIS platform to carry out data transformations, user can pre-set data transformations type, when data transformations task start, SSIS platform can perform corresponding data conversion operation according to presetting conversion types automatically, obtain target data, analyze data transformations result further for the ease of user, set up the corresponding relation of target data and data to be transformed.Particularly, by corresponding with the target data that its conversion obtains for the data of successful conversion in data to be transformed and export, or be stored in further in target database.
Step S106: obtain conversion successful data in data to be transformed and data relay miss data to be transformed according to conversion results, wherein, transform successful data and refer to that in data to be transformed, successful conversion is the data of target data, transform miss data and refer to the unsuccessful data being converted into target data in data to be transformed.
After treating conversion data execution conversion processing, obtain the conversion successful data in data to be transformed and data relay miss data to be transformed, such as, different marks can be added for data relay successful data to be transformed and data relay miss data to be transformed in data transformations process, according to identifying the conversion successful data and data relay miss data to be transformed distinguished in data to be transformed, also in conversion process, the conversion successful data in data to be transformed can be taken out the memory headroom being stored in other positions from data to be transformed, thus conversion successful data in data to be transformed and data relay miss data to be transformed can be distinguished according to the position of content space.
Step S108: the conversion successful data in data to be transformed is stored to first object memory location, and the conversion miss data in data to be transformed is stored to the second target storage position.
First object memory location and the second target storage position can be any two not identical memory locations.The embodiment of the present invention is by obtaining the conversion successful data in data conversion process in data to be transformed and the conversion miss data in data to be transformed, and the conversion successful data in data to be transformed and the conversion miss data in data to be transformed are stored in different memory location respectively, realize automatically recording in data transformations process and change into merit data and transform miss data, thus can obtain easily and fast and check conversion successful data and transform miss data, solve and search the low problem of data transformations information efficiency that is successful and data transformations failure.In addition, according to transforming successful data and transforming the defect that miss data can analyze current data conversion quickly and accurately, increase work efficiency, reduce error rate.
Preferably, the conversion successful data in data to be transformed is stored to first object memory location, and the conversion miss data in data to be transformed is stored to the second target storage position comprises: obtain preset data file layout; And be stored to first object memory location according to the conversion successful data in preset data file layout data just to be transformed, and according to preset data file layout, the conversion miss data in data to be transformed is stored to the second target storage position.
Preset data file layout refers to the file layout that data store, and such as, can be TXT file layout, can be Excel form, also can be alternative document form.After the file layout that setting data stores, by the conversion successful data in data to be transformed and the conversion miss data in data to be transformed according to the stored in file format pre-set, such as, be stored as Excel form.The file layout that the embodiment of the present invention can select data to store flexibly, is convenient to the follow-up use treating the conversion successful data in conversion data and the conversion miss data in data to be transformed of user.
Preferably, after the conversion successful data obtained according to conversion results in data to be transformed and data relay miss data to be transformed, method also comprises: transform failed type according to transforming successful data and transforming miss data parsing conversion miss data; And export the failed type of conversion miss data conversion.
Data transformations unsuccessfully may cause due to reasons such as character length, character types, character sequences.Particularly, the type of multiple rule to data transformations failure can be pre-set detect.Such as, character length, character types, character sequence is detected successively.Be described for character length, first the character length all transforming successful data is detected, obtain maximum character length and minimum character length, traversal transforms miss data again, judge whether the character that each transforms miss data is between maximum character length and minimum character length respectively, particularly, a threshold value can be set, if transform character length in miss data and be not in by calculating the ratio that data volume between maximum character length and minimum character length and character length are in the data volume between maximum character length and minimum character length, if this ratio is greater than preset value, determine that the type of this data transformations failure is that character length is inconsistent, otherwise continue other and transform failed type detection, such as, detect character types, if detect the character string being letter formation all transforming successful data, and detect that transforming miss data is not all the character string be made up of letter, now determine that the type of data transformations failure is that character types are inconsistent.After detecting the failed type of default multiple conversions respectively, output detections result, such as, detect be character length do not mate cause data transformations failure time, can output character length not mate, user can transform failed root according to this testing result rapidly locating further, improves the efficiency of searching data transformations failure cause.
As can be seen from the above description, present invention achieves following technique effect:
The embodiment of the present invention is by obtaining data to be transformed, treat conversion data and perform conversion processing, obtain conversion results, conversion successful data in data to be transformed and data relay miss data to be transformed is obtained according to conversion results, and the conversion successful data in data to be transformed is stored to first object memory location, and the conversion miss data in data to be transformed is stored to the second target storage position.Conversion successful data in data to be transformed and the conversion miss data in data to be transformed are stored in different memory location by the embodiment of the present invention respectively, realize automatically recording in data transformations process and change into merit data and transform miss data, thus can obtain easily and fast and check conversion successful data and transform miss data, solve and search the low problem of data transformations information efficiency that is successful and data transformations failure.In addition, the embodiment of the present invention further according to the reason of the conversion successful data in data to be transformed and the conversion miss data automatic analysis data transformations failure in data to be transformed, can improve the efficiency of searching data transformations failure cause.
It should be noted that, can perform in the computer system of such as one group of computer executable instructions in the step shown in the process flow diagram of accompanying drawing, and, although show logical order in flow charts, but in some cases, can be different from the step shown or described by order execution herein.
According to the another aspect of the embodiment of the present invention, provide a kind of data transformations signal conditioning package, this data transformations signal conditioning package may be used for the data transformations information processing method performing the embodiment of the present invention, and the method for the embodiment of the present invention also can be performed by the data transformations signal conditioning package of the embodiment of the present invention.
Fig. 2 is the schematic diagram of the data transformations signal conditioning package according to the embodiment of the present invention, and as shown in Figure 2, this data transformations signal conditioning package comprises: the first acquiring unit 10, conversion unit 20, second acquisition unit 30 and storage unit 40.
First acquiring unit 10, for obtaining data to be transformed.
Data transformations refers to and raw data is converted into target data according to preset rules, such as, and data type conversion, Data Integration, data mining etc.Data to be transformed refer to the data needing to perform conversion processing, and can be the data obtained from database, also can be the data obtained from other electrical forms, such as, and the data obtained from Excel form.
Preferably, the first acquiring unit 10 comprises: the first acquisition module, for obtaining source data from presetting database; First memory module, for being stored to the 3rd target storage position by source data; And second acquisition module, for obtaining data to be transformed from the 3rd target storage position.
Conversion unit 20, performing conversion processing for treating conversion data, obtaining conversion results.
Alternatively, conversion unit 20 comprises: determination module, for determining default conversion types; Conversion module, for being target data according to presetting conversion types by data transformations to be transformed, wherein, the type of target data is for presetting conversion types; Set up module, for setting up the corresponding relation of target data and data to be transformed; And output module, for exporting the corresponding relation of target data and data to be transformed.
Second acquisition unit 30, for obtaining conversion successful data in data to be transformed and data relay miss data to be transformed according to conversion results, wherein, transform successful data and refer to that in data to be transformed, successful conversion is the data of target data, transform miss data and refer to the unsuccessful data being converted into target data in data to be transformed.
After treating conversion data execution conversion processing, obtain the conversion successful data in data to be transformed and data relay miss data to be transformed, such as, different marks can be added for data relay successful data to be transformed and data relay miss data to be transformed in data transformations process, according to identifying the conversion successful data and data relay miss data to be transformed distinguished in data to be transformed, also in conversion process, the conversion successful data in data to be transformed can be taken out the memory headroom being stored in other positions from data to be transformed, thus conversion successful data in data to be transformed and data relay miss data to be transformed can be distinguished according to the position of content space.
Storage unit 40, for the conversion successful data in data to be transformed is stored to first object memory location, and is stored to the second target storage position by the conversion miss data in data to be transformed.
First object memory location and the second target storage position are any two not identical memory locations.After point other places store conversion successful data in data to be transformed and data relay miss data to be transformed, user can in conjunction with the data transformations exception recorded in the conversion successful data in data to be transformed and data relay miss data to be transformed and data transformations daily record and failed information, such as, the log information of record when SSIS platform carries out data transformations, express-analysis goes out the reason of data transformations failure.
Preferably, storage unit 40 comprises: the 3rd acquisition module, for obtaining preset data file layout; And second memory module, for being stored to first object memory location according to the conversion successful data in preset data file layout data just to be transformed, and according to preset data file layout, the conversion miss data in data to be transformed is stored to the second target storage position.
Preferably, this device also comprises: resolution unit, for transforming failed type according to transforming successful data and transforming miss data parsing conversion miss data; And output unit, transform the failed type of miss data conversion for exporting.
Data transformations unsuccessfully may cause due to reasons such as character length, character types, character sequences.Particularly, the type of multiple rule to data transformations failure can be pre-set detect.Such as, character length, character types, character sequence is detected successively.
The embodiment of the present invention obtains data to be transformed by the first acquiring unit 10, converting unit 20 is treated conversion data and is performed conversion processing, obtain conversion results, second acquisition unit 30 obtains conversion successful data in data to be transformed and data relay miss data to be transformed according to conversion results, and the conversion successful data in data to be transformed is stored to first object memory location by storage unit 40, and the conversion miss data in data to be transformed is stored to the second target storage position.By the conversion successful data in data to be transformed and the conversion miss data in data to be transformed are stored in different memory location respectively, realize automatically recording in data transformations process and change into merit data and transform miss data, thus can obtain easily and fast and check conversion successful data and transform miss data, solve and search the low problem of data transformations information efficiency that is successful and data transformations failure.
Obviously, those skilled in the art should be understood that, above-mentioned of the present invention each module or each step can realize with general calculation element, they can concentrate on single calculation element, or be distributed on network that multiple calculation element forms, alternatively, they can realize with the executable program code of calculation element, thus, they can be stored and be performed by calculation element in the storage device, or they are made into each integrated circuit modules respectively, or the multiple module in them or step are made into single integrated circuit module to realize.Like this, the present invention is not restricted to any specific hardware and software combination.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.