CN111625581A - System data processing method adopting button to start service - Google Patents

Abstract

The invention discloses a system data processing method adopting a button to start service, which comprises the following steps: deploying webservice service; receiving starting information sent from the outside through the webservice service, and starting a button task; establishing a joblog and a jobdetail log, and storing starting information; and (5) completing the operation of the key joband judging whether the operation is successful or not. The method can refine the data processing flow of the system, establish a program framework, determine the interfaces between the module and other modules, determine the main algorithm of the functions specified for realizing the demand analysis, provide guidance for the detailed design, the test and the writing of the maintenance document, enhance the usability and the user experience of the exchange operation, have strong practicability and wide application range, and are easy to popularize.

Description

System data processing method adopting button to start service

Technical Field

The invention relates to the technical field of computers, in particular to a system data processing method which is high in practicability and adopts a button starting service.

Background

The effective utilization of information resources is a core problem of information-based construction, the existing data resources are dispersed in power authorities, district and county construction offices and direct units, and data of enterprises, personnel, engineering and the like only always keep the current latest condition, so that the precipitation of historical data is lacked, and the data between businesses is also lacked in effective association. And the information resource directory system and the information resource exchange system are lacked, so that the cross-region and cross-department information resource sharing is difficult to realize. In addition, a large amount of implicit information resources are not effectively utilized, and some basic services are repeatedly built.

Nowadays, with the rapid development of science and technology and the continuous promotion of informatization, the data volume of human beings is explosively increased, the value contained in the data is increasingly emphasized, and an era of large-scale sharing and application is opened. The data sharing is realized, more people can fully use the existing data resources, the repeated labor and cost of data collection, data acquisition and the like are reduced, and the social cost is reduced. Due to different data sources, data quality, data formats and data contents are different, and sometimes, the problems of incapability of conversion or information loss after conversion are encountered, and the sharing of data among various groups and systems is seriously hindered.

The keytle is used as a few open source ETL tools, and is widely applied by the advantages of strong data extraction, conversion and loading capacity, friendly graphical interface, abundant data warehouse adaptation, big data support, high-efficiency execution efficiency, expandability and the like.

Disclosure of Invention

In view of the above, the first aspect of the present invention is to provide a system data processing method using a key start service. The problems existing in the background art can be solved.

The purpose of the first aspect of the invention is realized by the following technical scheme:

the system data processing method adopting the button startup service comprises the following steps:

step S1, the webservice service is deployed;

step S2: receiving starting information sent from the outside through the webservice service, and starting a button task;

step S3: establishing a joblog and a jobdetail log, and storing starting information;

step S4: and (3) completing the operation of the button jobe, judging whether the operation is successful, if the operation is successful, storing the operation success information to the logs joblog and jobdetail, and if the operation is failed, storing the operation failure information to the logs joblog and jobdetail.

Specifically, in step S3, the start information includes a keytleid, a logid, and xmlPack.

Specifically, in step S4, the determination is as follows: and querying the jobid through the logid to judge whether the data volume of the data source is 0, if so, judging that the operation is successful, and if not, judging that the operation is failed.

Specifically, in step 2, there may be a plurality of button tasks to be started.

It is an object of a second aspect of the invention to provide a computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method as described above when executing the computer program.

It is an object of a third aspect of the invention to provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method as previously described.

The invention has the beneficial effects that: the method can refine the data processing flow of the system, establish a program framework, determine the interfaces between the module and other modules, determine the main algorithm of the functions specified for realizing the demand analysis, provide guidance for the detailed design, the test and the writing of the maintenance document, enhance the usability and the user experience of the exchange operation, have strong practicability and wide application range, and are easy to popularize.

Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof.

Drawings

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings, in which:

fig. 1 is a schematic business logic flow diagram according to a first embodiment of the present invention;

FIG. 2 is a schematic business logic flow diagram according to a second embodiment of the present invention;

FIG. 3 is a schematic view of an operation interface of "calculate difference data warehouse entry" created in the third embodiment;

FIG. 4 is a schematic diagram illustrating an operation process of a third configuration operation according to an embodiment;

fig. 5 to fig. 18 are schematic views of operation interfaces for performing various specific operations in the third embodiment.

Detailed Description

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are illustrative of the invention only and are not limiting upon the scope of the invention.

The terms of art relevant to the present invention are defined as follows:

web Service: the network modules are self-describing and self-contained available network modules, can execute specific Service functions, can exchange data among applications implemented according to the Web Service specification regardless of the language, platform or internal protocol used by the applications, can enable different applications running on different machines to exchange data or be integrated without the aid of additional and special third-party software or hardware, and are easy to deploy because the applications are based on some conventional industry standards and some existing technologies, such as XML and HTTP. Web Service reduces the cost of application interfaces. The Web Service provides a general mechanism for the integration of the business process of the whole enterprise and even among a plurality of organizations, and is suitable for building an SOA architecture.

Example one

As shown in fig. 1, in this embodiment, the webservice receives the start information sent by the data center, starts a button task, and returns that the job start is successful. The start-up information includes: keytleid, logic, xmlPack. In this embodiment, only one key job can be started, and specifically, the method includes the following steps:

step S1, the webservice service is deployed;

step S2: receiving starting information sent from the outside through the webservice service, and starting a button task;

step S3: establishing a joblog and a jobdetail log, and storing starting information; in this embodiment, the start information includes a keytleid, a logic, and an xmlPack.

Step S4: and (3) completing the operation of the button jobe, judging whether the operation is successful, if the operation is successful, storing the operation success information to the joblog and the jobdetail, and if the operation is failed, storing the operation failure information to the joblog and the jobdetail, wherein the judgment mode is as follows: and querying the jobid through the logid to judge whether the data volume of the data source is 0, if so, judging that the operation is successful, and if not, judging that the operation is failed.

The header information of the input parameters of the present embodiment is shown in the following table:

serial number

Require that

Name (R)

Description of the invention

Data type

Field constraints

1.

Y

kettleid

Key number

String

2.

Y

logid

Log numbering

String

3.

N

xmlPack

Parameter(s)

String

Header information of the returned values is shown in the following table

Serial number	Name (R)	Description of the invention	Data type	Remarks for note
					1.	success	Whether or not to succeed	boolean	true success/false error
2.	message	Information of errors	String

Example two

As shown in fig. 2, in this embodiment, a web service receives a plurality of pieces of start information sent by a data center, starts a plurality of key tasks, and returns a job start success. The start-up information includes: keytleid, logic, xmlPack. The number of the button tasks started in this embodiment may be plural.

Specifically, the difference from the first embodiment is that in step S2, a plurality of button tasks are started in the present embodiment.

EXAMPLE III

The method of the invention is adopted for carrying out the example of specific development and application:

the method comprises the following steps: the V _ OA _ XZ _ ZRZT view of the information center OA system is extracted to a data center library.

The fields in view V _ OA _ XZ _ ZRZT are described as follows:

encoding	Name (R)	Type (B)	Length of
				GUIDCode	Unique number	varchar(50)	50
DJBM	Registration code	varchar(50)	50
				SJDW	Design unit	varchar(200)	200
SGZCBDW	General construction contract unit	varchar(200)	200
				JLDW	Supervision unit	varchar(200)	200
ZTLB	Subject classes	varchar(50)	50
				SGDWZZJGDM	Construction unit organization code	varchar(50)	50
JLZZJGDM	Proctoring organization code	varchar(50)	50
				ZZJGDM	Organization code	varchar(50)	50
XMBM	Item coding	varchar(50)	50
				JSDWZZJGDM1	Organization code	1 of construction organization	varchar(50)	50
JSDW	Construction unit	varchar(200)	200
				SGFBDW	Construction sub-packaging unit	varchar(200)	200
SGFBZZJGDM	Construction sub-packaging organization code	varchar(50)	50
				YWLCBM	Ywlcbm	varchar(50)	50
BYFZSGDW	Termite prevention and control construction unit	varchar(50)	50
				SGZCBAQSCXKZ	Construction general contract safety production permit	varchar(100)	100
DLBM	DLBM	varchar(50)	50
				intime	intime	datetime
ProcID	ProcID	uniqueidentifier

The development steps are as follows:

firstly, the analysis and planning of the data access requirement are completed,

analyzing data access requirements, and listing the following collected and converted loading information:

source database IP address	10.1.9.143
		Source database port	1433
Account number of source database	OA_Readonly
		Source database password	***
Source database name	ViewDB_NGZ
		Source tables or views	V_OA_XZ_ZRZT
Period of time	Every morning 1:00

First, the V _ OA _ XZ _ ZRZT field type is analyzed to determine the extraction method.

It can be seen that the view has a field GUIDCode that can be the primary key, but no field for delta identification, such as a common delta ID or last update time field. Thereby determining the extraction method to be an incremental method.

The new creation of the data table is completed through the following steps:

1. the temporary table is established according to a table establishing rule and used for storing extracted source table data, the temporary table needs to be placed in a DataCenterTemp database, and the structure of SJZX is used, the target table naming rule is that a data source unit is added in front of the table name of a source table, such as XXZX (the data source unit is an information center) _ OA (affiliated system) _ V _ OA _ XZ _ ZRZT (source table name), and the table name XXZX _ V _ OA _ XZ _ ZZT is finally obtained because of two OAs, and the table structure is consistent with the source.

2. And establishing a temporary table DataCenterTemp.SJZX.XXZX _ V _ OA _ XZ _ ZRZT _ MID according to the table establishing rule for storing the calculated difference data. The temporary table is used for storing difference data, whether the temporary table is newly built or not is determined according to actual needs, and if full extraction is adopted, the temporary intermediate table does not need to be newly built.

3. And establishing a target table DataCenter, dbo, XXZX, V, OA, XZ, ZRZT according to the table establishing rule, wherein the target table is used for storing formal data. The naming rule is the same as above, and 4 fields are additionally added

4、sjly varchar(2000),kettle_job varchar(2000),SJZX_RKSJ datetime,SJZX_GXSJ datetime

5. And establishing a target table DataCenterHistory.dbo.XXZX _ V _ OA _ XZ _ ZRZT according to the table establishing rule for storing historical data. The naming rule is the same as above, and 5 fields are additionally added

6、sjly varchar(2000),kettle_job varchar(2000),SJZX_RKSJ datetime,SJZX_GXSJ datetime,Vesion_Num_Main varchar(100)

7. Note that: if the table structure is modified, such as adding new fields, the field sequence of the target table, the history table and the temporary table must be ensured to be in one-to-one correspondence, and the field cannot be directly added behind the version _ Num _ Main.

Secondly, performing button operation development

1. Configuration work

As shown in fig. 3, a job is created at Spoon, named "calculate difference data warehouse". And (3) creating a naming rule of the job: the target table is directly preceded by "JOB _", such as the target table is: XXZX _ V _ OA _ XZ _ ZRZT, then the new job name is: JOB _ XXZX _ V _ OA _ XZ _ zzt; path rule of new job: and storing the operation and the conversion according to the rule of 'unit-collection-JOB/TRAN'.

As shown in fig. 4, START is the identifier of the job entry, and "set variable" is used to set the variable value in the job, and a plurality of partial parameters are converted, so that a variable method can be used, which is not mandatory, but a plurality of variables can be used conveniently. The setup variables must be modified by selecting templates for additional operations.

The method comprises the steps of reading data volume of READ _ LINES operation, inputting data volume of INPUT _ LINES operation, writing data volume of WRITE _ LINES operation, outputting data volume of OUTPUT _ LINES operation, updating data volume of UPDATED _ LINES operation, deleting data volume of DEFLED _ LINES operation, a source table READ by READ _ OBJECT operation, a target table written by WRITE _ OBJECT operation, a main key of a table of.

"TRAN _ DC _ JOB _ START" calls a common conversion whose purpose is to execute the conversion, generates a log ID of the JOB, and the data center foreground can display the JOB execution log by reading the ID, regardless of data extraction. Common components, without modification.

"TRAN _ DC _ JOB _ ID" is used to obtain the log ID of the JOB and generate a variable, which is also a common transformation, common component, without modification.

The icon for checking whether the table exists is used for checking whether the source table exists, if the source table is not connected, an error is directly reported, and whether the database link configured in the source table is the current operation data source needs to be checked.

The "TRAN _ XXZX _ V _ OA _ TAB" icon is a common transformation for calling OA extraction source tables to extract data from the source system library to DataCenterTemp.SJZX.XXZX _ V _ OA _ XZ _ ZRZT, and some are common components to be checked whether the common components are the unit system where the current job is located. For example, the OA system has a common component TRAN _ XXZX _ V _ OA _ TAB, but some tables have boolean fields, which are not supported by the sql server, so that the common component cannot be used, and a unique conversion of the current table needs to be created to convert the boolean into Integer, refer to the example: TRAN _ XXZX _ V _ OA _ g _ RDYACPB _ TAB

"calculate difference data" is to compare the difference between DataCenterTemp.SJZX.XXZX _ V _ OA _ XZ _ ZRZT and DataCenterTemp.dbo.XXZX _ V _ OA _ XZ _ ZRZT, calculate difference data and store the difference data in DataCenterTemp.SJZX.XXZX _ V _ OA _ XZ _ ZRZT _ MID;

the "difference data storage" is to insert and update the difference data dataCenterTemp.SJZX.XXZX _ V _ OA _ XZ _ ZRZT _ MID calculated in the previous step into the target table dataCenter.dbo.XXZX _ V _ OA _ XZ _ ZRZT

The purpose of this common transformation "TRAN _ DC _ TB _ DELETE" is to DELETE data that has been deleted in the source table, but that is still held in the target table;

the purpose of this common translation of "TRAN _ DC _ XXZX _ TB _ COUNT" is to calculate the total amount of data written to the system tables for use by the data center foreground for the target tables. In addition, whether the metadata model is configured or not and judgment of the read data volume and the write data volume are added.

The purpose of the common conversion of "TRAN _ COUNT _ LOG" is to calculate the amount of data read, input, output, write, update, delete of this run of the job

In the figure, the dotted line indicates that the output of the component in the previous step is running in parallel, the red circle indicates the trend of error, the green arrow indicates the trend of success, and the yellow lock indicates the forced trend regardless of success and failure. The conversion naming rules are as follows:

(1) entering the file into a temporary library: TRAN _ target Table name _ FILE

(2) Table into temporary repository: TRAN _ target table name _ TAB

(3) And (4) putting the total amount of temporary stocks into a formal stock: TRAN _ target Table name _ QL

(4) And (4) incremental temporary library entering a formal library: TRAN _ target Table name _ ZL

(5) And (3) inserting and updating the temporary library into the formal library in a full amount: TRAN _ target Table name _ CRGX

(6) Calculating difference data into a temporary table: TRAN _ target table name _ MID

Conversion path rule:

(1) stored under the corresponding unit-acquisition-TRAN directory

Job naming parameters: the job must set two naming parameters:

(1) the LogId is consistent as shown in the figure, and the function is that the resource center calls the operation in the foreground and transmits the operation to an operation LogId for positioning the operation log.

(2) filename _ path, data source information, if the collected file is put in storage, the path of the available file, if the collected file is put in storage, the data source table or view name is written.

2. Configuring data acquisition start logs

As shown in fig. 5, the log of the job is configured, and the log of the button operation is saved in the data table.

As shown in fig. 6, the log of the conversion is configured and the log of the conversion operation is saved in a table.

3. Configuring job reception parameters

As shown in fig. 7, it is necessary to set these two parameters to the job.

4. Calculating difference data

As shown in fig. 8, reading the temporary table storing the source table data must be sorted according to the primary key. As shown in FIG. 9, the difference between the two sides is compared, and the difference between the comparison is new, changed, deleted, and identified except that the identified is not processed, and all the differences are inserted into the middle table.

5. Differential data warehousing

As shown in FIG. 10, GUIDCODE queries the target table as a query primary key, updates if any, and inserts if not.

Third, data center operation configuration

1. Metadata model update

As shown in fig. 11, since the above table is added to the metadata model "information center OA system", the data model must be updated at the system, and the newly added job is successfully executed.

And inputting a local operation website by the browser, and entering an enterprise service platform.

And opening metadata management-metadata model-finding the information center OA system, clicking to acquire metadata, checking an analysis report, and replacing with the latest version.

2. Increase work

As shown in fig. 12 and 13, the browser inputs the local operation platform website to enter the enterprise service platform.

Opening job scheduling management-job configuration information; clicking a 'newly added' button; the job type 'ETL task' is selected, and the job newly built at the spaon client can be found by clicking 'job parameters'.

3. Information of source unit

Data collection configuration-Source information-Source Unit information-the information of the source unit for the job is added as shown in FIG. 14, if it already exists, it need not be added.

4. Source system information

As shown in FIG. 15, the data collection configuration-Source information-Source System information-adds the source system information for the job as shown, if it already exists, then no addition is required.

5. Data table interface

Data acquisition configuration-data interface setting-data table interface, as shown in fig. 16, as shown in the figure, data table interface information is added, and the main function is to map the corresponding relation of units, systems, tables and target tables of the source table. And adding a new job and adding a new mapping relation.

6. Button job information

As shown in fig. 17, opening the data acquisition configuration, i.e., the button job information, the designed job can be associated with the newly added data table interface in the previous step.

7. Job log list

As shown in fig. 18, the configuration of the source unit, the system, the data table interface, and the key job information has the functions of looking up the job log and quickly searching the source target of the job, so as to conveniently locate the error position of the job. Once a job is in error, the information related to the job can be known clearly.

It should be noted that any process or method descriptions in flow charts of the present invention or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and that the scope of the preferred embodiments of the present invention includes additional implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, and the program may be stored in a computer readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.

Finally, the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all of them should be covered by the claims of the present invention.

Claims (6)

1. The system data processing method adopting the button start service is characterized in that: the method comprises the following steps:

step S1, the webservice service is deployed;

step S2: receiving starting information sent from the outside through the webservice service, and starting a button task;

step S3: establishing a joblog and a jobdetail log, and storing starting information;

step S4: and (3) completing the operation of the button jobe, judging whether the operation is successful, if the operation is successful, storing the operation success information to the logs joblog and jobdetail, and if the operation is failed, storing the operation failure information to the logs joblog and jobdetail.

2. The system data processing method using a button startup service according to claim 1, wherein: in step S3, the start information includes a keytleid, a logic, and xmlPack.

3. The system data processing method using a button startup service according to claim 2, wherein: in step S4, the determination method is as follows: and querying the jobid through the logid to judge whether the data volume of the data source is 0, if so, judging that the operation is successful, and if not, judging that the operation is failed.

4. A system data processing method using a key start service according to claim 1, 2 or 3, wherein: in step 2, there may be a plurality of button tasks to be started.

5. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory and capable of running on the processor, wherein: the processor, when executing the computer program, implements the method of any of claims 1-4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when executed by a processor, implements the method of any one of claims 1-4.

Images