CN108536745A - Tables of data extracting method, terminal, equipment and storage medium based on Shell - Google Patents
Tables of data extracting method, terminal, equipment and storage medium based on Shell Download PDFInfo
- Publication number
- CN108536745A CN108536745A CN201810196485.4A CN201810196485A CN108536745A CN 108536745 A CN108536745 A CN 108536745A CN 201810196485 A CN201810196485 A CN 201810196485A CN 108536745 A CN108536745 A CN 108536745A
- Authority
- CN
- China
- Prior art keywords
- data
- tables
- shell
- table name
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Abstract
The invention discloses a kind of tables of data extracting method, terminal, equipment and storage medium based on Shell, wherein this method includes：Identify the tables of data in Shell scripts；Extract the table name of the tables of data；Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and object table；The corresponding data information of different types of tables of data is obtained, and acquired different types of data information is exported into same default document.The present invention artificial need not search the relevant tables of data of each script troublesomely, farthest simplify arrangement and more new technological process, and can save a large amount of human resources by improved tables of data extracting method.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of tables of data extracting method based on Shell, terminal,
Equipment and storage medium.
Background technology
Shell is a free programming language, for realizing that automatic and interactive task is communicated, without people's
Intervene.Script, which can be created, using it is used for realizing that Shell then can be according to the prompt of program to order or program offer input
Mock standard input is supplied to the input that program needs to realize that interactive program executes.
In the application of existing Shell scripts, Shell scripts have often related to more tables of data, if passed through
Manually each Shell scripts are arranged, to obtain the tables of data in Shell scripts, extraction process can be caused to take very much,
And workload is very big；In addition, the sentence of the tables of data in Shell scripts can become with the modification of application version
Change, if by manual sorting and updating these information and also needing to expend a large amount of manpowers, and sorts out the tables of data come and also hold very much
Easily there is mistake.
Invention content
In view of this, the embodiment of the present invention provides a kind of tables of data extracting method based on Shell, terminal, equipment and deposits
Storage media can farthest simplify arrangement and more new technological process, and can save a large amount of human resources.
On the one hand, the tables of data extracting method based on Shell that an embodiment of the present invention provides a kind of, this method include：
Identify the tables of data in Shell scripts；
Extract the table name of the tables of data；
Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and target
Table；
The corresponding data information of different types of tables of data is obtained, and acquired different types of data information is exported
To in same default document.
On the other hand, an embodiment of the present invention provides a kind of, and the tables of data based on Shell extracts terminal, the terminal packet
It includes：
Recognition unit, for identification tables of data in Shell scripts；
Extraction unit, the table name for extracting the tables of data；
Taxon, for being classified to the tables of data according to the table name extracted, wherein the tables of data includes
Source table and object table；
Acquiring unit, for obtaining the corresponding data information of different types of tables of data, and by acquired different type
Data information export into same default document.
Another aspect, the embodiment of the present invention additionally provide a kind of tables of data extraction equipment based on Shell comprising：
Memory, for storing the program for realizing tables of data extracting method；And
Processor, the program for running the realization tables of data extracting method stored in the memory are as above to execute
The method.
It is described computer-readable to deposit in another aspect, the embodiment of the present invention additionally provides a kind of computer readable storage medium
Storage media storage there are one either more than one program the one or more programs can by one or more than one
Processor execute, to realize method as described above.
The embodiment of the present invention is by identifying the tables of data in Shell scripts；Extract the table name of the tables of data；According to being carried
The table name taken classifies to the tables of data, wherein the tables of data includes source table and object table；It obtains different types of
The corresponding data information of tables of data, and acquired different types of data information is exported into same default document.This hair
Bright embodiment by improved tables of data extracting method, artificial need not troublesomely search the relevant tables of data of each script,
Arrangement and more new technological process are farthest simplified, and a large amount of human resources can be saved.
Description of the drawings
Technical solution in order to illustrate the embodiments of the present invention more clearly, below will be to needed in embodiment description
Attached drawing is briefly described, it should be apparent that, drawings in the following description are some embodiments of the invention, general for this field
For logical technical staff, without creative efforts, other drawings may also be obtained based on these drawings.
Fig. 1 is a kind of schematic flow diagram of tables of data extracting method based on Shell provided in an embodiment of the present invention；
Fig. 2 is a kind of schematic flow diagram of tables of data extracting method based on Shell provided in an embodiment of the present invention；
Fig. 3 is a kind of schematic flow diagram of tables of data extracting method based on Shell provided in an embodiment of the present invention；
Fig. 4 is a kind of schematic flow diagram of tables of data extracting method based on Shell provided in an embodiment of the present invention；
Fig. 5 is a kind of schematic flow diagram for tables of data extracting method based on Shell that another embodiment of the present invention provides；
Fig. 6 is a kind of schematic block diagram of tables of data extraction terminal based on Shell provided in an embodiment of the present invention；
Fig. 7 is a kind of another schematic block diagram of tables of data extraction terminal based on Shell provided in an embodiment of the present invention；
Fig. 8 is a kind of another schematic block diagram of tables of data extraction terminal based on Shell provided in an embodiment of the present invention；
Fig. 9 is a kind of another schematic block diagram of tables of data extraction terminal based on Shell provided in an embodiment of the present invention；
Figure 10 is a kind of another schematic frame of tables of data extraction terminal based on Shell provided in an embodiment of the present invention
Figure；
Figure 11 is a kind of structure composition signal of tables of data extraction equipment based on Shell provided in an embodiment of the present invention
Figure.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
It should be appreciated that ought use in this specification and in the appended claims, term " comprising " and "comprising" instruction
Described feature, entirety, step, operation, the presence of element and/or component, but one or more of the other feature, whole is not precluded
Body, step, operation, element, component and/or its presence or addition gathered.
It is also understood that the term used in this description of the invention is merely for the sake of the mesh for describing specific embodiment
And be not intended to limit the present invention.As description of the invention and it is used in the attached claims, unless on
Other situations are hereafter clearly indicated, otherwise " one " of singulative, "one" and "the" are intended to include plural form.
Referring to Fig. 1, Fig. 1 is a kind of signal of the tables of data extracting method based on Shell provided in an embodiment of the present invention
Flow chart.This method may operate in smart mobile phone (such as Android phone, IOS mobile phones), tablet computer, laptop
And in the terminals such as smart machine.Data dimension generation method described in the embodiment of the present invention need not be looked into manually troublesomely
It looks for the relevant tables of data of each script, farthest simplify arrangement and more new technological process, and a large amount of human resources can be saved.
Fig. 1 is the schematic flow diagram of the tables of data extracting method provided in an embodiment of the present invention based on Shell.The method comprising the steps of
S101~S104.
S101 identifies the tables of data in Shell scripts.
In embodiments of the present invention, the tables of data refers to connecting database by SQL statement in Shell scripts,
And the relevant tables of data called out from database；In Shell scripts connect database and call tables of data be in order to
The data in database are obtained, it, can be by obtaining the data in database, to reach monitoring number in daily maintenance work
According to the purpose of certain information in library, to further understand the performance of equipment in real time.
Identify the tables of data in Shell scripts, it can be by identifying that the keyword in SQL statement be realized, for example, can be with
It is inserted into sentence " insert into " by identification, to identify the tables of data be inserted into sentence and be followed by；Identification can be passed through
Query statement " select*from ", to identify tables of data that query statement is followed by；It can also be by identifying more newspeak
Sentence " update ", to identify the tables of data for updating sentence and being followed by；It can also be by identifying cancel statement " delete
From ", to identify tables of data etc. that cancel statement is followed by.
S102 extracts the table name of the tables of data.
In embodiments of the present invention, after by identifying the tables of data in Shell scripts, identified tables of data is extracted
Table name, for example, in being inserted into sentence " insert into { TABLENAME } ", the table name for the tables of data extracted is
“TABLENAME”；In query statement " select*from { USERNAME } ", the table name for the tables of data extracted is
The table name of " USERNAME ", the tables of data extracted in update sentence " update { DBNAME } " are " DBNAME ", are being deleted
The table name for the tables of data extracted in sentence " delete from { KBNAME } " is " KBNAME ".
S103 classifies to the tables of data according to the table name extracted, wherein the tables of data include source table and
Object table.
It in embodiments of the present invention, will after being extracted to the table name of tables of data by a series of keyword of SQL statements
Tables of data table name is stored in temporary file, and the type of the tables of data includes source table and object table, wherein according to tables of data
Table name can be to the method that the type of tables of data is classified：If tables of data table name is an independent character string, and the independence
Have that space or line feed and table name front follow before and after character string is from keywords, then the type of the tables of data is
Source table, if tables of data table name is an independent character string, and before having space or line feed and table name before and after the respective character string
What face followed is non-from keywords, then the type of the tables of data is object table, optionally, non-from keywords can be
The SQL statements keyword such as into, update.
Further, as shown in Fig. 2, the step S103 includes step S201~S202.
S201 determines character string corresponding with the tables of data table name.
In embodiments of the present invention, the character string refers to a string of characters corresponding to tables of data table name, due to data
Table table name can be made of number, letter, underscore, so the character string can also be by number, letter, underscore group
At character.
S202 classifies to the tables of data according to the character string.
In embodiments of the present invention, classified to the tables of data according to the character string, the method and data of classification
SQL statement keyword before table table name is related, and the method for classification can be：If tables of data table name is an independent character string,
And have that space or line feed and table name front follow before and after the respective character string is from keywords, then the tables of data
Type be source table, if tables of data table name is an independent character string, and have before and after the respective character string space or line feed with
And what table name front followed is non-from keywords, then the type of the tables of data is object table, optionally, non-from is crucial
Word can be the SQL statements keyword such as into, update.Classified to the tables of data by the character string, Ke Yipai
Interference except invalid data table to valid data table, such as from_unixtime, as the character that this from is followed by does not belong to
It will be identified as invalid data table in defined content, such case is not just considered.
S104 obtains the corresponding data information of different types of tables of data, and acquired different types of data is believed
In breath output to same default document.
In embodiments of the present invention, the tables of data is the JOB contingency tables of Hadoop, and the JOB contingency tables of the Hadoop are
It is write, and is stored in corresponding database using Hadoop sentences and SQL statement, the JOB of Hadoop is associated with
The table name of table is written in corresponding Shell scripts, when needing to identify the JOB contingency tables of Hadoop, first extracts Shell scripts
In table name, that is, have identified in the type for having arrived which JOB contingency table and these JOB contingency tables involved in script
Belong to source table or object table.
It should be noted that source table refers to the table of table and external relations type database inside Hadoop, the word of source table
Have that space or line feed and table name front follow before and after symbol string is from keywords；Object table is referred to by writing mode
It is divided into and is inserted into object table and coverage goal table, such as insert into tableA, this is exactly plug-type object table, insert
Overwrite tableB, this is exactly the object table of cover type, and the default document can be the data in presetting database
Table, for example, keyword and its related content are captured from the script of JOB contingency tables, and by the content record captured to temporarily
In file, these are completed in the hdfs levels of Hadoop, and the result in temporary file is then loaded into Hadoop's
In Hive tables, data information is output in specified default oracle database by the data in Hive tables by Sqoop modes,
Specifically, exporting data information into the tables of data pre-established in the default oracle database.Optionally, user
The tables of data that pre-establishes for being stored with data information can be formed an Oracle Pkg (Oracle packaging,
Oracle package files), if desired the tables of data optimizes, and only needs to optimize this Oracle Pkg.
Further, if as shown in figure 3, the tables of data is source table, the step S104 includes step S301~S303.
The source table is divided into inside sources table and external source table by S301.
In embodiments of the present invention, the inside sources table refers to table (such as the Hive of Hadoop inside Hadoop
Table), the external source table refers to the table of external relations type database.
S302 obtains the inside sources table and the corresponding data information of external source table.
In embodiments of the present invention, the data information includes table information, field information etc., and wherein table information can be table
Name, table type etc., field information can be field name, field type etc..
S303 exports acquired data information into default document.
In embodiments of the present invention, the default document can be the data pre-established in default oracle database
Table, specifically, acquired data information can be exported to the data pre-established in the default oracle database
In table, and user can will be stored with the tables of data of data information pre-established and form an Oracle Pkg
(Oracle packaging, Oracle package file), if desired the tables of data optimizes, and only needs to optimize this
Oracle Pkg, so that it may preset the preset data table established in oracle database to reach optimization.
Further, if as shown in figure 4, the tables of data be object table, the step S104 include step S401~
S403。
The object table is divided into and is inserted into object table and coverage goal table by S401.
In embodiments of the present invention, the table being inserted into object table such as SQL statement " insert into tableA "
The type of tableA, the object table are determined by the SQL statement keyword before tables of data table name, if into is followed by slotting
Enter object table, the table tableB in the coverage goal table such as SQL statement " insert overwrite tableB ", the mesh
The type of mark table is determined by the SQL statement keyword before tables of data table name, if overwrite is followed by coverage goal
Table.
S402 obtains the insertion object table and the corresponding data information of coverage goal table.
In embodiments of the present invention, the data information includes table information, field information etc., and wherein table information can be table
Name, table type etc., field information can be field name, field type etc..
S403 exports acquired data information into default document.
In embodiments of the present invention, the default document can be the data pre-established in default oracle database
Table, specifically, acquired data information can be exported to the data pre-established in the default oracle database
In table, and user can will be stored with the tables of data of data information pre-established and form an Oracle Pkg
(Oracle packaging, Oracle package file), if desired the tables of data optimizes, and only needs to optimize this
Oracle Pkg, so that it may preset the preset data table established in oracle database to reach optimization.
As seen from the above, the embodiment of the present invention is by identifying the tables of data in Shell scripts；Extract the table of the tables of data
Name；Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and object table；It obtains
The corresponding data information of different types of tables of data, and acquired different types of data information is exported to same default text
In shelves.It is related artificial need not to search each script troublesomely by improved tables of data extracting method for the embodiment of the present invention
Tables of data, farthest simplify arrangement and more new technological process, and a large amount of human resources can be saved.
Referring to Fig. 5, Fig. 5 is a kind of signal of tables of data extracting method based on Shell provided in an embodiment of the present invention
Flow chart.This method may operate in smart mobile phone (such as Android phone, IOS mobile phones), tablet computer, laptop
And in the terminals such as smart machine.As shown in figure 5, the method comprising the steps of S501~S506.
S501 traverses the Shell scripts according to preset keyword.
In embodiments of the present invention, it is when being created by using first ergodic data table when being traversed to Shell scripts
Between short rule, the rule of rear ergodic data table settling time length is traversed, to realize the number of the progress to Shell scripts
According to the traversal of table, Shell scripts are carried out to be short to the traversal rule that creation time is grown from creation time, can be improved to Shell
The efficiency of script processing.
S502 positions the tables of data in the Shell scripts according to the result of traversal.
In embodiments of the present invention, using the traversing result to the Shell scripts tables of data in Shell scripts
Position is shown, by the location information of shown tables of data to determine the tables of data in the Shell scripts
Position.
S503 identifies the tables of data in Shell scripts.
In embodiments of the present invention, the tables of data refers to connecting database by SQL statement in Shell scripts,
And the relevant tables of data called out from database；In Shell scripts connect database and call tables of data be in order to
The data in database are obtained, it, can be by obtaining the data in database, to reach monitoring number in daily maintenance work
According to the purpose of certain information in library, to further understand the performance of equipment in real time.
Identify the tables of data in Shell scripts, it can be by identifying that the keyword in SQL statement be realized, for example, can be with
It is inserted into sentence " insert into " by identification, to identify the tables of data be inserted into sentence and be followed by；Identification can be passed through
Query statement " select*from ", to identify tables of data that query statement is followed by；It can also be by identifying more newspeak
Sentence " update ", to identify the tables of data for updating sentence and being followed by；It can also be by identifying cancel statement " delete
From ", to identify tables of data etc. that cancel statement is followed by.
S504 extracts the table name of the tables of data.
In embodiments of the present invention, after by identifying the tables of data in Shell scripts, identified tables of data is extracted
Table name, for example, in being inserted into sentence " insert into { TABLENAME } ", the table name for the tables of data extracted is
“TABLENAME”；In query statement " select*from { USERNAME } ", the table name for the tables of data extracted is
The table name of " USERNAME ", the tables of data extracted in update sentence " update { DBNAME } " are " DBNAME ", are being deleted
The table name for the tables of data extracted in sentence " delete from { KBNAME } " is " KBNAME ".
S505 classifies to the tables of data according to the table name extracted, wherein the tables of data include source table and
Object table.
It in embodiments of the present invention, will after being extracted to the table name of tables of data by a series of keyword of SQL statements
Tables of data table name is stored in temporary file, and the type of the tables of data includes source table and object table, wherein according to tables of data
Table name can be to the method that the type of tables of data is classified：If tables of data table name is an independent character string, and the independence
Have that space or line feed and table name front follow before and after character string is from keywords, then the type of the tables of data is
Source table, if tables of data table name is an independent character string, and before having space or line feed and table name before and after the respective character string
What face followed is non-from keywords, then the type of the tables of data is object table, optionally, non-from keywords can be
The SQL statements keyword such as into, update.
S506 obtains the corresponding data information of different types of tables of data, and acquired different types of data is believed
In breath output to same default document.
In embodiments of the present invention, the tables of data is the JOB contingency tables of Hadoop, and the JOB contingency tables of the Hadoop are
It is write, and is stored in corresponding database using Hadoop sentences and SQL statement, the JOB of Hadoop is associated with
The table name of table is written in corresponding Shell scripts, when needing to identify the JOB contingency tables of Hadoop, first extracts Shell scripts
In table name, that is, have identified in the type for having arrived which JOB contingency table and these JOB contingency tables involved in script
Belong to source table or object table.
It should be noted that source table refers to the table of table and external relations type database inside Hadoop, the word of source table
Have that space or line feed and table name front follow before and after symbol string is from keywords；Object table is referred to by writing mode
It is divided into and is inserted into object table and coverage goal table, such as insert into tableA, this is exactly plug-type object table, insert
Overwrite tableB, this is exactly the object table of cover type, and the default document can be the data in presetting database
Table, for example, keyword and its related content are captured from the script of JOB contingency tables, and by the content record captured to temporarily
In file, these are completed in the hdfs levels of Hadoop, and the result in temporary file is then loaded into Hadoop's
In Hive tables, data information is output in specified default oracle database by the data in Hive tables by Sqoop modes,
Specifically, exporting data information into the tables of data pre-established in the default oracle database.Optionally, user
Can will be stored with the tables of data of data information pre-established and formed Oracle Pkg (Oracle packaging,
Oracle package files), if desired the tables of data optimizes, and only needs to optimize this Oracle Pkg.
Referring to Fig. 6, a kind of corresponding above-mentioned tables of data extracting method based on Shell, the embodiment of the present invention also propose one
Tables of data of the kind based on Shell extracts terminal, which includes：Recognition unit 101, extraction unit 102, taxon
103, acquiring unit 104.
Wherein, the recognition unit 101, for identification tables of data in Shell scripts.In embodiments of the present invention, institute
It states tables of data to refer to connecting database by SQL statement in Shell scripts, and is called out from database relevant
Tables of data；It is in order to obtain the data in database, in daily fortune to connect database in Shell scripts and call tables of data
It ties up in work, it can be by obtaining the data in database, to achieve the purpose that certain information in monitoring data library, thus into one
Step understands the performance of equipment in real time.
Identify the tables of data in Shell scripts, it can be by identifying that the keyword in SQL statement be realized, for example, can be with
It is inserted into sentence " insert into " by identification, to identify the tables of data be inserted into sentence and be followed by；Identification can be passed through
Query statement " select*from ", to identify tables of data that query statement is followed by；It can also be by identifying more newspeak
Sentence " update ", to identify the tables of data for updating sentence and being followed by；It can also be by identifying cancel statement " delete
From ", to identify tables of data etc. that cancel statement is followed by.
Extraction unit 102, the table name for extracting the tables of data.In embodiments of the present invention, by identifying Shell
After tables of data in script, the table name of identified tables of data is extracted, for example, being inserted into sentence " insert into
In { TABLENAME } ", the table name for the tables of data extracted is " TABLENAME "；In query statement " select*from
In { USERNAME } ", the table name for the tables of data extracted is " USERNAME ", is carried in update sentence " update { DBNAME } "
The table name for the tables of data got is " DBNAME ", the tables of data extracted in cancel statement " delete from { KBNAME } "
Table name be " KBNAME ".
Taxon 103, for being classified to the tables of data according to the table name extracted, wherein the tables of data packet
Include source table and object table.In embodiments of the present invention, the table name of tables of data is carried out by a series of keyword of SQL statements
After extraction, tables of data table name is stored in temporary file, the type of the tables of data includes source table and object table, wherein
Can be to the method that the type of tables of data is classified according to tables of data table name：If tables of data table name is an independent character
String, and have that space or line feed and table name front follow before and after the respective character string is from keywords, then the number
Type according to table is source table, if tables of data table name is an independent character string, and has space before and after the respective character string or changes
What row and table name front followed is non-from keywords, then the type of the tables of data is object table, optionally, non-from
Keyword can be the SQL statements keyword such as into, update.
Acquiring unit 104, for obtaining the corresponding data information of different types of tables of data, and by acquired inhomogeneity
The data information of type is exported into same default document.In embodiments of the present invention, the JOB that the tables of data is Hadoop is associated with
The JOB contingency tables of table, the Hadoop are write using Hadoop sentences and SQL statement, and are stored in corresponding number
According in library, the table name of the JOB contingency tables of Hadoop is written in corresponding Shell scripts, as the JOB for needing identification Hadoop
When contingency table, the table name in Shell scripts is first extracted, that is, have identified and which JOB contingency table arrived involved in script,
And the type of these JOB contingency tables belongs to source table or object table.
It should be noted that source table refers to the table of table and external relations type database inside Hadoop, the word of source table
Have that space or line feed and table name front follow before and after symbol string is from keywords；Object table is referred to by writing mode
It is divided into and is inserted into object table and coverage goal table, such as insert into tableA, this is exactly plug-type object table, insert
Overwrite tableB, this is exactly the object table of cover type, and the default document can be the data in presetting database
Table, for example, keyword and its related content are captured from the script of JOB contingency tables, and by the content record captured to temporarily
In file, these are completed in the hdfs levels of Hadoop, and the result in temporary file is then loaded into Hadoop's
In Hive tables, data information is output in specified default oracle database by the data in Hive tables by Sqoop modes,
Specifically, exporting data information into the tables of data pre-established in the default oracle database.Optionally, user
Can will be stored with the tables of data of data information pre-established and formed Oracle Pkg (Oracle packaging,
Oracle package files), if desired the tables of data optimizes, and only needs to optimize this Oracle Pkg.
As seen from the above, the embodiment of the present invention is by identifying the tables of data in Shell scripts；Extract the table of the tables of data
Name；Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and object table；It obtains
The corresponding data information of different types of tables of data, and acquired different types of data information is exported to same default text
In shelves.It is related artificial need not to search each script troublesomely by improved tables of data extracting method for the embodiment of the present invention
Tables of data, farthest simplify arrangement and more new technological process, and a large amount of human resources can be saved.
As shown in fig. 7, the taxon 103, including：
Determination unit 1031, for determining character string corresponding with the tables of data table name.In embodiments of the present invention,
The character string refers to a string of characters corresponding to tables of data table name, due to tables of data table name can be by number, letter, under
Scribing line composition, so the character string can also be the character being made of number, letter, underscore.
Classification subelement 1032, for being classified to the tables of data according to the character string.In the embodiment of the present invention
In, classified to the tables of data according to the character string, the method for classification and the SQL statement before tables of data table name are crucial
Word is related, and the method for classification can be：If tables of data table name is an independent character string, and has space before and after the respective character string
Or line feed and what table name front followed is from keywords, then the type of the tables of data is source table, if tables of data table
Name is an independent character string, and that have that space or line feed and table name front follow before and after the respective character string is non-from
Keyword, then the type of the tables of data is object table, optionally, non-from keywords can be the SQL such as into, update
Statement keyword.Classified to the tables of data by the character string, invalid data table can be excluded to valid data table
Interference, such as from_unixtime, as this from characters being followed by be not belonging to as defined in content will be identified as
Invalid data table, such case are not just considered.
If as shown in figure 8, the tables of data be source table, the acquiring unit 104, including：
First execution unit 1041, for the source table to be divided into inside sources table and external source table.In the embodiment of the present invention
In, the inside sources table refers to the table (such as Hive tables of Hadoop) inside Hadoop, and the external source table refers to
The table of external relations type database.
First obtains subelement 1042, for obtaining the inside sources table and the corresponding data information of external source table.At this
In inventive embodiments, the data information includes table information, field information etc., and wherein table information can be table name, table type etc.,
Field information can be field name, field type etc..
First output unit 1043, for exporting acquired data information into default document.The default document
Can be the tables of data pre-established in default oracle database, specifically, acquired data information can be exported
Into the tables of data pre-established in the default oracle database, and user can will be stored with the pre- of data information
The one Oracle Pkg (Oracle packaging, Oracle package file) of tables of data and formation first established, if desired should
Tables of data optimizes, and only needs to optimize this Oracle Pkg, so that it may be built with reaching in the default oracle database of optimization
Vertical preset data table.
If as shown in figure 9, the tables of data be object table, the acquiring unit 104, including：
Second execution unit 1044 is inserted into object table and coverage goal table for the object table to be divided into.In the present invention
In embodiment, the table tableA being inserted into object table such as SQL statement " insert into tableA ", the object table
Type is determined by the SQL statement keyword before tables of data table name, if into is followed by insertion object table, the covering
Table tableB in object table such as SQL statement " insert overwrite tableB ", the type of the object table is by tables of data
SQL statement keyword before table name determines, if overwrite is followed by coverage goal table.
Second obtains subelement 1045, for obtaining the insertion object table and the corresponding data information of coverage goal table.
In embodiments of the present invention, the data information includes table information, field information etc., and wherein table information can be table name, table class
Type etc., field information can be field name, field type etc..
Second output unit 1046, for exporting acquired data information into default document.Implement in the present invention
In example, data information is output in specified oracle database by Sqoop modes, and forms an Oracle Pkg,
If desired tables of data is optimized, only needs the Pkg for optimizing this Oracle.
Referring to Fig. 10, a kind of corresponding above-mentioned tables of data extracting method based on Shell, the embodiment of the present invention also propose one
Tables of data of the kind based on Shell extracts terminal, which includes：Traversal Unit 201, positioning unit 202, recognition unit
203, extraction unit 204, taxon 205, acquiring unit 206.
Wherein, the Traversal Unit 201, for being traversed to the Shell scripts according to preset keyword.In this hair
In bright embodiment, when being traversed to Shell scripts, be by using the short rule of first ergodic data table creation time, after
The ergodic data table settling time rule of length is traversed, right to which realization is to the traversal of the tables of data of the progress of Shell scripts
Shell scripts carry out being short to the traversal rule that creation time is grown from creation time, can improve the effect handled Shell scripts
Rate.
Positioning unit 202, for being positioned to the tables of data in the Shell scripts according to the result of traversal.At this
In inventive embodiments, position of the tables of data in Shell scripts is shown using the traversing result to the Shell scripts
Come, by the location information of shown tables of data to be positioned to the tables of data in the Shell scripts.
Recognition unit 203, for identification tables of data in Shell scripts.In embodiments of the present invention, the tables of data refers to
Be that database, and the relevant tables of data called out from database are connected by SQL statement in Shell scripts；
In Shell scripts connect database and call tables of data be in order to obtain the data in database, in daily maintenance work,
It can be by obtaining the data in database, to achieve the purpose that certain information in monitoring data library, to further real-time
Solve the performance of equipment.
Identify the tables of data in Shell scripts, it can be by identifying that the keyword in SQL statement be realized, for example, can be with
It is inserted into sentence " insert into " by identification, to identify the tables of data be inserted into sentence and be followed by；Identification can be passed through
Query statement " select*from ", to identify tables of data that query statement is followed by；It can also be by identifying more newspeak
Sentence " update ", to identify the tables of data for updating sentence and being followed by；It can also be by identifying cancel statement " delete
From ", to identify tables of data etc. that cancel statement is followed by.
Extraction unit 204, the table name for extracting the tables of data.In embodiments of the present invention, by identifying Shell
After tables of data in script, the table name of identified tables of data is extracted, for example, being inserted into sentence " insert into
In { TABLENAME } ", the table name for the tables of data extracted is " TABLENAME "；In query statement " select*from
In { USERNAME } ", the table name for the tables of data extracted is " USERNAME ", is carried in update sentence " update { DBNAME } "
The table name for the tables of data got is " DBNAME ", the tables of data extracted in cancel statement " delete from { KBNAME } "
Table name be " KBNAME ".
Taxon 205, for being classified to the tables of data according to the table name extracted, wherein the tables of data packet
Include source table and object table.In embodiments of the present invention, the table name of tables of data is carried out by a series of keyword of SQL statements
After extraction, tables of data table name is stored in temporary file, the type of the tables of data includes source table and object table, wherein
Can be to the method that the type of tables of data is classified according to tables of data table name：If tables of data table name is an independent character
String, and have that space or line feed and table name front follow before and after the respective character string is from keywords, then the number
Type according to table is source table, if tables of data table name is an independent character string, and has space before and after the respective character string or changes
What row and table name front followed is non-from keywords, then the type of the tables of data is object table, optionally, non-from
Keyword can be the SQL statements keyword such as into, update.
Acquiring unit 206, for obtaining the corresponding data information of different types of tables of data, and by acquired inhomogeneity
The data information of type is exported into same default document.In embodiments of the present invention, the JOB that the tables of data is Hadoop is associated with
The JOB contingency tables of table, the Hadoop are write using Hadoop sentences and SQL statement, and are stored in corresponding number
According in library, the table name of the JOB contingency tables of Hadoop is written in corresponding Shell scripts, as the JOB for needing identification Hadoop
When contingency table, the table name in Shell scripts is first extracted, that is, have identified and which JOB contingency table arrived involved in script,
And the type of these JOB contingency tables belongs to source table or object table.
It should be noted that source table refers to the table of table and external relations type database inside Hadoop, the word of source table
Have that space or line feed and table name front follow before and after symbol string is from keywords；Object table is referred to by writing mode
It is divided into and is inserted into object table and coverage goal table, such as insert into tableA, this is exactly plug-type object table, insert
Overwrite tableB, this is exactly the object table of cover type, and the default document can be the data in presetting database
Table, for example, keyword and its related content are captured from the script of JOB contingency tables, and by the content record captured to temporarily
In file, these are completed in the hdfs levels of Hadoop, and the result in temporary file is then loaded into Hadoop's
In Hive tables, data information is output in specified default oracle database by the data in Hive tables by Sqoop modes,
Specifically, exporting data information into the tables of data pre-established in the default oracle database.Optionally, user
Can will be stored with the tables of data of data information pre-established and formed Oracle Pkg (Oracle packaging,
Oracle package files), if desired the tables of data optimizes, and only needs to optimize this Oracle Pkg.
In hardware realization, unit 101 identified above, extraction unit 102, taxon 103, acquiring unit 104 etc. can
To be embedded in the form of hardware or independently of in the device of data processing, data processing equipment can also be stored in a software form
Memory in, execute the corresponding operation of above each unit so that processor calls.The processor can be central processing list
First (CPU), microprocessor, microcontroller etc..
The above-mentioned tables of data extraction terminal based on Shell can be implemented as a kind of form of computer program, computer journey
Sequence can be run on computer equipment as shown in figure 11.
Figure 11 is a kind of structure composition schematic diagram of the tables of data extraction equipment based on Shell of the present invention.The equipment can be with
It is terminal, can also be server, wherein terminal can be smart mobile phone, tablet computer, laptop, desktop computer, a
The electronic device with communication function such as personal digital assistant and wearing formula device.Server can be independent server, also may be used
To be server cluster that multiple servers form.Referring to Fig.1 1, which includes being connected by system bus 501
Processor 502, non-volatile memory medium 503, built-in storage 504 and the network interface 505 connect.Wherein, the computer equipment
500 non-volatile memory medium 503 can storage program area 5031 and computer program 5032,5032 quilt of computer program
When execution, processor 502 may make to execute a kind of tables of data extracting method based on Shell.The processing of the computer equipment 500
Device 502 supports the operation of entire computer equipment 500 for providing calculating and control ability.The built-in storage 504 is non-volatile
Property storage medium 503 in computer program 5032 operation provide environment, when which is executed by processor, can make
It obtains processor 502 and executes a kind of tables of data extracting method based on Shell.The network interface 505 of computer equipment 500 be used for into
Row network communication such as sends the task dispatching of distribution.It will be understood by those skilled in the art that structure shown in Figure 11, only
With the block diagram of the relevant part-structure of application scheme, the computer equipment being applied thereon to application scheme is not constituted
Restriction, specific computer equipment may include than more or fewer components as shown in the figure, or the certain components of combination, or
There is person different components to arrange.
Wherein, the processor 502 executes following operation：
Identify the tables of data in Shell scripts；
Extract the table name of the tables of data；
Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and target
Table；
The corresponding data information of different types of tables of data is obtained, and acquired different types of data information is exported
To in same default document.
In one embodiment, the processor 502 also executes following operation：
The Shell scripts are traversed according to preset keyword；
The tables of data in the Shell scripts is positioned according to the result of traversal.
In one embodiment, described to be classified to the tables of data according to the table name extracted, including：
Determine character string corresponding with the tables of data table name；
Classified to the tables of data according to the character string.
In one embodiment, described to obtain the corresponding data of different types of tables of data if the tables of data is source table
Information, and acquired different types of data information is exported into same default document, including：
The source table is divided into inside sources table and external source table；
Obtain the inside sources table and the corresponding data information of external source table；
Acquired data information is exported into default document.
In one embodiment, described to obtain the corresponding number of different types of tables of data if the tables of data is object table
It is believed that breath, and acquired different types of data information is exported into same default document, including：
The object table is divided into and is inserted into object table and coverage goal table；
Obtain the insertion object table and the corresponding data information of coverage goal table；
Acquired data information is exported into default document.
It will be understood by those skilled in the art that the embodiment of the tables of data extraction equipment based on Shell shown in Figure 11
The restriction to the tables of data extraction equipment specific composition based on Shell is not constituted, in other embodiments, based on Shell's
Tables of data extraction equipment may include either combining certain components or different components than illustrating more or fewer components
Arrangement.For example, in some embodiments, the tables of data extraction equipment based on Shell only includes memory and processor, in this way
Embodiment in, the structure and function of memory and processor are consistent with embodiment illustrated in fig. 11, and details are not described herein.
The present invention provides a kind of computer readable storage medium, computer-readable recording medium storage there are one or one
A procedure above, the one or more programs can be executed by one or more than one processor, with realize with
Lower step：
Identify the tables of data in Shell scripts；
Extract the table name of the tables of data；
Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and target
Table；
The corresponding data information of different types of tables of data is obtained, and acquired different types of data information is exported
To in same default document.
In one embodiment, following steps are also realized：
The Shell scripts are traversed according to preset keyword；
The tables of data in the Shell scripts is positioned according to the result of traversal.
In one embodiment, described to be classified to the tables of data according to the table name extracted, including：
Determine character string corresponding with the tables of data table name；
Classified to the tables of data according to the character string.
In one embodiment, described to obtain the corresponding data of different types of tables of data if the tables of data is source table
Information, and acquired different types of data information is exported into same default document, including：
The source table is divided into inside sources table and external source table；
Obtain the inside sources table and the corresponding data information of external source table；
Acquired data information is exported into default document.
In one embodiment, described to obtain the corresponding number of different types of tables of data if the tables of data is object table
It is believed that breath, and acquired different types of data information is exported into same default document, including：
The object table is divided into and is inserted into object table and coverage goal table；
Obtain the insertion object table and the corresponding data information of coverage goal table；
Acquired data information is exported into default document.
Present invention storage medium above-mentioned includes：Magnetic disc, CD, read-only memory (Read-Only Memory,
The various media that can store program code such as ROM).
Unit in all embodiments of the invention can pass through universal integrated circuit, such as CPU (Central
Processing Unit, central processing unit), or pass through ASIC (Application Specific Integrated
Circuit, application-specific integrated circuit) it realizes.
Step in tables of data extracting method of the embodiment of the present invention based on Shell can progress sequence according to actual needs
It adjusts, merge and deletes.
Unit in tables of data extraction terminal of the embodiment of the present invention based on Shell can be closed according to actual needs
And it divides and deletes.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can readily occur in various equivalent modifications or replace
It changes, these modifications or substitutions should be covered by the protection scope of the present invention.Therefore, protection scope of the present invention should be with right
It is required that protection domain subject to.
Claims (10)
1. a kind of tables of data extracting method based on Shell, which is characterized in that the method includes：
Identify the tables of data in Shell scripts；
Extract the table name of the tables of data；
Classified to the tables of data according to the table name extracted, wherein the tables of data includes source table and object table；
The corresponding data information of different types of tables of data is obtained, and acquired different types of data information is exported to same
In one default document.
2. the method as described in claim 1, which is characterized in that described before the tables of data in the identification Shell scripts
Method further includes：
The Shell scripts are traversed according to preset keyword；
The tables of data in the Shell scripts is positioned according to the result of traversal.
3. the method as described in claim 1, which is characterized in that described to be divided the tables of data according to the table name extracted
Class, including：
Determine character string corresponding with the tables of data table name；
Classified to the tables of data according to the character string.
4. the method as described in claim 1, which is characterized in that if the tables of data is source table, the acquisition is different types of
The corresponding data information of tables of data, and acquired different types of data information is exported into same default document, including：
The source table is divided into inside sources table and external source table；
Obtain the inside sources table and the corresponding data information of external source table；
Acquired data information is exported into default document.
5. the method as described in claim 1, which is characterized in that if the tables of data is object table, the acquisition different type
The corresponding data information of tables of data, and acquired different types of data information is exported into same default document, is wrapped
It includes：
The object table is divided into and is inserted into object table and coverage goal table；
Obtain the insertion object table and the corresponding data information of coverage goal table；
Acquired data information is exported into default document.
6. a kind of tables of data based on Shell extracts terminal, which is characterized in that the terminal includes：
Recognition unit, for identification tables of data in Shell scripts；
Extraction unit, the table name for extracting the tables of data；
Taxon, for being classified to the tables of data according to the table name extracted, wherein the tables of data includes source table
And object table；
Acquiring unit, for obtaining the corresponding data information of different types of tables of data, and by acquired different types of number
It is believed that in breath output to same default document.
7. terminal as claimed in claim 6, which is characterized in that the terminal further includes：
Traversal Unit, for being traversed to the Shell scripts according to preset keyword；
Positioning unit, for being positioned to the tables of data in the Shell scripts according to the result of traversal.
8. terminal as claimed in claim 6, which is characterized in that the taxon, including：
Determination unit, for determining character string corresponding with the tables of data table name；
Classification subelement, for being classified to the tables of data according to the character string.
9. a kind of tables of data extraction equipment based on Shell, which is characterized in that including：
Memory, for storing the program for realizing tables of data extracting method；And
Processor, the program for running the realization tables of data extracting method stored in the memory, to execute as right is wanted
Seek 1-5 any one of them methods.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage there are one or
More than one program, the one or more programs can be executed by one or more than one processor, to realize
Method as described in any one in claim 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2018/101880 WO2019161645A1 (en) | 2018-02-24 | 2018-08-23 | Shell-based data table extraction method, terminal, device, and storage medium |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810156612 | 2018-02-24 | ||
| CN2018101566128 | 2018-02-24 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108536745A true CN108536745A (en) | 2018-09-14 |
| CN108536745B CN108536745B (en) | 2021-03-16 |
Family
ID=63483448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810196485.4A Active CN108536745B (en) | 2018-02-24 | 2018-03-09 | Shell-based data table extraction method, terminal, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108536745B (en) |
| WO (1) | WO2019161645A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109359160A (en) * | 2018-10-12 | 2019-02-19 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN110647564A (en) * | 2019-08-14 | 2020-01-03 | 中国平安财产保险股份有限公司 | Hive table establishing method, electronic device and computer readable storage medium |
| CN111460241A (en) * | 2020-04-26 | 2020-07-28 | 甬矽电子(宁波)股份有限公司 | Data query method and device, electronic equipment and storage medium |
| CN113190603A (en) * | 2021-04-28 | 2021-07-30 | 中国邮政储蓄银行股份有限公司 | Data processing method, data processing device, computer readable storage medium and processor |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111984659B (en) * | 2020-07-28 | 2023-07-21 | 招联消费金融有限公司 | Data updating method, device, computer equipment and storage medium |
| CN116578651B (en) * | 2023-07-12 | 2023-11-17 | 北京集度科技有限公司 | Data table structure synchronization method, system and equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101944128A (en) * | 2010-09-25 | 2011-01-12 | 中兴通讯股份有限公司 | Data export and import method and device |
| US20130060816A1 (en) * | 2011-09-07 | 2013-03-07 | International Business Machines Corporation | Transforming hierarchical language data into relational form |
| US20130173664A1 (en) * | 2011-12-28 | 2013-07-04 | Xiaohui Xue | Mapping non-relational database objects into a relational database model |
| CN104536987A (en) * | 2014-12-08 | 2015-04-22 | 联动优势电子商务有限公司 | Data query method and device |
| CN104866595A (en) * | 2015-05-29 | 2015-08-26 | 北京京东尚科信息技术有限公司 | Method and apparatus for adding transaction control to relational database script |
| CN105868204A (en) * | 2015-01-21 | 2016-08-17 | 中国移动（深圳）有限公司 | Method and apparatus for converting script language SQL of Oracle |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11265293A (en) * | 1998-03-17 | 1999-09-28 | Nec Corp | Script processor |
| CN102375826B (en) * | 2010-08-13 | 2014-12-31 | 中国移动通信集团公司 | Structured query language script analysis method, device and system |
| CN107169023A (en) * | 2017-04-07 | 2017-09-15 | 广东精点数据科技股份有限公司 | Data lineage analysis system and method based on sql semantic automatic analysis |
-
2018
- 2018-03-09 CN CN201810196485.4A patent/CN108536745B/en active Active
- 2018-08-23 WO PCT/CN2018/101880 patent/WO2019161645A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101944128A (en) * | 2010-09-25 | 2011-01-12 | 中兴通讯股份有限公司 | Data export and import method and device |
| US20130060816A1 (en) * | 2011-09-07 | 2013-03-07 | International Business Machines Corporation | Transforming hierarchical language data into relational form |
| US20130173664A1 (en) * | 2011-12-28 | 2013-07-04 | Xiaohui Xue | Mapping non-relational database objects into a relational database model |
| CN104536987A (en) * | 2014-12-08 | 2015-04-22 | 联动优势电子商务有限公司 | Data query method and device |
| CN105868204A (en) * | 2015-01-21 | 2016-08-17 | 中国移动（深圳）有限公司 | Method and apparatus for converting script language SQL of Oracle |
| CN104866595A (en) * | 2015-05-29 | 2015-08-26 | 北京京东尚科信息技术有限公司 | Method and apparatus for adding transaction control to relational database script |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109359160A (en) * | 2018-10-12 | 2019-02-19 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN110647564A (en) * | 2019-08-14 | 2020-01-03 | 中国平安财产保险股份有限公司 | Hive table establishing method, electronic device and computer readable storage medium |
| CN110647564B (en) * | 2019-08-14 | 2023-11-24 | 中国平安财产保险股份有限公司 | Hive table building method, electronic device and computer readable storage medium |
| CN111460241A (en) * | 2020-04-26 | 2020-07-28 | 甬矽电子(宁波)股份有限公司 | Data query method and device, electronic equipment and storage medium |
| CN111460241B (en) * | 2020-04-26 | 2024-01-23 | 甬矽电子(宁波)股份有限公司 | Data query method and device, electronic equipment and storage medium |
| CN113190603A (en) * | 2021-04-28 | 2021-07-30 | 中国邮政储蓄银行股份有限公司 | Data processing method, data processing device, computer readable storage medium and processor |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019161645A1 (en) | 2019-08-29 |
| CN108536745B (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108536745A (en) | Tables of data extracting method, terminal, equipment and storage medium based on Shell | |
| CN105094707B (en) | A kind of data storage, read method and device | |
| US9817876B2 (en) | Enhanced mechanisms for managing multidimensional data | |
| CN108536761A (en) | Report data querying method and server | |
| CN103838672A (en) | Automated testing method and device for all-purpose financial statements | |
| CN109508355A (en) | A kind of data pick-up method, system and terminal device | |
| CN104750472B (en) | The resource package management method and device of a kind of terminal applies | |
| CN107766431B (en) | Parameterization removing function method and system based on grammar parsing | |
| CN109522332A (en) | Customer profile data merging method, device, equipment and readable storage medium storing program for executing | |
| CN108399072A (en) | Five application page update method and device | |
| CN109657177A (en) | The generation method of the page, device, storage medium and computer equipment after upgrading | |
| CN109669933A (en) | Transaction data intelligent processing method, device and computer readable storage medium | |
| CN109857803A (en) | Method of data synchronization, device, equipment, system and computer readable storage medium | |
| CN117238433B (en) | Method for automatically isolating document data based on Libreoffice | |
| CN107832448A (en) | Database operation method, device and equipment | |
| CN102915344B (en) | SQL (structured query language) statement processing method and device | |
| CN109298882A (en) | Management method, computer readable storage medium and the terminal device of interface | |
| CN108776702A (en) | A kind of data make a report on page user-defined visual configuration method | |
| CN104657164B (en) | Software upgrading treating method and apparatus | |
| CN106557307A (en) | The processing method and processing system of business datum | |
| CN106802928B (en) | Power grid historical data management method and system | |
| CN104933077B (en) | Rule-based multifile information analysis method | |
| CN108415998A (en) | Using dependence update method, terminal, equipment and storage medium | |
| CN106599241A (en) | Big data visual management method for GIS software | |
| CN107943912B (en) | A kind of response type Resource TOC data visualization management method, terminal and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |