CN108984757A - A kind of data lead-in method and equipment - Google Patents
A kind of data lead-in method and equipment Download PDFInfo
- Publication number
- CN108984757A CN108984757A CN201810790544.0A CN201810790544A CN108984757A CN 108984757 A CN108984757 A CN 108984757A CN 201810790544 A CN201810790544 A CN 201810790544A CN 108984757 A CN108984757 A CN 108984757A
- Authority
- CN
- China
- Prior art keywords
- data
- imported
- thread
- file
- importing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The purpose of the application is to provide a kind of data lead-in method, and the application is by scanning first and determining data to be imported and its affiliated platform relevant information；Corresponding transformation rule when determining that data import according to the platform relevant information；Finally, the data to be imported are written to target storage table according to the transformation rule, it realizes and passes through transformation rule being automatically imported determining data to be imported to target storage table, the problem of not only avoiding the low efficiency artificially imported in available data importing process, it also improves data and imports efficiency, data to be imported different simultaneously correspond to different transformation rules, so as to improve the robustness of data importing.
Description
Technical field
This application involves computer field more particularly to a kind of data lead-in method and equipment.
Background technique
With the basic link of system engineering and automatic control when computer progress data processing.This data processing method
Through the every field of social production and social life, development and application greatly affect human social development into
Journey.During computer processing data, importing data is an essential link
In existing traditional data importing process, often just for the file of ad hoc rules, progress is artificially led manually
Enter, lead to traditional data importing process inefficiency, robustness is low.
Summary of the invention
The purpose of the application is to provide a kind of data lead-in method and equipment, is imported with solving data in the prior art
Low efficiency and the low problem of robustness.
According to the one aspect of the application, a kind of data lead-in method is provided, this method comprises:
Scan and determine data to be imported and its affiliated platform relevant information；
Corresponding transformation rule when determining that data import according to the platform relevant information；
The data to be imported are written to target storage table according to the transformation rule.
Further, in above-mentioned data lead-in method, the scanning simultaneously determines that data to be imported and its affiliated platform are related
Information, comprising:
Starting scanning thread, by the scanning thread in real time/timing scan at least one target directory under file,
In, the file under each target directory is used to indicate the initial data that the correspondence platform of storage provides；
When the file of preset data type is arrived in scanning under the target directory, data to be imported and its affiliated flat are determined
Platform relevant information.
Further, in above-mentioned data lead-in method, the starting scans thread, real-time/fixed by the scanning thread
When scan file under at least one target directory before, further includes:
Judge to whether there is task in execution under the target directory,
If it does not exist, then start scanning thread；
If it exists, then to the task execution in execution after carry out starting scanning thread.
Further, described scanning to arrive preset data type under the target directory in above-mentioned data lead-in method
File when, determine data to be imported and its affiliated platform relevant information, comprising:
When scanning is to the corresponding file of preset data type under the target directory, by it is under the target directory,
File corresponding with the preset data type is determined as the data to be imported, and determines the affiliated platform of data to be imported
Relevant information.
Further, described that the data to be imported are written according to the transformation rule in above-mentioned data lead-in method
To before target storage table, further includes:
It based on the data to be imported and its affiliated platform relevant information, establishes data and imports task, while by the number
State to be imported is set as according to importing task；
Data importing task is inserted into form of a message and is imported in message queue, is imported with triggering the data
Task, wherein it includes the data to be imported that the data, which import task,.
Further, described that the data to be imported are written according to the transformation rule in above-mentioned data lead-in method
To target storage table, comprising:
Starting imports thread, executes the data by the importing thread and imports task；
The data to be imported are written to the target storage table according to the transformation rule.
Further, described that the data to be imported are written according to the transformation rule in above-mentioned data lead-in method
To target storage table, comprising:
Starting imports thread, is found out from the message queue by the importing thread and imports task with the data
Corresponding message, and the corresponding data of the message are imported into task and are updated to execution state；
Load corresponding transformation rule when data import；
The data to be imported are written to the target storage table according to the transformation rule.
Further, in above-mentioned data lead-in method, the transformation rule includes file verification rule and data mapping rule
Then,
It is wherein, described that the data to be imported are written to the target storage table according to the transformation rule, comprising:
Based on the file verification rule to file under the target directory, corresponding with the preset data type into
Row file verification, and the state that the file is arranged is preparation state；
Content authentication is carried out to the data mapping ruler based on the parameter information for obtaining the file；
When the file verification and content authentication pass through, then according to the data mapping ruler by the number to be imported
According to write-in to the target storage table.
Further, in above-mentioned data lead-in method, it is described according to the data mapping ruler by the data to be imported
It is written to the target storage table, further includes:
The data to be imported and its corresponding related information are written to the mesh according to the data mapping ruler
Mark storage table.
Further, described according to the conversion if the data to be imported are compressed package in above-mentioned data lead-in method
The data to be imported are written to target storage table rule further include:
Processing will be unziped it to compress data to be imported existing for packet form；
The data to be imported obtained after decompression are written to target storage table according to the transformation rule；
After the completion of write-in, the data to be imported obtained after the decompression are deleted, and by the data after the completion of write-in
It is backed up.
Further, in above-mentioned data lead-in method, the method also includes:
Creation for data import scanning thread and at least one import thread and for monitor the scanning thread with
Import a listening thread of thread；
By the listening thread, the monitoring log imported for real-time monitored data is obtained；
Wherein, mutually indepedent between the scanning thread, importing thread and listening thread.
According to the another aspect of the application, a kind of non-volatile memory medium is additionally provided, being stored thereon with computer can
Reading instruction when the computer-readable instruction can be executed by processor, makes the processor realize such as above-mentioned data importing side
Method.
It according to the another aspect of the application, additionally provides a kind of data and imports equipment, wherein the equipment includes:
One or more processors；
Non-volatile memory medium, for storing one or more computer-readable instructions,
When one or more of computer-readable instructions are executed by one or more of processors, so that one
Or multiple processors realize such as above-mentioned data lead-in method.
Compared with prior art, the application is by scanning first and determining data to be imported and its affiliated platform correlation letter
Breath；Corresponding transformation rule when determining that data import according to the platform relevant information；Finally, according to the transformation rule by institute
It states data to be imported to be written to target storage table, realizes and pass through transformation rule being automatically imported determining data to be imported to mesh
The problem of marking storage table, not only avoiding the low efficiency artificially imported in available data importing process, also improves data and leads
Enter efficiency, while different data to be imported correspond to different transformation rules, so as to improve the robustness of data importing.Into
One step, the application also creates for the scanning thread of data importing and at least one importing thread and for monitoring the scanning
Thread and the listening thread for importing thread, wherein the scanning thread imports between thread and listening thread independently of each other,
By the listening thread, to obtain the monitoring log imported for real-time monitored data, not only may be implemented to import data
The monitoring of the real time status of process, the different times that can also be imported by the monitoring log real-time recorded data, so as to rear
Continuous data can be accurately located out fault by the monitoring log when importing error, and then accelerate to solve in data importing process
Error.
Detailed description of the invention
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon:
Fig. 1 shows a kind of flow diagram of data lead-in method according to the application one aspect.
The same or similar appended drawing reference represents the same or similar component in attached drawing.
Specific embodiment
The application is described in further detail with reference to the accompanying drawing.
In a typical configuration of this application, terminal, the equipment of service network and trusted party include one or more
Processor (CPU), input/output interface, network interface and memory.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/or
The forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flashRAM).Memory is showing for computer-readable medium
Example.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer readable instructions, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory (SRAM), moves
State random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electric erasable
Programmable read only memory (EEPROM), flash memory or other memory techniques, read-only disc read only memory (CD-ROM) (CD-ROM),
Digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or
Any other non-transmission medium, can be used for storage can be accessed by a computing device information.As defined in this article, computer
Readable medium does not include non-temporary computer readable media (transitory media), such as the data-signal and carrier wave of modulation.
Fig. 1 shows a kind of data lead-in method according to the application one aspect, in multi-platform data docking operation
In, by obtaining from multiple platforms in order to realize that the data of data docking are directed into the data importing process of unified database
In, the method comprising the steps of S11, step S12 and step S13, wherein
The step S11 is scanned and is determined data to be imported and its affiliated platform relevant information；Here, the platform phase
Closing information includes the information such as platform names, Platform Type and platform release；Wherein, the Platform Type can be including but it is unlimited
It is the ERP systems such as Oracle EBS platform, SAP, Kingdee and UF in including.
The step S12, corresponding transformation rule when determining that data import according to the platform relevant information；
The data to be imported are written to target storage table according to the transformation rule by the step S13.
S11 to step S13 realizes the leading automatically by determining data to be imported by transformation rule through the above steps
The problem of entering to target storage table, not only avoiding the low efficiency artificially imported in available data importing process, also improves
Data import efficiency, while different data to be imported correspond to different transformation rules, so as to improve the Shandong of data importing
Stick.
In embodiments herein, the step S11 scans and determines data to be imported and its affiliated platform correlation letter
Breath, comprising:
Starting scanning thread, by the scanning thread in real time/timing scan at least one target directory under file,
In, the file under each target directory is used to indicate the initial data that the correspondence platform of storage provides；Here, each platform
There are its corresponding mating interfaces during carrying out data docking for type, then under target directory: identical data docking
When the corresponding file of mating interface, there is the initial data obtained from a platform by mating interface under this document；When need
When carrying out data importing to the data in the file under target directory, then needs to start scanning thread, be opened for executing at any time
The scan task of dynamic real-time/timing is led with scanning from the file under one or more target directories to needing to carry out data
The file entered determines data to be imported and its affiliated when the file of preset data type is arrived in scanning under the target directory
Platform relevant information determines corresponding data to be imported and its affiliated platform phase when needing to carry out data importing by scanning thread
Close information.Here, the preset data type can be include but is not limited to be include the original being used to indicate from multi-platform middle acquisition
Beginning data state (such as when obtaining successfully, file suffixes .ok；When obtaining failure, file suffixes is .error etc.), and
It is used to indicate from the initial data of multi-platform middle acquisition and whether malfunctions or omit, it, can be from mesh by the preset data type
The successful file of data acquisition can then be determined as to be imported by the lower scanning of heading record to file corresponding when data acquisition success
Data avoid scanning from causing to the data for obtaining failure or error subsequent to ensure correctness when subsequent progress data importing
Data, which import, introduces a chain of mistake.
Due under the target directory that is scanning there are when task in execution, if scanning the file under the target directory
When, the operations such as possible task in execution can be updated the file under the target directory, modify lead to the target scanned
There are errors for file under catalogue, if or the task in execution executed when scanning first time, and carry out mesh second
When the lower file scan of heading record, if the task in execution is still in commission, the file under the target directory is in the presence of being weighed
The case where multiple scanning, causes to clash, in order to avoid above-mentioned error and conflict, in one embodiment of the application, the step
Starting scanning thread in S11, by the scanning thread in real time/timing scan at least one target directory under file before,
Further include:
Judge with the presence or absence of task in execution under the target directory, if it does not exist when any task in execution, then
All Files of the starting scanning thread to be scanned under target directory；If it exists, then complete to the task execution in execution
Bi Hou, so that being avoided there is no starting scanning thread is carried out when the task in being carrying out again to target under the target directory
The error and conflict that file under catalogue is likely to occur when being scanned.
Not depositing in real time steps S11 when under the target directory of confirmation scanning is explained further in the form of code below
In any task in execution, just starting scanning thread carries out the file under in real time/timing scan target directory:
SELECT dpc.value1, dci.source_sys_key
FROM dc_custom_info dci, dc_param_configs dpc
WHERE
Dci.enable_flag='Y'
AND dpc.enabled_flag='Y'
AND dci.source_sys_key=dpc.source_key
AND dpc.CODE='INBOUND'
AND dpc.value1IS NOT NULL
AND dpc.value1<>”
AND dci.source_sys_key NOT IN(
SELECT
dptq.source_sys_key
FROM
dc_process_task_queue dptq
WHERE
Dptq.task_type_code='ALL2DC'
AND dptq.enabled_flag='Y'
AND dptq.source_sys_key=dci.source_sys_key
AND dptq.status_code IN(
' RUNNING',
' ERROR',
'PENDING'
)
Certainly, the All Files under at least one target directory are swept by scanning thread in the step S11
When retouching and determining data to be imported, it is that target directory of at least one carries out concurrent scan to this, realizes while scanning all
File under target directory accelerates the progress that data import, to improve rapidly to scan and determine data to be imported
Data import efficiency.
In the present embodiment, the text worked as the scanning under the target directory and arrive preset data type in the step S11
When part, data to be imported and its affiliated platform relevant information are determined, comprising:
When scanning is to the corresponding file of preset data type under the target directory, by it is under the target directory,
File corresponding with the preset data type is determined as the data to be imported, and determines the affiliated platform of data to be imported
Relevant information.
For example, the data type of the file under target directory can have successfully obtains successful initial data at platform
Corresponding file can also have and obtain corresponding file when omitting or being wrong occur in initial data from platform, therefore in order to accurate
Ground is got for the subsequent valid data for carrying out data importing, then when carrying out the scanning of All Files under target directory, if
When scanning the corresponding file of existing preset data type under target directory, then by it is under the target directory, with the present count
It is determined as data to be imported (for example, being .ok's by the suffix of all storage files under target directory according to the corresponding file of type
File is determined as data to be imported, and suffix is that the appearance such as .error are omitted or the file of mistake is not confirmed as number to be imported then
According to), and the affiliated platform relevant information being provided according to file under the target directory, corresponding with the preset data type,
So as to subsequent transformation rule when importing for data, Jin Ershi can be determined according to the corresponding affiliated platform relevant information of this document
Now the accurate and effective of the file under target directory is imported.
In one embodiment of the application, the data to be imported are written to mesh according to the transformation rule by the step S13
Before mark storage table, further includes:
It based on the data to be imported and its affiliated platform relevant information, establishes data and imports task, while by the number
State to be imported is set as according to importing task；
Data importing task is inserted into form of a message and is imported in message queue, is imported with triggering the data
Task, wherein it includes the data to be imported that the data, which import task,.
For example, according to determining data to be imported and its affiliated platform relevant information (such as platform names, Platform Type and
Platform release etc.) it establishes the corresponding data of the data to be imported and imports task, meanwhile, it sets data importing task to wait lead
Enter state pending, then data importing task is inserted into message queue in form of a message, so that the data import
Task is inserted into message queue in form of a message, and then can be triggered by detecting in message queue there are when message
The data import task, hence into the data lead-in stage to data to be imported.
Ought scan under the target directory to default in real time steps S13 is explained further in the form of code below
When the corresponding file of data type, file under the target directory, corresponding with the preset data type is determined as institute
Data to be imported are stated, and establishes data and imports task, while setting state to be imported for data importing task
Pending:
Importing in real time steps S13 based on data creation data to be imported is explained further in the form of code below
After task, which is imported into task and is inserted into importing message queue in form of a message:
RabbitContent rabbitContent=new RabbitContent ()；
Map<String, Object>msgCotent=new HashMap<String, Object>()；；
MsgCotent.put (" sourceSysKey ", task.getSysKey ())；
MsgCotent.put (" taskQueueID ", task.gettaskQueueID ())；
MsgCotent.put (" processGroupID ", task.getprocessID ())；
MsgCotent.put (" processNode ", -1)；
rabbitContent.set MsgCotent(msgCotent)；
rabbitContent.setExchange(GlobalInfo.TOPIC_DATAIMP_EXCHANGE_BASE)；
rabbitContent.setRoutingKey(GlobalInfo.TOPIC_DATAIMP_QUOTA)；
Rabbit.sendTopicMessage(rabbitContent)；
Log.info (" SendMqToOwner is sent to the MQ of OWNER are as follows: "+task.toString ())；
In one embodiment of the application, the data to be imported are written to mesh according to the transformation rule by the step S13
Mark storage table, comprising:
Starting imports thread, executes the data by the importing thread and imports task；
The data to be imported are written to the target storage table according to the transformation rule.
Here, the importing thread is independent from each other with scanning thread, wherein the scanning thread is that real-time/timing is right
File under all target directories carries out in real time/timing scan, which is in real-time or start by set date state, and
During the importing thread is then the message in real-time reception message queue, when in message queue there are when message, just meeting
Starting imports thread at once, is found out from the message queue by the importing thread corresponding with data importing task
Message, and the corresponding data of the message are imported into task and are updated to execution state running；And according to the number to be imported
The data to be imported are written into target storage mark table the transformation rule determined according to affiliated platform relevant information, and realization, which is treated, to be led
Enter the importing of data.
Be explained further in the form of code below in the step S13 by the importing thread from the message team
Message corresponding with data importing task is found out in column, and the corresponding data of the message are imported into task and are updated
To execute state:
Then the present embodiment loads the data to be imported in the parameter information etc. of the target directory of its affiliated platform, is somebody's turn to do
Parameter information mainly includes loading the data to be imported corresponding transformation rule when data import, wherein the transformation rule can
To carry out the file verification rule of file verification for the filename to the file under the target directory and for number to be imported
According to the data mapping ruler etc. for carrying out content map, after the transformation rule when having loaded data to be imported and having carried out data importing,
The data to be imported are written to the target storage table according to the transformation rule, realize the importing to data to be imported.
Further, corresponding transformation rule is illustrated when being imported in the form of code to the load data below:
Then the present embodiment, the data to be imported being written to institute according to the transformation rule in the step S13
State target storage table, comprising:
Target directory corresponding to the data importing task is scanned again, based on this document verification rule to the target mesh
File under record, corresponding with the preset data type carries out file verification, for verifying in rule match with this document
File, the state that this document is arranged is preparation state, that is, prepares to enter the corresponding data to be imported of this document into data importing
Preparation state；The difference for remembering the mapping ruler of the target storage table mapped to according to required for file, by data pair to be imported
The file answered is grouped, to take the work for being grouped concurrent mode while carrying out data importing.Then, it loads out and corresponds to
The detailed rules information of data mapping ruler corresponding to group, and generated according to the detailed rules information of the data mapping ruler
It is inserted into sentence, such as:
SELECT
distinct ditd.file_column_sequence,
ditd.file_column_name,
ditd.file_column_type,
ditd.file_column_default_value,
ditd.table_column_name,
ditd.table_column_default_value,
ditd.org_type,
ditd.table_column_sequence
FROM
dc_interface_table_master ditm,
dc_interface_table_details ditd
WHERE
Ditm.id=ditd.interface_table_id
AND ditm.interface_code=?
order BY ditd.table_column_sequence；
Then, the parameter information (such as the character strings such as storage time of file) based on the acquisition file is to the data
Mapping ruler carries out content authentication, to guarantee the accuracy of the data mapping ruler；When the file verification and content authentication are equal
By when, then the data to be imported are written to the target storage table according to the data mapping ruler.
In the present embodiment, if the data to be imported are compressed package, the step S13 will according to the data mapping ruler
The data to be imported are written to the target storage table further include:
Processing will be unziped it to compress data to be imported existing for packet form；
The data to be imported obtained after decompression are written to target storage table according to the transformation rule；
After the completion of All Files corresponding etc. the data to be imported are written into, will be obtained after the decompression wait lead
Enter data (decompressing files) deletion, and the data (importing successful file) after the completion of write-in are backed up, and is stored in standby
It is deleted in file, will not only have been introduced into successful data to be imported, saves memory source, will also import successfully
Data backed up with guarantee import after the completion of data effective storage.
Further, the data to be imported being written to institute according to the data mapping ruler in the step S13
State target storage table, further includes:
The data to be imported and its corresponding related information are written to the mesh according to the data mapping ruler
Mark storage table.Wherein, the related information includes the affiliated platform of the data to be imported, there are data friendships with the data to be imported
The tissue corresponding with the data to be imported such as mutual one or more platforms and the application scenarios of the data to be imported and tissue
Relationship.Below in the form of code further in the step S13 by related information corresponding to the data to be imported
Write-in to the target storage table explains:
When S13 succeeds and do not malfunction to the importing of data to be imported and its related information through the above steps, it can establish
Cleaning task simultaneously starts, to realize importing and cleaning to data to be imported.
Data lead-in method in one embodiment of the application further include: the scanning thread and at least that creation is imported for data
One imports thread and the listening thread for monitoring the scanning thread and importing thread, wherein the scanning thread is led
Enter between thread and listening thread independently of each other, by the listening thread, to obtain the prison imported for real-time monitored data
Log is listened, the monitoring of the real time status to data importing process not only may be implemented, can also be remembered in real time by the monitoring log
The different times that data import are recorded, so that when follow-up data is imported and malfunctioned error can be accurately located by the monitoring log
Place, and then accelerate to solve the error in data importing process.
In addition, another embodiment of the application additionally provides a kind of non-volatile memory medium, being stored thereon with computer can
Reading instruction when the computer-readable instruction can be executed by processor, makes the processor realize such as above-mentioned data lead-in method.
In addition, another embodiment of the application, which additionally provides a kind of data, imports equipment, wherein the equipment includes:
One or more processors；
Non-volatile memory medium, for storing one or more computer-readable instructions, when one or more of meters
Calculation machine readable instruction is executed by one or more of processors, so that one or more of processors realize such as above-mentioned data
Introductory technique.
Here, the detailed content of each embodiment in the equipment imported for data, for details, reference can be made to this for counting
According to the corresponding part of introduction method embodiment, here, repeating no more.
In conclusion the application is by scanning first and determining data to be imported and its affiliated platform relevant information；According to
The platform relevant information determines corresponding transformation rule when data import；Finally, according to the transformation rule by described wait lead
Enter data to be written to target storage table, realizes and stored being automatically imported to target for determining data to be imported by transformation rule
Table, the problem of not only avoiding the low efficiency artificially imported in available data importing process, also improve data and import efficiency,
Data to be imported different simultaneously correspond to different transformation rules, so as to improve the robustness of data importing.Further,
The application also create for data import scanning thread and at least one import thread and for monitor the scanning thread with
Import a listening thread of thread, wherein the scanning thread imports between thread and listening thread independently of each other, to pass through
The listening thread obtains the monitoring log imported for real-time monitored data, not only may be implemented to data importing process
The monitoring of real time status, the different times that can also be imported by the monitoring log real-time recorded data, so as in follow-up data
Fault can be accurately located out when importing error by the monitoring log, and then accelerates to solve going out in data importing process
It is wrong.
It should be noted that the application can be carried out in the assembly of software and/or software and hardware, for example, can adopt
With specific integrated circuit (ASIC), general purpose computer or any other realized similar to hardware device.In one embodiment
In, the software program of the application can be executed to implement the above steps or functions by processor.Similarly, the application
Software program (including relevant data structure) can be stored in computer readable recording medium, for example, RAM memory,
Magnetic or optical driver or floppy disc and similar devices.In addition, hardware can be used to realize in some steps or function of the application, example
Such as, as the circuit cooperated with processor thereby executing each step or function.
In addition, a part of the application can be applied to computer program product, such as computer program instructions, when its quilt
When computer executes, by the operation of the computer, it can call or provide according to the present processes and/or technical solution.
And the program instruction of the present processes is called, it is possibly stored in fixed or moveable recording medium, and/or pass through
Broadcast or the data flow in other signal-bearing mediums and transmitted, and/or be stored according to described program instruction operation
In the working storage of computer equipment.Here, including a device according to one embodiment of the application, which includes using
Memory in storage computer program instructions and processor for executing program instructions, wherein when the computer program refers to
When enabling by processor execution, method and/or skill of the device operation based on aforementioned multiple embodiments according to the application are triggered
Art scheme.
It is obvious to a person skilled in the art that the application is not limited to the details of above-mentioned exemplary embodiment, Er Qie
In the case where without departing substantially from spirit herein or essential characteristic, the application can be realized in other specific forms.Therefore, no matter
From the point of view of which point, the present embodiments are to be considered as illustrative and not restrictive, and scope of the present application is by appended power
Benefit requires rather than above description limits, it is intended that all by what is fallen within the meaning and scope of the equivalent elements of the claims
Variation is included in the application.Any reference signs in the claims should not be construed as limiting the involved claims.This
Outside, it is clear that one word of " comprising " does not exclude other units or steps, and odd number is not excluded for plural number.That states in device claim is multiple
Unit or device can also be implemented through software or hardware by a unit or device.The first, the second equal words are used to table
Show title, and does not indicate any particular order.
Claims (13)
1. a kind of data lead-in method, wherein the described method includes:
Scan and determine data to be imported and its affiliated platform relevant information；
Corresponding transformation rule when determining that data import according to the platform relevant information；
The data to be imported are written to target storage table according to the transformation rule.
2. according to the method described in claim 1, wherein, the scanning simultaneously determines data to be imported and its affiliated platform correlation letter
Breath, comprising:
Starting scanning thread, by the scanning thread in real time/timing scan at least one target directory under file, wherein
File under each target directory is used to indicate the initial data that the correspondence platform of storage provides；
When the file of preset data type is arrived in scanning under the target directory, data to be imported and its affiliated platform phase are determined
Close information.
3. the starting scans thread according to the method described in claim 2, wherein, by the scanning thread in real time/periodically
Before scanning the file under at least one target directory, further includes:
Judge to whether there is task in execution under the target directory,
If it does not exist, then start scanning thread；
If it exists, then to the task execution in execution after carry out starting scanning thread.
4. according to the method described in claim 2, wherein, described ought scan under the target directory arrives preset data type
When file, data to be imported and its affiliated platform relevant information are determined, comprising:
When scanning file corresponding to preset data type under the target directory, by under the target directory and institute
It states the corresponding file of preset data type and is determined as the data to be imported, and determine that the affiliated platform of data to be imported is related
Information.
5. according to the method described in claim 2, wherein, it is described according to the transformation rule by the data to be imported be written to
Before target storage table, further includes:
It based on the data to be imported and its affiliated platform relevant information, establishes data and imports task, while the data being led
Enter task and is set as state to be imported；
Data importing task is inserted into form of a message and is imported in message queue, is appointed with triggering the data and importing
Business, wherein it includes the data to be imported that the data, which import task,.
6. according to the method described in claim 5, wherein, it is described according to the transformation rule by the data to be imported be written to
Target storage table, comprising:
Starting imports thread, executes the data by the importing thread and imports task；
The data to be imported are written to the target storage table according to the transformation rule.
7. according to the method described in claim 6, wherein, it is described according to the transformation rule by the data to be imported be written to
Target storage table, comprising:
Starting imports thread, is found out from the message queue by the importing thread corresponding with data importing task
Message, and the corresponding data of the message are imported into task and are updated to execution state；
Load corresponding transformation rule when data import；
The data to be imported are written to the target storage table according to the transformation rule.
8. according to the method described in claim 7, wherein, the transformation rule includes file verification rule and data mapping rule
Then,
It is wherein, described that the data to be imported are written to the target storage table according to the transformation rule, comprising:
Text is carried out to file under the target directory, corresponding with the preset data type based on the file verification rule
Part verification, and the state that the file is arranged is preparation state；
Content authentication is carried out to the data mapping ruler based on the parameter information for obtaining the file；
When the file verification and content authentication pass through, then the data to be imported are write according to the data mapping ruler
Enter to the target storage table.
9. described to be write the data to be imported according to the data mapping ruler according to the method described in claim 8, wherein
Enter to the target storage table, further includes:
The data to be imported and its corresponding related information are written to the target according to the data mapping ruler and are deposited
Store up table.
10. described according to the conversion if the data to be imported are compressed package according to the method described in claim 9, wherein
The data to be imported are written to target storage table rule further include:
Processing will be unziped it to compress data to be imported existing for packet form；
The data to be imported obtained after decompression are written to target storage table according to the transformation rule；
After the completion of write-in, the data to be imported obtained after the decompression are deleted, and the data after the completion of write-in are carried out
Backup.
11. according to claim 1 to method described in Arbitrary Term in 10, wherein the method also includes:
Creation is for the scanning thread of data importing and at least one importing thread and for monitoring the scanning thread and importing
One listening thread of thread；
By the listening thread, the monitoring log imported for real-time monitored data is obtained；
Wherein, mutually indepedent between the scanning thread, importing thread and listening thread.
12. a kind of non-volatile memory medium, is stored thereon with computer-readable instruction, the computer-readable instruction can be located
When managing device execution, the processor is made to realize the method as described in any one of claims 1 to 11.
13. a kind of data import equipment, wherein the equipment includes:
One or more processors；
Non-volatile memory medium, for storing one or more computer-readable instructions,
When one or more of computer-readable instructions are executed by one or more of processors, so that one or more
A processor realizes the method as described in any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810790544.0A CN108984757A (en) | 2018-07-18 | 2018-07-18 | A kind of data lead-in method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810790544.0A CN108984757A (en) | 2018-07-18 | 2018-07-18 | A kind of data lead-in method and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108984757A true CN108984757A (en) | 2018-12-11 |
Family
ID=64549633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810790544.0A Pending CN108984757A (en) | 2018-07-18 | 2018-07-18 | A kind of data lead-in method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108984757A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008264A (en) * | 2019-03-04 | 2019-07-12 | 广州易朋软件有限公司 | The collecting method and device of Cost Accounting System |
| CN110222009A (en) * | 2019-05-28 | 2019-09-10 | 咪咕文化科技有限公司 | A kind of Hive storage abnormal document automatic processing method and device |
| WO2021238902A1 (en) * | 2020-05-25 | 2021-12-02 | 中兴通讯股份有限公司 | Data import method and apparatus, service platform, and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103699967A (en) * | 2013-12-19 | 2014-04-02 | 国家电网公司 | Panoramic information platform based on electric network regulation and control integration |
| CN105335412A (en) * | 2014-07-31 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Method and device for data conversion and data migration |
| CN105426502A (en) * | 2015-11-26 | 2016-03-23 | 福州大学 | Social network based person information search and relational network drawing method |
| CN107506451A (en) * | 2017-08-28 | 2017-12-22 | 泰康保险集团股份有限公司 | abnormal information monitoring method and device for data interaction |
| CN107766456A (en) * | 2017-09-27 | 2018-03-06 | 山东浪潮通软信息科技有限公司 | Data parsing converting system and method based on file interaction between supplementary pension system |
| CN108108949A (en) * | 2016-11-25 | 2018-06-01 | 杭州王道科技有限公司 | Speed passage through customs method and the platform of information synergism |
-
2018
- 2018-07-18 CN CN201810790544.0A patent/CN108984757A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103699967A (en) * | 2013-12-19 | 2014-04-02 | 国家电网公司 | Panoramic information platform based on electric network regulation and control integration |
| CN105335412A (en) * | 2014-07-31 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Method and device for data conversion and data migration |
| CN105426502A (en) * | 2015-11-26 | 2016-03-23 | 福州大学 | Social network based person information search and relational network drawing method |
| CN108108949A (en) * | 2016-11-25 | 2018-06-01 | 杭州王道科技有限公司 | Speed passage through customs method and the platform of information synergism |
| CN107506451A (en) * | 2017-08-28 | 2017-12-22 | 泰康保险集团股份有限公司 | abnormal information monitoring method and device for data interaction |
| CN107766456A (en) * | 2017-09-27 | 2018-03-06 | 山东浪潮通软信息科技有限公司 | Data parsing converting system and method based on file interaction between supplementary pension system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008264A (en) * | 2019-03-04 | 2019-07-12 | 广州易朋软件有限公司 | The collecting method and device of Cost Accounting System |
| CN110008264B (en) * | 2019-03-04 | 2020-12-25 | 广州易朋软件有限公司 | Data acquisition method and device of cost accounting system |
| CN110222009A (en) * | 2019-05-28 | 2019-09-10 | 咪咕文化科技有限公司 | A kind of Hive storage abnormal document automatic processing method and device |
| CN110222009B (en) * | 2019-05-28 | 2021-08-06 | 咪咕文化科技有限公司 | Method and device for automatically processing Hive warehousing abnormal file |
| WO2021238902A1 (en) * | 2020-05-25 | 2021-12-02 | 中兴通讯股份有限公司 | Data import method and apparatus, service platform, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109656934B (en) | Source Oracle database DDL synchronization method and device based on log analysis | |
| CN109271326B (en) | Cloud database testing method and device, equipment and storage medium thereof | |
| WO2019047482A1 (en) | Application program compatibility testing method, apparatus, computer device and storage medium | |
| CA3131079A1 (en) | Test case generation method and device, computer equipment and storage medium | |
| CN110457311B (en) | Method, server and storage medium for automatically generating reconciliation document | |
| WO2019085466A1 (en) | Association test method and system, application server, and computer readable storage medium | |
| CN108984757A (en) | A kind of data lead-in method and equipment | |
| US8862937B2 (en) | Method and system for migrating data from multiple sources | |
| CN113448862B (en) | Software version testing method and device and computer equipment | |
| CN109741022B (en) | Method, device, storage medium and equipment for sharing electronic government front-end information | |
| CN108874441B (en) | Board card configuration method, device, server and storage medium | |
| WO2019148657A1 (en) | Method for testing associated environments, electronic device and computer readable storage medium | |
| CN107357721B (en) | Method and device for testing system | |
| CN107766354A (en) | A kind of method and apparatus for being used to ensure data correctness | |
| CN116599881A (en) | Cloud platform tenant modeling test method, device, equipment and storage medium | |
| CN111078529B (en) | Client writing module testing method and device and electronic equipment | |
| CN111125743B (en) | Authority management method, system, computer device and computer readable storage medium | |
| CN111694724B (en) | Test method and device of distributed form system, electronic equipment and storage medium | |
| US20120317073A1 (en) | Replication Support for Procedures with Arguments of Unsupported Types | |
| CN113704123B (en) | Interface testing method, device, equipment and storage medium | |
| CN114896161A (en) | File construction method and device based on artificial intelligence, computer equipment and medium | |
| CN114969215A (en) | Method for automatically loading data in batches on data warehouse source layer and related equipment | |
| CN114064678A (en) | Event data processing method and device and terminal equipment | |
| CN108415814B (en) | Method for automatically recording field change, application server and computer readable storage medium | |
| CN115705297A (en) | Code call detection method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181211 |