CN109241033A - The method and apparatus for creating real-time data warehouse - Google Patents

The method and apparatus for creating real-time data warehouse Download PDF

Info

Publication number
CN109241033A
CN109241033A CN201810955493.2A CN201810955493A CN109241033A CN 109241033 A CN109241033 A CN 109241033A CN 201810955493 A CN201810955493 A CN 201810955493A CN 109241033 A CN109241033 A CN 109241033A
Authority
CN
China
Prior art keywords
data
real
database
accessed
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810955493.2A
Other languages
Chinese (zh)
Inventor
张爱芸
王科
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201810955493.2A priority Critical patent/CN109241033A/en
Publication of CN109241033A publication Critical patent/CN109241033A/en
Pending legal-status Critical Current

Links

Abstract

The invention discloses a kind of method and apparatus for creating real-time data warehouse, are related to field of computer technology.One specific embodiment of this method includes: to determine data to be accessed according to preset database configuration information;Wherein, the database configuration information includes at least database name and table name;According to the database name and table name, the data to be accessed are sent in real time in different message queues;By the data real-time storage to be accessed in message queue to data warehouse table, to realize real-time data warehouse.The embodiment can quickly restore the data of database on line, realize that real-time data warehouse reduces the pressure of data query on line to improve data user rate.

Description

The method and apparatus for creating real-time data warehouse
Technical field
The present invention relates to field of computer technology more particularly to a kind of method and apparatus for creating real-time data warehouse.
Background technique
Data warehouse (Data Warehouse) is to provide all types number for the decision-making process of all ranks of enterprise According to the strategy set of support.With the gradually expansion of business, demand of the user to real time data is higher and higher, and more people wish Real-time data can be found.But current data warehouse, what is stored mostly is the off-line data daily processed, i.e. the same day Data can only can just see tomorrow, real-time data can not be provided.If desired data are inquired in real time, it can only be by directly looking into Database on line is ask to realize.
However, at least there are the following problems in the prior art for inventor's discovery in realizing process of the present invention:
When the data volume of inquiry is larger, the pressure of on-line checking can be very big.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of method and apparatus for creating real-time data warehouse, can quickly go back The data of database on former line realize real-time data warehouse, to improve data user rate, and reduce data on line and look into The pressure of inquiry.
To achieve the above object, according to an aspect of an embodiment of the present invention, a kind of creation real-time data warehouse is provided Method, comprising: according to preset database configuration information, determine data to be accessed;Wherein, the database configuration information Including at least database name and table name;According to the database name and data table name, the data to be accessed are sent in real time To in different message queues;It is real-time to realize by the data real-time storage to be accessed in message queue to data warehouse table Data warehouse.
Optionally, according to preset database configuration information, determine that data to be accessed include: according to preset database Configuration information, the log in read line in database, using the log as data to be accessed.
Optionally, the mode of the log is row mode.
Optionally, the database configuration information further includes table structure information;By the number to be accessed in message queue Before real-time storage to database table, the method also includes: according to the table structure information, determine the table of database on line Structure;Create data warehouse table identical with the table structure.
To achieve the above object, according to another aspect of an embodiment of the present invention, a kind of creation real-time data warehouse is provided Device, comprising: data determining module to be accessed determines data to be accessed for according to preset database configuration information; Wherein, the database configuration information includes at least database name and table name;Real time message subscribing module, for according to the number According to library name and data table name, the data to be accessed are sent in real time in different message queues;Real-time storage module is used In by the data real-time storage to be accessed in message queue to data warehouse table, to realize real-time data warehouse.
Optionally, the data determining module to be accessed is also used to: according to preset database configuration information, in read line Log in database, using the log as data to be accessed.
Optionally, the mode of the log is row mode.
Optionally, the database configuration information further includes table structure information;The real-time storage module is also used to: according to The table structure information, determines the table structure of database on line;Create data warehouse table identical with the table structure.
To achieve the above object, according to an embodiment of the present invention in another aspect, providing a kind of electronic equipment, comprising: one A or multiple processors;Storage device, for storing one or more programs, when one or more of programs are one Or multiple processors execute, so that one or more of processors realize the creation real-time data warehouse of the embodiment of the present invention Method.
To achieve the above object, another aspect according to an embodiment of the present invention, provides a kind of computer-readable medium, On be stored with computer program, the creation real-time data warehouse of the embodiment of the present invention is realized when described program is executed by processor Method.
One embodiment in foregoing invention has the following advantages that or the utility model has the advantages that because using according to preset database Configuration information determines data to be accessed;Wherein, the database configuration information includes at least database name and table name;According to The data to be accessed are sent in different message queues by the database name and data table name in real time;By message team Data real-time storage to be accessed to data warehouse table in column can be quick to realize the technological means of real-time data warehouse The data of database on line are restored, real-time data warehouse are realized, so that data user rate is improved, to reduce data on line The pressure of inquiry.
Further effect possessed by above-mentioned non-usual optional way adds hereinafter in conjunction with specific embodiment With explanation.
Detailed description of the invention
Attached drawing for a better understanding of the present invention, does not constitute an undue limitation on the present invention.Wherein:
Fig. 1 is the schematic diagram of the main flow of the method for creation real-time data warehouse according to an embodiment of the present invention;
Fig. 2 is the schematic diagram of the main modular of the device of creation real-time data warehouse according to an embodiment of the present invention;
Fig. 3 is that the embodiment of the present invention can be applied to exemplary system architecture figure therein;
Fig. 4 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention Figure.
Specific embodiment
Below in conjunction with attached drawing, an exemplary embodiment of the present invention will be described, including the various of the embodiment of the present invention Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize It arrives, it can be with various changes and modifications are made to the embodiments described herein, without departing from scope and spirit of the present invention.Together Sample, for clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.
Fig. 1 is the schematic diagram of the main flow of the method for creation real-time data warehouse according to an embodiment of the present invention, such as Fig. 1 It is shown, this method comprises:
Step S101: according to preset database configuration information, data to be accessed are determined;Wherein, the database is matched Confidence breath includes at least database name and table name;
Step S102: according to the database name and data table name, the data to be accessed are sent to difference in real time Message queue in;
Step S103: by the data real-time storage to be accessed in message queue to data warehouse table, to realize number in real time According to warehouse.
For step S101, user can be pre-configured with the configuration information for needing the database (such as MySQL) from line, such as IP, port numbers, database name, table name and table structure information (such as field information), access way (table, standard scale are divided in a point library) Deng.Configuration file can be generated based on the configuration information, data to be accessed can be determined according to configuration file.
Specifically, determining that the process of data to be accessed may include steps of:
According to preset database configuration information, log in read line in database, using the log as to be accessed Data.
Wherein, log can refer to the binlog log of MySQL database.Wherein, binlog is a binary format File, for recording user to the SQL statement information of database update, such as change database table and the SQL for changing content Sentence (Structured Query Language, structured query language) can all be recorded in binlog, therefore in binlog What is stored is all the change message of database.
For step S102, in practical applications, quantity on line in database may more than one, such as can will not It stores with the data that service line generates into database on multiple and different lines, or can be by same service line different time The data of generation or the storage of different types of data are into database on different lines.Therefore, it creates in the present embodiment more The theme of a message queue, each message queue is different, and theme can be corresponding with data source, specifically, theme can be with number Corresponding according to library, more specifically, theme can be corresponding with the table in database.Therefore, in the present embodiment, can according to The source for accessing data, which is sent in message queue corresponding with its source, specifically, can basis Data to be accessed from database name, which is sent in corresponding message queue, more specifically, can be with According to data to be accessed from database name and data table name, which is sent to corresponding message queue In.
As specific example, message queue can be realized by kafka.Wherein, kafka is a kind of point of high-throughput Cloth distribution subscription message system, it can handle the everything flow data in the website of consumer's scale.
In an alternate embodiment of the invention, the mode of the log is row mode (row mode).
Under row mode, log will record the form that every data line is modified, and then at the end slave, (from end) is right again Identical data are modified, and the data to be modified only are recorded, and only value (value) does not have the case where multilist is associated with.? Under row mode, the context-sensitive information of the SQL statement of execution can not be recorded in log, only needs to record that Item record be modified, what is modified as, thus the log content under row mode can be perfectly clear record every a line The details of data modification.Therefore, using the log of row mode, the data of database on line can preferably be restored.
For step S103, can be determined on line according to the table structure information in preset database configuration information Then the table structure of database creates data warehouse table identical with the table structure, the data to be accessed in message queue are real When store into data warehouse table, to create real-time data warehouse.
The method of the creation real-time data warehouse of the embodiment of the present invention, can quickly restore the data of database on line, real Existing real-time data warehouse reduces the pressure of data query on line to improve data user rate.
As specific example, real-time data warehouse can (Hadoop be a distributed system basis using Hadoop Framework) technological frame, it can be good at the inquiry and calculating of supporting big data quantity, compared to database on direct information trunk (MySQL) for, the pressure of data query is greatly reduced.
Since data access data warehouse in real time, data mart modeling is not yet carried out, therefore in inquiry, user needs to pass through position Point, timestamp, major key, subregion cooperation, to be inquired.Wherein, site refers to the character string identification of record data read time. Timestamp refers to the number of seconds passed through since on January 1st, 1970 (midnight of UTC/GMT), does not consider leap second.
The method of the embodiment of the present invention can be regarded as two tasks on the whole.First task is to access task in real time, The task is used to need the data to be accessed of access data warehouse to be in real time sent in different message queues database on line. Second part is real-time distributed tasks, for the data to be accessed in message queue to be sent in database table, is realized real-time Data warehouse.
As specific example, above-mentioned real-time access task and real-time distributed tasks can use Fregata and k8s reality It is existing.Fregata is a lightweight, and super fast, the frame on a large scale based on the machine learning based on spark, it is provided The scalaAPI (application programming interface write using the programming language of more normal forms) of high level.Fregata is considered as It is a real-time data imputing system management platform, it is believed that be the pipe of a series of generated tasks, table during data access Platform is a management client.K8s, that is, Kubernetes is the Open Source Platform for automating container operation, these operation packets It includes and is extended between deployment, scheduling and node cluster.K8s can be regarded as bottom and be responsible for the system layers such as task execution, resource allocation The logic of database data to be accessed in client, such as crawl line, the logic etc. for reading binlog is deployed on k8s Mirror image script can all go to pull script in k8s when the task in Fregata executes every time, executed in k8s and processing, knot Fruit and operation monitoring are the clients shown in fregata.Real-time access task and real-time distributed tasks operate on k8s, The unnecessary wasting of resources is not will cause using only the resource of needs so as to which dynamically dilatation can be carried out to application.
Fig. 2 is the schematic diagram of the main modular of the device 200 of creation real-time data warehouse according to an embodiment of the present invention, such as Shown in Fig. 2, which includes:
Data determining module 201 to be accessed, for determining data to be accessed according to preset database configuration information; Wherein, the database configuration information includes at least database name and table name;
Real time message subscribing module 202 is used for according to the database name and data table name, by the data to be accessed It is sent in different message queues in real time;
Real-time storage module 203, for by the data real-time storage to be accessed in message queue to data warehouse table, with Realize real-time data warehouse.
Optionally, the data determining module to be accessed 201 is also used to: according to preset database configuration information, being read Log on line in database, using the log as data to be accessed.
Optionally, the mode of the log is row mode.
Optionally, the database configuration information further includes table structure information;The real-time storage module is also used to: according to The table structure information, determines the table structure of database on line;Create data warehouse table identical with the table structure.
The device of the realization real-time data warehouse of the embodiment of the present invention, can quickly restore the data of database on line, real Existing real-time data warehouse reduces the pressure of data query on line to improve data user rate.
Method provided by the embodiment of the present invention can be performed in above-mentioned apparatus, has the corresponding functional module of execution method and has Beneficial effect.The not technical detail of detailed description in the present embodiment, reference can be made to method provided by the embodiment of the present invention.
The device of above-mentioned creation real-time data warehouse may operate on k8s, so as to can dynamically to application into Row dilatation not will cause the unnecessary wasting of resources using only the resource of needs.
Fig. 3 shows the method that can apply the creation real-time data warehouse of the embodiment of the present invention or creation real time data storehouse The exemplary system architecture 300 of the device in library.
As shown in figure 3, system architecture 300 may include terminal device 301,302,303, network 304 and server 305. Network 304 between terminal device 301,302,303 and server 305 to provide the medium of communication link.Network 304 can be with Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be used terminal device 301,302,303 and be interacted by network 304 with server 305, to receive or send out Send message etc..Various telecommunication customer end applications, such as the application of shopping class, net can be installed on terminal device 301,302,303 The application of page browsing device, searching class application, instant messaging tools, mailbox client, social platform software etc..
Terminal device 301,302,303 can be the various electronic equipments with display screen and supported web page browsing, packet Include but be not limited to smart phone, tablet computer, pocket computer on knee and desktop computer etc..
Server 305 can be to provide the server of various services, such as utilize terminal device 301,302,303 to user The shopping class website browsed provides the back-stage management server supported.Back-stage management server can believe the product received The data such as breath inquiry request carry out the processing such as analyzing, and processing result (such as target push information, product information) is fed back to Terminal device.
It should be noted that the method for creation real-time data warehouse is generally by server provided by the embodiment of the present invention 305 execute, and correspondingly, realize that the device of real-time data warehouse is generally positioned in server 305.
It should be understood that the number of terminal device, network and server in Fig. 3 is only schematical.According to realization need It wants, can have any number of terminal device, network and server.
Below with reference to Fig. 4, it illustrates the computer systems 400 for the terminal device for being suitable for being used to realize the embodiment of the present invention Structural schematic diagram.Terminal device shown in Fig. 4 is only an example, function to the embodiment of the present invention and should not use model Shroud carrys out any restrictions.
As shown in figure 4, computer system 400 includes central processing unit (CPU) 401, it can be read-only according to being stored in Program in memory (ROM) 402 or be loaded into the program in random access storage device (RAM) 403 from storage section 408 and Execute various movements appropriate and processing.In RAM 403, also it is stored with system 400 and operates required various programs and data. CPU 401, ROM 402 and RAM 403 are connected with each other by bus 404.Input/output (I/O) interface 405 is also connected to always Line 404.
I/O interface 405 is connected to lower component: the importation 406 including keyboard, mouse etc.;It is penetrated including such as cathode The output par, c 407 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.;Storage section 408 including hard disk etc.; And the communications portion 409 of the network interface card including LAN card, modem etc..Communications portion 409 via such as because The network of spy's net executes communication process.Driver 410 is also connected to I/O interface 405 as needed.Detachable media 411, such as Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 410, in order to read from thereon Computer program be mounted into storage section 408 as needed.
Particularly, disclosed embodiment, the process described above with reference to flow chart may be implemented as counting according to the present invention Calculation machine software program.For example, embodiment disclosed by the invention includes a kind of computer program product comprising be carried on computer Computer program on readable medium, the computer program include the program code for method shown in execution flow chart.? In such embodiment, which can be downloaded and installed from network by communications portion 409, and/or from can Medium 411 is dismantled to be mounted.When the computer program is executed by central processing unit (CPU) 401, system of the invention is executed The above-mentioned function of middle restriction.
It should be noted that computer-readable medium shown in the present invention can be computer-readable signal media or meter Calculation machine readable storage medium storing program for executing either the two any combination.Computer readable storage medium for example can be --- but not Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.Meter The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, just of one or more conducting wires Taking formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only storage Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device, Or above-mentioned any appropriate combination.In the present invention, computer readable storage medium can be it is any include or storage journey The tangible medium of sequence, the program can be commanded execution system, device or device use or in connection.And at this In invention, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal, Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for By the use of instruction execution system, device or device or program in connection.Include on computer-readable medium Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc. are above-mentioned Any appropriate combination.
Flow chart and block diagram in attached drawing are illustrated according to the system of various embodiments of the invention, method and computer journey The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction It closes to realize.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard The mode of part is realized.Described module also can be set in the processor, for example, can be described as: a kind of processor packet It includes sending module, obtain module, determining module and first processing module.Wherein, the title of these modules is under certain conditions simultaneously The restriction to the unit itself is not constituted, for example, sending module is also described as " sending picture to the server-side connected The module of acquisition request ".
As on the other hand, the present invention also provides a kind of computer-readable medium, which be can be Included in equipment described in above-described embodiment;It is also possible to individualism, and without in the supplying equipment.Above-mentioned calculating Machine readable medium carries one or more program, when said one or multiple programs are executed by the equipment, makes Obtaining the equipment includes:
According to preset database configuration information, data to be accessed are determined;Wherein, the database configuration information is at least Including database name and table name;
According to the database name and table name, the data to be accessed are sent in real time in different message queues;
By the data real-time storage to be accessed in message queue to data warehouse table, to realize real-time data warehouse.
The technical solution of the embodiment of the present invention can quickly restore the data of database on line, realize real-time data warehouse, To improve data user rate, the pressure of data query on line is reduced.
Above-mentioned specific embodiment, does not constitute a limitation on the scope of protection of the present invention.Those skilled in the art should be bright It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and substitution can occur.It is any Made modifications, equivalent substitutions and improvements etc. within the spirit and principles in the present invention, should be included in the scope of the present invention Within.

Claims (10)

1. a kind of method for creating real-time data warehouse characterized by comprising
According to preset database configuration information, data to be accessed are determined;Wherein, the database configuration information includes at least Database name and table name;
According to the database name and table name, the data to be accessed are sent in real time in different message queues;
By the data real-time storage to be accessed in message queue to data warehouse table, to realize real-time data warehouse.
2. the method according to claim 1, wherein being determined to be accessed according to preset database configuration information Data include:
According to preset database configuration information, log in read line in database, using the log as number to be accessed According to.
3. according to the method described in claim 2, it is characterized in that, the mode of the log is row mode.
4. according to the method described in claim 2, it is characterized in that, the database configuration information further includes table structure information;
Before by the data real-time storage to database table to be accessed in message queue, the method also includes:
According to the table structure information, the table structure of database on line is determined;
Create data warehouse table identical with the table structure.
5. a kind of device for creating real-time data warehouse characterized by comprising
Data determining module to be accessed, for determining data to be accessed according to preset database configuration information;Wherein, institute Database configuration information is stated including at least database name and table name;
Real time message subscribing module, for according to the database name and data table name, the data to be accessed to be sent out in real time It send into different message queues;
Real-time storage module, for by the data real-time storage to be accessed in message queue to data warehouse table, to realize reality When data warehouse.
6. device according to claim 5, which is characterized in that the data determining module to be accessed is also used to:
According to preset database configuration information, log in read line in database, using the log as number to be accessed According to.
7. device according to claim 6, which is characterized in that the mode of the log is row mode.
8. device according to claim 6, which is characterized in that the database configuration information further includes table structure information;
The real-time storage module is also used to:
According to the table structure information, the table structure of database on line is determined;
Create data warehouse table identical with the table structure.
9. a kind of electronic equipment characterized by comprising
One or more processors;
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real The now method as described in any in claim 1-4.
10. a kind of computer-readable medium, is stored thereon with computer program, which is characterized in that described program is held by processor The method as described in any in claim 1-4 is realized when row.
CN201810955493.2A 2018-08-21 2018-08-21 The method and apparatus for creating real-time data warehouse Pending CN109241033A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810955493.2A CN109241033A (en) 2018-08-21 2018-08-21 The method and apparatus for creating real-time data warehouse

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810955493.2A CN109241033A (en) 2018-08-21 2018-08-21 The method and apparatus for creating real-time data warehouse

Publications (1)

Publication Number Publication Date
CN109241033A true CN109241033A (en) 2019-01-18

Family

ID=65071328

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810955493.2A Pending CN109241033A (en) 2018-08-21 2018-08-21 The method and apparatus for creating real-time data warehouse

Country Status (1)

Country Link
CN (1) CN109241033A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110032554A (en) * 2019-04-10 2019-07-19 北京字节跳动网络技术有限公司 Management method, device, storage medium and the electronic equipment of data warehouse table
WO2021208774A1 (en) * 2020-04-17 2021-10-21 第四范式(北京)技术有限公司 Method and apparatus for assisting machine learning model to go online

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101105793A (en) * 2006-07-11 2008-01-16 阿里巴巴公司 Data processing method and system of data library
CN105243067A (en) * 2014-07-07 2016-01-13 北京明略软件系统有限公司 Method and apparatus for realizing real-time increment synchronization of data
CN105989163A (en) * 2015-03-04 2016-10-05 中国移动通信集团福建有限公司 Data real-time processing method and system
CN106354434A (en) * 2016-08-31 2017-01-25 中国人民大学 Log data storing method and system
CN107402963A (en) * 2017-06-20 2017-11-28 阿里巴巴集团控股有限公司 Search for construction method, the method for pushing and device and equipment of incremental data of data
CN107590158A (en) * 2016-07-08 2018-01-16 北京京东尚科信息技术有限公司 A kind of method and apparatus for obtaining data source modification information
CN107679097A (en) * 2017-09-08 2018-02-09 广州汉邮通信有限公司 A kind of distributed data processing method, system and storage medium
CN107704590A (en) * 2017-09-30 2018-02-16 深圳市华傲数据技术有限公司 A kind of data processing method and system based on data warehouse
CN107704597A (en) * 2017-10-13 2018-02-16 携程旅游网络技术(上海)有限公司 Relevant database to Hive ETL script creation methods
CN107958082A (en) * 2017-12-15 2018-04-24 杭州有赞科技有限公司 Offline increment synchronization method and system of the database to data warehouse
EP3332336A1 (en) * 2015-08-05 2018-06-13 AB Initio Technology LLC Selecting queries for execution on a stream of real-time data

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101105793A (en) * 2006-07-11 2008-01-16 阿里巴巴公司 Data processing method and system of data library
CN105243067A (en) * 2014-07-07 2016-01-13 北京明略软件系统有限公司 Method and apparatus for realizing real-time increment synchronization of data
CN105989163A (en) * 2015-03-04 2016-10-05 中国移动通信集团福建有限公司 Data real-time processing method and system
EP3332336A1 (en) * 2015-08-05 2018-06-13 AB Initio Technology LLC Selecting queries for execution on a stream of real-time data
CN107590158A (en) * 2016-07-08 2018-01-16 北京京东尚科信息技术有限公司 A kind of method and apparatus for obtaining data source modification information
CN106354434A (en) * 2016-08-31 2017-01-25 中国人民大学 Log data storing method and system
CN107402963A (en) * 2017-06-20 2017-11-28 阿里巴巴集团控股有限公司 Search for construction method, the method for pushing and device and equipment of incremental data of data
CN107679097A (en) * 2017-09-08 2018-02-09 广州汉邮通信有限公司 A kind of distributed data processing method, system and storage medium
CN107704590A (en) * 2017-09-30 2018-02-16 深圳市华傲数据技术有限公司 A kind of data processing method and system based on data warehouse
CN107704597A (en) * 2017-10-13 2018-02-16 携程旅游网络技术(上海)有限公司 Relevant database to Hive ETL script creation methods
CN107958082A (en) * 2017-12-15 2018-04-24 杭州有赞科技有限公司 Offline increment synchronization method and system of the database to data warehouse

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110032554A (en) * 2019-04-10 2019-07-19 北京字节跳动网络技术有限公司 Management method, device, storage medium and the electronic equipment of data warehouse table
CN110032554B (en) * 2019-04-10 2022-04-01 北京字节跳动网络技术有限公司 Management method and device of database table, storage medium and electronic equipment
WO2021208774A1 (en) * 2020-04-17 2021-10-21 第四范式(北京)技术有限公司 Method and apparatus for assisting machine learning model to go online

Similar Documents

Publication Publication Date Title
CN107844324A (en) Customer terminal webpage redirects treating method and apparatus
CN108510081A (en) machine learning method and platform
CN110310034A (en) A kind of service orchestration applied to SaaS, business flow processing method and apparatus
CN109036425A (en) Method and apparatus for operating intelligent terminal
CN109726094A (en) The method and apparatus of pressure test
CN109905286A (en) A kind of method and system of monitoring device operating status
CN107506218A (en) The management method and management system of a kind of configuration file
CN109002440A (en) Method, apparatus and system for big data multidimensional analysis
CN109189835A (en) The method and apparatus of the wide table of data are generated in real time
CN109241033A (en) The method and apparatus for creating real-time data warehouse
CN108804327A (en) A kind of method and apparatus of automatic Data Generation Test
CN107861933A (en) The method and apparatus for generating O&M form
CN109683998A (en) Internationalize implementation method, device and system
CN110019350A (en) Data query method and apparatus based on configuration information
CN110427304A (en) O&M method, apparatus, electronic equipment and medium for banking system
CN110347654A (en) A kind of method and apparatus of online cluster features
CN108920222A (en) A kind of method and device for business processing of rule-based engine
CN102185863A (en) Intelligent data interactive publishing system and method between server and client
CN110019258A (en) The method and apparatus for handling order data
CN109117420A (en) operation log recording method and device
CN110472207A (en) List generation method and device
CN109960212A (en) Task sending method and device
CN109614603A (en) Method and apparatus for generating information
CN109992495A (en) The method and apparatus of interface testing
CN110019539A (en) A kind of method and apparatus that the data of data warehouse are synchronous

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20190118

RJ01 Rejection of invention patent application after publication