WO2021169268A1 - Data processing method, apparatus and device, and storage medium - Google Patents

Data processing method, apparatus and device, and storage medium Download PDF

Info

Publication number
WO2021169268A1
WO2021169268A1 PCT/CN2020/117504 CN2020117504W WO2021169268A1 WO 2021169268 A1 WO2021169268 A1 WO 2021169268A1 CN 2020117504 W CN2020117504 W CN 2020117504W WO 2021169268 A1 WO2021169268 A1 WO 2021169268A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
update
database
query
log
Prior art date
Application number
PCT/CN2020/117504
Other languages
French (fr)
Chinese (zh)
Inventor
黄祥铮
李钊
万书武
李均
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2021169268A1 publication Critical patent/WO2021169268A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2308—Concurrency control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A data processing method, comprising the following steps: when a data update of a source database is detected, collecting an update log when the source database is updated, and parsing the update log to extract update data in the update log (S10); when the update data is of a first data type, adding the update data into a preset message queue, processing the update data in the preset message queue by means of a streaming data engine to obtain standard data, and synchronizing the standard data to a first database (S20); when the update data is of a second data type, encapsulating the update data and synchronizing the encapsulated update data to a second database in the memory (S30); and when a data query request is received, acquiring a query sentence corresponding to the data query request, and querying the first database or the second database to acquire target data corresponding to the query sentence (S40).

Description

数据处理方法、装置、设备和存储介质Data processing method, device, equipment and storage medium
本申请要求于2020年2月28日提交中国专利局、申请号为202010133454.1，发明名称为“数据处理方法、装置、设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on February 28, 2020, the application number is 202010133454.1, and the invention title is "data processing methods, devices, equipment and storage media", the entire content of which is incorporated by reference In this application.
技术领域Technical field
本申请涉及数据处理领域，尤其涉及数据处理方法、装置、设备和存储介质。This application relates to the field of data processing, in particular to data processing methods, devices, equipment and storage media.
背景技术Background technique
随着大数据技术的发展，越来越多个业务平台都有各自的业务数据，为了统一管理各个业务平台产生的业务数据，需要进行业务数据的同步。With the development of big data technology, more and more business platforms have their own business data. In order to uniformly manage the business data generated by each business platform, it is necessary to synchronize the business data.
发明人意识到，当前数据同步主要采用以下两种方式：方式一固定周期（例如，每天凌晨）对源数据库进行数据同步，这样的数据同步方法存在滞后性，用户不可以实时地查询到更新数据；方式二通过高频度查询源数据库来进行数据同步，这种方式对源数据库侵入性较高，只能识别增量数据，检测不到删除与更新数据，数据完整性达不到需求。The inventor realizes that the current data synchronization mainly adopts the following two methods: Method 1: The source database is synchronized at a fixed period (for example, every morning). This data synchronization method has lag, and users cannot query the updated data in real time. ; Method 2 performs data synchronization by frequently querying the source database. This method is more intrusive to the source database and can only identify incremental data. Delete and update data cannot be detected, and data integrity cannot meet the requirements.
技术解决方案Technical solutions
本申请的主要目的在于提供一种数据处理方法、装置、设备和存储介质，旨在解决当前数据同步时完整性和实时性不兼顾的技术问题。The main purpose of this application is to provide a data processing method, device, equipment, and storage medium, aiming to solve the current technical problem of inconsistent data synchronization with integrity and real-time performance.
为实现上述目的，本申请提供数据处理方法，所述数据处理方法包括以下步骤：In order to achieve the above objective, this application provides a data processing method, which includes the following steps:
在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；When an update of the source database data is detected, collect an update log when the source database is updated, and parse the update log to extract update data in the update log;
在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；When the update data is of the first data type, the update data is added to a preset message queue, the update data in the preset message queue is processed by a streaming data engine to obtain standard data, and the standard Data is synchronized to the first database;
在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；When the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory;
在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。When a data query request is received, a query statement corresponding to the data query request is obtained, the first database or the second database is queried, and the target data corresponding to the query statement is obtained.
此外，为实现上述目的，本申请还提供一种数据处理装置，所述数据处理装置包括：In addition, in order to achieve the above objective, this application also provides a data processing device, the data processing device including:
请求接收模块，用于在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；The request receiving module is configured to collect the update log when the source database is updated when the source database data update is detected, and parse the update log to extract the update data in the update log;
第一同步模块，用于在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；The first synchronization module is configured to add the update data to the preset message queue when the update data is of the first data type, and process the update data in the preset message queue through the streaming data engine to obtain a standard Data, and synchronize the standard data to the first database;
第二同步模块，用于在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；The second synchronization module is configured to encapsulate the update data and synchronize the encapsulated update data to the second database in the memory when the update data is of the second data type;
数据查询模块，用于在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。The data query module is configured to obtain the query sentence corresponding to the data query request, query the first database or the second database, and obtain the target data corresponding to the query sentence when the data query request is received.
此外，为实现上述目的，本申请还提供一种数据处理设备，所述数据处理设备包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述计算机程序被所述处理器执行时实现如下步骤：In addition, in order to achieve the above object, the present application also provides a data processing device, the data processing device includes: a memory, a processor, and a computer program stored in the memory and running on the processor, the When the computer program is executed by the processor, the following steps are implemented:
在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；When an update of the source database data is detected, collect an update log when the source database is updated, and parse the update log to extract update data in the update log;
在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；When the update data is of the first data type, the update data is added to a preset message queue, the update data in the preset message queue is processed by a streaming data engine to obtain standard data, and the standard Data is synchronized to the first database;
在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；When the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory;
在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。When a data query request is received, a query statement corresponding to the data query request is obtained, the first database or the second database is queried, and the target data corresponding to the query statement is obtained.
此外，为实现上述目的，本申请还提供计算机存储介质，所述计算机存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如下步骤：In addition, in order to achieve the above object, the present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；When an update of the source database data is detected, collect an update log when the source database is updated, and parse the update log to extract update data in the update log;
在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；When the update data is of the first data type, the update data is added to a preset message queue, the update data in the preset message queue is processed by a streaming data engine to obtain standard data, and the standard Data is synchronized to the first database;
在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；When the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory;
在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。When a data query request is received, a query statement corresponding to the data query request is obtained, the first database or the second database is queried, and the target data corresponding to the query statement is obtained.
本申请实施例提出的一种数据处理方法、装置、设备和存储介质，数据处理设备通过解析更新日志提取更新日志中的更新数据，与轮询数据同步的方式相比，减少了对源数据库的影响，实现了源数据库无感知的数据同步操作；在更新数据为第一数据类型时，数据处理设备将更新数据添加到预设消息队列中，通过流式数据引擎处理预设消息队列中的更新数据进行数据同步，实现了数据的匀速同步，在更新数据为第二数据类型时，数据处理设备封装更新数据并将封装后的更新数据进行同步，即，数据处理设备根据更新数据的数据类型进行分流同步，减小了数据处理设备的硬件压力，避免了预设消息队列中更新数据过量导致资源不足的情况发生，在接收到数据查询请求时，数据处理设备获取数据查询请求对应的查询语句，查询第一数据库或第二数据库，获取查询语句对应的目标数据，实现了数据实时更新快速查询。The data processing method, device, device, and storage medium proposed in the embodiments of this application. The data processing device extracts update data in the update log by parsing the update log. Compared with the polling data synchronization method, the data processing device reduces the amount of data on the source database. Influence, realize the data synchronization operation without perception of the source database; when the update data is the first data type, the data processing device adds the update data to the preset message queue, and processes the update in the preset message queue through the streaming data engine The data is synchronized with data to achieve uniform synchronization of data. When the updated data is of the second data type, the data processing device encapsulates the updated data and synchronizes the encapsulated updated data, that is, the data processing device performs data processing according to the data type of the updated data. Shunt synchronization reduces the hardware pressure of the data processing equipment, and avoids the occurrence of insufficient resources due to excessive update data in the preset message queue. When a data query request is received, the data processing equipment obtains the query statement corresponding to the data query request. The first database or the second database is queried, and the target data corresponding to the query sentence is obtained, which realizes real-time data update and fast query.
附图说明Description of the drawings
图1是本申请实施例方案涉及的硬件运行环境的装置结构示意图；FIG. 1 is a schematic diagram of a device structure of a hardware operating environment involved in a solution of an embodiment of the present application;
图2为本申请数据处理方法第一实施例的流程示意图；2 is a schematic flowchart of the first embodiment of the data processing method of this application;
图3为本申请数据处理装置一实施例的功能模块示意图。FIG. 3 is a schematic diagram of functional modules of an embodiment of a data processing device according to the present application.
本发明的实施方式Embodiments of the present invention
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.
如图1所示，图1是本申请实施例方案涉及的硬件运行环境的服务器（又叫数据处理设备，其中，数据处理设备可以是由单独的数据处理装置构成，也可以是由其他装置与数据处理装置组合形成）结构示意图。As shown in Figure 1, Figure 1 is a hardware operating environment server (also called a data processing device) involved in the embodiment of the present application. The data processing device can be composed of a separate data processing device, or it can be composed of other devices and The data processing device is combined to form a schematic diagram of the structure.
本申请实施例服务器指一个管理资源并为用户提供服务的计算机，通常分为文件服务器、数据库服务器和应用程序服务器。运行以上软件的计算机或计算机系统也被称为服务器。相对于普通PC(personal computer)个人计算机来说，服务器在稳定性、安全性、性能等方面都要求较高；如图1所示，该服务器可以包括：处理器1001，例如中央处理器（Central Processing Unit，CPU），网络接口1004，用户接口1003，存储器1005，通信总线1002、芯片组、磁盘系统、网络等硬件等。其中，通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏（Display）、输入单元比如键盘（Keyboard），可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口（如无线保真WIreless-FIdelity，WIFI接口）。存储器1005可以是高速随机存取存储器（random access memory，RAM），也可以是稳定的存储器（non-volatile memory），例如磁盘存储器。存储器1005可选的还可以是独立于前述处理器1001的存储装置。The server in the embodiment of the present application refers to a computer that manages resources and provides services for users, and is generally divided into a file server, a database server, and an application server. The computer or computer system running the above software is also called a server. Compared with ordinary PC (personal computer) personal computers, the server has higher requirements in terms of stability, security, performance, etc.; as shown in Figure 1, the server may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), network interface 1004, user interface 1003, memory 1005, communication bus 1002, chipset, disk system, network and other hardware. Among them, the communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as WIreless-FIdelity, WIFI interface). The memory 1005 may be a high-speed random access memory (random access memory, RAM), or a stable memory (non-volatile memory), such as a magnetic disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
可选地，服务器还可以包括摄像头、RF（Radio Frequency，射频）电路，传感器、音频电路、WiFi模块；输入单元，比显示屏，触摸屏；网络接口可选除无线接口中除WiFi外，蓝牙、探针等。本领域技术人员可以理解，图1中示出的服务器结构并不构成对服务器的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。Optionally, the server may also include a camera, RF (Radio Frequency, radio frequency) circuit, sensor, audio circuit, WiFi module; input unit, display screen, touch screen; network interface can be selected in addition to wireless interface in addition to WiFi, Bluetooth, probe, etc. Those skilled in the art can understand that the server structure shown in FIG. 1 does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or a combination of certain components, or different component arrangements.
如图1所示，该计算机软件产品存储在一个存储介质（存储介质：又叫计算机存储介质、计算机介质、可读介质、可读存储介质、计算机可读存储介质或者直接叫介质等，存储介质可以是非易失性可读存储介质，如RAM、磁碟、光盘）中，包括若干指令用以使得一台终端设备（可以是手机，计算机，服务器，空调器，或者网络设备等）执行本申请各个实施例所述的方法，作为一种计算机存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及计算机程序。As shown in Figure 1, the computer software product is stored in a storage medium (storage medium: also called computer storage medium, computer medium, readable medium, readable storage medium, computer readable storage medium, or directly called medium, etc., storage medium It can be a non-volatile readable storage medium, such as RAM, magnetic disk, optical disk, and includes several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute this application In the method described in each embodiment, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a computer program.
在图1所示的服务器中，网络接口1004主要用于连接后台数据库，与后台数据库进行数据通信；用户接口1003主要用于连接客户端（客户端，又叫用户端或终端，本申请实施例终端可以固定终端也可以是移动终端，如，带联网功能的智能空调、智能电灯、智能电源、智能音箱、自动驾驶汽车、PC、智能手机、平板电脑、电子书阅读器、便携计算机等，终端中包含传感器比如光传感器、运动传感器以及其他传感器，在此不再赘述），与客户端进行数据通信；而处理器1001可以用于调用存储器1005中存储的计算机程序，并执行本申请以下实施例提供的数据处理方法中的步骤。In the server shown in FIG. 1, the network interface 1004 is mainly used to connect to a back-end database and perform data communication with the back-end database; the user interface 1003 is mainly used to connect to a client (client, also called a client or terminal, in this embodiment of the application) The terminal can be a fixed terminal or a mobile terminal, such as smart air conditioners with networking functions, smart lights, smart power supplies, smart speakers, self-driving cars, PCs, smart phones, tablets, e-book readers, portable computers, etc., terminals It contains sensors such as light sensors, motion sensors and other sensors, which will not be repeated here), to communicate with the client; and the processor 1001 can be used to call the computer program stored in the memory 1005 and execute the following embodiments of the application Provide the steps in the data processing method.
基于上述硬件运行环境提出了本申请数据处理方法的实施例。An embodiment of the data processing method of the present application is proposed based on the above hardware operating environment.
参照图2，在本申请数据处理方法的第一实施例中，所述数据处理方法包括：2, in the first embodiment of the data processing method of the present application, the data processing method includes:
步骤S10，在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据。Step S10, when the source database data update is detected, the update log when the source database is updated is collected, and the update log is parsed to extract the update data in the update log.
本实施例中的数据处理方法应用于数据处理设备，数据处理设备是数据存储平台和数据查询平台的硬件载体，数据存储平台是可视化批流数据接入平台，支持调度批处理作业和实时流式作业，可以实时启动，监控，或停止批处理/实时作业；数据查询平台是可视化实时报表分析平台，支持不同实时计算引擎（计算引擎例如SparkSQL、Kylin或Phoenix）进行分析查询。The data processing method in this embodiment is applied to a data processing device. The data processing device is a hardware carrier of a data storage platform and a data query platform. The data storage platform is a visual batch stream data access platform that supports scheduling batch processing jobs and real-time streaming Jobs can be started, monitored, or stopped batch processing/real-time jobs in real time; the data query platform is a visual real-time report analysis platform that supports different real-time computing engines (computing engines such as SparkSQL, Kylin or Phoenix) for analysis and query.
数据处理设备与源数据库通信连接，源数据库可以理解为业务数据库，通常情况下的源数据库为结构化的关系型数据库数据，数据处理设备负责将源数据库中的数据同步到数据存储平台，以支持数据查询平台进行数据的快速查询准确分析，具体地：The data processing equipment communicates with the source database. The source database can be understood as a business database. Under normal circumstances, the source database is structured relational database data. The data processing equipment is responsible for synchronizing the data in the source database to the data storage platform to support The data query platform conducts fast query and accurate analysis of data, specifically:
数据处理设备实时地监测与自身通信连接的源数据库，即，数据处理设备获取源数据库操作指令，数据处理设备判断源数据库操作指令是否为删除指令、添加指令或者修改指令，若源数据库操作指令是删除指令、添加指令或者修改指令，数据处理设备判定源数据库更新，数据处理设备在检测到源数据库数据更新时，数据处理设备采集源数据库更新时的更新日志。The data processing equipment monitors the source database that is in communication with itself in real time, that is, the data processing equipment obtains the source database operating instructions, and the data processing equipment determines whether the source database operating instructions are delete instructions, add instructions, or modify instructions. If the source database operating instructions are Delete instructions, add instructions or modify instructions, the data processing device determines that the source database is updated, and when the data processing device detects that the source database is updated, the data processing device collects the update log when the source database is updated.
数据处理设备对更新日志进行解析，获取更新日志中的更新数据，即，数据处理设备获取更新日志中的源数据库操作指令，及源数据库操作指令对应的更新数据；数据处理设备获取更新数据的数据类型，数据类型的划分标准不作具体限定，例如，数据处理设备将更新数据划分为文本数据和数值数据；或者数据处理设备将更新数据划分为业务类型和非业务类型，本实施例中数据处理设备将更新数据按照数据量的划分为第一数据类型和第二数据类型，数据处理设备根据更新数据的数据类型，确定更新数据的同步路径，具体地：The data processing device parses the update log and obtains the update data in the update log. That is, the data processing device obtains the source database operation instruction in the update log and the update data corresponding to the source database operation instruction; the data processing device obtains the update data data Type, the data type classification standard is not specifically limited. For example, the data processing device divides the update data into text data and numerical data; or the data processing device divides the update data into business type and non-business type. In this embodiment, the data processing device The update data is divided into a first data type and a second data type according to the amount of data, and the data processing device determines the synchronization path of the update data according to the data type of the update data, specifically:
步骤S20，在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库。Step S20: When the update data is of the first data type, add the update data to a preset message queue, process the update data in the preset message queue through a streaming data engine to obtain standard data, and The standard data is synchronized to the first database.
数据处理设备确定更新数据为第一数据类型时，数据处理设备将更新数据添加到预设消息队列中（预设消息队列是指预先设置的用于存放更新数据的容器），数据处理设备通过流式数据引擎处理预设消息队列中的更新数据得到分布式数据流，分布式数据流中包含标准数据，数据处理设备将标准数据同步至第一数据库，例如，第一数据库为HIVE数据库（HIVE是基于HADOOP的一个数据仓库）。When the data processing device determines that the update data is of the first data type, the data processing device adds the update data to the preset message queue (the preset message queue refers to a pre-set container for storing the update data), and the data processing device passes the stream The distributed data engine processes the updated data in the preset message queue to obtain a distributed data stream. The distributed data stream contains standard data. The data processing equipment synchronizes the standard data to the first database. For example, the first database is the HIVE database (HIVE is A data warehouse based on HADOOP).
本实施例数据处理设备将更新数据保存至预设消息队列，通过流式数据引擎处理预设消息队列中的更新数据，实现了消息数据的均匀处理，减小了数据处理设备的压力。The data processing device of this embodiment saves the update data to the preset message queue, and processes the update data in the preset message queue through the streaming data engine, which realizes uniform processing of message data and reduces the pressure on the data processing device.
步骤S30，在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库。Step S30, when the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory.
在更新数据为第二数据类型时，数据处理设备封装更新数据，即，数据处理设备将更新数据映射到一个封装协议的净荷中，然后，数据处理设备填充对应协议的包头，形成封装协议的数据包，数据处理设备将封装后的更新数据同步至内存中的第二数据库。When the update data is the second data type, the data processing device encapsulates the update data, that is, the data processing device maps the update data to the payload of an encapsulation protocol, and then the data processing device fills the header of the corresponding protocol to form the encapsulation protocol For data packets, the data processing device synchronizes the encapsulated update data to the second database in the memory.
本实施例中数据处理设备将更新数据按照数据类型同步至两个数据库，即数据处理设备在确定数据类型为第二数据类型时，数据处理设备占用内存进行数据的同步，避免了更新数据较多时，造成预设消息队列过长导致的资源不足的情况发生，同时数据分类存在可以方便数据的快速查询，具体地：In this embodiment, the data processing device synchronizes the updated data to the two databases according to the data type, that is, when the data processing device determines that the data type is the second data type, the data processing device occupies the memory to synchronize the data, which avoids the time when there is a lot of updated data. , Resulting in insufficient resources caused by the pre-determined message queue being too long, and the existence of data classification can facilitate rapid data query, specifically:
步骤S40，在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。Step S40: When a data query request is received, a query sentence corresponding to the data query request is acquired, the first database or the second database is queried, and the target data corresponding to the query sentence is acquired.
数据处理设备接收数据查询请求，数据查询请求的触发方式不作具体限定，例如，用户输入查询语句触发数据查询请求，数据处理设备接收到数据查询请求时，数据处理设备获取数据查询请求对应的查询语句，数据处理设备根据查询语句确定第一数据库或者第二数据库，以获取数据查询请求对应的目标数据，具体地，包括：The data processing device receives the data query request, and the triggering method of the data query request is not specifically limited. For example, the user inputs a query statement to trigger the data query request. When the data processing device receives the data query request, the data processing device obtains the query statement corresponding to the data query request. The data processing device determines the first database or the second database according to the query sentence to obtain the target data corresponding to the data query request, specifically including:
步骤a1，在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；Step a1, when a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are obtained;
步骤a2，在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；Step a2, when the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
步骤a3，在所述数据类型为第二数据类型时，查询所述第二数据库获取所述查询语句对应的第二目标数据。Step a3, when the data type is the second data type, query the second database to obtain the second target data corresponding to the query sentence.
即，数据处理设备在接收到数据查询请求时，获取数据查询请求对应的查询语句，及查询语句中的数据类型；在数据类型为第一数据类型时，数据处理设备查询第一数据库获取查询语句对应的第一目标数据；在数据类型为第二数据类型时，查询第二数据库获取所述查询语句对应的第二目标数据，本实施例中按照数据类型查询对应的数据库，实现了更新数据的快速查询，有效地避免了数据拥堵的情况发生。That is, when the data processing device receives the data query request, it obtains the query statement corresponding to the data query request and the data type in the query statement; when the data type is the first data type, the data processing device queries the first database to obtain the query statement Corresponding first target data; when the data type is the second data type, query the second database to obtain the second target data corresponding to the query statement. In this embodiment, the corresponding database is queried according to the data type to achieve the update of the data Quick query, effectively avoiding data congestion.
在本实施例中数据处理设备通过解析更新日志提取更新日志中的更新数据，与轮询数据同步的方式相比，减少了对源数据库的影响，实现了源数据库无感知的数据同步操作；在更新数据为第一数据类型时，数据处理设备将更新数据添加到预设消息队列中，通过流式数据引擎处理预设消息队列中的更新数据进行数据同步，实现了数据的匀速同步，在更新数据为第二数据类型时，数据处理设备封装更新数据并将封装后的更新数据进行同步，即，数据处理设备根据更新数据的数据类型进行分流同步，减小了数据处理设备的硬件压力，避免了预设消息队列中更新数据过量导致资源不足的情况发生，在接收到数据查询请求时，数据处理设备获取数据查询请求对应的查询语句，查询第一数据库或第二数据库，获取查询语句对应的目标数据，实现了数据实时更新快速查询In this embodiment, the data processing device extracts the update data in the update log by parsing the update log. Compared with the polling data synchronization method, the impact on the source database is reduced, and the source database is not aware of the data synchronization operation; When the update data is the first data type, the data processing device adds the update data to the preset message queue, and uses the streaming data engine to process the update data in the preset message queue for data synchronization, achieving uniform data synchronization. When the data is the second data type, the data processing device encapsulates the updated data and synchronizes the encapsulated updated data, that is, the data processing device performs shunt synchronization according to the data type of the updated data, which reduces the hardware pressure of the data processing device and avoids In order to prevent the occurrence of insufficient resources due to excessive update data in the preset message queue, when a data query request is received, the data processing device obtains the query statement corresponding to the data query request, queries the first database or the second database, and obtains the query statement corresponding to the query statement. Target data, real-time data update and fast query
进一步地，在本申请第一实施例的基础上，提出了本申请数据处理方法的第二实施例。Further, on the basis of the first embodiment of the present application, a second embodiment of the data processing method of the present application is proposed.
本实施例是第一实施例中步骤S10之前的步骤，本实施例与本申请第一实施例的区别在于：This embodiment is a step before step S10 in the first embodiment. The difference between this embodiment and the first embodiment of this application lies in:
在接收到源数据库监控指令时，将所述源数据库监控指令发送至预设数据采集器，以使所述预设数据采集器启动并注册源数据库对应的监控实例；When receiving the source database monitoring instruction, sending the source database monitoring instruction to the preset data collector, so that the preset data collector starts and registers the monitoring instance corresponding to the source database;
通过所述监控实例采集源数据库操作指令，将所述源数据库操作指令添加至预设日志模板，生成更新日志。The source database operation instructions are collected through the monitoring instance, the source database operation instructions are added to a preset log template, and an update log is generated.
数据处理设备中预设数据采集器，预设数据采集器是指用于数据监控采集的工具，例如，预设数据采集器可以是Canal组件，Canal组件的数据采集原理为：Canal组件模拟mysql 主从交互协议，Canal组件作为mysql从节点，向mysql 主节点发送dump协议，mysql主节点收到dump请求，开始推送Binlog日志给从节点，Canal组件解析Binlog日志供客户端使用。The data collector is preset in the data processing equipment. The preset data collector refers to the tool used for data monitoring and collection. For example, the preset data collector can be a Canal component. The data collection principle of the Canal component is: the Canal component simulates the mysql master From the interactive protocol, the Canal component acts as a mysql slave node and sends a dump protocol to the mysql master node. The mysql master node receives the dump request and starts to push the Binlog log to the slave node. The Canal component parses the Binlog log for use by the client.
数据处理设备接收源数据库监控指令，源数据库监控指令的触发方式不作具体限定，数据处理设备在接收到源数据库监控指令时，数据处理设备将源数据库监控指令发送至预设数据采集器，以使预设数据采集器启动并注册源数据库对应的监控实例，数据处理设备通过监控实例采集源数据库操作指令，数据处理设备将源数据库操作指令添加至预设日志模板（预设日志模板是指预先设置的包括空格字符和格式命令的模板），生成更新日志，数据处理设备接收到更新日志时，判定源数据库数据更新。The data processing device receives the source database monitoring instruction, and the triggering method of the source database monitoring instruction is not specifically limited. When the data processing device receives the source database monitoring instruction, the data processing device sends the source database monitoring instruction to the preset data collector so that The preset data collector starts and registers the monitoring instance corresponding to the source database, the data processing device collects the source database operation instructions through the monitoring instance, and the data processing device adds the source database operation instructions to the preset log template (the preset log template refers to the preset log template). (Includes space characters and format command templates), generates an update log, and when the data processing device receives the update log, it determines that the source database data is updated.
例如，数据处理设备向Canal组件发送源数据库监控指令，Canal组件接收源数据库监控指令，Canal组件启动监控实例源数据库，Canal组件向数据处理设备注册该监控实例，数据处理设备实时监控实例的状况，获取监控实例采集到的源数据库操作指令，数据处理设备将源数据库操作指令添加至预设日志模板，生成更新日志。For example, the data processing device sends a source database monitoring instruction to the Canal component, the Canal component receives the source database monitoring instruction, the Canal component starts to monitor the instance source database, the Canal component registers the monitoring instance with the data processing device, and the data processing device monitors the status of the instance in real time. Obtain the source database operation instructions collected by the monitoring instance, and the data processing device adds the source database operation instructions to the preset log template to generate an update log.
本实施例中数据处理设备通过预设数据采集器采集源数据库的更新日志，减少了数据的同步周期，使得更新数据同步周期，从原来的若干天缩短到分钟级别，便于数据处理设备对应的数据查询平台能够在分析决策层更早的发现业务端的变化状况，从而改进生产经营方案或者措施，提高经营效率。In this embodiment, the data processing device collects the update log of the source database through the preset data collector, which reduces the data synchronization period, so that the update data synchronization period is shortened from the original several days to the minute level, which is convenient for the data corresponding to the data processing device. The query platform can find changes in the business side earlier at the analysis and decision-making level, thereby improving production and operation plans or measures, and improving operating efficiency.
进一步地，基于本申请的上述实施例，提出了本申请数据处理方法的第三实施例。Further, based on the foregoing embodiments of the present application, a third embodiment of the data processing method of the present application is proposed.
本实施例是第一实施例中步骤S30之后的步骤，本实施例与上述实施例的区别在于：This embodiment is a step after step S30 in the first embodiment. The difference between this embodiment and the foregoing embodiment lies in:
提取所述第二数据库中预设时间间隔之前产生的更新数据，将所述更新数据输入至预设消息队列中；Extracting update data generated before a preset time interval in the second database, and inputting the update data into a preset message queue;
调用流式计算引擎对所述预设消息队列中的所述更新数据进行处理，得到分布式数据流；Calling a streaming computing engine to process the update data in the preset message queue to obtain a distributed data stream;
将处理后的分布式数据流中的标准数据同步至第一数据库。Synchronize the standard data in the processed distributed data stream to the first database.
由于数据处理设备的内存可用空间是有限的，数据处理设备会对内存进行自动清除，为了避免内存中保存的更新数据被删除，本实施例中数据处理设备提取第二数据库中预设时间间隔之前产生的更新数据，数据处理设备将更新数据输入至预设消息队列中；其中，预设时间间隔是指预先设置的时间节点，预设时间间隔可以根据具体情况灵活设置，例如，预设时间间隔受到内存可用存储空间和更新日志数据量的影响，当内存可用存储空间较大时，预设时间间隔可以设置为第一时间间隔，当内存可用存储空间较小时，预设时间间隔可以设置为第二时间间隔；或者当更新日志数据量小时，预设时间间隔可以设置为第一时间间隔，当更新日志数据量大时，预设时间间隔可以设置为第二时间间隔。Since the available memory space of the data processing device is limited, the data processing device will automatically clear the memory. In order to prevent the updated data stored in the memory from being deleted, the data processing device in this embodiment extracts the data from the second database before the preset time interval For the generated update data, the data processing device inputs the update data into the preset message queue; among them, the preset time interval refers to the time node set in advance, and the preset time interval can be flexibly set according to the specific situation, for example, the preset time interval Affected by the available storage space of the memory and the amount of update log data, when the available storage space of the memory is large, the preset time interval can be set to the first time interval, and when the available storage space of the memory is small, the preset time interval can be set to the first time interval. Two time intervals; or when the amount of update log data is small, the preset time interval can be set to the first time interval, and when the amount of update log data is large, the preset time interval can be set to the second time interval.
数据处理设备调用流式计算引擎对预设消息队列中的更新数据进行处理，得到分布式数据流，数据处理设备将处理后的分布式数据流中的标准数据同步至第一数据库。The data processing device calls the streaming computing engine to process the updated data in the preset message queue to obtain a distributed data stream, and the data processing device synchronizes the standard data in the processed distributed data stream to the first database.
本实施例中数据处理设备将内存中第二数据库中的更新数据再次同步至第一数据库，避免了由于内存自动清理导致的数据删除的情况，保证了数据的持久性，同时使得第一数据库与源数据库的相同，方便更新数据统一管理。In this embodiment, the data processing device synchronizes the updated data in the second database in the memory to the first database again, avoiding data deletion caused by automatic memory cleaning, ensuring data durability, and making the first database and The same source database is convenient for unified management of updated data.
进一步地，基于本申请第三实施例，提出了本申请数据处理方法的第四实施例。Further, based on the third embodiment of the present application, a fourth embodiment of the data processing method of the present application is proposed.
本实施例是第一实施例中步骤S40的细化步骤，本实施例与上述实施例的区别在于：This embodiment is a detailed step of step S40 in the first embodiment. The difference between this embodiment and the foregoing embodiment lies in:
在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
在所述数据类型为第二数据类型时，获取所述查询语句中的数据更新时间，若所述数据更新时间在所述预设时间间隔之前，则查询所述第一数据库获取所述查询语句对应的第二目标数据，若所述数据更新时间在所述预设时间间隔之后，则查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, obtain the data update time in the query sentence, and if the data update time is before the preset time interval, query the first database to obtain the query sentence For the corresponding second target data, if the data update time is after the preset time interval, query the second database to obtain the second target data corresponding to the query sentence.
即，数据处理设备在接收到数据查询请求时，获取数据查询请求对应的查询语句，及查询语句中的数据类型；数据处理设备在数据类型为第一数据类型时，查询第一数据库获取所述查询语句对应的第一目标数据。That is, when the data processing device receives the data query request, it obtains the query statement corresponding to the data query request and the data type in the query statement; when the data type is the first data type, the data processing device queries the first database to obtain the data type. The first target data corresponding to the query statement.
在数据类型为第二数据类型时，数据处理设备获取查询语句中的数据更新时间，并判断数据更新时间是否在预设时间间隔之前，其中，预设时间间隔与第三实施例的相同，若数据更新时间在预设时间间隔之前，则查询第一数据库获取查询语句对应的第二目标数据，若数据更新时间在预设时间间隔之后，则查询第二数据库获取查询语句对应的第二目标数据。When the data type is the second data type, the data processing device obtains the data update time in the query sentence and determines whether the data update time is before the preset time interval, where the preset time interval is the same as that of the third embodiment, if If the data update time is before the preset time interval, query the first database to obtain the second target data corresponding to the query sentence; if the data update time is after the preset time interval, query the second database to obtain the second target data corresponding to the query statement .
在本实施例中由于数据处理设备会将内存中的更新数据再次同步到第一数据库，因此在数据查询过程中，数据处理设备可以根据数据更新时间，确定目标数据库，进行数据的快速查询。In this embodiment, since the data processing device resynchronizes the updated data in the memory to the first database, during the data query process, the data processing device can determine the target database according to the data update time, and perform rapid data query.
进一步地，基于本申请上述实施例，提出了本申请数据处理方法的第五实施例。Further, based on the foregoing embodiments of the present application, a fifth embodiment of the data processing method of the present application is proposed.
本实施例是第一实施例中步骤S10的细化，本实施例与上述实施例的区别在于：This embodiment is a refinement of step S10 in the first embodiment. The difference between this embodiment and the foregoing embodiment lies in:
在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志获得所述更新日志中的源数据库操作指令；When the source database data update is detected, collect the update log when the source database is updated, and parse the update log to obtain the source database operation instruction in the update log;
若所述更新日志中的源数据库操作指令是删除指令时，对第一数据库执行所述删除指令；If the source database operation instruction in the update log is a delete instruction, execute the delete instruction on the first database;
若所述更新日志中的源数据库操作指令不是删除指令时，解析所述更新日志提取所述更新日志中的更新数据。If the source database operation instruction in the update log is not a delete instruction, the update log is parsed to extract update data in the update log.
即，数据处理设备在检测到源数据库数据更新时，数据处理设备采集源数据库更新时的更新日志，解析更新日志获得所述更新日志中的源数据库操作指令；数据处理设备判断源数据库操作指令是否为删除指令，若更新日志中的源数据库操作指令是删除指令时，数据处理设备对第一数据库执行删除指令；若更新日志中的源数据库操作指令不是删除指令时，数据处理设备解析更新日志提取所述更新日志中的更新数据。That is, when the data processing device detects that the source database data is updated, the data processing device collects the update log when the source database is updated, analyzes the update log to obtain the source database operation instruction in the update log; the data processing device determines whether the source database operation instruction is To delete instructions, if the source database operation instruction in the update log is a delete instruction, the data processing device executes the delete instruction on the first database; if the source database operation instruction in the update log is not a delete instruction, the data processing device parses the update log and extracts The update data in the update log.
本实施例中数据处理设备对更新日志进行解析，确定更新日志中的源数据库操作指令的类型，在源数据库操作指令是删除指令时，数据处理设备直接执行删除指令，保证了源数据库同步的完整性，避免了源数据库中数据删除没有被同步的情况。In this embodiment, the data processing device parses the update log to determine the type of the source database operation instruction in the update log. When the source database operation instruction is a delete instruction, the data processing device directly executes the delete instruction, ensuring the integrity of the source database synchronization It avoids the situation that data deletion in the source database is not synchronized.
进一步地，基于本申请上述实施例，提出了本申请数据处理方法的第六实施例。Further, based on the foregoing embodiments of the present application, a sixth embodiment of the data processing method of the present application is proposed.
本实施例是第一实施例中步骤S10之后的步骤，本实施例与上述实施例的区别在于：This embodiment is a step after step S10 in the first embodiment. The difference between this embodiment and the foregoing embodiment lies in:
获取预设消息队列的当前数据量，判断所述当前数据量是否为零；Acquiring the current data volume of the preset message queue, and judging whether the current data volume is zero;
若所述当前数据量为零，则将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库If the current data volume is zero, the update data is added to the preset message queue, the update data in the preset message queue is processed by the streaming data engine to obtain standard data, and the standard data is synchronized To the first database
若所述当前数据量不为零，则获取所述更新数据的数据类型。If the current data amount is not zero, the data type of the update data is acquired.
数据处理设备获取到更新数据之后，数据处理设备获取预设消息队列的当前数据量（预设消息队列与第一实施例中的相同，本实施例不作赘述），数据处理设备判断当前数据量是否为零；若当前数据量为零，即，数据同步压力较少，数据处理设备在数据同步时不进行分流，数据处理设备直接将更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将标准数据同步至第一数据库。After the data processing device obtains the updated data, the data processing device obtains the current data volume of the preset message queue (the preset message queue is the same as that in the first embodiment, which will not be repeated in this embodiment), and the data processing device determines whether the current data volume is Zero; if the current data volume is zero, that is, the data synchronization pressure is less, the data processing equipment does not split the data during data synchronization, and the data processing equipment directly adds the updated data to the preset message queue and processes it through the streaming data engine The updated data in the preset message queue obtains standard data, and the standard data is synchronized to the first database.
若当前数据量不为零，数据处理设备获取更新数据的数据类型，数据处理设备根据更新数据的数据类型对更新数据进行分流，即，数据处理设备执行第一实施例中的步骤S20在更新数据为第一数据类型时，将更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；步骤S30在更新数据为第二数据类型时，封装更新数据并将封装后的更新数据同步至内存中的第二数据库。If the current data volume is not zero, the data processing device obtains the data type of the update data, and the data processing device diverges the update data according to the data type of the update data. That is, the data processing device executes step S20 in the first embodiment to update the data. When it is the first data type, add update data to the preset message queue, process the update data in the preset message queue by a streaming data engine to obtain standard data, and synchronize the standard data to the first database; In step S30, when the updated data is of the second data type, the updated data is encapsulated and the encapsulated updated data is synchronized to the second database in the memory.
本实施例中数据处理设备根据预设消息队列中的当前数据量确定数据同步的方法，使得数据同步更加高效合理。In this embodiment, the data processing device determines the data synchronization method according to the current data volume in the preset message queue, so that the data synchronization is more efficient and reasonable.
此外，参照图3，本申请实施例还提出一种数据处理装置，所述数据处理装置包括：In addition, referring to FIG. 3, an embodiment of the present application also proposes a data processing device, and the data processing device includes:
请求接收模块10，用于在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；The request receiving module 10 is configured to collect the update log when the source database is updated when the source database data update is detected, and parse the update log to extract the update data in the update log;
第一同步模块20，用于在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；The first synchronization module 20 is configured to add the update data to a preset message queue when the update data is of the first data type, and process the update data in the preset message queue through a streaming data engine to obtain Standard data, and synchronize the standard data to the first database;
第二同步模块30，用于在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；The second synchronization module 30 is configured to encapsulate the update data and synchronize the encapsulated update data to the second database in the memory when the update data is of the second data type;
数据查询模块40，用于在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。The data query module 40 is configured to obtain a query statement corresponding to the data query request, query the first database or the second database, and obtain target data corresponding to the query statement when a data query request is received.
在一实施例中，所述的数据处理装置，包括：In an embodiment, the data processing device includes:
监控发送模块，用于在接收到源数据库监控指令时，将所述源数据库监控指令发送至预设数据采集器，以使所述预设数据采集器启动并注册源数据库对应的监控实例；The monitoring sending module is configured to send the source database monitoring instruction to the preset data collector when receiving the source database monitoring instruction, so that the preset data collector starts and registers the monitoring instance corresponding to the source database;
日志生成模块，用于通过所述监控实例采集源数据库操作指令，将所述源数据库操作指令添加至预设日志模板，生成更新日志。The log generation module is configured to collect source database operation instructions through the monitoring instance, add the source database operation instructions to a preset log template, and generate an update log.
在一实施例中，所述数据查询模块40，包括：In an embodiment, the data query module 40 includes:
信息获取单元，用于在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；The information obtaining unit is configured to obtain the query statement corresponding to the data query request and the data type in the query statement when the data query request is received;
第一查询单元，用于在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；The first query unit is configured to query the first database to obtain the first target data corresponding to the query sentence when the data type is the first data type;
第二查询单元，用于在所述数据类型为第二数据类型时，查询所述第二数据库获取所述查询语句对应的第二目标数据。The second query unit is configured to query the second database to obtain the second target data corresponding to the query sentence when the data type is the second data type.
在一实施例中，所述的数据处理装置，包括：In an embodiment, the data processing device includes:
信息提取模块，用于提取所述第二数据库中预设时间间隔之前产生的更新数据，将所述更新数据输入至预设消息队列中；An information extraction module, configured to extract update data generated before a preset time interval in the second database, and input the update data into a preset message queue;
数据处理模块，用于调用流式计算引擎对所述预设消息队列中的所述更新数据进行处理，得到分布式数据流；A data processing module, configured to call a streaming computing engine to process the update data in the preset message queue to obtain a distributed data stream;
数据同步模块，用于将处理后的分布式数据流中的标准数据同步至第一数据库。The data synchronization module is used to synchronize the standard data in the processed distributed data stream to the first database.
在一实施例中，所述数据查询模块40，包括：In an embodiment, the data query module 40 includes:
信息获取单元，用于在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；The information obtaining unit is configured to obtain the query statement corresponding to the data query request and the data type in the query statement when the data query request is received;
第一查询单元，用于在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；The first query unit is configured to query the first database to obtain the first target data corresponding to the query sentence when the data type is the first data type;
所述第二查询单元，用于在所述数据类型为第二数据类型时，获取所述查询语句中的数据更新时间，若所述数据更新时间在所述预设时间间隔之前，则查询所述第一数据库获取所述查询语句对应的第二目标数据，若所述数据更新时间在所述预设时间间隔之后，则查询所述第二数据库获取所述查询语句对应的第二目标数据。The second query unit is configured to obtain the data update time in the query sentence when the data type is the second data type, and if the data update time is before the preset time interval, query the The first database obtains the second target data corresponding to the query sentence, and if the data update time is after the preset time interval, the second database is queried to obtain the second target data corresponding to the query sentence.
在一实施例中，所述请求接收模块10，包括：In an embodiment, the request receiving module 10 includes:
指令获取单元，用于在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志获得所述更新日志中的源数据库操作指令；The instruction acquisition unit is configured to collect an update log when the source database is updated when the source database data update is detected, and parse the update log to obtain the source database operation instruction in the update log;
数据删除单元，用于若所述更新日志中的源数据库操作指令是删除指令时，对第一数据库执行所述删除指令；A data deletion unit, configured to execute the deletion instruction on the first database if the source database operation instruction in the update log is a deletion instruction;
日志解析单元，用于若所述更新日志中的源数据库操作指令不是删除指令时，解析所述更新日志提取所述更新日志中的更新数据。The log analysis unit is configured to analyze the update log to extract update data in the update log if the source database operation instruction in the update log is not a delete instruction.
在一实施例中，所述的数据处理装置，包括：In an embodiment, the data processing device includes:
获取判断模块，用于获取预设消息队列的当前数据量，判断所述当前数据量是否为零；An obtaining and judging module for obtaining the current data volume of the preset message queue, and judging whether the current data volume is zero;
数据同步模块，用于若所述当前数据量为零，则将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库The data synchronization module is configured to add the update data to a preset message queue if the current data volume is zero, process the update data in the preset message queue through a streaming data engine to obtain standard data, and Synchronize the standard data to the first database
类型获取模块，用于若所述当前数据量不为零，则获取所述更新数据的数据类型。The type obtaining module is configured to obtain the data type of the update data if the current data amount is not zero.
其中，数据处理装置的各个功能模块实现的步骤可参照本申请数据处理方法的各个实施例，此处不再赘述。For the steps implemented by each functional module of the data processing device, reference may be made to each embodiment of the data processing method of the present application, which will not be repeated here.
此外，本申请实施例还提出一种计算机存储介质，所述计算机存储介质可以是非易失性的，也可以是易失性的。In addition, the embodiments of the present application also propose a computer storage medium. The computer storage medium may be non-volatile or volatile.
所述计算机存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现上述实施例提供的数据处理方法中的操作。A computer program is stored on the computer storage medium, and when the computer program is executed by a processor, the operations in the data processing method provided in the foregoing embodiments are implemented.
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体/操作/对象与另一个实体/操作/对象区分开来，而不一定要求或者暗示这些实体/操作/对象之间存在任何这种实际的关系或者顺序；术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者系统不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者系统所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者系统中还存在另外的相同要素。It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity/operation/object from another entity/operation/object, and do not necessarily require or imply these There is any such actual relationship or sequence between entities/operations/objects; the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, An article or system includes not only those elements, but also other elements that are not explicitly listed, or include elements inherent to the process, method, article, or system. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or system that includes the element.
对于装置实施例而言，由于其基本相似于方法实施例，所以描述得比较简单，相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中作为分离部件说明的单元可以是或者也可以不是物理上分开的。可以根据实际的需要选择中的部分或者全部模块来实现本申请方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solution of the present application. Those of ordinary skill in the art can understand and implement it without creative work.
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。The serial numbers of the foregoing embodiments of the present application are for description only, and do not represent the superiority or inferiority of the embodiments.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本申请各个实施例所述的方法。Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disks, optical disks), including several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.
以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。The above are only the preferred embodiments of the application, and do not limit the scope of the patent for this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of the application, or directly or indirectly applied to other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims (20)

  1. 一种数据处理方法，其中，所述数据处理方法包括以下步骤：A data processing method, wherein the data processing method includes the following steps:
    在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；When an update of the source database data is detected, collect an update log when the source database is updated, and parse the update log to extract update data in the update log;
    在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；When the update data is of the first data type, the update data is added to a preset message queue, the update data in the preset message queue is processed by a streaming data engine to obtain standard data, and the standard Data is synchronized to the first database;
    在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；When the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory;
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。When a data query request is received, a query statement corresponding to the data query request is obtained, the first database or the second database is queried, and the target data corresponding to the query statement is obtained.
  2. 如权利要求1所述的数据处理方法，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤之前，包括：The data processing method of claim 1, wherein when the source database data update is detected, before the step of collecting the update log when the source database is updated, and parsing the update log to extract the update data in the update log ,include:
    在接收到源数据库监控指令时，将所述源数据库监控指令发送至预设数据采集器，以使所述预设数据采集器启动并注册源数据库对应的监控实例；When receiving the source database monitoring instruction, sending the source database monitoring instruction to the preset data collector, so that the preset data collector starts and registers the monitoring instance corresponding to the source database;
    通过所述监控实例采集源数据库操作指令，将所述源数据库操作指令添加至预设日志模板，生成更新日志。The source database operation instructions are collected through the monitoring instance, the source database operation instructions are added to a preset log template, and an update log is generated.
  3. 如权利要求1所述的数据处理方法，其中，所述在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据的步骤，包括：The data processing method according to claim 1, wherein when a data query request is received, the query sentence corresponding to the data query request is obtained, the first database or the second database is queried, and the The steps of querying the target data corresponding to the sentence include:
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
    在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
    在所述数据类型为第二数据类型时，查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, query the second database to obtain the second target data corresponding to the query sentence.
  4. 如权利要求1所述的数据处理方法，其中，所述在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库的步骤之后，包括：The data processing method of claim 1, wherein after the step of encapsulating the updated data and synchronizing the encapsulated updated data to the second database in the memory when the updated data is of the second data type ,include:
    提取所述第二数据库中预设时间间隔之前产生的更新数据，将所述更新数据输入至预设消息队列中；Extracting update data generated before a preset time interval in the second database, and inputting the update data into a preset message queue;
    调用流式计算引擎对所述预设消息队列中的所述更新数据进行处理，得到分布式数据流；Calling a streaming computing engine to process the update data in the preset message queue to obtain a distributed data stream;
    将处理后的分布式数据流中的标准数据同步至第一数据库。Synchronize the standard data in the processed distributed data stream to the first database.
  5. 如权利要求4所述的数据处理方法，其中，所述在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据的步骤，包括：The data processing method according to claim 4, wherein when the data query request is received, the query sentence corresponding to the data query request is obtained, the first database or the second database is queried, and the The steps of querying the target data corresponding to the sentence include:
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
    在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
    在所述数据类型为第二数据类型时，获取所述查询语句中的数据更新时间，若所述数据更新时间在所述预设时间间隔之前，则查询所述第一数据库获取所述查询语句对应的第二目标数据，若所述数据更新时间在所述预设时间间隔之后，则查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, obtain the data update time in the query sentence, and if the data update time is before the preset time interval, query the first database to obtain the query sentence For the corresponding second target data, if the data update time is after the preset time interval, query the second database to obtain the second target data corresponding to the query sentence.
  6. 如权利要求1-5任意一项所述的数据处理方法，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤，包括：The data processing method according to any one of claims 1-5, wherein when the source database data update is detected, the update log when the source database is updated is collected, and the update log is parsed to extract the data in the update log. The steps to update the data include:
    在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志获得所述更新日志中的源数据库操作指令；When the source database data update is detected, collect the update log when the source database is updated, and parse the update log to obtain the source database operation instruction in the update log;
    若所述更新日志中的源数据库操作指令是删除指令时，对第一数据库执行所述删除指令；If the source database operation instruction in the update log is a delete instruction, execute the delete instruction on the first database;
    若所述更新日志中的源数据库操作指令不是删除指令时，解析所述更新日志提取所述更新日志中的更新数据。If the source database operation instruction in the update log is not a delete instruction, the update log is parsed to extract update data in the update log.
  7. 如权利要求1-5任意一项所述的数据处理方法，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤之后，包括：The data processing method according to any one of claims 1-5, wherein when the source database data update is detected, the update log when the source database is updated is collected, and the update log is parsed to extract the data in the update log. After the steps to update the data, include:
    获取预设消息队列的当前数据量，判断所述当前数据量是否为零；Acquiring the current data volume of the preset message queue, and judging whether the current data volume is zero;
    若所述当前数据量为零，则将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库If the current data volume is zero, the update data is added to the preset message queue, the update data in the preset message queue is processed by the streaming data engine to obtain standard data, and the standard data is synchronized To the first database
    若所述当前数据量不为零，则获取所述更新数据的数据类型。If the current data amount is not zero, the data type of the update data is acquired.
  8. 一种数据处理装置，其中，所述数据处理装置包括：A data processing device, wherein the data processing device includes:
    请求接收模块，用于在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；The request receiving module is configured to collect the update log when the source database is updated when the source database data update is detected, and parse the update log to extract the update data in the update log;
    第一同步模块，用于在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；The first synchronization module is configured to add the update data to the preset message queue when the update data is of the first data type, and process the update data in the preset message queue through the streaming data engine to obtain a standard Data, and synchronize the standard data to the first database;
    第二同步模块，用于在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；The second synchronization module is configured to encapsulate the update data and synchronize the encapsulated update data to the second database in the memory when the update data is of the second data type;
    数据查询模块，用于在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。The data query module is configured to obtain the query sentence corresponding to the data query request, query the first database or the second database, and obtain the target data corresponding to the query sentence when the data query request is received.
  9. 一种数据处理设备，其中，所述数据处理设备包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述计算机程序被所述处理器执行时实现如下步骤：A data processing device, wherein the data processing device includes a memory, a processor, and a computer program stored in the memory and running on the processor, and when the computer program is executed by the processor To achieve the following steps:
    在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；When an update of the source database data is detected, collect an update log when the source database is updated, and parse the update log to extract update data in the update log;
    在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；When the update data is of the first data type, the update data is added to a preset message queue, the update data in the preset message queue is processed by a streaming data engine to obtain standard data, and the standard Data is synchronized to the first database;
    在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；When the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory;
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。When a data query request is received, a query statement corresponding to the data query request is obtained, the first database or the second database is queried, and the target data corresponding to the query statement is obtained.
  10. 如权利要求9所述的数据处理设备，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤之前，包括：The data processing device according to claim 9, wherein when the source database data update is detected, before the step of collecting the update log when the source database is updated, and parsing the update log to extract the update data in the update log ,include:
    在接收到源数据库监控指令时，将所述源数据库监控指令发送至预设数据采集器，以使所述预设数据采集器启动并注册源数据库对应的监控实例；When receiving the source database monitoring instruction, sending the source database monitoring instruction to the preset data collector, so that the preset data collector starts and registers the monitoring instance corresponding to the source database;
    通过所述监控实例采集源数据库操作指令，将所述源数据库操作指令添加至预设日志模板，生成更新日志。The source database operation instructions are collected through the monitoring instance, the source database operation instructions are added to a preset log template, and an update log is generated.
  11. 如权利要求9所述的数据处理设备，其中，所述在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据的步骤，包括：The data processing device according to claim 9, wherein when the data query request is received, the query sentence corresponding to the data query request is obtained, the first database or the second database is queried, and the The steps of querying the target data corresponding to the sentence include:
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
    在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
    在所述数据类型为第二数据类型时，查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, query the second database to obtain the second target data corresponding to the query sentence.
  12. 如权利要求9所述的数据处理设备，其中，所述在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库的步骤之后，包括：The data processing device according to claim 9, wherein when the update data is of the second data type, after the step of encapsulating the update data and synchronizing the encapsulated update data to the second database in the memory ,include:
    提取所述第二数据库中预设时间间隔之前产生的更新数据，将所述更新数据输入至预设消息队列中；Extracting update data generated before a preset time interval in the second database, and inputting the update data into a preset message queue;
    调用流式计算引擎对所述预设消息队列中的所述更新数据进行处理，得到分布式数据流；Calling a streaming computing engine to process the update data in the preset message queue to obtain a distributed data stream;
    将处理后的分布式数据流中的标准数据同步至第一数据库。Synchronize the standard data in the processed distributed data stream to the first database.
  13. 如权利要求12所述的数据处理设备，其中，所述在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据的步骤，包括：The data processing device according to claim 12, wherein when the data query request is received, the query sentence corresponding to the data query request is obtained, the first database or the second database is queried, and the The steps of querying the target data corresponding to the sentence include:
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
    在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
    在所述数据类型为第二数据类型时，获取所述查询语句中的数据更新时间，若所述数据更新时间在所述预设时间间隔之前，则查询所述第一数据库获取所述查询语句对应的第二目标数据，若所述数据更新时间在所述预设时间间隔之后，则查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, obtain the data update time in the query sentence, and if the data update time is before the preset time interval, query the first database to obtain the query sentence For the corresponding second target data, if the data update time is after the preset time interval, query the second database to obtain the second target data corresponding to the query sentence.
  14. 如权利要求9-13任意一项所述的数据处理设备，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤，包括：The data processing device according to any one of claims 9-13, wherein when the source database data update is detected, the update log when the source database is updated is collected, and the update log is parsed to extract the The steps to update the data include:
    在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志获得所述更新日志中的源数据库操作指令；When the source database data update is detected, collect the update log when the source database is updated, and parse the update log to obtain the source database operation instruction in the update log;
    若所述更新日志中的源数据库操作指令是删除指令时，对第一数据库执行所述删除指令；If the source database operation instruction in the update log is a delete instruction, execute the delete instruction on the first database;
    若所述更新日志中的源数据库操作指令不是删除指令时，解析所述更新日志提取所述更新日志中的更新数据。If the source database operation instruction in the update log is not a delete instruction, the update log is parsed to extract update data in the update log.
  15. 如权利要求9-13任意一项所述的数据处理设备，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤之后，包括：The data processing device according to any one of claims 9-13, wherein when the source database data update is detected, the update log when the source database is updated is collected, and the update log is parsed to extract the After the steps to update the data, include:
    获取预设消息队列的当前数据量，判断所述当前数据量是否为零；Acquiring the current data volume of the preset message queue, and judging whether the current data volume is zero;
    若所述当前数据量为零，则将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库If the current data volume is zero, the update data is added to the preset message queue, the update data in the preset message queue is processed by the streaming data engine to obtain standard data, and the standard data is synchronized To the first database
    若所述当前数据量不为零，则获取所述更新数据的数据类型。If the current data amount is not zero, the data type of the update data is acquired.
  16. 一种计算机存储介质，其中，所述计算机存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如下步骤：A computer storage medium, wherein a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, the following steps are implemented:
    在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据；When an update of the source database data is detected, collect an update log when the source database is updated, and parse the update log to extract update data in the update log;
    在所述更新数据为第一数据类型时，将所述更新数据添加到预设消息队列中，通过流式数据引擎处理所述预设消息队列中的更新数据得到标准数据，并将所述标准数据同步至第一数据库；When the update data is of the first data type, the update data is added to a preset message queue, the update data in the preset message queue is processed by a streaming data engine to obtain standard data, and the standard Data is synchronized to the first database;
    在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库；When the update data is of the second data type, encapsulate the update data and synchronize the encapsulated update data to the second database in the memory;
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据。When a data query request is received, a query statement corresponding to the data query request is obtained, the first database or the second database is queried, and the target data corresponding to the query statement is obtained.
  17. 如权利要求16所述的计算机存储介质，其中，所述在检测到源数据库数据更新时，采集源数据库更新时的更新日志，解析所述更新日志提取所述更新日志中的更新数据的步骤之前，包括：The computer storage medium of claim 16, wherein when the source database data update is detected, the update log of the source database update is collected, and the update log is analyzed to extract the update data in the update log before the step ,include:
    在接收到源数据库监控指令时，将所述源数据库监控指令发送至预设数据采集器，以使所述预设数据采集器启动并注册源数据库对应的监控实例；When receiving the source database monitoring instruction, sending the source database monitoring instruction to the preset data collector, so that the preset data collector starts and registers the monitoring instance corresponding to the source database;
    通过所述监控实例采集源数据库操作指令，将所述源数据库操作指令添加至预设日志模板，生成更新日志。The source database operation instructions are collected through the monitoring instance, the source database operation instructions are added to a preset log template, and an update log is generated.
  18. 如权利要求16所述的计算机存储介质，其中，所述在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据的步骤，包括：The computer storage medium according to claim 16, wherein when the data query request is received, the query sentence corresponding to the data query request is obtained, the first database or the second database is queried, and the The steps of querying the target data corresponding to the sentence include:
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
    在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
    在所述数据类型为第二数据类型时，查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, query the second database to obtain the second target data corresponding to the query sentence.
  19. 如权利要求16所述的计算机存储介质，其中，所述在所述更新数据为第二数据类型时，封装所述更新数据并将封装后的更新数据同步至内存中的第二数据库的步骤之后，包括：The computer storage medium of claim 16, wherein after the step of encapsulating the update data and synchronizing the encapsulated update data to the second database in the memory when the update data is of the second data type ,include:
    提取所述第二数据库中预设时间间隔之前产生的更新数据，将所述更新数据输入至预设消息队列中；Extracting update data generated before a preset time interval in the second database, and inputting the update data into a preset message queue;
    调用流式计算引擎对所述预设消息队列中的所述更新数据进行处理，得到分布式数据流；Calling a streaming computing engine to process the update data in the preset message queue to obtain a distributed data stream;
    将处理后的分布式数据流中的标准数据同步至第一数据库。Synchronize the standard data in the processed distributed data stream to the first database.
  20. 如权利要求19所述的计算机存储介质，其中，所述在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，查询所述第一数据库或所述第二数据库，获取所述查询语句对应的目标数据的步骤，包括：The computer storage medium according to claim 19, wherein when the data query request is received, the query sentence corresponding to the data query request is obtained, the first database or the second database is queried, and the The steps of querying the target data corresponding to the sentence include:
    在接收到数据查询请求时，获取所述数据查询请求对应的查询语句，及所述查询语句中的数据类型；When a data query request is received, the query sentence corresponding to the data query request and the data type in the query sentence are acquired;
    在所述数据类型为第一数据类型时，查询所述第一数据库获取所述查询语句对应的第一目标数据；When the data type is the first data type, query the first database to obtain the first target data corresponding to the query sentence;
    在所述数据类型为第二数据类型时，获取所述查询语句中的数据更新时间，若所述数据更新时间在所述预设时间间隔之前，则查询所述第一数据库获取所述查询语句对应的第二目标数据，若所述数据更新时间在所述预设时间间隔之后，则查询所述第二数据库获取所述查询语句对应的第二目标数据。When the data type is the second data type, obtain the data update time in the query sentence, and if the data update time is before the preset time interval, query the first database to obtain the query sentence For the corresponding second target data, if the data update time is after the preset time interval, query the second database to obtain the second target data corresponding to the query sentence.
PCT/CN2020/117504 2020-02-28 2020-09-24 Data processing method, apparatus and device, and storage medium WO2021169268A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010133454.1 2020-02-28
CN202010133454.1A CN111414416B (en) 2020-02-28 2020-02-28 Data processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
WO2021169268A1 true WO2021169268A1 (en) 2021-09-02

Family

ID=71491034

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/117504 WO2021169268A1 (en) 2020-02-28 2020-09-24 Data processing method, apparatus and device, and storage medium

Country Status (2)

Country Link
CN (1) CN111414416B (en)
WO (1) WO2021169268A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114625502A (en) * 2022-03-03 2022-06-14 盐城金堤科技有限公司 Word-throwing task processing method and device, storage medium and electronic equipment
CN114817267A (en) * 2022-05-16 2022-07-29 通联支付网络服务股份有限公司 Method, apparatus and medium for generating data report
CN115391325A (en) * 2022-10-31 2022-11-25 深圳曼顿科技有限公司 Energy data management method, device, equipment and medium
CN115982285A (en) * 2023-03-10 2023-04-18 北京集度科技有限公司 Data processing method, device and computer readable storage medium

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111414416B (en) * 2020-02-28 2023-05-23 平安科技（深圳）有限公司 Data processing method, device, equipment and storage medium
CN112073520B (en) * 2020-09-09 2022-08-09 北京金山云网络技术有限公司 Data processing method and device and electronic equipment
CN112182001A (en) * 2020-09-27 2021-01-05 浪潮云信息技术股份公司 Method, apparatus and medium for incremental synchronization of database to dynamic ES index library
CN112347114A (en) * 2020-10-29 2021-02-09 有半岛(北京)信息科技有限公司 Data query system, data query method, data query device and server
CN112434062A (en) * 2020-11-26 2021-03-02 平安普惠企业管理有限公司 Quasi-real-time data processing method, device, server and storage medium
CN113282611B (en) * 2021-06-29 2024-04-23 深圳平安智汇企业信息管理有限公司 Method, device, computer equipment and storage medium for synchronizing stream data
CN113870641A (en) * 2021-09-29 2021-12-31 上海乐项信息技术有限公司 Simulation training method and system for live broadcast of tape goods
CN113722401B (en) * 2021-11-04 2022-02-01 树根互联股份有限公司 Data caching method and device, computer equipment and readable storage medium
CN114372064B (en) * 2022-03-22 2022-07-12 飞狐信息技术（天津）有限公司 Data processing apparatus, method, computer readable medium and processor

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120158799A1 (en) * 2010-12-17 2012-06-21 Microsoft Corporation Automatically matching data sets with storage components
CN104346479A (en) * 2014-11-26 2015-02-11 北京奇虎科技有限公司 Database synchronization method and database synchronization device
CN107040578A (en) * 2016-12-30 2017-08-11 平安科技（深圳）有限公司 Method of data synchronization, device and system
CN107180116A (en) * 2017-06-28 2017-09-19 努比亚技术有限公司 A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium
CN107404540A (en) * 2017-09-01 2017-11-28 北京奇虎科技有限公司 The synchronous method and device of data
CN111414416A (en) * 2020-02-28 2020-07-14 平安科技（深圳）有限公司 Data processing method, device, equipment and storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8626681B1 (en) * 2011-01-04 2014-01-07 Google Inc. Training a probabilistic spelling checker from structured data
US9953081B2 (en) * 2015-02-26 2018-04-24 Accenture Global Services Limited System for linking diverse data systems
CN109726074A (en) * 2018-08-31 2019-05-07 网联清算有限公司 Log processing method, device, computer equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120158799A1 (en) * 2010-12-17 2012-06-21 Microsoft Corporation Automatically matching data sets with storage components
CN104346479A (en) * 2014-11-26 2015-02-11 北京奇虎科技有限公司 Database synchronization method and database synchronization device
CN107040578A (en) * 2016-12-30 2017-08-11 平安科技（深圳）有限公司 Method of data synchronization, device and system
CN107180116A (en) * 2017-06-28 2017-09-19 努比亚技术有限公司 A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium
CN107404540A (en) * 2017-09-01 2017-11-28 北京奇虎科技有限公司 The synchronous method and device of data
CN111414416A (en) * 2020-02-28 2020-07-14 平安科技（深圳）有限公司 Data processing method, device, equipment and storage medium

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114625502A (en) * 2022-03-03 2022-06-14 盐城金堤科技有限公司 Word-throwing task processing method and device, storage medium and electronic equipment
CN114817267A (en) * 2022-05-16 2022-07-29 通联支付网络服务股份有限公司 Method, apparatus and medium for generating data report
CN115391325A (en) * 2022-10-31 2022-11-25 深圳曼顿科技有限公司 Energy data management method, device, equipment and medium
CN115982285A (en) * 2023-03-10 2023-04-18 北京集度科技有限公司 Data processing method, device and computer readable storage medium
CN115982285B (en) * 2023-03-10 2023-07-07 北京集度科技有限公司 Data processing method, apparatus and computer readable storage medium

Also Published As

Publication number Publication date
CN111414416A (en) 2020-07-14
CN111414416B (en) 2023-05-23

Similar Documents

Publication Publication Date Title
WO2021169268A1 (en) Data processing method, apparatus and device, and storage medium
US20150170070A1 (en) Method, apparatus, and system for monitoring website
CN109492073B (en) Log search method, log search apparatus, and computer-readable storage medium
CN109656963B (en) Metadata acquisition method, apparatus, device and computer readable storage medium
WO2019109518A1 (en) Data list uploading method and terminal thereof
CN107341033A (en) A kind of data statistical approach, device, electronic equipment and storage medium
WO2021164253A1 (en) Method and device for real-time multidimensional analysis of user behaviors, and storage medium
CN103312544A (en) Method, equipment and system for controlling terminals during log file reporting
CN102916854A (en) Traffic statistical method and device and proxy server
WO2021169275A1 (en) Sdn network device access method and apparatus, computer device, and storage medium
CN110928681A (en) Data processing method and device, storage medium and electronic device
CN113642300A (en) Report generation method and device, electronic equipment and computer readable medium
CN109426597A (en) Application performance monitoring method, device, equipment, system and storage medium
CN112445861B (en) Information processing method, device, system and storage medium
CN111367686A (en) Service interface calling method and device, computer equipment and storage medium
CN103440333A (en) Data processing method and device
CN103647666A (en) Method and apparatus for counting call detail record (CDR) messages and outputting results in real time
CN110633191A (en) Method and system for monitoring service health degree of software system in real time
CN115858672A (en) Power terminal management method and device, electronic equipment and storage medium
CN112788592B (en) Data transmission processing method for adding wake-up time
CN107347024A (en) A kind of method and apparatus for storing Operation Log
CN113487340A (en) Service decoupling processing method, device, equipment and storage medium
CN115114316A (en) Processing method, device, cluster and storage medium for high-concurrency data
CN112506886A (en) Multi-source service operation log acquisition method and system
CN109684159A (en) Method for monitoring state, device, equipment and the storage medium of distributed information system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20922283

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20922283

Country of ref document: EP

Kind code of ref document: A1