CN115017171A - Data processing method and device in multi-time zone scene, electronic equipment and medium - Google Patents
Data processing method and device in multi-time zone scene, electronic equipment and medium Download PDFInfo
- Publication number
- CN115017171A CN115017171A CN202210942783.XA CN202210942783A CN115017171A CN 115017171 A CN115017171 A CN 115017171A CN 202210942783 A CN202210942783 A CN 202210942783A CN 115017171 A CN115017171 A CN 115017171A
- Authority
- CN
- China
- Prior art keywords
- data
- time
- time zone
- service
- service data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Abstract
The present disclosure provides a data processing method, apparatus, electronic device and medium in a multi-time zone scene, the method comprising: acquiring system time of a time zone in which a data warehouse is located; determining the local time of the time zone of the business system according to the system time and the time difference between the time zone of the business system and the time zone of the data warehouse; and starting a data dispatching batch corresponding to the system time under the condition that the local time of the time zone in which the service system is located is zero, so as to dispatch target service data corresponding to the service system from a full data table set containing the service data of the service system. According to the scheme, when the data dispatching batch dispatching data is started, the local time of the time zone where the service system is located is zero, and at the moment, the service data of the service system in the previous day are all generated, so that the complete data of the previous day can be dispatched and obtained, and the completeness and the correctness of the service data of the service system in each time zone output by the data warehouse are ensured.
Description
Technical Field
The present disclosure relates to the field of big data and data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a medium in a multi-time-zone scenario.
Background
With the development of the internet and the improvement of business capability of enterprises, more and more excellent enterprises no longer satisfy the domestic market, and the expansion of overseas business is gradually and actively developed.
However, the development of overseas business presents new challenges to enterprises. For example, how to implement aggregation processing of cross-time-zone data distributed in various regions around the world and ensure that complete data of the previous day of each time zone is obtained becomes a problem to be solved urgently for each offshore enterprise.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, embodiments of the present disclosure provide a data processing method and apparatus in a multi-time zone scenario, an electronic device, and a medium.
According to an aspect of the present disclosure, a data processing method in a multi-time zone scene is provided, including:
acquiring system time of a time zone in which a data warehouse is located;
determining the local time of the time zone of the business system according to the system time and the time difference between the time zone of the business system and the time zone of the data warehouse;
and under the condition that the local time of the time zone in which the service system is located is zero, starting a data dispatching batch corresponding to the system time to dispatch target service data corresponding to the service system from a full data table set containing the service data of the service system, wherein the target service data is the service data generated by the service system on the day before the local time, and the service data carries time zone information of the time zone in which the service system is located and the generation time of the service data.
According to another aspect of the present disclosure, there is provided a data processing apparatus in a multi-time zone scene, including:
the time acquisition module is used for acquiring the system time of the time zone in which the data warehouse is located;
the time determining module is used for determining the local time of the time zone of the business system according to the system time and the time difference between the time zone of the business system and the time zone of the data warehouse;
and the data scheduling module is used for starting a data scheduling batch corresponding to the system time under the condition that the local time of the time zone in which the service system is located is zero, so as to schedule target service data corresponding to the service system from a full data table set containing the service data of the service system, wherein the target service data is the service data generated by the service system on the day before the local time, and the service data carries time zone information of the time zone in which the service system is located and the generation time of the service data.
According to another aspect of the present disclosure, there is provided an electronic device including:
a processor; and
a memory for storing a program, wherein the program is stored in the memory,
wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method of data processing in a multi-temporal scenario according to the preceding aspect.
According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the data processing method in a multi-temporal scenario according to the previous aspect.
According to another aspect of the present disclosure, a computer program product is provided, which includes a computer program, wherein the computer program, when executed by a processor, implements the data processing method in the multi-temporal scenario of the foregoing aspect.
One or more technical solutions provided in the embodiments of the present disclosure determine the local time of the time zone in which the business system is located by obtaining the system time of the time zone in which the data warehouse is located, according to the system time, and a time difference between the time zone in which the business system is located and the time zone in which the data warehouse is located, and start a data scheduling batch corresponding to the system time when the local time of the time zone in which the business system is located is zero, to schedule target business data corresponding to the business system from a full data table set including the business data of the business system, where the target business data is business data generated by the business system on a day before the local time, and the business data carries time zone information of the time zone in which the business system is located and the generation time of the business data. By adopting the scheme, when the data dispatching batch dispatching data is started, the local time of the time zone in which the service system is located is zero, and at the moment, the service data of the previous day of the service system are all generated, so that the complete data of the previous day can be dispatched and obtained, and the integrity and the correctness of the service data of the service system of each time zone output by the data warehouse are ensured.
Drawings
Further details, features and advantages of the disclosure are disclosed in the following description of exemplary embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 shows a flow diagram of a data processing method in a multi-time zone scenario according to an example embodiment of the present disclosure;
FIG. 2 shows a schematic block diagram of a data processing apparatus in a multi-time zone scenario according to an exemplary embodiment of the present disclosure;
FIG. 3 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description. It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Before explaining aspects of the present disclosure in detail, terms related to the present disclosure are explained as follows:
the Data Warehouse, known in english under the name Data Warehouse, may be abbreviated as DW or DWH. A data warehouse is a strategic set that provides all types of data support for all levels of decision-making processes of an enterprise. It is a single data store created for analytical reporting and decision support purposes. Providing the functions of guiding business process improvement, monitoring time, cost, quality and control for enterprises needing business intelligence;
t +1, representing that the timeliness of the output data of the data warehouse is the statistical result of the output data 1 day after the data generation time T, and similarly, T +2, representing that the timeliness of the output data of the data warehouse is the statistical result of the output data 2 days after the data generation time T;
and the data full link is idempotent, and the output result can be ensured to be consistent with the previous result when the data task is repeatedly executed for multiple times according to the full link.
The following describes a data processing method, an apparatus, an electronic device, and a medium in a multi-time zone scenario provided by the present disclosure with reference to the drawings.
With the development of enterprises, more and more enterprises expand overseas markets. At present, for aggregation processing of data distributed around the world, two methods are generally adopted: one way is that data is integrated into a unified data center, and T +1 data scheduling is performed once according to 0 point of a time zone in which a data warehouse is located, so that the timeliness of data in other time zones cannot be guaranteed, T +2 may occur in the maximum delay, and the integrity of data obtained by scheduling cannot be guaranteed; another way is to build an independent data warehouse at the location where the overseas business is developed, and perform T +1 data scheduling according to the 0 point of the local time zone, which results in the multiplied construction cost and maintenance cost of the data warehouse.
In order to solve the above problems, the present disclosure provides a data processing method in a multi-time zone scenario, the system time of a time zone in which a data warehouse is located is obtained, and the local time of the time zone in which a business system is located is determined according to the system time, a time difference between the time zone in which the business system is located and the time zone in which the data warehouse is located, and when the local time of the time zone in which the business system is located is zero, a data scheduling batch corresponding to the system time is started to schedule target business data corresponding to the business system from a full data table set containing the business data of the business system, where the target business data is business data generated by the business system on the day before the local time, and the business data carries time zone information of the time zone in which the business system is located and the generation time of the business data. By adopting the scheme, when the data dispatching batch dispatching data is started, the local time of the time zone where the service system is located is zero, and at the moment, the service data of the service system in the previous day are all generated, so that the complete data of the previous day can be dispatched and obtained, the integrity and the correctness of the service data of the service system in each time zone output by the data warehouse are ensured, the idempotent of the data full link is ensured, the relevant offshore enterprises can be effectively assisted to aggregate the processed data, and the data value is mined.
Fig. 1 shows a flowchart of a data processing method in a multi-time zone scenario, which may be performed by a data processing apparatus in the multi-time zone scenario, where the apparatus may be implemented in software and/or hardware, and may be generally integrated in an electronic device, where the electronic device includes a data warehouse, a data center, and the like, according to an example embodiment of the present disclosure.
As shown in fig. 1, the data processing method in the multi-time zone scenario may include the following steps:
The time zone in which the data warehouse is located, that is, the time zone to which the geographical location in which the data warehouse is located belongs, for example, if the data warehouse is in beijing, the time zone in which the data warehouse is located is the eighty-th area, and the acquired system time is the beijing time.
Illustratively, the system time of the time zone in which the data warehouse is located can be obtained in real time.
Illustratively, the system time for the time zone in which the data warehouse is located may be obtained periodically. For example, the system time of the time zone in which the data warehouse is located may be acquired every 10 minutes, half an hour, and the like. It should be noted that, the time interval for acquiring the system time of the time zone in which the data warehouse is located should be able to ensure that the system time acquired at a certain time is an integer, i.e., an integer time of 0 point (24 points), 1 point, 2 points, 3 points, etc., until 23 points.
And 102, determining the local time of the time zone of the business system according to the system time and the time difference between the time zone of the business system and the time zone of the data warehouse.
The business system is set in at least one region for enterprise to expand global business, the business systems in different regions adopt the same technical architecture, the same data infrastructure and logic design, and the business systems in different regions are configured according to local conditions. The number of the business systems may be one or more, and the business systems may be set according to actual business expansion requirements of an enterprise, which is not limited by the present disclosure. The scheme disclosed by the invention is also applicable to a scene of sharing the physical configuration of the service in multiple places, and the subsequent steps are not influenced.
As is well known, the world is divided into 24 time zones, the time difference between two adjacent time zones is 1 hour, and the time in the east is earlier than that in the northwest. In the embodiment of the present disclosure, after the system time of the time zone in which the data warehouse is located is obtained, the local time of the time zone in which the business system is located may be determined according to the system time and the time difference between the time zone in which the business system is located and the time zone in which the data warehouse is located.
The time difference between the time zone of the business system and the time zone of the data warehouse can be determined according to the time zone number of the difference between the time zone of the business system and the time zone of the data warehouse. For example, if the time zone in which the business system is located is the east nine area and the time zone in which the data warehouse is located is the east eight area, the time difference between the two areas is 1. For another example, if the time zone of the business system is the seven east zones and the time zone of the data warehouse is the eight east zones, the time difference between the two zones is-1
Exemplarily, assuming that the time zone of the data warehouse is the eight east zone, the system time is 2022, 6 month, 22 day 12:00, if the time zone of a certain service system is the nineteenth zone, the local time of the time zone of the service system is 12:00+ (9-8) =13:00, that is, the local time is 2022 years, 6 months, 22 days, 13: 00.
exemplarily, assuming that the time zone of the data warehouse is the eight east zone, the system time is 2022, 6 month, 22 day 12:00, the time zone in which a certain service system is located is west five zone, the local time of the time zone in which the service system is located is 12:00+ (-5-8) = -1:00+24:00-1 day =23:00, namely the local time is 2022 years, 6 months, 21 days, 23: 00.
step 103, when the local time of the time zone in which the service system is located is zero, starting a data scheduling batch corresponding to the system time to schedule target service data corresponding to the service system from a full data table set containing the service data of the service system, wherein the target service data is service data generated by the service system on the day before the local time, and the service data carries time zone information of the time zone in which the service system is located and generation time of the service data.
The full-scale data table set comprises a plurality of database tables, and each database table records service data reported by databases corresponding to the service systems of all parts of the world, that is, the databases corresponding to the service systems of all parts of the world report the service data to a unified data warehouse, and the data warehouse performs convergence processing on the received service data to generate the full-scale data table set. The service data recorded in each database table carries time zone information of the time zone in which the service system is located and generation time of the service data, and the time zone information and/or the generation time may be included in the service data reported by the database corresponding to the service system, or may be added to the data warehouse in the later stage, which is not limited by the present disclosure.
In the embodiment of the present disclosure, after determining the local time of the time zone in which each service system is located, it may further determine whether each local time is zero (0), if the local time of the time zone in which a certain service system is located is 0, it indicates that all data of the previous day of the service system have been generated, and may start a data scheduling batch corresponding to the system time, so as to schedule the target service data corresponding to the service system from the full data table set.
The data dispatching batch is configured to dispatch according to a point 0 of a time zone in which the service system is located, and the service data generated one day before the service system is acquired, so that the acquired target service data is the service data generated one day before the local time by the service system.
Exemplarily, assuming that the time zone in which the data warehouse is located is an eighty zone, the system time is 7/month/6/day/3, and the local time of the business system in which the time zone is an eighty zone is 7/month/6/day/0, at this time, the data scheduling batch corresponding to the 3-point time preconfigured in the data warehouse is started, and business data with time zone information of the eighty zone and time information of 7/month/5/0 to 7/month/5/23: 59 are scheduled from the full data table set as the target business data of the business system.
Further, after the target service data corresponding to the service system is acquired, the data warehouse may perform operations such as aggregation, association, calculation, and the like according to the model design based on the acquired target service data, and generate and cover the data model of each intermediate layer. In the calculation, in a calculation scenario with a strict data requirement, the acquired target service data may be filtered before calculation. For example, according to the zero time of the time zone in which the data warehouse is located, data before the zero time is filtered out, so that it is ensured that the service data participating in calculation are the data of the current day. For another example, according to the zero time of the time zone in which the data warehouse is located, the data after the 0 point on the day is calculated is filtered, so that the data on the previous day is analyzed. In a data elastic application scenario, that is, a timely index update is allowed according to generated data, in this scenario, the data may not be filtered, but a refresh deadline is newly added to prompt the timeliness of the data user, for example, a prompt message that "the data in the day is not generated and is still changed" is shown to the data user. And then, for the detail data of each region of business generated by processing the target business data, directly exporting the detail data of each region of business to flow back to each region of business system according to actual requirements. In addition, for data needing multidimensional statistics, multidimensional cross statistics is carried out, and the final result is distributed and returned to each local business system.
The data processing method in a multi-time zone scenario of the embodiment of the present disclosure obtains a system time of a time zone in which a data warehouse is located, and determines a local time of the time zone in which a business system is located according to the system time, a time difference between the time zone in which the business system is located and the time zone in which the data warehouse is located, and starts a data scheduling batch corresponding to the system time when the local time of the time zone in which the business system is located is zero, so as to schedule target business data corresponding to the business system from a full data table set including the business data of the business system, where the target business data is the business data generated by the business system on the day before the local time, and the business data carries time zone information of the time zone in which the business system is located and the generation time of the business data. By adopting the scheme disclosed by the invention, when the data dispatching batch dispatching data is started, the local time of the time zone in which the service system is located is zero, and at the moment, the service data of the service system in the previous day is completely generated, so that the complete data of the previous day can be dispatched and obtained, and the completeness and the correctness of the service data of the service system in each time zone output by the data warehouse are ensured.
In an optional implementation manner of the present disclosure, the databases corresponding to the service systems in each time zone may synchronize data to a unified data warehouse, and the data warehouse performs grouping and combining processing on the service data generated by each system, so as to generate a full data table set including the service data in all time zones. Thus, in the embodiments of the present disclosure, the method further includes:
receiving service data reported by a database corresponding to the service system according to a preset period;
and generating the full data table set according to the service data.
The database corresponding to the service system is used for storing the service data generated by the service system and synchronizing the service data to the data warehouse for processing. In the design process of each database, the storage of various time data is added, and the business data is stored in a data format of a timestamp, wherein the stored business data can carry time zone information and time strings, the time zone information is represented by a time zone of the location of a business system, the time strings are represented by time generated by the business data, and the time strings at least comprise months, days, hours and minutes, and can also comprise years.
In the embodiment of the present disclosure, the database corresponding to the service system may report the service data generated by the service system according to a preset period, where the preset period may be preset according to an actual requirement, for example, the preset period may be set to one hour. And the data warehouse receives the service data reported by the database and generates a full data table set according to the received service data.
For example, when generating a full-size data table set according to business data, a data warehouse may store business data reported by each received local database in a partitioned manner according to dates and hours, and then merge local data stored in the partitioned manner, for example, the same type of business data may be merged and stored in the same database table to obtain a plurality of database tables storing different types of data, thereby obtaining the full-size data table set including all local data.
Exemplarily, when generating a full-volume data table set according to business data, the data warehouse may write the business data with the same business logic into the same database table based on the data merging rule of the same database table, to obtain at least one database table; and constructing the full-scale data table set by utilizing the at least one database table. In the embodiment of the present disclosure, the same database table is used to store business data with the same business logic. By writing the service data of the same service logic into the same database table, grouping and merging of the service data are realized, and a full data table set containing all regional data can be obtained.
In an optional embodiment of the present disclosure, by receiving service data reported by a database corresponding to a service system according to a preset period and generating a full data table set according to the service data, service data in different time zones are aggregated in a unified data warehouse for grouping and combining, so that construction and maintenance costs of the data warehouse are saved, and by generating the full data table set according to the service data, grouping and combining of service data in various regions are realized.
In the embodiment of the present disclosure, the service data of each time zone stored in the full data table set carries the time zone information of the time zone in which the service system is located and the generation time of the service data, and when the database corresponding to each service system reports the service data to the data warehouse, the reported service data may not include the time zone information. Thus, in an optional implementation of the present disclosure, before the generating the full data table set according to the service data, the method further includes:
judging whether the received service data carries time zone information or not;
under the condition that the received service data do not carry the time zone information, acquiring a target database identifier of a database reporting the received service data;
inquiring a corresponding relation table of a preset database identifier and a located time zone, and determining a target time zone corresponding to the target database identifier;
and marking the target time zone on the received service data as the time zone information of the received service data.
The corresponding relation table between the database identification and the located time zone can be preset. For an enterprise expanding overseas business, it is known in which region the overseas business is developed, and the time zone to which the region belongs, so that the business system deployed in the region and the time zone to which the database belongs are determined, unique database identifiers can be allocated for the databases deployed in different regions, the corresponding relation between each database identifier and the time zone in which the database identifier is located is established, the corresponding relation table between the database identifier and the time zone in which the database identifier is located is obtained, and the corresponding relation table is imported into the database.
In the embodiment of the present disclosure, after receiving the service data reported by the databases corresponding to the service systems, the data warehouse may check the received service data, determine whether the received service data carries time zone information, and if the received service data lacks time zone information, may obtain a database identifier corresponding to the database reporting the service data, which is called a target database identifier, where the database identifier is used to uniquely represent the corresponding database, and one database corresponds to a unique database identifier, that is, the database identifiers correspond to the databases one to one. Then, the corresponding relation table between the preset database identifier and the time zone can be inquired, the target time zone corresponding to the target database identifier is determined, and the target time zone is used as the time zone information of the received service data and is marked on the received service data. Therefore, time zone marking is carried out on the service data which does not carry time zone information, and conditions are provided for subsequently scheduling the service data of each time zone.
In the embodiment of the present disclosure, when the database corresponding to each service system reports the service data to the data warehouse, the reported service data may not include the generation time of the service data, and for this situation, the data warehouse may mark the generation time of the received service data. Thus, in an optional implementation of the present disclosure, before the generating the full data table set according to the service data, the method further includes:
judging whether the received service data carries generation time or not;
and under the condition that the received service data does not carry the generation time, marking the current time of the received service data on the received service data as the generation time.
In the embodiment of the disclosure, when the data warehouse receives the service data reported by the database, the current time of receiving the service data may be recorded, and after the data warehouse receives the service data reported by the database corresponding to each service system, the data warehouse may check the received service data, determine whether the received service data carries the generation time, and if the received service data lacks the generation time, may use the current time of receiving the service data as the generation time and mark the generation time on the received service data. Therefore, the generation time mark is realized for the service data which does not carry the generation time, and conditions are provided for the subsequent scheduling of the service data of the previous day of each time zone.
Considering that the time difference between the time zones is an integral hour, for example, the difference between two adjacent time zones is one hour, the east eight zone is three hours earlier than the east five zone, and the like, if the local time of the time zone of the business system determined according to the system time of the time zone of the data warehouse and the time difference between the time zone of the business system and the time zone of the data warehouse is zero, the system time of the time zone of the data warehouse should be an integral point, for example, when the time zone of the data warehouse is the east eight zone, and the system time is 3 points, the local time of the business system in the east five zone is 0 point. Therefore, in an optional embodiment of the present disclosure, before determining the local time of the time zone in which the business system is located according to the system time, the time difference between the time zone in which the business system is located and the time zone in which the data warehouse is located, the method further includes: determining the system time to be an integer point. Therefore, the local time of the time zone of the business system is determined only when the system time of the time zone of the data warehouse is the whole point, so that unnecessary time conversion operation can be avoided, and computing resources are saved.
With the continuous expansion of enterprise services, the services of an enterprise may be distributed all over the world, service systems are deployed in twenty-four time zones of the world, for such a scenario, a data warehouse may be applied to a full-time zone scenario, where the full-time zone is twenty-four time zones, the data warehouse is applied to the full-time zone scenario, that is, service data generated by the service systems deployed in the twenty-four time zones are all reported to a unified data warehouse, and the data warehouse performs data scheduling by using hourly batches, that is, twenty-four data scheduling batches are started at each integral point to perform data scheduling. Thus, in an optional implementation manner of the present disclosure, in a case that the data warehouse is applied to a full time zone scenario, each whole time of a time zone in which the data warehouse is located corresponds to one data dispatching batch, which is equivalent to one time zone corresponding to one data dispatching batch, and twenty-four data dispatching batches are set in the data warehouse, the method further includes: and when the time of the whole point of the time zone of the data warehouse arrives, starting all data dispatching batches to dispatch the service data corresponding to the service system of each time zone from the full data table set.
In the embodiment of the present disclosure, when the data warehouse is applied to a full time zone scenario, 24 data dispatching batches are set in the data warehouse, and when the hour of the time zone in which the data warehouse is located arrives, that is, the system time of the time zone in which the data warehouse is located is the hour, all (24) data dispatching batches are started to perform data dispatching, so as to dispatch the service data corresponding to the service system in each time zone from the full data table set. That is, every time the system time of the time zone in which the data warehouse is located reaches the whole point, all data scheduling matching is started to perform data scheduling. It can be understood that, in all the started data dispatching batches, the local time of the time zone where the service system is located is 0, the dispatching is the complete data of the previous day of the service system, and the dispatching of other data dispatching batches is the service data of the previous hour generated by the service system of the corresponding time zone, so that the T + (1 hour) data dispatching is realized.
For example, if the time zone where the data warehouse is located is the eighty zone, the system time is 3 o ' clock, and the local time of the business system in the eighty zone is 0 o ' clock, the data dispatching batch corresponding to the 3 o ' clock is all data generated by the business system in the eighty zone in the previous day, and the timeliness of the generated data is T + (1 day); the local time of the service system in the east nine area is 4 o ' clock, the data dispatching batch corresponding to the 23 o ' clock dispatches the data generated in the first hour of the east nine area, and when the system time of the time zone in which the data warehouse is located is 3 o ' clock, the data dispatching batch for dispatching the data of the service system in the east nine area dispatches the service data generated in the 4 hour of the service system on the day, so as to realize the T + (1 hour) data dispatching.
In the embodiment of the disclosure, when the data warehouse is applied to the full time zone scene, each integral point time of the time zone in which the data warehouse is located corresponds to one data scheduling batch, and when the integral point time of the time zone in which the data warehouse is located arrives, all data scheduling batches are started to schedule the service data corresponding to the service system of each time zone from the full data table set, so that the data scheduling of the hour level in the full time zone scene is realized while the data of the previous day of the service system at the local time zero point is completely scheduled.
The exemplary embodiment of the present disclosure also provides a data processing apparatus in a multi-time zone scenario. Fig. 2 shows a schematic block diagram of a data processing apparatus in a multi-time zone scenario according to an exemplary embodiment of the present disclosure, and as shown in fig. 2, the data processing apparatus 20 in the multi-time zone scenario includes: a time acquisition module 201, a time determination module 202 and a data scheduling module 203.
The time obtaining module 201 is configured to obtain system time of a time zone where the data warehouse is located;
a time determining module 202, configured to determine local time of the time zone in which the business system is located according to the system time and a time difference between the time zone in which the business system is located and the time zone in which the data warehouse is located;
a data scheduling module 203, configured to start a data scheduling batch corresponding to the system time when the local time of the time zone in which the service system is located is zero, so as to schedule target service data corresponding to the service system from a full data table set including service data of the service system, where the target service data is service data generated by the service system on a day before the local time, and the service data carries time zone information of the time zone in which the service system is located and generation time of the service data.
Optionally, the data processing apparatus 20 in the multi-time zone scenario further includes:
the data receiving module is used for receiving the service data reported by the database corresponding to the service system according to a preset period;
and the generating module is used for generating the full data table set according to the service data.
Optionally, the generating module is further configured to:
writing the service data with the same service logic into the same database table based on the data merging rules of the same database table to obtain at least one database table;
and constructing the full data table set by utilizing the at least one database table.
Optionally, the data processing apparatus 20 in the multi-time zone scenario further includes:
the first judging module is used for judging whether the received service data carries time zone information or not;
an identifier obtaining module, configured to obtain a target database identifier of a database that reports the received service data when the received service data does not carry the time zone information;
the time zone determining module is used for inquiring a corresponding relation table of a preset database identifier and the time zone where the preset database identifier is located, and determining a target time zone corresponding to the target database identifier;
a first marking module, configured to mark the target time zone on the received service data as time zone information of the received service data.
Optionally, the data processing apparatus 20 in the multi-time zone scenario further includes:
the second judging module is used for judging whether the received service data carries the generation time;
and the second marking module is used for marking the current time of the received service data as the generation time on the received service data under the condition that the received service data does not carry the generation time.
Optionally, the data processing apparatus 20 in the multi-time zone scenario further includes:
a determining module for determining the system time as an integer.
Optionally, when the data warehouse is applied to a full time zone scenario, each integral point time of the time zone in which the data warehouse is located corresponds to one data scheduling batch, and the data scheduling module 203 is further configured to:
and when the time of the whole point of the time zone of the data warehouse arrives, starting all data dispatching batches to dispatch the service data corresponding to the service system of each time zone from the full data table set.
The data processing device in the multi-time zone scene provided by the embodiment of the disclosure can execute any data processing method in the multi-time zone scene applicable to the electronic equipment provided by the embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method. Reference may be made to the description of any method embodiment of the disclosure that may not be described in detail in the embodiments of the apparatus of the disclosure.
An exemplary embodiment of the present disclosure also provides an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program, when executed by the at least one processor, is for causing the electronic device to perform a data processing method in a multi-time zone scenario according to an embodiment of the present disclosure.
The exemplary embodiments of the present disclosure also provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform a data processing method in a multi-temporal scenario according to an embodiment of the present disclosure.
The exemplary embodiments of the present disclosure also provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform a data processing method in a multi-time zone scenario according to an embodiment of the present disclosure.
Referring to fig. 3, a block diagram of a structure of an electronic device 1100, which may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 3, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data necessary for the operation of the device 1100 may also be stored. The calculation unit 1101, the ROM 1102, and the RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.
A number of components in electronic device 1100 connect to I/O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the electronic device 1100, and the input unit 1106 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device. Output unit 1107 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, a video/audio output terminal, a vibrator, and/or a printer. Storage unit 1108 may include, but is not limited to, a magnetic or optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, WiFi devices, WiMax devices, cellular communication devices, and/or the like.
The computing unit 1101 can be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The calculation unit 1101 performs the respective methods and processes described above. For example, in some embodiments, the data processing method in a multi-time zone scenario may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 1100 via the ROM 1102 and/or the communication unit 1109. In some embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform the data processing method in a multi-time zone scenario.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
Claims (10)
1. A data processing method under a multi-time zone scene, wherein the method comprises the following steps:
acquiring system time of a time zone in which a data warehouse is located;
determining the local time of the time zone of the business system according to the system time and the time difference between the time zone of the business system and the time zone of the data warehouse;
and under the condition that the local time of the time zone in which the service system is located is zero, starting a data dispatching batch corresponding to the system time to dispatch target service data corresponding to the service system from a full data table set containing the service data of the service system, wherein the target service data is the service data generated by the service system on the day before the local time, and the service data carries time zone information of the time zone in which the service system is located and the generation time of the service data.
2. The data processing method in a multi-temporal scenario as recited in claim 1, wherein the method further comprises:
receiving service data reported by a database corresponding to the service system according to a preset period;
and generating the full data table set according to the service data.
3. The data processing method under the multi-time zone scenario according to claim 2, wherein the generating the full data table set according to the service data includes:
writing the service data with the same service logic into the same database table based on the data merging rules of the same database table to obtain at least one database table;
and constructing the full-scale data table set by utilizing the at least one database table.
4. The data processing method under the multi-time zone scenario of claim 2, wherein before the generating the full data table set according to the service data, the method further comprises:
judging whether the received service data carries time zone information or not;
under the condition that the received service data do not carry the time zone information, acquiring a target database identifier of a database reporting the received service data;
inquiring a corresponding relation table of a preset database identifier and a located time zone, and determining a target time zone corresponding to the target database identifier;
and marking the target time zone on the received service data as the time zone information of the received service data.
5. The data processing method in a multi-temporal scenario as recited in claim 4, wherein the method further comprises:
judging whether the received service data carries generation time or not;
and under the condition that the received service data does not carry the generation time, marking the current time of the received service data on the received service data as the generation time.
6. A data processing method in a multi-time zone scenario as claimed in any of claims 1-5, wherein prior to said determining the local time of the time zone in which the business system is located, the method further comprises:
determining the system time to be an integer point.
7. The data processing method in a multi-time zone scenario of any one of claims 1 to 5, wherein, when the data warehouse is applied in a full-time zone scenario, each integral point in time of the time zone in which the data warehouse is located corresponds to a data dispatching batch, the method further comprises:
and when the time of the whole point of the time zone of the data warehouse arrives, starting all data dispatching batches to dispatch the service data corresponding to the service system of each time zone from the full data table set.
8. A data processing apparatus in a multi-time zone scenario, comprising:
the time acquisition module is used for acquiring the system time of the time zone in which the data warehouse is located;
the time determining module is used for determining the local time of the time zone of the business system according to the system time and the time difference between the time zone of the business system and the time zone of the data warehouse;
and the data scheduling module is used for starting a data scheduling batch corresponding to the system time under the condition that the local time of the time zone in which the service system is located is zero, so as to schedule target service data corresponding to the service system from a full data table set containing the service data of the service system, wherein the target service data is the service data generated by the service system on the day before the local time, and the service data carries time zone information of the time zone in which the service system is located and the generation time of the service data.
9. An electronic device, comprising:
a processor; and
a memory for storing a program, wherein the program is stored in the memory,
wherein the program comprises instructions which, when executed by the processor, cause the processor to carry out the method of data processing in a multi-temporal scenario according to any one of claims 1-7.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the data processing method in the multi-temporal scenario according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210942783.XA CN115017171A (en) | 2022-08-08 | 2022-08-08 | Data processing method and device in multi-time zone scene, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210942783.XA CN115017171A (en) | 2022-08-08 | 2022-08-08 | Data processing method and device in multi-time zone scene, electronic equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115017171A true CN115017171A (en) | 2022-09-06 |
Family
ID=83065798
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210942783.XA Pending CN115017171A (en) | 2022-08-08 | 2022-08-08 | Data processing method and device in multi-time zone scene, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115017171A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116775771A (en) * | 2023-08-23 | 2023-09-19 | 北京逐风科技有限公司 | Data synchronization method, device, system and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000042553A2 (en) * | 1999-01-15 | 2000-07-20 | Harmony Software, Inc. | Method and apparatus for processing business information from multiple enterprises |
| US20020122355A1 (en) * | 2001-01-10 | 2002-09-05 | Williams Joseph F. | Universal time zone conversion |
| CN103092959A (en) * | 2013-01-18 | 2013-05-08 | 中兴通讯股份有限公司 | Processing method and device for task time zone |
| US20130132470A1 (en) * | 2011-11-22 | 2013-05-23 | Christian Nicholas Vardin | Method and System for Storing Time-Dependent Data |
| CN103294774A (en) * | 2013-05-10 | 2013-09-11 | 中国工商银行股份有限公司 | Multi-timezone-based device and multi-timezone-based method for batch loading of data warehouses |
| CN107888654A (en) * | 2017-09-30 | 2018-04-06 | 北京诺君安信息技术股份有限公司 | A kind of method updated the data by time zone and server |
| CN112685447A (en) * | 2020-12-17 | 2021-04-20 | 贵阳语玩科技有限公司 | Cross-time-zone operation database system, data query method, medium, and server |
-
2022
- 2022-08-08 CN CN202210942783.XA patent/CN115017171A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000042553A2 (en) * | 1999-01-15 | 2000-07-20 | Harmony Software, Inc. | Method and apparatus for processing business information from multiple enterprises |
| US20020122355A1 (en) * | 2001-01-10 | 2002-09-05 | Williams Joseph F. | Universal time zone conversion |
| US20130132470A1 (en) * | 2011-11-22 | 2013-05-23 | Christian Nicholas Vardin | Method and System for Storing Time-Dependent Data |
| CN103092959A (en) * | 2013-01-18 | 2013-05-08 | 中兴通讯股份有限公司 | Processing method and device for task time zone |
| CN103294774A (en) * | 2013-05-10 | 2013-09-11 | 中国工商银行股份有限公司 | Multi-timezone-based device and multi-timezone-based method for batch loading of data warehouses |
| CN107888654A (en) * | 2017-09-30 | 2018-04-06 | 北京诺君安信息技术股份有限公司 | A kind of method updated the data by time zone and server |
| CN112685447A (en) * | 2020-12-17 | 2021-04-20 | 贵阳语玩科技有限公司 | Cross-time-zone operation database system, data query method, medium, and server |
Non-Patent Citations (1)
| Title |
|---|
| 许有志等: "基于多Agent的主动数据仓库的研究", 《计算机工程与设计》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116775771A (en) * | 2023-08-23 | 2023-09-19 | 北京逐风科技有限公司 | Data synchronization method, device, system and medium |
| CN116775771B (en) * | 2023-08-23 | 2024-01-26 | 北京逐风科技有限公司 | Data synchronization method, device, system and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107341643B (en) | Visual process display method and system | |
| CN111400356B (en) | Data query method, device and equipment | |
| US20200402648A1 (en) | Generating high confidence refills for unified workforce management | |
| CN113505520A (en) | Method, device and system for supporting heterogeneous federated learning | |
| CN115017171A (en) | Data processing method and device in multi-time zone scene, electronic equipment and medium | |
| US7310562B2 (en) | System and method for scheduling mold manufacturing | |
| CN113377872B (en) | Offline synchronization method, device and equipment of online system data in big data center | |
| CN115617480A (en) | Task scheduling method, device and system and storage medium | |
| CN114218199A (en) | Visual Portal system with data interaction and analysis functions | |
| CN114564249A (en) | Recommendation scheduling engine, recommendation scheduling method, and computer-readable storage medium | |
| US8700656B2 (en) | Method and system for implementing an on-demand scheduler | |
| CN112102099A (en) | Policy data processing method and device, electronic equipment and storage medium | |
| CN111275557A (en) | Method and device for controlling risk of resource management | |
| CN110852700A (en) | Data sorting method and device for workflow system | |
| US9021095B2 (en) | Method and system for implementing an on-demand scheduler in a mobile device | |
| US20230196292A1 (en) | Planning method and planning apparatus for production capacity allocation | |
| US11941441B2 (en) | Data movement and monitoring system | |
| CN114201501A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN116703092A (en) | Batch task scheduling method, device, equipment and medium based on resource allocation | |
| CN116109094A (en) | Resource scheduling method and system, storage medium and terminal | |
| JP2023177933A (en) | Member management support system | |
| CN117271104A (en) | Resource arrangement method, device, electronic equipment and storage medium | |
| CN114493108A (en) | Dispatcher operation recommendation method, recommendation device and recommendation system | |
| CN116679850A (en) | Automatic card punching result display method, device, computer equipment and storage medium | |
| CN116244320A (en) | Data pushing method and device, computer storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220906 |