CN111338668A - Method and device for upgrading code in real-time computing - Google Patents
Method and device for upgrading code in real-time computing Download PDFInfo
- Publication number
- CN111338668A CN111338668A CN202010092831.1A CN202010092831A CN111338668A CN 111338668 A CN111338668 A CN 111338668A CN 202010092831 A CN202010092831 A CN 202010092831A CN 111338668 A CN111338668 A CN 111338668A
- Authority
- CN
- China
- Prior art keywords
- task
- processing
- code
- data
- backup
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Abstract
This specification provides a method and apparatus for code upgrade in real-time computing, the method comprising: the method comprises the steps of obtaining an upgrading code of a current running code after receiving a code upgrading instruction of the current running code, triggering a processing kernel to run the upgrading code, starting a backup task to call data in a database according to a version sequence for processing, monitoring processing data versions of a main task and the backup task, stopping the main task by the processing kernel when the processing data versions of the main task and the backup task are the same, and enabling the processing kernel to continue to perform real-time calculation on the database by taking the backup task as a new main task. The method is used for upgrading codes, the real-time calculation tasks are operated at each moment, and the operated tasks are calculated in real time aiming at the database, so that the real-time calculation instantaneity is ensured.
Description
Technical Field
The present description relates to the field of real-time computing technologies, and in particular, to a method and an apparatus for code upgrade in real-time computing.
Background
The real-time calculation is a calculation capable of feeding back a calculation result in real time, emphasizes real-time performance, and has the characteristic of low delay. Based on this, the real-time calculation is widely applied in various application scenes, especially scenes with strong real-time performance such as early warning and online service.
The tasks of real-time computation are generated based on code execution. In order to improve the computing power of real-time computation and further shorten the delay of data processing, it is necessary to occasionally upgrade the code of real-time computation. Currently, the code upgrading method is to stop the currently running real-time computation, restart the real-time computing system, call and run a new upgrade code in the restarting process to start a new upgraded real-time computing task, and start real-time processing on data in a database after the new real-time computing task is started.
Disclosure of Invention
In view of the foregoing, the present specification provides a method and apparatus for code upgrade in real-time computing. One embodiment of the present description implements: and after receiving the code upgrading instruction, triggering the running upgrading code to start the backup task to call the data in the database according to the version sequence for processing, wherein the currently running main task is not stopped in the process, and the main task continues to call the data in the database according to the version sequence for processing. And in the stage of simultaneous operation of the main task and the backup task, monitoring the processing data versions of the main task and the backup task, stopping the main task when the processing data versions of the main task and the backup task are the same, and taking the backup task as a new main task to continue to perform real-time calculation on the database. In the code upgrading process, the real-time calculation task is operated at each moment, and the operated task carries out real-time calculation aiming at the database, so that the real-time calculation instantaneity is ensured.
According to one aspect of the present specification, there is provided a method for code upgrade in real-time computing, comprising: after a code upgrading instruction of a current running code is received, an upgrading code of the current running code is obtained, a main task of the current running code is used for carrying out real-time calculation on a database, the database has multi-version data storage capacity, and the main task calls corresponding version data in the database to carry out processing; triggering a processing kernel to run the upgrading code so as to start a backup task to call the data in the database according to the version sequence for processing, wherein the initial version of the data called by the backup task is set to be no later than the data version processed by the main task when the code upgrading instruction is received; monitoring whether the current processing data versions of the main task and the backup task are the same; and when the current processing data versions of the main task and the backup task are the same, enabling the processing core to stop the main task and enabling the processing core to take the backup task as a new main task to continue to perform real-time calculation on the database.
Optionally, in an example of the above aspect, further comprising: monitoring whether the current data processing progress of the main task and the backup task is the same; and when the current processing data versions of the main task and the backup task are the same, the step of stopping the main task by the processing core and taking the backup task as a new main task by the processing core to continue to perform real-time calculation on the database comprises the following steps: and when the current processing data versions and the current data processing progress of the main task and the backup task are the same, stopping the main task by the processing kernel, and taking the backup task as a new main task by the processing kernel to continue to perform real-time calculation on the database.
Optionally, in an example of the above aspect, it is determined that the data processing progress of the primary task and the backup task is the same according to at least one of the following manners: for the same data belonging to the same processed data version, the time difference between the ending processing time point of the backup task and the ending processing time point of the main task is within a first numerical range; and aiming at the same data belonging to the same processing data version, the backup task and the main task are in the same processing stage in the process of processing the same data.
Optionally, in an example of the above aspect, determining that the processing data versions of the primary task and the backup task are the same is performed according to the following: the time difference value between the ending processing time point of the backup task for the data corresponding to the processed data version and the ending processing time point of the main task for the data corresponding to the processed data version is within a second numerical range.
Optionally, in an example of the above aspect, triggering the processing core to execute the upgrade code includes: installing the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and triggering the processing core to run the upgrade code; or triggering a code installation device to install the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and enabling the processing core to run the upgrade code.
Optionally, in one example of the above aspect, the database comprises HBase or Kudu.
Optionally, in an example of the above aspect, further comprising: and synchronizing the processing results of the main task and the backup task to a search engine.
Optionally, in an example of the above aspect, the search engine performs deduplication processing on processing results of the main task and the backup task.
According to another aspect of the present specification, there is also provided an apparatus for code upgrade in real-time computing, comprising: the method comprises the steps that after an upgrade code obtaining unit receives a code upgrade instruction of a current running code, the upgrade code of the current running code is obtained, a main task of the current running code is used for carrying out real-time calculation on a database, the database has multi-version data storage capacity, and the main task calls corresponding version data in the database to carry out processing; a code running triggering unit triggers a processing kernel to run the upgrading code so as to start a backup task to call the data in the database according to the version sequence for processing, wherein the initial version of the data called by the backup task is set to be no later than the data version processed by the main task when the main task receives the code upgrading instruction; a task monitoring unit monitors whether the current processing data versions of the main task and the backup task are the same or not; and when the current processing data versions of the main task and the backup task are the same, the task switching unit enables the processing kernel to stop the main task and enables the processing kernel to take the backup task as a new main task to continue to perform real-time calculation on the database.
Optionally, in an example of the above aspect, the task monitoring unit further: monitoring the current data processing progress of the main task and the backup task; and the task switching unit further: and when the current processing data versions and the current data processing progress of the main task and the backup task are the same, stopping the main task by the processing kernel, and taking the backup task as a new main task by the processing kernel to continue to perform real-time calculation on the database.
Optionally, in an example of the above aspect, the code execution triggering unit further: installing the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and triggering the processing core to run the upgrade code; or triggering a code installation device to install the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and enabling the processing core to run the upgrade code.
Optionally, in one example of the above aspect, the database comprises HBase or Kudu.
Optionally, in an example of the above aspect, further comprising: and the processing result synchronization unit synchronizes the processing results of the main task and the backup task to a search engine.
According to another aspect of the present specification, there is also provided a computing device comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for code upgrade in real-time computing as described above.
According to another aspect of the present specification, there is also provided a machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the method for code upgrade in real-time computing as described above.
Drawings
A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the drawings, similar components or features may have the same reference numerals. The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the detailed description serve to explain the embodiments of the invention. In the drawings:
FIG. 1 is a diagram illustrating the stages involved in a code upgrade approach in the prior art;
FIG. 2 illustrates a flow diagram of a method for code upgrade in real-time computing of an embodiment of the present description;
FIG. 3 illustrates a schematic diagram of one example of a database with multi-version data storage capabilities of an embodiment of the present description;
FIG. 4 illustrates a schematic diagram of one example of monitoring processing data versions of a primary task and a backup task in accordance with an embodiment of the present description;
FIG. 5 is a schematic diagram illustrating one example of monitoring the processing data versions and data processing progress of the primary and backup tasks of the present specification;
FIG. 6 illustrates a block diagram of an apparatus for code upgrade in real-time computing of an embodiment of the present description; and
FIG. 7 is a block diagram of a computing device for a method of code upgrade in real-time computing in accordance with an embodiment of the present description.
Detailed Description
The subject matter described herein will be discussed with reference to example embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thereby implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. Changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as needed. In addition, features described with respect to some examples may also be combined in other examples.
As used herein, the term "include" and its variants mean open-ended terms in the sense of "including, but not limited to. The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first," "second," and the like may refer to different or the same object. Other definitions, whether explicit or implicit, may be included below. The definition of a term is consistent throughout the specification unless the context clearly dictates otherwise.
In this specification, the term "data version" refers to a timestamp corresponding to data in a database, and if the timestamp corresponding to the data is different, the version is different.
In the present specification, the term "data processing progress" refers to a processing progress for data in one version.
Real-time computing tasks are generated based on code execution. In order to improve the computing power of real-time computation and further shorten the delay of data processing, it is necessary to occasionally upgrade the code of real-time computation. Currently, the code upgrade method is to stop the currently running real-time computing task and then restart the real-time computing system to run the upgrade code to start the upgraded new real-time computing task.
Fig. 1 is a schematic diagram illustrating various stages involved in a code upgrading method in the prior art. As shown in FIG. 1, t1Time of day indicates the receipt of an upgrade order, t2The time indicates the end of the code upgrade. Then at t1The stage before the moment is before the code upgrade, at t1Time to t2The phase between moments is code risingStage process at t2The stage after the moment is after the code upgrade.
At t1Before the code before the moment is upgraded, the running real-time computing task is generated based on the running code to be upgraded, and the real-time computing task carries out real-time processing on the data in the database.
At the time of reaching t1At that time, the upgrade instruction is received, and at this time, the currently running real-time computing task (i.e., the original real-time computing task in fig. 1) is stopped, and the operation of code upgrade is started to be executed.
At t1Time to t2Restarting the real-time computing system in the process of code upgrading at the moment, calling a new upgrading code in the restarting process and running the upgrading code to start a new real-time computing task after upgrading until t2The time of day.
At t2At the moment, the new real-time computing task is started and completed and starts to be executed, which indicates that the code upgrading process is finished.
At t2And after the code at the moment is upgraded, the running new real-time computing task carries out real-time processing on the data in the database, wherein the new real-time computing task is generated based on the running of the upgraded code.
However, in the whole process shown in fig. 1, there are real-time computing tasks to process data in the database only in two stages, namely before and after code upgrade, and no real-time computing task is running in the process of code upgrade, so that data written into the database at this stage cannot be processed in time, which affects real-time performance of data processing and also affects real-time performance of data processing of subsequent writing.
In order to solve the above problems, the present specification provides a method and apparatus for code upgrade in real-time computing. The embodiments of the present specification realize: and after receiving the code upgrading instruction, triggering the running upgrading code to start the backup task to call the data in the database according to the version sequence for processing, wherein the currently running main task is not stopped in the process, and the main task continues to call the data in the database according to the version sequence for processing. And in the stage of simultaneous operation of the main task and the backup task, monitoring the processing data versions of the main task and the backup task, stopping the main task when the processing data versions of the main task and the backup task are the same, and taking the backup task as a new main task to continue to perform real-time calculation on the database. In the code upgrading process, the real-time calculation task is operated at each moment, and the operated task carries out real-time calculation aiming at the database, so that the real-time calculation instantaneity is ensured.
A method and an apparatus for code upgrade in real-time computing according to embodiments of the present specification will be described in detail below with reference to the accompanying drawings.
FIG. 2 illustrates a flow diagram of a method for code upgrade in real-time computing of an embodiment of the present description.
The method for upgrading codes in real-time computing provided by the specification can be applied to a code upgrading center, and the code upgrading center can be a device or a system with a code upgrading function. The code upgrading center may be configured to perform an upgrading operation on a currently running code, that is, to stop running the currently running code and start running an upgrading code of the code.
As shown in fig. 2, at 210, after a code upgrade instruction of the current running code is received, an upgrade code of the current running code is acquired.
In this specification, the code upgrade instruction is used to instruct to perform an upgrade operation on a code, and the code upgrade instruction may carry an identifier of a code to be upgraded, so as to determine the code to be upgraded. For example, if the current running code is identified as 12 in this specification, the information contained in one field of the received code upgrade instruction is 12.
The code upgrading instruction can be sent by a worker through the management background, and can also be sent after the management background detects the upgrading code of the currently running code.
In this specification, the current running code and the upgrade code may be stored locally, at which time the upgrade code of the current running code is acquired locally. The update code may also be stored in other storage devices, for example, an OSS (object storage Service) for storing the code, and at this time, the update code of the current running code is obtained from the OSS.
Executing a code performs a task, and the task of the currently executed code is the task that the code performs at runtime. In this specification, the task of currently running code is the main task, which is to perform real-time computation on the database.
In this specification, the data stored in the database includes data to be processed, and when there is data to be processed, an operating task may write the data to be processed into the database, and a subsequent operating task reads the data from the database to perform processing.
The database has multi-version data storage capability, i.e., the database can store a plurality of different versions of data. The version of the data refers to a timestamp corresponding to the data, and the timestamp corresponding to the data may be a time point when the data is written into the database or a specified time point. In this specification, a data version may serve as a consumption site for a task to perform real-time computation, indicating a location of data currently processed by the real-time computed task.
FIG. 3 illustrates a schematic diagram of one example of a database with multi-version data storage capabilities of an embodiment of the present description. As shown in fig. 3, the database includes five versions for data a and five versions for data B. Each version of data corresponds to a time stamp, for example, data A with a version of 8:00:00 indicates that the time for writing the data A into the database is 8 o' clock. In this specification, a piece of data (for example, data a) is described as an example.
When there are multiple versions for the same data in the database, the multiple versions may be arranged in reverse order of time, i.e., with the latest version of the data in front. Thus, when the data needs to be processed, the version of the data read from the database is the latest version of the data.
Taking FIG. 3 as an example, data A is the price of the product, 1 Yuan at 8:00:00, 2 Yuan at 8:05:00, 3 Yuan at 8:10:00, 4 Yuan at 8:15:00, and 5 Yuan at 8:20: 00. When the data A needs to be processed, the price of the data A read from the database is 5 yuan.
In one example of the present description, the database may include HBase or Kudu, although other databases with multi-version data storage capabilities may also be included. The examples in this specification will be described with HBase as an example.
In this specification, based on the multi-version data storage capability of the database, the primary task may call the corresponding version data in the database to process. In one example, the data may be stored in the database in the order of the timestamps, and the main task performs data processing in turn in the order of the timestamps.
Taking the data a in fig. 3 as an example, the data a of each version is sequentially stored in the database according to the sequence of the timestamps from morning to evening, and the data a is sequentially processed by the main task according to the sequence of the timestamps from morning to evening.
At 220, the processing core is triggered to run upgrade code to initiate a backup task to call the data in the database in version order for processing.
In this specification, the processing core is used to start and run code. In one example, the code upgrading center is in communication connection with the processing core, and after the upgrading code is acquired, the code upgrading center sends a code running instruction to the processing core to trigger the processing core to run the upgrading code. In one example, the code upgrade center and the processing core may belong to the same system or device, or may belong to different systems or devices, respectively.
In this specification, the version order is the order of the timestamps, that is, the backup task calls the data in the database in turn from the morning to the evening according to the timestamps for processing. The backup task calls the data according to the same version sequence as the main task.
In this specification, the initial version of the backup task refers to the version of the first data called after the backup task is started. Taking data a in fig. 3 as an example, if it is determined that 8:05:00 is the initial version, the backup task takes data a of 8:05:00 as the first data to start calling and processing, and then calls data a of 8:10:00 to process, and calls and processes in this order.
The initial version of the data called by the backup task is set to be no later than the version of the data processed by the primary task when the code upgrade instruction is received. In one example, the starting version of data called by the backup task may be the version of data that the primary task processed when the code upgrade instructions were received. Taking data a in fig. 3 as an example, when the code upgrade instruction is received, and the version of data currently processed by the primary task is 8:15:00, 8:15:00 may be determined as the initial version of the backup task.
In another example, the starting version of data called by the backup task may be set to be earlier than the version of data processed by the primary task when the code upgrade instruction is received. The starting version may be a specified version of the data version that was processed earlier by the primary task when the code upgrade instruction was received. Taking data a in fig. 3 as an example, when a code upgrade instruction is received, the data version processed by the main task is 8:15:00, and the versions earlier than the data version are 8:00:00, 8:05:00, and 8:10:00, then 8:05:00 may be determined as the initial version of the backup task.
In an example of this specification, after the upgrade code is obtained, the upgrade code is installed locally, the installed upgrade code is loaded into an execution environment of the processing core, and then the processing core is triggered to run the upgrade code.
In another example of the present specification, the operation of installing the upgrade code is performed by a third party apparatus (e.g., a code installation apparatus). After the upgrade code is acquired, the code installation device may be triggered to install the upgrade code, load the installed upgrade code into an execution environment of the processing core, and cause the processing core to run the upgrade code. In one example, the triggering may be sending a code installation instruction to the code installation apparatus, where the code installation instruction may carry an identification of the upgrade code, storage location information, and the like. In another example, the triggering may be sending the upgrade code to a code installation apparatus, which installs after receiving the upgrade code.
At 230, the current processing data versions of the primary task and the backup task are monitored.
In this specification, when the primary task and the backup task are running simultaneously, the processing data version of the backup task is earlier than that of the primary task until the processing data versions of the backup task and the primary task are the same. In this process, in an example, the processing data versions of the main task and the backup task may be monitored in real time, and in this example, the code upgrade center may directly monitor the processing data versions of the main task and the backup task and changes of the processing data versions. In another example, the processing kernel may send the processing data versions of the primary task and the backup task to the code upgrade center, and the code upgrade center determines whether the received processing data versions of the primary task and the backup task are the same.
In this specification, the processing capacity of the backup task is greater than that of the main task, and for example, the backup task has more resources such as CPU resources and the number of machines than the main task. Thus, the processing time of the backup task is less than the processing time of the main task for the same version of the same data. Therefore, in the process of simultaneously operating the main task and the backup task, the processing speed of the backup task is higher than that of the main task, and finally the backup task can catch up with the main task, namely the processing data version of the backup task is the same as that of the main task.
FIG. 4 illustrates a schematic diagram of one example of monitoring processing data versions of a primary task and a backup task in accordance with an embodiment of the present description. As shown in FIG. 4, the initial version of the backup task is 8:00:00, while the processed data version of the primary task is 8:10: 00. Then, the main task and the backup task are both processed sequentially according to the version sequence direction (i.e. the direction indicated by the arrow in fig. 4), and the processing data versions of the main task and the backup task are monitored in the process, for example, when the processing data version of the main task is monitored to be 8:15:00, the processing data version of the backup task is 8:10:00 until the processing data versions of the main task and the backup task are the same, and are both 8:20: 00.
At 240, a determination is made as to whether the current processing data versions of the primary task and the backup task are the same. If not, flow returns to block 230 to continue monitoring the processed data versions of the primary and backup tasks. If so, flow proceeds to block 250.
In one example of the present specification, it may be determined that the current processing data versions of the primary task and the backup task are the same according to the following manner.
The time difference value between the ending processing time point of the backup task for the data corresponding to the processed data version and the ending processing time point of the main task for the data corresponding to the processed data version is in a second numerical range.
In this example, the processed data version is the version to which the processed completed data corresponds. The processing time of the backup task is the same as or similar to that of the main task aiming at the same data version. The second range of values may be specified. For example, the second value range is 2 seconds, the data with the data version of 8:00:00 includes data a and data B, and for the processing of the data corresponding to the data version, it is monitored that the processing ending time point of the backup task is 8:00:05, the processing ending time point of the main task is 8:00:06, the difference between the two processing ending time points is 1 second, and within the range of 2 seconds, it can be determined that the currently processed data versions of the main task and the backup task are the same at this time.
Of course, in addition to the above, it may be determined whether the current processing data versions of the primary task and the backup task are the same in other ways. For example, it may be directly detected whether the respective current processing data versions of the primary task and the backup task are the same.
In an example of the present specification, on the basis of the operation performed at the above-described block 230, the data processing progress of the main task and the backup task may also be monitored. A data processing progress is said for a version, and when processing a version of data, the monitored data processing progress is the processing progress of the version of data. Taking fig. 3 as an example, when the processing data version of the main task is 8:00:00, the monitored data processing progress of the main task is a processing progress for 8:00: 00.
And monitoring the current processing data versions and the current data processing progress of the main task and the backup task, and judging whether the processing data versions and the data processing progress of the main task and the backup task are the same. When the current processing data version and the current data processing progress of the primary and backup tasks are the same, flow proceeds to block 250.
Fig. 5 is a schematic diagram illustrating an example of monitoring the processing data versions and data processing progress of the main task and the backup task according to the present specification. As shown in fig. 5, the processing for the data a includes four phases, where the data of the initial version of the backup task is the data a of 8:00:00, and the data processed by the main task is the data a of 8:10:00, until the backup task starts processing the data a of 8:20:00 at 8:20:30, and the data processing progress of the main task on the data a of 8:20:00 is the second phase end. At 8:20:40, the data processing progress of the backup task and the main task for the data A at 8:20:00 is the same, and both the backup task and the main task are finished in the third stage, and the processing data version and the data processing progress of the main task and the backup task are the same.
The above example in this specification implements: whether the main task and the backup task are synchronized can be judged more accurately by using two factors of the processing data version and the data processing progress, so that the time point when the main task and the backup task are synchronized can be accurately obtained, the main task and the backup task are switched at the time point, and the data delayed in processing can be reduced or even avoided.
In one example of the present specification, it is determined that the data processing schedules of the main task and the backup task are the same according to at least one of the following manners.
In the first mode, for the same data belonging to the same processing data version, the time difference between the end processing time point of the backup task and the end processing time point of the main task is within a first numerical range.
In this first manner, a first range of values may be specified. For example, the processing procedure of the data a of the version 8:20:00 includes four stages, and when the end processing time points when the backup task and the main task process the data a of the version 8:20:00 are both 8:20:05, it may be determined that the data processing schedules of the main task and the backup task are the same.
In the second way, for the same data belonging to the same processing data version, the processing stage of the backup task is the same as that of the main task in the process of processing the same data.
In the above manner, the processing procedure for each data can be divided into a plurality of stages. In one example, the corresponding information is fed back after each stage is completed. Based on the above, when the backup task and the main task feed back the same information and the feedback time difference of the same information is within the specified value range, the processing stages of the backup task and the main task are the same.
For example, for data a of 8:15:00, the backup task feeds back information a indicating that the first stage is completed at 8:15:05, and the main task feeds back the information a at 8:15:07, so that it can be determined that the data processing progress of the main task and the backup task is the same at this time and both the main task and the backup task are completed at the first stage.
At block 250, the processing core is caused to stop the primary task and is caused to continue real-time computation against the database with the backup task as the new primary task.
Taking fig. 3 as an example, when it is monitored that the processed data versions of the main task and the backup task are both 8:10:00, the main task is stopped, the backup task is used as a new main task to continue processing the data a of 8:10:00 until the processing of the data a of 8:10:00 is completed, and then the data a of 8:15:00 and the data a of 8:20:00 are sequentially calculated in real time.
In one example of the present specification, after determining that the primary task is stopped, the code of the primary task may be deleted to save storage space.
In one example of the present specification, after obtaining the processing results of the primary task and the backup task, the processing results of the primary task and the backup task may be synchronized to the search engine. In this example, the database may include data corresponding to the index in the search engine.
In one example of the present specification, when the processing results of the main task and the backup task are processing results for the same data, the search engine may perform deduplication processing on the processing results of the main task and the backup task, and retain only one of the processing results.
Fig. 6 shows a block diagram of an apparatus for code upgrade in real-time computing (hereinafter simply referred to as code upgrade apparatus) 600 according to an embodiment of the present specification. As shown in fig. 6, the code upgrading apparatus 600 includes an upgrade code acquiring unit 610, a code running triggering unit 620, a task monitoring unit 630, and a task switching unit 640.
The upgrade code obtaining unit 610 obtains the upgrade code of the current running code after receiving the code upgrade instruction of the current running code, a main task of the current running code is used for performing real-time calculation for a database, the database has multi-version data storage capability, and the main task calls corresponding version data in the database to perform processing. In one example of the present specification, the database includes HBase or Kudu.
The code running triggering unit 620 triggers the processing core to run the upgrade code to start the backup task to call the data in the database in the version order for processing, and the initial version of the data called by the backup task is set to be no later than the data version processed by the main task when receiving the code upgrade instruction. In one example of the present specification, the code execution triggering unit 620 further: installing an upgrade code, loading the installed upgrade code into an execution environment of a processing kernel, and triggering the processing kernel to run the upgrade code; or triggering the code installation device to install the upgrade code and loading the installed upgrade code into the execution environment of the processing core, and enabling the processing core to run the upgrade code.
The task monitoring unit 630 monitors whether the current processing data versions of the primary task and the backup task are the same.
When the current processing data versions of the main task and the backup task are the same, the task switching unit 640 enables the processing core to stop the main task, and enables the processing core to take the backup task as a new main task to continue real-time calculation for the database.
In one example of the present specification, the task monitoring unit 630 may further monitor the current data processing progress of the main task and the backup task. The task switching unit 640 further: and when the current processing data versions and the current data processing progress of the main task and the backup task are the same, stopping the main task by the processing kernel, and taking the backup task as a new main task by the processing kernel to continue to perform real-time calculation on the database.
In one example of the present specification, the code upgrading apparatus 600 may further include a processing result synchronizing unit that synchronizes the processing results of the main task and the backup task to the search engine.
Embodiments of a method and apparatus for code upgrade in real-time computing according to the present description are described above with reference to fig. 1 through 6.
The code upgrading apparatus in the present specification may be implemented by hardware, or may be implemented by software, or a combination of hardware and software. The software implementation is taken as an example, and is formed by reading corresponding computer program instructions in the storage into the memory for operation through the processor of the device where the software implementation is located as a logical means. In this specification, the code upgrading apparatus may be implemented by, for example, a computing device.
FIG. 7 is a block diagram of a computing device 700 for a method of code upgrade in real-time computing in accordance with an embodiment of the present description.
As shown in fig. 7, computing device 700 may include at least one processor 710, storage (e.g., non-volatile storage) 720, memory 730, and communication interface 740, and the at least one processor 710, storage 720, memory 730, and communication interface 740 are connected together via a bus 760. The at least one processor 710 executes at least one computer-readable instruction (i.e., the elements described above as being implemented in software) stored or encoded in memory.
In one embodiment, computer-executable instructions are stored in the memory that, when executed, cause the at least one processor 710 to: after a code upgrading instruction of a current running code is received, an upgrading code of the current running code is obtained, a main task of the current running code is used for carrying out real-time calculation aiming at a database, the database has multi-version data storage capacity, and the main task calls corresponding version data in the database to carry out processing; triggering the processing kernel to run the upgrading code so as to start a backup task to call the data in the database according to the version sequence for processing, wherein the initial version of the data called by the backup task is set to be no later than the data version processed by the main task when the code upgrading instruction is received; monitoring whether the current processing data versions of the main task and the backup task are the same; and when the current processing data versions of the main task and the backup task are the same, stopping the main task by the processing kernel, and taking the backup task as a new main task by the processing kernel to continue to perform real-time calculation on the database.
It should be appreciated that the computer-executable instructions stored in the memory, when executed, cause the at least one processor 710 to perform the various operations and functions described above in connection with fig. 2-6 in the various embodiments of the present description.
According to one embodiment, a program product, such as a machine-readable medium, is provided. A machine-readable medium may have instructions (i.e., elements described above as being implemented in software) that, when executed by a machine, cause the machine to perform various operations and functions described above in connection with fig. 2-6 in the various embodiments of the present specification.
Specifically, a system or apparatus may be provided which is provided with a readable storage medium on which software program code implementing the functions of any of the above embodiments is stored, and causes a computer or processor of the system or apparatus to read out and execute instructions stored in the readable storage medium.
In this case, the program code itself read from the readable medium can realize the functions of any of the above-described embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code form part of the present invention.
Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-R, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs), magnetic tapes, nonvolatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or from the cloud via a communications network.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Not all steps and elements in the above flows and system structure diagrams are necessary, and some steps or elements may be omitted according to actual needs. The execution order of the steps is not fixed, and can be determined as required. The apparatus structures described in the above embodiments may be physical structures or logical structures, that is, some units may be implemented by the same physical entity, or some units may be implemented by a plurality of physical entities, or some units may be implemented by some components in a plurality of independent devices.
The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration," and does not mean "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described technology. However, the techniques may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.
Although the embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the embodiments of the present disclosure are not limited to the specific details of the embodiments, and various simple modifications may be made to the technical solutions of the embodiments of the present disclosure within the technical spirit of the embodiments of the present disclosure, and all of them fall within the scope of the embodiments of the present disclosure.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the description is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (15)
1. A method for code upgrade in real-time computing, comprising:
after a code upgrading instruction of a current running code is received, an upgrading code of the current running code is obtained, a main task of the current running code is used for calculating a database in real time, the database has multi-version data storage capacity, and the main task calls corresponding version data in the database to process;
triggering a processing kernel to run the upgrading code so as to start a backup task to call the data in the database according to the version sequence for processing, wherein the initial version of the data called by the backup task is set to be no later than the data version processed by the main task when the code upgrading instruction is received;
monitoring whether the current processing data versions of the main task and the backup task are the same; and
and when the current processing data versions of the main task and the backup task are the same, stopping the main task by the processing core, and taking the backup task as a new main task by the processing core to continue to perform real-time calculation on the database.
2. The method of claim 1, further comprising:
monitoring whether the current data processing progress of the main task and the backup task is the same; and
when the current processing data versions of the main task and the backup task are the same, stopping the main task by the processing core, and taking the backup task as a new main task by the processing core to continue real-time calculation for the database comprises:
and when the current processing data versions and the current data processing progress of the main task and the backup task are the same, stopping the main task by the processing kernel, and taking the backup task as a new main task by the processing kernel to continue to perform real-time calculation on the database.
3. The method of claim 2, wherein the data processing progress of the primary task and the backup task is determined to be the same according to at least one of:
for the same data belonging to the same processed data version, the time difference between the ending processing time point of the backup task and the ending processing time point of the main task is within a first numerical range; and
and aiming at the same data belonging to the same processing data version, in the process of processing the same data, the processing stage of the backup task is the same as that of the main task.
4. A method according to any one of claims 1 to 3, wherein it is determined that the current processing data versions of the primary task and the backup task are the same according to:
the time difference value between the ending processing time point of the backup task for the data corresponding to the processed data version and the ending processing time point of the main task for the data corresponding to the processed data version is within a second numerical range.
5. The method of claim 1 or 2, wherein triggering a processing core to execute the upgrade code comprises:
installing the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and triggering the processing core to run the upgrade code; or
And triggering a code installation device to install the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and enabling the processing core to run the upgrade code.
6. The method according to claim 1 or 2, wherein the database comprises HBase or Kudu.
7. The method of claim 1 or 2, further comprising:
and synchronizing the processing results of the main task and the backup task to a search engine.
8. The method of claim 7, wherein the search engine deduplicates processing results of the primary task and the backup task.
9. An apparatus for code upgrade in real-time computing, comprising:
the system comprises an upgrade code acquisition unit, an upgrade code acquisition unit and a data processing unit, wherein the upgrade code acquisition unit is used for acquiring an upgrade code of a current operation code after receiving a code upgrade instruction of the current operation code, a main task of the current operation code is used for carrying out real-time calculation aiming at a database, the database has multi-version data storage capacity, and the main task calls corresponding version data in the database to carry out processing;
a code running triggering unit, which triggers the processing kernel to run the upgrade code so as to start a backup task to call the data in the database according to the version sequence for processing, wherein the initial version of the data called by the backup task is set to be no later than the data version processed by the main task when receiving the code upgrade instruction;
the task monitoring unit is used for monitoring whether the current processing data versions of the main task and the backup task are the same or not; and
and the task switching unit is used for stopping the main task by the processing kernel when the current processing data versions of the main task and the backup task are the same, and enabling the processing kernel to take the backup task as a new main task to continue to perform real-time calculation on the database.
10. The apparatus of claim 9, the task monitoring unit further to:
monitoring the current data processing progress of the main task and the backup task; and
the task switching unit further:
and when the current processing data versions and the current data processing progress of the main task and the backup task are the same, stopping the main task by the processing kernel, and taking the backup task as a new main task by the processing kernel to continue to perform real-time calculation on the database.
11. The apparatus of claim 9 or 10, wherein the code execution triggering unit is further to:
installing the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and triggering the processing core to run the upgrade code; or
And triggering a code installation device to install the upgrade code, loading the installed upgrade code into an execution environment of the processing core, and enabling the processing core to run the upgrade code.
12. The apparatus according to claim 9 or 10, wherein the database comprises HBase or Kudu.
13. The apparatus of claim 9 or 10, further comprising:
and the processing result synchronization unit synchronizes the processing results of the main task and the backup task to a search engine.
14. A computing device, comprising:
at least one processor, and
a memory coupled with the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method of any of claims 1-8.
15. A machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010092831.1A CN111338668B (en) | 2020-02-14 | 2020-02-14 | Method and device for upgrading code in real-time computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010092831.1A CN111338668B (en) | 2020-02-14 | 2020-02-14 | Method and device for upgrading code in real-time computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111338668A true CN111338668A (en) | 2020-06-26 |
| CN111338668B CN111338668B (en) | 2022-05-10 |
Family
ID=71181562
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010092831.1A Active CN111338668B (en) | 2020-02-14 | 2020-02-14 | Method and device for upgrading code in real-time computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111338668B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1190833A (en) * | 1997-02-11 | 1998-08-19 | 埃克斯塞尔开关公司 | Redundancy arrangement for telecommunications system |
| US20070294574A1 (en) * | 2006-06-05 | 2007-12-20 | Dmp Electronics Inc. | Dual computer for system backup and being fault-tolerant |
| CN101145119A (en) * | 2006-09-15 | 2008-03-19 | 中兴通讯股份有限公司 | Data synchronization method in upgrading process |
| CN101996083A (en) * | 2009-08-21 | 2011-03-30 | 中兴通讯股份有限公司 | Mirror image updating method and device |
| CN102081552A (en) * | 2009-12-01 | 2011-06-01 | 华为技术有限公司 | Method, device and system for transferring from physical machine to virtual machine on line |
| CN102855314A (en) * | 2012-08-27 | 2013-01-02 | 济南大学 | Real-time database backup system and method based on FPGA (field programmable gate array) |
| CN106487863A (en) * | 2015-09-02 | 2017-03-08 | 阿里巴巴集团控股有限公司 | A kind of based on the gray scale dissemination method of distributed system, client and platform |
| CN108170464A (en) * | 2017-12-27 | 2018-06-15 | 联想(北京)有限公司 | A kind of version upgrading method, service platform and electronic equipment |
| US20190212999A1 (en) * | 2018-01-11 | 2019-07-11 | Macronix International Co., Ltd. | System boot code clone |
| CN110046172A (en) * | 2018-12-29 | 2019-07-23 | 阿里巴巴集团控股有限公司 | In line computation data processing method and system |
-
2020
- 2020-02-14 CN CN202010092831.1A patent/CN111338668B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1190833A (en) * | 1997-02-11 | 1998-08-19 | 埃克斯塞尔开关公司 | Redundancy arrangement for telecommunications system |
| US20070294574A1 (en) * | 2006-06-05 | 2007-12-20 | Dmp Electronics Inc. | Dual computer for system backup and being fault-tolerant |
| CN101145119A (en) * | 2006-09-15 | 2008-03-19 | 中兴通讯股份有限公司 | Data synchronization method in upgrading process |
| CN101996083A (en) * | 2009-08-21 | 2011-03-30 | 中兴通讯股份有限公司 | Mirror image updating method and device |
| CN102081552A (en) * | 2009-12-01 | 2011-06-01 | 华为技术有限公司 | Method, device and system for transferring from physical machine to virtual machine on line |
| CN102855314A (en) * | 2012-08-27 | 2013-01-02 | 济南大学 | Real-time database backup system and method based on FPGA (field programmable gate array) |
| CN106487863A (en) * | 2015-09-02 | 2017-03-08 | 阿里巴巴集团控股有限公司 | A kind of based on the gray scale dissemination method of distributed system, client and platform |
| CN108170464A (en) * | 2017-12-27 | 2018-06-15 | 联想(北京)有限公司 | A kind of version upgrading method, service platform and electronic equipment |
| US20190212999A1 (en) * | 2018-01-11 | 2019-07-11 | Macronix International Co., Ltd. | System boot code clone |
| CN110046172A (en) * | 2018-12-29 | 2019-07-23 | 阿里巴巴集团控股有限公司 | In line computation data processing method and system |
Non-Patent Citations (3)
| Title |
|---|
| VAMYIP: "Gracefully reload Gunicorn for changes to Python Code", 《HTTPS://DISCUSS.ERPNEXT.COM/T/GRACEFULLY-RELOAD-GUNICORN-FOR-CHANGES-TO-PYTHON-CODE/55207》 * |
| WANGKUN_DHCC: "Nginx运行时二进制更新代码分析", 《HTTPS://BLOG.CSDN.NET/WANGKUN_DHCC/ARTICLE/DETAILS/8613930》 * |
| 隐林: "DataWorks支持PyODPS类型任务", 《HTTPS://WWW.SOHU.COM/A/231546698_612370》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111338668B (en) | 2022-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180285216A1 (en) | Virtual Machine Recovery Method and Virtual Machine Management Device | |
| CN112383610B (en) | Synchronous processing method and system for block chain state data | |
| CN110895484A (en) | Task scheduling method and device | |
| CN110825495A (en) | Container cloud platform recovery method, device, equipment and readable storage medium | |
| CN110225078B (en) | Application service updating method, system and terminal equipment | |
| CN110895488B (en) | Task scheduling method and device | |
| CN112596771A (en) | Method and system for uninterrupted service upgrade | |
| US20220335011A1 (en) | System and Method for Eliminating Full Rescan Synchronizations on Service Restarts | |
| CN110895483A (en) | Task recovery method and device | |
| CN110895487A (en) | Distributed task scheduling system | |
| CN111258850A (en) | Method and device for updating software information based on Linux system | |
| US20130086572A1 (en) | Generation apparatus, generation method and computer readable information recording medium | |
| CN107463390B (en) | Software upgrading method and upgrading server | |
| CN111538585A (en) | Js-based server process scheduling method, system and device | |
| CN112328602B (en) | Method, device and equipment for writing data into Kafka | |
| CN111338668B (en) | Method and device for upgrading code in real-time computing | |
| CN110895486A (en) | Distributed task scheduling system | |
| CN110895485A (en) | Task scheduling system | |
| CN107094160B (en) | Method and system for downloading data file | |
| CN110888739B (en) | Distributed processing method and device for delayed tasks | |
| CN116560802A (en) | Virtual machine load-based virtual machine self-adaptive thermal migration method and system | |
| CN114500128B (en) | Flow control charging method, device, system, electronic equipment, medium and product | |
| CN112685063B (en) | Feature library updating method, device, network equipment and readable storage medium | |
| CN111026529B (en) | Task stopping method and device of distributed task processing system | |
| CN112905341A (en) | Distributed load balancing service information continuous inheritance method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |