CN112860576A - Business processing method, device and equipment based on gray level verification - Google Patents
Business processing method, device and equipment based on gray level verification Download PDFInfo
- Publication number
- CN112860576A CN112860576A CN202110279326.2A CN202110279326A CN112860576A CN 112860576 A CN112860576 A CN 112860576A CN 202110279326 A CN202110279326 A CN 202110279326A CN 112860576 A CN112860576 A CN 112860576A
- Authority
- CN
- China
- Prior art keywords
- gray level
- gray
- service
- database
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012795 verification Methods 0.000 title claims abstract description 100
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 75
- 238000004519 manufacturing process Methods 0.000 claims abstract description 51
- 238000012545 processing Methods 0.000 claims abstract description 48
- 230000008569 process Effects 0.000 claims abstract description 46
- 238000012360 testing method Methods 0.000 claims abstract description 39
- 238000010367 cloning Methods 0.000 claims abstract description 12
- 230000001360 synchronised effect Effects 0.000 claims description 40
- 238000013515 script Methods 0.000 claims description 37
- 238000012986 modification Methods 0.000 claims description 11
- 230000004048 modification Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 6
- 230000003044 adaptive effect Effects 0.000 claims 3
- 238000013523 data management Methods 0.000 description 10
- 230000006872 improvement Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000001419 dependent effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000013522 software testing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/368—Test management for test version control, e.g. updating test cases to a new software version
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The embodiment of the specification provides a business processing method, a business processing device and business processing equipment based on gray level verification. The method comprises the following steps: receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services; acquiring gray level service data from a gray level database through the gray level configuration information; the gray database comprises a database for gray testing obtained by cloning a production database; adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment; and processing the gray level service according to the gray level service program and the gray level service data. By the method, the gray scale verification process can be completed by utilizing the gray scale service data under the condition of ensuring the authenticity of the data.
Description
Technical Field
The embodiment of the specification relates to the technical field of big data, in particular to a business processing method, a business processing device and business processing equipment based on gray level verification.
Background
With the development of internet technology, the requirements of users on application software are higher and higher. In order to adapt to technical update and user requirements, developers often need to continuously update versions, and correspondingly, software development processes also need to be continuously tested. Due to the limitation of the test environment, research and development personnel generally cannot verify data processing and analysis algorithms through real production data. Some problems may need to be discovered after release. In order to solve the above problems, the testing process is closer to the actual application environment, and the application can be tested in a gray level verification manner.
The gray scale verification is a testing mode of providing a newly developed version to a part of target testing users and providing the software of the original version to other users in the software testing process. In the gray scale verification process, the actual application effect of the software can be determined more truly by collecting the use process of a target test user and problem feedback. And gradually expanding the number of target test users along with the progress of the verification process, and finally finishing version replacement.
Because the gray scale test process cannot affect the original version and data, the existing gray scale test usually additionally designs a set of application and data to provide a target test user with gray scale test. Such a method is not only time-consuming, but also often causes a certain deviation of the test result from the actual situation, thereby affecting the test process. Therefore, a method for conveniently and accurately performing gray level verification is needed.
Disclosure of Invention
An embodiment of the present specification aims to provide a business processing method, device and equipment based on gray level verification, so as to solve the problem of how to conveniently and accurately perform gray level verification.
In order to solve the above technical problem, an embodiment of the present specification provides a business processing method based on gray level verification, including: receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services; acquiring gray level service data from a gray level database through the gray level configuration information; the gray database comprises a database for gray testing obtained by cloning a production database; adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment; and processing the gray level service according to the gray level service program and the gray level service data.
An embodiment of the present specification further provides a business processing apparatus based on grayscale verification, including: the gray level service receiving module is used for receiving the gray level service sent by the test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services; the gray scale service data acquisition module is used for acquiring gray scale service data from a gray scale database through the gray scale configuration information; the gray database comprises a database for gray testing obtained by cloning a production database; the business program adjusting module is used for adjusting the business program into a gray business program by utilizing the gray configuration information; the gray scale service program is a program adapted to a gray scale verification environment; and the gray level service processing module is used for processing the gray level service according to the gray level service program and the gray level service data.
The embodiment of the present specification further provides a business processing device based on gray level verification, which includes a memory and a processor; the memory to store computer program instructions; the processor to execute the computer program instructions to implement the steps of: receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services; acquiring gray level service data from a gray level database through the gray level configuration information; the gray database comprises a database for gray testing obtained by cloning a production database; adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment; and processing the gray level service according to the gray level service program and the gray level service data.
As can be seen from the technical solutions provided by the embodiments of the present specification, after receiving a gray-scale service, the embodiments of the present specification obtain gray-scale service data for processing the gray-scale service based on gray-scale configuration information included in the gray-scale service, and then adjust a service program for executing the gray-scale service according to the gray-scale configuration information so that the service program is adapted to a gray-scale verification environment. And completing the gray level service by using the adjusted gray level service program and the gray level service data, thereby realizing a gray level verification process. By the method, the data used in the gray level verification process is independent of the actual production data, and the data used in the verification process is consistent with the actual production data, so that the accuracy of the gray level verification process is ensured. In addition, the method can effectively utilize data in the big data cloud service, and further guarantees the effect of the gray level verification process.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the specification, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a block diagram of a gray scale verification system according to an embodiment of the present disclosure;
fig. 2 is a flowchart of a business processing method based on gray level verification according to an embodiment of the present disclosure;
FIG. 3 is a flowchart of a task allocation method according to an embodiment of the present disclosure;
FIG. 4 is a flowchart of a task allocation method according to an embodiment of the present disclosure;
fig. 5 is a block diagram of a business processing apparatus based on grayscale verification according to an embodiment of the present disclosure;
fig. 6 is a block diagram of a business processing device based on grayscale verification according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without any creative effort shall fall within the protection scope of the present specification.
In order to better understand the inventive concept of the present application, a grayscale verification system according to an embodiment of the present specification will be described first. As shown in FIG. 1, the grayscale verification system 100 can include a grayscale database 110, a business program cloning component 120, a data management component 130, and a grayscale verification execution component 140.
The grayscale verification system 100 is a system for implementing a particular grayscale verification process. Specifically, the gray scale verification system 100 may receive a gray scale service sent by a user, adjust the gray scale service to a version suitable for gray scale verification, and obtain gray scale service data corresponding to the gray scale service, thereby completing a processing flow of the gray scale service.
The grayscale database 110 may be used to store grayscale service data. Since data is essential for gray level verification, the gray level database 110 needs to provide corresponding data to ensure the gray level verification process is performed successfully. The gray database 110 may be created in the form of a separate gray library within the production database. Specifically, the grayscale database 110 may be, for example, a distributed database supporting multiple types of Hive or HBase. When processing the gray level service, only the gray level database 110 has complete read-write permission, and only the production database has read permission, thereby ensuring that the gray level verification does not affect the original production data.
The service program cloning component 120 is configured to clone the version program package submitted by the user, and generate the version program package meeting the gray level verification requirement. The grey-scale service submitted by the user can comprise a version program package. The version program package is used for indicating the flow of business processing. Specifically, the version package may be a package developed based on a big data platform in a platform, and includes contents such as a version document and a version program. The version program can perform configuration management on the jobs, for example, job groups in the jobs can be distinguished according to different service functions, and in the same job group, the job, the running batch frequency and the dependency relationship among the jobs can also be configured, so as to better execute the processing flow of the services.
The data management component 130 is used to obtain data required in the grayscale verification process and maintain the grayscale database 110. A specific maintenance procedure may be to update data in the grayscale database 110. Specifically, the data management component 130 may copy the data that meets the synchronization requirement to the grayscale database 110 after screening the data in the production database.
The grayscale verification execution component 140 is used to process the grayscale service after configuration is completed. Specifically, the processing can be performed based on preset logic set by the system, and the cloned gray version program package can also be directly submitted to a big data service cloud and verified by a big data scheduling service. Correspondingly, certain gray level test users are configured in the big data service cloud, and the isolation of the gray level verification process and the actual production process is ensured.
The grayscale verification system 100 may be connected to a client. The client specifically may refer to a client corresponding to a target test user corresponding to the grayscale verification. The client may receive a gray level verification request of a user, and further send the gray level verification request to the gray level verification system 100 as a gray level service.
Based on the gray scale verification system, a business processing method based on gray scale verification in the embodiment of the present specification is introduced. The execution subject of the business processing method based on the gray scale verification is the gray scale verification system. As shown in fig. 2, the business processing method based on grayscale verification may include the following specific implementation steps.
S210: receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to the gray business.
The test user may be the user specified by the grayscale verification process. In the gray level verification process, the tested application is only opened to a part of users, and the opened users are the testing users.
The gray level service is the service submitted by the test user. In the gray level verification process, other users except for the test outdoor still use the original production database, so that the data called by the gray level service needs to be distinguished. When the client submits the gray level service of the user, the data acquisition source and the configuration information cannot be set in advance, and the transmitted gray level service is still the service set according to the original processing logic under the ordinary condition, so the gray level service can be configured into the service conforming to the gray level verification process.
The gray level service may include a service program and gray level configuration information. The business program is used for indicating the flow of business processing. Specifically, the service program may be a package developed based on a big data platform in a platform, and includes contents such as a version document and a version program. The version program can perform configuration management on the jobs, for example, job groups in the jobs can be distinguished according to different service functions, and in the same job group, the job, the running batch frequency and the dependency relationship among the jobs can also be configured, so as to better execute the processing flow of the services.
The gray configuration information may specify a mapping relationship between the gray service and a gray verification system, including a data index and an environment configuration corresponding to the gray service. The data index may be a category of specific grayscale service data required for the grayscale service. The environment configuration can be used for describing environment configuration information of the gray level business corresponding to the gray level verification system, so that the gray level business after configuration can be processed by the gray level verification system.
In some embodiments, the grayscale configuration information may include at least one of an application name, a grayscale database category, a database identification, a data table identification, a data synchronization period, and a data synchronization frequency.
Correspondingly, a gray level service identifier may exist in the gray level service, and is used for identifying that the gray level service is a service for gray level verification, so that a gray level verification system can distinguish a normal service from a gray level service conveniently.
S220: acquiring gray level service data from a gray level database through the gray level configuration information; the gray scale database comprises a database for gray scale testing obtained by cloning a production database.
The gray level database is a database for storing gray level service data. The data called by the original version of the application is sourced from the production database. The production database stores data provided for the user to interact with the original application. In order not to affect the data in the production database, the grey scale database needs to be independent from the production database. In order to ensure that the gray scale service data in the gray scale database is close to the actual application scene, the gray scale database may be a database for gray scale testing obtained by cloning the production database.
Based on a data management component in the gray scale verification system, the data in the production database can be cloned to the gray scale database. Specifically, the data management component may locate database tables corresponding to the production environment, screen data meeting synchronization requirements from the database tables, and synchronize the data to the same-name table of the grayscale library. The data management component maintains a synchronization configuration table in which libraries and tables involved in gray level verification are recorded. The specific structure of the synchronization configuration table may be as shown in table 1 below.
TABLE 1
Correspondingly, after the synchronous operation is performed, the data management component may further perform status recording on each synchronous operation by using the cluster synchronization information table. The specific structure of the cluster synchronization information table may be as shown in table 2 below.
TABLE 2
In some embodiments, the grayscale database includes a Hive table-based storage database, and the production database includes an underlying Hadoop cluster-based HDFS file system. The grayscale database may be constructed by first receiving a synchronization job. The synchronization job is used for instructing to add the gray scale service data in the gray scale database, and specifically, may be instructing to copy the data in the production database to the gray scale database. The synchronization operation may also identify whether this is the first synchronization operation. If the data in the production database is not synchronized before the gray database, the synchronization operation is the first synchronization operation; if the gray database has been synchronized with the data in the database, the synchronization operation is not the first synchronization operation. The data in the gray database can be obtained in different ways according to different types corresponding to the synchronous operation.
In the case where the synchronization job is a first synchronization job, data meeting the synchronization requirement in the production database may be scanned as data to be synchronized, and the data to be synchronized may be copied to the grayscale database.
Hive table data organizes management data files by time as a partition directory. Because the first synchronization is carried out, the gray library has no data, and the production data needing to be synchronized is all data files meeting the batch requirements under the corresponding application library configured in the synchronization configuration table. And scanning a data file directory positioned in the HDFS according to the corresponding application, library name and table name information of the synchronous configuration table, further scanning subdirectories which take batch dates as partitions and screen the data file directory which meets the synchronous requirement.
In some embodiments, the synchronization requirement may be that a difference between a date to which the data corresponds and a preset date is less than a synchronization date difference. When the difference between the number of days between the date batch corresponding to the acquired data and the current date batch is smaller than the synchronous date difference value, the data can be considered to meet the synchronous requirement. The synchronization date difference value can be set according to the requirements of practical application, and can be 10 days for example. A specific synchronization operation may be to separate the absolute path of the data file that meets the synchronization requirements by a line break, and write the absolute path to the specified HDFS file (e.g., HDFS:// hash 1/srcFileList. cfg) in sequence. And simultaneously writing the path into the Src _ hdfsList _ path column corresponding to the current sync _ id in the cluster synchronization information table.
And the data is screened by utilizing the synchronous date difference value, so that the data synchronized to the gray level database is the recent data, the timeliness of the data in the gray level database is ensured, and the accuracy of the gray level verification process is improved.
And under the condition that the synchronous operation is not the first synchronous operation, acquiring a data modification date corresponding to the gray-scale service data. The data modification date may be the date that the data was last updated. Thereafter, a grayscale database synchronization date may also be obtained. The grayscale database synchronization date includes a date that the grayscale database was last updated.
And comparing the data modification date of each data with the gray database synchronization date, and under the condition that the data modification date is earlier than the gray database synchronization date, indicating that the data is not modified since the last synchronization data, updating the data does not influence the application of the data, and determining the gray business data corresponding to the data modification date as the data to be updated.
If the data modification date of certain data is not earlier than the grayscale database synchronization date, the data is called and modified since the last database synchronization, and records in actual production are covered if the data is directly covered, so that the data does not need to be processed.
After determining the data to be updated, scanning data corresponding to the data to be updated in a production database as data to be synchronized, and copying the data to be synchronized to a gray database to cover the data to be updated. Specifically, the absolute path of the data file meeting the synchronization requirement is divided by a line break and written into the specified HDFS file (e.g., HDFS:// hash 1/srcFileList. cfg) in sequence. And simultaneously writing the path into the Src _ hdfsList _ path column corresponding to the current sync _ id in the cluster synchronization information table.
Through the process, the data timeliness of the gray database is guaranteed, meanwhile, the coverage of actual production data is avoided, and the smooth proceeding of the gray verification process is guaranteed.
In some embodiments, the grayscale database may further include an HBase table-based database, and correspondingly, the production database may include an unstructured database. Data in the HBase table generally corresponds to a time stamp record, and the data needing to be synchronized is determined according to the time stamp record. And taking the data with the timestamp in accordance with the preset time range as the data to be synchronized, and copying the data to be synchronized to a gray database.
During specific execution, the line keys of the full-table scanning line key can be used for acquiring the data lines of the source table line by line, and the line keys and the time stamps of each line of data are extracted. And if the time stamp accords with the time range configured by the synchronous configuration table and the row key cannot retrieve the data in the target grey table, writing the complete information of the data row into the target grey table, and if the time stamp does not accord with the writing rule, not performing the data synchronization. And processing the data lines of the source table line by line until the data of the source table is scanned. And after synchronization is finished, updating a cluster synchronization information table, setting the process of the synchronization operation marked by the sync _ id as successful, and marking an end _ time timestamp.
The above process is summarized in combination with fig. 3. When synchronizing data in the grayscale database, step 301 is performed first, and after the data management component starts a synchronization job, the data management component determines the type of the synchronization job. If the operation is the first synchronous operation of the Hive table, the data files of the cluster HDFS can be produced by the mobile phone according to the first synchronous mode, and synchronization is waited; and if the Hive table is not synchronized for the first time, screening the data files of the production cluster HDFS meeting the synchronization requirement according to a non-first-time synchronization mode, and waiting for synchronization. For the synchronization of the Hive table, after the above steps 302 or 303 are performed, step 304 may be performed to perform the data synchronization of the Hive table. If the type of the synchronization job is HBase table synchronization, step 305 is executed to perform data synchronization according to the synchronization mode of the HBase table.
Because the data synchronization process occupies network resources and disk IO, and further affects the execution of other concurrent services, the gray scale service data in the gray scale database can be acquired at the service low peak in order to reduce the influence of the data synchronization process on the system. The traffic low period includes a period in which traffic processing amount is lower than a preset traffic threshold, for example, may be a morning period.
Because there is a large difference between the grayscale database stored based on the Hive table and the database stored based on the HBase table, before receiving the synchronization job, the data management component may also scan the cluster synchronization information table, determine the data synchronization type as Hive or HBase, and then perform the corresponding synchronization operation.
S230: adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment.
Since the business program included in the grayscale business is still a corresponding program when processing a general business, the way of acquiring data by the program, the flow of executing the business by the program, and the environment configuration of the program itself are different from those of the grayscale verification process. Therefore, in order to ensure normal processing of the service, the service program may be adjusted to a gray level service program based on the gray level configuration information, where the gray level service program is a program adapted to the gray level verification environment.
Specifically, the flow of acquiring the gray level service program may be as shown in fig. 4. After the step 401 is executed, the service script and the service configuration file are obtained from the service program, the step 402 is executed to modify the gray scale configuration item, the step 403 is executed to modify the gray scale service script, and finally the step 404 is executed to integrate the gray scale configuration item and the gray scale service script to generate the gray scale service program.
In some embodiments, the service program may include a service script and a service configuration file. These two types of files can be distinguished by the directory in which the file is located and the suffix of the file. The code script is usually hql file or jar package, and the job dependent configuration file is usually csv or xls file. And the version program cloning component scans the directories and the files one by the file folders generated after decompressing the version packet corresponding to the service program, and then locates and acquires the application program script and the operation dependent configuration file.
In the adjusting process, the configuration items in the service configuration file can be modified into the gray level configuration items corresponding to the gray level verification, the service scripts are modified into the gray level service scripts suitable for the gray level verification, and then the gray level service programs are obtained by integrating the gray level configuration items and the gray level service scripts.
The service configuration file is mainly used for distinguishing the operation groups in the service according to the specific functions of the service. The service profile may be composed of a plurality of configuration record rows, each configuration row for representing an application job. Specifically, the service configuration file may include at least one configuration item of a job group name, a job name, a unified scheduling component script path, a scheduling parameter, a run batch frequency, and a job dependency relationship.
Wherein the unified scheduling component script path configuration item represents a common script required by the current job. The specific common components are divided into program execution, data import and data export. Wherein data import and data export are associated with Hive or HBase databases and resources, e.g., a production job calls Hive script parameters to Hive warehouse production library and production queue, a production job executes HBase program to HBase Namespace, version clone component needs to replace production library and physical resources with grayscale library and physical resources. The dependency configuration item between the jobs represents other job sets depended on by the current job, the job set can be empty, one or more, and each job of the job set is indicated by a job group name and a job name. The dependency relationship among the operations defines the execution sequence of all the operations in the operation group, and all other operations which are required to be depended by each operation and meet the execution requirement are completely executed.
When the gray configuration items are modified specifically, attention can be paid to analyzing each configuration item, firstly, the name of a job group is changed into a gray job group added with a gray verification mark, then, a unified scheduling component script is analyzed, and scheduling parameters are replaced and set according to the category of the scheduling component; and finally, analyzing and replacing the content of the dependency relationship between the jobs, wherein the related production job group is changed into a gray level job group.
When the service script is modified into the gray level service script, certain difference exists according to different types of the database.
When the gray database is a database based on a Hive table, the cloning core of the Hive script is to replace the database and the resource queue, when the script operates the table, a specific database needs to be indicated, and before each script is executed, the used resource queue needs to be determined. Therefore, the resource queue and the script code corresponding to the service script can be determined, the resource queue is replaced by the queue corresponding to the gray database, and the code related to the production database in the script code is replaced by the code corresponding to the gray database.
When the gray scale database is based on the HBase table, the program code itself is not adjusted, but only the configuration file or the configuration parameters of the HBase program are adjusted, so that the configuration file and the configuration parameters of the service script can be adjusted to be the gray scale configuration file and the gray scale configuration parameters corresponding to the gray scale verification environment.
The specific modification process may also be adjusted according to other conditions in the practical application, and is not limited to the above example, and is not described herein again.
After the gray level service script and the gray level configuration item are obtained, the version program cloning component can repackage the gray level service script and the gray level configuration item into a gray level version package according to a mode of packaging and producing the version package, so that the service processing can be completed under the condition of being suitable for gray level verification.
S240: and processing the gray level service according to the gray level service program and the gray level service data.
When the modified gray scale service program and gray scale service data are obtained, the gray scale service program and the gray scale service data are already adapted to the gray scale verification process, so that the gray scale service program and the gray scale service data can be directly utilized to complete the gray scale service processing. When the gray level business is processed, the processing logic of the system can be utilized to complete the processing of the business, or the gray level business program and the gray level business data can be directly packaged and submitted to a big data cloud service for processing. The specific processing procedure may be set according to an actual application situation, and is not limited to the above example, and is not described herein again.
As can be seen from the introduction of the above embodiment, after receiving the gray scale service, the method obtains the gray scale service data for processing the gray scale service based on the gray scale configuration information included in the gray scale service, and then adjusts the service program for executing the gray scale service according to the gray scale configuration information so as to adapt to the environment of gray scale verification. And completing the gray level service by using the adjusted gray level service program and the gray level service data, thereby realizing a gray level verification process. By the method, the data used in the gray level verification process is independent of the actual production data, and the data used in the verification process is consistent with the actual production data, so that the accuracy of the gray level verification process is ensured. In addition, the method can effectively utilize data in the big data cloud service, and further guarantees the effect of the gray level verification process.
A business processing apparatus based on gray scale verification according to an embodiment of the present specification is introduced based on the business processing method based on gray scale verification corresponding to fig. 2. As shown in fig. 5, the business processing apparatus based on grayscale verification includes the following modules.
A gray level service receiving module 510, configured to receive a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to the gray business.
A gray scale service data obtaining module 520, configured to obtain gray scale service data from a gray scale database according to the gray scale configuration information; the gray scale database comprises a database for gray scale testing obtained by cloning a production database.
A business program adjusting module 530, configured to adjust the business program into a gray business program by using the gray configuration information; the gray scale service program is a program adapted to a gray scale verification environment.
And the gray level service processing module 540 is configured to process the gray level service according to the gray level service program and the gray level service data.
Based on the business processing method based on gray level verification corresponding to fig. 2, an embodiment of the present specification provides a business processing device based on gray level verification. As shown in fig. 6, the business process device based on grayscale verification may include a memory and a processor.
In this embodiment, the memory may be implemented in any suitable manner. For example, the memory may be a read-only memory, a mechanical hard disk, a solid state disk, a U disk, or the like. The memory may be used to store computer program instructions.
In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or processor and a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, an embedded microcontroller, and so forth. The processor may execute the computer program instructions to perform the steps of: receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services; acquiring gray level service data from a gray level database through the gray level configuration information; the gray database comprises a database for gray testing obtained by cloning a production database; adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment; and processing the gray level service according to the gray level service program and the gray level service data.
In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
From the above description of the embodiments, it is clear to those skilled in the art that the present specification can be implemented by software plus the necessary first hardware platform. Based on such understanding, the technical solutions of the present specification may be essentially or partially implemented in the form of software products, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments of the present specification.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The description is operational with numerous first or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
While the specification has been described with examples, those skilled in the art will appreciate that there are numerous variations and permutations of the specification that do not depart from the spirit of the specification, and it is intended that the appended claims include such variations and modifications that do not depart from the spirit of the specification.
Claims (13)
1. A business processing method based on gray level verification is characterized by comprising the following steps:
receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services;
acquiring gray level service data from a gray level database through the gray level configuration information; the gray database comprises a database for gray testing obtained by cloning a production database;
adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment;
and processing the gray level service according to the gray level service program and the gray level service data.
2. The method of claim 1, wherein the gray scale configuration information comprises at least one of an application name, a gray scale database category, a database identification, a data table identification, a data synchronization period, and a data synchronization frequency.
3. The method of claim 1, wherein the grayscale database comprises a Hive table-based database; the grayscale database is obtained by:
receiving synchronous operation; the synchronous operation is used for indicating that gray level service data are added in a gray level database;
under the condition that the synchronous operation is the first synchronous operation, scanning data meeting the synchronous requirement in a production database as data to be synchronized; the production database comprises an HDFS file system based on a bottom layer Hadoop cluster;
and copying the data to be synchronized into a gray database.
4. The method of claim 3, wherein the synchronization requirement includes that a difference between a date to which the data corresponds and a preset date is less than a synchronization date difference.
5. The method of claim 3, wherein after receiving the synchronization job, further comprising:
acquiring a data modification date corresponding to the gray-scale service data under the condition that the synchronous operation is not the first synchronous operation;
under the condition that the data modification date is earlier than the synchronization date of the gray level database, determining the gray level service data corresponding to the data modification date as the data to be updated; the grayscale database synchronization date comprises the date that the grayscale database updated data last time;
scanning data corresponding to the data to be updated in a production database to serve as data to be synchronized;
and copying the data to be synchronized to a gray database to cover the data to be updated.
6. The method of claim 1, wherein the grayscale database comprises an HBase table-based database; the grayscale database is obtained by:
receiving synchronous operation; the synchronous operation is used for indicating that gray level service data are added in a gray level database;
acquiring a time stamp of data in a production database; the production database comprises an unstructured database;
taking the data with the timestamp in accordance with the preset time range as the data to be synchronized;
and copying the data to be synchronized to a gray database.
7. The method of claim 3 or 6, wherein the grayscale service data in the grayscale database is obtained during low-peak periods of service; the traffic low peak period comprises a period when the traffic processing capacity is lower than a preset traffic threshold value.
8. The method of claim 1, wherein said utilizing said gray-scale configuration information to adjust said business process to a gray-scale business process comprises:
acquiring a service configuration file and a service script in the service program;
modifying the configuration items in the service configuration file into gray level configuration items corresponding to gray level verification;
modifying the service script into a gray level service script adaptive to gray level verification;
and integrating the gray level configuration item and the gray level service script to obtain a gray level service program.
9. The method of claim 8, wherein the business profile includes at least one configuration item of a job group name, a job name, a unified scheduling component script path, a scheduling parameter, a run batch frequency, and a job dependency.
10. The method of claim 8, wherein the grayscale database comprises a Hive table-based database; the step of modifying the service script into a gray level service script adaptive to gray level verification comprises the following steps:
determining a resource queue and a script code corresponding to the service script;
replacing the resource queue with a queue corresponding to a gray database;
and replacing codes related to the production database in the script codes with codes corresponding to the gray database.
11. The method of claim 8, wherein the grayscale database comprises an HBase table-based database; the step of modifying the service script into a gray level service script adaptive to gray level verification comprises the following steps:
and adjusting the configuration file and the configuration parameters of the service script into a gray level configuration file and gray level configuration parameters corresponding to a gray level verification environment.
12. A business processing device based on gray level verification is characterized by comprising:
the gray level service receiving module is used for receiving the gray level service sent by the test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services;
the gray scale service data acquisition module is used for acquiring gray scale service data from a gray scale database through the gray scale configuration information; the gray database comprises a database for gray testing obtained by cloning a production database;
the business program adjusting module is used for adjusting the business program into a gray business program by utilizing the gray configuration information; the gray scale service program is a program adapted to a gray scale verification environment;
and the gray level service processing module is used for processing the gray level service according to the gray level service program and the gray level service data.
13. A business processing device based on gray level verification comprises a memory and a processor;
the memory to store computer program instructions;
the processor to execute the computer program instructions to implement the steps of: receiving a gray level service sent by a test user; the gray level service comprises a service program and gray level configuration information; the business program is used for expressing the execution flow of the gray level business; the gray configuration information is used for indicating data indexes and environment configurations corresponding to gray services; acquiring gray level service data from a gray level database through the gray level configuration information; the gray database comprises a database for gray testing obtained by cloning a production database; adjusting the service program into a gray level service program by utilizing the gray level configuration information; the gray scale service program is a program adapted to a gray scale verification environment; and processing the gray level service according to the gray level service program and the gray level service data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110279326.2A CN112860576A (en) | 2021-03-16 | 2021-03-16 | Business processing method, device and equipment based on gray level verification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110279326.2A CN112860576A (en) | 2021-03-16 | 2021-03-16 | Business processing method, device and equipment based on gray level verification |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112860576A true CN112860576A (en) | 2021-05-28 |
Family
ID=75994640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110279326.2A Pending CN112860576A (en) | 2021-03-16 | 2021-03-16 | Business processing method, device and equipment based on gray level verification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860576A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114238099A (en) * | 2021-12-09 | 2022-03-25 | 北京五八信息技术有限公司 | Method and device for gray scale test, electronic equipment and readable medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824745A (en) * | 2015-01-04 | 2016-08-03 | 中国移动通信集团湖南有限公司 | Grey level launch method and device |
| CN109828779A (en) * | 2018-12-26 | 2019-05-31 | 中国平安人寿保险股份有限公司 | Environment changing method, apparatus, terminal installation and storage medium |
| CN109857428A (en) * | 2018-12-29 | 2019-06-07 | 大唐软件技术股份有限公司 | Gray scale dissemination method for IOM system |
| CN111046036A (en) * | 2019-11-05 | 2020-04-21 | 深信服科技股份有限公司 | Data synchronization method, device, system and storage medium |
| CN111090452A (en) * | 2019-11-11 | 2020-05-01 | 福建天泉教育科技有限公司 | Service environment switching method and computer-readable storage medium |
| CN111324377A (en) * | 2020-03-11 | 2020-06-23 | 上海东普信息科技有限公司 | Application gray level release method, system, device and storage medium |
-
2021
- 2021-03-16 CN CN202110279326.2A patent/CN112860576A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824745A (en) * | 2015-01-04 | 2016-08-03 | 中国移动通信集团湖南有限公司 | Grey level launch method and device |
| CN109828779A (en) * | 2018-12-26 | 2019-05-31 | 中国平安人寿保险股份有限公司 | Environment changing method, apparatus, terminal installation and storage medium |
| CN109857428A (en) * | 2018-12-29 | 2019-06-07 | 大唐软件技术股份有限公司 | Gray scale dissemination method for IOM system |
| CN111046036A (en) * | 2019-11-05 | 2020-04-21 | 深信服科技股份有限公司 | Data synchronization method, device, system and storage medium |
| CN111090452A (en) * | 2019-11-11 | 2020-05-01 | 福建天泉教育科技有限公司 | Service environment switching method and computer-readable storage medium |
| CN111324377A (en) * | 2020-03-11 | 2020-06-23 | 上海东普信息科技有限公司 | Application gray level release method, system, device and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114238099A (en) * | 2021-12-09 | 2022-03-25 | 北京五八信息技术有限公司 | Method and device for gray scale test, electronic equipment and readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230126005A1 (en) | Consistent filtering of machine learning data | |
| US10719510B2 (en) | Tiering with pluggable storage system for parallel query engines | |
| US10740093B2 (en) | Advanced packaging techniques for improving work flows | |
| US10621211B2 (en) | Language tag management on international data storage | |
| RU2500023C2 (en) | Document synchronisation on protocol not using status information | |
| CN108804306B (en) | Method and system for automatic test system | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| CN108920698A (en) | A kind of method of data synchronization, device, system, medium and electronic equipment | |
| CN103617176A (en) | Method for achieving automatic synchronization of multi-source heterogeneous data resources | |
| CN111324610A (en) | Data synchronization method and device | |
| US11762654B2 (en) | Processing framework for in-system programming in a containerized environment | |
| CN109669925B (en) | Management method and device of unstructured data | |
| CN104765840A (en) | Big data distributed storage method and device | |
| CN108205560B (en) | Data synchronization method and device | |
| CN113297320A (en) | Distributed database system and data processing method | |
| Caneill et al. | The debsources dataset: two decades of free and open source software | |
| US20170169069A1 (en) | Data integrity checking in a distributed filesystem using object versioning | |
| US10831709B2 (en) | Pluggable storage system for parallel query engines across non-native file systems | |
| US11397750B1 (en) | Automated conflict resolution and synchronization of objects | |
| CN112308727A (en) | Insurance claim settlement service processing method and device | |
| CN112860576A (en) | Business processing method, device and equipment based on gray level verification | |
| CN113672556A (en) | Batch file migration method and device | |
| WO2022159391A1 (en) | Handling system-characteristics drift in machine learning applications | |
| CN113220706A (en) | Component product query method, device, equipment and medium | |
| CN113792026A (en) | Deployment method and device of database script and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210528 |