CN118132407A - Code quality determining method and device - Google Patents
Code quality determining method and device Download PDFInfo
- Publication number
- CN118132407A CN118132407A CN202211530268.7A CN202211530268A CN118132407A CN 118132407 A CN118132407 A CN 118132407A CN 202211530268 A CN202211530268 A CN 202211530268A CN 118132407 A CN118132407 A CN 118132407A
- Authority
- CN
- China
- Prior art keywords
- code
- standard
- reviewed
- data table
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012552 review Methods 0.000 claims abstract description 88
- 238000012545 processing Methods 0.000 claims abstract description 48
- 238000003860 storage Methods 0.000 claims description 27
- 230000006835 compression Effects 0.000 claims description 15
- 238000007906 compression Methods 0.000 claims description 15
- 238000012549 training Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 238000005192 partition Methods 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 13
- 238000011161 development Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004904 shortening Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000002860 competitive effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3608—Software analysis for verifying properties of programs using formal methods, e.g. model checking, abstract interpretation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method and a device for determining code quality, and relates to the technical field of big data. The specific implementation mode of the method comprises the following steps: acquiring a code to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables; inputting the code to be reviewed into a pre-trained code quality determination model; determining whether the format of the script header meets the requirements of the standard script header and whether the processing mode of each data table meets the requirements of the standard data table according to the output of the code quality determining model, and generating a target review result table; and determining the quality grade of the code to be reviewed by using the target review result table. According to the embodiment, automatic evaluation of the codes can be realized, so that the evaluation efficiency of the codes is improved, labor cost is greatly reduced, the accuracy of the code evaluation is improved, and code errors can be timely revised according to the evaluation result, so that the online period of a program is greatly shortened, the user requirements of frequent iteration are met, and the user experience is improved.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a method and an apparatus for determining code quality.
Background
The code review refers to judging whether the writing of the code accords with the writing standard or not or judging the quality of the code according to the error rate of the code by reading the code by current developers, reviewers in other links and the like.
In the existing code review process, a manual review mode is generally adopted, and the reviewers of a plurality of departments are integrated to judge the codes from the operation characteristics of each department so as to ensure that the program can be smoothly on line.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
In the existing code review process, because different developers have different writing habits (whether comments are added or not) and the review rules of the codes are countless, the review work of the database script is extremely heavy, the labor cost is high, and the normal online of the program is directly influenced due to extremely easy mistakes of manual review, so that the subsequent normal production cannot be ensured.
Disclosure of Invention
In view of this, the embodiment of the invention provides a method and a device for determining code quality, which can judge the data table format, script format, parameter format, task scheduling format and the like of a program code and determine whether the development of the program code meets the requirements, thereby greatly saving labor cost, improving review efficiency, finding code problems in time and maintaining, greatly improving the online speed of the program, greatly shortening the optimization period of user experience, and improving the competitiveness of products.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method for determining code quality, including:
Acquiring a code to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables;
inputting the code to be reviewed into a pre-trained code quality determination model;
Determining whether the format of the script header meets the requirements of a standard script header and whether the processing mode of each data table meets the requirements of a standard data table according to the output of the code quality determining model, and generating a target review result table;
and determining the quality grade of the code to be reviewed by using the target review result table.
Optionally, the standard data table comprises a standard table type, a standard table naming mode, a standard table storage mode and a standard table compression mode; the determining whether the processing mode of each data table meets the requirement of the standard data table comprises the following steps:
Determining whether the table type of each data table is any one of the following according to the standard table type: an internal table, an external table, a partition table, and a bucket table;
Extracting the data table names of the data tables, and determining whether the data table names accord with a standard table naming mode or not;
and judging whether the storage format and the compression format of each data table are a standard table storage mode and a standard table compression mode or not respectively.
Optionally, the determining whether the format of the script header meets the requirement of the standard script header includes:
And extracting a script head catalog of the script head, and determining whether the script head catalog comprises a program name, a project name, a business party, a product party, a project background, a project target, a project output, a source table, a temporary table, a target result table and a version definition.
Optionally, the code quality determination model includes a standard parameter usage mode, and further includes:
Judging whether the date parameter of the code to be reviewed uses sysdate functions or not according to the standard parameter using mode of the date parameter, if so, determining that the date parameter of the code to be reviewed does not meet the requirements;
And analyzing the time period of each small file merging operation of the code to be reviewed, determining whether the time period of the small file merging operation belongs to a preset time period, and if so, determining that the small file merging parameters accord with the standard parameter using mode.
Optionally, the code quality determination model includes a standard task scheduling manner, and further includes:
Comparing the task names of the scheduling tasks of the code to be reviewed with the data table names of the executed data tables of the corresponding scheduling tasks, and determining whether the naming mode of the scheduling tasks meets the requirements of a standard scheduling task mode;
And judging whether the dependency relationship of each scheduling task is correct or not according to the task content of each scheduling task of the code to be reviewed.
Optionally, the method further comprises:
Acquiring a sample code;
In a training stage, a code quality determining model respectively judges whether the processing modes of a sample script header and each sample data table of the sample code meet the requirements of the standard script header and the standard data table format according to a preset standard script header and standard data table format;
Generating a sample review result table of the sample script head and the processing mode of each sample data table;
And comparing the sample review result list with a sample code review result calibrated in advance, and performing iterative training to obtain the code quality determination model.
Optionally, the processing mode of the data table in the code to be reviewed is SQL language.
According to still another aspect of the embodiment of the present invention, there is provided a code quality determining apparatus including:
The acquisition module is used for acquiring codes to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables;
the input module is used for inputting the code to be reviewed into a pre-trained code quality determination model;
The output module is used for determining whether the format of the script header meets the requirements of the standard script header and whether the processing mode of each data table meets the requirements of the standard data table according to the output of the code quality determination model, and generating a target review result table;
And the quality determining module is used for determining the quality grade of the code to be reviewed by using the target review result table.
According to another aspect of an embodiment of the present invention, there is provided an electronic device for determining code quality, including:
One or more processors;
Storage means for storing one or more programs,
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the code quality determination method provided by the present invention.
According to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements the method of determining the quality of code provided by the present invention.
One embodiment of the above invention has the following advantages or benefits: the code of the program to be reviewed is reviewed by using a pre-trained code quality determination model, whether the code to be reviewed meets the requirements or not is determined according to a standard script head, a standard table type, a standard table naming mode, a standard table storage mode, a standard table compression mode, a standard parameter use mode, a standard scheduling task mode and the like specified by the code quality determination model, a target review result table comprising judgment results of all items is generated, so that the quality of the code to be reviewed is determined, the technical means of automatic review of the code is realized, the technical problems that in the existing code review process, the review work of a database script is extremely heavy, the labor cost is high, the error is extremely easy, the normal online of the program cannot be guaranteed, the subsequent normal production cannot be guaranteed are solved, and further the development of the program code can be judged according with the requirements or not, so that the labor cost is greatly saved, the review efficiency is improved, the code problem is timely found, the online speed of the program is greatly improved, the user experience is greatly shortened, and the competitive effect of the product is greatly improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
Fig. 1 is a schematic diagram of a main flow of a code quality determination method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main flow of a method of generating a code quality determination model according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of a method of generating a target review results table according to an embodiment of the invention;
fig. 4 is a schematic diagram of main modules of a code quality determination apparatus according to an embodiment of the present invention;
FIG. 5 shows an exemplary system architecture diagram of a code quality determination method or code quality determination apparatus suitable for application to embodiments of the present invention;
Fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of main flow of a code quality determining method according to an embodiment of the present invention, and as shown in fig. 1, the code quality determining method of the present invention includes the steps of:
The existing program codes usually need to be developed jointly by people of different departments (such as a data platform part, an operation data product part, an operation quality safety part and the like), and in order to ensure perfect online of the program, code testing of each link is required to be qualified, so that code review becomes an indispensable link for online of the program.
The existing manual review mode needs to be carried out by cooperation of multiple departments, on one hand, the code review period is greatly prolonged, and the review efficiency is low; on the other hand, due to different writing habits, selection of writing languages and the like (for example, zhang three is convenient to read, codes are written to annotate, python language is used for writing, lisi four is not used for writing any annotation, shell is used for writing), the development standards of codes are uneven, unified management and control are lacked, and the development of codes is bigger and is irregularly found for a long time; moreover, the development of codes has many same or similar functions, and repeated development of different personnel also causes waste of development cost, department operation cost and the like. In addition, with the rapid increase of the user demands, the update frequency and the volume of the program codes also increase linearly, and the existing single-thread item-by-item auditing mode not only wastes time, but also is low in efficiency, the error rate is also increased, the script quality is low, and hidden danger is brought to the running of the program.
By the method for determining the code quality, automatic review is carried out on the program codes, particularly the database SQL codes corresponding to the program, so that the code review efficiency is improved, the repeated work of the reviewer can be greatly reduced, the developer can also pre-judge in advance by adopting the method, timely adjust errors, improve the code accuracy, avoid the time consumed by subsequent correction, optimize the online flow of the whole program and improve the work enthusiasm of the reviewer.
Step S101, acquiring a code to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables.
In the embodiment of the present invention, the code to be reviewed corresponds to the program to be reviewed, and any type of programming language may be used, such as perl language, shell language, python language, SQL language, etc.
Further, the code to be reviewed is written by using the python language, the database of the program to be reviewed is written by using the SQL language and is embedded into a program framework written by using the python language, and the database is used as the database of the program to be reviewed and used for inquiring, storing and other operations, and comprises a plurality of data tables.
In the embodiment of the invention, the code to be reviewed comprises a script header and a processing mode of a plurality of data tables, wherein the script header is written in the python language, and the processing mode of the data tables is SQL language.
In the embodiment of the present invention, the program to be reviewed may be a program for any link such as WMS system (Warehouse MANAGEMENT SYSTEM ), express system, TMS system (Transportation management system), cold chain system, spare part library, etc.
And step S102, inputting the code to be reviewed into a pre-trained code quality determination model.
In the embodiment of the invention, the code quality determination model is used for evaluating the code to be evaluated, so that the quality grade of the code to be evaluated is determined, namely whether the code to be evaluated is qualified, whether errors exist or not is determined, so that the program to be evaluated is normally run on line under the condition that the code to be evaluated is qualified, and the code errors are pointed out under the condition that the code to be evaluated has the errors, so that a developer corrects the code errors.
In the embodiment of the present invention, as shown in fig. 2, the method for generating the code quality determination model of the present invention includes the following steps:
Step S201, a sample code is acquired.
In the embodiment of the invention, the sample code comprises a sample script header and a processing mode of a plurality of sample data tables.
Step S202, in the training stage, the code quality determining model respectively judges whether the processing modes of the sample script header and each sample data table of the sample code meet the requirements of the standard script header and the standard data table format according to the preset standard script header and standard data table format.
In the embodiment of the invention, the code quality determination model configures a standard script header and a standard data table format in advance so as to evaluate and judge whether the sample code meets the requirement. The standard script header, standard data table format may be txt file or excel table, etc.
In embodiments of the present invention, either a single program or a gadget may be developed to pre-configure standard script header and standard data table formats.
In step S2021, according to a preset standard script header, the code quality determining model determines whether the sample script header of the sample code meets the requirement of the standard script header.
In the embodiment of the invention, the standard script header comprises a program name (such as app_exp_route_sum_d.py), a project name (i.e. a project name such as express experience class T+1 project), a service party (such as service personnel identification), a product party (such as product responsible person identification), a project background (such as development express experience class T+1 project for realizing online up of the service data T+1 in order to facilitate management of the service data by managers of various levels of services), a project target (such as finishing processing of the service data such as product type, claim amount segment, region, site, liability strip line, liability cause and the like, in order to meet the service query requirement), project output (i.e. the final presentation form of the project), code description (e.g. data table processing of T+1 express delivery foreign list loss rate/breakage rate), parameters (some script call fixed parameters, mode description of parameter call can be written out, such as disabling function, whether to add small file merging parameters, etc.), rules (e.g. generating T+1 data daily), period (e.g. frequency of day, week, month, season, year, etc.), source table (refer to the original data table processed by the code to be reviewed, for example, the source table comprises fdm_ judgement _biz_ judgement _chain, which represents the main table of responsibility judgment; dim_ kdt _duty_ dept _cd, which represents a service provision manual table, app_app_ea_del_ waybill _detail_d, which represents a commodity stream split-medium and medium piece distribution detail daily table, fdm_bd_ waybill _v_ waybill _c_chain, which represents a medium and medium piece manifest extension table), and, temporary tables (for example, temporary tables including tmp.tmp_t1exp_route_ kdfz _ord_det representing molecular detail temporary tables; tmp.tmp_t1exp_route_ kdfz _ord_sum representing denominator summary temporary tables), target tables (i.e., final result tables, for example, target tables of app_exp_route_sum_d representing results tables after the loss/breakage rate of the t+1 express delivery external form has undergone responsibility judgment), versions (for example, versions including v 1.0-20200218-developer identification-first draft; v 1.1-20200318-developer identification-modification content-modification reason, etc.) are generated in the intermediate processing.
In the embodiment of the invention, the code quality determination model analyzes the sample codes in the training stage. And extracting a sample script header part corresponding to the standard script header in the sample code according to a code template defined in advance, and judging whether the sample script header meets the requirements. The code template comprises a script header part, an SQL definition part and other parts, and the sequence of each part can be freely adjusted.
Further, the code quality determination model may parse the sample code using a scripting language (e.g., perl language, shell language, python language, JAVA language, etc.).
In the embodiment of the invention, the code quality determination model judges whether a sample script header of a sample code comprises various definitions such as a program name, a project name, a service party, a product party, a project background, a project target, a project output, a code description, parameters, rules, a period, a source table, a temporary table, a target result table, a version and the like, and if the sample script header comprises all the above items, the sample script header of the sample code is determined to meet the requirements; if the sample script header is missing any one of the above items, determining that the sample script header of the sample code is not satisfactory.
Further, in the training stage, the code quality determination model can extract missing items of the sample script header for display, so that a developer revises the sample script header according to the missing items of the sample script header, and the sample script header meets the requirements of the standard script header.
In step S2022, the code quality determining model determines whether the processing mode of each sample data table meets the requirement of the standard data table format according to the preset standard data table format.
In the embodiment of the invention, the code quality determination model analyzes the sample codes in the training stage. According to a code template defined in advance, a sample SQL definition part corresponding to a standard data table format in a sample code is extracted, the processing modes of all sample data tables are matched, and whether the processing modes of all sample data tables meet the requirements is judged, namely, the SQL definition part comprises the processing modes of all sample data tables (a source table, a temporary table, a target table and the like).
In the embodiment of the invention, the standard data table format comprises a standard table type, a standard table naming mode, a standard table storage mode and a standard table compression mode, and specifically:
In the training stage, the code quality determination model determines whether the table type of each sample data table meets the requirements according to the standard table type, for example, whether the type of each data table is any of the following types: an internal table, an external table, a partition table, and a bucket table.
The code quality determining model determines whether the table naming of each sample data table meets the requirement according to the standard table naming mode, for example, whether the naming mode of the sample detail table is app_det_d/m/q/y and the naming mode of the sample summary table is app_sum_d/m/q/y; wherein app corresponds to the program name to be reviewed, det is detail, det represents a list, sum represents a summary list, d, m, q, y represents day, month, season, year.
The code quality determining model determines whether the table storage of each sample data table meets the requirements according to the standard table storage mode, for example, whether the storage mode of each data table is in any of the following formats: textfile, sequence, ORC, parquet.
The code quality determining model determines whether the table compression of each sample data table meets the requirements according to the standard table compression mode, for example, whether the compression mode of each sample data table is in any of the following formats: bzip2, gzip, deflate, snappy, lzo, etc.
And step S203, generating a sample review result table of the processing mode of the sample script header and each sample data table.
In the embodiment of the present invention, according to the determination result in step S202, a sample review result table is generated, where the sample review result table includes a sample script header and a determination result of a processing manner of each data table, and may be embodied in forms of scoring items, opposite hooks "v", and cross "x", etc.
And step S204, comparing the sample review result list with a sample code review result calibrated in advance, and performing iterative training.
In the embodiment of the invention, a sample review result table is compared with a sample code review result calibrated in advance, a code quality determination model is adjusted, and iterative training is carried out on the code quality determination model.
Step S205, obtaining the code quality determination model.
In the embodiment of the invention, a final code quality determination model is determined according to training results of multiple iterations.
In the embodiment of the invention, the training process of the code quality determination model further comprises the following steps:
In the embodiment of the invention, the code quality determination model analyzes the sample codes in the training stage. According to a pre-defined code template, extracting other parts of samples in the sample code, matching each sample parameter and each sample scheduling task, and judging whether the execution process of each sample parameter and each sample scheduling task meets the requirements, namely, the other parts comprise the execution process of each sample parameter and each sample scheduling task.
The code quality determining model judges whether the use of the sample parameters meets the requirements according to the standard parameter using mode, for example, the sample parameters are date parameters, and the standard parameter using mode prescribes a function for prohibiting the use of the date parameters sysdate, so that whether each date parameter of the sample code is defined by a sysdate function is judged to judge whether the use of the sample parameters meets the requirements; for another example, the sample parameters are small file merging parameters, the small files are used for storing data in the cluster server, and excessive small files influence the efficiency of data query and use, so that small file merging operation needs to be added, and the standard parameter use mode prescribes a small file merging conversion time period, so that whether each small file merging operation of the sample codes is in a preset conversion time period is judged, and whether the use of the sample parameters meets the requirements is judged;
The code quality determining model judges whether the execution process of the sample scheduling task meets the requirement according to a standard scheduling task mode, for example, whether the naming of the scheduling task corresponds to an executed data table, the name of the sample scheduling task is exe_app_pf_account_info_det_d on the assumption that the table name is app_pf_account_info_det_d, and the sample scheduling task corresponds to an upper table, and the corresponding sample scheduling task can be found through one upper table; for example, if the dependency relationship of the scheduling task is correct, it is determined whether the data table B exists in the dependency relationship of the data table a in the sample scheduling task, assuming that the processing data table a depends on the data table B.
Further, the sample review result table also includes judgment results of use of sample parameters, execution process of sample scheduling task, and the like.
According to the embodiment of the invention, the automatic review of the code can be realized by the method for determining the code quality determination model, so that the review efficiency of the code is improved, the labor cost is greatly reduced, the accuracy of the code review is improved, and code errors can be timely revised according to the review result, so that the online period of a program is greatly shortened, the user requirement of frequent iteration is met, and the user experience is improved.
And step S103, determining whether the format of the script header meets the requirements of the standard script header and whether the processing mode of each data table meets the requirements of the standard data table according to the output of the code quality determination model, and generating a target review result table.
In the embodiment of the invention, whether the format of the script header of the code to be reviewed meets the requirement of the standard script header and whether the processing mode of each data table of the code to be reviewed meets the requirement of the standard data table are determined according to the output of the code quality determination model, and a target review result table is generated.
In the embodiment of the present invention, as shown in fig. 3, the method for generating the target review result table of the present invention includes the following steps:
In step S301, the code quality determination model analyzes whether the script header of the code to be reviewed meets the requirements of the standard script header.
In the embodiment of the invention, the code quality determination model can extract a script header catalog of the script header to determine whether the script header catalog comprises a program name, a project name, a business party, a product party, a project background, a project target, a project output, a source table, a temporary table, a target result table and a version definition.
In step S302, the code quality determination model analyzes whether the table types of the data tables of the code to be reviewed meet the requirements of standard table types.
In the embodiment of the invention, the code quality determination model can determine whether the table type of each data table is any one of the following according to the standard table type: an internal table, an external table, a partition table, and a bucket table.
In step S303, the code quality determination model analyzes whether the data table names of the data tables of the code to be reviewed meet the requirements of the standard table naming method.
In the embodiment of the invention, the code quality determination model can extract the data table names of all the data tables and determine whether the data table names accord with the standard table naming mode.
And S304, analyzing whether the compression format of each data table of the code to be reviewed meets the requirement of a standard table compression mode by the code quality determination model.
In the embodiment of the invention, the code quality determination model can determine whether the compression format of each data table is any one of the following: bzip2, gzip, deflate, snappy, lzo.
In step S305, the code quality determination model analyzes whether the storage format of each data table of the code to be reviewed meets the requirements of the standard table storage mode.
In the embodiment of the invention, the code quality determination model can determine whether the storage format of each data table is any one of the following: textfile, sequence, ORC, parquet.
Step S306, the code quality determining model analyzes whether the date parameter of the code to be reviewed meets the requirement of the standard parameter using mode.
In the embodiment of the invention, the code quality determination model can judge whether the date parameter of the code to be reviewed uses sysdate functions according to the standard parameter use mode of the date parameter, and if so, the date parameter of the code to be reviewed is determined to be not in accordance with the requirement of the standard parameter use mode; if not, determining that the date parameter of the code to be reviewed meets the requirement of the standard parameter using mode.
Step S307, the code quality determination model analyzes whether the doclet merging operation of the code to be reviewed meets the requirement of the standard parameter using mode.
In the embodiment of the invention, the code quality determination model can analyze the time period of each small file merging operation of the code to be reviewed to determine whether the time period of the small file merging operation belongs to a preset time period or not, and if so, determine that the small file merging parameters accord with the standard parameter using mode; if not, determining that the small file merging parameters do not accord with the standard parameter using mode.
In step S308, the code quality determination model analyzes whether the task names of the scheduled tasks of the code to be reviewed meet the requirements of the standard scheduled task mode.
In the embodiment of the invention, the code quality determination model can compare the task names of the scheduling tasks of the code to be reviewed with the data table names of the executed data tables of the corresponding scheduling tasks to determine whether the naming mode of the scheduling tasks meets the requirements of the standard scheduling task mode.
Step S309, the code quality determination model analyzes whether the dependency relationship of each scheduling task of the code to be reviewed meets the requirements of the standard scheduling task mode.
In the embodiment of the invention, the code quality determination model can judge whether the dependency relationship of each scheduling task is correct according to the task content of each scheduling task of the code to be reviewed, for example, the task content of the scheduling task is the processing data table A, and the execution of the data table A depends on the result of the data table B, so if the dependency relationship of the scheduling task indicates that the execution of the scheduling task depends on the data table B, the dependency relationship of the scheduling task is correct.
Step S310, determining a target review result table according to the analysis results of the steps S301-S309.
In the embodiment of the invention, the target review result table comprises the review results of the script header of the code to be reviewed, the table type of each data table, the data table name of each data table, the compression format of each data table, the storage format of each data table, the date parameter, the small file merging operation, the task name of each scheduling task, the dependency relationship of each scheduling task and the like.
According to the method for generating the target review result table, disclosed by the embodiment of the invention, the code to be reviewed can be analyzed, and the review result of each sub item of the code to be reviewed can be determined, so that the target review result table is generated, the reference in the subsequent code quality determination is facilitated, the code review efficiency is greatly improved, the labor cost is reduced, and the review accuracy is improved.
And step S104, determining the quality grade of the code to be reviewed by using the target review result table.
In the embodiment of the invention, the quality grades comprise qualification, revision and disqualification, and the quality grades of the codes to be reviewed are distinguished according to the review results of all the items in the target review result table. The quality grade is qualified, which means that the code is qualified, and the test can be directly carried out or the code can be directly put on line; the quality grade is that the part of codes to be revised have problems, and a developer is required to revise the part of codes; a quality grade of disqualification indicates that the code is disqualified, and the code has excessive problems and needs to be redeveloped.
In the embodiment of the invention, the target review result table can be displayed in a visual mode, operations such as downloading, sending, sharing and the like of the target review result table can be provided, humanization and convenience of code review are improved, a developer can quickly position a revision according to the displayed target review result table, code errors can be corrected, the review period is greatly shortened, and the online efficiency of a program is improved.
In the embodiment of the invention, the code to be reviewed is obtained; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables; inputting the code to be reviewed into a pre-trained code quality determination model; determining whether the format of the script header meets the requirements of a standard script header and whether the processing mode of each data table meets the requirements of a standard data table according to the output of the code quality determining model, and generating a target review result table; and determining the quality grade of the code to be reviewed by utilizing the target review result table, judging the data table format, the script format, the parameter format, the task scheduling format and the like of the program code, and determining whether the development of the program code meets the requirements or not, thereby greatly saving labor cost, improving review efficiency, finding code problems in time and maintaining, greatly improving the online speed of the program, greatly shortening the optimization period of user experience, and improving the competitiveness of the product.
Fig. 4 is a schematic diagram of main modules of a code quality determining apparatus according to an embodiment of the present invention, and as shown in fig. 4, a code quality determining apparatus 400 of the present invention includes:
An acquisition module 401, configured to acquire a code to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables.
In this embodiment of the present invention, the obtaining module 401 is configured to obtain a code to be reviewed, where the code to be reviewed corresponds to a program to be reviewed, and includes a script header, a processing manner of a plurality of data tables, a parameter usage manner, an execution process of a scheduling task, and the like.
An input module 402, configured to input the code to be reviewed into a pre-trained code quality determination model.
In the embodiment of the invention, the code quality determination model is used for evaluating the code to be evaluated, so that the quality grade of the code to be evaluated is determined, namely whether the code to be evaluated is qualified, whether errors exist or not is determined, so that the program to be evaluated is normally run on line under the condition that the code to be evaluated is qualified, and the code errors are pointed out under the condition that the code to be evaluated has the errors, so that a developer corrects the code errors. Wherein the input module 402 inputs the code to be reviewed into a pre-trained code quality determination model.
And an output module 403, configured to determine, according to the output of the code quality determination model, whether the format of the script header meets the requirement of the standard script header, and whether the processing manner of each data table meets the requirement of the standard data table, and generate a target review result table.
In this embodiment of the present invention, the output module 403 determines, according to the output of the code quality determination model, whether the format of the script header of the code to be reviewed meets the requirement of the standard script header, whether the processing mode of each data table meets the requirement of the standard data table, whether the usage mode of each parameter meets the requirement of the standard parameter usage mode, and whether the execution process of each scheduling task meets the requirement of the standard scheduling task mode, so as to generate the target review result table.
And the quality determining module 404 is used for determining the quality grade of the code to be reviewed by using the target review result table.
In the embodiment of the present invention, the quality determining module 404 distinguishes the quality level of the code to be reviewed according to the review result of each item in the target review result table.
In the embodiment of the invention, the data table format, the script format, the parameter format, the task scheduling format and the like of the program codes can be judged through the acquisition module, the input module, the output module, the quality determination module and other modules, and whether the development of the program codes meets the requirements or not is determined, so that the labor cost is greatly saved, the evaluation efficiency is improved, the code problems are timely found and maintained, the online speed of the program is greatly improved, the user experience optimization period is greatly shortened, and the product competitiveness is improved.
Fig. 5 shows an exemplary system architecture diagram of a code quality determination method or a code quality determination apparatus suitable for application to an embodiment of the present invention, as shown in fig. 5, the exemplary system architecture of the code quality determination method or the code quality determination apparatus of the embodiment of the present invention includes:
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 105. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 505 via the network 504 using the terminal devices 501, 502, 503 to receive or send messages or the like. Various communication client applications, such as a code review class application, a shopping class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, and the like, may be installed on the terminal devices 501, 502, 503.
The terminal devices 501, 502, 503 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server providing support for code review-type websites browsed by the user using the terminal devices 501, 502, 503. The background management server may perform analysis and the like on the received data of the code to be reviewed and the like, and feed back the processing result (for example, the quality level of the code to be reviewed) to the terminal devices 501, 502, 503.
It should be noted that, the method for determining the code quality provided in the embodiment of the present invention is generally performed by the server 505, and accordingly, the device for determining the code quality is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 6 is a schematic structural diagram of a computer system suitable for a terminal device or a server for implementing an embodiment of the present invention, and as shown in fig. 6, a computer system 600 of a terminal device or a server of an embodiment of the present invention includes:
A Central Processing Unit (CPU) 601, which can execute various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the system 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes an acquisition module, an input module, an output module, and a quality determination module. The names of these modules do not limit the module itself in some cases, for example, the output module may also be described as a "module for generating a target review result table according to the output of the code quality determination model, determining whether the format of the script header meets the requirements of the standard script header, and whether the format of each data table meets the requirements of the standard data table".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: acquiring a code to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables; inputting the code to be reviewed into a pre-trained code quality determination model; determining whether the format of the script header meets the requirements of a standard script header and whether the processing mode of each data table meets the requirements of a standard data table according to the output of the code quality determining model, and generating a target review result table; and determining the quality grade of the code to be reviewed by using the target review result table.
In the existing manual review mode, a plurality of departments such as a constant review agent (including an architect, a bazaar manager, a basic model manager and the like), a data platform part, an operation data product part, an operation quality safety part and the like are required to cooperate in multiple ways, and as the frequency of submitting SQL script codes is greatly increased, the review work of the review agent corresponding to offline big data is also linearly increased, a great amount of time is required, the efficiency is low, the frequency of errors such as in the twinkling of an eye and omission of manual review is high, the code quality is reduced, and the risk of loopholes is extremely high.
According to the technical scheme provided by the embodiment of the invention, the evaluation efficiency can be greatly improved, the automatic evaluation is realized, and the repeated work of the evaluation personnel is reduced; the developer of the code to be reviewed can also perform script self-checking and pre-judging in advance, so that time waste such as later revising, adjusting and the like is avoided; the method of the invention can be embedded into the existing big data platform, and SQL code verification is carried out in an online link, thus improving the platform function and having wide application scenarios.
According to the technical scheme provided by the embodiment of the invention, the accuracy and consistency of the code review are greatly improved, the automatic output of the review result report can be realized without the cooperation of multiple persons, the review efficiency is improved, and the accuracy of the code review is improved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for determining code quality, comprising:
Acquiring a code to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables;
inputting the code to be reviewed into a pre-trained code quality determination model;
Determining whether the format of the script header meets the requirements of a standard script header and whether the processing mode of each data table meets the requirements of a standard data table according to the output of the code quality determining model, and generating a target review result table;
and determining the quality grade of the code to be reviewed by using the target review result table.
2. The method of claim 1, wherein the standard data table comprises a standard table type, a standard table naming method, a standard table storage method, a standard table compression method; the determining whether the processing mode of each data table meets the requirement of the standard data table comprises the following steps:
Determining whether the table type of each data table is any one of the following according to the standard table type: an internal table, an external table, a partition table, and a bucket table;
Extracting the data table names of the data tables, and determining whether the data table names accord with a standard table naming mode or not;
and judging whether the storage format and the compression format of each data table are a standard table storage mode and a standard table compression mode or not respectively.
3. The method of claim 2, wherein determining whether the format of the script header meets the requirements of a standard script header comprises:
And extracting a script head catalog of the script head, and determining whether the script head catalog comprises a program name, a project name, a business party, a product party, a project background, a project target, a project output, a source table, a temporary table, a target result table and a version definition.
4. The method of claim 1, wherein the code quality determination model includes a standard parameter usage pattern, further comprising:
Judging whether the date parameter of the code to be reviewed uses sysdate functions or not according to the standard parameter using mode of the date parameter, if so, determining that the date parameter of the code to be reviewed does not meet the requirements;
And analyzing the time period of each small file merging operation of the code to be reviewed, determining whether the time period of the small file merging operation belongs to a preset time period, and if so, determining that the small file merging parameters accord with the standard parameter using mode.
5. The method of claim 1, wherein the code quality determination model comprises a standard scheduling task manner, further comprising:
Comparing the task names of the scheduling tasks of the code to be reviewed with the data table names of the executed data tables of the corresponding scheduling tasks, and determining whether the naming mode of the scheduling tasks meets the requirements of a standard scheduling task mode;
And judging whether the dependency relationship of each scheduling task is correct or not according to the task content of each scheduling task of the code to be reviewed.
6. The method as recited in claim 1, further comprising:
Acquiring a sample code;
In a training stage, a code quality determining model respectively judges whether the processing modes of a sample script header and each sample data table of the sample code meet the requirements of the standard script header and the standard data table format according to a preset standard script header and standard data table format;
Generating a sample review result table of the sample script head and the processing mode of each sample data table;
And comparing the sample review result list with a sample code review result calibrated in advance, and performing iterative training to obtain the code quality determination model.
7. The method of claim 1, wherein the processing mode of the data table in the code to be reviewed is SQL language.
8. A code quality determining apparatus, comprising:
The acquisition module is used for acquiring codes to be reviewed; the code to be reviewed comprises a script head and a processing mode of a plurality of data tables;
the input module is used for inputting the code to be reviewed into a pre-trained code quality determination model;
The output module is used for determining whether the format of the script header meets the requirements of the standard script header and whether the processing mode of each data table meets the requirements of the standard data table according to the output of the code quality determination model, and generating a target review result table;
And the quality determining module is used for determining the quality grade of the code to be reviewed by using the target review result table.
9. An electronic device for determining code quality, comprising:
One or more processors;
Storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211530268.7A CN118132407A (en) | 2022-12-01 | 2022-12-01 | Code quality determining method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211530268.7A CN118132407A (en) | 2022-12-01 | 2022-12-01 | Code quality determining method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118132407A true CN118132407A (en) | 2024-06-04 |
Family
ID=91239464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211530268.7A Pending CN118132407A (en) | 2022-12-01 | 2022-12-01 | Code quality determining method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118132407A (en) |
-
2022
- 2022-12-01 CN CN202211530268.7A patent/CN118132407A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111061833B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US11367008B2 (en) | Artificial intelligence techniques for improving efficiency | |
| US9411575B2 (en) | Systems and methods for quality assurance automation | |
| CN110837356B (en) | Data processing method and device | |
| US10560484B2 (en) | Managing access in one or more computing systems | |
| US11237824B2 (en) | Tracking related changes with code annotations | |
| CN109840192B (en) | Method and device for automatic testing | |
| WO2020143313A1 (en) | Code modification method and apparatus, computer-readable medium and electronic device | |
| CN111367530B (en) | Code auditing method, terminal and server based on custom auditing rules | |
| CN110674117A (en) | Data modeling method and device, computer readable medium and electronic equipment | |
| CN101582138A (en) | Dynamic business processing system and method | |
| CN112947919A (en) | Method and device for constructing service model and processing service request | |
| CN112988576B (en) | Method and device for determining development process of software project | |
| CN110895761B (en) | After-sales service application information processing method and device | |
| US20200167263A1 (en) | Automated source code writing style evaluation, validation, and modification | |
| US20210089992A1 (en) | Method for automated code reviewer recommendation | |
| CN118132407A (en) | Code quality determining method and device | |
| CN114625640B (en) | Software test project management method and device, storage medium and electronic equipment | |
| CN107229487B (en) | Code processing method and device | |
| CN114693116A (en) | Method and device for detecting code review validity and electronic equipment | |
| US20140279132A1 (en) | Buyer assignment for requisitions lines | |
| CN113626379A (en) | Research and development data management method, device, equipment and medium | |
| US20120216105A1 (en) | System and method for creating non-functional requirements for implemented technology | |
| CN114331116A (en) | Data assessment method, equipment, medium and product for business system | |
| CN113487224A (en) | Content processing method, apparatus, device, medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |