CN110968511A - Recommendation engine testing method, device, computing equipment and system - Google Patents

Abstract

The invention discloses a method for testing a recommendation engine, which is suitable for being executed in computing equipment, wherein the computing equipment is respectively in communication connection with the recommendation engine and a service platform, and the method comprises the following steps: acquiring log information recorded on a service platform, and dividing the log information into plan data, scheduling data, unit data and strategy data, wherein each kind of data has a plan identifier to which the data belongs; selecting scheduling data, unit data and strategy data with the same plan identification to assemble to generate a test case, wherein the test case comprises engine data constructed based on the selected data, a test request for testing the engine data and an expected return result of the test request; and sending the engine data to a recommendation engine, sending the test request to the recommendation engine to obtain an actual return result, and verifying the actual return result according to an expected return result. The invention also discloses a corresponding testing device, a corresponding computing device and a corresponding system of the recommendation engine.

Description

Recommendation engine testing method, device, computing equipment and system

Technical Field

The invention relates to the field of automatic testing, in particular to a method, a device, computing equipment and a system for testing a recommendation engine.

Background

Automated testing is one of the commonly used testing means in the iterative regression process of a project. Compared with other project tests, the test difficulty and complexity of the advertisement engine are far higher than those of other tests, mainly because the test data structure is large and not easy to maintain, and the data structure body needs to be prepared again in each test, which is time-consuming and labor-consuming.

The existing automatic engine regression scheme stores data into EXCEL to be read and processed, and also stores data into a database to be processed. But data production is still through manual creation and manual combination, and finally is entered into EXCEL or a database through a page. The method has obvious effect when the use cases are few and the logic of the engine service is simple. However, when the tester constructs the data, the tester may have hand error to cause the data to be incorrect, so that the subsequent test flow is blocked. Moreover, when the data structure of the method is complex, it takes time to generate a complete test data, and the difficulty of other personnel receiving the test regression service is increased. Therefore, it is desirable to provide a more automated and simple engine regression testing method.

Disclosure of Invention

In view of the above, the present invention provides a method, apparatus, computing device and system for testing a recommendation engine in an effort to solve, or at least solve, the above-identified problems.

According to an aspect of the present invention, there is provided a method for testing a recommendation engine, adapted to be executed in a computing device, the computing device being communicatively connected to the recommendation engine and a service platform, respectively, the method including the steps of: acquiring log information recorded on a service platform, and dividing the log information into plan data, scheduling data, unit data and strategy data, wherein the data respectively correspond to release time, release positions, oriented groups and release styles of recommended contents, and each data has a plan identifier to which the data belongs; selecting scheduling data, unit data and strategy data with the same plan identification for assembly to generate a test case, wherein the test case comprises engine data constructed based on the selected data, a test request for testing the engine data and an expected return result of the test request; and sending the engine data to a recommendation engine, sending the test request to the recommendation engine to obtain an actual return result, and verifying the actual return result according to an expected return result.

Optionally, in the testing method according to the present invention, the plan data has a plan identification and plan information; the scheduling data comprises plan identification, scheduling identification, position identification and scheduling information; the unit data comprises a plan identifier, a scheduling identifier, a unit identifier and unit information; the policy information includes a plan identifier, a schedule identifier, a unit identifier, a policy identifier, and policy information.

Optionally, in the testing method according to the present invention, further comprising the steps of: and respectively storing the divided plan data, scheduling data, unit data and strategy data into a plan library, a scheduling library, a unit library and a strategy library.

Optionally, in the testing method according to the present invention, a testing platform is disposed in the computing device, and option boxes of the plan identifier, the schedule identifier, the unit identifier, and the policy identifier are disposed in the testing platform, respectively.

Optionally, in the testing method according to the present invention, the step of selecting scheduling data, unit data and strategy data having the same plan identifier for assembly includes: based on the plan identifier selected by the tester in the plan option box, automatically generating a schedule identifier, a unit identifier and a strategy identifier which are selectable under the plan identifier in other option boxes; and acquiring corresponding plan information, scheduling information, unit information and strategy information from the database for assembly based on the options of the tester in each option box.

Optionally, in the testing method according to the present invention, the test case further includes media information of the content delivery platform, where the media information includes at least one of a request path, a request header, and a request parameter.

Optionally, in the testing method according to the present invention, the engine data includes at least one engine sub-data, and each engine sub-data includes a set of schedule data, unit data, and policy data associated with each other.

Optionally, in the test method according to the present invention, the request structure is included in the test request, and the request structure includes a user tag, where the user tag belongs to a targeted group of unit data in the engine data.

Optionally, in the test method according to the present invention, the test case includes at least one test step, each test step has a corresponding check point, and each check point has a corresponding expected return result.

Optionally, in the test method according to the invention, the expected returned results comprise at least one of the following types: expecting an accurate check value to return, expecting an accurate check value not to appear, the expected check point fuzzy matches any value, the expected check point is in the range list, expecting an accurate probability value to appear, expecting an accurate frequency check number to appear.

Optionally, in the testing method according to the present invention, further comprising the steps of: assembling a plurality of test cases belonging to the same plan identification, and executing all the test cases under the plan identification in batch; and calculating and displaying the test cases which are successfully executed and the test cases which are failed to be executed in the case list corresponding to the plan identification.

Optionally, in the testing method according to the present invention, the log information recorded by the service platform is sent to a kafka message queue, and the step of acquiring the log information recorded by the service platform includes: the consumer creating the kafka in the computing device retrieves the log information from the kafka message queue.

Optionally, in the testing method according to the present invention, the service platform records log information by a logstack method, and the database is a Mysql library.

According to another aspect of the present invention, there is provided a recommendation engine testing apparatus adapted to be resident in a computing device for execution, the computing device being communicatively connected to a recommendation engine and a service platform, respectively, the apparatus comprising: the data acquisition module is suitable for acquiring log information recorded on a service platform and dividing the log information into plan data, scheduling data, unit data and strategy data, wherein the data respectively correspond to the release time, release position, oriented group and release style of recommended content, and each data has a plan identifier; the test case generation module is suitable for selecting scheduling data, unit data and strategy data with the same plan identification to be assembled to generate a test case, and the test case comprises engine data constructed based on the selected data, a test request for testing the engine data and an expected return result of the test request; and the case execution module is suitable for sending the engine data to the recommendation engine, sending a test request to the recommendation engine to obtain an actual return result, and further verifying the actual return result according to an expected return result.

According to yet another aspect of the present invention, there is provided a computing device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs when executed by the processors implement the steps of the method of testing of the recommendation engine as described above.

According to a further aspect of the invention there is provided a readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, implement the steps of the method of testing a recommendation engine as described above.

According to the technical scheme of the invention, the original data of the service platform is recorded into the test platform through a program in a data driving mode, so that a tester does not need to care about the internal format of the data any more, and can generate a complete data structure body of the request engine only by combining the data according to the keywords. And simultaneously, directly sending a request to the engine through a program simulation service end, and verifying the engine result according to preset verification logic. Moreover, the invention can also add in steps according to the actual scene of the use case, which is equivalent to that one use case contains a plurality of use cases, and a plurality of test points can be tested in one use case. Furthermore, the invention can also increase the dimensionality of the test case plan, different test plans can be combined in each iteration, and regression verification is carried out on the engine according to the change of the iteration update.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.

FIG. 1 shows a block diagram of a recommendation engine testing system 100 according to one embodiment of the present invention;

FIG. 2 shows a block diagram of a computing device 200, according to one embodiment of the invention;

FIG. 3 illustrates a flow diagram of a method 300 of testing a recommendation engine in accordance with one embodiment of the present invention;

FIGS. 4a-4d are data structure diagrams illustrating planning data, scheduling data, unit data, and strategy data, respectively, according to an embodiment of the present invention;

FIGS. 5a-5c are diagrams illustrating, respectively, assembling test cases, creating media information, and checkpoints, according to one embodiment of the invention;

FIG. 6 illustrates a schematic diagram of a test request according to one embodiment of the invention;

FIG. 7 is a diagram illustrating actual returned results of a test request, according to one embodiment of the invention;

8a-8c show schematic diagrams of planned use case assembly, planned execution results, and execution result details, respectively, according to one embodiment of the invention; and

FIG. 9 shows a block diagram of a recommendation engine testing device 900 according to one embodiment of the present invention;

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

FIG. 1 shows a schematic diagram of a recommendation engine testing system 100, according to one embodiment of the present invention. As shown in fig. 1, the recommendation engine testing system 100 includes a service platform 110, a message queue 120, a computing device 130, and a recommendation engine 140. Wherein the service platform 110 is communicatively coupled to the message queue 120, and the computing device 130 is communicatively coupled to the message queue 120 and the recommendation engine 140, respectively. In addition, the system 100 may further include a database (not shown), which may reside in the computing device 140 as a local database, or may be disposed outside the computing device 140 as a remote database, and the present invention does not limit the deployment manner of the database.

It should be noted that the recommendation engine testing system 100 of fig. 1 is merely exemplary, and in a specific practical situation, there may be different numbers of service platforms 110, message queues 120, computing devices 130 and recommendation engines 140 in the system 100, and the present invention does not limit the number of devices included in the system 100.

Generally, the service personnel configures recommendation conditions of recommended content (such as advertisements) on the service platform 110, such as a planning layer, a scheduling layer, a unit layer and a strategy layer for setting the recommended content. The planning layer represents the delivery time, delivery period, etc. of the recommended content, and the scheduling layer represents the delivery position (e.g., advertisement position) of the recommended content. The unit layer represents the targeted group of the recommended content, such as targeted delivery to a certain person, a person in a certain region, a person of a certain gender, a person of a certain age group, and the like. The strategy layer represents the delivery format of the recommended content, such as delivering advertisements of a text chain, delivering advertisements of a text in a picture or delivering advertisements of a text in a picture, and the like.

A logstack log collection node is deployed in the service platform 110, and creates a producer of kafka, and sends collected log information of the service platform to the message queue 120. The message queue 120 is optionally a kafka message queue.

The computing device 130 serves as an engine testing platform, and creates a kafka consumer to obtain log information from the message queue, and generates a test case based on the log information and sends the test case to the recommendation engine 140, so as to test the recommendation engine 140 based on the test case.

The recommendation engine 140 may be an advertisement recommendation engine that may push particular advertisements to particular user populations within particular time periods based on configuration in the business platform 110. The recommendation engine 140 may be a single server, or a server cluster composed of a plurality of servers, or a cloud computing service center, and a plurality of servers used for constituting the server cluster or the cloud computing service center may reside in a plurality of geographic locations.

According to an embodiment of the present invention, the service platform 110, the message queue 120, the computing device 130 and the recommendation engine 140 in the recommendation engine testing system 100 can be implemented by the computing device 200 as described below. FIG. 2 shows a block diagram of a computing device 200, according to one embodiment of the invention.

In a basic configuration 202, computing device 200 typically includes system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.

Depending on the desired configuration, the processor 204 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.

Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, the application 222 can be arranged to operate with program data 224 on an operating system. The program data 224 comprises instructions, and in the computing device 200 according to the invention the program data 224 comprises instructions for executing the method for testing 300 a recommendation engine.

Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.

A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.

Computing device 200 may be implemented as a server, such as a file server, a database server, an application server, a WEB server, etc., or as part of a small-sized portable (or mobile) electronic device, such as a cellular telephone, a Personal Digital Assistant (PDA), a wireless WEB-watch device, an application specific device, or a hybrid device that include any of the above functions. Computing device 200 may also be implemented as a personal computer including both desktop and notebook computer configurations. In some embodiments, the computing device 200 is configured to execute the recommendation engine testing method 300.

FIG. 3 shows a flow diagram of a method 300 for testing a recommendation engine, according to one embodiment of the invention. The method 300 is performed in a computing device, such as the computing device 200, for automated regression testing by a recommendation engine.

As shown in fig. 3, the method begins at step S310. In step S310, log information recorded on the service platform is obtained, and the log information is divided into plan data, schedule data, unit data, and policy data, where the data respectively correspond to a release time, a release position, a targeted group, and a release style of the recommended content, and each data has a plan identifier to which the data belongs.

The kafka producer is created in the service platform, records log information through a logstack method, and sends the recorded log information to a kafka message queue. Therefore, when obtaining the log information recorded in the service platform in step S310, a consumer of kafka may be created in the computing device and obtain the log information from the kafka message queue.

Optionally, in step S310, the divided plan data, schedule data, unit data and strategy data may also be stored in a plan library, a schedule library, a unit library and a strategy library in the database. The databases may be Mysql libraries.

As described above, the log information recorded in the service platform includes a schedule layer, a scheduling layer, a unit layer, and a policy layer, where one schedule may have multiple scheduling, one scheduling may have multiple units, and one unit may have multiple policies. Based on the method, the content of each piece of log information can be divided into plan data, scheduling data, unit data and strategy data which respectively represent the release time, the release position, the directional group and the release format. The releasing time comprises releasing starting and stopping time, and the oriented group comprises at least one of crowd orientation, operating system orientation, brand orientation, region orientation, strategy time period orientation and strategy weight. The policy time period is, for example, a display time period of a certain format of content (e.g., text chain advertisement), and the policy weight is, for example, a display frequency weight of a certain format of content.

FIGS. 4a-4d are diagrams illustrating the structure of planning data, scheduling data, unit data, and policy data, respectively, according to one embodiment of the invention. The unit data comprises plan identification, schedule identification, unit identification (unit ID) and unit information, and the strategy information comprises the plan identification, the schedule identification, the unit identification, strategy identification (strategyID, also called creative identification or material identification) and the strategy information. In addition, the data also respectively have a plan name, a schedule name, a unit name and a strategy name.

For the plan identifier, the scheduling identifier, the unit identifier and the strategy identifier, if the log information only contains the plan identifier, the log information is divided into plan data and stored in a plan library. If only the first two kinds of identification exist, the data are divided into schedule data and stored in a schedule library. If only the first three kinds of identifiers exist at the same time, the identifiers are divided into unit data and stored in a unit library. If the four kinds of identification exist at the same time, the identification is divided into strategy data and stored in a strategy library. By means of the four identifications, a plurality of schedules belonging to the same plan, a plurality of units belonging to the same schedule and a plurality of strategies belonging to the same unit can be determined. The data portions of FIGS. 4a-4d represent scheduling information, unit information, and policy information, respectively, for storing detailed scheduling content, and the like. Knowing the unit identifier, the scheduling identifier and the plan identifier in the strategy data, the corresponding associated data can be searched in the corresponding unit library, the scheduling library and the plan library.

Subsequently, in step S320, scheduling data, unit data and strategy data with the same plan identifier are selected for assembly, and a test case is generated, where the test case includes engine data constructed based on the selected data, a test request for testing the engine data, and an expected return result of the test request.

According to an embodiment, a test platform may be provided in the computing device, as shown in fig. 5a, option boxes with a plan identifier, a schedule identifier, a unit identifier, and a policy identifier are respectively provided in the test platform, and may be configured in each option box in a keyword selection manner. Based on the above, when the scheduling data, the unit data and the strategy data with the same plan identifier are selected for assembly, the selectable scheduling identifier, the unit identifier and the strategy identifier under the plan identifier can be automatically generated in other option boxes based on the plan identifier selected by the tester in the plan option box. Similarly, after the schedule identifier is selected, the unit identifier selectable under the schedule identifier is displayed in the option box of the unit identifier, and the policy identifier selectable under the unit identifier is displayed in the option box of the policy identifier. Then, based on the options of the tester in each option box (i.e. the selected scheduling identifier, unit identifier and strategy identifier), the corresponding plan information, scheduling information, unit information and strategy information are obtained from the database for assembly.

Generally, the engine data includes at least one engine sub-data, each engine sub-data includes a set of schedule data, unit data, and policy data associated with each other, and one schedule data is associated with a plurality of schedule data, one schedule data is associated with a plurality of unit data, and one unit data is associated with a plurality of material data. Here, a schedule identifier, a unit identifier, and a policy identifier may be configured as a complete set to form an engine subdata. But sometimes a complete configuration may have more than two schedules, or more than two units, more than two strategies. For example, to detect A, B two locations of cells, A, B cell identifiers of two locations can be added simultaneously to form one engine sub-data. Similarly, a plurality of strategies may exist in the test scenario, for example, if two strategies are to be tested and can appear alternately, two strategies are necessarily used in the case, and the strategy identifications of the two strategies can be added at the same time to form one engine subdata.

As shown in fig. 5a, a plan of "engine automation", a schedule of "5587", a unit of "engine automation scenario 01-unit 02", a policy of "engine automation scenario 01-policy 02", as a complete piece of engine sub-data. The tester may also add other elements and other policies to the configuration, and may initiate another configuration to select new schedules, elements and policies. And finally, forming complete engine data of a test case by the generated multiple pieces of engine subdata.

For example, if the tester performs two configurations, the first configuration after selecting the keyword is: user a displays the text chain advertisement in 5587 ranks, and the configuration of the second selection keyword is as follows: user B ranks and displays a two-figure advertisement in 5587. Thus, based on the data in the database such as the planning library and the scheduling library in step S310, an engine sub-data may be assembled based on the two configurations, and the two engine sub-data constitute the complete engine data of the test case. At this time, the test case is subjected to two-step detection, namely, whether the advertisement delivered in the area A is returned to the user in the area A is detected, and whether the advertisement delivered in the area B is returned to the user in the area B is detected.

According to another embodiment, the test case further includes media information of the content delivery platform, where the media information includes at least one of a media name, a request path, a request header, a request mode, and a request parameter. As shown in fig. 5a and 5b, the test platform needs to be configured with the media information, for example, a media platform DSP to select content to be delivered, each media has a different request header and a different request body, and therefore needs to be created separately, which is convenient for uniform use in use cases.

According to another embodiment, a request structure is included in the test request, the request structure including a user tag belonging to a targeted population of unit data in the engine data. The user tags may include, for example, at least one of a geographic tag, an age tag, a gender tag, a device tag, and an income tag. The geographic label may be represented by an ip address or a domain name, or a home location of the ip address or the domain name, for example, by city. The age label may be expressed by its actual age, or may be expressed by the age after 80, 85, 90, or the like. The present invention is not limited to the specific expression form of each tag. In addition, the media information may be included in a request structure of the test request.

FIG. 6 shows a schematic diagram of a test request. The test request is actually a test request simulating a user, wherein the user tag is identified as the same attribute as the element in the engine data. For example, if the unit data constituting the engine data refers to the a-place orientation, the simulation user should also be the a-place user, and the user tag in the test request is also the a-place user tag.

Further, the request structure may also carry an expected version and/or an expected style of the user, that is, the request structure includes an expected schedule identifier and/or an expected policy identifier of the user, and the expected schedule identifier and the expected policy identifier are the same as the schedule identifier and the policy identifier in the test case. In this way, after the simulated user sends the test request, it can be detected whether the advertisement with the expected style will appear in the expected position. At this time, the expected return result should include the result corresponding to the expected version and the expected style.

In addition, the test case may include at least one test step, each test step having a corresponding checkpoint, each checkpoint having a corresponding expected return result. As above, the two-step detection of a and B will have two check points, each of which will have a corresponding returned result. Or testing whether the advertisement can be put or not when the plan is started, and testing whether the advertisement can be stopped after the plan is suspended. In the first step, whether the advertisement appears or not is judged to verify whether the engine function is correct or not, and in the second step, a pause request is sent to the engine to judge whether the advertisement does not appear or not. As shown in fig. 5c, different business scenarios have different checkpoints and expected return results, which are typically written in jsonPath, including assertion logic. For example, { "checkType":1, "checkPoint": { "ads [0]. creative _ ID":50303} }) is a material ID check method.

The expected return result is also a preset check logic, which may include at least one of the following check types:

for type 1, assuming that it is detected that the user in the area a can receive the advertisement delivered in the area a, the return result includes the identifier in the area a (50302), and the test case is executed correctly. For type 3, assuming that an advertisement can be received by detecting a place a or B, the returned result contains either the identifier of the place a or B, which represents that the test case executes correctly.

Subsequently, in step S330, the engine data is sent to the recommendation engine, and the actual return result is obtained after the test request is sent to the recommendation engine, and then the actual return result is verified according to the expected return result.

After the engine data is sent to the recommendation engine, the recommendation engine takes the pushable content data, such as advertisement data. After the case is successfully created, debugging can be performed to obtain an actual return result of the case, and whether each sub-step of the case is successfully executed or not can be directly written in the actual return result based on the check logic preset by a tester. FIG. 7 shows a diagram of the actual returned result, including two substeps step1 and step2, both verified, according to one embodiment of the invention. The use case execution succeeds by representing the creation of the use case.

According to an embodiment, the method 300 may further assemble a plurality of test cases belonging to the same plan identifier, and execute all test cases under the plan identifier in batch; and calculating and displaying the test cases which are successfully executed and the test cases which are failed to be executed in the case list corresponding to the plan identification. The assembly case plan is actually to put the written cases into the same execution plan to realize the function of executing the cases in batch. The function range influenced by each code updating iteration is different, so that different plans are assembled according to different iterations, and the tested function can be tested more pertinently.

8a-8c, the present invention assembles a use case plan based on the iterative versions, and the create use case plan interface may select the use case created in step S320, input the use case plan description, and the line of business to which the use case plan belongs. For example, under the "top CPM" plan, there are multiple use cases such as "crowd oriented", "operating system oriented" and "brand oriented", and these multiple use cases can be assembled to execute the "top CPM" plan in batch. In addition, multiple plans such as 'microblog case', 'netease case' and the like are occasionally encountered, and after each plan is executed in batches, the number of failed use cases, the number of successful use cases and the execution source of the use cases can be counted. The details of the use case can be further displayed by viewing the use case list. And the case failing in the plan can be executed again after being clicked in, so that the cause of the problem is confirmed.

The use case execution sources include platform timed execution and development code submission triggers. The former can be realized by Jenkins configuration timing tasks, for example, the timing tasks are set to be triggered once every 2 hours, and the purpose is to poll whether the environment monitoring function is normal. The latter is to automatically trigger the execution of the case after the submitted code is developed, so as to verify whether the submitted code influences the function points covered by the previous cases. When the development code is submitted to trigger deployment, a client script of the platform is called, and the client script is responsible for receiving a platform smoking test triggered by the code module, counting successful cases or failed cases and carrying out mail alarming.

In summary, the invention stores the real data sent to the engine by the service platform to the test platform through the logstack node, and assembles the engine data in a keyword data combination manner, thereby improving the efficiency of assembling the engine data and greatly improving the accuracy of the data. The method can flexibly configure the multimedia delivery, the provided verification scheme can cover more than 90% of request verification of a recommendation engine, and the data is positioned according to the JSON data structure path, so that the method is convenient to start and uniform in writing. In addition, steps can be added according to the actual scene of the use case, which is equivalent to that one use case contains a plurality of use cases, and a plurality of test points can be tested in one use case. Moreover, the cases can be combined to form a test set, different test cases can be combined aiming at different versions in an iteration mode, and each test regression is guaranteed to have pertinence. When the use case fails to be executed, the use case execution source can be checked, and the failure reason can be located in an auxiliary mode.

FIG. 9 shows a block diagram of a recommendation engine testing apparatus 900 according to one embodiment of the present invention, the apparatus 900 may reside in the computing device 200. As shown in fig. 9, the apparatus 900 includes: a data acquisition module 910, a use case generation module 920, and a use case execution module 930.

The data obtaining module 910 obtains log information recorded on a service platform, and divides the log information into plan data, scheduling data, unit data, and policy data, where the data respectively correspond to a release time, a release position, a targeted group, and a release style of a recommended content, and each data has a plan identifier.

According to one embodiment, the data retrieval module 910 may retrieve log information from a kafka message queue by a consumer that creates kafka in a computing device. The data obtaining module 910 may further store the divided planning data, scheduling data, unit data, and material data into a planning library, a scheduling library, a unit library, and a material library, respectively. The data acquisition module 910 may perform processing corresponding to the processing described above in step S310, and the detailed description thereof is omitted.

The use case generating module 920 selects scheduling data, unit data, and strategy data with the same plan identifier to assemble, and generates a test case, where the test case includes engine data constructed based on the selected data, a test request for testing the engine data, and an expected return result of the test request.

According to one embodiment, the use case generation module 920 may automatically generate the schedule identifier, the unit identifier and the material identifier selectable under the schedule identifier in other option boxes based on the schedule identifier selected by the tester in the schedule option box. And then acquiring corresponding plan information, scheduling information, unit information and material information from the database for assembly based on the options of the tester in each option box. The use case generation module 920 may perform processing corresponding to the processing described above in step S320, and details thereof are not repeated here.

The case execution module 930 sends the engine data to the recommendation engine, and sends the test request to the recommendation engine to obtain an actual return result, and then verifies the actual return result according to an expected return result. The use case execution module 930 may perform processing corresponding to the processing described above in step S330, and details thereof are not repeated here.

According to an embodiment, the use case execution module 930 may further assemble a plurality of test use cases belonging to the same plan identifier, and execute all test use cases under the plan identifier in batch; and calculating and displaying the test cases which are successfully executed and the test cases which are failed to be executed in the case list corresponding to the plan identification.

According to the technical scheme, the invention provides an automatic scheme for testing the recommendation engine based on the recorded data, which can automatically generate complete engine data by recording the data of the pushing engine of the service end and selecting keywords on the test platform, thereby effectively ensuring the correctness and the high efficiency of the data. In addition, the invention also provides a test set dimension, so that the subsequent iteration is facilitated according to different versions, a plurality of single cases are combined into a test case set (case plan), and test regression is performed in a targeted manner. The test case assembled based on the log recording and keyword selecting modes has the characteristics of good user experience, and quick, accurate and reliable data assembly.

A7, the method according to any one of A1-A6, wherein the engine data includes at least one engine subdata, each engine subdata includes a set of schedule data, unit data and policy data associated with each other. A8, the method of any one of A1-A7, wherein the test request includes a request structure, the request structure includes a user tag, and the user tag belongs to a directed group of unit data in the engine data. A9, the method of any one of A1-A8, wherein the test case includes at least one test step, each test step has a corresponding checkpoint, and each checkpoint has a corresponding expected return. A10, the method of A9, wherein the expected return results include at least one of: expecting an accurate check value to return, expecting an accurate check value not to appear, the expected check point fuzzy matches any value, the expected check point is in the range list, expecting an accurate probability value to appear, expecting an accurate frequency check number to appear.

A11, the method of any one of A1-A10, further comprising the steps of: assembling a plurality of test cases belonging to the same plan identification, and executing all the test cases under the plan identification in batch; and calculating and displaying the test cases which are successfully executed and the test cases which are failed to be executed in the case list corresponding to the plan identification. A12, the method as in a1, wherein the log information recorded by the service platform is sent to a kafka message queue, and the step of acquiring the log information recorded by the service platform includes: a consumer creating a kafka in the computing device, retrieving the log information from the kafka message queue. A13, the method as in A5, wherein the service platform records log information by logstack method, and the database is Mysql database.

The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.

In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the recommendation engine testing method of the present invention according to instructions in the program code stored in the memory.

By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.

In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.

As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense with respect to the scope of the invention, as defined in the appended claims.

Claims (10)

1. A method for testing a recommendation engine, adapted to be executed in a computing device, the computing device being communicatively connected to the recommendation engine and a service platform, respectively, the method comprising the steps of:

acquiring log information recorded on the service platform, and dividing the log information into plan data, scheduling data, unit data and strategy data, wherein the data respectively correspond to the release time, release position, oriented group and release style of recommended content, and each data has a plan identifier;

selecting scheduling data, unit data and strategy data with the same plan identification to assemble to generate a test case, wherein the test case comprises engine data constructed based on the selected data, a test request for testing the engine data and an expected return result of the test request; and

and sending the engine data to a recommendation engine, sending the test request to the recommendation engine to obtain an actual return result, and verifying the actual return result according to the expected return result.

2. The method of claim 1, wherein,

the plan data has a plan identification and plan information;

the scheduling data comprises plan identification, scheduling identification, position identification and scheduling information;

the unit data comprises a plan identifier, a scheduling identifier, a unit identifier and unit information;

the strategy information comprises a plan identifier, a scheduling identifier, a unit identifier, a strategy identifier and strategy information.

3. The method of claim 1 or 2, further comprising the step of:

and respectively storing the divided plan data, scheduling data, unit data and strategy data into a plan library, a scheduling library, a unit library and a strategy library.

4. The method of any one of claims 2-3, wherein a test platform is disposed in the computing device, and option boxes of the plan identifier, the schedule identifier, the unit identifier, and the policy identifier are disposed in the test platform, respectively.

5. The method of claim 4, wherein the step of selecting scheduling data, unit data and strategy data having the same schedule identification for assembly comprises:

based on the plan identifier selected by the tester in the plan option box, automatically generating a schedule identifier, a unit identifier and a strategy identifier which are selectable under the plan identifier in other option boxes; and

and acquiring corresponding plan information, scheduling information, unit information and strategy information from the database for assembly based on the options of the tester in each option box.

6. The method according to any one of claims 1-5, wherein the test case further includes media information of a content delivery platform, and the media information includes at least one of a request path, a request header, and a request parameter.

7. A recommendation engine testing apparatus adapted to be resident for execution in a computing device communicatively coupled to a recommendation engine and a service platform, respectively, the apparatus comprising:

the data acquisition module is suitable for acquiring the log information recorded on the service platform and dividing the log information into plan data, scheduling data, unit data and strategy data, wherein the data respectively correspond to the release time, release position, oriented group and release style of the recommended content, and each data has a plan identifier;

the test case generation module is suitable for selecting scheduling data, unit data and strategy data with the same plan identification to be assembled to generate a test case, wherein the test case comprises engine data constructed based on the selected data, a test request for testing the engine data and an expected return result of the test request; and

and the case execution module is suitable for sending the engine data to a recommendation engine, sending the test request to the recommendation engine to obtain an actual return result, and further verifying the actual return result according to the expected return result.

8. A computing device, comprising:

at least one processor; and

at least one memory including computer program instructions;

the at least one memory and the computer program instructions are configured to, with the at least one processor, cause the computing device to perform the method of any of claims 1-6.

9. A readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any of claims 1-6.

10. A recommendation engine testing system comprising:

the service platform is suitable for recording log information;

the computing device of claim 8, adapted to generate a test case based on log information recorded by the service platform, the test case including engine data, a test request, and an expected return result; and

and the recommendation engine is suitable for returning a result to the recommendation engine after receiving the engine data and the test request sent by the computing equipment so that the computing equipment can verify the returned result.

Images