CN113296653A

CN113296653A - Simulation interaction model construction method, interaction method and related equipment

Info

Publication number: CN113296653A
Application number: CN202110850474.5A
Authority: CN
Inventors: 吴宗翰; 吴俊骅; 刘冰; 姜毅
Original assignee: Alibaba Cloud Computing Ltd
Current assignee: Alibaba Cloud Computing Ltd
Priority date: 2021-07-27
Filing date: 2021-07-27
Publication date: 2021-08-24
Anticipated expiration: 2041-07-27
Also published as: CN113296653B

Abstract

The embodiment of the application provides a simulation interaction model construction method, an interaction method and related equipment, wherein the simulation interaction model construction method comprises the following steps: acquiring interactive data of a user and an application program in advance; constructing a simulation interaction model at least based on the interaction data, wherein the simulation interaction model provides interaction operations which are associated with the pre-acquired interaction data, so that the interaction operations provided by the simulation interaction model are used for calling the pre-acquired interaction data; and generating a simulation interaction page which is corresponding to the simulation interaction model and is used for user interaction. The embodiment of the application can reduce the realization cost of the immersive interactive experience and improve the universality.

Description

Simulation interaction model construction method, interaction method and related equipment

Technical Field

The embodiment of the application relates to the technical field of user interaction, in particular to a simulation interaction model construction method, an interaction method and related equipment.

Background

When an internet service provider publicizes or introduces a new internet application program (application program for short) or a new function of the application program, or when a user uses the new application program or the new function of the application program for the first time, interactive guidance is often needed to guide the user how to use the application program; particularly, for some applications involving complex user operations, effective interactive guidance enables users to use the applications more quickly. The interactive guidance can guide the interactive experience of the user and the application program through visual guidance, visual action, sound guidance and other modes, so that the immersive interactive experience is created based on the sensory experience and the cognitive experience of the user in the interactive process of the user and the application program.

The immersive interactive experience undoubtedly provides a way for users to use some applications involving complex user operations. However, each application program has its own unique technology stack and technology implementation when being developed, and if the immersive interactive experience is implemented by modifying the code of the application program, there is undoubtedly a large modification cost; and the immersive interactive experience of each application program needs to be realized by respectively modifying the codes of each application program, so that the generalization is difficult to realize. Therefore, how to provide an immersive interactive experience scheme to reduce the implementation cost of the immersive interactive experience and improve the universality becomes a problem to be solved by those skilled in the art.

Disclosure of Invention

In view of this, embodiments of the present application provide a simulation interaction model construction method, an interaction method, and related devices, so as to construct a simulation interaction model in which a user interacts with an application program through a simulation technique, thereby providing an immersive interaction experience for the user and the application program in a simulation layer, reducing implementation cost of the immersive interaction experience, and improving generality.

In order to achieve the above object, the embodiments of the present application provide the following technical solutions.

In a first aspect, an embodiment of the present application provides a method for building a simulation interaction model, where the method includes:

acquiring interactive data of a user and an application program in advance;

constructing a simulation interaction model at least based on the interaction data, wherein the simulation interaction model provides interaction operations which are associated with the pre-acquired interaction data, so that the interaction operations provided by the simulation interaction model are used for calling the pre-acquired interaction data;

and generating a simulation interaction page which is corresponding to the simulation interaction model and is used for user interaction.

In a second aspect, an embodiment of the present application provides an interaction method, where the method includes:

displaying a simulation interaction page corresponding to the simulation interaction model;

detecting a current interactive operation on the simulation interactive page, wherein the current interactive operation is provided by a simulation interactive model;

and calling interactive data associated with the current interactive operation, and displaying a corresponding interactive effect on the simulation interactive page, wherein the interactive data is interactive data of a user and an application program, which is acquired in advance.

In a third aspect, an embodiment of the present application provides a simulation interaction model building apparatus, including:

the data acquisition module is used for acquiring interactive data of a user and an application program in advance;

a model construction module for constructing a simulation interaction model based at least on the interaction data, wherein the simulation interaction model provides interaction operations associated with pre-acquired interaction data such that the interaction operations provided by the simulation interaction model are used to invoke the pre-acquired interaction data;

and the page generation module is used for generating a simulation interaction page which is corresponding to the simulation interaction model and is used for user interaction.

In a fourth aspect, an embodiment of the present application provides an interaction apparatus, including:

the page display module is used for displaying a simulation interaction page corresponding to the simulation interaction model;

the operation detection module is used for detecting current interactive operation on the simulation interactive page, and the current interactive operation is provided by a simulation interactive model;

and the data calling and displaying module is used for calling interactive data associated with the current interactive operation and displaying a corresponding interactive effect on the simulation interactive page, wherein the interactive data is interactive data of a user and an application program, which is acquired in advance.

In a fifth aspect, an embodiment of the present application provides a model building apparatus, including: at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform a method of modeling a simulated interaction model as described above in relation to the first aspect.

In a sixth aspect, an embodiment of the present application provides an interactive front end, including: at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform the interaction method as described in the second aspect above.

In a seventh aspect, an embodiment of the present application provides a storage medium storing one or more computer-executable instructions that, when executed, implement the simulation interaction model building method according to the first aspect or the interaction method according to the second aspect.

In an eighth aspect, an embodiment of the present application provides a program, which when executed by an electronic device, implements the simulation interaction model building method according to the first aspect described above, or the interaction method according to the second aspect described above.

The method for constructing the simulation interaction model can acquire interaction data of a user and an application program in advance in the real interaction process of the user and the application program; constructing a simulation interaction model at least based on the interaction data, wherein the simulation interaction model provides interaction operations which are associated with the pre-acquired interaction data, so that the interaction operations provided by the simulation interaction model are used for calling the pre-acquired interaction data; and generating a simulation interaction page which is corresponding to the simulation interaction model and is used for user interaction. Therefore, the simulation interaction model constructed in the embodiment of the application can provide interaction operation in a simulation environment, and the interaction operation can directly call interaction data acquired in advance without acquiring the interaction data through a real interaction interface; therefore, the simulation interaction model constructed in the embodiment of the application can enable the user interaction process to be carried out in a simulation environment without calling a real interaction interface, so that code transformation of the application program is reduced, and the application program can be used universally in a plurality of application programs. According to the embodiment of the application, the simulation interaction model for interaction between the user and the application program is built through a simulation technology, so that immersive interaction experience is provided for the user and the application program on a simulation level, the realization cost of the immersive interaction experience is reduced, and the universality is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a diagram illustrating constraints of an immersive interactive experience.

Fig. 2 is a schematic diagram of a system architecture for building a simulation interaction model according to an embodiment of the present application.

Fig. 3 is a flowchart of a simulation interaction model building method provided in the embodiment of the present application.

Fig. 4 is a schematic diagram of a stage of building a simulation interaction model according to an embodiment of the present application.

Fig. 5a is a flowchart of collecting interaction data according to an embodiment of the present application.

FIG. 5b is a diagram of an example of a native API of a browser.

Fig. 5c is a flowchart illustrating an implementation of the injection extractor according to an embodiment of the present application.

FIG. 5d is a diagram illustrating the technical architecture of MarkUp.

Fig. 5e is a schematic diagram of a situation resulting in the collected data not being unique.

Fig. 5f is a schematic diagram of the formation of the unique identifier of the interaction data.

FIG. 6a is a sub-phase schematic diagram of the simulation modeling phase.

FIG. 6b is a schematic diagram of the components of the simulation interaction model.

FIG. 6c is a diagram of an example implementation of a page snapshot technique.

FIG. 6d is a schematic diagram of element capture and display of a page.

FIG. 6e is a schematic diagram of a build page.

Fig. 7a is a flowchart of an interaction method according to an embodiment of the present application.

Fig. 7b is a schematic diagram of an interaction implementation provided in the embodiment of the present application.

Fig. 8a is a block diagram of a simulation interaction model building apparatus according to an embodiment of the present application.

Fig. 8b is a block diagram of a model building apparatus according to an embodiment of the present application.

Fig. 8c is a block diagram of an interaction device provided in the embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Immersive experience: immersion Experience (Flow Experience), also known as immersion Theory (Flow Theory). When a user enters the immersive experience, the user has conditions such as high challenges and encouragement (highlevels of exchange and outlay), concentration (focused attention), and enhanced interaction and telepresence (enhanced by interaction and telepresence). The immersive experience referred to in the embodiment of the application is immersive interactive experience, and particularly refers to the purpose of using software assistance to achieve the purposes of challenging, encouraging, attentive concentration and interaction and the like in the interactive experience process of a user and an application program in the field of internet, so that the user is immersed in the immersive interactive experience.

The inventors of the present application found that: the application program to realize the immersive interactive experience is often restricted by three items, and has the problems of difficult generalization and high realization cost. FIG. 1 illustrates a constraint diagram of an immersive interactive experience. As shown in fig. 1, the three constraints include: real online environment constraints, business stability constraints, and application program diversity constraints. The real online environment restriction refers to a restriction from an online environment generated in consideration of the running operation of an application program after the application program is online, and includes: interface authentication problems, resource occupation problems, security problems, interface delay problems, and the like. The service stability constraint mainly means that when an application program iterates, the immersive interactive experience also needs to be adjusted accordingly, otherwise, the stability of the immersive interactive experience may be degraded, for example, an experience content of the immersive interactive experience is wrong. The stability of the immersive interactive experience directly affects the quality of the immersive interactive experience. The difference of the application programs is restricted by the difference of the application programs, and front-end frameworks, interface format definitions and account systems used by different application programs are different, so that the realization cost of the immersive interactive experience is high, and the immersive interactive experience is difficult to copy and use. Based on the above constraints, if the codes of the application programs are directly modified to realize the immersive interactive experience, the problems of high realization cost and difficult generalization are brought.

Based on the method, the embodiment of the application provides a novel immersive interactive experience scheme, the immersive interactive experience scheme can design the interactive process of the user and the application program into a simulation process, and a simulation interactive model for interaction is constructed based on a simulation technology; through the simulation interaction model, the user can realize immersive interaction experience in the interaction guidance of the application program without logging in an account and calling an interface.

The construction scheme of the simulation interaction model provided by the embodiment of the present application and the interaction scheme based on the simulation interaction model will be introduced below.

Fig. 2 schematically shows a system architecture diagram for building a simulation interaction model according to an embodiment of the present application. As shown in fig. 2, the system architecture for building the simulation interaction model may include: simulation front end 210, application server 220, simulation server 230. The simulation front end 210 is a client used for constructing a simulation interaction model, and the simulation front end 210 may be deployed on an electronic device such as a mobile phone and a PC (personal computer). In some embodiments, the emulation front-end 210 may provide a browser (e.g., an H5-based browser). The application server 220 is a server device for providing service support of the application, and the simulation front end 210 can interact with the application server 220 and generate interaction data during the real interaction process between the user and the application. In some embodiments, the application server 220 may support H5 (HTML 5, Hypertext 5) based services. The simulation Server (Mock Server) 230 may be a service device deployed on the network side used to build a simulation interaction model. The simulation interaction model constructed in the embodiment of the application can be stored in a simulation server.

In some embodiments, the user of the emulation front-end 210 may be a technician (e.g., an operator) of an internet service provider. That is, the simulation front end 210 may be a client used in constructing the simulation interaction model according to the embodiment of the present application, and the user thereof may be a technician of an internet service provider. For example, a technician of an internet service provider may build a simulated interaction model using the simulation front end 210 based on the interaction guidance requirements of the application. And when the specific interactive guidance is carried out, the interactive guidance object can be a user using the application program, and at the moment, the user can carry out the specific interactive guidance by using the interactive front end based on the simulation interactive model. In an optional implementation, a front end used in a stage of building a simulation interaction model in the embodiment of the present application is a simulation front end, and a user of the front end may be a technician of an internet service provider; the front end used in the interactive guidance stage is an interactive front end, and the user of the interactive front end can be a user using the application program.

As an alternative implementation, fig. 3 shows an alternative flowchart of a simulation interaction model building method provided in an embodiment of the present application. As shown in fig. 3, the process may include the following steps.

In step S310, interaction data between the user and the application is obtained in advance.

The simulation front end may provide the ability for a user to actually interact with the application, e.g., the simulation front end may provide a browser so that the user can actually interact with the application through the browser. In the real interaction process of the user and the application program, the embodiment of the application can acquire the interaction data of the user and the application program in advance. In some embodiments, the user herein may be a technician (e.g., an operator) of an internet service provider; in other embodiments, the user may also be a user using the application, that is, the embodiment of the present application may also support building a simulation interaction model based on a real interaction process between the user and the application.

In some embodiments, the emulation front-end can interact with the application server through a native interface for interaction requests to enable real user interaction with the application. The native interface can send an interactive request to the application server based on the interactive operation of the user and obtain interactive response data fed back by the application server. As an optional implementation, in the embodiment of the application, the native interface may be hijacked to hijack the interactive data between the user and the application program in the real interactive process between the user and the application program, so as to obtain the interactive data between the user and the application program in advance.

In some embodiments, step S310 may be implemented by the emulation front-end, for example, the emulation front-end hijacks a native interface to obtain the interaction data of the user and the application program in advance. In other embodiments, step S310 may be implemented by an emulation server, for example, the emulation server may obtain the interaction data hijacked by the emulation front end in advance.

In step S311, a simulation interaction model is constructed based on at least the interaction data, wherein the simulation interaction model provides interaction operations, and the interaction operations are associated with the pre-acquired interaction data, so that the interaction operations provided by the simulation interaction model are used for calling the pre-acquired interaction data.

After interaction data of a user and an application program is acquired in advance, the embodiment of the application can construct a simulation interaction model at least based on the interaction data. It should be noted that, the simulation interaction model may provide interaction operations for the user at the simulation level, and present corresponding interaction effects after the provided interaction operations are executed. In order to avoid that a real interaction interface needs to be called for user interaction realized based on the simulation interaction model, the embodiment of the application can associate interaction operation provided by the simulation interaction model with interaction data acquired in advance in step S310, so that the interaction data acquired in advance in step S310 can be directly called through the interaction operation provided by the simulation interaction model.

In some embodiments, the simulated interaction model may provide multiple interaction scenarios of a user with an application, and provide a set of interactions under each interaction scenario. Each interactive operation in the set of interactive operations may be associated with the interactive data that is obtained in advance in step S310, so that when performing interactive guidance, a user performs a certain interactive operation based on the simulation interaction model. Therefore, the simulation interaction model constructed in the embodiment of the application can enable the user interaction process to be carried out in a simulation environment without calling a real interaction interface, so that code transformation of the application program is reduced, and the application program can be used universally in a plurality of application programs.

In some embodiments, step S310 may send an interaction request to the application server based on the interaction of the user, so as to obtain the interaction data between the user and the application in advance, and in the embodiments of the present application, the interaction corresponding to the obtained interaction data may be determined to be prefetched, that is, the interaction for obtaining the interaction data is determined to be triggered; furthermore, the interactive operation for triggering the acquisition of the interactive data can be provided by the simulation interactive model, and the interactive operation is associated with the interactive data acquired in advance.

In some embodiments, step S311 may be implemented by an emulation front-end execution. Optionally, after the simulation front end constructs the simulation interaction model, the simulation interaction model may be uploaded to the simulation server, so that the simulation interaction model may be acquired from the simulation server when user interaction is subsequently implemented. In other embodiments, step S311 may be implemented by a simulation server, for example, the simulation server may construct a simulation interaction model based on the interaction data hijacked by the simulation front end.

In step S312, a simulation interaction page for user interaction corresponding to the simulation interaction model is generated.

When the user interaction is realized, the simulation interaction page for the user interaction can be provided, so that the user can realize the user interaction (such as interaction guidance) based on the simulation environment under the simulation interaction page. Therefore, after the simulation interaction model is constructed, the simulation interaction page corresponding to the simulation interaction model can be generated, so that the subsequent user interaction process can be conveniently used.

In some embodiments, step S312 may be implemented by the simulation front end execution. In other embodiments, step S312 may also be implemented by the simulation server.

As an alternative implementation, FIG. 4 is a schematic diagram illustrating a phase of building a simulation interaction model according to an embodiment of the present application. As shown in FIG. 4, the stage of building a simulated interaction model may include: a script information entry phase 410, a data collection phase 420, a simulation modeling phase 430, and a version setup phase 440. In some embodiments, the illustrated stages may be implemented by the simulation front end, and the simulation front end uploads the output result of the stages to the simulation server; for example, the simulation front-end may collect user interaction data with the application during a data collection phase 420, build a simulation interaction model during a simulation modeling phase 430, and build a version of the simulation interaction model during a version build phase 440. After the simulation front end realizes the construction and the version determination of the simulation interaction model, the simulation front end can upload the acquired interaction data and the constructed simulation interaction model to the simulation server so as to be used in the user interaction process in the subsequent simulation environment. Of course, the implementation of the above phase by the simulation front end is only an example, and the embodiment of the present application does not exclude the case where the simulation server implements the above partial phase, which may depend on the technical implementation configuration. The following describes the implementation of the above stages by taking as an example that the simulation front end mainly implements the above stages.

In the script information entry stage 410, a user (e.g., a technician of an internet service provider) may fill out script basic information of the simulated interaction model, such as: name of the simulation interaction model, application program used for interaction, interaction scene (also called interaction case) of the user and the application program, and the like. In some embodiments, the present application may provide a build page of the simulation interaction model, where the build page may be displayed at the front end of the simulation, and a user may input the basic script information in the build page. It should be noted that, the simulation interaction model for user interaction provided in the embodiment of the present application may also be referred to as a script, and the name of the simulation interaction model may also be referred to as a script name.

During the data collection phase 420, the simulation front end can collect user interaction data with the application. In some embodiments, the simulation front end may collect interactive data of a user and an application program in an interactive scene in a recording environment, and then record the interactive data to the simulation server to achieve pre-acquisition of the interactive data. Optionally, after the interactive data is collected by the simulation front end, the interactive data may be stored in the local database, and then the interactive data stored in the local database is uploaded to the simulation server after the construction and the version determination of the simulation interactive model are implemented.

In some embodiments, the build page may provide a start recording button that the simulation front end may enter the recording environment when the user clicks. Under the recording environment, the simulation front end can hijack the interactive data of the user and the application program under the interactive scene by hijacking the native interface used for the interactive request.

In the simulation modeling phase 430, the simulation front end can build a simulation interaction model based on the collected interaction data. Optionally, the simulation interaction model constructed by the simulation front end may be stored in the local database, and then the simulation interaction model stored in the local database is uploaded to the simulation server after the version determination of the simulation interaction model is implemented.

In some embodiments, the simulated interaction model may be viewed as a transcript, which may include one or more steps. The steps may refer to simulating an interaction scenario (e.g., an interaction case) under an interaction model, the steps including one or more substeps. The sub-step refers to a segment descriptor in an interaction scenario under the simulation interaction model, and may include a run operation and/or a rollback operation. A run operation refers to running a set of interactions under a sub-step to introduce or guide the use of the functionality of an application. The rollback operation may backtrack an interactive operation in which an interactive error occurred or a previous interactive operation. The interactive operation is the smallest granularity in the simulation interaction model and can be regarded as an Action (Action) to be performed, such as clicking, focusing, inputting data, and the like.

Based on this, in the case that the simulation interaction model is regarded as a script, in the simulation modeling stage, the process of constructing the simulation interaction model in the embodiment of the present application may at least include: creating a step under the scenario, creating a sub-step under the step, creating a run operation and/or a rollback operation under the sub-step, creating a plurality of interactive operations under the run operation and/or the rollback operation. For example, the embodiment of the application may create steps under the scenario, sub-steps under the scenario, and various interactive operations included in the running operation and/or the rollback operation under the sub-steps based on the interaction scenario of the user and the application program entered in the scenario information entry stage, so that the simulation interaction model may provide the interactive operations, and the interactive operations in the running operation and/or the rollback operation under the sub-steps may directly call the interactive data acquired in the data acquisition stage.

In the version setup phase 440, the embodiments of the present application may implement version management of the simulation interaction model.

In some embodiments, in order to facilitate a user to perform an interactive operation in a simulation environment, the embodiments of the present application may provide a simulation page corresponding to a simulation interaction model, so that the user may perform the interactive operation in the simulation page corresponding to the simulation interaction model.

According to the method and the device, the interactive process of the user and the application program can be designed into a simulation process by constructing the simulation interactive model for interaction; after the user executes the interactive operation provided by the simulation interactive model, the interactive data acquired in advance can be directly called without acquiring the interactive data through a real interactive interface, so that the immersive interactive experience of the user and the application program can be realized in a simulation environment. Interactive guidance of applications, for example, may be implemented in a simulated environment and provide an immersive interactive experience.

The following describes alternative implementation processes of the embodiment of the present application in the data acquisition phase 420, the simulation modeling phase 430, and the version establishment phase 440.

In the data collection phase 420, the emulation front-end may refresh the page where the user interacts with the application program, and hijack the interactive data by hijacking the native interface of the emulation front-end used to send the interactive request before the page is reloaded. The above process may be performed in a recording environment. As an alternative implementation, fig. 5a exemplarily shows an alternative flowchart for collecting interaction data provided by an embodiment of the present application. In some embodiments, this flow may be performed by the simulation front end. As shown in fig. 5a, the process may include the following steps.

In step S510, enter the recording environment, and trigger refreshing the page where the user interacts with the application.

In some embodiments, when the user clicks the start recording button of the build page, the emulation front-end can enter the recording environment and trigger a page reload (i.e., trigger a refresh of the page that the user interacts with the application).

In step S511, before the page reloads, the native interface of the emulation front-end for the interactive request is hijacked.

In the real interaction process between the user and the Application program, the simulation front end may send an interaction request to the Application server through a native API (Application Programming Interface) for the interaction request and obtain corresponding interaction response data fed back by the Application server, so as to complete the real interaction between the user and the Application program. For example, the simulation front end may load a page where a user interacts with the application program, and the user may perform an interaction operation on the page, so that the simulation front end may send an interaction request to the application server through the native API, and further obtain interaction response data fed back by the application server.

According to the embodiment of the application, the native API of the simulation front end for the interactive request can be hijacked to hijack the interactive response data corresponding to the interactive request, so that the interactive data can be acquired. In some embodiments, the native API of the emulation front-end for the interactive request can be hijacked by the injection extractor. The extractor may be configured to extract the interface information. For example, the embodiment of the application can forward the native API used by the emulation front end for the interactive request to the proxy API through the injection extractor, so as to hijack the native API.

In step S512, an interaction request is sent based on the hijacked native interface, and interaction response data corresponding to the interaction request is captured.

Based on the hijacked native interface, the embodiment of the application can send the interaction request to the application server through the native interface, and capture the interaction response data which is fed back by the application server and corresponds to the interaction request. In some embodiments, if hijacking the native API is achieved by transferring the native API to a proxy API, the embodiment of the present application may initiate an interaction request based on the native API, and capture corresponding interaction response data through the proxy API.

After the interaction request and the corresponding interaction response data are collected through the hijack native interface, the simulation front end can store the interaction request and the corresponding interaction response data (the interaction request and the corresponding interaction response data can form interaction data) into a local database, and then upload the interaction data in the local database to a simulation server after a simulation interaction model is built and a model version is determined.

In some embodiments, the interaction request may include information such as a request parameter, a request header, and the like, and the embodiment of the present application may capture and encapsulate the request parameter, the request header, and cookies of the simulation front end (for example, cookies of a browser) through a proxy API, and then record the captured data together with corresponding real interaction response (response) data to the simulation server. In an alternative implementation, the simulation front end may store the data in the local database, and then record the data stored in the local database into the simulation server.

Because the embodiment of the application does not realize immersive interactive experience in a real interactive environment, but realizes immersive interactive experience in a simulation environment based on the constructed simulation interactive model, in order to ensure the accuracy and the authenticity of the simulation interactive model in the realization of simulation interaction, the data used for constructing the simulation interactive model in the embodiment of the application can be derived from interactive data of a user and an application program in the real interactive process, including interactive requests of the user and the application program in the real interactive process and corresponding interactive response data. Based on this, the embodiment of the application acquires the interactive data in the real interactive process between the user and the application program in advance, associates the interactive data acquired in advance with the interactive operation provided by the subsequently constructed simulation interactive model, and enables the user interaction realized based on the simulation interactive model to directly call the real interactive data acquired in advance through the interactive operation without acquiring the real interactive data through a real interactive interface.

In an example, taking the simulation front end as an example to provide a Browser, in the embodiment of the present application, a Native Browser api (Native Browser api) used by the Browser for the interaction request is converted into a Native Browser proxy api (proxy Native Browser api) to hijack a Native interface used by the simulation front end for the interaction request.

FIG. 5b is a diagram illustrating an example of a browser's native API transition. As shown in fig. 5b, the Browser may make the interaction request through a native Browser API (native Browser API), such as XMLHttpRequest or fetch, and the native Browser API may be regarded as a native API for the Browser to use in the interaction request. For example, through a native browser API such as XMLHttpRequest or fetch, the browser may send an interaction request to the application server and obtain corresponding interaction response data based on the interaction operation of the user. The XMLHttpRequest is an API of the client, and a convenient channel is provided for the communication between the browser and the server; fetch provides a JavaScript interface for accessing and manipulating specific parts of an HTTP (HyperText Transfer Protocol) pipeline, such as requests and responses. The embodiment of the application can transfer the Native Browser API (application programming interface) such as XMLHttpRequest or fetch and the like to obtain a Native Browser proxy API (proxy Native Browser API). The native browser proxy API may have the following functions: and capturing information such as request parameters of the interaction request, a request packet header and cookies of the browser, and capturing interaction response data. Furthermore, the native browser agent API can return interactive response data to the native browser API after recording the interactive request and the corresponding interactive response data to the emulation server, so that the data recording under the recording environment can normally continue to run.

In some embodiments, after the native browser APIs such as XMLHttpRequest or fetch are transferred to obtain the native browser proxy API, the embodiment of the present application may implement hijacking of the interaction response data corresponding to the interaction request based on the hijacking configuration information. The hijacking configuration information may indicate at least: initiating an interaction request by a native XMLHttpRequest or Fetch API, and capturing corresponding interaction response data by a local browser proxy API; then, packaging request parameters, a request packet header and cookies of the interactive request by using a local browser agent API, and sending the packaged request parameters, the request packet header and the cookies together with interactive response data to a simulation server for storage; and finally, returning the interactive response data to the native XMLHttpRequest or Fetch API, so that the recording environment can normally continue to run.

It should be noted that the native API exposed by the mainstream browser for the interactive Request generally follows the W3C (World Wide Web Consortium) protocol, and includes an XMLHttpRequest native API based on the XHR (XML Http Request) protocol and a new generation fetch API. Some mainstream request libraries are third-party libraries encapsulated based on XMLHttpRequest or fetch. Therefore, the embodiment of the application realizes data acquisition by hijacking the native API of the simulation front end for the interactive request, can solve the problem that different request libraries may be relied on in different systems, and reduces the adaptation cost of data acquisition.

In some embodiments, the present application can inject the data into the extractor through the injector before the user reloads the page interacting with the application program, so as to hijack the native interface of the simulation front end for the interaction request. In order to avoid the situation of missing capture of interface information of the native API, the embodiment of the present application needs to precisely define the timing of injecting the native interface by the extractor, so as to ensure that the timing of hijacking the native interface is necessarily before the native interface sends the interaction request. As an alternative implementation, fig. 5c shows a flowchart of an alternative implementation of the injection extractor according to the embodiment of the present application. In some embodiments, the flow may be implemented by simulation front-end execution. As shown in fig. 5c, the process may include the following steps.

In step S520, before the page reloads, a tag is written in localStorage of the emulation front-end to inform the emulation front-end that the injector needs to be enabled to inject the extractor at the native interface.

After triggering the refresh of the page where the user interacts with the application and before the page is reloaded, the embodiment of the present application may write a tag in localStorage of the emulation front-end to indicate that the injector needs to be enabled to inject the extractor into the native interface when the page is reloaded.

In step S521, upon page reload, the injector is loaded; the injector activates an extractor based on the tag detected in localStorage to inject the extractor into a native interface.

In some embodiments, the present application may trigger the emulation front end to refresh the page by a window reload method to call an interface used in a page loading phase in a real interaction environment. The emulated front-end (e.g., browser) will then go through the onload state (which occurs when the page exits) and enter the process of reloading the page resources. In this process, due to the lifecycle mechanism of the browser extension program, when the document read state of the page is a loading stage, the browser loads a code for starting a content script of the document and running the content script, and at this time, the embodiment of the present application may implement loading of an injector. The extractor is activated when the injector detects a tag in the localStorage, thereby injecting the extractor into the native interface to activate the capability of requesting to hijack the native interface.

Since the whole process is performed synchronously, the embodiment of the present application can ensure that the injection timing of the extractor is certainly faster than the occurrence time point of the first interactive request of the native API. Therefore, after the extractor completes injection and the simulation front end completes preparation of the start environment, the embodiment of the application can enter a stage in which the document ready is interactive, at this time, resources related to the page are loaded and operated, and the first interactive request of the native API occurs at this interactive stage.

In one example, the injector may exist as a network content script (network content script); taking a browser as an example, an implementation of loading a content script for starting a document according to an embodiment of the present application may include: due to the lifecycle mechanism of the Chrome Extension program, when a page enters a document ready state as a loading stage, the Chrome Extension program loads all content scripts with a run _ at state as document _ start and runs a code. The injector then activates the extractor when it detects the tag in localStorage to activate the capability of requesting to hijack the native API. The run _ at is a description attribute of a manifest file in the Chrome extension program, and is used for describing the operation opportunity of the content script.

In one example, the embodiment of the present application may use a recording tool (e.g., MarkUp) to record the interactive data, and store the recorded interactive data in an indexedDB of Chrome background. The Chrome background is a centralized program in the Chrome extended program, the program is isolated from other Chrome threads and other background programs, and the life cycle of the program is the same as that of the browser. The operation element in the Chrome Background can reach the browser level, namely has higher authority and can call the browser level API. IndexEDDB is a browser-provided local database that can be created and operated on by web scripts.

Taking the example of providing a MarkUp recording tool, MarkUp is a tool for recording by using Chrome Extension as a carrier, and the design idea of Chrome Extension needs to be followed in the architecture. Fig. 5d is a schematic diagram illustrating the technical architecture of MarkUp, which can be referred to. Referring to fig. 5d, when the user clicks the enabled plug-in, the Chrome background (browser backend) receives the request from the emulation front end and loads the content script named App render by the content script loader. App render will load the interactive components of MarkUp into the page of the emulated front end. In addition, the two content scripts of Request and Error Handler are set to run _ at state, document _ start, so when the page is loaded, the emulation front-end automatically loads the two content scripts. The Request content script is a main plug-in for data hijacking, and when the function of data hijacking is enabled, the plug-in hijacks a currently recorded interface and transmits data to a storage listener of the background script so as to store the data into an indexedDB in the background work space. The design mode avoids the problem that Chrome Extension does not share data in pages with different domain names, and the MarkUp realizes the modeling capability in a plurality of application programs. And the Error handler plug-in is used for handling exception conditions, for example, a MarkUp plug-in is enabled simultaneously in a plurality of pages or the account check fails, and if the MarkUp plug-in is determined not to pass, the Error handler writes an Error DOM (Document Object Model) in the client page to inform of Error information.

It should be further noted that, during the data collection process, a problem of interface duplication of the collected data may be encountered, and therefore, if no complete mechanism is provided for uniquely identifying the collected data, a situation that an erroneous data is requested in a subsequent simulation-based interaction process may occur. Based on this, in further some embodiments, the present application embodiments may provide a unique identifier for the interaction data collected during the data collection phase.

As an example, fig. 5e schematically shows a situation that results in the collected data not being unique, and different situations will be described below.

Cross-application interactive requests return different data using the same interface, which occurs in a cross-application scenario. For example, several applications may call the same interface, but different applications may return different data based on the same interface, and not identifying the data retrieved in this case may result in a situation where a subsequent simulation-based interactive process requests erroneous data.

The interactive requests of the same application program use the same interface to return different data, and the situation occurs in a scene that the interactive requests of the same application program use the same interface. An example of this is that the interface used for the interactive request is a stateful interface, such as when the data is requested for the first time, the data is in a loading state, and when the data is requested based on the same interface again, the data becomes in a loading successful state (success).

The same interface supports different interactive request methods at the same time. This occurs, for example, in the RestFul (restul is a design style and development method of web applications) API where the same interface has multiple method definitions, so that even if the URL (Uniform Resource Locator) of the interface is the same, different request methods result in different returned data results.

The application has a polling interface. This occurs in the context of a polling interface of an application, an example of which is where the application server requires processing time to return data, so the application server returns tickets to have the emulation front-end interrogate at intervals. the ticket mechanism means that some information which is not changed frequently is packaged and sent to the simulation front end, and then the simulation front end takes the information every time of requesting, so that the application program server can reduce the times of library checking or remote calling.

The query or body parameters in the interface are different. In this case, the query parameter or the body parameter in the same interface is different, which results in the query parameter or the body parameter in the interactive request (e.g. query request) being different, and thus results in the data returned by the interface being different.

cookies differ resulting in different data. In this case, the returned data results are different due to different cookies carried in the interactive request (e.g., request for identity verification).

Of course, the above-listed cases that result in the collected data not being unique are merely exemplary, and the embodiments of the present application do not exclude other cases that may result in the collected data not being unique.

In some embodiments, in order to make the data collected by the embodiment of the present application unique, the embodiment of the present application may split the interactive request into a plurality of request parameters, including any of the following items: url of request, query of request, Method of request, body of request, origin of request, cookies, data Type of request (Content-Type), and user defined key in header of request (head). Based on the Request parameters for splitting the interactive Request, the embodiment of the present application may form a unique identifier of the interactive data based on the Request parameters, an identifier of the simulation interaction model (for marking the simulation interaction model, such as script identifier), an ID of the interactive operation (Action ID for marking a specific user operation), and a polling Counter (Poll Request Counter). Fig. 5f shows an exemplary schematic representation of the formation of the unique identification of the interaction data, to which reference may be made.

In some embodiments, the generating of the unique identifier of the interaction data may be performed by the simulation server based on the unique identifier rule of the data. In the data acquisition stage of the embodiment of the application, after the interactive response data is hijacked by the simulation front end, the interactive request and the corresponding interactive response data can be sent to the simulation server. After the simulation server obtains the interaction request and the corresponding interaction response data, it may perform route analysis (e.g., perform route analysis by using a spring wrapper method). In an optional implementation of routing parsing, the embodiment of the present application may use generalized rule parsing, and only match rules, not entity paths; through the route analysis mode, the embodiment of the application can allow only the source (origin) of the message to be changed in the simulation process, and other contents are kept unchanged to realize the access to the simulation server. The spring wrapper parsing process distributes models according to model types, and sends the models to a model manager to obtain corresponding scene/body models, and the scene/body models are loaded into service models. According to the loaded model, the embodiment of the application can perform specific analysis on the interactive request to obtain the relevant contents such as the headers, the cookies and the like of the interactive request, and the whole process can be called as a generalization disassembly technology. Through the generalization disassembly technology, the embodiment of the application can more flexibly expand the interface and support the disassembly capability of diversified interaction requests.

After the interactive Request is disassembled, the simulation server may form a unique identifier of the interactive data based on the disassembled multiple Request parameters, the ID of the simulation interactive model, the ID of the interactive operation, and a polling Counter (Poll Counter).

In some embodiments, the ID of the interactive operation provided by the embodiments of the present application, and the unique identifier of the interactive data; according to the embodiment of the application, after the interactive data of the user and the application program is obtained in advance, the corresponding relation between the ID of the interactive operation and the unique identification of the interactive data can be set based on the ID of the interactive operation for triggering the obtaining of the interactive data and the unique identification of the interactive data, so that the association between the interactive operation and the interactive data obtained in advance is realized. For example, if a user executes a certain interactive operation provided by the simulation interactive model, based on the ID of the interactive operation, the interactive data associated with the interactive operation may be called according to the corresponding relationship between the ID of the interactive operation and the unique identifier of the interactive data in the embodiment of the present application. In some further embodiments, the interaction data obtained in advance, the correspondence between the ID of the interaction operation and the unique identifier of the interaction data may be stored in the simulation server, and when the user performs the simulation interaction using the interaction front end, if a certain interaction operation provided by the simulation interaction model is executed at the interaction front end, the interaction front end may indicate the ID of the interaction operation to the simulation server, so that the simulation server may feed back the interaction data corresponding to the unique identifier of the corresponding interaction data to the interaction front end, so as to implement invoking the interaction data associated with the interaction operation provided by the simulation interaction model.

In the embodiment of the application, a script information input stage is first entered before a data acquisition stage, and in the script information input stage, a user can fill basic information related to a simulation interaction model (such as a script), such as the script name, an application program used, an interaction scene of the user and the application program, and the like. After the basic information is filled in, the embodiment of the application enters a data acquisition stage. In the data acquisition stage, after a user clicks a recording start button, the simulation front end triggers page refreshing and injects the page refreshing into a capturer to hijack the native interface, so as to hijack the interactive data between the user and the application program in the recording environment and record the interactive data into the simulation server. Therefore, in the subsequent simulation modeling stage, the embodiment of the application can construct the simulation interaction model for user interaction based on the basic information input in the script information input stage and the interaction data recorded in the data acquisition stage.

In the simulation modeling phase 430, embodiments of the present application may build a simulation interaction model for user interaction. FIG. 6a schematically illustrates a sub-phase diagram of the simulation modeling phase 430. As shown in FIG. 6a, the implementation of the simulation modeling phase may include the following sub-phases: a create step sub-phase 431, a create sub-step sub-phase 432, a create operation sub-phase 433, an interface recording sub-phase 434, a model commissioning sub-phase 435, and a model storage sub-phase 436.

In a create step sub-stage 431, the present application embodiments may create steps under a scenario based on a simulated interaction model (e.g., scenario) including one or more steps. In some embodiments, the step may be an interaction scenario under a simulation interaction model, and the embodiment of the present application may create the step under the scenario based on the interaction scenario between the user and the application program, which is entered in the script information entry stage.

In create sub-step sub-stage 432, based on the steps comprising one or more sub-steps, embodiments of the present application may create sub-steps under the steps. In some embodiments, the sub-steps refer to segment descriptors in an interaction scenario under the simulated interaction model, which may include a run operation and/or a rollback operation.

In the create operation sub-phase 433, the embodiment of the present application may create each interactive operation included in the run operation and/or the rollback operation under the sub-phase, so that the simulation interactive model provides the interactive operation. In some embodiments, the interactive operation may be recorded based on the interactive operation corresponding to the interactive data recorded in the data acquisition stage.

Since the user still involves calling the native interface in the interaction process with the application program, in the interface recording sub-stage 434, the embodiment of the present application can record the native interface called by the interaction operation. It should be noted that, in the interface recording sub-stage, the native interface is obtained by recording, and does not involve the call of the subsequent real native interface.

The creating step sub-stage, the creating operation sub-stage and the interface recording sub-stage can build the simulation interaction model layer by layer from the coarse granularity (step) to the minimum granularity (interaction operation) of the simulation interaction model, and can realize the direct calling of the interaction data acquired in the data acquisition stage through the interaction operation of the minimum granularity. FIG. 6b is a schematic diagram illustrating the components of the simulated interaction model, which may be referred to. As shown in fig. 6b, the simulation interaction model (i.e. scenario) may include a plurality of steps, one step may correspond to an interaction scenario of the user and the application, and one step may represent an interaction of a function point of the application, such as an interaction of introducing a form usage manner, an interaction of explaining how to perform an application function opening, and the like; a step may comprise a plurality of sub-steps, one sub-step for example introducing a form interaction, an interaction describing the definition of the opening function, etc., and one sub-step may comprise an ordered set of interaction operations, and the ordered set of interaction operations may form a run operation and a rollback operation; the operation of running refers to a set of interactive operations required for completing a sub-step, such as interactive operations related to selecting a form, expanding the form, highlighting the form, inputting example values and the like when introducing the interaction of the form; the rollback operation represents the backtracking interactive operation to support the interactive error or the previous step of interactive operation; it can be seen that sub-steps may include a plurality of interactions, each interaction being associated with interaction data collected during a data collection phase.

It should be noted that the composition of the above simulation interaction model is only an example, and the simulation interaction model may also be changed according to actual interaction requirements, such as a series connection of a scenario and a scenario to form a theater, and an subdivision of interaction operations into operation sources, besides the breakdown of the granularity of the scenario, steps, sub-steps, interaction operations, and the like.

After the simulation interaction model is built, in the model commissioning sub-phase 435, the embodiment of the present application may commission the simulation interaction model to check whether the built simulation interaction model meets expectations, and if not, the modification and editing (for example, interaction operation provided by re-creating the simulation interaction model) may be continued on the simulation interaction model. After the simulated interaction model is in anticipation, at model store sub-stage 436, embodiments of the present application may store the simulated interaction model in a local database. For example, the simulated interaction model is stored to indexedDB in the chrome background.

In the version establishing stage 440, in order to avoid the dilemma that the constructed simulation interaction model follows fast iteration due to frequent iteration of the online environment, the embodiment of the present application provides a model version management technology to solve the constraint of service stability.

In some embodiments, in a case that a simulation interaction page corresponding to a simulation interaction model is provided to implement simulation interaction of a user, the embodiment of the present application may establish a version of the simulation interaction model by using a page snapshot technique. As an alternative implementation, FIG. 6c is a diagram illustrating an example implementation of a page snapshot technique. As shown in connection with FIG. 6c, the page snapshot technique can be divided into a tool-side portion and a back-side portion. In the tool end part, the embodiment of the present application may request the current domain name through the Fetch method to obtain the original HTML text. Original HTML text here refers to the content of the page before the page has not been processed by document. In some embodiments, the whole simulation page (Web page) can be regarded as a black box in the simulation process, so that it is necessary to obtain the original HTML text. Through the document objects, the embodiments of the present application can obtain all the loaded Scripts, which may include some non-file Type loaded Scripts, and therefore, filtering needs to be performed through a Filter (Filter Remote Type Scripts) to obtain a true JS file path. Links objects can be accessed to all loaded links tags, including links and CSS (Cascading Style Sheets) resources. By using a Filter (Filter CSS links), the embodiments of the present application may Filter out the true CSS resource paths. Finally, through the labels such as video, img, audio and the like, the resource path of the picture, the sound and the video can be obtained.

After all the required paths and contents are obtained, the emulation front-end can send the paths and contents to the emulation server. The emulation server can request Files (Fetch Files) according to a file path. Since the JS and the CSS may include path information, it is necessary to scan the content and replace the path information with the emulated server path. Finally, the embodiment of the present application may store all files in an OSS (Object Storage Service) and wait for an interactive procedure to call.

After the version establishment is completed, the interactive data and the simulation interactive model stored in the indexedDB can be uploaded to the simulation server by the embodiment of the application.

In the process of simulation modeling, there is also a relatively difficult problem of how to capture elements in a page of an application. Because elements in a real environment tend to be nested layer by layer, it is quite difficult to accurately determine the elements that trigger an interaction request without perceiving the entire underlying practice. Based on this, the embodiment of the application provides a set of design schemes for element capture and display. FIG. 6d shows an element capture and presentation diagram of a page. The element capture and presentation may be implemented by an emulation front-end execution.

In some embodiments, after the simulation front end loads the recording tool, the embodiments of the present application may select an element in the recording tool. In an alternative implementation, the simulation front end may obtain the current coordinates of the cursor by listening for cursor (e.g., mouse) events. Based on the obtained current coordinate and a coordinate range (elementFromPoint) corresponding to an element in the browser native interface, an element corresponding to the current coordinate may be determined in the embodiment of the present application.

After determining the element corresponding to the current coordinate, the embodiment of the application may generate and store a path (path) corresponding to the element. In some embodiments, to avoid the ID of an element being generated by a random number in some special cases, the embodiments of the present application may turn off the function of generating a path based on the ID of the element, and instead generate a path based on the class (type) of the element, the element name. In addition, in terms of threshold, since the officially proposed threshold is 1000, in order to avoid the path lengthening caused by too much nth-child, the embodiment of the present application may change threshold to 1, mainly because nth-child is used to avoid misjudgment, for example, in table list, ul and other elements, the use of nth-child is more secure than the use of element name or class. Therefore, in alternative implementations, the embodiments of the present application may generate the path corresponding to an element based on the class and the element name of the element, or based on the nth-child of the element. nth-child is an element selector for matching child elements in a parent element.

After determining the path of the element, the embodiment of the application may upload the path of the element to the simulation server for subsequent invocation. In some embodiments that invoke elements, the front end may reverse the elements according to a path based on the elements fed back by the emulation server and fetch the element instances. After the element instance is obtained, the front end cannot directly use the element instance to operate the interactive event, because there is no way to ensure that the selected element is necessarily the element that is to trigger the interactive operation, for example, the element that actually triggers the interactive operation is a button element, but when the element is selected, a parent container div having the same width, height and position as the button may be selected. There are many solutions to this problem, for example, the real button can be found through drilling down and recorded, or the button can be triggered by the bubbling mechanism of the browser after drilling down to the bottom layer, but these methods are risky, and the really riskless method is to simulate the user behavior to trigger the click event. Therefore, in the embodiment of the application, after the element is acquired, the boundary of the element is acquired through the getBoundingClientRect API, the center point position of the element is acquired through the boundary, and finally, a mouse click event is simulated on the center point position to achieve an action of simulating click. Wherein getbackingclientrect can be used to obtain a set of positions of an element relative to a window.

In some embodiments, embodiments of the present application may generate a path for an element based on an open source library. An open source library such as a finder, which is a library that is dedicated to converting elements into paths.

In an optional application example, the simulated interaction model constructed by the embodiment of the application can be used for a cloud computing platform to enable a user to realize an immersive interaction experience in a simulated environment through the capability of the cloud computing platform.

In an alternative example, when an internet service provider needs to promote, introduce a new application, or a new function of an application, the operator of the internet service provider may open a build page for building a simulated interaction model using the simulation front-end. The operator can input the script basic information of the simulation interaction model on the construction page based on the new application program function or the new application program function, and then click the recording button, so that when the operator uses the new application program or the new application program function, the simulation front end can record the real interaction data of the operator. In one example, FIG. 6e illustrates a diagram of a build page that introduces how the charging function of an application is enabled; for example, when the charging function is newly activated by the application program, and the internet service provider needs to publicize and introduce how to activate the charging function to the user, the operator of the internet service provider can open the corresponding construction page by using the simulation front end, and then click the recording button, so that the simulation front end can record the real interactive data of the operator in the interactive process of activating the charging function by hijacking the native API in the process of activating the charging function by the operator. Furthermore, in the next operation of building the page, the operator may create a step, a sub-step, a running operation and/or a rollback operation of the simulation interaction model, and an interaction operation under the running operation and/or the rollback operation, and associate the interaction operation in the simulation interaction model with the recorded interaction data to implement building the simulation interaction model. Furthermore, when the internet service provider invites the user to use a new application program or a new function of the application program, the user can acquire the simulation interaction page corresponding to the simulation interaction model by using the interaction front end, perform interaction operation based on guidance under the simulation interaction page, acquire interaction data without calling an interface, and realize interaction guidance when the user uses the new application program or the new function of the application program.

In some embodiments, the internet service provider referred to above may be a big data service provider, a cloud computing service provider. The big data service provider can provide services of a plurality of application programs for a user through the big data platform, namely the service provision of the application programs can be realized based on the big data platform, and the simulation interaction model constructed in the embodiment of the application can be used universally in interaction guidance of the application programs of the big data platform. For example, a simulated interaction model may be made up of a plurality of scripts, one of which is considered to be an application. In other embodiments, the internet provider referred to above may also be a service provider for a single application. In an application example, the application program referred to in the embodiments of the present application may be an application program for a B-side (enterprise-side), for example, the big data platform may be a big data platform for providing services for the B-side, that is, the big data platform may provide a plurality of application program products for providing services for the B-side.

In some embodiments, the application embodiments may support providing interactive guidance for new user experience applications and may also support providing interactive guidance for new functionality of old user experience applications. In an alternative implementation, since the simulation interaction model is constructed by a technician at an internet service provider based on an interaction process with an application program, the sources of interaction data used for guiding the interaction between a new user and an old user may be the same, for example, all based on data recording of the technician at the internet service provider.

The construction scheme of the simulation interaction model provided by the embodiment of the application takes data acquisition, simulation modeling and version establishment as key stages; in the data acquisition stage, the interactive data of the real interactive environment is acquired and stored in the simulation server, so that the problems of interface safety, interface authentication and the like caused by calling a real interface possibly in the immersive interactive experience process are solved, and therefore, the marginal effect cannot be generated. Furthermore, the simulation modeling process can be performed in a visual mode, the modeling tool is loaded in the page through the Chrome Content Script and is recorded, and the modeling cost is greatly reduced. Furthermore, the simulation modeling can be carried out on the basis of the Chrome extension program, so that the simulation modeling can support the realization of the browser end, the interactive process is considered in the design of the modeling mode, the model is disassembled into the script, the steps, the sub-steps and the interactive operation, and the interactive capability under the simulation environment is achieved by the aid of the element capturing scheme. Further, the embodiment of the application provides a version management scheme of the simulation interaction model, and the problem that the simulation interaction model must be iterated together due to fast iteration of the application program in the immersive interaction experience is solved.

The following describes an interaction method provided in an embodiment of the present application. The interaction method described below can be implemented based on the above-constructed simulation interaction model. As an alternative implementation, fig. 7a shows an alternative flow of the interaction method provided in the embodiment of the present application. The interaction method can be implemented by an interaction front end. The interaction front-end may be considered a client for user interaction. As shown in fig. 7a, the process may include the following steps.

In step S710, a simulation interaction page corresponding to the simulation interaction model is displayed.

In step S711, a current interactive operation is detected on the simulation interactive page, where the current interactive operation is provided by a simulation interactive model.

In step S712, the interactive data associated with the current interactive operation is called, and the corresponding interactive effect is displayed on the simulation interactive page, where the interactive data is the interactive data of the user and the application program, which is acquired in advance.

Based on the interaction method provided by the embodiment of the application, the interaction front end can perform interaction guidance when the user uses the application program. Firstly, the interactive front end can obtain a simulation interactive page corresponding to the simulation interactive model from the simulation server. In some embodiments, the simulation interaction page may expose objects (e.g., elements) for performing interaction operations, and interaction operations that may be performed in the simulation interaction page are provided by the simulation interaction model and associated with pre-acquired interaction data stored in the simulation server; therefore, when the interactive front end detects the current interactive operation executed by the simulation interactive page, the interactive front end can call the pre-acquired interactive data corresponding to the current interactive operation (for example, call the pre-acquired interactive request and the corresponding interactive response data corresponding to the current interactive operation) from the simulation server, and display the corresponding interactive effect on the simulation interactive page, so as to realize the response and effect display of the current interactive operation executed by the simulation interactive page.

In some embodiments, if the simulation interaction model (e.g., scenario) includes a plurality of steps, and each step includes a plurality of sub-steps, each sub-step may include a run operation and a rollback operation, and the run operation and the rollback operation respectively include a plurality of interaction operations; the running operation may introduce or guide the function of the application through multiple interactive operations when performing interactive guidance, for example, when guiding the user to use the form of the application, the interactive front end may guide the user to perform interactive operations such as selecting the form, expanding the form, highlighting the form, inputting example values, and the like through multiple interactive operations included in the running operation of the introduction form. The rollback operation may occur when an error occurs in the interactive operation of the user, and the rollback operation may backtrack the interactive operation in which the interactive error occurs or a previous interactive operation through a plurality of interactive operations. In some embodiments, the rollback operation is different from the run operation, and the interaction operations included in the rollback operation can be separately defined; because the simulation interaction is not like playing a movie, the simulation interaction has no reversibility and cannot be traced back using a running operation. For example, when a query form is interacted with, its backtracking may be to empty the form and empty the query terms, which is information that is not available from the interaction of the query form.

In some embodiments, the interaction front-end may prompt the user for a current interaction to be performed or automatically perform the current interaction. With reference to the interaction implementation diagram shown in fig. 7b, when the interaction front end prompts the user that the current interaction operation needs to be executed or automatically executes the current interaction operation, the current interaction operation may be loaded by the interaction front end; the interactive front-end may then enter an initialization state to prepare the execution environment for performing the current interactive operation, such as modifying the ID of the current interactive operation onto the url column, flushing the cache of the previous interactive operation, waiting for the object of the next interactive operation to appear, and so on. Then, the interaction front end may highlight, on the simulation interaction page, an object corresponding to the current interaction operation (for example, highlight an element related to the current interaction operation), and bind a corresponding response listener to listen to a state of completion of the response event. After the current interactive operation is automatically executed or manually executed by a user, a response event is triggered, and the interactive front end can fetch corresponding interactive data and perform effect display even if the current interactive operation is finished. Then, the interactive front end can unbind the corresponding snoop response event to complete the detection snoop. Furthermore, the interactive front end can also be bound with a timer to time the completion time of the current interactive operation so as to support the function of finishing the interactive operation at fixed time. In the above process, if any execution fails, the interactive front end may enter a rollback operation link to perform operation backtracking.

It should be noted that the embodiment of the present application may determine whether the current interactive operation is actively triggered by the interactive front end or waits for manual triggering by the user based on the configuration.

In some embodiments, the embodiment of the present application may also support charging for the user to use the interactive guidance, for example, when an enterprise at the B-end uses a certain application or an application provided by a big data platform, the embodiment of the present application may provide the interactive guidance of the application, and provide charging information required by the interactive guidance on a simulated interactive page where the interactive guidance is performed.

In some embodiments, different databases or data modules may be set based on different service types, and interactive data acquired in advance under one service type may be stored in a database corresponding to the service type; for example, after the interaction data of the user and the application program is acquired in advance, the acquired interaction data may be stored in a database corresponding to the service type of the application program. Therefore, after a simulation interaction model corresponding to a certain service type is constructed, aiming at user interaction guidance under the service type, the embodiment of the application can provide interaction operation under the service type through the simulation interaction model corresponding to the service type, and the interaction operation under the service type is associated with interaction data stored in a database corresponding to the service type. Furthermore, after a user performs a certain interactive operation based on the simulation interaction model corresponding to the service type, the embodiment of the application may call interactive data associated with the interactive operation from the database corresponding to the service type, so as to implement user interaction guidance under the service type. For example, the simulation interaction model provides interaction operations for: and calling the interaction data acquired in advance from a database corresponding to the service type of the application program. Through the setting, the embodiment of the application can set different databases for different service types to store the interaction data acquired in advance under different service types, so that when a user performs interaction operation of a certain service type, the embodiment of the application can call the associated interaction data acquired in advance from the corresponding database to support interaction guidance of users of different service types.

According to the method and the device, after the interactive operation is executed on the simulation interactive page, the interactive data corresponding to the interactive operation is not acquired through a real interactive interface, but is called from the simulation server. Therefore, in the process of realizing user interaction based on the simulation interaction model, the realization cost of immersive interaction experience can be reduced, and the method and the device can be commonly used among a plurality of application programs.

While various embodiments have been described above in connection with what are presently considered to be the embodiments of the disclosure, the various alternatives described in the various embodiments can be readily combined and cross-referenced without conflict to extend the variety of possible embodiments that can be considered to be the disclosed and disclosed embodiments of the disclosure.

In the following, the simulation interaction model building apparatus provided in the embodiment of the present application is introduced, and the apparatus content described below may be regarded as a model building device (the model building device may be a simulation front end or a simulation server), and is a functional module that is required to implement the simulation interaction model building method provided in the embodiment of the present application. The device content described below may be referred to in correspondence with the above description.

As an alternative implementation, fig. 8a shows an alternative block diagram of a simulation interaction model building apparatus provided in an embodiment of the present application. The simulation interaction model building device can be applied to model building equipment. The model building device may be a simulation front end or a simulation server, and may be determined according to technical configuration. As shown in fig. 8a, the apparatus may include:

a data obtaining module 810, configured to obtain interaction data between a user and an application in advance;

a model construction module 811 for constructing a simulation interaction model based on at least the interaction data, wherein the simulation interaction model provides interaction operations associated with pre-acquired interaction data such that the interaction operations provided by the simulation interaction model are used to invoke the pre-acquired interaction data;

and a page generating module 812, configured to generate a simulation interaction page for user interaction corresponding to the simulation interaction model.

In some embodiments, the simulated interaction model is a transcript, the transcript comprising a step comprising a sub-step comprising a run operation and/or a rollback operation, the run operation and/or the rollback operation comprising a plurality of interaction operations; the running operation introduces or guides the application program to use through a plurality of interactive operations, and the rollback operation backtracks the interactive operation with an interactive error or the previous interactive operation through the plurality of interactive operations.

In some embodiments, model building module 811 for building a simulated interaction model based at least on the interaction data comprises:

creating a script based on the interactive scene of the user and the application program;

a sub-step under the step of creating, said sub-step comprising a run operation and/or a rollback operation;

a plurality of interactive operations under run operations and/or rollback operations are created such that the simulated interaction model provides interactive operations.

In some embodiments, the interaction data comprises an interaction request and corresponding interaction response data; the interactive operation provided by the simulation interactive model is used for calling the pre-acquired interactive data and comprises the following steps:

and the interactive operation provided by the simulation interactive model is used for calling the interactive request and the corresponding interactive response data which are acquired in advance.

In some further embodiments, the apparatus may further include a function module configured to:

acquiring script basic information of a simulation interaction model, wherein the script basic information comprises: name of the simulation interaction model, application program used for interaction and interaction scene of the application program.

In some embodiments, the model building module 811 for creating a scenario based on a user's interaction scenario with an application includes:

and creating a script based on the interactive scenes in the script basic information.

In some embodiments, the data obtaining module 810, configured to obtain the interaction data of the user and the application in advance, includes:

entering a recording environment, and triggering and refreshing a page where a user interacts with an application program;

hijacking a native interface used for an interactive request before reloading the page;

and sending an interaction request based on the hijacked native interface, and capturing interaction response data corresponding to the interaction request.

In some embodiments, the data acquisition module 810 for hijacking a native interface for the interactive request before the page reload comprises:

before the page reloads, writing a tag in a local storage of the emulation front-end to inform the emulation front-end that an injector needs to be started to inject the extractor into the native interface;

loading the injector upon a page reload; the injector initiates a retriever based on the tag detected in the local storage to inject the retriever into the native interface.

In some embodiments, the apparatus may further be configured to:

decomposing an interactive request in interactive data into a plurality of request parameters; generating a unique identifier of the interactive data based on the plurality of request parameters, the identifier of the simulation interactive model and the ID of the interactive operation corresponding to the interactive request;

and/or, acquiring the current coordinate of the cursor by monitoring the cursor event; determining an element corresponding to the current coordinate based on the current coordinate and a coordinate range corresponding to the element; generating a path corresponding to the element based on the type and the element name of the element or based on an element selector of the element;

and/or storing the interaction data to a local database;

and/or, the simulation interaction model is operated in a test mode, and after the simulation interaction model is successfully operated in the test mode, the simulation interaction model is stored in a local database;

and/or uploading the interaction data in the local database and the simulation interaction model to a simulation server;

and/or establishing a version of the simulation interaction model based on the page snapshot technology.

The functions shown above may be deployed in one functional module or in multiple functional modules respectively, and the functions of the multiple functional modules may be crossed.

In some embodiments, the application is provided by a big data platform, and the big data platform provides a plurality of applications; the simulation interaction model supports interaction guidance of a plurality of application programs, one application program corresponds to one script of the simulation interaction model, the simulation interaction model comprises a plurality of scripts, and one script comprises one or more steps.

The embodiment of the application also provides a model building device, which can be a simulation front end or a simulation server. The model construction equipment can be loaded with the simulation interaction model construction device to realize the simulation interaction model construction method provided by the embodiment of the application. In some embodiments, fig. 8b illustrates an alternative block diagram of a model building apparatus provided by embodiments of the present application. As shown in fig. 8b, the model building apparatus may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete mutual communication through the communication bus 4.

Alternatively, the communication interface 2 may be an interface of a communication module for performing network communication.

Alternatively, the processor 1 may be a CPU (central Processing Unit), a GPU (Graphics Processing Unit), an NPU (embedded neural network processor), an FPGA (Field Programmable Gate Array), a TPU (tensor Processing Unit), an AI chip, an asic (application Specific Integrated circuit), or one or more Integrated circuits configured to implement the embodiments of the present application.

The memory 3 may comprise a high-speed RAM memory and may also comprise a non-volatile memory, such as at least one disk memory.

The memory 3 stores one or more computer-executable instructions, and the processor 1 calls the one or more computer-executable instructions to execute the simulation interaction model building method provided by the embodiment of the application.

The embodiment of the present application further provides a storage medium, where the storage medium may store one or more computer-executable instructions, and when the one or more computer-executable instructions are executed, the method for building the simulation interaction model provided in the embodiment of the present application may be implemented.

In the following, the interaction device provided in the embodiment of the present application is introduced, and the device content described below may be regarded as a functional module that is required to be arranged by the interaction front end to implement the interaction method provided in the embodiment of the present application. The device content described below may be referred to in correspondence with the above description.

As an alternative implementation, fig. 8c shows an alternative block diagram of the interaction apparatus provided in the embodiment of the present application. As shown in fig. 8c, the apparatus may include:

a page display module 820, configured to display a simulation interaction page corresponding to the simulation interaction model;

an operation detection module 821, configured to detect a current interaction operation on the simulation interaction page, where the current interaction operation is provided by a simulation interaction model;

and a data retrieving and displaying module 822, configured to invoke interactive data associated with the current interactive operation, and display a corresponding interactive effect on the simulation interactive page, where the interactive data is interaction data of a user and an application program, which is acquired in advance.

In some embodiments, the simulated interaction model is a transcript, the transcript comprising a step comprising a sub-step comprising a run operation and/or a rollback operation, the run operation and/or the rollback operation comprising a plurality of interaction operations; the running operation introduces or guides the application program to use through a plurality of interactive operations, and the rollback operation backtracks the interactive operation with an interactive error or the previous interactive operation through the plurality of interactive operations; the current interactive operation detected on the simulation interactive page is an interactive operation under running operation or rollback operation;

before the operation detection module detects the current interactive operation on the simulation interactive page, the device can further set a function module to be further used for:

loading the current interactive operation and entering an initialization state to prepare an execution environment for executing the current interactive operation;

on the simulation interaction page, highlighting an object corresponding to the current interaction operation, and binding a corresponding response monitor to monitor the state of completion of a response event;

wherein the current interaction operation may be performed automatically or manually by a user.

The embodiment of the present application further provides an interactive front end, where the interactive front end may implement the interactive method provided by the embodiment of the present application by loading the interactive device described above. An alternative configuration of the interactive front-end may be seen in connection with fig. 8b, including: the interactive device comprises at least one memory and at least one processor, wherein the memory stores one or more computer-executable instructions, and the processor calls the one or more computer-executable instructions to execute the interactive method provided by the embodiment of the application.

Embodiments of the present application also provide a storage medium, where the storage medium may store one or more computer-executable instructions, and when executed, the one or more computer-executable instructions implement the interaction method provided in the embodiments of the present application.

Embodiments of the present application further provide a program, where the program is executed by an electronic device (for example, a model building device or an interaction front end), to implement the simulation interaction model building method provided in the embodiments of the present application, or the interaction method provided in the embodiments of the present application.

Although the embodiments of the present application are disclosed above, the present application is not limited thereto. Various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present disclosure, and it is intended that the scope of the present disclosure be defined by the appended claims.

Claims

1. A simulation interaction model building method, wherein the method comprises the following steps:

acquiring interactive data of a user and an application program in advance;

2. The method of claim 1, wherein the simulation interaction model comprises a step comprising a sub-step comprising a run operation and/or a rollback operation comprising a plurality of interaction operations; the running operation introduces or guides the application program to use through a plurality of interactive operations, and the rollback operation backtracks the interactive operation with an interactive error or the previous interactive operation through the plurality of interactive operations;

the building a simulated interaction model based at least on the interaction data comprises:

3. The method of claim 1, wherein the interaction data comprises an interaction request and corresponding interaction response data; the interactive operation provided by the simulation interactive model is used for calling the pre-acquired interactive data and comprises the following steps:

the interactive operation provided by the simulation interactive model is used for calling the interactive request and the corresponding interactive response data which are acquired in advance;

the method further comprises the following steps:

acquiring script basic information of a simulation interaction model, wherein the script basic information comprises: simulating the name of an interactive model, an application program used interactively and an interactive scene of the application program;

the method for creating the script based on the interactive scenes of the user and the application program comprises the following steps:

4. The method of claim 1, wherein the pre-acquiring user interaction data with the application comprises:

5. The method of claim 4, wherein hijacking a native interface for interactive requests prior to page reload comprises:

6. The method of claim 1, wherein the method further comprises:

and/or storing the interaction data to a local database;

7. The method of claim 2, wherein the application is provided by a big data platform, and the big data platform provides a plurality of applications; the simulation interaction model supports interaction guidance of a plurality of application programs, one application program corresponds to one script of the simulation interaction model, the simulation interaction model comprises a plurality of scripts, and one script comprises one or more steps.

8. An interaction method, wherein the method comprises:

9. The method of claim 8, wherein the simulation interaction model comprises a step comprising a sub-step comprising a run operation and/or a rollback operation comprising a plurality of interaction operations; the running operation introduces or guides the application program to use through a plurality of interactive operations, and the rollback operation backtracks the interactive operation with an interactive error or the previous interactive operation through the plurality of interactive operations; the current interactive operation detected on the simulation interactive page is an interactive operation under running operation or rollback operation;

before the simulation interaction page detects a current interaction operation, the method further comprises:

highlighting an object corresponding to the current interactive operation on the simulation interactive page, and binding a corresponding response monitor to monitor the state of completion of a response event;

10. A model building apparatus, comprising: at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform the simulated interaction model building method of any of claims 1-7.

11. An interactive front-end, comprising: at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform the interaction method of any of claims 8-9.

12. A storage medium, wherein the storage medium stores one or more computer-executable instructions that, when executed, implement the method of modeling simulation interactions according to any of claims 1-7, or the method of interacting according to any of claims 8-9.

13. A program, wherein the program, when executed by an electronic device, implements a method of constructing a simulated interaction model as claimed in any one of claims 1 to 7 or an interaction method as claimed in any one of claims 8 to 9.