CN117193751A

CN117193751A - Service code generation method, device, equipment and storage medium

Info

Publication number: CN117193751A
Application number: CN202310989238.0A
Authority: CN
Inventors: 郑荣福; 李文涛; 朱世涛; 罗一风
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2023-08-07
Filing date: 2023-08-07
Publication date: 2023-12-08

Abstract

The application discloses a service code generation method, a device, equipment and a storage medium, which are used for automatically generating corresponding service codes according to requirements. The method acquires the multi-mode information of the target service; decomposing the multi-mode information by adopting the multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model; for each business sub-flow: inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process; and merging the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service. The service codes corresponding to the service are generated through the multi-mode information based on the service, so that the generated service codes are more accurate, the automation of service code generation is realized through adopting the multi-mode basic model and the code generation model, no manual intervention is needed, the use threshold is reduced, the development period is shortened, and the method is easy to realize and low in cost.

Description

Service code generation method, device, equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a service code generating method, device, equipment, and storage medium.

Background

A wide variety of business systems are used in business day-to-day offices, and people often need to communicate information between the various systems, including: text copying, uploading and downloading files and the like; however, these systems may be provided or developed by different manufacturers at different times or belong to different units, so that no system interface channel between the systems can complete automatic transfer of information; but rather require manual effort to effect information transfer between the systems. In the related art, robot process automation (Robotic process automation, RPA) is adopted to realize interaction between systems, but the method needs participation of service operators and process developers, has high use threshold and long development period.

Disclosure of Invention

In view of the above, the present application provides a service code generating method, apparatus, device and storage medium, so as to facilitate automatic generation of corresponding service codes according to requirements without manual intervention.

In a first aspect, an embodiment of the present application provides a service code generating method, where the method includes:

acquiring multi-mode information of a target service;

decomposing the multi-mode information by adopting a multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model;

For each business sub-flow: inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process;

and merging the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service.

In the application, the service codes corresponding to the services are generated through the multi-mode information based on the services, so that the generated service codes are more accurate, and the automation of the service code generation is realized through adopting the multi-mode basic model and the code generation model, so that the manual intervention is not needed, the use threshold is reduced, the development period is shortened, and the method is easy to realize and has low cost.

In some possible embodiments, the obtaining the multi-modal information of the target service includes:

responding to a code generation instruction triggered by a user, and determining that the code generation instruction corresponds to a target service;

and acquiring multi-mode information corresponding to the target service from a memory based on the target service.

In some possible embodiments, after the determining that the code generation instruction corresponds to a target service, the method further includes:

if the multi-mode information corresponding to the target service is not obtained in the memory based on the target service, generating a multi-mode information configuration prompt; the multi-mode information configuration prompt is used for prompting a technician to configure multi-mode information corresponding to the target service;

And outputting and displaying the multi-mode information configuration prompt.

In some possible embodiments, before the decomposing operation on the multimodal information using the multimodal base model, the method further includes:

the method comprises the steps of obtaining multi-modal information of a training service, wherein the multi-modal information comprises an image sequence, a voice sequence, a mouse input sequence and a keyboard input sequence corresponding to the training service;

constructing a first training sample set by adopting the multi-mode information of the training service;

inputting a first training sample in the first training sample set into an initial multi-modal basic model, training the initial multi-modal basic model in an iterative mode until a preset convergence condition is met, and taking the initial multi-modal basic model with the iteration ended as the multi-modal basic model.

In some possible embodiments, each round of iterative process of the initial multi-modal base model is as follows:

inputting the first training samples in the first training sample set into the initial multi-mode basic model to obtain a business sub-process output by the initial multi-mode basic model;

determining a loss value of the initial multi-mode basic model according to a preset business sub-process of the first training sample and a business sub-process output by the initial multi-mode basic model;

And adjusting model parameters of the initial multi-mode basic model according to the loss value of the initial multi-mode basic model.

In some possible embodiments, before the inputting the business sub-process into the pre-trained code generation model, the method further includes:

acquiring service sub-flows of a plurality of training services;

constructing a second training sample set by adopting the service sub-flow of the training service;

inputting a second training sample in the second training sample set into an initial code generation model, training the initial code generation model in an iterative mode until a preset convergence condition is met, and taking the initial code generation model with the iteration end as the code generation model.

In some possible embodiments, each iteration of the initial code generation model proceeds as follows:

inputting the second training samples in the second training sample set into the initial code generation model to obtain service codes output by the initial code generation model;

determining a loss value of the initial code generation model according to a preset service code of the second training sample and a service sub-process output by the initial code generation model;

And adjusting model parameters of the initial code generation model according to the loss value of the initial code generation model.

In some possible embodiments, after the obtaining the service code corresponding to the service sub-flow, the method further includes:

performing accuracy judgment on the service codes;

if the service code is determined to have errors, generating a corresponding error prompt according to the error type of the service code;

and outputting and displaying the error prompt.

In a second aspect, an embodiment of the present application provides a service code generating apparatus, where the apparatus includes:

the acquisition module is used for acquiring the multi-mode information of the target service;

the decomposition module is used for decomposing the multi-mode information by adopting a multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model;

a code generating module, configured to, for each service sub-flow: inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process;

and the merging module is used for merging the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service.

In some possible embodiments, when the acquiring module executes the multi-mode information of the target service, the acquiring module is specifically configured to:

In some possible embodiments, after the acquiring module executes the determining that the code generation instruction corresponds to the target service, the acquiring module is further configured to:

and outputting and displaying the multi-mode information configuration prompt.

In some possible embodiments, before the decomposition module performs the decomposition operation on the multimodal information using the multimodal base model, the decomposition module is further configured to:

In some possible embodiments, before the code generation module performs inputting the business sub-process into a pre-trained code generation model, the code generation module is further configured to:

Acquiring service sub-flows of a plurality of training services;

In some possible embodiments, after the code generating module executes the service code corresponding to the service sub-flow, the code generating module is further configured to:

Performing accuracy judgment on the service codes;

and outputting and displaying the error prompt.

In a third aspect, another embodiment of the present application also provides an electronic device, including at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the methods provided by the embodiments of the first aspect of the present application.

In a fourth aspect, another embodiment of the present application further provides a computer readable storage medium, where the computer readable storage medium stores a computer program for causing a computer to perform any one of the methods provided by the embodiments of the first aspect of the present application.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of a first related art of a service code generating method according to an embodiment of the present application;

fig. 2 is a schematic diagram of a second related art of a service code generating method according to an embodiment of the present application;

fig. 3 is a schematic diagram of a third related technology of a service code generating method according to an embodiment of the present application;

fig. 4 is an application scenario schematic diagram of a service code generating method according to an embodiment of the present application;

fig. 5 is an overall flow diagram of a service code generating method according to an embodiment of the present application;

fig. 6 is a schematic diagram of multi-mode information of a service code generating method according to an embodiment of the present application;

fig. 7 is a schematic diagram of a multi-mode information configuration prompting flow of a service code generating method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a multi-modal basic model training process of a business code generation method according to an embodiment of the present application;

FIG. 9 is a schematic diagram of an iteration process of each round of an initial multi-modal base model of a service code generation method according to an embodiment of the present application;

fig. 10 is a schematic diagram of a code generation model training flow of a service code generation method according to an embodiment of the present application;

FIG. 11 is a schematic diagram of an iterative process of each round of an initial code generation model of a service code generation method according to an embodiment of the present application;

fig. 12 is a schematic diagram of a service code accuracy determining flow of a service code generating method according to an embodiment of the present application;

fig. 13 is another schematic diagram of an overall flow of a service code generating method according to an embodiment of the present application;

fig. 14 is a schematic diagram of a service code generating system frame of a service code generating method according to an embodiment of the present application;

fig. 15 is a schematic diagram of a central architecture of a service code generating method according to an embodiment of the present application;

fig. 16 is a schematic diagram of a device for generating a service code according to an embodiment of the present application;

fig. 17 is a schematic diagram of an electronic device of a service code generating method according to an embodiment of the present application.

Detailed Description

For a better understanding of the technical solution of the present application, the following detailed description of the embodiments of the present application refers to the accompanying drawings.

It should be understood that the described embodiments are merely some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terminology used in the embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be understood that the term "and/or" as used herein is merely one way of describing an association of associated objects, meaning that there may be three relationships, e.g., a and/or b, which may represent: the first and second cases exist separately, and the first and second cases exist separately. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

The inventor researches and discovers that various business systems are used in daily offices of enterprises, and people generally need to transfer information among the systems, including: text copying, uploading and downloading files and the like; however, these systems may be provided or developed by different manufacturers at different times or belong to different units, so that no system interface channel between the systems can complete automatic transfer of information; instead, the mouse and the keyboard are needed to be used manually, and information transfer between the systems is realized through operations such as inputting, copying, pasting, uploading, downloading and the like. This process is often a highly repeatable, low added value workflow. RPA is employed in the related art to solve this problem; RPA essentially replaces a person with a software robot to perform mouse and keyboard operations, thereby enabling automation of cross-system information transfer.

RPAs are generally classified into attended RPAs and unattended RPAs. The attended RPA means that part of the business processes are automated by using the RPA technology, and the other part of the business processes are limited by the technical level or the business standardization level and cannot be automated; therefore, the RPA procedure is completed and the system operation authority is handed over to the service personnel. The unattended RPA means that the whole business process is completely executed by the RPA without manual intervention; the RPA procedure execution is triggered by a condition and the RPA procedure execution status is monitored by a monitor.

The flow of RPA in the related art can be divided into four steps:

A. description of the flow: the description clearly requires the implementation of automated business operational flow steps including the objects of the operations, the operations performed, the timing of the operations. The description of the flow may be output in the form of a document such as Word processor (Word), form processor (Excel), extensible markup language (Extensible Markup Language, XML), etc. This step is typically done by both the business operator and the flow developer.

B. And (3) creating a flow: appropriate components are selected to implement the flow defined above. Where a component may be a function, a service, a functional module, etc. This step is typically done by a flow developer.

C. The flow is improved: the created flow is tried to be operated, and whether an operation result is consistent with an expected result is judged; and if the operation results are inconsistent, modifying the flow definition and the flow implementation until the operation results meet the expectations. This step is typically done by both the business operator and the flow developer.

D. The flow is operated: and formally running the process, automatically processing the service, and monitoring the execution condition of the process. This step is typically taken care of by an attendant or a monitoring program.

As described above, three steps of flow description, flow creation and flow improvement require participation of business operators and flow developers, and have high use threshold and long development period.

For easy understanding, the following describes in detail a service code generating method according to an embodiment of the present application with reference to the accompanying drawings:

firstly, a service code generation flow in the related art is described:

as shown in fig. 1, the RPA process in the related art is divided into two steps: flow description and flow creation. Wherein, the flow is described: the service operators and RPA process developers comb the service process needing automation, determine the process details and output the service process description in a document form. And (3) creating a flow: the RPA flow developer selects a proper flow template according to the business flow description, and then adds, deletes and revises components in the template; or creating a new flow, selecting a proper assembly to build the RPA flow from the beginning; thus obtaining a complete RPA flow.

The method is poorly applicable. The flow description and flow creation require the RPA flow developer to participate in order to complete. Secondly, the development period of the method is long. In the process of flow description and flow creation, the business operators and RPA flow developers are required to repeatedly communicate and align, so that the understanding of both parties to the business flow is consistent, and the communication time is longer. For business processes without proper templates, RPA processes need to be built from scratch, and development time is longer.

As shown in fig. 2, in other related art schemes, the RPA procedure is divided into the following steps:

A. description of the flow: the service operator performs service operation demonstration, and the monitoring program records the change of an operation interface (screen recording) and the input of a mouse and a keyboard. I.e. the flow is described as a time sequence of three dimensions, screen, mouse, keyboard.

B. And (3) creating a flow:

a) Slicing the flow description by using a model;

b) Predicting each fragment by using a classification model to obtain a category, wherein one category corresponds to one preset module;

c) And sequentially combining the preset modules corresponding to each fragment to form an RPA flow.

The method has limited flow description expression capability. The description of the flow is a time sequence, and complex flows such as branch processing, circulation and the like cannot be described; and the universality is poor and the cost is high. The interface difference of application systems of enterprises is extremely large, and the operation modes of the systems are also quite different; if the RPA manufacturer provides a unified fragmentation model and classification model, good fragmentation and classification results cannot be obtained for each enterprise; if an enterprise is used to train a set of segmentation and classification models for each RPA, the cost can be high; and if the interface is changed after the application system of the enterprise is upgraded, a set of segmentation and classification models trained for the enterprise also fail, and the models need to be retrained.

As shown in fig. 3, in other related art schemes, the RPA procedure is divided into the following steps:

A. description of the flow: the business operator describes the business process in a short natural language.

B. And (3) creating a flow:

a) A disassembly model is preset to decompose the business process into a plurality of sub-processes; the sub-flow is described by natural language;

b) The sub-process obtains a preset module sequence through model identification;

c) And combining the preset module sequences obtained by the sub-processes to form a complete RPA process.

The method has poor universality and high cost. An enterprise fine tuning model needs to be applied for each RPA. The same natural language is used to describe a task, and the operations that need to be performed at different enterprises are different. For example: "please help export the delivery resume of the candidate", some enterprises may be log-in mailbox export resume, and some enterprises may be log-in specified system export resume. Thus, a separate training model for each RPA application enterprise is required.

In view of the above problems, embodiments of the present application provide a service code generating method, apparatus, device, and storage medium, for solving the above problems. The inventive concept of the present application can be summarized as follows: acquiring multi-mode information of a target service; decomposing the multi-mode information by adopting the multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model; for each business sub-flow: inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process; and merging the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service.

The following describes a service code generating method provided by the embodiment of the present application:

fig. 4 is an application scenario diagram of a service code generating method according to an embodiment of the present application. The drawings include: a server 10, a memory 20, and a terminal device 30; wherein:

the server 10 acquires the multi-modal information of the target service from the memory 20; decomposing the multi-mode information by adopting the multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model; for each business sub-flow: inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process; and carrying out combination processing on the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service, and displaying the service codes in the terminal equipment 30.

In the description of the present application, only a single server 10, memory 20, terminal device 30 is described in detail, but it should be understood by those skilled in the art that the illustrated server 10, memory 20, terminal device 30 are intended to represent the operation of the server 10, memory 20, terminal device 30 in relation to the technical solution of the present application. And not implying a limitation on the number, type, location, etc. of servers 10, memories 20, terminal devices 30. It should be noted that the underlying concepts of the exemplary embodiments of this application are not altered if additional modules are added to or individual modules are removed from the illustrated environment.

It should be noted that, the service code generating method provided by the present application is not only suitable for the application scenario shown in fig. 4, but also suitable for any device with a service code generating requirement.

Fig. 5 is a schematic flow chart of a service code generating method according to an embodiment of the present application, where:

in step 501: and acquiring multi-mode information of the target service.

In step 502: and decomposing the multi-mode information by adopting the multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model.

In step 503: for each business sub-flow: and inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process.

In step 504: and merging the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service.

In order to facilitate further understanding of a service code generation method provided by an embodiment of the present application, the following details of steps in fig. 5 are described below:

in some possible embodiments, the obtaining multi-modal information of the target service may be specifically implemented as: responding to a code generation instruction triggered by a user, and determining that the code generation instruction corresponds to a target service; and acquiring multi-mode information corresponding to the target service from the memory based on the target service.

In the present application, in order to ensure the accuracy of the obtained service code, the target service is described using multi-modal information in the present application. The multi-modal information comprises images, voice, a mouse and a keyboard input sequence; before the steps in fig. 5 are implemented, operation demonstration is performed by a service person for each service flow of each service which may occur, and simultaneously, each operation step is illustrated by using voice in the demonstration process, and a recording tool is used for recording an image sequence, a voice sequence, a mouse input sequence and a keyboard input sequence in the demonstration process, so that multi-mode information of the service flow is obtained. The mouse input sequence comprises time, screen position and operation type, wherein the operation type comprises left key click, right key click, left key double click, right key double click and the like. The keyboard inputs include time, input method status, key inputs. In this embodiment, this stage is to perform the next process decomposition after the completion of the multi-modal description of the entire business process.

For example: as shown in fig. 6, the target traffic is: and (3) exporting 23-set AA plan school delivery information on the A website, recording a demonstration process by using a recording tool, and performing explanation by using voice, namely opening an XX browser, entering the A website, clicking to log in, inputting an account number and a password, logging in, clicking a project management button, inputting the AA plan in a search box, clicking a search, clicking an export button after the search is completed, clicking a download, and completing. The time, the screen position and the operation type of the service personnel clicking the mouse each time are recorded in the mouse input sequence, and the time, the input method state and the key input of the service personnel using the keyboard each time are recorded in the keyboard input sequence.

In some possible embodiments, in order to ensure that a corresponding service code can be generated for each target service, the steps shown in fig. 7 are performed after determining that the code generation instruction corresponds to the target service:

in step 701: if the multi-mode information corresponding to the target service is not obtained from the memory based on the target service, generating a multi-mode information configuration prompt; the multi-mode information configuration prompt is used for prompting a technician to configure multi-mode information corresponding to the target service.

In step 702: and outputting and displaying the multi-mode information configuration prompt.

For example: and determining that the target service corresponding to the code generation instruction is the service 8, and if the multi-mode information corresponding to the service 8 is not acquired in the memory, generating a multi-mode information configuration prompt of 'please configure the multi-mode information of the service 8', and outputting and displaying the multi-mode information.

By the method, each target service can generate a corresponding service code, and the experience of the user is guaranteed, so that the method has universality.

In some possible embodiments, the target business is broken down into multiple sub-flows using the generalized summarization capabilities of the multimodal base model for multimodal information, where each sub-flow is also a description of the use of multiple modalities. Wherein the multi-template description of the sub-process includes text and objects, the text being a natural language description of the sub-process; the object is a multi-modal representation of the sub-process, including the name, image, website, web page element, etc. of the object corresponding to the sub-process, and which information the object specifically contains depends on what kind of object is; for example, if the object is a software object, the multi-modal description of the sub-flow of the object needs to include a software name, an absolute path of a software executable file, a software icon, and a software icon position; if the object is a button object on a web page, the multi-modal description of the sub-flow of the object needs to include text on the button, a button image, a button position, and a web page element corresponding to the button. The multiple sub-processes obtained in the present application may be a sequential sequence, or may be jumps or loops.

Wherein, the multi-modal base model in the embodiment of the present application is trained by the steps shown in fig. 8, wherein:

in step 801: the method comprises the steps of obtaining multi-modal information of a training service, wherein the multi-modal information comprises an image sequence, a voice sequence, a mouse input sequence and a keyboard input sequence corresponding to the training service.

For example: aiming at the training service 1, the training service 2, the training service 3, the training service … … and the training service N, the multi-mode information corresponding to each training service in the training service 1, the training service 2, the training service 3, the training service … … and the training service N is obtained.

In step 802: and constructing a first training sample set by adopting the multi-mode information of the training service.

Continuing with the above example, the first training sample set constructed includes: training service 1, training service 2, training services 3 and … …, and training service N respectively correspond to the multi-modal information.

In step 803: inputting a first training sample in a first training sample set into an initial multi-modal base model, training the initial multi-modal base model in an iterative mode until a preset convergence condition is met, and taking the initial multi-modal base model with the iteration end as the multi-modal base model.

In some possible embodiments, each round of iterative process of the initial multi-modal base model may be implemented as steps as shown in fig. 9, wherein:

In step 901: and inputting the first training samples in the first training sample set into the initial multi-mode basic model to obtain a business sub-process output by the initial multi-mode basic model.

In step 902: and determining a loss value of the initial multi-mode basic model according to the preset business sub-flow of the first training sample and the business sub-flow output by the initial multi-mode basic model.

In step 903: and adjusting model parameters of the initial multi-mode basic model according to the loss value of the initial multi-mode basic model.

For example: as shown in fig. 6, the multi-modal information corresponding to the first training sample 1 is determined as an image sequence: image 1, image 2, image 3; the voice sequence is as follows: opening an XX browser, entering an A website, clicking to log in, inputting an account number and a password, logging in, clicking a project management button, inputting an AA plan in a search box, clicking to search, clicking to export the button after the search is completed, clicking to download, and completing the process; the mouse input sequence is Left Click (100,10,1), left Click (100,10000,1), left Click (1800,900,1), left Click (1600,800,1), left Click (1600,700,1), left Click (1600,600,1); wherein Left Click represents Left Click of the mouse, the first two bits in the bracket represent positions in the screen clicked by the mouse, and the last bit in the bracket represents operation type of the mouse; keyboard input sequence: zp.xxxx.cn; thirdly, stretching; zhangsan; AA planning.

The preset business sub-process corresponding to the training sample 1 comprises the following steps: sub-flow 1, open XX browser (image 1) to enter A website (zp. Xxxx. Cn); sub-process 2, click "login" (text, image 2, web page element); sub-flow 3, input "user name" (Zhangsan) and "password" (zhangsan); sub-process 4, click "login"; sub-flow 5, click "project management button" (image 3); sub-process 6, inputting "AA plan" in "search box"; sub-process 7, clicking the "search" button; sub-process 8, wait for searching to finish; sub-process 9, clicking the "export" button; sub-flow 10, clicking the "download" button.

The determining the business sub-process of the initial multi-mode basic model output by the training sample 1 comprises the following steps: sub-flow 1, open XX browser (image 1) to enter A website (zp. Xxxx. Cn); sub-process 2, click "login" (text, image 2, web page element); sub-flow 3, input "user name" (Zhangsan) and "password" (zhangsan); sub-process 4, click "login"; sub-flow 5, click "project management button" (image 3); sub-process 7, clicking the "search" button; sub-process 8, wait for searching to finish; sub-process 9, clicking the "export" button; sub-flow 10, clicking the "download" button.

Determining a loss value of the initial multi-mode basic model according to the preset business sub-flow of the training sample 1 and the business sub-flow of the training sample 1 output by the initial multi-mode basic model, and adjusting model parameters of the initial multi-mode basic model according to the loss value.

In some possible embodiments, after the sub-flows are obtained, for each sub-flow, a code generation model is used to generate a service code, i.e. a preset module sequence, corresponding to the sub-flow. The code generation model here is trained to the requirements of different RPA vendors. The preset module in the application can be a function or a functional module, and the input of the preset module is an object in the sub-flow description. The object uses a multi-mode information description mode to describe the difference between business systems of different companies, namely, the difference of system interfaces is described by the attributes of icons, websites, webpage elements, texts and the like in the object; therefore, natural language description in the sub-flow description can be unified among companies, so that only one code generator is required to be trained according to respective requirements of RPA manufacturers, the code generator is directly available for all companies, and special model training is not required to be performed for the companies using RPA software.

The code generation model in the embodiment of the application is obtained by training through the steps shown in fig. 10:

in step 1001: and acquiring service sub-flows of a plurality of training services.

For example: acquiring a service sub-process of the training service A, wherein the service sub-process of the training service A comprises: sub-flow 1, sub-flow 2, sub-flow 3, sub-flow 4, sub-flow 5, sub-flow 6, sub-flow 7, sub-flow 8, sub-flow 9, sub-flow 10.

In step 1002: and constructing a second training sample set by adopting a service sub-process of the training service.

Continuing with the above example, sub-flow 1, sub-flow 2, sub-flow 3, sub-flow 4, sub-flow 5, sub-flow 6, sub-flow 7, sub-flow 8, sub-flow 9, sub-flow 10 are respectively used as second training samples to construct a second training sample set.

In step 1003: inputting a second training sample in the second training sample set into the initial code generation model, training the initial code generation model in an iterative mode until a preset convergence condition is met, and taking the initial code generation model after iteration is used as a code generation model.

In some possible embodiments, each round of iterative process of the initial code generation model may be implemented as steps as shown in fig. 11, wherein:

In step 1101: and inputting the second training samples in the second training sample set into the initial code generation model to obtain the service codes output by the initial code generation model.

In step 1102: and determining a loss value of the initial code generation model according to the preset service code of the second training sample and the service sub-flow output by the initial code generation model.

In step 1103: and adjusting model parameters of the initial code generation model according to the loss value of the initial code generation model.

For example: as shown in fig. 6, in the sub-process 1, a XX browser (image 1) is opened to enter an a website (zp.xxxx.cn), and the preset service code corresponding to the sub-process 1 is xxxx.xxxx.xxxxx; inputting the sub-flow 1 into an initial code generation model to obtain a service code which is output by the initial code generation model as XXXX.XX.XXXXXX; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-process 1 and the service code of the sub-process 1 output by the initial code generation model.

The sub-flow 2 is click "login" (text, image 2, webpage element), and the preset service code corresponding to the sub-flow 2 is AAAAA; inputting the sub-process 2 into an initial code generation model to obtain a service code output by the initial code generation model as AAAAA. And obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 2 and the service code of the sub-flow 2 output by the initial code generation model.

The sub-flow 3 is to input a user name (Zhangsan) and a password (zhangsan), and the preset service code corresponding to the sub-flow 3 is BBBBB.BBB.BBBBB; inputting the sub-flow 3 into an initial code generation model to obtain a service code output by the initial code generation model as BBB, BBB and BBB; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 3 and the service code of the sub-flow 3 output by the initial code generation model.

The sub-process 4 is clicking on "login", and the preset service code corresponding to the sub-process 4 is CCCCC.CCC.CCCCCC; inputting the sub-process 4 into an initial code generation model to obtain a service code which is output by the initial code generation model as CCCCCC. And obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 4 and the service code of the sub-flow 4 output by the initial code generation model.

The sub-process 5 is to click a project management button (image 3), and the preset service code corresponding to the sub-process 5 is DDDDD.DDD.DDDDDDD; inputting the sub-process 5 into an initial code generation model to obtain a service code output by the initial code generation model as DDDDD.DDDDD.DDDDDDD; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 5 and the service code of the sub-flow 5 output by the initial code generation model.

The sub-process 6 is to input an AA plan in a search box, and the preset service code corresponding to the sub-process 6 is EEEEE.EEE.EEEEEEEEE; inputting the sub-process 6 into an initial code generation model to obtain a service code which is EEEEEEE, EEEEEEEEEEE and is output by the initial code generation model; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 6 and the service code of the sub-flow 6 output by the initial code generation model.

The sub-flow 7 is to click a 'search' button, and the preset service code corresponding to the sub-flow 7 is FFFFF.FFF.FFFFF; inputting the sub-process 7 into an initial code generation model to obtain a service code which is FFFFF.FFFFF.FFFFF and is output by the initial code generation model; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-process 7 and the service code of the sub-process 7 output by the initial code generation model.

The sub-flow 8 waits for the completion of the search, and the preset service code corresponding to the sub-flow 8 is GGGGG.GGG.GGGGGGG; inputting the sub-flow 8 into an initial code generation model to obtain a business code which is GGGGG, GGGGGGG and is output by the initial code generation model; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 8 and the service code of the sub-flow 8 output by the initial code generation model.

The sub-flow 9 is to click the "export" button, and the preset service code corresponding to the sub-flow 9 is HHHHH.HHHHHHH; inputting the sub-flow 9 into an initial code generation model to obtain a service code which is output by the initial code generation model as HHHHH.HHHHHHHHHH; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 9 and the service code of the sub-flow 9 output by the initial code generation model.

The sub-process 10 is to click the "download" button, and the preset service code corresponding to the sub-process 10 is iiiii.iii.iiiiiii; inputting the sub-flow 10 into an initial code generation model to obtain a service code of IIIII.IIIII.IIIIIIIIII output by the initial code generation model; and obtaining a loss value of the initial code generation model based on the preset service code corresponding to the sub-flow 10 and the service code of the sub-flow 10 output by the initial code generation model.

In some possible embodiments, in order to ensure that the obtained service code is accurate, a feedback mechanism is added for each service sub-flow after the service code corresponding to the service sub-flow is obtained, so that necessary information or ambiguous information which is not notified in the code generation process is supplemented and corrected, and the probability that the automatically generated RPA flow needs to be modified is reduced. And judging whether each business code is abnormal or not according to the business code corresponding to each business sub-flow. If the service code is determined to be missing or ambiguous, the service operator is queried in a voice, image and text mode. The absence of the service code means that necessary information which cannot be obtained based on the current multi-mode information of the target service or that the obtained sub-flow is inaccurate due to the problem of the multi-mode basic model capability. Business code ambiguity refers to ambiguity in the multimodal information of the acquired target business (e.g., speech may be misidentified) or ambiguity in the output sub-process caused by the multimodal base model capability problem. The steps shown in fig. 12 may be specifically implemented, wherein:

In step 1201: and judging the accuracy of the service codes.

In step 1202: if the service code is determined to have errors, generating a corresponding error prompt according to the error type of the service code.

In step 1203: and outputting and displaying the error prompt.

In the embodiment of the application, the error prompt can be semantic feedback, image feedback and text feedback; the voice feedback in the application converts the questions to be inquired into voice and plays the voice; image feedback is to define a range on the operation interface, such as drawing a region on the software operation interface, asking what button to click at the bottom; text feedback may be printed on the screen, for example, where the business operator speaks a specified word at the time of the explanation, and where there are two homophones that match the specified word, it is necessary to print it by text, and ask what the specified word is. After feedback, the service operator gives a corresponding supplementary explanation, and the steps in fig. 5 are executed again based on the supplementary explanation until the generated service code is accurate.

It should be noted that the above three error prompting methods provided by the present application are only one embodiment, and are not limited to the error prompting method, and a technician can set the error prompting method according to the needs during the implementation.

After adding the feedback mechanism, the flow of the present application is shown in fig. 13, wherein:

in step 1301: and acquiring multi-mode information of the target service.

In step 1302: and decomposing the multi-mode information by adopting the multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model.

In step 1303: for each business sub-flow: and inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process.

In step 1304: determining the accuracy of the service code of each service sub-process, determining whether the service code is wrong, if so, entering step 1305, otherwise, entering step 1307;

in step 1305: outputting an error prompt;

in step 1306: and merging the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service.

In step 1307: a supplemental description is received.

Based on the same inventive concept, the present application also provides a service code generating system, as shown in fig. 14, comprising: the system comprises a flow mining module, a flow recording module, a flow creating module, other intelligent processing modules, a flow executing/monitoring module, a flow robot management module and a visualization module; wherein:

The flow digging module: for discovering which of the workflows are suitable for automation using RPA techniques;

the flow recording module: the description of the workflow can be various modes such as documents, videos and the like;

the flow creation module: the method comprises the steps of realizing an executable RPA flow by using a template, a preset module, a new module and the like;

other intelligent processing modules: a module for handling a particular task, for example: invoice identification, contract comparison, seal identification and the like;

flow execution/monitoring module: the method comprises the steps of starting an RPA flow robot and monitoring the execution state of the RPA flow robot;

the flow robot management module: the method is used for distributing tasks of the flow robots, managing resources, coordinating among the flow robots and managing the coordination of the flow robots and people;

and a visualization module: the method is used for robot state visualization, data analysis and the like.

In summary, as shown in fig. 15, the central architecture of the present application acquires multi-modal information of a target service, where the multi-modal information includes video image information, language information, a mouse input sequence, and a keyboard input sequence; then decomposing the multi-mode information by adopting the multi-mode basic model to obtain at least one business sub-process output by the multi-mode basic model, wherein the obtained business sub-process also adopts multi-mode description; then, inputting the business sub-process into a pre-trained code generation model for each sub-process to obtain a business code corresponding to the business sub-process; and finally, carrying out combination processing on the service codes corresponding to each service sub-flow to obtain the service codes corresponding to the target service.

Based on the same inventive concept, after describing a service code generating method provided by an embodiment of the present application, as shown in fig. 16, a service code generating apparatus 1600 provided by an embodiment of the present application is described below, where the apparatus includes:

an obtaining module 16001, configured to obtain multi-modal information of a target service;

the decomposition module 16002 is configured to perform a decomposition operation on the multimodal information by using a multimodal basic model, so as to obtain at least one business sub-process output by the multimodal basic model;

a code generation module 16003 for, for each business sub-process: inputting the business sub-process into a pre-trained code generation model to obtain a business code corresponding to the business sub-process;

and a merging module 16004, configured to merge the service codes corresponding to each service sub-flow, to obtain the service codes corresponding to the target service.

In some possible embodiments, when the obtaining module 16001 performs multi-modal information of the target service, the method specifically is used for:

In some possible embodiments, after the obtaining module 16001 executes the determination that the code generation instruction corresponds to the target service, the method further comprises:

and outputting and displaying the multi-mode information configuration prompt.

In some possible embodiments, before the decomposing module 16002 performs the decomposing operation on the multimodal information using the multimodal base model, the decomposing module is further configured to:

In some possible embodiments, before the code generation module 16003 performs the inputting of the business sub-process into a pre-trained code generation model, it is further configured to:

acquiring service sub-flows of a plurality of training services;

In some possible embodiments, after the code generating module 16003 executes the service code corresponding to the service sub-flow, the code generating module is further configured to:

performing accuracy judgment on the service codes;

and outputting and displaying the error prompt.

Corresponding to the embodiment, the application also provides electronic equipment. Fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present application, where the electronic device 1700 may include: a processor 1701, a memory 1702 and a communication unit 1703. The components may communicate via one or more buses, and it will be appreciated by those skilled in the art that the configuration of the electronic device shown in the drawings is not limiting of the embodiments of the application, as it may be a bus-like structure, a star-like structure, or include more or fewer components than shown, or may be a combination of certain components or a different arrangement of components.

Wherein the communication unit 1703 is configured to establish a communication channel, so that the electronic device may communicate with other devices. Receiving user data sent by other devices or sending user data to other devices.

The processor 1701, which is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, performs various functions of the electronic device and/or processes data by running or executing software programs and/or modules stored in the memory 1702, and invoking data stored in the memory. The processor may be comprised of integrated circuits (integrated circuit, ICs), such as a single packaged IC, or may be comprised of packaged ICs that connect multiple identical or different functions. For example, the processor 1701 may include only a central processing unit (central processing unit, CPU). In the embodiment of the invention, the CPU can be a single operation core or can comprise multiple operation cores.

The memory 1702 for storing the instructions for execution by the processor 1701 may be implemented by any type of volatile or nonvolatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

The execution of the instructions in memory 1702, when executed by processor 1701, enables electronic device 1700 to perform some or all of the steps of the embodiment shown in fig. 7.

In a specific implementation, the present invention further provides a computer storage medium, where the computer storage medium may store a program, where the program may include some or all of the steps in each embodiment of the calling method provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.

It will be apparent to those skilled in the art that the techniques of embodiments of the present invention may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in essence or what contributes to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments of the present invention.

The same or similar parts between the various embodiments in this specification are referred to each other. In particular, for the device embodiment and the terminal embodiment, since they are substantially similar to the method embodiment, the description is relatively simple, and reference should be made to the description in the method embodiment for relevant points.

Claims

1. A method for generating a service code, the method comprising:

acquiring multi-mode information of a target service;

2. The method of claim 1, wherein the obtaining multi-modal information of the target service comprises:

3. The method of claim 2, wherein after the determining that the code generation instruction corresponds to a target service, the method further comprises:

and outputting and displaying the multi-mode information configuration prompt.

4. The method of claim 1, wherein prior to said decomposing said multimodal information with a multimodal base model, the method further comprises:

5. The method of claim 4, wherein each iteration of the initial multi-modal base model proceeds as follows:

6. The method of claim 1, wherein prior to said entering the business sub-process into the pre-trained code generation model, the method further comprises:

acquiring service sub-flows of a plurality of training services;

7. The method of claim 6, wherein each iteration of the initial code generation model proceeds as follows:

8. The method of claim 1, wherein after the service codes corresponding to the service sub-flows are obtained, the method further comprises:

performing accuracy judgment on the service codes;

and outputting and displaying the error prompt.

9. A service code generation apparatus, the apparatus comprising:

10. An electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the electronic device to perform the method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run controls a device in which the computer readable storage medium is located to perform the method according to any one of claims 1-8.