CN113822215A - Equipment operation guide file generation method and device, electronic equipment and storage medium - Google Patents
Equipment operation guide file generation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113822215A CN113822215A CN202111149207.1A CN202111149207A CN113822215A CN 113822215 A CN113822215 A CN 113822215A CN 202111149207 A CN202111149207 A CN 202111149207A CN 113822215 A CN113822215 A CN 113822215A
- Authority
- CN
- China
- Prior art keywords
- screenshot
- operation interface
- annotation
- classification
- screenshots
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
Abstract
The invention relates to an artificial intelligence technology, and discloses a method for generating an equipment operation guide file, which comprises the following steps: carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets; performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot; carrying out image content description on the annotation screenshot to obtain an image content text; marking a corresponding operation interface screenshot by using each image content text; and combining and packaging all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide file. The invention also relates to a block chain technology, and the screenshot set of the operation interface can be stored in a block chain link point. The invention also provides a device, equipment and a medium for generating the equipment operation guide file. The invention can improve the efficiency of generating the device operation guide file.
Description
Technical Field
The present invention relates to artificial intelligence technologies, and in particular, to a method and an apparatus for generating an equipment operation guidance file, an electronic device, and a storage medium.
Background
In recent years, with the continuous development of artificial intelligence technology, various banking outlets realize self-service transaction of the business of users through various devices. However, the operation of some devices is not very user-friendly, and the software version of the device is often updated iteratively, so that the operation guidance of the device is very necessary for the user.
However, the operation guide generation scheme of the conventional device needs to perform screenshot description on all operation screenshots of all devices, and the efficiency of the device operation guide generation method is low.
Disclosure of Invention
The invention provides a method and a device for generating a device operation guide file, electronic equipment and a computer readable storage medium, and mainly aims to improve the efficiency of generating the device operation guide file.
In order to achieve the above object, a method for generating a device operation guidance file provided by the present invention includes:
acquiring an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time;
carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets;
performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
carrying out image content description on the annotation screenshot to obtain an image content text;
marking a corresponding operation interface screenshot by using each image content text;
and combining and packaging all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide file.
Optionally, the classifying the operation interface and filtering similar screenshots of the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classified screenshot sets includes:
acquiring an operation interface title corresponding to the operation interface screenshot;
summarizing all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set;
and eliminating similar operation interface screenshots in the initial classification screenshot set to obtain the classification screenshot set.
Optionally, the obtaining the classification screenshot set by removing the similar operation interface screenshot in the initial classification screenshot set includes:
converting each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;
calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain screenshot similarity;
and eliminating similar operation interface screenshots in the initial classification screenshot sets according to the screenshot similarity to obtain the corresponding classification screenshot sets.
Optionally, the converting, by the utilization, each screenshot of the operation interface in the initial classification screenshot set into a vector to obtain a screenshot vector, where the method includes:
selecting any one of the operation interface screenshots in the initial classification screenshot set and inputting the screenshot into a pre-constructed deep learning network model;
and acquiring output values of all nodes of the full-connection layer in the deep learning network model to construct vectors, and acquiring the corresponding screenshot vectors.
Optionally, the calculating, according to the screenshot vector, a similarity between any two operation interface screenshots in the initial classification screenshot set to obtain a screenshot similarity includes:
calculating the screenshot similarity by using the following formula:
wherein, XiThe i-th element, Y, representing the screenshot vector XiAnd the Sim represents the similarity between the screenshot vector X corresponding to the screenshot X of the operation interface and the screenshot vector Y corresponding to the screenshot Y of the operation interface, and the n represents the vector dimensions of the screenshot vector X and the screenshot vector Y.
Optionally, before performing image region division annotation on each operation interface screenshot in the classification screenshot set by using a preset region division model, the method further includes:
acquiring a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set comprises a divided region coordinate and a divided region annotation;
selecting a historical operation interface screenshot in the historical operation interface screenshot set, inputting the historical operation interface screenshot into a preset initial region division model, and obtaining a division region prediction coordinate and a division region annotation prediction value;
determining a real value of the divided region annotation according to the divided region annotation;
calculating by using a preset first loss function according to the divided region annotation real value and the divided region annotation predicted value to obtain an annotation loss value;
calculating by using a preset second loss function according to the divided region coordinates and the predicted region coordinates to obtain region division loss values;
calculating according to the annotation loss value and the region division loss value to obtain a target loss value;
when the target loss value is larger than a preset loss threshold value, adjusting model parameters of the initial region division model, and returning to the step of selecting a historical operation interface screenshot in the historical operation interface screenshot set and inputting the historical operation interface screenshot into the preset initial region division model;
and when the target loss value is not greater than a preset loss threshold value, outputting the initial region division model to obtain the region division model.
Optionally, the combining and packaging all the labeled screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guidance file includes:
splicing all the marked screenshot of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide picture;
and converting the operation guide diagram into a preset file format to obtain the operation guide file.
In order to solve the above problem, the present invention further provides an apparatus for generating a device operation guidance file, including:
the screenshot annotation module is used for acquiring an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time; carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets; performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
the image content description module is used for carrying out image content description on the annotation screenshot to obtain an image content text; marking a corresponding operation interface screenshot by using each image content text;
and the file generation module is used for combining and packaging all the marked screenshot of the operation interface according to the sequence of the corresponding screenshot time to obtain the operation guide file.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one computer program; and
and the processor executes the computer program stored in the memory to realize the device operation guidance file generation method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the device operation guidance file generation method described above.
The method, the device, the electronic device and the readable storage medium for generating the device operation guidance file improve the efficiency of generating the device operation guidance file.
Drawings
Fig. 1 is a schematic flowchart of a method for generating a device operation guidance file according to an embodiment of the present invention;
fig. 2 is a schematic block diagram of an apparatus for generating a device operation guidance file according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device for implementing a method for generating a device operation guidance file according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention provides a method for generating a device operation guide file. The execution subject of the device operation guidance file generation method includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiments of the present application. In other words, the device operation guidance file generation method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a blockchain platform. The server includes but is not limited to: the cloud server can be an independent server, or can be a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
Referring to fig. 1, which is a schematic flow diagram of a device operation guidance file generation method according to an embodiment of the present invention, in an embodiment of the present invention, the device operation guidance file generation method includes:
s1, obtaining an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time;
in detail, the screenshot of the operation interface in the embodiment of the present invention is a screenshot of a test operation interface when some intelligent devices perform a UAT test on a certain software iteration version, where the intelligent devices include, but are not limited to: bank outlets, self-service cash registers, etc. For example: and the bank outlet equipment A has an iteration version of V1.0.1.0, and performs screenshot on the operation interface when in the UAT test stage.
Specifically, the embodiment of the invention can capture the operation interface of the software operation according to the operation sequence of the software when the UAT test ring of a certain iteration version of the software is tested, so as to obtain the operation interface capture set.
In another embodiment of the invention, the operation interface cut-out set can be stored in a block chain node, and the data access efficiency is improved by using the characteristic of high throughput of the block chain node.
S2, carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets;
in the embodiment of the invention, because the operation guide corresponding to the operation screenshot corresponding to different operation interfaces is different, the operation screenshot of the same operation interface has the same operation interface title.
In detail, in the embodiment of the present invention, an operation interface title corresponding to the operation interface screenshot is obtained; and summarizing all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set.
Further, because the operation interface screenshots of similar operation steps in the classification screenshot set are also similar, and highly similar operation interface screenshots can be guided by operations labeled with the same label, in order to improve the generation efficiency of the operation guide, the similar operation interface screenshots in the initial classification screenshot set need to be removed to obtain the classification screenshot set.
Further, in the embodiment of the present invention, the obtaining the classification screenshot set by removing the similar operation interface screenshot in the initial classification screenshot set includes:
converting each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;
calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain screenshot similarity;
in detail, in the embodiment of the invention, similarity calculation is performed on two screenshot vectors corresponding to two screenshot of the operation interface, so that corresponding screenshot similarity is obtained.
Optionally, in the embodiment of the present invention, the similarity calculation is performed by using the following formula:
wherein, XiThe i-th element, Y, representing the screenshot vector XiAnd the Sim represents the similarity between the screenshot vector X corresponding to the screenshot X of the operation interface and the screenshot vector Y corresponding to the screenshot Y of the operation interface, and the n represents the vector dimensions of the screenshot vector X and the screenshot vector Y.
And eliminating similar operation interface screenshots in the initial classification screenshot set according to the screenshot similarity to obtain a corresponding classification screenshot set.
Further, in the embodiment of the present invention, converting each screenshot of the initial classification screenshot set into a vector to obtain a screenshot vector, including:
selecting any one of the operation interface screenshots in the initial classification screenshot set and inputting the screenshot into a pre-constructed deep learning network model;
and acquiring output values of all nodes of the full-connection layer in the deep learning network model to construct vectors, and acquiring the corresponding screenshot vectors.
Optionally, in the embodiment of the present invention, the deep learning network model is an artificial intelligence model, and the deep learning network model is CNN (Convolutional Neural Networks).
Further, in the embodiment of the present invention, the screening and filtering of the similar operation interface screenshots in the initial classification screenshot set by using the pre-constructed deep learning network model includes:
in detail, in the embodiment of the present invention, obtaining output values of all nodes of a full connection layer in the deep learning network model to perform vector construction, so as to obtain the corresponding screenshot vector, where the vector construction includes:
and longitudinally combining the output values of all the nodes according to the sequence of the corresponding nodes in the full connection layer to obtain the screenshot vector.
For example: the full connection layer is provided with 3 nodes which are respectively a first node, a second node and a third node in sequence, the output values of the operation interface screenshot nodes of the operation interface screenshot A are 3,5 and 1 in total, wherein the output value of the node 1 is the output of the first node, the output value of the node 3 is the output of the second node, the output value of the node 5 is the output of the third node, and the output values of the three nodes in the operation interface screenshot node output value set of the operation interface screenshot A are longitudinally combined in the node sequence to obtain the operation interface screenshot feature vector of the operation interface screenshot A
In detail, in the embodiment of the present invention, the removing, according to the screenshot similarity, a similar operation interface screenshot in the initial classification screenshot set to obtain a corresponding classification screenshot set includes:
and removing any one of the two operation interface screenshots corresponding to the screenshot similarity which is greater than a preset similarity threshold in the initial splitting screenshot set to obtain the classification screenshot set. Optionally, the similarity threshold in the embodiment of the present invention may be 90%, and when the similarity of the screenshots corresponding to the two operator interfaces is greater than the similarity threshold, it indicates that the screenshots of the two operator interfaces are highly similar, and only one of the screenshots is retained.
S3, performing image area division annotation on each operation interface screenshot in the classification screenshot set by using a preset area division model to obtain an annotation screenshot;
in the embodiment of the present invention, the screenshot of the operation interface includes different page controls, for example: the method comprises the steps of inputting a frame, selecting a single or multiple option, pulling down the option, pressing a button, handwriting a signature frame, taking a license picture and the like, and dividing different page control areas in the screenshot of the annotation operation interface due to different page controls and different operation guides corresponding to different page controls, so that the embodiment of the invention divides and annotates the image area of each screenshot of the operation interface in the classified screenshot set to obtain the annotation screenshot.
In detail, in the embodiment of the present invention, a preset region division model is used to perform image region division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot, and optionally, the region division model is a Deep CNN model.
Further, before performing image region division annotation on each operation interface screenshot in the classification screenshot set by using a preset region division model, the method in the embodiment of the present invention further includes:
acquiring a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set comprises a divided region coordinate and a divided region annotation;
optionally, the divided region annotation in the embodiment of the present invention includes but is not limited to: an input box, a single/multiple option, a pull-down option, a button, a handwritten signature box, a certificate photo page, etc.
Selecting a historical operation interface screenshot in the historical operation interface screenshot set, inputting the historical operation interface screenshot into a preset initial region division model, and obtaining a division region prediction coordinate and a division region annotation prediction value;
determining a real value of the divided region annotation according to the divided region annotation;
for example: the segmented region annotation is "input box", then the corresponding segmented region annotation true value is: an input box: 1.
calculating by using a preset first loss function according to the divided region annotation real value and the divided region annotation predicted value to obtain an annotation loss value;
calculating by using a preset second loss function according to the divided region coordinates and the predicted region coordinates to obtain region division loss values;
optionally, in this embodiment of the present invention, the first loss function and the second loss function may be: logarithmic loss functions, quadratic loss functions, cross-entropy loss functions, and the like.
Calculating according to the annotation loss value and the region division loss value to obtain a target loss value;
when the target loss value is larger than a preset loss threshold value, adjusting model parameters of the initial region division model, and returning to the step of selecting a historical operation interface screenshot in the historical operation interface screenshot set and inputting the historical operation interface screenshot into the preset initial region division model; and when the target loss value is not greater than a preset loss threshold value, outputting the initial region division model to obtain the region division model.
S4, carrying out image content description on the annotation screenshot to obtain an image content text;
in detail, in the embodiment of the present invention, a preset picture description (Image capture) technology is used to describe the Image content of the annotation screenshot, so as to obtain the Image content text. Optionally, in the embodiment of the present invention, image content description is performed on the annotation screenshot by using a GRU, each divided region in the annotation screenshot is extracted to perform intent recognition, an intent text is obtained, the intent text is combined with the annotation corresponding to the divided region, and a divided region content text is obtained, for example: and the annotation corresponding to the divided area is a text box, the corresponding intention text is 'fill in personal information', the corresponding content text of the divided area is 'fill in personal information in the text box', and further, all the content texts of the divided area corresponding to the annotation screenshot are combined to obtain the image content text.
For example: and the annotation screenshot comprises a name input box and a product type radio box, and then the knowledge screenshot is subjected to image description, and the obtained image content text is that the name is input in the input box and a product type needing to be handled is selected.
S5, marking a corresponding operation interface screenshot by using each image content text;
in detail, in the embodiment of the present invention, in order to associate the image content text with the corresponding operation interface screenshot, a user can conveniently perform image-text comparison and browsing, and each image content text is used to label the corresponding operation interface screenshot. In detail, in the embodiment of the present invention, an annotation text box is constructed in the screenshot of the operation interface, and the image content text is input into the annotation text box to realize annotation of the screenshot of the operation interface.
And S6, combining and packaging all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide file.
In detail, in the embodiment of the invention, all the marked screenshot of the operation interface are subjected to image splicing according to the sequence of the corresponding screenshot time to obtain an operation guide image; and converting the operation guide diagram into a preset file format to obtain the operation guide file. Optionally, the preset file format in the embodiment of the present invention may be a PDF format or a doc format.
Fig. 2 is a functional block diagram of the device operation guidance file generating apparatus according to the present invention.
The device operation guidance file generating apparatus 100 according to the present invention may be installed in an electronic device. According to the implemented functions, the device operation guidance file generation apparatus may include a screenshot annotation module 101, an image content description module 102, and a file generation module 103, which may also be referred to as a unit, and refer to a series of computer program segments that can be executed by a processor of an electronic device and can perform fixed functions, and are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the screenshot annotation module 101 is configured to obtain an operation interface screenshot set, where the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes corresponding screenshot time; carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets; performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
in detail, the screenshot of the operation interface in the embodiment of the present invention is a screenshot of a test operation interface when some intelligent devices perform a UAT test on a certain software iteration version, where the intelligent devices include, but are not limited to: bank outlets, self-service cash registers, etc. For example: and the bank outlet equipment A has an iteration version of V1.0.1.0, and performs screenshot on the operation interface when in the UAT test stage.
Specifically, the embodiment of the invention can capture the operation interface of the software operation according to the operation sequence of the software when the UAT test ring of a certain iteration version of the software is tested, so as to obtain the operation interface capture set.
In another embodiment of the invention, the operation interface cut-out set can be stored in a block chain node, and the data access efficiency is improved by using the characteristic of high throughput of the block chain node.
In the embodiment of the invention, because the operation guide corresponding to the operation screenshot corresponding to different operation interfaces is different, the operation screenshot of the same operation interface has the same operation interface title.
In detail, in the embodiment of the present invention, the screenshot annotation module 101 obtains an operation interface title corresponding to the operation interface screenshot; and summarizing all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set.
Further, since the operation interface screenshots of similar operation steps in the classification screenshot set are also similar, and highly similar operation interface screenshots may be guided by operations labeled the same, the screenshot annotation module 101 needs to remove the similar operation interface screenshot in the initial classification screenshot set to obtain the classification screenshot set in order to improve the generation efficiency of the operation guide.
Further, in the embodiment of the present invention, the obtaining, by the screenshot annotation module 101, the classification screenshot set by removing the similar operation interface screenshot in the initial classification screenshot set includes:
converting each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;
calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain screenshot similarity;
in detail, in the embodiment of the invention, similarity calculation is performed on two screenshot vectors corresponding to two screenshot of the operation interface, so that corresponding screenshot similarity is obtained.
Optionally, in the embodiment of the present invention, the similarity calculation is performed by using the following formula:
wherein, XiThe i-th element, Y, representing the screenshot vector XiAnd the Sim represents the similarity between the screenshot vector X corresponding to the screenshot X of the operation interface and the screenshot vector Y corresponding to the screenshot Y of the operation interface, and the n represents the vector dimensions of the screenshot vector X and the screenshot vector Y.
And eliminating similar operation interface screenshots in the initial classification screenshot set according to the screenshot similarity to obtain a corresponding classification screenshot set.
Further, in the embodiment of the present invention, the screenshot annotation module 101 converts each screenshot of the operation interface in the initial classification screenshot set into a vector, so as to obtain a screenshot vector, where the screenshot vector includes:
selecting any one of the operation interface screenshots in the initial classification screenshot set and inputting the screenshot into a pre-constructed deep learning network model;
and acquiring output values of all nodes of the full-connection layer in the deep learning network model to construct vectors, and acquiring the corresponding screenshot vectors.
Optionally, in the embodiment of the present invention, the deep learning network model is an artificial intelligence model, and the deep learning network model is CNN (Convolutional Neural Networks).
Further, in the embodiment of the present invention, the screenshot annotation module 101 performs filtering on similar operation interface screenshots in the initial classification screenshot set by using a pre-constructed deep learning network model, including:
in detail, in the embodiment of the present invention, obtaining output values of all nodes of a full connection layer in the deep learning network model to perform vector construction, so as to obtain the corresponding screenshot vector, where the vector construction includes:
and longitudinally combining the output values of all the nodes according to the sequence of the corresponding nodes in the full connection layer to obtain the screenshot vector.
For example: the full connection layer is provided with 3 nodes which are respectively a first node, a second node and a third node in sequence, the output values of the operation interface screenshot nodes of the operation interface screenshot A are 3,5 and 1 in total, wherein the output value of the node 1 is the output of the first node, the output value of the node 3 is the output of the second node, the output value of the node 5 is the output of the third node, and the output values of the three nodes in the operation interface screenshot node output value set of the operation interface screenshot A are longitudinally combined in the node sequence to obtain the operation interface screenshot feature vector of the operation interface screenshot A
In detail, in the embodiment of the present invention, the screenshot annotation module 101 eliminates similar operation interface screenshots in the initial classification screenshot set according to the screenshot similarity, so as to obtain a corresponding classification screenshot set, including:
and removing any one of the two operation interface screenshots corresponding to the screenshot similarity which is greater than a preset similarity threshold in the initial splitting screenshot set to obtain the classification screenshot set. Optionally, the similarity threshold in the embodiment of the present invention may be 90%, and when the similarity of the screenshots corresponding to the two operator interfaces is greater than the similarity threshold, it indicates that the screenshots of the two operator interfaces are highly similar, and only one of the screenshots is retained.
In the embodiment of the present invention, the screenshot of the operation interface includes different page controls, for example: the method comprises the steps of inputting a frame, selecting a single or multiple option, pulling down the option, pressing a button, handwriting a signature frame, taking a license picture and the like, and dividing different page control areas in the screenshot of the annotation operation interface due to different page controls and different operation guides corresponding to different page controls, so that the embodiment of the invention divides and annotates the image area of each screenshot of the operation interface in the classified screenshot set to obtain the annotation screenshot.
In detail, in the embodiment of the present invention, the screenshot annotation module 101 performs image area division annotation on each screenshot of the operation interface in the classification screenshot set by using a preset area division model to obtain an annotation screenshot, and optionally, the area division model is a Deep CNN model.
Further, before the screenshot annotation module 101 performs image area division annotation on each operation interface screenshot in the classification screenshot set by using a preset area division model, the method further includes:
acquiring a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set comprises a divided region coordinate and a divided region annotation;
optionally, the divided region annotation in the embodiment of the present invention includes but is not limited to: an input box, a single/multiple option, a pull-down option, a button, a handwritten signature box, a certificate photo page, etc.
Selecting a historical operation interface screenshot in the historical operation interface screenshot set, inputting the historical operation interface screenshot into a preset initial region division model, and obtaining a division region prediction coordinate and a division region annotation prediction value;
determining a real value of the divided region annotation according to the divided region annotation;
for example: the segmented region annotation is "input box", then the corresponding segmented region annotation true value is: an input box: 1.
calculating by using a preset first loss function according to the divided region annotation real value and the divided region annotation predicted value to obtain an annotation loss value;
calculating by using a preset second loss function according to the divided region coordinates and the predicted region coordinates to obtain region division loss values;
optionally, in this embodiment of the present invention, the first loss function and the second loss function may be: logarithmic loss functions, quadratic loss functions, cross-entropy loss functions, and the like.
Calculating according to the annotation loss value and the region division loss value to obtain a target loss value;
when the target loss value is larger than a preset loss threshold value, adjusting model parameters of the initial region division model, and returning to the step of selecting a historical operation interface screenshot in the historical operation interface screenshot set and inputting the historical operation interface screenshot into the preset initial region division model; and when the target loss value is not greater than a preset loss threshold value, outputting the initial region division model to obtain the region division model.
The image content description module 102 is configured to perform image content description on the annotation screenshot to obtain an image content text; marking a corresponding operation interface screenshot by using each image content text;
in detail, in the embodiment of the present invention, the Image content description module 102 performs Image content description on the annotation screenshot by using a preset Image description (Image capture) technology, so as to obtain the Image content text. Optionally, in the embodiment of the present invention, image content description is performed on the annotation screenshot by using a GRU, each divided region in the annotation screenshot is extracted to perform intent recognition, an intent text is obtained, the intent text is combined with the annotation corresponding to the divided region, and a divided region content text is obtained, for example: and the annotation corresponding to the divided area is a text box, the corresponding intention text is 'fill in personal information', the corresponding content text of the divided area is 'fill in personal information in the text box', and further, all the content texts of the divided area corresponding to the annotation screenshot are combined to obtain the image content text.
For example: and the annotation screenshot comprises a name input box and a product type radio box, and then the knowledge screenshot is subjected to image description, and the obtained image content text is that the name is input in the input box and a product type needing to be handled is selected.
In detail, in the embodiment of the present invention, in order to associate the image content text with the corresponding operation interface screenshot, and facilitate the user to perform image-text comparison and browsing, the image content description module 102 uses each image content text to label the corresponding operation interface screenshot. In detail, in the embodiment of the present invention, an annotation text box is constructed in the screenshot of the operation interface, and the image content text is input into the annotation text box to realize annotation of the screenshot of the operation interface.
The file generation module 103 is configured to combine and encapsulate all the labeled screenshots of the operation interface according to the sequence of the corresponding screenshot time, so as to obtain an operation guidance file.
In detail, in the embodiment of the present invention, the file generation module 103 performs picture splicing on all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time, so as to obtain an operation guidance picture; and converting the operation guide diagram into a preset file format to obtain the operation guide file. Optionally, the preset file format in the embodiment of the present invention may be a PDF format or a doc format.
Fig. 2 is a schematic structural diagram of an electronic device implementing the device operation guidance file generation method according to the present invention.
The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may further include a computer program, such as a device operation guidance file generating program, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The memory 11 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 may be used not only to store application software installed in the electronic device and various types of data, such as codes of the device operation guidance file generation program, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (e.g., device operation guidance file generation programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The communication bus 12 may be a PerIPheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
Fig. 2 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 2 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power source may also include any component of one or more dc or ac power sources, recharging devices, power failure classification circuits, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Optionally, the communication interface 13 may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.
Optionally, the communication interface 13 may further include a user interface, which may be a Display (Display), an input unit (such as a Keyboard (Keyboard)), and optionally, a standard wired interface, or a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The device operation guidance file generation program stored in the memory 11 of the electronic device is a combination of a plurality of computer programs, and when running in the processor 10, can realize:
acquiring an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time;
carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets;
performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
carrying out image content description on the annotation screenshot to obtain an image content text;
marking a corresponding operation interface screenshot by using each image content text;
and combining and packaging all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide file.
Specifically, the processor 10 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the computer program, which is not described herein again.
Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. The computer readable medium may be non-volatile or volatile. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
Embodiments of the present invention may also provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program may implement:
acquiring an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time;
carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets;
performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
carrying out image content description on the annotation screenshot to obtain an image content text;
marking a corresponding operation interface screenshot by using each image content text;
and combining and packaging all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide file.
Further, the computer usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method for generating a device operation guidance file, the method comprising:
acquiring an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time;
carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets;
performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
carrying out image content description on the annotation screenshot to obtain an image content text;
marking a corresponding operation interface screenshot by using each image content text;
and combining and packaging all the marked screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide file.
2. The method for generating the device operation guidance file according to claim 1, wherein the step of performing operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets comprises:
acquiring an operation interface title corresponding to the operation interface screenshot;
summarizing all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set;
and eliminating similar operation interface screenshots in the initial classification screenshot set to obtain the classification screenshot set.
3. The method for generating the device operation guidance file according to claim 2, wherein the obtaining the classification screenshot set by removing the similar operation interface screenshot in the initial classification screenshot set comprises:
converting each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;
calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain screenshot similarity;
and eliminating similar operation interface screenshots in the initial classification screenshot sets according to the screenshot similarity to obtain the corresponding classification screenshot sets.
4. The method for generating a device operation guidance file according to claim 2, wherein the converting each screenshot of the initial classification screenshot set into a vector to obtain a screenshot vector comprises:
selecting any one of the operation interface screenshots in the initial classification screenshot set and inputting the screenshot into a pre-constructed deep learning network model;
and acquiring output values of all nodes of the full-connection layer in the deep learning network model to construct vectors, and acquiring the corresponding screenshot vectors.
5. The method for generating the device operation guidance file according to claim 4, wherein the step of calculating the similarity between any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain the screenshot similarity comprises the steps of:
calculating the screenshot similarity by using the following formula:
wherein, XiThe i-th element, Y, representing the screenshot vector XiAnd the Sim represents the similarity between the screenshot vector X corresponding to the screenshot X of the operation interface and the screenshot vector Y corresponding to the screenshot Y of the operation interface, and the n represents the vector dimensions of the screenshot vector X and the screenshot vector Y.
6. The method for generating the device operation guidance file according to claim 1, wherein before performing image region division annotation on each operation interface screenshot in the classification screenshot set by using a preset region division model, the method further comprises:
acquiring a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set comprises a divided region coordinate and a divided region annotation;
selecting a historical operation interface screenshot in the historical operation interface screenshot set, inputting the historical operation interface screenshot into a preset initial region division model, and obtaining a division region prediction coordinate and a division region annotation prediction value;
determining a real value of the divided region annotation according to the divided region annotation;
calculating by using a preset first loss function according to the divided region annotation real value and the divided region annotation predicted value to obtain an annotation loss value;
calculating by using a preset second loss function according to the divided region coordinates and the predicted region coordinates to obtain region division loss values;
calculating according to the annotation loss value and the region division loss value to obtain a target loss value;
when the target loss value is larger than a preset loss threshold value, adjusting model parameters of the initial region division model, and returning to the step of selecting a historical operation interface screenshot in the historical operation interface screenshot set and inputting the historical operation interface screenshot into the preset initial region division model;
and when the target loss value is not greater than a preset loss threshold value, outputting the initial region division model to obtain the region division model.
7. The method for generating the device operation guidance file according to any one of claims 1 to 6, wherein the step of combining and packaging all the labeled screenshots of the operation interface according to the sequence of the corresponding screenshot time to obtain the operation guidance file comprises the steps of:
splicing all the marked screenshot of the operation interface according to the sequence of the corresponding screenshot time to obtain an operation guide picture;
and converting the operation guide diagram into a preset file format to obtain the operation guide file.
8. An apparatus for generating a device operation guidance file, comprising:
the screenshot annotation module is used for acquiring an operation interface screenshot set, wherein the operation interface screenshot set comprises a plurality of operation interface screenshots, and each operation interface screenshot comprises corresponding screenshot time; carrying out operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot sets to obtain a plurality of classification screenshot sets; performing image area division annotation on each operation interface screenshot in the classification screenshot set to obtain an annotation screenshot;
the image content description module is used for carrying out image content description on the annotation screenshot to obtain an image content text; marking a corresponding operation interface screenshot by using each image content text;
and the file generation module is used for combining and packaging all the marked screenshot of the operation interface according to the sequence of the corresponding screenshot time to obtain the operation guide file.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the apparatus operation guidance file generating method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the device operation guidance file generating method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111149207.1A CN113822215A (en) | 2021-09-29 | 2021-09-29 | Equipment operation guide file generation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111149207.1A CN113822215A (en) | 2021-09-29 | 2021-09-29 | Equipment operation guide file generation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113822215A true CN113822215A (en) | 2021-12-21 |
Family
ID=78915911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111149207.1A Pending CN113822215A (en) | 2021-09-29 | 2021-09-29 | Equipment operation guide file generation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113822215A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114554133A (en) * | 2022-02-22 | 2022-05-27 | 联想(北京)有限公司 | Information processing method and device and electronic equipment |
-
2021
- 2021-09-29 CN CN202111149207.1A patent/CN113822215A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114554133A (en) * | 2022-02-22 | 2022-05-27 | 联想(北京)有限公司 | Information processing method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112528616B (en) | Service form generation method and device, electronic equipment and computer storage medium | |
| CN113157927B (en) | Text classification method, apparatus, electronic device and readable storage medium | |
| CN113704614A (en) | Page generation method, device, equipment and medium based on user portrait | |
| CN113961473A (en) | Data testing method and device, electronic equipment and computer readable storage medium | |
| CN112988963A (en) | User intention prediction method, device, equipment and medium based on multi-process node | |
| CN112860905A (en) | Text information extraction method, device and equipment and readable storage medium | |
| CN115917613A (en) | Semantic representation of text in a document | |
| CN114398557A (en) | Information recommendation method and device based on double portraits, electronic equipment and storage medium | |
| CN114491047A (en) | Multi-label text classification method and device, electronic equipment and storage medium | |
| CN113868528A (en) | Information recommendation method and device, electronic equipment and readable storage medium | |
| CN113822215A (en) | Equipment operation guide file generation method and device, electronic equipment and storage medium | |
| CN113626605B (en) | Information classification method, device, electronic equipment and readable storage medium | |
| CN113704474B (en) | Bank outlet equipment operation guide generation method, device, equipment and storage medium | |
| CN115982454A (en) | User portrait based questionnaire pushing method, device, equipment and storage medium | |
| CN115146064A (en) | Intention recognition model optimization method, device, equipment and storage medium | |
| CN114708461A (en) | Multi-modal learning model-based classification method, device, equipment and storage medium | |
| CN114996386A (en) | Business role identification method, device, equipment and storage medium | |
| CN114385815A (en) | News screening method, device, equipment and storage medium based on business requirements | |
| CN113343102A (en) | Data recommendation method and device based on feature screening, electronic equipment and medium | |
| CN113486646A (en) | Product report issuing method and device, electronic equipment and readable storage medium | |
| CN113313211A (en) | Text classification method and device, electronic equipment and storage medium | |
| CN113434650B (en) | Question-answer pair expansion method and device, electronic equipment and readable storage medium | |
| CN115204158B (en) | Data isolation application method and device, electronic equipment and storage medium | |
| CN114185617B (en) | Service call interface configuration method, device, equipment and storage medium | |
| CN113221888B (en) | License plate number management system test method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |