CN112231032A - Software interface element access method and device combining RPA and AI - Google Patents
Software interface element access method and device combining RPA and AI Download PDFInfo
- Publication number
- CN112231032A CN112231032A CN202011125498.6A CN202011125498A CN112231032A CN 112231032 A CN112231032 A CN 112231032A CN 202011125498 A CN202011125498 A CN 202011125498A CN 112231032 A CN112231032 A CN 112231032A
- Authority
- CN
- China
- Prior art keywords
- interface
- anchor point
- information
- interface element
- software
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000010586 diagram Methods 0.000 claims abstract description 61
- 238000012015 optical character recognition Methods 0.000 claims abstract description 21
- 238000005516 engineering process Methods 0.000 claims abstract description 15
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 5
- 238000013473 artificial intelligence Methods 0.000 description 25
- 238000013461 design Methods 0.000 description 14
- 238000004801 process automation Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000001788 irregular Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
Abstract
The disclosure provides an access method, device, equipment and storage medium for software interface elements combining RPA and AI. The method for accessing the software interface element provided by the embodiment comprises the steps of obtaining a template characteristic diagram corresponding to a current software interface; determining target anchor point information on the current software interface by adopting an Optical Character Recognition (OCR) technology according to the template feature map; determining the distribution information of at least one interface element on the current software interface according to the target anchor point information; and executing access operation on the interface element according to the distribution information. Therefore, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Description
Technical Field
The present disclosure relates to the field of Automation technologies, and in particular, to an RPA (robot Process Automation) and an AI (Artificial Intelligence), and more particularly, to an access method and an access device for software interface elements that combine an RPA and an AI.
Background
In the field of Robot Process Automation (RPA), in order to implement Automation of a Process, a software robot needs to frequently access control elements (interface elements for short) on a software interface and operate on the interface elements to execute corresponding operation tasks.
Artificial Intelligence (AI) is a new technology science for researching and developing theories, methods, techniques and application systems for simulating, extending and expanding human Intelligence. Research in the field of artificial intelligence includes robotics, language recognition, image recognition, natural language processing, and expert systems, among others.
In the prior art, a software robot generally acquires software Interface elements through a Microsoft User Interface Automation (UIA) Interface, and then executes corresponding operation tasks.
However, when the software interface does not support UIA interface access, the software robot cannot normally access the software interface elements.
Disclosure of Invention
The disclosure provides an access method, an access device, an access equipment and a storage medium for software interface elements combining RPA and AI, which can access interface elements on a software interface which does not support UIA interface, and improve the access capability of a software robot.
In a first aspect, the present disclosure provides an access method for a software interface element combining RPA and AI, including:
acquiring a template characteristic diagram corresponding to a current software interface;
determining target anchor point information on the current software interface by adopting an Optical Character Recognition (OCR) technology according to the template feature map;
determining the distribution information of at least one interface element on the current software interface according to the target anchor point information;
and executing access operation on the interface element according to the distribution information.
In a possible design, the obtaining a template feature map corresponding to a current software interface includes:
searching a template characteristic diagram matched with the software identification information from a memory according to the software identification information of the current software interface; the memory is pre-loaded with a template characteristic diagram and a mapping relation between the template characteristic diagram and software identification information; alternatively, the first and second electrodes may be,
and searching a template characteristic diagram matched with the image of the current software interface from a memory according to the image of the current software interface.
In one possible design, the target anchor information includes: the anchor point area, the association information between the anchor point area and the interface element; wherein the anchor region comprises: any one or more of a form-invariant icon element, a text element, a key element.
In one possible design, determining distribution information of at least one interface element on the current software interface according to the target anchor point information includes:
determining the distribution information of at least one interface element on the current software interface according to the anchor point area, the association information between the anchor point area and the interface element; the distribution information includes: coordinate information of at least one shape point of the interface element, and size information of the interface element; wherein the shape points are used for defining the area contained by the interface element; the associated information includes: and the coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element.
In one possible design, before obtaining the template feature map corresponding to the current software interface, the method further includes:
and generating a template characteristic diagram according to the image of the software interface.
In one possible design, the generating a template feature map according to an image of a software interface includes:
when the software is operated, intercepting an image of a software interface;
marking an anchor point area and interface elements in the image of the software interface;
determining association information between the anchor point area and the interface element according to the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element;
and generating a template characteristic diagram corresponding to the software according to the image of the software interface, the marked anchor point area and the interface element and the associated information between the anchor point area and the interface element.
In a second aspect, the present disclosure also provides an access device for software interface elements that combine RPA and AI, including:
the acquisition module is used for acquiring a template characteristic diagram corresponding to the current software interface;
a first determining module, configured to determine, according to the template feature map, target anchor point information on the current software interface by using an Optical Character Recognition (OCR) technique;
the second determining module is used for determining the distribution information of at least one interface element on the current software interface according to the target anchor point information;
and the access module is used for executing access operation on the interface element according to the distribution information.
In one possible design, the obtaining module is specifically configured to:
searching a template characteristic diagram matched with the software identification information from a memory according to the software identification information of the current software interface; the memory is pre-loaded with a template characteristic diagram and a mapping relation between the template characteristic diagram and software identification information; alternatively, the first and second electrodes may be,
and according to the image of the current software interface, the template feature map matched with the image of the current software interface is stored in a memory.
In one possible design, the target anchor information includes: the anchor point area, the association information between the anchor point area and the interface element; wherein the anchor region comprises: any one or more of a form-invariant icon element, a text element, a key element.
In one possible design, the second determining module is specifically configured to:
determining the distribution information of at least one interface element on the current software interface according to the anchor point area, the association information between the anchor point area and the interface element; the distribution information includes: coordinate information of at least one shape point of the interface element, and size information of the interface element; wherein the shape points are used for defining the area contained by the interface element; the associated information includes: and the coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element.
In one possible design, further comprising: a template generation module to:
and generating a template characteristic diagram according to the image of the software interface.
In one possible design, the template generation module: the method is specifically used for:
when the software is operated, intercepting an image of a software interface;
marking an anchor point area and interface elements in the image of the software interface;
determining association information between the anchor point area and the interface element according to the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element;
and generating a template characteristic diagram corresponding to the software according to the image of the software interface, the marked anchor point area and the interface element and the associated information between the anchor point area and the interface element.
In a third aspect, the present disclosure also provides an electronic device, including:
a processor; and the number of the first and second groups,
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform any one of the methods of the first aspect of accessing software interface elements in conjunction with RPA and AI via execution of the executable instructions.
In a fourth aspect, the disclosed embodiments also provide a storage medium, on which a computer program is stored, where the program, when executed by a processor, implements any one of the methods for accessing a software interface element combining an RPA and an AI in the first aspect.
The disclosure provides an access method, device, equipment and storage medium for software interface elements by combining RPA and AI, which is implemented by acquiring a template characteristic diagram corresponding to a current software interface; determining target anchor point information on the current software interface by adopting an OCR technology according to the template characteristic diagram; determining the distribution information of at least one interface element on the current software interface according to the target anchor point information; and executing access operation on the interface element according to the distribution information. Therefore, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
FIG. 1 is a diagram illustrating an application scenario of an access method for software interface elements that combine RPA and AI according to an example embodiment of the present disclosure;
FIG. 2 is a flow diagram illustrating a method for accessing software interface elements that combine RPA and AI according to an example embodiment of the present disclosure;
FIG. 3 is a flow diagram illustrating a method of accessing software interface elements that combine RPA and AI according to another example embodiment of the present disclosure;
FIG. 4 is a block diagram illustrating an access device that combines software interface elements of an RPA and an AI according to an example embodiment of the present disclosure;
FIG. 5 is a block diagram illustrating an access device that combines software interface elements of an RPA and an AI according to another example embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device shown in the present disclosure according to an example embodiment.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present disclosure and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In the field of Robot Process Automation (RPA), in order to implement Automation of a Process, a software robot needs to frequently access control elements (interface elements for short) on a software interface and operate on the interface elements to execute corresponding operation tasks. In the prior art, a software robot generally acquires software Interface elements through a Microsoft User Interface Automation (UIA) Interface, and then executes corresponding operation tasks. However, when the software interface does not support UIA interface access, the software robot cannot normally access the software interface elements.
In view of the above technical problems, the present disclosure provides an access method, an access device, an access apparatus, and a storage medium for software interface elements in combination with an RPA and an AI, which can perform access processing on interface elements on a software interface that does not support a UIA interface, and improve the access capability of a software robot. Fig. 1 is a diagram illustrating an application scenario of an access method of a software interface element combining an RPA and an AI according to an example embodiment of the present disclosure, as shown in fig. 1, since a relative position of a control element in an interface or a size and a shape of the control element itself may be dynamically changed. For example, the chat input box area can be arbitrarily dragged to be enlarged or reduced in a PC edition WeChat interface. Therefore, in the embodiment of the present disclosure, a stable and unchangeable region in the software interface may be labeled as an anchor region. The anchor point area can be any one or more of icon elements, text elements and key elements with unchanged shapes, but the anchor point area is only and stably searchable in the interface; the number of anchor regions is not limited and is determined by the complexity of the software interface. In particular, an anchor region may be understood as a reference region whose form must remain unchanged. For example, if a circular icon is used as the anchor point region, the shape of the icon is necessarily circular regardless of where the icon is displayed on the interface. Besides the common elements such as icons, text characters on the interface can be set as anchor areas, for example, text characters are generally available in the lower right corner of an input box, such as "confirm", "cancel", "send", and the like, and the areas corresponding to the text characters can be used as anchor areas to define the positions of the input box. Similarly, the key element may be a control with a certain shape, and may also be an anchor point region for positioning the distribution region of the interface element as long as the shape of the key element remains unchanged. It should be noted that, the number and type of anchor points can be selected by those skilled in the art according to the actual software interface. For example, when the area of an interface element on a software interface is large and the area of the interface element is irregular, more anchor points are needed to define the interface element. In addition, the icon element, the text element or the key element corresponding to the anchor area needs to be uniquely present on the interface, for example, if two icon elements with the same shape or two text elements with the same text exist on the interface, the element area corresponding to the icon element or the text element cannot be used as the anchor area. The target anchor information includes, in addition to anchor region information, association information between the anchor region and the interface element. Target anchor point information is recorded in the template characteristic diagram, and the target anchor point information in the template characteristic diagram can be directly extracted. As in the PC-version wechat interface shown in fig. 1, the emoticon element may be used as an anchor point 1, and the send button may be used as an anchor point 2. The control area is an interface element which the robot needs to access, such as a chat information input box area in a PC edition WeChat interface. The distribution information of the interface element may be described by a shape point, which may be a vertex of the interface element or a center point of the interface element. The distribution information of the rectangular interface element may be described by four vertices, and the distribution information of the circular interface element may be described by a center point. For example, a circular interface element (circular button), knowing the center position and the radius of the circle, the area of the interface element can be determined. According to the coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element, the coordinate of the shape point of the interface element can be quickly determined, and further the information such as the position coordinate, the size and the like of the interface element is determined. For example, in the PC-version wechat interface, the position information of the shape point at the upper left corner of the control area can be determined according to the area information of the anchor point 1, and the position information of the shape point at the lower right corner of the control area can be determined according to the area information of the anchor point 2, so as to determine the distribution information of the control area. After the distribution information of the interface elements is acquired, the interface elements can be accessed, for example, the interface elements are picked up and simulated.
By applying the method, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Fig. 2 is a flowchart illustrating an access method of a software interface element combining an RPA and an AI according to an example embodiment of the present disclosure, and as shown in fig. 2, the method provided in this embodiment may include:
In this embodiment, the software robot may search the template feature map matched with the software identification information from the memory according to the software identification information of the current software interface; the device comprises a memory, a control module, a display module and a display module, wherein the memory is pre-loaded with a template characteristic diagram and a mapping relation between the template characteristic diagram and software identification information; alternatively, the software robot may search the memory for a template feature map matching the image of the current software interface according to the image of the current software interface.
Specifically, in one embodiment, the template feature map and the identification information of the software interface may be stored in the memory, and the corresponding template feature map may be searched and obtained through the identification information of the current software interface. The identification information of the software interface may include an application name, interface information such as a home page, a dialog interface, and the like, and information such as a software version number. In another embodiment, the image recognition technology can search the template feature map matched with the image of the current software interface from the memory according to the image of the current software interface.
It should be noted that, in this embodiment, the manner of obtaining the template feature map is not limited, and the template feature map may be obtained through identification information of a software interface, or may be matched through an image recognition technology, or may be a combination of the two.
And step 102, determining target anchor point information on the current software interface by adopting an Optical Character Recognition (OCR) technology according to the template characteristic diagram.
In this embodiment, the target anchor point information includes: the anchor point area, the association information between the anchor point area and the interface element; wherein, the anchor point region includes: any one or more of a form-invariant icon element, a text element, a key element.
Specifically, a form-invariant region in the software interface may be labeled as an anchor region. The anchor point area can be any one or more of icon elements, text elements and key elements with unchanged shapes, but the anchor point area is only and stably searchable in the interface; the number of anchor regions is not limited and is determined by the complexity of the software interface. The target anchor information includes, in addition to anchor region information, association information between the anchor region and the interface element. Target anchor point information is recorded in the template characteristic diagram, and the target anchor point information in the template characteristic diagram can be directly extracted.
In this embodiment, the target anchor point information may be recorded in the template feature map in a text form, for example, an anchor point region in the template feature map is marked in a text form, and association information between the anchor point region and the interface element is recorded, so that after the software robot acquires the template feature map, the software robot may extract the target anchor point information from the template feature map by using an OCR technology.
And 103, determining the distribution information of at least one interface element on the current software interface according to the target anchor point information.
In this embodiment, determining distribution information of at least one interface element on the current software interface according to the target anchor point information includes: determining the distribution information of at least one interface element on the current software interface according to the anchor point area, the association information between the anchor point area and the interface element; the distribution information includes: coordinate information of at least one shape point of the interface element, and size information of the interface element; wherein, the shape point is used for defining the area contained by the interface element; the associated information includes: and coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element.
In particular, the relative position of the control element in the interface or the size and shape of the control element itself may be dynamically changed. For example, the chat input box area can be arbitrarily dragged to be enlarged or reduced in a PC edition WeChat interface. Therefore, the association information between the anchor point area and the interface element indicates the relationship between the control element and the anchor point, and the position and the size of the control element are determined by the stable and unchangeable anchor point. The distribution information of the interface element may be described by a shape point, which may be a vertex of the interface element or a center point of the interface element. The distribution information of the rectangular interface element may be described by four vertices, and the distribution information of the circular interface element may be described by a center point. For example, a circular interface element (circular button), knowing the center position and the radius of the circle, the area of the interface element can be determined. According to the coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element, the coordinate of the shape point of the interface element can be quickly determined, and further the information such as the position coordinate, the size and the like of the interface element is determined.
And 104, executing access operation on the interface element according to the distribution information.
In this embodiment, after the distribution information of the interface element is acquired, the interface element may be accessed, for example, the interface element is picked up and simulated.
In this embodiment, a template feature map corresponding to a current software interface is obtained; determining target anchor point information on a current software interface by adopting an OCR technology according to the template characteristic diagram; determining the distribution information of at least one interface element on the current software interface according to the target anchor point information; and according to the distribution information, performing access operation on the interface element. Therefore, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Fig. 3 is a flowchart illustrating a method for accessing a software interface element by combining an RPA and an AI according to another exemplary embodiment of the present disclosure, and as shown in fig. 3, the method provided in this embodiment may include:
In the embodiment, when the software is operated, an image of a software interface is intercepted; marking an anchor point area and interface elements in an image of a software interface; determining association information between the anchor point area and the interface element according to the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element; and generating a template characteristic diagram corresponding to the software according to the image of the software interface, the labeled anchor point area and the interface element and the associated information between the anchor point area and the interface element.
Specifically, the software interface can be intercepted, and the template characteristic diagram corresponding to the software interface is generated in a manual or automatic labeling mode. In the implementation process, aiming at a software interface to be accessed by the robot, firstly, an image of the interface is intercepted, and the generation of a custom template is realized on the basis of the image. Then, the anchor region is labeled. The step is to mark the region with unchanged shape in the software interface as the anchor region. The anchor area may be an icon or text, but the anchor area must be unique and stably searchable in the interface; the number of anchor regions is not limited and is determined by the complexity of the software interface. Then, the control element area is labeled. The areas where the control elements in the software interface are located can be marked, and the areas are the interface elements which the robot needs to access. And then, establishing the incidence relation between the control element and the anchor point. The relative position of the control element in the interface or the size and shape of the control element can be dynamically changed. For example, the chat input box area can be arbitrarily dragged to be enlarged or reduced in a PC edition WeChat interface. Therefore, the control element needs to be related to the anchor point, and the position and the size of the control element are determined by the stable and unchangeable anchor point. And finally, generating a template characteristic diagram corresponding to the software according to the image of the software interface, the marked anchor point area and the associated information between the anchor point area and the interface element.
In this embodiment, when information such as an anchor point region, a region where a control element is located, and association information between the anchor point region and the interface element is marked in a software interface, the information may be marked in a text form, so that the generated template feature map has target anchor point information in a text form.
And step 203, determining the target anchor point information on the current software interface by adopting an OCR technology according to the template characteristic diagram.
And 204, determining the distribution information of at least one interface element on the current software interface according to the target anchor point information.
And step 205, executing an access operation on the interface element according to the distribution information.
In this embodiment, please refer to the related description in step 101 to step 104 in the method shown in fig. 2 for the specific implementation process and technical principle of step 202 to step 205, which is not described herein again.
In this embodiment, a template feature map corresponding to a current software interface is obtained; determining target anchor point information on a current software interface by adopting an OCR technology according to the template characteristic diagram; determining the distribution information of at least one interface element on the current software interface according to the target anchor point information; and according to the distribution information, performing access operation on the interface element. Therefore, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
In addition, the embodiment can also generate the template characteristic diagram according to the image of the software interface, so that the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Fig. 4 is a schematic structural diagram illustrating an access device of a software interface element combining an RPA and an AI according to an example embodiment of the present disclosure. As shown in fig. 4, the access device for software interface elements combining RPA and AI according to this embodiment may include:
the obtaining module 31 is configured to obtain a template feature map corresponding to a current software interface;
a first determining module 32, configured to determine, according to the template feature map, target anchor point information on the current software interface by using an Optical Character Recognition (OCR) technique;
a second determining module 33, configured to determine, according to the target anchor point information, distribution information of at least one interface element on the current software interface;
and the access module 34 is configured to perform an access operation on the interface element according to the distribution information.
In one possible design, the obtaining module 31 is specifically configured to:
searching a template characteristic diagram matched with the software identification information from a memory according to the software identification information of the current software interface; the device comprises a memory, a control module, a display module and a display module, wherein the memory is pre-loaded with a template characteristic diagram and a mapping relation between the template characteristic diagram and software identification information; alternatively, the first and second electrodes may be,
and according to the image of the current software interface, matching the template characteristic graph with the image of the current software interface from the memory.
In one possible design, the target anchor information includes: the anchor point area, the association information between the anchor point area and the interface element; wherein, the anchor point region includes: any one or more of a form-invariant icon element, a text element, a key element.
In one possible design, the second determining module 33 is specifically configured to:
determining the distribution information of at least one interface element on the current software interface according to the anchor point area, the association information between the anchor point area and the interface element; the distribution information includes: coordinate information of at least one shape point of the interface element, and size information of the interface element; wherein, the shape point is used for defining the area contained by the interface element; the associated information includes: and coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element.
The apparatus provided in this embodiment may be used to implement the technical solution of the method embodiment shown in fig. 2, and the implementation principle and the technical effect are similar, which are not described herein again.
In this embodiment, a template feature map corresponding to a current software interface is obtained; determining target anchor point information on a current software interface by adopting an OCR technology according to the template characteristic diagram; determining the distribution information of at least one interface element on the current software interface according to the target anchor point information; and according to the distribution information, performing access operation on the interface element. Therefore, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Based on the embodiment shown in fig. 4, fig. 5 is a schematic structural diagram of an access apparatus for combining software interface elements of an RPA and an AI according to another exemplary embodiment of the present disclosure, and as shown in fig. 5, the access apparatus for combining software interface elements of an RPA and an AI provided in this embodiment further includes:
a template generation module 35 configured to:
and generating a template characteristic diagram according to the image of the software interface.
In one possible design, template generation module 35: the method is specifically used for:
when the software is operated, intercepting an image of a software interface;
marking an anchor point area and interface elements in an image of a software interface;
determining association information between the anchor point area and the interface element according to the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element;
and generating a template characteristic diagram corresponding to the software according to the image of the software interface, the labeled anchor point area and the interface element and the associated information between the anchor point area and the interface element.
The apparatus provided in this embodiment may be used to implement the technical solutions of the method embodiments shown in fig. 2 and fig. 3, and the implementation principles and technical effects are similar, which are not described herein again.
In this embodiment, a template feature map corresponding to a current software interface is obtained; determining target anchor point information on a current software interface by adopting an OCR technology according to the template characteristic diagram; determining the distribution information of at least one interface element on the current software interface according to the target anchor point information; and according to the distribution information, performing access operation on the interface element. Therefore, the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
In addition, the embodiment can also generate the template characteristic diagram according to the image of the software interface, so that the interface elements on the software interface which does not support the UIA interface can be accessed, the access capability of the software robot is improved, the implementation mode is simple, and the result is stable and reliable.
Fig. 6 is a schematic structural diagram of an electronic device shown in the present disclosure according to an example embodiment. As shown in fig. 6, the present embodiment provides an electronic device 40, including:
a processor 401; and the number of the first and second groups,
a memory 402 for storing executable instructions of the processor, which may also be a flash (flash memory);
wherein the processor 401 is configured to perform the respective steps of the above-described method via execution of executable instructions. Reference may be made in particular to the description relating to the preceding method embodiment.
Alternatively, the memory 402 may be separate or integrated with the processor 401.
When the memory 402 is a device independent of the processor 401, the electronic device 40 may further include:
a bus 403 for connecting the processor 401 and the memory 402.
The present embodiment also provides a readable storage medium, in which a computer program is stored, and when at least one processor of the electronic device executes the computer program, the electronic device executes the methods provided by the above various embodiments.
The present embodiment also provides a program product comprising a computer program stored in a readable storage medium. The computer program can be read from a readable storage medium by at least one processor of the electronic device, and the execution of the computer program by the at least one processor causes the electronic device to implement the methods provided by the various embodiments described above.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present disclosure, and not for limiting the same; while the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.
Claims (13)
1. A method for accessing software interface elements by combining RPA and AI, comprising:
acquiring a template characteristic diagram corresponding to a current software interface;
determining target anchor point information on the current software interface by adopting an Optical Character Recognition (OCR) technology according to the template feature map;
determining the distribution information of at least one interface element on the current software interface according to the target anchor point information;
and executing access operation on the interface element according to the distribution information.
2. The method according to claim 1, wherein the obtaining of the template feature map corresponding to the current software interface comprises:
searching a template characteristic diagram matched with the software identification information from a memory according to the software identification information of the current software interface; the memory is pre-loaded with a template characteristic diagram and a mapping relation between the template characteristic diagram and software identification information; alternatively, the first and second electrodes may be,
and searching a template characteristic diagram matched with the image of the current software interface from a memory according to the image of the current software interface.
3. The method of claim 1, wherein the target anchor information comprises: the anchor point area, the association information between the anchor point area and the interface element; wherein the anchor region comprises: any one or more of a form-invariant icon element, a text element, a key element.
4. The method of claim 3, wherein determining distribution information of at least one interface element on a current software interface according to the target anchor point information comprises:
determining the distribution information of at least one interface element on the current software interface according to the anchor point area, the association information between the anchor point area and the interface element; the distribution information includes: coordinate information of at least one shape point of the interface element, and size information of the interface element; wherein the shape points are used for defining the area contained by the interface element; the associated information includes: and the coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element.
5. The method according to any one of claims 1-4, further comprising, before obtaining the template feature map corresponding to the current software interface:
and generating a template characteristic diagram according to the image of the software interface.
6. The method of claim 5, wherein generating the template feature map from the image of the software interface comprises:
when the software is operated, intercepting an image of a software interface;
marking an anchor point area and interface elements in the image of the software interface;
determining association information between the anchor point area and the interface element according to the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element;
and generating a template characteristic diagram corresponding to the software according to the image of the software interface, the marked anchor point area and the interface element and the associated information between the anchor point area and the interface element.
7. An access device for software interface elements that combine RPA and AI, comprising:
the acquisition module is used for acquiring a template characteristic diagram corresponding to the current software interface;
a first determining module, configured to determine, according to the template feature map, target anchor point information on the current software interface by using an Optical Character Recognition (OCR) technique;
the second determining module is used for determining the distribution information of at least one interface element on the current software interface according to the target anchor point information;
and the access module is used for executing access operation on the interface element according to the distribution information.
8. The apparatus of claim 7, wherein the target anchor information comprises: the anchor point area, the association information between the anchor point area and the interface element; wherein the anchor region comprises: any one or more of a form-invariant icon element, a text element, a key element.
9. The apparatus of claim 8, wherein the second determining module is specifically configured to:
determining the distribution information of at least one interface element on the current software interface according to the anchor point area, the association information between the anchor point area and the interface element; the distribution information includes: coordinate information of at least one shape point of the interface element, and size information of the interface element; wherein the shape points are used for defining the area contained by the interface element; the associated information includes: and the coordinate conversion relation between the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element.
10. The apparatus of any one of claims 7-9, further comprising: a template generation module to:
and generating a template characteristic diagram according to the image of the software interface.
11. The apparatus of claim 10, wherein the template generation module: the method is specifically used for:
when the software is operated, intercepting an image of a software interface;
marking an anchor point area and interface elements in the image of the software interface;
determining association information between the anchor point area and the interface element according to the coordinate information corresponding to the anchor point area and the coordinate information corresponding to the interface element;
and generating a template characteristic diagram corresponding to the software according to the image of the software interface, the marked anchor point area and the interface element and the associated information between the anchor point area and the interface element.
12. An electronic device, comprising:
a processor; and the number of the first and second groups,
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of accessing the software interface element that combines RPA and AI of any of claims 1-6 via execution of the executable instructions.
13. A storage medium on which a computer program is stored, which program, when executed by a processor, implements the method of accessing a software interface element that combines RPA and AI according to any one of claims 1 to 6.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911254717 | 2019-12-10 | ||
| CN2019112547178 | 2019-12-10 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112231032A true CN112231032A (en) | 2021-01-15 |
Family
ID=74118574
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011125498.6A Pending CN112231032A (en) | 2019-12-10 | 2020-10-20 | Software interface element access method and device combining RPA and AI |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112231032A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113176921A (en) * | 2021-04-30 | 2021-07-27 | 上海云扩信息科技有限公司 | Automatic customization system of desktop |
| CN115455227A (en) * | 2022-09-20 | 2022-12-09 | 上海弘玑信息技术有限公司 | Graphical interface element searching method, electronic device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110119609A1 (en) * | 2009-11-16 | 2011-05-19 | Apple Inc. | Docking User Interface Elements |
| CN104899146A (en) * | 2015-06-19 | 2015-09-09 | 安一恒通（北京）科技有限公司 | Image matching technology based software stability test method and device |
| CN107463327A (en) * | 2017-07-20 | 2017-12-12 | 福建网龙计算机网络信息技术有限公司 | A kind of method and terminal for obtaining interface control element position information |
| CN109408367A (en) * | 2018-09-03 | 2019-03-01 | 深圳壹账通智能科技有限公司 | A kind of method and terminal of the control element identifying interactive interface |
| CN110413529A (en) * | 2019-07-31 | 2019-11-05 | 中国工商银行股份有限公司 | Applied to the test method of electronic equipment, device, calculate equipment and medium |
-
2020
- 2020-10-20 CN CN202011125498.6A patent/CN112231032A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110119609A1 (en) * | 2009-11-16 | 2011-05-19 | Apple Inc. | Docking User Interface Elements |
| CN104899146A (en) * | 2015-06-19 | 2015-09-09 | 安一恒通（北京）科技有限公司 | Image matching technology based software stability test method and device |
| CN107463327A (en) * | 2017-07-20 | 2017-12-12 | 福建网龙计算机网络信息技术有限公司 | A kind of method and terminal for obtaining interface control element position information |
| CN109408367A (en) * | 2018-09-03 | 2019-03-01 | 深圳壹账通智能科技有限公司 | A kind of method and terminal of the control element identifying interactive interface |
| CN110413529A (en) * | 2019-07-31 | 2019-11-05 | 中国工商银行股份有限公司 | Applied to the test method of electronic equipment, device, calculate equipment and medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113176921A (en) * | 2021-04-30 | 2021-07-27 | 上海云扩信息科技有限公司 | Automatic customization system of desktop |
| CN113176921B (en) * | 2021-04-30 | 2023-09-19 | 上海云扩信息科技有限公司 | Desktop automatic customizing system |
| CN115455227A (en) * | 2022-09-20 | 2022-12-09 | 上海弘玑信息技术有限公司 | Graphical interface element searching method, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20140099002A1 (en) | Apparatus and method for providing digital drawing | |
| CN112231033A (en) | Software interface element matching method and device combining RPA and AI | |
| US20140068553A1 (en) | Wireframe Recognition and Analysis Engine | |
| CN112231032A (en) | Software interface element access method and device combining RPA and AI | |
| US9355338B2 (en) | Image recognition device, image recognition method, and recording medium | |
| JP2017111731A (en) | Information processing system, information processing method, and program | |
| EP3913532A2 (en) | Object area measurement method, apparatus, storage medium and computer product | |
| CN112231034A (en) | Software interface element identification method and device combining RPA and AI | |
| CN112288882A (en) | Information display method and device, computer equipment and storage medium | |
| CN109858402B (en) | Image detection method, device, terminal and storage medium | |
| CN110672102B (en) | Visual auxiliary robot initialization positioning method, robot and readable storage medium | |
| CN115035367A (en) | Picture identification method and device and electronic equipment | |
| CN112817447B (en) | AR content display method and system | |
| CN117033309A (en) | Data conversion method and device, electronic equipment and readable storage medium | |
| CN115359203B (en) | Three-dimensional high-precision map generation method, system and cloud platform | |
| JP2020013567A (en) | Image processing method, image processing device, and image processing apparatus | |
| Mooser et al. | An augmented reality interface for mobile information retrieval | |
| CN108874360B (en) | Panoramic content positioning method and device | |
| CN114511653A (en) | Progress tracking with automatic symbol detection | |
| CN110909739B (en) | Picture identification and operation method and device, computer equipment and storage medium | |
| CN108733304A (en) | A kind of automatic identification and processing hand-written character method, apparatus | |
| JP7359207B2 (en) | Polygon search method | |
| CN112911266A (en) | Implementation method and system of Internet of things practical training system based on augmented reality technology | |
| Satish et al. | ARTSAM: Augmented Reality App for Tool Selection in Aircraft Maintenance | |
| CN115455227B (en) | Element searching method of graphical interface, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |