WO2021087818A1 - Method, apparatus and system for capturing knowledge in software - Google Patents

Abstract

A method for capturing knowledge in software, the method comprising the following steps: S1, acquiring a captured screen of the software and identifying a target and pixel coordinates thereof on the basis of the captured screen, wherein the target comprises a plurality of functional regions, icons or characters on the captured screen, iteratively performing step S1 to acquire a list of maximum values and minimum values on an X-coordinate axis and a Y-coordinate axis respectively corresponding to the target and the pixels thereof, and storing the list in a first database, and then performing the following steps: S2, capturing, online, a software-based event on a software operation apparatus, and mapping the event into the list to acquire a software operation corresponding to the event; and S3, extracting an event stream-based knowledge graph on the basis of a plurality of software operations, and storing the knowledge graph in a second database. The method is high in speed and efficiency, has no impact on the operations of a user with regards to the software, and is applicable to various types of software, in particular modeling software.

Description

Software knowledge capture method, device and system Technical field

The present invention relates to the field of software, in particular to a method, device and system for capturing modeling knowledge.

Background technique

A large amount of useful knowledge data is generated during the user's operation of the software, and it is very meaningful to obtain and use such knowledge data. For example, a large amount of knowledge data is generated in the process of engineering modeling. These knowledge data, especially expert knowledge, are actually very useful. This is because based on this knowledge can help junior engineers provide reference and guidance in their modeling work, so it can improve the efficiency and quality of modeling work. The above process is called knowledge capture and reuse.

Nowadays, there are three main ways to capture knowledge. The first way is to manually input and construct knowledge. Specifically, the necessary knowledge is acquired by interviewing experts, and then the information is constructed manually, but this is a waste of time and requires the cooperation of experts. The second way is to get information from the log file. Specifically, software event log files are generally regarded as the most important resource, which provides first-hand process information. However, most of the time, business modeling software does not provide users with these record files, so it is impossible for users to identify and capture knowledge in this way. The third way is to develop the capture function based on the application program interface. Specifically, some software can provide an application program interface, which can be used to develop customized functions, so this is another function that recognizes knowledge and captures it.

Therefore, the problems that need to be solved in the prior art for knowledge capture of modeling are mainly: some modeling software does not provide users with record files including modeling process information, nor does it provide users with application program interfaces to capture modeling process information. . and. The development of the user interface is represented by graphs, not script files, which makes it more difficult to identify and capture knowledge in the modeling process.

Summary of the invention

The first aspect of the present invention provides a knowledge capture method for music software, which includes the following steps: S1, acquiring a screenshot of the software and performing recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes the screenshot Iteratively execute step S1 to obtain a list of the target and its pixels corresponding to the maximum and minimum values on the X-coordinate axis and the Y-coordinate axis, and save it in the first database. , And then perform the following steps: S2, online capture of events based on the software on the software operating device, and map the events in the list to obtain the software operations corresponding to the events; S3, extract the events based on multiple software operations The knowledge graph of the stream is stored in the second database.

Further, the pixel coordinates are the coordinates of the target relative to the window where the target is located.

Further, the step S1 also includes the following steps: S11, acquiring a screenshot of the software and converting the screenshot into a gray image; S12, performing image segmentation on the screenshot of the gray image based on each target, so as to divide the Each target is positioned in the gray image to obtain the pixel coordinates of the target; S13, using an image matching algorithm to recognize an image-based target, and using an optical character recognition function to recognize a text-based target.

Further, the step S1 further includes the following step: capturing a simulation operation based on the software on the software operating device to obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein, The simulation operation is realized by traversing and simulating user operations by software.

Further, the simulation operation includes operations on the main interface of the software and its secondary interfaces, wherein the secondary interface is the first interface obtained by performing keyboard, mouse, and keyboard operations once from the main interface of the software, And all operations on the first interface and its sub-interfaces.

Further, the list also includes the time stamp when the operation occurred, the name of the related sub-panel, the name of the parent panel, and the key value corresponding to the operation, wherein the key value includes coordinates, input text, and input value.

The second aspect of the present invention provides a knowledge capture system for music software, including: a processor; and a memory coupled with the processor, the memory having instructions stored therein, and the instructions cause the The electronic device performs an action, and the action includes: A1, acquiring a screenshot of the software and performing recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes a plurality of functional areas, icons, or For text, perform step A1 iteratively to obtain a list of the target and its pixels corresponding to the maximum and minimum values on the X-coordinate axis and Y-coordinate axis, and save it in the first database, and then perform the following steps: A2, online Capture an event based on the software on the software operating device, and map the event to the list to obtain the software operation corresponding to the event;

A3. Refine the knowledge graph based on the event stream based on multiple software operations and save it in the second database.

Further, the pixel coordinates are the coordinates of the target relative to the window where the target is located.

Further, the step A1 also includes the following steps: A11, acquiring a screenshot of the software and converting the screenshot into a gray image; A12, performing image segmentation on the screenshot of the gray image based on each target, so as to divide the Each target is located in the gray image to obtain the pixel coordinates of the target; A13, the image matching algorithm is used to recognize the image-based target, and the optical character recognition function is used to recognize the text-based target.

Further, the step A1 further includes the following step: capturing a simulation operation based on the software on the software operating device to obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein, The simulation operation is realized by traversing and simulating user operations by software.

Further, the simulation operation includes operations on the main interface of the software and its secondary interfaces, wherein the secondary interface is the first interface obtained by performing keyboard, mouse, and keyboard operations once from the main interface of the software, And all operations on the first interface and its sub-interfaces.

Further, the list also includes the time stamp when the operation occurred, the name of the related sub-panel, the name of the parent panel, and the key value corresponding to the operation, wherein the key value includes coordinates, input text, and input value.

A third aspect of the present invention provides a knowledge capture device for music software, which includes: a knowledge learning module that obtains a screenshot of the software and performs recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes the Take a screenshot of a plurality of functional areas, icons or texts until the target and its pixels correspond to the maximum and minimum lists on the X-coordinate axis and the Y-coordinate axis and save them in the first database: knowledge acquisition device , It captures events based on the software on the software operating device online, maps the events in the list to obtain the software operations corresponding to the events, and refines the knowledge graph based on the event flow based on multiple software operations, and saves them In the second database.

The fourth aspect of the present invention provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-executable instructions that, when executed, cause at least one processor to execute the present invention. The method of the first aspect of the invention.

The fifth aspect of the present invention provides a computer-readable medium on which computer-executable instructions are stored, and when executed, the computer-executable instructions cause at least one processor to perform the method described in the first aspect of the present invention.

Translating the user's mouse and keyboard events into user interface goal-oriented events will consume some time online, sometimes lasting several hundred milliseconds, which will affect the modeling operations performed by customers at the same time. The knowledge acquisition mechanism of the software provided by the present invention can learn knowledge offline in advance and save the knowledge in the database, and combine the simple and rapid mapping function to recognize the information, which greatly improves the efficiency.

The invention can be applied to many softwares, especially engineering software, and is especially suitable for modeling software. The present invention does not depend on the function of the software itself, nor does it interfere with the modeling process, so it can run independently. The knowledge capture function provided by the present invention runs in the background, so it will not disturb, influence or change the user's operation of the software. The present invention is based on operating system functions, such as windows hook or screenshot function, so it can be applied to a variety of software without affecting the original functions or user interfaces of these software.

The present invention is not only suitable for a single software tool, but can also be used for multi-software. For example, when the user needs to switch between multiple software tools, such as co-simulation (multi-software), knowledge is also Can be captured.

Description of the drawings

Fig. 1 is a schematic structural diagram of a knowledge capturing device for modeling according to a specific embodiment of the present invention;

2 is a schematic diagram of the main interface of the modeling software according to a specific embodiment of the present invention;

3 is a schematic diagram of a primary interface and a secondary interface of the modeling software according to a specific embodiment of the present invention;

4 is a schematic diagram of another level interface of the main interface of the modeling software according to a specific embodiment of the present invention;

Fig. 5 is a schematic diagram of an event flow of a knowledge capture method for modeling according to a specific embodiment of the present invention;

Fig. 6 is a schematic diagram of an event flow of a knowledge capture method for modeling according to another specific embodiment of the present invention;

Fig. 7 is an object characteristic table of a modeled knowledge acquisition mechanism according to a specific embodiment of the present invention.

Detailed ways

The specific embodiments of the present invention will be described below with reference to the accompanying drawings.

The present invention provides a software knowledge capture mechanism, which has offline software-based related information acquisition, combined with related information corresponding to the online software operation in the software operation process, to analyze the software operation intention to refine the knowledge graph based on the event flow, In order to complete the entire online and offline combination of modeling software knowledge capture. In particular, the software is modeling software, and the present invention will be described below in conjunction with modeling software.

Fig. 1 is a schematic structural diagram of a knowledge capturing device for modeling according to a specific embodiment of the present invention. The user usually performs modeling operations on the modeling software 300, where the modeling software 300 is installed on the modeling device, and the knowledge learning module 100 and the knowledge acquisition device 200 provided by the present invention are also embedded on the modeling device. It is used to capture modeling knowledge offline and users perform modeling process at the same time. The present invention mainly includes two modules: a knowledge learning module 100 and a knowledge acquisition device 200. The knowledge learning module 100 is an offline operation module, and the knowledge acquisition device 200 is an online operation module, which is executed simultaneously during the modeling process. Wherein, the knowledge learning module 100 includes two sub-modules, namely a primary learning module 110 and a secondary learning module 120. Among them, the primary learning module 110 is used to learn the knowledge of the main interface of the modeling software, and the secondary learning module 120 is used to learn the knowledge of the secondary interface of the main interface of the modeling software. Specifically, the primary learning module 110 includes a first image acquisition module 111 and a first optical character recognition module 112, and the secondary learning module 120 includes an event simulation module 121, a second image acquisition module 121, and a second optical character recognition module 122. Wherein, the knowledge learning module 100 performs offline knowledge learning data stored in the first database DB1.

Further, the knowledge acquisition device 200 includes an event capture module 210 and a refinement analysis device 220. The mouse capture module 211 in the event capture module 210 is used to capture mouse operations, and the keyboard capture module 212 in the event capture module 210 is used to capture keyboard operations. The refining analysis module 220 further includes two sub-modules, including a mapping module 221 and a refining module 222, respectively. Wherein, the mapping module 221 maps the event based on the modeling software on the online capturing modeling device to the data in the first database DB1 to obtain the modeling operation issued to the modeling software of the modeling device corresponding to the event . The refinement module 222 refines the knowledge graph based on the event flow based on a plurality of modeling operations, and saves the knowledge graph in the second database DB2.

In this embodiment, illustratively, the modeling device is a personal computer, and the operating system of the personal computer is Windows. Further, the simulation operation includes operations on the main interface of the software and its secondary interfaces, wherein the secondary interface is the first interface obtained by performing keyboard, mouse, and keyboard operations once from the main interface of the software, And all operations on the first interface and its sub-interfaces.

The first aspect of the present invention provides a method for capturing modeling knowledge, which includes the following steps.

First, step S1 is executed. The knowledge learning module 100 obtains a screenshot of the modeling software and performs recognition based on the screenshot execution target and its pixel coordinates, where the target includes a plurality of functional areas on the modeling software screenshot, Icon or text. The primary learning module 110 in the knowledge learning module 100 is used to learn the knowledge of the main interface of the modeling software, and the secondary learning module 120 is used to learn the knowledge of the secondary interface of the main interface of the modeling software.

Specifically, the step S1 includes a sub-step S11, a sub-step S12, and a sub-step S13.

First, sub-step S11 is executed. The first image acquisition module 111 acquires a screenshot of the main interface of the modeling software 300 and converts the screenshot of the main interface into a gray image. The image color conversion refers to the conversion of the color RGB image of the main interface into a gray image. The gray image needed to perform the subsequent process. The main interface of the modeling software 300 shown in FIG. 2 is a converted gray image.

Then sub-step S12 is executed. The image segmentation device (not used) in the primary learning module 110 performs image segmentation on the screenshot of the gray image based on each target, wherein the target includes a complex number on the screenshot of the modeling software. Functional areas, icons or text. Therefore, image segmentation is to locate the main functional areas in the image to obtain the location of each target in the gray image, such as the command bar 510, the tool bar 520, and the white modeling area 530 of the main interface 500 in FIG. 2.

Finally, sub-step S13 is executed. The first image acquisition module 111 uses an image matching algorithm to recognize image-based objects, and the first optical character recognition module 112 uses an optical character recognition function to recognize text-based objects.

Therefore, what the primary learning module 110 needs to learn is to open the target name and location of the main interface of the modeling software 300, and the relationship between the targets on the main interface.

Fig. 2 is a schematic diagram of the main interface of the modeling software according to a specific embodiment of the present invention. As shown in FIG. 2, the main interface 500 of the modeling software includes a command bar 510, a tool bar 520 and a modeling area 530. Wherein, the toolbar 520 has multiple tabs, and after clicking each tab, each tab also has multiple buttons to choose from. The following uses the label 521 "Resources" in the toolbar 520 as an example to describe the image recognition of the present invention. Among them, as shown in Figure 2, the label 521 "Resources" is in the functional area of the toolbar, the minimum value of its pixel coordinates on the X axis X _min = 345, and the maximum value of its pixel coordinates on the X axis X _max = 405. The minimum value of the coordinates on the Y axis Y _min =175, and the maximum value of its pixel coordinates on the Y axis Y _max =191. Since the label 521 "Resources" is a rectangle, the coordinates of the upper left corner of the rectangle are (X _min , Y _min ), and the coordinates of the lower right corner of the rectangle are (X _max , Y _max ), so if the mouse clicks in this area , It is considered that the operator has clicked the label 521 "Resources".

In this embodiment, after clicking the label 521 "Resources", the operator further selects the button 5211 "worker" in the label 521 "Resources". Therefore, the present invention further performs pattern recognition after finding the functional area to correspond to the label 521" Button 5211 "worker" on Resources". Among them, as shown in Figure 2, the button 5211 "worker" is in the functional area of the toolbar, and the minimum value of its pixel coordinates on the X axis X _min =410, and the maximum value of its pixel coordinates on the X axis X _max =450. The minimum value of the pixel coordinates on the Y axis Y _min =196, and the maximum value of the pixel coordinates on the Y axis Y _max =234. Since the button 5211 "worker" is a rectangle, the coordinates of the upper left corner of the rectangle are (X _min , Y _min ), and the coordinates of the lower right corner of the rectangle are (X _max , Y _max ), so if the mouse clicks in this area , It is considered that the operator has clicked the button 5211 "worker".

According to a variation of this embodiment, it is assumed that the operator is operating the icon 541 "MaterialFlow" in the side window 540 "basis" on the left side of the modeling software main interface 500 as shown in FIG. 2, and the icon 541 "MaterialFlow" Located in the window on the left side of the main interface 500 of the modeling software shown throughout, the side window 540 "basis" is a tree structure. Among them, as shown in Figure 2, the icon 541 "MaterialFlow" is in the functional area of the toolbar, and its pixel coordinate is the minimum value X _min ＝6 on the X axis, and its pixel coordinate is the maximum value X _max ＝104 on the X axis. The minimum value of the coordinate on the Y axis Y _min =18, and the maximum value of its pixel coordinate on the Y axis Y _max =202. Since the icon 541 "MaterialFlow" is a rectangle, the coordinates of the upper left corner of the rectangle are (X _min , Y _min ), and the coordinates of the lower right corner of the rectangle are (X _max , Y _max ), so if the mouse clicks in this area , It is considered that the operator has clicked the icon 541 "MaterialFlow".

In summary, the present invention has different algorithms for image recognition for different types of icons in the software interface.

According to a specific embodiment of the present invention, the optical character recognition function is introduced next.

Locate the action in the first event record table in the screenshot, and determine the object selected by the action based on the characteristics of the object based on image recognition. Based on the action and its location and screenshot, the action can be placed in the first Positioning in the screenshot. This embodiment can also determine the above actions based on the image recognition algorithm. In this embodiment, it needs to determine the object selected by the mouse.

As shown in the table shown in Figure 7, the mouse action objects and their types, characteristics, and positions are listed on the sub-panel "Models.Frame.TestStation1". Among them, the objects are recorded in the form of images, and the object types include numerical boxes, Label items, check boxes, and menu items. Among them, the object feature of the value box is "A rectangle with a large blank area" (A rectangle with a large blank area), and its position is "on the left of the mouse cursor" (on the left of mouse click). The object feature of the label item is "A narrow rectangle-like shape, but no 4 edge-contour" (A narrow rectangle-like shape, but no 4 edge-contour), and its position is "at the mouse cursor position" (at the mouse position). The object feature of the check box is "A square blank area" (A square blank), and its position is "On the right of the mouse" (On the right of the mouse). The object feature of the menu item is "The position on the Y coordinate is very close to the origin of the window (the upper left corner of the window), and there is no outline near the mouse position" (Y position is quite close to the origin of the window (top left corner of the window) ), no contour around the mouse position, and its position is "at the mouse position".

As shown in FIG. 4, an area of interest is generated according to the object type and object position, and the area of interest is stored as an enlarged sub-image. The optical character recognition module (not used) can determine the object type based on the above object characteristics, and based on the object type and object position, generate a region of interest (ROI) as shown in FIG. 4, where the interest is The area ROI is generated based on the position of the object. The optical character recognition module crops the original screenshot shown in FIG. 4 into a sub-image ROI' based on the region of interest ROI. In order to improve the quality of text extraction, the mapping module 221 enlarges the above-mentioned sub-image ROI'.

Then, the optical character recognition module executes the optical character recognition function to obtain the text on the sub-image, and the information in the timetable is matched with the text of the object and a second event record table is generated. The recognition module (not used) is used to perform Optical Character Recognition (OCR, Optical Character Recognition). When the ROI of the first screenshot is enlarged and cut into a sub-image, the recognition module converts the text on the sub-image Extracted, therefore, we can get the name of the value box and the text "Processing time: Const" on the sub-image. Therefore, therefore, the knowledge structure constructed by the present invention for the main interface 500 of the modeling software is as shown in the following table:

Table 1 The structure of the main interface 500 of the modeling software 300

Among them, the main components of the main interface 500 of the modeling software include a command bar 510, a tool bar 520, and a side window 540. The command bar 510 has a minimum X _min and a maximum X _{max of} its pixels on the X axis, and its pixel coordinates in Y. The minimum value Y _min and the maximum maximum value Y _{max of the} axis have other required information. The sub-components of the command bar 510 include the label 511 "Home", the label 512 "Debugger", and the label 513 "Window"... Among them, the label 511 "Home" has the minimum value X _min and the maximum value X of its pixels on the X axis. _{max and the minimum value Y min} and maximum maximum value Y _{max of} its pixel coordinates on the Y axis. The label 512 "Debugger" has the minimum value X _min and maximum value X _{max of} its pixels on the X axis and the minimum value of its pixel coordinates on the Y axis. The value Y _min and the maximum value Y _max, etc. The sub-components of the toolbar 520 include the label 522 "Material Flow", the label 523 "Fluids"... and the minimum and maximum values of the pixels on the X axis and the Y axis, respectively. In the same way, the secondary components of the side window 540 include the label 541 "Material Flow", the label 542 "Fluids"... and the minimum and maximum values of its pixels on the X axis and the Y axis, respectively. The sub-component label 511 "Home" includes the buttons "Event Controller", "Reset" and "Start/Stop" and the minimum and maximum values of their pixels on the X-axis and Y-axis, respectively. Similarly, the secondary component label 512 "Debugger" includes button 1 and button 2, the label 513 "Window" includes button 3 and button 4, and the label 522 "Material Flow" includes the buttons "Connector", "Source" and "SingleProc". The label 523 "Fluids" includes button 5 and button 6, the secondary component label 541 "Material Flow" includes buttons "Connector", "Source" and "SingleProc", and the secondary component label 542 "Fluids" includes button 7 and button 8.

Therefore, the present invention decomposes the construction of the main software interface into multiple layers. The knowledge included is hierarchical. The first layer includes main components, each layer of main components includes multiple secondary components, and each layer requires multiple components. Buttons. Each of the above-mentioned layers includes the area covered by the minimum and maximum values of its pixels on the X-axis and Y-axis, respectively, covering the value range, or some other information is used to describe these components. Therefore, this knowledge is goal-oriented, and it can be saved in many formats, such as JSON files, ontology files, xml files, or others.

Next, take a screenshot of the target identification and positioning of the secondary interface 600 under the main interface 500 of the modeling software 300. The target identification and positioning of the secondary interface 600 is similar to that of the main interface 500. The target identification and positioning of the primary interface and the secondary interface are The difference of the mechanism is that the main interface 600 needs to be operated to expand the secondary interface, and the relationship between multiple targets needs to be paid attention to.

As shown in Figure 1, the secondary learning module 120 is used to acquire knowledge of the secondary interfaces below the main interface of the modeling software, where the secondary interface is an interface displayed after the main interface of the modeling software is operated once, and it mainly includes Secondary window list or drop-down menu. Exemplarily, the secondary interface 6 is activated by clicking some buttons under the main interface. In order to realize automation, the simulator 121 is designed as an operation of the secondary interface displayed after the main interface of the modeling software once. Similarly, the operation includes but not limited to keyboard operation, mouse operation, and touch screen operation. Preferably, the simulator 121 is a mouse click simulator, which simulates the function of completing the main interface of the click modeling software, and simulates the action of the mouse click to open the secondary interface that needs to be learned. Among them, the goals of the secondary interface include many types and have many levels, and the layout of the secondary interface is very detailed.

Specifically, the emulator 121 provided by the present invention opens a secondary interface, and when the target secondary interface is opened, a screenshot of the secondary interface is automatically taken to obtain an image of a window, and then the automatic learning process is executed. The secondary interface knowledge learning process is similar to the primary interface knowledge learning process, including graphics color conversion, graphics segmentation, pattern matching and optical character recognition functions. Similarly, sub-step S21, sub-step S22, and sub-step S23 need to be executed.

In the same way, the knowledge learned from the secondary interface is hierarchical and goal-oriented. The difference between the primary interface and the secondary interface is that the secondary interface has a different hierarchical structure, so each secondary interface is regarded as a target, and its main element is different from other targets in the type. Specifically, a secondary interface type of SingleProc includes a menu bar 610, a tab bar 620, and other areas, such as character boxes 630 and 640, and check boxes. The menu bar 610 has the minimum value X _min and maximum value X _max of its pixels on the X axis and the minimum value Y _min and maximum value Y _{max of} its pixel coordinates on the Y axis. The menu bar 610 has the label 610 "Navigate" and the label 620 "View", label 630 "Tools", label 640 "Help", label 610 "Navigate", label 620 "View", label 630 "Tools" and label 640 "Help" also have their smallest pixels on the X axis. The value X _min and the maximum value X _{max and the minimum value Y min} and the maximum value Y _{max of} the pixel coordinates on the Y axis. Further, the label 612 "Navigate" also includes the list item 6121 "Open Location", the list item 6122 "Open Origin", the list item 6123 "Go to Origin" and the list item 6124 "Open Class Alt+Home", of which the list item 6121 "Open Location", list item 6122 "Open Origin", list item 6123 "Go to Origin" and list item 6124 "Open Class Alt+Home" also have the minimum X _min and maximum X _{max of its pixels on the X axis. And the minimum value Y min} and maximum maximum value Y _{max of} its pixel coordinates on the Y axis. The label column 620 includes the label 621 "Time", the label 622 "Set-up" and the label 623 "Failures". The label 621 "Time", the label 622 "Set-up", the label 623 "Failures"... also have their respective pixels. The minimum value X _min and maximum value X _max on the X axis and the minimum value Y _min and maximum value Y _{max of} its pixel coordinates on the Y axis. The label 621 "Time" includes the character box 6211 "Processing Time", the character box 6212 "Set-up Time"... the character box 6211 "Processing Time", the character box 6212 "Set-up Time", etc. also have their pixels in X. The minimum X _min and maximum X _max of the axis and the minimum Y _min and maximum Y _{max of} its pixel coordinates on the Y axis. The label 622 "Set-up" includes the buttons "Entrance" and "Exit". The buttons "Entrance" and "Exit" also have the minimum X _min and maximum X _{max of} the pixels on the X axis, and the pixel coordinates in the The minimum value Y _min and the maximum value Y _{max of the} Y axis.

Therefore, in this embodiment, the knowledge structure constructed by the present invention for the secondary interface of the main interface 500 of the modeling software is as shown in the following table:

Table 2 The structure of the secondary interface of the main interface 500 of the modeling software 300

Therefore, the present invention decomposes the construction of the software secondary interface into multiple layers. The secondary interface is regarded as a target, which includes different components, and each component is also regarded as a target, which can have secondary components, and each Each component has coordinate values. The types of components include, for example, a digital fan, a command button, or a switch key. Then the mapping function module will track this hierarchical structure to illustrate the event target name and location.

Iteratively executed to step S1, to obtain a list of the target pixel and respectively correspond to the X-axis and Y axes, and the maximum and minimum values stored in the first database DB _1. According to a preferred embodiment of the present invention, all operations of the main interface 500 of the modeling software and all of its secondary interfaces are targeted for identification and positioning, that is, the targets and their pixels of all operations in all subsequent modeling processes are separated corresponding to the X-axis and Y coordinate axes, and a list of maximum and minimum values stored in the first database DB _1. In this way, no matter what operation the user performs on the modeling software, he can recognize his modeling intention and operation, and capture knowledge. Target recognition and positioning of all operations on the main interface 500 of the modeling software and all of its secondary interfaces can be achieved by traversing the software to simulate user clicks.

Further, the step S1 further includes the following step: capturing a simulation operation based on the software on the software operating device to obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein, The simulation operation is realized by traversing and simulating user operations by software.

Then step S2 is executed. The event capturing module 210 of the knowledge acquisition device 200 captures an event based on the modeling software 300 on the modeling device online, and the mapping module 221 maps the event to the target and its pixels correspond to the X coordinate axis and In the list of the maximum value and the minimum value on the Y coordinate axis, the modeling operation issued to the modeling software 300 of the modeling device corresponding to the event can be obtained.

Specifically, the mouse capture module 211 captures mouse actions based on modeling software, and the keyboard capture module 212 is used to capture keyboard actions. In addition, the event capture module 210 preferably further includes a touch screen capture module (not shown), which is used to capture touch screen actions. Therefore, once the event capturing module 210 of the knowledge acquisition device 200 online captures the operations of the mouse, keyboard, and touch screen on the modeling device based on the modeling software 300, the Windows hook function will capture the interaction between the mouse, keyboard, and touch screen. Information related to the operation of the screen, including the above-mentioned actions and the time stamp when the above-mentioned actions occurred, the names of the related sub-panels, the names of the parent panels, and the key values of the above-mentioned actions on the modeling software generate a first event record table. Wherein, the key value includes coordinates, input text, and input value. Each keyboard, mouse, and touch screen operation is regarded as an event, and the first event record table shown below is generated:

Table 3 The first event record table

Timestamp	action	Subpanel name	Window type	Window position	Mother panel name	Key value
Thu Mar26 14:59:30 2019	Mouse left down	Models.Frame.TestStation1	SingleProc	(1090, 454)	.Models.Frame	(1340,630)
Thu Mar26 14:59:32 2019	Key down	Models.Frame.TestStation1	SingleProc	(1090, 454)	.Models.Frame	Numpad2
Thu Mar26 14:59:33 2019	Key down	Models.Frame.TestStation1	SingleProc	(1090, 454)	.Models.Frame	Numpad0

Optionally, Table 1 saves event records in .csv format. As shown in Table 1 and Figure 4, the time stamp represents the time when the action was executed. Specifically, "Thu Mar2614:59:30 2019" means that it was done at 14:59:30 PM on March 26, 2019. "Mouse left down" is the action of clicking the left mouse button, and then "Key down" is made on "Thu Mar26 14:59:322019" at 14:59:32 PM on March 26, 2019, which is the keyboard The typing action, followed by "Thu Mar26 14:59:33 2019" at 14:59:33 PM on March 26, 2019, a "Key down" was made, that is, the keyboard typing action, the input character was "20". This means that the engineer has completed a left mouse click on the user interface of the modeling software and entered information through the keyboard, and the information includes two characters. Then, the name of the sub-panel and the name of the parent panel both indicate the panel on the user interface of the modeling software. In this embodiment, the key value represents the coordinates on the user interface. Therefore, the position of the left mouse button above is specifically in the sub-panel. On the panel "Models.Frame.TestStation1", the specific coordinates are (1340,630). Wherein, the sub-panel "Models.Frame.TestStation1" belongs to the parent panel ".Models.Frame". In addition, the window position of "Models.Frame.TestStation1" is the coordinates (1090, 454)". However, the above information is not enough to identify the object name. Therefore, the window system (Windows Hook) cannot obtain enough meaningful modeling software For example, the name of the text box (text box name) or the name of the value box (box name), which requires the help of screenshots of modeling software.

Further, the mapping module 221 maps the event to the target and its pixels in a list corresponding to the maximum and minimum values on the X-coordinate axis and the Y-coordinate axis, respectively, wherein, according to the above-mentioned embodiment of the present invention, The list of the target and its pixels corresponding to the maximum and minimum values on the X-coordinate axis and the Y-coordinate axis respectively includes Table 1 and Table 2 above. It should be noted that the structure of the main interface 500 is shown in Table 1. Table 2 is one of the secondary interfaces 600 of the primary interface 500, and the primary interface 500 must also include other secondary interfaces.

In this embodiment, since it is known in Table 3 that the window type is "SingleProc", it is mapped to the secondary interface with the window type of "SingleProc", and the mapped target is "time label". Among them, each event is not isolated from each other, some events need to refer to some previous events. For example, in this embodiment, if the label clicked here is not a label switch, the default label should be "time label". Then what needs to be mapped is the secondary target to the window "Processing times" and the number box. Among them, it can be seen from Table 3 that this embodiment has completed a left mouse click at the position (1340, 630), and the upper left position of "Models.Frame.TestStation1" is the coordinates (1090, 454)", so on the window "TestStation1" The mouse click position should be (1340-1090, 630-454), and the relative coordinates should be (250, 176). Through the mapping function, we know that the user clicked on the secondary interface of "SingleProc" at the position (250, 176) , And it is within the "Processing times" range of the time label.

So so far, the information we have obtained is: the user clicked the number box of "Processing times" with the left mouse button.

Further, the pixel coordinates are the coordinates of the target relative to the window where the target is located. For example, take the coordinates of the upper left corner of the software window where the target is located as the origin, because the position of the target relative to the upper left corner of the window where it is located remains unchanged.

Finally performing step S3, the extraction module 222 based on a plurality of modeling operations to extract the knowledge-based event stream pattern, and stored in the second database DB _2. According to the foregoing, although we have the target names and related events as shown in the first event list in Table 3, we have relevant information about scattered target object trends, but we also need to refine and analyze the user’s modeling intentions, so we need to combine the above information Interpreted as meaningful process-oriented related information, expressed as follows:

Table 4 The second event record table

Specifically, as shown in Table 4, at 14:59:30 PM on March 26, 2019, the user displayed the sub-panel "Models.Frame.TestStation1" of the parent panel "Technomatix Observer_example1_rev3_0.spp" on the modeling software interface. The upper coordinate is (250, 176) to complete the action of a left mouse click. Then, at 14:59:33 PM on March 26, 2019, the user completed two keyboard inputs on the sub-panel "Models.Frame.TestStation1" of the parent panel ".Models.Frame" on the modeling software interface In the action, the user sequentially presses the numeric key 2 (Numpad2) and numeric key 0 (Numpad2) of the keyboard. Therefore, as shown in the second event record table of Table 4, after analysis by the refining module 222, the user’s modeling intention on the modeling software is the event "Set_item[Process time]on_sub_panel" and "Keyboard input[20]for[Process time]on_sub_panel". As shown in Figure 2, the event "Set_item[Process time] on_sub_panel" indicates that the modeling intention is to issue a command to operate the modeling software to set "Process times" (execution time) on the modeling device in response to the above-mentioned mouse action. "Keyboard input[20] for [Process time] on_sub_panel" indicates that the modeling intention is to issue a command to the modeling device to operate the modeling software input execution time of "20" in response to the above keyboard action. It should be noted that, in Figure 4, the modeling software automatically recognizes "20" input by the user as "20:00".

Finally, these events and data nodes are saved as a knowledge graph and saved in the second database DB ₂ for subsequent analysis. The events shown in Table 4 are stored as the knowledge graph shown in Figure 5. As shown in Figure 5, the type of event 1 is mouse action, and the timestamp of event 1 "Set_item[Process time]on_sub_panel" is "Thu Apr 26 14:59:31 2018", the location of event 1 is "Technomatix Observer_example1_rev3_0.spp-.[.Models.Frame.Test Station1]", and the key value of event 1 is the coordinate "(1340, 630)". Among them, the type of event 2 "Keyboard input[20]for[Process time]on_sub_panel" is keyboard action, the time stamp of event 1 is "Thu Apr 26 14:59:33 2018", and the location of event 2 is "Technomatix Observer_example1_rev3_0. spp-.[.Models.Frame.Test Station1]", the key value of event 2 is the keyboard input value "20". The relationship between event 1 and event 2 is: the previous event of event 2 is "event 1", and the next event of event 1 is "event 2".

According to another preferred embodiment of the present invention, the user uses the left mouse button to click the four components in the label 522 "Material Flow" in the toolbar 520 in the main interface 500, and drag them to the modeling area 530. The components are "Source", "SingleProc", "ParallelProc" and "Drain", then double-click to open the component "SingleProc", and then enter "2:00" in the number box following "Processing time". Then record the above user operation as the first event record table as shown below:

Table 5 The first event record table

Timestamp	action	Subpanel name	Types of	Window position	Mother panel name	Key value
3/26/2019 14:53:19	Mouse left down	.Models.Frame	main	NA	example.spp	(419,219)
3/26/2019 14:53:22	Mouse left down	.Models.Frame	main	NA	example.spp	(515,551)
3/26/2019 14:53:24	Mouse left down	.Models.Frame	main	NA	example.spp	(501,211)
3/26/2019 14:53:27	Mouse left down	.Models.Frame	main	NA	example.spp	(587,546)
3/26/2019 14:53:30	Mouse left down	.Models.Frame	main	NA	example.spp	(541,218)
3/26/2019 14:53:32	Mouse left down	.Models.Frame	main	NA	example.spp	(673,552)
3/26/2019 14:53:34	Mouse left down	.Models.Frame	main	NA	example.spp	(464,218)
3/26/2019 14:53:37	Mouse left down	.Models.Frame	main	NA	example.spp	(757,550)
3/26/2019 14:53:39	Mouse left down	.Models.Frame	main	NA	example.spp	(590,550)
3/26/2019 14:53:41	Mouse left down	.Models.Frame.SingleProc	SingleProc	(383, 200)	.Models.Frame	(631,375)
3/26/2019 14:53:44	key down	.Models.Frame.SingleProc	SingleProc	(383, 200)	.Models.Frame	Numpad2

Next, the mapping module 221 performs the mapping function, and the refining module 222 performs the refining function. According to the foregoing, although we have the target names and related events as shown in the first event list in Table 5, we have relevant information about the trends of scattered target objects, but we also need to refine and analyze the user's modeling intentions, so we need to combine the above information Interpreted as meaningful process-oriented related information, expressed as follows:

Table 6 The third event record table

Specifically, as shown in Table 6, at 14:53:22 PM on March 26, 2018, the coordinates of the user on the sub-panel ".Model.Frame" of the parent panel "Example.spp" on the modeling software interface are (515,551) Completed a click of the left mouse button. The event "Create_object[Source]" indicates that the modeling intention is to send a command to the modeling device to operate the modeling interface of the modeling software to establish the first element Source in response to the above-mentioned mouse action. At 14:53:27 pm on March 26, 2019, the user completed a left mouse button at (587,546) on the sub-panel ".Model.Frame" of the parent panel "Example.spp" on the modeling software interface The action of the click. The event "Create_object[SingleProc]" indicates that the modeling intention is to send a command to the modeling device to operate the modeling interface of the modeling software in response to the mouse action. At 14:53:32 PM on March 26, 2019, the user completed a left mouse button at (673,552) on the sub-panel ".Model.Frame" of the parent panel "Example.spp" on the modeling software interface. The action of the click. The event "Create_object[ParallelProc]" indicates that the modeling intention is to issue a command to the modeling device to operate the ParallelProc component on the modeling interface of the modeling software in response to the above-mentioned mouse action. At 14:53:37 PM on March 26, 2019, the user completed a left mouse button on the sub-panel ".Model.Frame" of the parent panel "Example.spp" on the modeling software interface at (757,550) The action of the click. The event "Create_object[Drain]" indicates that the modeling intention is to issue a command to the modeling device to operate the modeling interface of the modeling software to establish the fourth element Drain in response to the above-mentioned mouse action. At 14:53:41 PM on March 26, 2019, the user completed a mouse at (248,175) on the sub-panel ".Model.Frame.SingleProc" of the parent panel "Example.spp" on the modeling software interface Left click action. The event "Set_item[Processing Time] on_sub_panel" indicates that the modeling intention is to send a command to operate the modeling software to set the "Process time" (execution time) of the modeling device in response to the mouse action described above. At 14:53:44 PM on March 26, 2019, the user completed a keyboard action on the sub-panel ".Model.Frame.SingleProc" of the parent panel "Example.spp" on the modeling software interface. The event "keyboard input[2] for [Processing Time] on_sub_panel" indicates that the modeling intention is to issue a command to operate the modeling software input execution time "2" to the modeling device in response to the above keyboard action. It should be noted that the modeling software automatically recognizes the "2" input by the user as "2:00".

The refinement module 222 refines the knowledge graph based on the event flow based on a plurality of modeling operations, and saves the knowledge graph in the second database DB ₂ . Among them, as shown in Figure 6, the knowledge graph of the event stream includes event 3, event 4, event 5, event 6, event 7 and event 8. The above events form an event stream in the order of execution. Among them, the type of event 3 "Create_object[Source]" is a mouse action, the time stamp of event 3 is "5/22/2018 10:53:20", and the location of event 3 is "Example.spp–[.Model.Frame ]", the key value of event 4 "Create_object[SingleProc]" is the coordinate "(515,551)". Among them, the type of event 4 is a mouse action, the time stamp of event 4 is "5/22/2018 10:53:23", the location of event 4 is "Example.spp–[.Model.Frame]", and the time stamp of event 4 is "Example.spp–[.Model.Frame]". The key value is the coordinate "(587,546)". Among them, the type of event 5 "Create_object[ParallelProc]" is a mouse action, the time stamp of event 5 is "5/22/2018 10:53:25", and the location of event 5 is "Example.spp–[.Model.Frame ]", the key value of event 5 is the coordinate "(673,552)". Among them, the type of event 6 "Create_object[Drain]" is a mouse action, the time stamp of event 6 is "5/22/2018 10:53:28", and the location of event 6 is "Example.spp–[.Model.Frame ]", the key value of event 6 is the coordinate "(757,550)". Among them, the type of event 7 is a mouse action, the time stamp of event 7 is "5/22/2018 10:53:34", and the location of event 7 is "Example.spp–[.Model.Frame.SingleProc]", the event 7 The key value of "Set_item[Processing Time]on_sub_panel" is the coordinate "(21,1187)". Among them, the type of event 8 is keyboard action, the time stamp of event 8 is "5/22/2018 10:53:37", and the location of event 8 "keyboard input[2]for[Processing Time]on_sub_panel" is "Example. spp–[.Model.Frame.SingleProc]", the key value of event 8 is keyboard input "2".

Among them, the relationship between event 3 and event 4 is: the previous event of event 4 is "event 3", and the next event of event 3 is "event 4". The relationship between event 4 and event 5 is: the previous event of event 5 is "event 4", and the subsequent event of event 4 is "event 5". The relationship between event 5 and event 6 is: the previous event of event 6 is event 5, and the next event of event 5 is "event 6". The relationship between event 6 and event 7 is: the previous event of event 7 is "event 6", the next event of event 6 is "event 7". The relationship between event 7 and event 8 is: the previous event of event 8 is "event 7", and the next event of event 7 is "event 8".

Since the image recognition and optical character recognition (OCR, Optical Character Recognition) functions used for the learning module need to consume a certain amount of time, the present invention sets the learning module to run offline, which is more efficient and faster.

The second aspect of the present invention provides a knowledge capture system for music software, including: a processor; and a memory coupled with the processor, the memory having instructions stored therein, and the instructions cause the The electronic device performs an action, and the action includes: A1, acquiring a screenshot of the software and performing recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes a plurality of functional areas, icons, or For text, perform step A1 iteratively to obtain a list of the target and its pixels corresponding to the maximum and minimum values on the X-coordinate axis and Y-coordinate axis, and save it in the first database, and then perform the following steps: A2, online Capture an event based on the software on the software operating device, and map the event to the list to obtain the software operation corresponding to the event;

A3. Refine the knowledge graph based on the event stream based on multiple software operations and save it in the second database.

Further, the pixel coordinates are the coordinates of the target relative to the window where the target is located.

Further, the step A1 also includes the following steps: A11, acquiring a screenshot of the software and converting the screenshot into a gray image; A12, performing image segmentation on the screenshot of the gray image based on each target, so as to divide the Each target is located in the gray image to obtain the pixel coordinates of the target; A13, the image matching algorithm is used to recognize the image-based target, and the optical character recognition function is used to recognize the text-based target.

Further, the step A1 further includes the following step: capturing a simulation operation based on the software on the software operating device to obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein, The simulation operation is realized by traversing and simulating user operations by software.

Further, the simulation operation includes operations on the main interface of the software and its secondary interfaces, wherein the secondary interface is the first interface obtained by performing keyboard, mouse, and keyboard operations once from the main interface of the software, And all operations on the first interface and its sub-interfaces.

Further, the list also includes the time stamp when the operation occurred, the name of the related sub-panel, the name of the parent panel, and the key value corresponding to the operation, wherein the key value includes coordinates, input text, and input value.

A third aspect of the present invention provides a knowledge capture device for music software, which includes: a knowledge learning module that obtains a screenshot of the software and performs recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes the Take a screenshot of a plurality of functional areas, icons or texts until the target and its pixels correspond to the maximum and minimum lists on the X-coordinate axis and the Y-coordinate axis and save them in the first database: knowledge acquisition device It captures events based on the software on the software operating device online, maps the events to the list to obtain the software operations corresponding to the events, and refines the knowledge graph based on the event flow based on multiple software operations, and saves them In the second database.

The fourth aspect of the present invention provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-executable instructions that, when executed, cause at least one processor to execute the present invention. The method of the first aspect of the invention.

The fifth aspect of the present invention provides a computer-readable medium on which computer-executable instructions are stored, and when executed, the computer-executable instructions cause at least one processor to perform the method described in the first aspect of the present invention.

Translating the user's mouse and keyboard events into user interface goal-oriented events will consume some time online, sometimes lasting several hundred milliseconds, which will affect the modeling operations performed by customers at the same time. The knowledge acquisition mechanism of the software provided by the present invention can learn knowledge offline in advance and save the knowledge in the database, and combine the simple and rapid mapping function to recognize the information, which greatly improves the efficiency.

The invention can be applied to many softwares, especially engineering software, and is especially suitable for modeling software. The present invention does not depend on the function of the software itself, nor does it interfere with the modeling process, so it can run independently. The knowledge capture function provided by the present invention runs in the background, so it will not disturb, influence or change the user's operation of the software. The present invention is based on operating system functions, such as windows hook or screenshot function, so it can be applied to a variety of software without affecting the original functions or user interfaces of these software.

The present invention is not only suitable for a single software tool, but can also be used for multi-software. For example, when the user needs to switch between multiple software tools, such as co-simulation (multi-software), knowledge is also Can be captured.

Although the content of the present invention has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as limiting the present invention. After those skilled in the art have read the above content, various modifications and alternatives to the present invention will be obvious. Therefore, the protection scope of the present invention should be defined by the appended claims. In addition, any reference signs in the claims should not be regarded as limiting the involved claims; the word "comprising" does not exclude other claims or devices or steps not listed in the specification; "first", "section Words such as "two" are only used to indicate names, and do not indicate any specific order.

Claims (15)

1. Software knowledge capture method, which includes the following steps:

   S1. Obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes a plurality of functional areas, icons or texts on the screenshot,

   Step S1 is performed iteratively to obtain a list of the target and its pixels corresponding to the maximum and minimum values on the X-coordinate axis and the Y-coordinate axis, and save them in the first database, and then perform the following steps:

   S2, online capturing an event based on the software on the software operating device, and mapping the event to the list to obtain the software operation corresponding to the event;

   S3: Refine the knowledge graph based on the event stream based on a plurality of software operations, and save it in the second database.
2. The software knowledge capture method according to claim 1, wherein the pixel coordinates are the coordinates of the target relative to the window where the target is located.
3. The software knowledge capture method according to claim 1, wherein the step S1 further comprises the following steps:

   S11. Obtain a screenshot of the software and convert the screenshot into a gray image;

   S12. Perform image segmentation on the screen capture of the gray image based on each target, so as to locate each target in the gray image to obtain the pixel coordinates of the target;

   S13, using an image matching algorithm to recognize an image-based target, and using an optical character recognition function to recognize a text-based target.
4. The software knowledge capture method according to claim 1, wherein said step S1 further comprises the following steps:

   Capture the simulation operation based on the software on the software operating device to obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein the simulation operation is realized by traversing the simulation user operation by the software .
5. The method for capturing knowledge of software according to claim 1, wherein the simulation operation includes an operation on the main interface of the software and its secondary interface, wherein the secondary interface is executed from the main interface of the software. The first interface obtained by one keyboard, mouse, and keyboard operation, and all operations on the first interface and its sub-interfaces.
6. The method for capturing knowledge of software according to claim 1, wherein the list further includes the time stamp when the operation occurs, the name of the related child panel, the name of the parent panel, and the key value corresponding to the operation, wherein , The key value includes coordinates, input text, and input value.
7. Software knowledge capture system, including:

   Processor; and

   A memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, cause the electronic device to perform actions, and the actions include:

   A1. Obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes a plurality of functional areas, icons or text on the screenshot,

   Step A1 is performed iteratively to obtain a list of the target and its pixels corresponding to the maximum and minimum values on the X-coordinate axis and the Y-coordinate axis, and save them in the first database, and then perform the following steps:

   A2. Online capture of events based on the software on the software operating device, and map the events to the list to obtain the software operation corresponding to the event;

   A3. Refine the knowledge graph based on the event stream based on multiple software operations and save it in the second database.
8. The software knowledge capture system according to claim 1, wherein the pixel coordinates are the coordinates of the target relative to the window where the target is located.
9. The software knowledge capture system according to claim 1, wherein the step A1 further comprises the following steps:

   A11. Obtain a screenshot of the software and convert the screenshot into a gray image;

   A12. Perform image segmentation on the screenshot of the gray image based on each target, so as to locate each target in the gray image to obtain the pixel coordinates of the target;

   A13, use image matching algorithms to recognize image-based targets, and use optical character recognition to recognize text-based targets.
10. The software knowledge capture system according to claim 1, wherein said step A1 further comprises the following steps:

    Capture the simulation operation based on the software on the software operating device to obtain a screenshot of the software and perform recognition based on the screenshot execution target and its pixel coordinates, wherein the simulation operation is realized by traversing the simulation user operation by the software .
11. The software knowledge capture system according to claim 1, wherein the simulation operation includes an operation on the main interface of the software and its secondary interface, wherein the secondary interface is executed from the main interface of the software The first interface obtained by one keyboard, mouse, and keyboard operation, and all operations on the first interface and its sub-interfaces.
12. The software knowledge capture system according to claim 1, wherein the list further includes the timestamp when the operation occurs, the name of the related child panel, the name of the parent panel, and the key value corresponding to the operation, wherein , The key value includes coordinates, input text, and input value.
13. Software knowledge capture device, including:

    A knowledge learning module that obtains a screenshot of the software and performs recognition based on the screenshot execution target and its pixel coordinates, wherein the target includes a plurality of functional areas, icons or text on the screenshot,

    Until the target and its pixels correspond to the maximum and minimum lists on the X-coordinate axis and the Y-coordinate axis, and save them in the first database:

    A knowledge acquisition device that online captures events based on the software on a software operating device, maps the events in the list to obtain the software operations corresponding to the events, and refines the knowledge graph based on the event flow based on a plurality of software operations , And save in the second database.
14. A computer program product that is tangibly stored on a computer-readable medium and includes computer-executable instructions that when executed cause at least one processor to execute any of claims 1 to 6 The method described in one item.
15. A computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed, cause at least one processor to execute the method according to any one of claims 1 to 6.

Images