CN113157471A - Debugging system and early warning system - Google Patents
Debugging system and early warning system Download PDFInfo
- Publication number
- CN113157471A CN113157471A CN202010498637.3A CN202010498637A CN113157471A CN 113157471 A CN113157471 A CN 113157471A CN 202010498637 A CN202010498637 A CN 202010498637A CN 113157471 A CN113157471 A CN 113157471A
- Authority
- CN
- China
- Prior art keywords
- debugging
- debug
- communication interface
- learning device
- error information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000002159 abnormal effect Effects 0.000 claims abstract description 85
- 238000004891 communication Methods 0.000 claims abstract description 82
- 238000012790 confirmation Methods 0.000 claims description 19
- 230000015654 memory Effects 0.000 claims description 13
- 239000000284 extract Substances 0.000 claims description 4
- 238000010200 validation analysis Methods 0.000 claims 5
- 238000003379 elimination reaction Methods 0.000 abstract description 10
- 230000008030 elimination Effects 0.000 abstract description 9
- 238000010801 machine learning Methods 0.000 abstract description 7
- 238000000034 method Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 230000005856 abnormality Effects 0.000 description 7
- 230000011218 segmentation Effects 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 230000009466 transformation Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/325—Display of status information by lamps or LED's
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/453—Help systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present disclosure relates to a debugging system and an early warning system. The debugging system comprises a communication interface device and a learning device. The communication interface device is in communication connection with the controlled device. The communication interface device comprises a capturing unit. The capturing unit captures an operation picture of the controlled device. And extracting abnormal error information from the operation picture when the controlled device is abnormal. The learning device is connected with the communication interface device. The learning device includes at least one debug subset. Any of the at least one debug subset includes an error model and an auto-debug step or a debug description of the corresponding error model, respectively. The learning device receives the abnormal error information through the communication interface device, and outputs corresponding automatic debugging steps or debugging descriptions to the controlled device when the abnormal error information is compared to be in accordance with the error model of each debugging subset. The corresponding automatic debugging steps or debugging descriptions are output by comparing the current abnormal error information with the error model, so that automatic and manual obstacle elimination is combined, and machine learning is carried out simultaneously.
Description
Technical Field
The present disclosure relates to a debugging system and an early warning system, and more particularly, to an automated debugging system and an early warning system.
Background
Generally, when a problem occurs in an electronic device such as a machine or a computer, the electronic device sends an alarm message to notify a field operator, and then the operator can perform obstacle elimination according to the alarm message.
However, when the operator has to replace the electronic device, such as shift or new person, there are some factors such as insufficient experience or unfamiliarity with the electronic device, and there is a problem that it is not known how to remove the obstacle or the operation procedure for removing the obstacle is wrong.
Therefore, the above problems are one of the important issues in the art.
Disclosure of Invention
One aspect of the present disclosure is directed to a debug system including a communication interface device and a learning device. The communication interface device is connected with the controlled device and the learning device in a communication mode. The communication interface device comprises a capturing unit. The capturing unit captures an operation picture of the controlled device. And extracting abnormal error information from the operation picture when the controlled device is abnormal. The learning device includes at least one debug subset. Any of the at least one debug subset includes an error model and an auto-debug step or a debug description of the corresponding error model, respectively. The learning device receives the abnormal error information through the communication interface device, and outputs corresponding automatic debugging steps or debugging descriptions to the controlled device when the abnormal error information is compared to be in accordance with the error model of each debugging subset.
In some embodiments, the debugging system further comprises a control device. The control device is connected with the communication interface device. The control device generates a control input signal. The communication interface device transmits the control input signal to the controlled device. When the abnormal error information does not accord with the error model of at least one debugging subset, the control device generates a debugging control input signal for eliminating the abnormal error information and transmits the debugging control input signal to the controlled device.
In some embodiments, the learning device receives the debug control input signal generated by the control device through the communication interface device when the abnormal error information does not match the error model of the at least one debug subset. The learning device receives abnormality error information of the controlled device through the communication interface device. The learning device builds a new debug subset from at least one debug subset according to the debug control input signal and the abnormal error information.
In some embodiments, the debugging system further comprises a control device. The control device is connected with the learning device. The control device generates a control input signal and outputs the control input signal to the learning device. When the abnormal error information does not accord with the error model of at least one debugging subset, the control device generates a debugging control input signal for eliminating the abnormal error information and transmits the debugging control input signal to the controlled device through the communication interface device.
In some embodiments, when the abnormal error information does not match the error model of at least one debug subset, the learning device analyzes the debug control input signal generated by the control device, and establishes a new debug subset according to the abnormal error information of the controlled device.
In some embodiments, the debugging system further comprises a remote computer and a control device. The remote computer is connected with the communication interface device. The control device is connected with a remote computer. The control device generates a control input signal and outputs the control input signal to the learning device through the remote computer. When the abnormal error information is not in accordance with the error model of the at least one debugging subset, the control device generates a debugging control input signal for eliminating the abnormal error information and transmits the debugging control input signal to the controlled device through the communication interface device.
In some embodiments, the debugging system further comprises a control device. The control device is connected with the communication interface device or the learning device. When the learning device outputs the debug account to the controlled device and the learning device receives a confirmation instruction corresponding to the debug account from the control device, the learning device outputs one or more automatic debug instructions corresponding to the debug account to the controlled device according to the confirmation instruction.
In some embodiments, the debug account includes an auto-debug account. The auto-debug specification corresponds to one or more auto-debug instructions. Upon receiving the confirmation instruction, the learning device outputs each of the one or more auto-debug instructions.
In some embodiments, the debug account includes a step-by-step debug account, and the step-by-step debug account includes a plurality of debug prompts. The debug prompts each correspond to one of the one or more debug commands. The confirmation instruction comprises a plurality of step-by-step confirmation instructions. When the learning device receives one of the step-by-step confirmation instructions, the learning device outputs one of one or more auto-debug instructions corresponding to the received step-by-step confirmation instruction.
In some embodiments, the debugging prompts include a guidance indicator, a guidance click range, or a memory playback movie. The guide index, the guide click range or the memory playback film are displayed on the display device corresponding to the control device in a picture-in-picture or overlapping mode.
Another aspect of the disclosure relates to an early warning system, which includes a communication interface device and a learning device. The communication interface device is connected to the controlled device. The communication interface device comprises a capturing unit. The capturing unit captures an operation picture of the controlled device. And extracting abnormal error information from the operation picture when the controlled device is abnormal. The learning device is connected with the communication interface device. The learning device detects the abnormal error information through the communication interface device. The learning device collects one or more past operation screens of the controlled device before the abnormal error information is generated or one or more past operation commands executed by the controlled device. The learning device generates at least one early warning model according to the collected one or more past operation pictures or one or more past operation instructions. And when the learning device judges that the at least one early warning model is met according to the current operation picture and the current input instruction of the controlled device, the learning device outputs a warning notice.
In some embodiments, the learning device includes a storage unit. The storage unit temporarily stores an operation picture when the controlled device operates. When the learning device detects the abnormal error information through the communication interface device, the learning device extracts one or more past operation pictures or one or more past operation instructions of the controlled device before the abnormal error information is generated from the operation pictures temporarily stored in the storage unit.
In some embodiments, the learning device analyzes one or more past operation screens before the generation of the abnormal error information to obtain the features of the past error screen. The warning notification output by the learning device prompts the past error screen feature.
In some embodiments, the learning device analyzes one or more past operation commands before the abnormal error information is generated to obtain the past error operation command. The warning notification output by the learning device prompts a past erroneous operation command.
In some embodiments, the learning device further comprises at least one debug subset. Any of the at least one debug subset includes an error model and an auto-debug step or a debug description of the corresponding error model, respectively. The learning device receives the abnormal error information through the communication interface device, and outputs corresponding automatic debugging steps or debugging descriptions to the controlled device when the abnormal error information is compared to be in accordance with the error model of each debugging subset.
In summary, by applying the above embodiments, the learning device generates the debug subset after performing segmentation and training according to the operation screen and the debug control input signal recorded by the communication interface device, and then outputs the corresponding automatic debug step or debug description by comparing the current abnormal error information with the error model, so as to achieve the purpose of combining automatic and manual obstacle elimination and simultaneously performing machine learning.
Drawings
Fig. 1A is a schematic diagram illustrating a debug system according to some embodiments of the present disclosure.
FIG. 1B is a schematic diagram illustrating another debugging system, according to some embodiments of the present disclosure.
FIG. 1C is a schematic diagram illustrating another debugging system according to some other embodiments of the present disclosure.
Fig. 2 is a flow chart illustrating a debugging method according to some embodiments of the present disclosure.
Fig. 3 is a diagram illustrating an operation screen according to some embodiments of the present disclosure.
Fig. 4 is a schematic diagram illustrating another operation screen according to some embodiments of the present disclosure.
Fig. 5 is a flow chart illustrating a warning method according to some embodiments of the present disclosure.
Fig. 6 is a flow chart illustrating an early warning model building method according to some embodiments of the present disclosure.
Wherein the reference numerals are as follows:
100a, 100b, 100c
A controlled device
A communication interface device
Picking unit
150
160
162
180
190
200
Operations S210, S220, S230, S240, S250, S260, S270, S280, S290
Prompt window T1, T2 and T3
Buttons B1, B2, B3, B4, B5.
V1, V2
M1, M2
MR.. image
500
600.. early warning model building method
S510, S520, S610, S620, S630
Detailed Description
The following detailed description of the embodiments is provided to better understand the aspects of the present disclosure, but the embodiments are not provided to limit the scope of the present disclosure, and the description of the structural operations is not provided to limit the order of execution thereof, and any structure resulting from the rearrangement of elements to produce a device with equivalent efficacy is within the scope of the present disclosure. Moreover, the drawings are for illustrative purposes only and are not drawn to scale in accordance with industry standard and conventional practice, and the dimensions of the various features may be arbitrarily increased or decreased for clarity of illustration. In the following description, the same elements will be described with the same reference numerals for ease of understanding.
The term (terms) used throughout the specification and claims has the ordinary meaning as commonly understood in the art, in the disclosure herein and in the claims, unless otherwise indicated. Certain words used to describe the disclosure are discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art in describing the disclosure.
Furthermore, as used herein, the terms "comprising," including, "" having, "" containing, "and the like are open-ended terms that mean" including, but not limited to. Further, as used herein, "and/or" includes any and all combinations of one or more of the associated listed items.
When an element is referred to as being "connected" or "coupled," it can be referred to as being "electrically connected" or "electrically coupled. "connected" or "coupled" may also be used to indicate that two or more elements are in mutual engagement or interaction.
Moreover, although terms such as "first," "second," …, etc., may be used herein to describe various elements, these terms are used merely to distinguish one element or operation from another element or operation described in similar technical terms. Unless the context clearly dictates otherwise, the terms do not specifically refer or imply an order or sequence nor are they intended to limit the invention.
Fig. 1A is a schematic diagram illustrating a debug system 100a according to some embodiments of the present disclosure. As shown in fig. 1A, the debugging system 100a includes a controlled device 120, a communication interface device 140, a learning device 160 and a control device 180. The communication interface device 140 connects the controlled device 120, the learning device 160, and the manipulation device 180. The communication interface device 140 includes an acquisition unit 142. Learning device 160 includes a storage unit 162. In one embodiment, the learning device 160 may be a remote computer disposed at a different location (e.g., a different room or a different block of the same room) from the controlled device 120.
Specifically, the communication interface device 140 may be a KVM switch or an iKVM switch (network-based switch). The retrieving unit 142 may be implemented by a processor in the communication interface device 140, or hardware or software capable of retrieving an operation screen of the controlled device 120.
The controlled device 120 may be a machine, a computer, a tool on a production line, etc., and is connected to the communication interface device 140 through a VGA terminal (Video Graphics Array connector) and/or a Universal Serial Bus (USB). The controlled device 120 may also be formed by connecting and combining various electronic devices, so as to be further connected to the communication interface device 140 through a VGA terminal and a USB.
The control device 180 can be an input device such as a keyboard, a mouse, a touch pad, a touch screen, a machine button, etc., and is connected to the communication interface device 140 through a Serial port and/or a Universal Serial Bus (USB).
The learning device 160 can be a computer, a tablet, a mobile phone, etc. and is connected to the communication interface device 140 through a VGA terminal, a DVI, an HDMI, a DP (Display Port) connection interface and/or a USB (universal serial bus). In some embodiments, the learning device 160 is connected to the communication interface device 140 through a Local Area Network (LAN) or the Internet (Internet). In addition, the learning device 160 can be located at the proximal end or the distal end of the controlled device 120, and the invention is not limited thereto.
The memory unit 162 may include one or more memories. The memory may be implemented by a Read Only Memory (ROM), a flash memory, a floppy disk, a hard disk, an optical disk, a flash disk, a usb disk, a magnetic tape, a database readable from a network, or any recording medium having the same functions as those of the present disclosure, which will be apparent to those skilled in the art.
Fig. 1B is a schematic diagram illustrating another debugging system 100B, according to some embodiments of the present disclosure. In the embodiment shown in fig. 1B, similar components to those in the embodiment of fig. 1A are denoted by the same reference numerals, and the contents thereof are already described in the previous paragraphs, and are not repeated herein. In contrast to the embodiment shown in fig. 1A, in the present embodiment, the debug system 100b further comprises a remote computer 150. The control device 180 is connected to the remote computer 150, and the remote computer 150 is connected to the communication interface device 140. Specifically, the console device 180 can be connected to the remote computer 150 via various serial ports and/or a universal serial bus. The remote computer 150 may be connected to the communication interface device 140 through a local area network or the internet.
It should be noted that the hardware devices or the connection lines are only for convenience of illustration, but the disclosure is not limited thereto. In addition, although fig. 1B shows a controlled device 120, a remote computer 150, a learning device 160, a communication interface device 140 and a control device 180, the numbers are merely examples for convenience of illustration and are not intended to limit the disclosure. That is, in some other embodiments, the communication interface device in the debugging system can be connected to a plurality of controlled devices and/or a plurality of learning devices.
Fig. 1C is a schematic diagram illustrating another debugging system 100C, according to some embodiments of the present disclosure. In the embodiment shown in fig. 1C, similar components to those in the embodiment of fig. 1A are denoted by the same reference numerals, and the contents thereof are already described in the previous paragraphs, and are not repeated herein. Compared to the embodiment shown in fig. 1A, in the present embodiment, the control device 180 included in the debugging system 100c is connected to the learning device 160. Specifically, the control device 180 can be connected to the learning device 160 through various serial ports and/or a universal serial bus. In this embodiment, the learning device 160 may be a remote computer disposed at a different location (e.g., a different room or a different block of the same room) from the controlled device 120.
It should be noted that the hardware devices or the connection lines are only for convenience of illustration, but the disclosure is not limited thereto. In addition, although one controlled device 120, one learning device 160, one communication interface device 140, and one manipulation device 180 are illustrated in fig. 1A and 1C, the numbers thereof are merely examples for convenience of explanation and are not intended to limit the present disclosure. That is, in some other embodiments, the communication interface device in the debugging system can be connected to a plurality of controlled devices and/or a plurality of learning devices.
Fig. 2 is a flow chart illustrating a debugging method 200 according to some embodiments of the present disclosure. For the sake of illustration, the detailed operation of the various elements of the debug system 100a, 100B or 100C will be described in conjunction with the embodiments shown in fig. 1A, 1B or 1C, but not limited thereto, and various modifications and alterations can be made by those skilled in the art without departing from the spirit and scope of the present disclosure.
The following paragraphs will first explain an example of an application of FIG. 2 to debug system 100a shown in FIG. 1A. First, in an embodiment, the communication interface device 140 detects whether an abnormality occurs in the controlled device 120 (as shown in operation S210). If no abnormality occurs in the controlled device 120, the communication interface device 140 continuously detects whether an abnormality occurs in the controlled device 120 (i.e., operation S210 is repeated). Specifically, when the controlled device 120 has an error message, a window is popped out of the error window, or a warning sound is generated, the communication interface device 140 determines that an abnormality occurs in the controlled device 120.
When the controlled device 120 is abnormal, the capturing unit 142 captures an operation screen of the controlled device 120 (as shown in operation S220). Then, the retrieving unit 142 extracts the abnormal error information from the operation screen (as shown in operation S230). Specifically, the capturing unit 142 can be used to extract the text and/or the image in the operation screen as the abnormal error information.
Next, the learning device 160 receives the abnormal error information through the communication interface device 140 (as shown in operation S240).
Then, the learning device 160 compares whether the abnormal error information matches the error model of each error in the at least one debug subset (operation S250). Specifically, the learning device 160 includes at least one debugging subset, and any one of the debugging subsets includes an error model and an automatic debugging step or a debugging description corresponding to the error model, respectively. In one embodiment, each debug subset is stored in memory units 162.
Here, when the abnormal error information matches the error model, the learning device 160 outputs a corresponding auto-debugging step or debug statement to the controlled device 120 (as shown in operation S260). Specifically, in some embodiments, when the abnormal error information matches the error model, the learning device 160 will directly output a corresponding auto-debugging step to the controlled device 120 for debugging, wherein the auto-debugging step may include one or more auto-debugging instructions. The debug command may be a sequence of one or more keyboard key entries or mouse click buttons. In this way, when the debug system 100a detects that the controlled device 120 is abnormal, it can output the corresponding auto-debug command to the controlled device 120 according to the comparison result between the abnormal error information and the error model, so as to perform automatic obstacle elimination.
In some other embodiments, the learning device 160 outputs a debug account to the controlled device 120 when the abnormal error information matches the error model, wherein the debug account can be a prompt window containing text or images. In further detail, after the learning device 160 outputs the debug account to the controlled device 120, the learning device 160 waits for the receiving of the confirmation instruction corresponding to the debug account from the control device 180, and outputs one or more automatic debug instructions corresponding to the debug account to the controlled device 120 according to the confirmation instruction. For example, the debug account may include an auto-debug account (e.g., as shown in FIG. 3 in hint window T1), which may correspond to one or more auto-debug instructions. When the learning device 160 receives the confirmation command (e.g., clicking the button B1 in fig. 3) from the control device 180, the learning device 160 outputs all the sdds corresponding to the confirmation command to the controlled device 120 for debugging.
As another example, the debug account includes a step-by-step debug account, which includes a plurality of debug prompts (e.g., prompt windows T2, T3 shown in FIG. 4). The debug prompts each correspond to one of the one or more debug instructions. The confirmation command includes a plurality of step-by-step confirmation commands (e.g., the operation of sequentially clicking the buttons B2 and B5 in FIG. 4). When the learning device 160 receives one of the step-by-step confirmation commands from the control device 180, the learning device 160 outputs an automatic debugging command corresponding to the step-by-step confirmation command.
In some embodiments, the debug prompts may include guide pointers (e.g., prompt windows T2 and T3 as shown in FIG. 4), guide click ranges (e.g., ranges M1 and M2 as shown in FIG. 4), or memory playback movies (e.g., video MR as shown in FIG. 4). The guidance index, the guidance click range or the memory playback movie are displayed on the display device corresponding to the control device 180 in a picture-in-picture manner or in a superposition manner. For example, the past operation screen is played in the window V1 shown in fig. 4, and the current operation screen is displayed in the window V2 and some of the instruction execution buttons B2 to B5 corresponding to the current operation screen are set.
It is to be noted that the operation screens shown in fig. 3 and 4 can be displayed by the display device 190 and viewed by the user. The display device 190 may be various displays, computer screens, tablets, mobile phones, etc., and the invention is not limited thereto. In addition, the display device 190 can be connected to the controlled device 120, the communication interface device 140, or the learning device 160 according to the position of the user for operation or the connection relationship of the control device 180.
In this way, when the debug system 100a detects that the controlled device 120 is abnormal, the corresponding automatic debug instruction or the step-by-step debug instruction can be output according to the comparison result between the abnormal error information and the error model, so that the user can select all the automatic debug instructions given by the debug system 100a through the confirmation instruction to perform automatic obstacle elimination, or determine whether to select the next step of automatic debug instruction given by the current debug system 100a according to the debug prompt through the step-by-step confirmation instruction. In other words, the user may also reject the automatic debugging command given by the debugging system 100a, and instead perform manual troubleshooting by the control device 180 (please refer to the following description of operations S270 to S290).
When the user rejects the automatic debugging command given by the debugging system 100a, or when the abnormal error information does not conform to the error model, the user can generate a debugging operation input signal for eliminating the abnormal error information by operating the operation control device 180, so as to further transmit the debugging operation input signal to the controlled device 120 (as shown in operation S270). In other words, the user can manually debug the controlled device 120 by operating the control device 180.
Next, the learning device 160 receives the debug control input signal generated by the control device 180 through the communication interface device 140 (as shown in operation S280). In some embodiments, operation S280 and operation S270 may be performed simultaneously. Specifically, the user generates the debug control input signals through the control device 180, and the communication interface device 140 receives the debug control input signals and transmits the debug control input signals to the controlled device 120 and the learning device 160.
Then, the learning device 160 updates or creates an additional debug subset from at least one debug subset according to the debug operation input signal and the abnormal error information (as shown in operation S290). Specifically, after the controlled device 120 is abnormal, the communication interface device 140 starts to capture the movie of the operation screen during the error elimination process and the debug control input signal generated from the control device 180, such as the keyboard input data, the mouse click coordinates, and other data. In addition, the communication interface device 140 also records the time of generation of these debug manipulation input signals corresponding to the operation screen movie. Then, the learning device 160 analyzes the interface transformation as the segmentation points by a Scene change detection algorithm (Scene change detection), and trains (training) the image and the input data between each segmentation point as a training data to update or establish a new debug subset (i.e., a new error model and an automatic debugging step or a debugging description corresponding to the error model). For example, the division point may be a window switch in the operation screen movie, or may be a preset keyboard shortcut key input or a mouse click on a specific button.
In this way, when the user rejects the automatic debugging command given by the debugging system 100a, or when the abnormal error information does not conform to the error model, the user can perform manual obstacle elimination through the control device 180, and the communication interface device 140 records and trains the operation screen movie and the debugging control input signal in the process, so that the learning device 160 can perform machine learning to update or establish a new error model and the corresponding automatic debugging command.
The following paragraphs will next describe an exemplary application of FIG. 2 to debug system 100B shown in FIG. 1B. When fig. 2 is applied to the debug system 100B shown in fig. 1B, in operation S270, the control device 180 generates a debug control input signal for eliminating the abnormal error information, so as to further transmit the debug control input signal to the controlled device 120 through the remote computer 150 and the communication interface device 140. In other words, the user can perform manual debugging on the controlled device 120 at a remote location through the control device 180 connected to the remote computer 150. In this embodiment, the remote computer 150 obtains the image of the controlled device 120 through the communication interface device 140, so that the user can control the controlled device 120 through the control device 180 at the remote location.
When fig. 2 is applied to the debug system 100B shown in fig. 1B, in operation S280B, the learning device 160 analyzes the debug manipulation input signal generated by the manipulation device 180. In this way, when the user rejects the automatic debugging command given by the debugging system 100b, or when the abnormal error information does not conform to the error model, the user can perform manual obstacle elimination through the control device 180 connected to the remote computer 150, and transmit the debugging control input signal to the controlled device 120 through the communication interface device 140 for debugging. The learning device 160 can record the debug control input signal through the communication interface device 140 and train the debug control input signal to perform machine learning, so as to update or establish a new error model and corresponding automatic debug command.
The following paragraphs will next describe an example of the application of FIG. 2 to debug system 100C shown in FIG. 1C. When fig. 2 is applied to the debug system 100C shown in fig. 1C, in operation S270, the control device 180 generates a debug control input signal for eliminating abnormal error information and transmits the debug control input signal to the controlled device 120 through the learning device 160 and the communication interface device 140. In other words, the user can perform manual debugging on the controlled device 120 remotely through the control device 180 connected to the learning device 160.
When fig. 2 is applied to the debug system 100c shown in fig. 1B, in operation S280, the learning device 160 analyzes the debug manipulation input signal generated by the manipulation device 180. In this way, when the user rejects the automatic debugging command given by the debugging system 100c, or when the abnormal error information does not conform to the error model, the user can perform manual obstacle elimination through the control device 180 connected to the learning device 160, so that the learning device 160 can record the debugging control input signal and perform training for machine learning, so as to update or establish a new error model and the corresponding automatic debugging command, and transmit the debugging control input signal to the controlled device 120 through the communication interface device 140 for debugging.
Please refer to fig. 5. Fig. 5 is a flow chart illustrating an early warning method 500 according to some embodiments of the present disclosure. As shown in fig. 5, the warning method 500 includes operations S510 and S520.
First, in operation S510, the learning device 160 determines whether at least one pre-warning model is satisfied according to the current operation screen and the current input command of the controlled device 120. When not, operation S510 is continued. When the agreement is made, operation S520 is performed, and the learning device 160 outputs a warning notification. Specifically, the warning notification may be a warning window containing text or images, a flashing warning light on the machine, or a warning sound.
Please refer to fig. 6. Fig. 6 is a flow chart illustrating an early warning model building method 600 according to some embodiments of the present disclosure. As shown in fig. 6, the early warning model building method 600 includes operations S610, S620, and S630.
First, in operation S610, the capturing unit 142 captures a past operation screen or a past operation command of the controlled apparatus 120. Next, in operation S620, whether an abnormality occurs in the controlled device 120 is detected. Operation S620 is similar to operation S210 and will not be described herein. When no abnormality occurs in the controlled device 120, operations S610 and S620 continue. When the controlled device 120 is abnormal, that is, when the controlled device 120 is determined to be abnormal through operation S620, operation S630 is performed, and the learning device 160 generates at least one warning model according to the collected one or more past operation screens or one or more past operation commands.
Specifically, the capturing unit 142 of the communication interface device 140 records the operation screen of the controlled device 120 and the operation command generated by the control device 180 at any time. When the controlled device 120 is abnormal, the learning device 160 receives the past operation screen during a period before the abnormal condition of the controlled device 120 occurs and/or the past operation command generated by the control device 180 during the period, such as the data of keyboard input data and mouse click coordinates, through the communication interface device 140. Further, the communication interface device 140 also records the time when these operation instructions are generated with respect to the operation screen. Then, the learning device 160 analyzes the interface transformation by a scene transformation algorithm and uses the interface transformation as the segmentation points, and trains the images and the operation commands between the segmentation points as a set of training data to establish the early warning model. For example, the division point may be a window switch in the operation screen movie, or may be a preset keyboard shortcut key input or a mouse click on a specific button.
In some embodiments, the memory unit 162 of the learning device 160 is used for temporarily storing the operation screen of the controlled device 120 during operation. When the learning device 160 detects the abnormal error information through the communication interface device 140, the learning device 160 extracts one or more past operation screens or one or more past operation commands of the controlled device 120 before the abnormal error information is generated from the operation screens temporarily stored in the storage unit 162.
In addition, the learning device 160 is further configured to analyze one or more past operation screens before the generation of the abnormal error information to obtain a feature of a past error screen, and when the warning notification is output by the learning device 160, the warning notification includes a prompt of the feature of the past error screen. The learning device 160 is further configured to analyze one or more past operation commands before the generation of the abnormal error information to obtain a past error operation command, and when the warning notification is output by the learning device 160, the warning notification includes a prompt of the past error operation command.
In this way, by performing machine learning on the operation screen and the operation command before the abnormal error information occurred in the past, and by determining whether the current operation screen and the current input command of the controlled device 120 conform to the early warning model, a warning notification can be issued before the abnormal error information may occur, so as to avoid repeatedly generating the same abnormal error information.
While the disclosed methods are illustrated and described herein as a series of steps or events, it will be appreciated that the order of the steps or events shown is not to be interpreted in a limiting sense. For example, some steps may occur in different orders and/or concurrently with other steps or events apart from those illustrated and/or described herein. In addition, not all illustrated steps may be required to implement one or more aspects or embodiments described herein. Furthermore, one or more steps herein may also be performed in one or more separate steps and/or stages.
It should be noted that, in the case of no conflict, the features and circuits in the respective drawings, embodiments and embodiments of the present disclosure may be combined with each other. The circuits shown in the figures are for illustration purposes only and are simplified to simplify the description and facilitate understanding, and are not meant to be limiting. In addition, each device, unit and element in the above embodiments can be implemented by various types of digital or analog circuits, or can be implemented by different integrated circuit chips, or integrated into a single chip. The foregoing is merely exemplary and the disclosure is not limited thereto.
In summary, by applying the above embodiments, the learning device 160 generates the debug subset after performing segmentation and training according to the operation screen and the debug control input signal recorded by the communication interface device 140, and then outputs the corresponding automatic debug step or debug description by comparing the current abnormal error information with the error model, so as to achieve the purpose of combining automatic and manual obstacle elimination and simultaneously performing machine learning.
Although the present disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the disclosure, and therefore, the scope of the disclosure should be determined by that defined in the appended claims.
Claims (15)
1. A debug system, comprising:
the communication interface device is in communication connection with a controlled device and comprises a capturing unit, the capturing unit captures an operation picture of the controlled device, and abnormal error information is extracted from the operation picture when the controlled device is abnormal; and
the learning device is connected with the communication interface device and comprises at least one debugging subset, any one of the at least one debugging subset comprises an error model and an automatic debugging step or a debugging description corresponding to the error model, the learning device receives the abnormal error information through the communication interface device, and the learning device outputs the automatic debugging step or the debugging description corresponding to the at least one debugging subset to the controlled device when the abnormal error information is compared to be matched with any one of the error models of the at least one debugging subset.
2. The debugging system of claim 1, further comprising:
the control device is connected with the communication interface device and generates a control input signal, and the communication interface device transmits the control input signal to the controlled device;
when the abnormal error information does not accord with the error model of the at least one debugging subset, the control device generates a debugging control input signal for eliminating the abnormal error information and transmits the debugging control input signal to the controlled device.
3. The system of claim 2, wherein the learning device receives the debug control input signal generated by the control device through the communication interface device when the error information does not match the error model of the at least one debug subset, the learning device receives the error information of the controlled device through the communication interface device, and the learning device establishes a new debug subset from the at least one debug subset according to the debug control input signal and the error information.
4. The debugging system of claim 1, further comprising:
the control device is connected with the learning device and generates a control input signal and outputs the control input signal to the learning device;
when the abnormal error information does not accord with the error model of the at least one debugging subset, the control device generates a debugging control input signal for eliminating the abnormal error information and transmits the debugging control input signal to the controlled device through the communication interface device.
5. The debugging system of claim 4, wherein the learning device analyzes the debug manipulation input signal generated by the manipulation device and creates a new debug subset according to the abnormal error information of the controlled device when the abnormal error information does not match the error model of the at least one debug subset.
6. The debugging system of claim 1, further comprising:
the remote computer is connected with the communication interface device; and
the control device is connected with the remote computer, generates a control input signal and outputs the control input signal to the learning device through the remote computer;
when the abnormal error information does not accord with the error model of the at least one debugging subset, the control device generates a debugging control input signal for eliminating the abnormal error information and transmits the debugging control input signal to the controlled device through the communication interface device.
7. The debugging system of claim 1, further comprising:
the control device is connected with the communication interface device or the learning device;
when the learning device outputs the debug account to the controlled device and the learning device receives a confirmation instruction corresponding to the debug account from the control device, the learning device outputs one or more automatic debug instructions corresponding to the debug account to the controlled device according to the confirmation instruction.
8. The debugging system of claim 7, wherein the debug account comprises an auto-debug account, the auto-debug account corresponding to the one or more auto-debug instructions, the learning device outputting each of the one or more auto-debug instructions upon receipt of the validation instruction.
9. The system of claim 7, wherein the debug account includes a step-by-step debug account, the debug account includes a plurality of debug prompts, each of the debug prompts corresponds to one of the one or more auto-debug commands, the validation command includes a plurality of step-by-step validation commands, and the learning device outputs one of the one or more auto-debug commands corresponding to the step-by-step validation command when the learning device receives one of the step-by-step validation commands.
10. The debugging system of claim 9, wherein the debugging hint comprises a guidance indicator, a guidance click range, or a memory playback movie, and wherein the guidance indicator, the guidance click range, or the memory playback movie is displayed in a PIP display or in an overlay manner on a display device corresponding to the control device.
11. An early warning system, comprising:
the communication interface device is connected to the controlled device and comprises a capturing unit, the capturing unit captures an operation picture of the controlled device, and abnormal error information is extracted from the operation picture when the controlled device is abnormal; and
the learning device is connected with the communication interface device, detects the abnormal error information through the communication interface device, and collects one or more past operation pictures of the controlled device or one or more past operation instructions executed by the controlled device before the abnormal error information is generated; and
the learning device generates at least one early warning model according to the collected one or more past operation pictures or one or more past operation instructions;
when the learning device judges that the at least one early warning model is met according to the current operation picture and the current input instruction of the controlled device, the learning device outputs a warning notice.
12. The warning system of claim 11, wherein the learning device includes a storage unit for temporarily storing the operation screen during operation of the controlled device,
when the learning device detects the abnormal error information through the communication interface device, the learning device extracts the one or more past operation pictures or the one or more past operation instructions of the controlled device before the abnormal error information is generated from the operation pictures temporarily stored in the storage unit.
13. The warning system of claim 11 wherein the learning device analyzes the one or more previous operation frames before the abnormal error message is generated to obtain a previous error frame characteristic, the warning notification output by the learning device indicating the previous error frame characteristic.
14. The warning system of claim 11, wherein the learning device analyzes the one or more past operation commands before the abnormal error message is generated to obtain a past error operation command, and the warning notification output by the learning device indicates the past error operation command.
15. The warning system of claim 11, wherein the learning device further comprises at least one debugging subset, each of the at least one debugging subset comprises an error model and an auto-debugging step or a debugging specification corresponding to the error model, the learning device receives the abnormal error information through the communication interface device, and the learning device outputs the corresponding auto-debugging step or the debugging specification to the controlled device when the abnormal error information is compared to match the error model of the at least one debugging subset.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW109102620 | 2020-01-22 | ||
| TW109102620A TWI741488B (en) | 2020-01-22 | 2020-01-22 | Debugging system and early warning system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113157471A true CN113157471A (en) | 2021-07-23 |
| CN113157471B CN113157471B (en) | 2024-05-24 |
Family
ID=76882232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010498637.3A Active CN113157471B (en) | 2020-01-22 | 2020-06-04 | Debug system and early warning system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113157471B (en) |
| TW (1) | TWI741488B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW563017B (en) * | 2002-06-25 | 2003-11-21 | Mitac Int Corp | Device and method for executing single-step interrupt debugging in first stage of computer booting through PCMCIA interface |
| US20040002994A1 (en) * | 2002-06-27 | 2004-01-01 | Brill Eric D. | Automated error checking system and method |
| CN1479204A (en) * | 2002-08-26 | 2004-03-03 | 联发科技股份有限公司 | Error eliminating device |
| CN1786924A (en) * | 2004-12-06 | 2006-06-14 | 明基电通股份有限公司 | Electronic appliance with error eliminating function and method thereof |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2717515A1 (en) * | 2012-06-30 | 2014-04-09 | Huawei Technologies Co., Ltd. | Virtual port monitoring method and device |
| US9094336B2 (en) * | 2013-03-15 | 2015-07-28 | Ixia | Methods, systems, and computer readable media for assisting with the debugging of conditions associated with the processing of test packets by a device under test |
| US10243862B2 (en) * | 2013-03-15 | 2019-03-26 | Gigamon Inc. | Systems and methods for sampling packets in a network flow |
| US10664766B2 (en) * | 2016-01-27 | 2020-05-26 | Bonsai AI, Inc. | Graphical user interface to an artificial intelligence engine utilized to generate one or more trained artificial intelligence models |
-
2020
- 2020-01-22 TW TW109102620A patent/TWI741488B/en active
- 2020-06-04 CN CN202010498637.3A patent/CN113157471B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW563017B (en) * | 2002-06-25 | 2003-11-21 | Mitac Int Corp | Device and method for executing single-step interrupt debugging in first stage of computer booting through PCMCIA interface |
| US20040002994A1 (en) * | 2002-06-27 | 2004-01-01 | Brill Eric D. | Automated error checking system and method |
| CN1479204A (en) * | 2002-08-26 | 2004-03-03 | 联发科技股份有限公司 | Error eliminating device |
| CN1786924A (en) * | 2004-12-06 | 2006-06-14 | 明基电通股份有限公司 | Electronic appliance with error eliminating function and method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| TW202129495A (en) | 2021-08-01 |
| TWI741488B (en) | 2021-10-01 |
| CN113157471B (en) | 2024-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10313627B2 (en) | Apparatus for playing back recorded video images related to event, and method thereof | |
| WO2014138097A1 (en) | Mobile device monitoring and testing | |
| CN104081784A (en) | Information processing device, information processing method, and program | |
| CN107211112A (en) | Image processing equipment, image processing system, image processing method and program | |
| CN113190106B (en) | Gesture recognition method and device and electronic equipment | |
| CN113253833A (en) | System and method for augmented reality applications | |
| US20210117040A1 (en) | System, method, and apparatus for an interactive container | |
| CN109669831A (en) | A kind of test method and electronic equipment | |
| CN106411649A (en) | Control method and device for remote play system | |
| US20180336435A1 (en) | Apparatus and method for classifying supervisory data for machine learning | |
| CN113157471B (en) | Debug system and early warning system | |
| JP2009116535A (en) | Automatic test system, automatic test method and program | |
| JPWO2010016251A1 (en) | Video processing device | |
| CN106789472A (en) | Consumer electronics control CEC function detecting methods, apparatus and system | |
| CN107995538B (en) | Video annotation method and system | |
| US11557065B2 (en) | Automatic segmentation for screen-based tutorials using AR image anchors | |
| KR20220167353A (en) | Methods, apparatuses, systems, media and computer devices for processing data | |
| CN112559340A (en) | Picture testing method, device, equipment and storage medium | |
| WO2021154510A1 (en) | Annotated decode of oscilloscope signals | |
| JPH0534181A (en) | Plant monitor system | |
| CN112307882A (en) | Image determination device and image determination system | |
| CN115357519B (en) | Test method, device, equipment and medium | |
| CN114780321A (en) | Detection method and detection device for control function on display screen and electronic equipment | |
| US11468657B2 (en) | Storage medium, information processing apparatus, and line-of-sight information processing method | |
| CN112817790B (en) | Method for simulating user behavior |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |