TWI693535B - Mouse cursor image detection comparison and feedback state judgment method - Google Patents

Abstract

The invention provides a mouse cursor image detection comparison and feedback state judgment method, which is based on a non-intrusive data acquisition system architecture and uses an image processing unit to read the mouse in the operation screen output by the machine controller Cursor image detection and comparison, the method includes first obtaining the cursor foreground image and background image set by the user, and using an algorithm to generate a cursor mask from the foreground image and background image, and then reading the mouse The cursor image is applied with the cursor mask as a sample comparison, which can return the information of the comparison result and the feedback status of the mouse cursor to the software control system, provide the subsequent correction system for the corresponding mouse cursor processing procedure, and confirm the mouse Whether the movement of the cursor conforms to the position controlled by the feedback and correction system to complete the closed-loop control of the mouse cursor, provide a more accurate machine control program, and can also be used to record the user operation process or provide a large number of user interfaces Intermediary of information.

Description

Mouse cursor image detection comparison and feedback state judgment method

The invention provides a mouse cursor image detection comparison and feedback state judgment method, in particular, the mouse cursor image in the operation screen of the machine can be read through the image processing unit for detection and comparison, and the comparison result The information and the feedback status of the mouse cursor are provided to the subsequent correction system for corresponding processing procedures to complete the closed-loop control of the mouse cursor.

According to the press, the advancement of information technology, the development of the network and the substantial improvement of computer computing power, and the continuous improvement of data collection and storage technology have also accelerated the accumulation, recording and acquisition of big data, and the use of cloud in industrial production machines When developing big data applications, one of the important tasks is to collect relevant machine data at the production line machine (or cloud end point). Since many important data come from the machine software user interface, the operator will Manipulate many screens on the user interface, this kind of operation behavior will help to understand the current data collection and analysis of the machine data adjustment, machine program interface related data acquisition and program related parameter settings currently being performed by the operator application.

The user interface of the general machine is the medium for interaction and information exchange between the system and the operator, including human-machine interaction and graphical user interface, so that the system can input data such as keyboard, mouse, touch pad, etc. through the operator The device operates the transmitted signal to control the operation screen on the machine, but when it is controlled by the signal transmitted unidirectionally with the mouse, it often depends on the operation system to be operated. Due to the limitation of system resources, the signal is lost, which makes it impossible to control the cursor of the mouse correctly. If the old machine is to be upgraded, it will also have difficulty in expanding its hardware equipment and modifying the software system. When the mouse controls data, in addition to affecting the load of its processor, the resources on the computer are also subject to many restrictions, including the processing procedures and performance of current machine programs; in addition, based on trade secrets, hardware and software Restrictions such as confidentiality agreements and computing resources cannot load the feedback program controlled by the relevant mouse on the target system by intrusive data acquisition, and the speed of the mouse cursor movement may also appear due to differences in the system settings of the other party Inconsistent movements cause the speed of the mouse cursor movement to be incorrectly predicted and cause errors. The above-mentioned factors will not be conducive to the collection of a large amount of data on the user interface. It needs to be redesigned by those engaged in this industry. To be effectively resolved.

Therefore, in view of the above-mentioned practice in collecting machine data, in addition to affecting the load of the processor, the system resources of the old machine are also subject to many restrictions and are not conducive to obtaining the user interface A large number of data problems and deficiencies, after many evaluations and considerations, and using years of research and development experience in this industry, this kind of mouse cursor image detection comparison and feedback status judgment method invention patent was born.

The main purpose of the present invention is that the non-intrusive data acquisition system can detect the mouse cursor image that the target machine side actually controls the mouse in the operation screen through the image processing unit without interfering with any production machine. After applying the cursor mask for sample comparison, the information of the comparison result and the feedback status of the mouse cursor can be returned to the software control system, and the subsequent correction system can be provided for corresponding processing procedures to ensure its transmission and control of the mouse. The correctness of the movement to complete the closed loop control of the mouse cursor, which not only does not affect the current production and load of the machine, It also provides a feedback correction mechanism in the control and monitoring of the mouse cursor, allowing the mouse cursor to provide more accurate and reliable mouse cursor position information and more accurate while allocating resources in the central processor or the operating system. The machine control program can also be used to record the user's operation flow and behavior, or provide an intermediate medium to retrieve a large amount of user interface information.

The secondary object of the present invention is that the image processing unit can be matched with the comparison operation to provide single or multiple position, number and stability information to design the mouse cursor position feedback state and correction process, including the cursor foreground and background set by the user Image, and apply a cursor mask, the concept of the area of concern and the area of inconsistency, the concept of obtaining the preset threshold through the interval of each brightness value interval on the horizontal axis of the image histogram, the design process and line of the comparison unit The comparison method of scanning, and in the judgment process of the feedback state mechanism of the mouse cursor, the mouse cursor is moved through the comparison result provided by the image processing unit, including the detection of the stability, multiple positions, and multiple positions of the mouse cursor in the entire design process. Mouse cursor judgment, analysis of past mouse cursor movements, recovery method of the mouse cursor returning to the starting position and then moving to the target position, use of the safe area, mouse cursor position compensation, error analysis, or other related mouse cursors Status etc.

Another object of the present invention is that the image processing unit can detect the image of the mouse cursor in the operation screen of the machine, instead of the function of the original human eye, and the field programmable logic gate array performs the comparison and detection of the image of the mouse cursor The process does not need to pass through the central processor of the target machine, in addition to reducing the operating load, and ensuring the real-time image processing, and the image processing unit returns the detection and comparison results, nor does it need to obtain the target machine in advance The movement speed of the mouse cursor on the platform system can be controlled by the coordinate record of the mouse cursor through the feedback and correction system, so that the current feedback status information such as the position, number and stability of the image of the mouse cursor can be correctly obtained to provide The traditional non-invasive data acquisition system has more accurate machine control procedures.

100‧‧‧machine controller

101‧‧‧ screen

102‧‧‧Keyboard/Mouse

200‧‧‧ Non-intrusive data retrieval system

201‧‧‧Image capture device

202‧‧‧Image processing unit

203‧‧‧ software control system

204‧‧‧Image output device

205‧‧‧Signal receiving device

206‧‧‧Human-machine interface simulation device

300‧‧‧Cursor mask

301‧‧‧Care area

302‧‧‧Don’t care area

303‧‧‧arrow cursor

304‧‧‧I-shaped cursor

305‧‧‧Shadow

The first figure is a system architecture diagram of the present invention for retrieving machine controller data under a non-invasive data retrieval system architecture.

The second figure is a flow chart of the steps of the image processing unit of the present invention as a mouse cursor image detection comparison.

The third figure is a schematic diagram of the present invention in which the cursor mask is placed on the mouse cursor image in the operation screen.

The fourth figure is a flow chart of the steps of the optimization process of the cursor mask of the present invention.

The fifth figure A is a schematic diagram of the cursor mask of the present invention through the threshold value to perform image binarization and area growth to calibrate the inside of the arrow cursor.

Fig. 5B is a schematic diagram of pixel filling of the I-shaped cursor of the present invention on the X axis.

The sixth figure is a flow chart of the image processing unit of the present invention reading the mouse cursor image and outputting the comparison result and the feedback status judgment through pattern comparison.

The seventh figure is a block diagram of the image processing unit of the present invention reading the mouse cursor image and outputting the comparison result and the feedback status judgment through pattern comparison.

Figure 8A is a flowchart (1) of the steps of the image processing unit of the present invention for determining the feedback state of the mouse cursor.

Figure 8B is a flowchart (2) of the steps of the image processing unit of the present invention for determining the feedback state of the mouse cursor.

Figure 8C is a flowchart (3) of the steps of the image processing unit of the present invention for determining the feedback state of the mouse cursor.

Figure 8D is a flowchart (4) of the steps of the image processing unit of the present invention for determining the feedback state of the mouse cursor.

In order to achieve the above-mentioned objectives and effects, the technical means and the structure adopted by the present invention, the drawings and the detailed description of the preferred embodiment of the present invention, the structure and function of which are as follows, fully understand.

Please refer to the first figure, which is a system architecture diagram of the present invention for capturing machine controller data under a non-invasive data acquisition system architecture. It can be clearly seen from the figure that the machine controller 100 of the present invention The installed machine operation program can generate the original operation screen of the control machine, and based on the structure of the non-intrusive data acquisition system 200, the image acquisition device 201 can be used to capture the operation screen output by the machine controller 100, and then separately Send to the non-intrusive data acquisition system 200 and image processing unit 202 to capture the information in the operation screen for the software control system 203 to simultaneously construct the machine simulation operation screen provided to the on-site operator, and the machine simulation operation A customized operation interface can also be added to the screen, and the post-machine simulation operation screen can be displayed on the screen 101 to which the machine belongs through the image output device 204, or the keyboard/slide can be received through the signal receiving device 205 The control signal of the mouse 102 is sent to the software control system 203 for processing, so that the human-machine interface simulation device 206 can simulate the keyboard/mouse 102 to send the corresponding control signal to the machine controller 100.

In this embodiment, the image processing unit 202 is a post-processing unit of the image of the machine operation screen captured by the image capturing device 201, and the image processing-related algorithm can be constructed on the field programmable logic gate array ( FPGA), special application integrated circuit (ASIC)/application-specific chip (ASSP), microprocessor or other chip inside, But not limited to the physical level, it can also provide high-speed image recognition, comparison, motion detection and mouse cursor position positioning for applications or other intermediary software and hardware, and the machine connected to the machine controller 100 In addition to including machines or equipment on the production line, it also has a computer system or embedded system that provides a user interface, including but not limited to processing modules, memory or communication modules, etc., and any operating system that uses a graphical interface window ( (Such as Windows, Linux, Unix or other operating systems).

Please refer to the second to seventh figures, which are flow charts of the steps of the image processing unit of the present invention as a mouse cursor image detection comparison, a schematic diagram of the cursor mask on the mouse cursor image in the operation screen, The flow chart of the steps of the cursor mask for optimization processing, the schematic diagram of the cursor mask through the threshold value to binarize the image and the area growth to mark the inside of the arrow cursor, the schematic diagram of the I-shaped cursor for pixel filling on the X axis, and image processing The unit reads the mouse cursor image through the sample comparison output comparison result and the feedback status judgment flowchart and block diagram. It can be clearly seen from the figure that when the image processing unit 202 performs the mouse cursor image detection in the operation screen When measuring the comparison method, the detection comparison method includes the following steps:

(S101) Acquire the cursor foreground image set by the user.

(S102) Acquire the cursor background image set by the user.

(S103) A cursor feature extraction algorithm is used to generate a cursor mask 300 (Select cursor contour extraction algorithm to obtain the cursor mask) from the foreground image and the background image.

(S104) The comparison unit reads the mouse cursor image, and applies the cursor mask 300 for sample comparison.

(S105) The image processing unit 202 provides the information of the comparison result and the feedback status of the mouse cursor.

It can be known from the above implementation steps that the image processing unit 202 in steps S101 and S102 first obtains the cursor foreground image and background image preset by the system, and the image size recommended by the system is 20x16, but the actual The size can be changed according to user needs, not limited to this, and the background image of the cursor is the part of the foreground image that does not contain the cursor; in step S103, the foreground image and the background image are transmitted through Related cursor feature extraction (such as point extraction, line extraction, edge extraction, contour linking and angle extraction, etc.) algorithm to generate a matching cursor mask 300, where the algorithm can be set according to the user's preset The complexity of the content of the foreground image and the background image can be expanded. The user can also make adjustments based on the extracted cursor mask 300 to facilitate the accuracy of the mouse cursor detection, and the cursor mask 300 includes the third The arrow and I-shaped cursor mask 300 divided by the upper left and lower half of the image in the figure, and the white blocks (such as the white pixels) in the arrow and I-shaped cursor mask 300 are: Care area 301, and the black block (such as the part composed of black pixels) is Don't care area 302.

In step S104, when the image processing unit 202 receives the original operation screen of the machine transmitted by the image capture device 201, the mouse cursor image in the operation screen can be read through the comparison unit, and the cursor mask 300 is applied as Sample comparison, as shown in the upper right and lower half images of the third image are the preview results of the left arrow and the I-shaped cursor mask 300 on the mouse cursor image, and The red blocks (such as the red pixels) in the cursor mask 300 are Don't care areas 302, and the other non-red block areas are Care areas 301.

As shown in the fourth figure, before performing the mouse cursor image comparison, the optimization processing for the cursor mask 300 first includes the following steps: (S201) use the Care area 301 in the cursor mask 300 as an image Histogram (Image histogram) operation, and for the calculation of the distance between each brightness value interval (Bin) on the horizontal axis.

(S202) Take a preset threshold value (Threshold value) from the maximum distance.

(S203) Perform image binary extraction through a preset threshold to extract the arrow cursor 303 and the I-shaped cursor 304 patterns, and then filter out the shadow 305.

(S204) The inside of the arrow area of the arrow cursor 303 is calibrated through the area growth.

(S205) The I-shaped cursor 304 performs pixel padding on the X axis.

As can be seen from the above implementation steps, the image processing unit 202 can use the Care area 301 in the cursor mask 300 as an image histogram operation, and draw each pixel of the brightness value (such as grayscale value or color value) in the image Number, for example, each pixel in the image can be represented by a brightness value in the range of 0 (black) to 255 (white), where the horizontal axis 0~255 represents different grayscale values, and the vertical axis represents the pixels of this grayscale value The proportion of the image, you can use the image histogram to understand how to adjust the brightness distribution, and calculate the distance between the brightness value intervals on the horizontal axis, and then choose a preset from the maximum distance Threshold value, you can use the preset threshold value to perform image binarization (Binarization), that is, the grayscale value of the pixel is set to 0 or 255, so that the entire image shows a clear black and white effect, usually gray The pixels whose order value exceeds the preset threshold are regarded as the background image, otherwise, they are the foreground image and are removed The background image is extracted from the arrow cursor 303 and the I-shaped cursor 304, and the shadow 305 of the cursor can be filtered out, and then the interior of the arrow area of the arrow cursor 303 is calibrated through the area growth using the diffusion method, I The glyph cursor 304 performs pixel padding in the X-axis direction (as shown in Figures 5A and 5B).

As shown in the sixth and seventh figures, the image processing unit 202 can read the mouse cursor image in the original operation screen of the machine through the comparison unit. For example, the image size can be Pix(20, W), that is The image is 20 pixels long in the horizontal axis direction and W pixels long in the vertical axis direction, and the cursor mask 300 is placed on the mouse cursor image, and then the image boundary and the image binarization are matched by image processing and related algorithms After performing mouse cursor image segmentation and line scan to scan the entire mouse cursor image line by line, you can perform pattern comparison by comparing the matrix of divided patterns, and at the same time calculate the score interval shown in Table 1 below (If the comparison matrix 20x16 is 1~320), and according to the threshold value of the score, the determination score of whether the mouse cursor image is stable can be determined through experiments. From Table 1, the score threshold of the arrow cursor image stability can be seen The value falls between 302 and 314, if it is an I-shaped cursor image, it falls between 314 and 316, and the image processing unit 202 can be used for mouse cursor image pattern comparison, and returns the comparison result information At the same time as the software control system 203 analyzes, it can also provide information such as the position, number, and image stability of the mouse cursor in the operation screen to confirm whether the movement of the mouse cursor conforms to the position controlled by the feedback and correction system.

In addition, when the comparison unit of the image processing unit 202 reads the mouse cursor image, the mouse cursor image can be encoded into a 20x16 matrix image in the same format as the cursor mask 300 through the encoder, and the cursor is stored in the line buffer After the image data of the mask 300, the image data of the cursor mask 300 is sequentially written into the comparison unit at different rates, thereby providing The comparison unit may apply the cursor mask image to the cursor mask 300 through the line scan comparison method for sample comparison, and calculate the above-mentioned mouse cursor score.

Please refer to Figures 8A to 8D, which are the flowchart (a), the flowchart (b), and the flowchart (c) of the step of judging the feedback state of the mouse cursor of the image processing unit of the present invention. Step flow chart (4), when the image processing unit 202 performs a method for determining the feedback state of the mouse cursor, the determination method includes the following steps: (S301) The system starts to execute the movement of the mouse cursor.

(S302) Wait for the image processing unit 202 to report that the state of the mouse cursor has stabilized or the waiting time has been exceeded, and then continue to execute step S303.

(S303) Determine whether a mouse cursor is found? If yes, go to step S304; if no, go to step S310.

(S304) Determine whether the mouse cursor has multiple positions? If yes, then hold Go to step S305; if not, go to step S306.

(S305) Provide an alarm message for multiple positions of the mouse cursor, and then continue to perform step S306.

(S306) Determine whether the position of the mouse cursor is correct? If yes, go to step S307; if no, go to step S315.

(S307) Determine whether there are multiple mouse cursors? If yes, go to step S308; if no, go to step S309.

(S308) Provide alarm information of multiple mouse cursors, and then continue to perform step S309.

(S309) It is determined that the mouse cursor moves successfully.

(S310) Is it the first execution? If yes, go to step S311; if no, go to step S312.

(S311) Perform the mouse cursor compensation operation, and then repeat step S302.

(S312) Acquire the position information of the last mouse cursor movement, and then proceed to step S313.

(S313) Is it the second or third execution? If yes, go to step S314; if no, go to step S316.

(S314) The mouse cursor returns to the starting position and then moves to the target position, and then repeats step S302.

(S315) Determine whether the number of mouse cursor retries has reached or exceeded the waiting time? If yes, repeat step S307; if no, repeat step S3 11.

(S316) Wait for the image processing unit 202 to report that the state of the mouse cursor has stabilized or the waiting time has been exceeded, and then continue to execute step S317.

(S317) The mouse cursor moves to a safe area, and then step S318 is continued.

(S318) Wait for the image processing unit 202 to report that the mouse cursor state has stabilized or the waiting time has been exceeded, and then continue to execute step S319.

(S319) Acquire the position information of the last mouse cursor movement, and then proceed to step S320.

(S320) Determine whether a mouse cursor is found? If not, go to step S321; if yes, go to step S322.

(S321) Provide an error message that the mouse cursor was not found.

(S322) Determine whether there is a difference between the positions of the second movement on the mouse cursor? If yes, go to step S323; if no, go to step S324.

(S323) Provide an error message that the mouse cursor cannot be controlled.

(S324) Provide an error message that the position of the mouse cursor is lost.

It can be seen from the above implementation steps that when the operator controls the keyboard/mouse to which the machine belongs, the system will start to move the mouse cursor, and the image processing unit 202 detects the operation screen output by the machine controller 100 After the mouse cursor image, the information of the mouse cursor image comparison result (including position, number, stability and score, etc.) is returned to the software control system 203 for analysis to complete the closed-loop control of the mouse cursor. When the software control system 203 executes the mouse cursor movement process, it first waits for the image processing unit 202 to report the mouse cursor The state has stabilized or the waiting time has been exceeded, and then determine whether a mouse cursor is found. If a mouse cursor is found, enter a multiple position judgment; if the multiple position is established, the image processing unit 202 provides an alarm for multiple positions of the mouse cursor Message (Warning multi-position) to the software control system 203, and then enter the judgment of the correctness of the mouse cursor position, if the position is correct, enter the judgment of multiple mouse cursors; if there are multiple mouse cursors established, the image processing The unit 202 provides warning messages (Warning multi-cursor) of multiple mouse cursors to the software control system 203, and determines that the mouse cursor moves successfully to complete the movement of the mouse cursor.

On the contrary, in the above step of judging whether the mouse cursor is found, if it is not found, it enters the judgment of whether it is the first execution; if it is the first execution, the feedback and correction system performs the mouse cursor compensation action , And then return to the step of waiting for the image processing unit 202 to report that the state of the mouse cursor has stabilized or has exceeded the waiting time; if it is judged that it is not the first time to execute, the position of the mouse cursor moved to the system last time is obtained Information, and then determine whether it is the second or third execution of the mouse cursor movement process, if it is the second or third execution, then return the mouse cursor to the starting position (Go to zero position), and then move Go to the target position, and then return to the step of waiting for the image processing unit 202 to report that the mouse cursor state has stabilized or the waiting time has been exceeded.

In addition, in the above step of determining whether the position of the mouse cursor is correct, if the position is incorrect, it is determined whether the number of retries has been reached or the waiting time has been exceeded; if the number of repetitions has been reached or the waiting time has been exceeded, return to the above In the step of judging multiple mouse cursors, otherwise, after performing the mouse cursor compensation operation, it returns to the aforementioned waiting image processing unit 2 02 In the step of reporting that the mouse cursor has stabilized or the waiting time has been exceeded.

Furthermore, in the step of determining whether it is the second or third execution, if it is determined that it is not the second or third execution, the subsequent error analysis of the mouse cursor will be performed. When the software control system 203 executes During the error analysis process, first wait for the image processing unit 202 to report that the mouse cursor status is stable or has exceeded the waiting time, and then move the mouse cursor to a safe area, and then perform another wait for the image processing unit 202 to report that the mouse cursor status is stable or The waiting time has been exceeded, and the position information of the last mouse cursor movement performed is obtained, and then it is determined whether the mouse cursor is found. If the mouse cursor is not found, the image processing unit 202 provides that the mouse cursor is not found (Cannot detect cursor) ) Error message to the software control system 203; if a mouse cursor is found, it is judged whether there is a difference between the positions of the second movement on the mouse cursor, and if there is a difference, the image processing unit 202 provides that the mouse cannot be controlled Error message of the cursor (Cannot control cursor); if there is no difference, the image processing unit 202 provides an error message of the lost cursor position (Lost cursor position), thereby providing subsequent detection feedback and correction system for corresponding processing procedures .

Therefore, the mouse cursor image detection comparison and feedback status judgment method of the present invention is based on the architecture of the non-invasive data acquisition system 200 and can be read by the image processing unit 202 without interfering with any production machine The machine controller 100 actually controls the mouse cursor image in the operation screen output by the keyboard/mouse 102, and applies a cursor mask 300 for sample comparison, and provides single or multiple positions after matching operation , Number and stability information to design the mouse cursor position feedback status and correction process, where the detection and comparison process includes the user-set cursor foreground and background image methods (such as foreground and background image acquisition, cursor Mask 300 generates concepts and adjusts the cursor pattern, etc.), and uses the cursor mask 300 as a pattern comparison method, including the design of Care area 301 and Don't care area 302, through the image histogram on the horizontal axis The interval of each brightness value interval on the above is the concept of obtaining a preset threshold, the design process of the comparison unit and the comparison method of line scanning, and the judgment process of the feedback state mechanism of the mouse cursor is provided by the image processing unit 202 Compare the results to move the mouse cursor, including detecting the stability of the mouse cursor in the entire design process, multiple positions of the mouse cursor, judgment of multiple mouse cursors, analysis of past mouse cursor movements, mouse cursor return to the starting position Then move to the target position recovery method, safe area usage method, mouse cursor position compensation, error analysis or other related mouse cursor status, etc.

Compared with the prior art, the aforementioned mouse cursor image detection comparison and feedback state judgment method of the present invention has the following beneficial effects:

(1) In the present invention, the feedback result of the mouse cursor image detection and comparison in the machine operation screen returned by the image processing unit 202 can provide a subsequent correction system to perform the corresponding mouse cursor processing procedure.

(2) In the present invention, if the image processing unit 202 executes the mouse cursor image detection and comparison process, it is processed by a field programmable logic gate array (FPGA), and does not need to pass through the central processing unit (CPU) in the host computer of the target machine system In addition to reducing the load on the CPU, it also ensures the real-time image processing.

(3) In the present invention, the image processing unit 202 detects the mouse cursor image in the operation screen of the machine, which can replace the function of the original human eye to confirm whether the movement of the mouse cursor conforms to the position controlled by the feedback and correction system to complete Closed loop control of mouse cursor.

(4) The present invention detects and compares through the image processing unit 202 As a result, the coordinate record of the position of the mouse cursor of the feedback and correction system can provide a more accurate machine control program of the traditional non-invasive data acquisition system.

(5) The present invention does not need to obtain the setting of the mouse cursor movement speed of the target machine end system in advance, and can correctly obtain the current feedback status information of the current mouse cursor position, number and stability level through the image processing unit 202 in real time .

(6) The feedback status information of the mouse cursor provided by the image processing unit 202 of the present invention can be used to record the user's operation flow and behavior, or to subsequently capture a large amount of user interface information for the collection and analysis of intermediary applications.

The above detailed description is for the description of a preferred feasible embodiment of the present invention, but this embodiment is not intended to limit the scope of the patent application of the present invention. Any other changes and modifications that have been completed without departing from the spirit of the art disclosed by the present invention Changes should be included in the scope of patents covered by the present invention.

In summary, the above mouse cursor image detection comparison and feedback status judgment method of the present invention can indeed achieve its efficacy and purpose when used, so the present invention is an invention with excellent practicability, which is in line with the invention patent You must file an application in accordance with the law. I hope that the review committee will grant the case as soon as possible to protect the inventor’s hard invention. If the Bureau has any doubts, please send me a letter and instruct the inventor to try their best to cooperate.

Claims (12)

A mouse cursor image detection comparison and feedback state judgment method, the method is based on a non-invasive data acquisition system architecture, using an image processing unit to read the mouse cursor image in the operation screen output by the machine controller For detection and comparison, the method includes the following steps: (A) Obtain the cursor foreground image set by the user; (B) Obtain the cursor background image set by the user; (C) Select the foreground image and the The background image generates a cursor mask; (D) read the mouse cursor image and apply the cursor mask for pattern comparison; (E) provide the information of the comparison result and the feedback status of the mouse cursor. The mouse cursor image detection comparison and feedback status judgment method as described in item 1 of the patent scope, wherein the image processing unit is a field programmable logic gate array, a special application integrated circuit, a microprocessor or an application program. The mouse cursor image detection comparison and feedback status judgment method as described in item 1 of the patent application scope, wherein the foreground image and the background image are obtained by the image processing unit from the image processing unit, and the system provides the cursor foreground image preset by the user Image with background image. The mouse cursor image detection comparison and feedback state determination method as described in item 1 of the patent scope, wherein the foreground image and the background image are generated by the cursor feature extraction algorithm to generate the compared cursor mask , Including arrow and I-shaped cursor mask. The mouse cursor image detection comparison and feedback state judgment method as described in item 1 of the patent scope, wherein the image processing unit reads the mouse cursor image through a comparison unit and applies the cursor mask to The mouse cursor image is divided into a caring area and a caring area, and then the pattern is compared. The mouse cursor image detection comparison and feedback state judgment method as described in item 5 of the patent application scope, wherein before the mouse cursor image comparison in step (D), the cursor mask is first optimized for the processing The method of optimization processing includes the following steps: (D1) Calculate the image histogram of the area of interest in the cursor mask, and calculate the distance between each brightness value interval on the horizontal axis; (D2) Take a preset threshold from the maximum distance; (D3) Binary image extraction through the preset threshold to extract the arrow cursor and I-shaped cursor patterns, and then filter out the shadow; (D4) The arrow area inside the arrow cursor is calibrated through the area growth; (D5) fill the pixel of the I-shaped cursor on the X axis. As described in item 6 of the patent application scope, the mouse cursor image detection comparison and feedback state judgment method, wherein the horizontal axis of the image histogram represents the different brightness values of the image, and the vertical axis represents the brightness value. The proportion of pixels in the image. The mouse cursor image detection comparison and feedback state judgment method as described in item 7 of the patent scope, wherein the image binarization process uses the pixel whose brightness value exceeds the preset threshold as the background image , Otherwise it is the foreground image, and the arrow cursor and I-shaped cursor pattern are extracted after removing the background image. The mouse cursor image detection comparison and feedback status judgment method as described in item 5 of the patent application scope, wherein the comparison unit performs color boundary and image binarization for the area of interest in the cursor mask The mouse cursor image is segmented, and the entire mouse cursor image is scanned line-by-line by line scanning, and then the score threshold for judging whether it is a stable mouse cursor image is calculated from the matrix of the comparison segmentation pattern. The mouse cursor image detection comparison and feedback state judgment method as described in item 1 of the patent scope, wherein the information of the comparison result in step (E) includes the position and number of the mouse cursor in the operation screen The stability of the image is used to confirm whether the movement of the mouse cursor conforms to the position controlled by the feedback and correction system. The mouse cursor image detection comparison and feedback state determination method as described in item 1 of the patent scope, wherein the image processing unit in step (E) performs the determination method of the mouse cursor feedback state, the determination method It includes the following steps: (E01) The system starts to execute the movement of the mouse cursor; (E02) waits for the image processing unit to report that the state of the mouse cursor has stabilized or has exceeded the waiting time; (E03) determines whether the mouse cursor is found , If yes, go to step (E04), if no, go to step (E10); (E04) judge whether the mouse cursor has multiple positions, if yes, go to step (E05), if no, go to step (E06); (E05) Provide the warning message of multiple positions of the mouse cursor, and then continue to perform step (E06); (E06) determine whether the position of the mouse cursor is correct, if yes, then execute step (E07), if not , Execute step (E15); (E07) determine whether there are multiple mouse cursors, if yes, execute step (E08), if not, execute step (E09); (E08) provide multiple mouse cursors Alarm message, and then continue to execute step (E09); (E09) Determine that the mouse cursor moves successfully; (E10) Determine whether it is the first execution. If yes, perform step (E11); if not, execute step (E12); (E11) Perform the mouse cursor For the compensation action, repeat step (E02); (E12) obtain the position information of the last mouse cursor movement, and then continue with step (E13); (E13) determine whether it is the second or third execution, If yes, go to step (E14), if not, go to step (E16); (E14) The mouse cursor returns to the starting position and then moves to the target position, and then repeat step (E02); (E15) to judge the slide Whether the number of mouse cursor retries has reached or exceeded the waiting time, if yes, repeat step (E07), if not, repeat step (E11); (E16) wait for the image processing unit to report that the mouse cursor status has been If it is stable or the waiting time has been exceeded, proceed to step (E17); (E17) The mouse cursor moves to a safe area, and then continue to step (E18); (E18) Wait for the image processing unit to report that the status of the mouse cursor has been If it is stable or the waiting time has been exceeded, proceed to step (E19); (E19) Obtain the position information of the last mouse cursor movement, and then continue to step (E20); (E20) Determine whether the mouse cursor is found, if not, go to step (E21), if yes, go to step (E22); (E21) provide an error message that the mouse cursor is not found; (E22) judge Is there any difference between the position of the second movement on the mouse cursor? If yes, then perform step (E23); if not, perform step (E24); (E23) provide an error message that the mouse cursor cannot be controlled; ( E24) Provide an error message that the position of the mouse cursor is lost. The mouse cursor image detection comparison and feedback state judgment method as described in item 1 of the patent scope, wherein the image processing unit in step (E) provides the mouse cursor feedback state to the software control system, In order to carry out subsequent detection feedback and correction system to carry out the corresponding processing procedures.

Images