WO2024013901A1

WO2024013901A1 - Match rate calculation device, match rate calculation method, and match rate calculation program

Info

Publication number: WO2024013901A1
Application number: PCT/JP2022/027611
Authority: WO
Inventors: 史拓横瀬; 公雄土川; 泰輔若杉; 諒内田; 晴夫大石
Original assignee: 日本電信電話株式会社
Priority date: 2022-07-13
Filing date: 2022-07-13
Publication date: 2024-01-18

Abstract

In the present invention, a background detection unit (15b) detects a background region in a template image for image matching. A calculation unit (15d) excludes the detected background region and calculates the match rate between the template image and a retrieved image.

Description

Match rate calculation device, match rate calculation method, and match rate calculation program

The present invention relates to a match rate calculation device, a match rate calculation method, and a match rate calculation program.

In PC operation automation such as RPA (Robotic Process Automation) and PC operation analysis, image matching is used to search for areas that match a template image on the captured image (search image) on the PC screen. (See Non-Patent Document 1). In image matching, in addition to the process of finding parts of the searched image that completely match the template image, the process of finding similar parts of the searched image whose match rate with the template image is greater than a predetermined threshold is used. be exposed.

However, with the conventional technology, it may be difficult to set a match rate threshold in image matching. For example, the match rate in image matching changes if the size of the area corresponding to the margin changes depending on how the template image is cut out. In addition, controls on the PC screen (operated elements) may have visual effects such as changing color when the mouse cursor is placed over the control (mouse over), and when non-search images change due to visual effects. The match rate will change. However, it is difficult for humans to notice changes in non-search images due to the size of the blank area of the template image or visual effects, so the resulting fluctuations in the match rate cause a discrepancy with human intuitive sense, leading to the determination that images are similar. It makes people feel that setting the threshold for the purpose is difficult.

The present invention has been made in view of the above, and an object of the present invention is to enable humans to easily set a match rate threshold by adjusting the match rate calculation in image matching to human intuitive sense. shall be.

In order to solve the above-mentioned problems and achieve the purpose, a match rate calculation device according to the present invention includes a background detection unit that detects a background area of a template image for image matching, and a background detection unit that excludes the detected background area. The image processing apparatus is characterized in that it includes a calculation unit that calculates a match rate between the template image and the searched image.

According to the present invention, it becomes possible for humans to easily set the match rate threshold in image matching.

FIG. 1 is a diagram for explaining an overview of a match rate calculation device. FIG. 2 is a diagram for explaining the outline of the match rate calculation device. FIG. 3 is a diagram for explaining the outline of the match rate calculation device. FIG. 4 is a diagram for explaining the outline of the match rate calculation device. FIG. 5 is a schematic diagram illustrating a schematic configuration of the match rate calculation device of this embodiment. FIG. 6 is a diagram for explaining the match rate calculation process. FIG. 7 is a diagram for explaining the processing of the background detection section. FIG. 8 is a diagram for explaining the processing of the edge detection section. FIG. 9 is a diagram for explaining the processing of the calculation unit. FIG. 10 is a diagram for explaining the processing of the calculation unit. FIG. 11 is a diagram for explaining the processing of the calculation unit. FIG. 12 is a diagram for explaining the processing of the calculation unit. FIG. 13 is a flowchart showing the match rate calculation processing procedure. FIG. 14 is a diagram for explaining the effect of the match rate calculation process. FIG. 15 is a diagram illustrating an example of a computer that executes a match rate calculation program.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.

[Overview of match rate calculation device]
FIGS. 1 to 4 are diagrams for explaining the outline of the match rate calculation device. Image matching is a process of comparing a searched image and a template image to find a position in the searched image that has a high degree of similarity to the template image. As illustrated in the concept in Figure 1, image matching involves shifting (scanning) the position within the searched image and comparing parts of the searched image with the same size as the template image to calculate the match rate for each position. do. If this match rate is greater than or equal to a predetermined threshold, it is determined that the position of the searched image is similar to the template image.

Note that in this embodiment, the image to be compared is a grayscale image in which each pixel takes a value of 0 to 255. However, the comparison is not limited to this, and may be a comparison between color images. Further, the match rate is normalized to 0% to 100%, that is, 0.0 to 1.0, and is set to 100% when there is a complete match. However, the range of the match rate is not limited to this. Further, as illustrated in FIG. 2, the match rate is the normalized sum of the absolute values of each pixel difference. However, the value is not limited to this, and may be a value calculated by, for example, the square root of the sum of squares of each pixel difference, a cosine similarity of each pixel difference, or the like.

Here, the match rate varies greatly depending on how the template image is cropped from the captured image, including the margins. The margin is the area around a control (operated element such as a button, check box, text box, etc.) on the PC screen to be operated, and is usually the background on the PC screen. For example, in the example shown in FIG. 3, the template image shown in FIG. 3(a) is cut out to be larger than the template image shown in FIG. 3(b), so that the margin portion is larger. In this case, the proportion of the different parts (the presence or absence of "R") to the whole becomes relatively small, so the match rate is as high as 98.9%.

In addition, if the color of a control such as a button on a PC screen changes due to a mouse over, etc., when the mouse cursor is placed on an element, the match rate will change even if there is no major change in appearance to the human eye. There are cases where For example, the search target image "registration" illustrated in FIG. 4 usually has a match rate of 100% with the template image "registration" as illustrated in FIG. 4(a). On the other hand, as illustrated in FIG. 4(b), when the color of "Registered" in the searched image changes due to mouse over, the match rate with "Registered" in the template image decreases, and "Reset" with different characters appears. The match rate is lower.

Therefore, the match rate calculation device of this embodiment detects the background region of the template image and excludes it from contributing to the match rate. This makes it possible to exclude the blank space of the template image, which has little meaning when specifying elements on the PC screen, thereby reducing the influence on the match rate.

Additionally, edge regions, which are regions with large changes compared to the surrounding areas, are detected, and the weight of their contribution to the match rate is increased. As a result, the match rate calculation device increases the contribution of more characteristic regions of the graphic design such as edges of the control image, and changes caused by visual effects such as mouse over are small, but in relatively large regions of the control. It becomes possible to reduce the influence of changes that occur on the match rate.

In this way, the match rate calculation device brings the change in match rate closer to the human sense, and calculates the match rate without being affected by how the template image is cropped or the mouse over, thereby improving the match rate in image matching. It becomes possible to easily set a threshold value for rate determination.

[Configuration of match rate calculation device]
FIG. 5 is a schematic diagram illustrating a schematic configuration of the match rate calculation device of this embodiment. As illustrated in FIG. 5, the match rate calculation device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input section 11, an output section 12, a communication control section 13, a storage section 14, and a control section 15. Be prepared.

The input unit 11 is realized using an input device such as a keyboard or a mouse, and inputs various instruction information such as starting processing to the control unit 15 in response to an input operation by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, and the like. For example, the output unit 12 displays the results of match rate calculation processing, which will be described later.

The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between an external device and the control unit 15 via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device that manages various types of information used in match rate calculation processing.

The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. In the storage unit 14, a processing program for operating the match rate calculation device 10, data used during execution of the processing program, etc. are stored in advance, or are temporarily stored each time processing is performed. Note that the storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

The control unit 15 is realized using a CPU (Central Processing Unit) or the like, and executes a processing program stored in a memory. Thereby, the control unit 15 functions as an acquisition unit 15a, a background detection unit 15b, an edge detection unit 15c, and a calculation unit 15d, and executes the match rate calculation process, as illustrated in FIG. Note that each or a part of these functional units may be implemented in different hardware. For example, the acquisition unit 15a may be implemented in hardware different from other functional units. Further, the control unit 15 may include other functional units.

The acquisition unit 15a acquires a searched image and a template image to be subjected to image matching processing. For example, the acquisition unit 15a acquires the searched image and template image generated by screen capture via the input unit 11 or from a user terminal, management device, or the like via the communication control unit 13.

Note that the acquisition unit 15a may acquire the searched image and template image to be processed in advance and store them in the storage unit 14, or may transfer them immediately to a subsequent functional unit without storing them in the storage unit 14. You can.

The background detection unit 15b detects the background area of the template image for image matching. Specifically, the background detection unit 15b detects pixels having the same color as the four outer sides of the template image as a background area.

Here, FIG. 6 is a diagram for explaining the match rate calculation process. FIG. 6(a) is an image to be searched for the processing described below, and FIG. 6(b) is a template image. In this embodiment, all images are grayscale images. In addition, since the searched image and template image are the same size, 20 pixels wide by 15 pixels high, there is no need to calculate the match rate multiple times while scanning the searched image, and only calculate the match rate once. calculate.

Further, FIG. 7 is a diagram for explaining the processing of the background detection section. For example, as shown in FIG. 7A, if the color information of pixels on the outer four sides of the template image are all the same, the background detection unit 15b uses that color as the background color. Then, the background detection unit 15b detects pixels of the same color inside from the pixels on the four sides as a background area.

The edge detection unit 15c detects edge regions of the template image. Specifically, the edge detection unit 15c detects a pixel having the maximum color difference between the target pixel of the template image and surrounding pixels as an edge region.

Here, FIG. 8 is a diagram for explaining the processing of the edge detection section. For example, as shown in FIG. 8A, the edge detection unit 15c calculates the maximum value among the absolute values of the differences between the target pixel of interest and its surrounding pixels. FIG. 8A illustrates a case where the center pixel is the target pixel, the lower right pixel has a color difference of 150, and the other pixels have the same color with a difference of 0. Then, as shown in FIG. 8B, the maximum value of the difference between each pixel and surrounding pixels is calculated for each pixel as a target pixel, and pixels for which the calculated value is greater than or equal to a predetermined threshold are defined as edge regions. In the example shown in FIG. 8B, pixels with a difference value of 150 that are shaded with diagonal lines are detected as edge regions.

Next, FIGS. 9 to 12 are diagrams for explaining the processing of the calculation unit. First, the calculation unit 15d excludes the detected background area and calculates the matching rate between the template image and the searched image. Here, the calculation unit 15d calculates the match rate using the difference value for each pixel between the template image and the search target image.

For example, the calculation unit 15d compares the template image illustrated in FIG. 9(a) with a comparison target portion of the same size as the template image of the searched image, and calculates the pixel size as illustrated in FIG. 9(b). Calculate the difference value for each.

Then, the calculation unit 15d identifies the background region of the template image from among the difference values for each pixel, as illustrated in FIG. 10(a). Further, as illustrated in FIG. 10B, the calculation unit 15d identifies pixels in the background region of the template image whose difference value is equal to or less than a predetermined threshold value (for example, 0) as background pixels. In FIG. 10(b), diagonally shaded pixels from the upper right to the lower left are specified as background pixels. The calculation unit 15d excludes the specified background pixel from the comparison target pixels for calculating the match rate.

Furthermore, the calculation unit 15d calculates the match rate between the template image and the searched image by adding a predetermined weight to the edge region. For example, as illustrated in FIG. 11A, the calculation unit 15d identifies an edge region of the template image among the pixel-by-pixel difference values, and identifies pixels in the edge region as weight-increasing pixels.

At this time, as illustrated in FIG. 11(b), the calculation unit 15d excludes pixels identified as background pixels as illustrated in FIG. 10(b) from the weight-increasing pixels. In FIG. 11(b), diagonally shaded pixels from the upper left to the lower right are specified as weight increasing pixels. Note that instead of prioritizing background pixels, priority may be given to weight increasing pixels, and pixels identified as weight increasing pixels may be excluded from the background pixels.

Then, the calculation unit 15d calculates the match rate between the comparison target portion of the searched image and the template image, as illustrated in FIG. 12. For example, the calculation unit 15d calculates the match rate using the ratio of the weighted sum of the difference values to the maximum possible value. That is, the calculation unit 15d calculates 1-(weighted sum of difference values)÷(maximum possible value of weighted sum of difference values) as the match rate.

Specifically, as illustrated in FIG. 12, the calculation unit 15d first calculates the weighted sum of the difference values between both images. In the example shown in FIG. 12, a predetermined weight of 2.0 is added to the sum of the difference values for the 133 weight increase pixels illustrated in FIG. 11(b). Furthermore, the sum of the difference values of 26 pixels excluding the 144 background pixels illustrated in FIG. 10(b) from the total 300 pixels is calculated.

Furthermore, the calculation unit 15d calculates the maximum value that the weighted sum of the difference values between both images can take. In the example shown in FIG. 12, the maximum value 255 of the difference value of each pixel is calculated as 255×133×2.0 by adding the weight 2.0 of the 133 weight increase pixels. Furthermore, the maximum possible value of 26 pixels excluding background pixels is calculated as 255×26.

Then, the calculation unit 15d calculates {1-(weighted sum of difference values)÷(maximum possible value of weighted sum of difference values)} as a match rate. In the example shown in FIG. 12, the match rate is calculated to be approximately 0.611=61.1%.

[Match rate calculation process]
Next, with reference to FIG. 13, match rate calculation processing by the match rate calculation device 10 according to the present embodiment will be described. FIG. 13 is a flowchart showing the match rate calculation processing procedure. The flowchart in FIG. 13 is started, for example, at the timing when the user instructs to start the apparatus.

First, the acquisition unit 15a acquires a search target image and a template image for image matching processing. Further, the background detection unit 15b detects a background area of a template image for image matching (step S1). For example, the background detection unit 15b detects pixels having the same color as the four outer sides of the template image as a background area.

Furthermore, the edge detection unit 15c detects an edge region of the template image (step S2). For example, the edge detection unit 15c detects, as an edge region, a pixel that has the maximum color difference between the target pixel and surrounding pixels in the template image.

Then, the calculation unit 15d calculates the match rate between the template image and the searched image, excluding the detected background area. Further, the calculation unit 15d adds a predetermined weight to the edge region and calculates the matching rate between the template image and the searched image (step S3). For example, the calculation unit 15d uses the difference value for each pixel between the template image and the searched image, and calculates the match rate as 1 - (weighted sum of difference values) ÷ (maximum possible value of the weighted sum of difference values). calculate. This completes a series of match rate calculation processes.

[effect]
As described above, in the match rate calculation device 10 of this embodiment, the background detection unit 15b detects the background area of the template image for image matching. The calculation unit 15d calculates the matching rate between the template image and the searched image, excluding the detected background area.

Specifically, the calculation unit 15d calculates the match rate using the difference value for each pixel between the template image and the searched image. Furthermore, the background detection unit 15b detects pixels having the same color as the four outer sides of the template image as a background area.

As a result, the match rate calculation device 10 is able to calculate the match rate by excluding the blank space of the template image, which has little meaning when specifying elements on the PC screen, and reducing the influence on the match rate. Become.

Additionally, the edge detection unit 15c detects edge areas of the template image. In that case, the calculation unit 15d adds a predetermined weight to the edge region and calculates the matching rate between the template image and the search target image.

Specifically, the edge detection unit 15c detects, as an edge region, a pixel that has the maximum color difference between the target pixel of the template image and surrounding pixels.

As a result, the match rate calculation device 10 calculates the match rate by emphasizing common designs such as the edges of control buttons and reducing the influence on the match rate of changes in color within the edge area due to mouse over. It becomes possible to do so.

Here, FIG. 14 is a diagram for explaining the effect of the match rate calculation process. Conventionally, as illustrated in FIG. 14A, the match rate with the searched image changes depending on how the template image is cut out from the captured image and the color changes due to mouse over. In the example shown in FIG. 14(a), the match rate for the searched image "Reset" with respect to the template image "Register" varies between 95% and 98% depending on the size of the blank space of the template image. . Furthermore, even if the size of the blank space is the same, the match rate is calculated to be the same 95% between the "reset" searched image and the "registered" image whose color changes due to mouse over.

On the other hand, according to the match rate calculation device 10 of this embodiment, as illustrated in FIG. 14(b), the influence of the margin part of the template image is eliminated, and the influence of color changes due to mouse over etc. is suppressed. Then, it is possible to calculate the match rate. In the example shown in Fig. 14(b), the match rate for the template image "Register" is 70% for "Register", whose color changes when the mouse is over the image, than for "Reset", the searched image. It is close to the feeling of 99%, which is high.

In this way, the match rate calculation device 10 is able to calculate the match rate in image matching by bringing the match rate closer to the human intuitive feeling and suppressing the effects of cropping methods, mouse overs, etc. Therefore, it becomes possible for humans to more easily set the matching rate threshold for determining that the search target image and the template image are similar.

[program]
It is also possible to create a program in which the processing executed by the match rate calculation device 10 according to the embodiment described above is written in a computer-executable language. As one embodiment, the match rate calculation device 10 can be implemented by installing a match rate calculation program that executes the above-described match rate calculation process on a desired computer as packaged software or online software. For example, by causing the information processing device to execute the above match rate calculation program, the information processing device can be made to function as the match rate calculation device 10. The information processing device referred to here includes a desktop or notebook personal computer. In addition, information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHSs (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). Further, the functions of the match rate calculation device 10 may be implemented in a cloud server.

FIG. 15 is a diagram showing an example of a computer that executes a match rate calculation program. Computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. These parts are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1031. Disk drive interface 1040 is connected to disk drive 1041. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041, for example. For example, a mouse 1051 and a keyboard 1052 are connected to the serial port interface 1050. For example, a display 1061 is connected to the video adapter 1060.

Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiments is stored in, for example, the hard disk drive 1031 or the memory 1010.

Further, the match rate calculation program is stored in the hard disk drive 1031, for example, as a program module 1093 in which commands to be executed by the computer 1000 are written. Specifically, a program module 1093 in which each process executed by the match rate calculation device 10 described in the above embodiment is described is stored in the hard disk drive 1031.

Further, data used for information processing by the match rate calculation program is stored as program data 1094 in, for example, the hard disk drive 1031. Then, the CPU 1020 reads out the program module 1093 and program data 1094 stored in the hard disk drive 1031 to the RAM 1012 as necessary, and executes each of the above-described procedures.

Note that the program module 1093 and program data 1094 related to the match rate calculation program are not limited to being stored in the hard disk drive 1031; for example, they may be stored in a removable storage medium and executed by the CPU 1020 via the disk drive 1041 or the like. May be read. Alternatively, the program module 1093 and program data 1094 related to the match rate calculation program are stored in another computer connected via a network such as a LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070. May be served.

Although embodiments to which the invention made by the present inventor is applied have been described above, the present invention is not limited by the description and drawings that form part of the disclosure of the present invention by this embodiment. That is, all other embodiments, examples, operational techniques, etc. made by those skilled in the art based on this embodiment are included in the scope of the present invention.

10 Match rate calculation device 11 Input section 12 Output section 13 Communication control section 14 Storage section 15 Control section 15a Acquisition section 15b Background detection section 15c Edge detection section 15d Calculation section

Claims

a background detection unit that detects a background area of a template image for image matching;
a calculation unit that calculates a match rate between the template image and the searched image by excluding the detected background area;
A match rate calculation device comprising:
further comprising an edge detection unit that detects an edge region of the template image,
The calculation unit adds a predetermined weight to the edge region and calculates a match rate between the template image and the searched image.
2. The match rate calculation device according to claim 1.
The match rate calculation device according to claim 1, wherein the calculation unit calculates the match rate using a pixel-by-pixel difference value between the template image and the searched image.
The match rate calculation device according to claim 1, wherein the background detection unit detects pixels having the same color as the four outer sides of the template image as a background area.
3. The match rate calculation device according to claim 2, wherein the edge detection unit detects a pixel having a maximum color difference between the target pixel of the template image and surrounding pixels as the edge region.
A match rate calculation method executed by a match rate calculation device, comprising:
a background detection step of detecting a background area of a template image for image matching;
a calculation step of calculating a match rate between the template image and the searched image by excluding the detected background area;
A match rate calculation method characterized by including.
a background detection step of detecting a background area of a template image for image matching;
a calculation step of calculating a match rate between the template image and the searched image by excluding the detected background area;
A match rate calculation program that causes a computer to execute.