TWI779836B - Keyboard file verification method based on image processing - Google Patents

Abstract

A keyboard file verification method based on image processing includes controling a processor to perform following operations: obtaining a keyboard file; generating a search index and a structural image according to the keyboard file; obtaining a template image from a template database according to the search index; performing a calibration operation to generate a plurality of candidate images, wherein the calibration operation includes adjusting the resolution of the structural image according to the resolution of the template image and executing a shifting operation according to the structural image to generate a plurality of candidate images; and comparing a key block within each of the candidate images with a key block within the template image, to generate a difference map and a comparison result.

Description

Keyboard File Verification Method Based on Image Processing

The present invention relates to image processing, in particular to a keyboard file verification method based on image processing.

The keyboard is one of the key components of the notebook computer. Before assembling the notebook computer product, it is necessary to ensure that the keyboard design in the specification and the physical keyboard sample are verified. Whether it's a specification design or a physical component, an anomalous or flawed design will add unnecessary cost to the production process.

Traditionally, verification of vendor-provided keyboard profiles has relied on manual labor. Quality control personnel need to visually inspect the discrepancies between the keyboard files provided by the suppliers and the reference keyboard designs in the database. However, there are a large number of keyboard samples to be inspected at any time on the production line. When the quality management personnel conduct visual inspection for a long time, it is easy to reduce the quality of the verification due to negligence.

In view of this, the present invention proposes a keyboard file verification method based on image processing. By combining multiple image processing technologies, the inconvenience of quality control personnel visually inspecting keyboard defects in the prior art can be improved, and the accuracy of defect inspection can be improved at the same time. Make sure the input keyboard design in the keyboard archive matches the reference design.

According to an embodiment of the present invention, a keyboard file verification method based on image processing includes controlling a processor to perform the following operations: obtain a keyboard file; generate a search index and a feature image according to the keyboard file; The template database obtains a template image; performs a correction operation according to the feature image, wherein the correction operation includes: adjusting the resolution of the feature image according to the resolution of the template image; and performing a displacement operation according to the feature image to generate multiple candidate images; and comparing the button blocks in each of the candidate images with the button blocks in the template image to generate a difference map and a comparison result.

In summary, the present invention proposes a keyboard file verification method based on image processing, which converts the original document of the keyboard design into a structured visual feature, and imitates the process of human visual perception, and combines the original document of the keyboard file with the template database Compare the template images in , and finally mark the perceived difference areas.

The above description of the disclosure and the following description of the implementation are used to demonstrate and explain the spirit and principle of the present invention, and provide a further explanation of the patent application scope of the present invention.

The detailed features and characteristics of the present invention are described in detail below in the implementation mode, and its content is enough to enable any person familiar with the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of the patent application and the drawings , anyone who is familiar with the related art can easily understand the ideas and features related to the present invention. The following examples are to further describe the concept of the present invention in detail, but not to limit the scope of the present invention in any way.

Fig. 1 is a flow chart of a keyboard file verification method based on image processing according to an embodiment of the present invention, the method is applicable to a personal computer or a network server with a processor, and each step in Fig. 1 is mainly executed by the processor .

Step S1 is "obtaining the keyboard file". In one embodiment, the keyboard file obtained by the processor from the storage device is a single-page portable document file (PDF). Figure 2 is an example of a keyboard file A schematic diagram of the content. The keyboard file includes the keyboard image and keyboard information. The keyboard image is, for example, a grayscale image obtained by shooting a keyboard sample with a camera, or a digital image file drawn by a keyboard designer with software. Keyboard information includes: project name, brand, supplier, country code, keyboard size, keyboard type (for example: large/medium/small keyboard, whether it is a backlit keyboard, whether it has a pointing stick, etc.), at least one of the manufacturing years By. The above keyboard information can be recorded in at least one of the content of the keyboard file, the file name of the keyboard file, and the storage path of the keyboard file, and the present invention does not limit the location of the keyboard information recording.

Step S2 is "generating search index and feature image according to the keyboard file", please refer to FIG. 3 , which is a detailed flow chart of step S2 in FIG. 1 .

Step S21 is "extracting the text part and the image part from the keyboard file". In one embodiment, the processor executes the Pymupdf software to extract the text part and the image part from the PDF file, wherein the text part includes the above-mentioned keyboard information, image Some keyboard images are in PNG file format.

Step S22 is "generating a search index according to the text part", in detail, the processor executes the program to set the content associated with the keyboard information in the text part as the search index.

Step S22 is "executing a plurality of image processing programs according to the keyboard image to generate a feature image", these image processing programs executed by the processor are used to extract the outline of the keyboard and locate the keyboard area. The keyboard outline includes the outer outline of the keyboard body and the grid lines of all keys.

In order to extract the outline of the keyboard, the processor converts the keyboard image into a gray-scale image, and then uses the Otsu algorithm (OTSU, or automatic threshold segmentation method) to degenerate the gray-scale image into a binary image. Figure 4 is an example of the corresponding Schematic diagram of a binarized image of a keyboard file. For the binarized image, the processor performs 8-connectivity in connected-component labeling (CCL) to find out the region belonging to the keyboard in the binarized image. The processor uses the function FindContour in OpenCV to obtain a rough outline of the keyboard.

In order to locate the keyboard area, the processor performs a filling operation according to the outline of the keyboard, and fills in the pixels inside the closed outline, thereby filtering out the non-closed outline area. The processor screens the area of the connected area, the degree of filling, and the size of the keyboard obtained from the keyboard information (which may include the aspect ratio or area of the keyboard), confirms whether the connected area surrounded by the outline of the keyboard belongs to the keyboard, and adjusts the two Threshold in value operations.

In addition, according to the binarized image after the filling operation, the processor further executes the open operation in the morphological processing, thereby filtering out the guiding lines used for indication in the original keyboard image. FIG. 5 is an example of a keyboard image The schematic diagram of the guide line, the guide line is the line on the right and below the keyboard in Figure 5, which is used to mark the length and width of the keyboard, but it is unnecessary information in the verification of the keyboard file.

Finally, the processor searches for adjacent objects, iteratively merges the padding key areas that are too close to each other, and finally merges the area belonging to the keyboard in the binarized image, and outputs the padding grid map. In other words, in the step of finding adjacent objects, there will be multiple unclustered areas, and the processor will combine these scattered small areas, and select the largest area among them as the keyboard area. FIG. 6 is a schematic diagram of a part of a filled grid map in an example.

The processor calculates the number of keys in the filled grid map, thereby confirming which type of keyboard is the large, medium and small keyboard, and then sets the confirmed keyboard type as the search index. The keyboard types exemplified above can also be set in step S22, which is not limited in the present invention.

The processor executes the Distance transform function in OpenCV according to the filled grid map, and calculates the distance between all the pixels on the inner edge of each key outline and the inner edge of the key outline to generate a distance map. The processor performs a binarization operation on the calculated multiple distance values, thereby separating the outline and the content. The processor performs a retraction operation on the filled grid map according to the distance map, detects whether a pixel belonging to the text of the button is encountered during the retraction process, and generates a bounding box (bounding box) surrounding the text of the button, and finally processes The device generates a context image according to the content of the bounding box. Please refer to Figure 7, which shows two sets of feature images generated by the two buttons in the above process. Each set of feature images includes: binarized image, filled grid map, distance map, and binarized distance after shrinking operation diagrams and text diagrams.

Please refer to FIG. 1 , step S3 is "acquire the template image from the template database according to the search index", and step S4 is "perform correction operation according to the characteristic image".

In step S3, the processor uses keyboard information such as brand, supplier, and country code as a search index to search in the template database, and can find one or more template images as a standard for subsequent comparisons. If two If there are more than one template image, each template image will be used for comparison. In step S4, the calibration operation includes resolution calibration and displacement calibration. Resolution correction is to adjust the resolution of the feature image to be consistent with the resolution of the template image obtained in step S4. In addition, each template image in the template database is also created in advance through the processes of steps S1 , S2 and S4 , so the resolutions of all template images have the same resolution.

After performing the resolution correction, the displacement operation is to use the enhanced correlation coefficient (Enhanced Correlation Coefficient, ECC) as a measure of similarity to predict the parameters of the motion model, and use the findTransformECC function in OpenCV to iteratively generate the motion model of the feature image, such as Homography matrix (Homography matrix) or affine transformation matrix (affine matrix), and set the tolerance error of this iterative method. When it is less than a certain level, it means that the image can achieve the function of aligning the image through the motion model, thereby achieving the effect of displacement. The processor can generate a plurality of candidate images according to the feature image and the motion model.

Please refer to FIG. 1 , step S5 is "execute block comparison procedure based on the candidate image and the template image". The block comparison program is based on the key comparison (patch-wise) for structural similarity (Structural Similarity Index Measure, SSIM) comparison, and performs wavelet hashing (wavelet hashing) algorithm before SSIM to make up for the SSIM The disadvantage is that for some images with simple structure, when SSIM compares to a completely black image, it may happen that the score generated by SSIM cannot reflect the difference. Therefore, the processor executes the wavelet hash algorithm to convert the candidate image into the frequency domain and generates a hash value, and then calculates the Hamming distance between the candidate image and the template image to reflect the degree of difference between the original keyboard image and the template image , if the difference is too large, the error map of the area will be highlighted and marked as an abnormal area for subsequent human re-judgment. If the difference is within the tolerance range, the image similarity inspection will continue based on SSIM.

Since the keyboard samples manufactured by different suppliers may have slight errors in the distance between the keys, in order to improve the fault tolerance and stability of the comparison, in the process of image similarity inspection based on SSIM, the present invention additionally introduces a displacement operation , move the block-scale feature image according to a specified direction (upward, downward, leftward, and rightward) and a specified length (eg, less than or equal to 5 pixels in length) to generate multiple displaced images. The processor performs a displacement operation based on the block scale to generate multiple displacement images, and then uses SSIM to calculate the similarity between each displacement image and the template image, and selects the displacement image corresponding to the one with the smallest error value for the subsequent process. In other words, the processor performs a small shift on the button portion of the candidate image to find a shifted image with the highest degree of alignment with the button portion of the template image. Through the above method, the present invention can reduce the area error caused by the difference in resolution.

Please refer to FIG. 1 , step S6 is "output difference map and comparison result". In detail, after step S5 is completed, the processor generates a candidate error map for each candidate image compared with the template image. The candidate error map is a grayscale image, and the pixel values in the image can reflect the degree of difference. The processor calculates the sum of differences in all keys according to each candidate error map, and the candidate image corresponding to the candidate error map with the smallest difference sum represents the best alignment with the template image. The processor further judges whether the sum of the differences is greater than a threshold value. If the judgment result is "Yes", it means that the difference between the keyboard image and the template image is too large, and it must be further confirmed by the quality control personnel. 8 is a schematic diagram of a template image, a keyboard image obtained from an input keyboard file, and an error map, wherein a plurality of white blocks in the error map represent differences between the keyboard image and the template image. In addition, the comparison result output by the processor includes the following files: description file, keyboard image, filled grid map, text map, template image most similar to the text map in the template database, error map, and mark. In one embodiment, the file format of the description file is Comma-Separated Values (CSV), and its content records the following information: brand, project name, supplier, file name of the keyboard file, and storage path of the keyboard file , country code, keyboard type, storage location of the template image, whether the comparison is successful, the number of different keys, and the reason for the difference, etc.

In summary, the present invention proposes a keyboard file verification method based on image processing, which can convert the original document of the keyboard design into structured visual features, and imitate the process of human visual perception, and combine the original document of the keyboard file with the template data The template image in the library is compared, and finally the perceived difference area is marked (for example, by circle selection).

Although the present invention is disclosed by the aforementioned embodiments, they are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, all changes and modifications are within the scope of patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the appended scope of patent application.

S1~S6, S21~S23: steps

Fig. 1 is the flow chart of the keyboard file verification method based on image processing of an embodiment of the present invention; Fig. 2 is a schematic diagram of the contents of a keyboard file in an example; Fig. 3 is the detailed flowchart of step among Fig. 1; 4 is a schematic diagram of a binarized image corresponding to a keyboard file in an example; FIG. 5 is a schematic diagram of guiding lines in an example keyboard image; Fig. 6 is a schematic diagram of a part of the filled grid map in an example; FIG. 7 is a schematic diagram of two sets of feature images in an example; and FIG. 8 is a schematic diagram of a template image, a keyboard image and an error map.

S1~S6: steps

Claims (7)

A keyboard file verification method based on image processing, including controlling a processor to perform the following operations: obtain a keyboard file; generate a search index and a feature image according to the keyboard file; obtain a template image from a template database according to the search index ; performing a correction operation according to the feature image, wherein the correction operation includes: adjusting the resolution of the feature image according to the resolution of the template image; and performing a displacement operation according to the feature image to generate a plurality of candidate images; comparing the button blocks in the candidate images with the button blocks in the template image to generate a difference map and a comparison result; wherein performing the displacement operation according to the feature image to generate the candidate images includes : generating a motion model according to an enhanced correlation coefficient, wherein the motion model is a homography matrix or an affine transformation matrix; and generating the candidate images according to the feature image and the motion model. The keyboard file verification method based on image processing as described in claim 1, wherein the keyboard file is in a portable file format, the keyboard file includes a keyboard information and a keyboard image, and the keyboard information is associated with the content and file of the keyboard file at least one of name and storage path. The keyboard file verification method based on image processing as described in claim 2, wherein generating the characteristic image according to the keyboard file includes: converting the keyboard image into a grayscale image; performing Otsu algorithm based on the grayscale image to generate a binary value image; performing a connected component labeling method according to the binarized image to locate a plurality of connected regions in the binarized image; obtaining a contour according to each of the connected regions and performing a filling operation according to the contour; and performing a filling operation according to each of the connected regions; The area of the connected region, a filling degree and a keyboard size in the keyboard information are screened to confirm whether each connected region belongs to a keyboard region, and a filled grid map is output. The keyboard file verification method based on image processing as described in claim 3, wherein generating the characteristic image according to the keyboard file includes: performing a distance conversion function according to the filling grid map to generate a distance map; according to the distance map to the filling The grid map performs a shrinking operation and generates a certain bounding box, wherein the bounding box is used to enclose a button text; and generates a text map according to the bounding box; wherein, the feature image includes the binarized image , the filled grid map, the distance map, the bounding box and the text map. The keyboard file verification method based on image processing as described in claim 4, further includes: calculating the number of a key in the filled grid map; and confirming a keyboard type according to the number of keys, and setting the keyboard type as the search index. The keyboard file verification method based on image processing as described in claim 1, wherein the key blocks in each of the candidate images are compared with the key blocks in the template image to generate the difference map and the comparison The results include: performing a wavelet hash algorithm to convert each of the candidate images into a frequency domain; calculating the structural similarity and Hamming distance of each of the candidate images and the template image to generate a plurality of candidate difference maps; and calculating a sum of difference quantities in each of the candidate difference maps, wherein the difference map is the One of the candidate difference maps with the smallest sum of difference amounts. The keyboard file verification method based on image processing as described in claim 6, wherein calculating the structural similarity between each of the candidate images and the template image includes: moving the feature image according to each of the candidate images, a specified direction and a specified length to generate multiple displacement images.

Images