JP6160138B2 - Moire removal method for pattern recognition, moire removal apparatus and program using this method - Google Patents

Description

  The present invention relates to a technique for recognizing a pattern in an image by processing an image generated by photographing a pattern such as a character (hereinafter referred to as “pattern image”). The present invention relates to a method implemented for removing moire fringes generated in a pattern image, a moire removal apparatus to which the method is applied, and a moire removal program.

  In recent portable terminal devices equipped with a camera such as a mobile phone, an application for pattern recognition using the camera (such as an OCR or a code reader) is often incorporated. In this type of application, there is a case where recognition processing using a photographed image is performed in the device, but the image is transmitted to an external server device, the recognition processing is performed in the server device, and the recognition result is transmitted to the mobile terminal. In some cases, it is returned to the apparatus (for example, see Patent Document 1).

  In this way, it is now possible to easily read characters and code information with a portable terminal device on hand, so this function is being used in various scenes, especially on the screen of mail order shopping. In order to read a character string such as a displayed receipt number, a display screen of a personal computer or a television is often used. However, in the image captured by the display screen, a lot of moire fringes are generated due to a shift in the pixel arrangement between the display and the camera, which may cause an error in recognition processing. Even if no recognition error occurs, extra processing may be performed using moiré fringes as character candidates, and processing efficiency may be significantly reduced.

  As a technique for removing moiré fringes from an image, a process is known in which a frequency component corresponding to the period of moiré fringes is extracted by Fourier transform and the frequency components are removed. For example, in Patent Document 2, an image generated by X-ray transmission imaging is set as a processing target, a one-dimensional fast Fourier transform is performed on the processing target image, and a peak frequency is extracted from a frequency distribution generated by the conversion. A method of generating a grid moiré pattern image (image representing noise) by performing two-dimensional inverse Fourier transform on the extracted peak frequency and subtracting the grid moiré pattern from the original image is described.

JP 2004-179851 A JP 2011-177373 A

  In the invention described in Patent Document 2, the direction in which the moire fringes are found (the horizontal direction of the image (X direction)), and the regularity of the moire fringes is high. The period of moire fringes can be specified by performing Fourier transform and extracting the peak frequency. However, in an image obtained by shooting a display screen, the direction and period of moire fringes vary depending on the position and tilt of the camera with respect to the screen, and moire fringes in the image are often irregular. It is difficult to remove moire fringes by such a method.

  The present invention pays attention to the above problem, and stably removes the moire fringes generated in the pattern image to be recognized without being influenced by the direction, period, regularity, etc. The task is to ensure efficiency.

  Prior to performing pattern recognition processing using a pattern image generated by photographing a predetermined pattern, the present invention applies a two-dimensional Fourier transform to the pattern image in order to remove moire fringes generated in the pattern image. And generating a spectral image, extracting a set of low-frequency components distributed in the center of the spectral image, and a region outside the set of low-frequency components extracted in the central portion of the spectral image, Extracting a set of frequency components satisfying a predetermined standard as noise, removing a set of frequency components of noise from a spectral image including the set of low-frequency components in the center, and removing noise Performing a two-dimensional inverse Fourier transform on the subsequent spectral image.

  The spectrum image generated by the two-dimensional Fourier transform on the image is a distribution in which the frequency is higher as the frequency component in each direction becomes farther from the center with the center position as the frequency zero. In the spectrum image generated from the pattern image, the contour part with large density change is distributed as a high frequency component in a place away from the center, while the part with low density change such as the inside of the pattern or the background is centered as a low frequency component. It becomes a state distributed in the part. When the pattern image includes a large number of moire fringes, a spectrum image including a large amount of high frequency components corresponding to the width of each fringe and the interval between the fringes is generated.

Focusing on the above phenomenon, in the present invention, the frequency component in the low frequency band including the component corresponding to the inside of the pattern to be recognized is maintained, the frequency component in the high frequency band including the moire fringe is removed, and the removal is performed. A pattern image from which main moire fringes are removed is generated by performing a two-dimensional inverse Fourier transform on the subsequent spectral image.
According to this method, since the moire fringes can be greatly reduced without being influenced by the direction of moire fringes, the variation in the period, the presence or absence of regularity, etc., the accuracy and efficiency of the recognition process can be improved. It becomes possible.

  In one embodiment of the present invention, a step of binarizing a spectral image generated by a two-dimensional Fourier transform with a predetermined intensity and a step of performing an expansion process on the binarized spectral image are further executed. A step of extracting a set of low frequency components and a step of extracting a set of noise frequency components on the binary spectrum image after the expansion processing, and a step of removing the set of noise frequency components This is performed on a multi-value spectrum image before binarization.

  According to the above-described embodiment, the frequency components distributed in the multi-valued spectrum image by Fourier transform are narrowed down to those having strong intensity by the binarization process, and the frequency components located in the vicinity are connected by the expansion process. it can. Therefore, a set of low frequency components and a set of frequency components of noise can be easily extracted by contour tracking processing, clustering processing, and the like. On the other hand, since the process of removing the frequency component of noise is performed on the original multi-valued spectral image, the low-frequency component image can be reproduced without any problem by the two-dimensional inverse Fourier transform.

  To extract a set of low-frequency components in the center, for example, a sample image is used in advance to determine the frequency distribution range of the image corresponding to the inside of the pattern, and the spectrum based on the setting data registered based on the result A region of a predetermined size is set at the center of the image, and frequency components distributed in this region are extracted. Alternatively, in this type of spectral image, it is possible to extract a connected body of frequency components that are connected in series including the center point, based on the fact that low frequency components tend to be distributed over a relatively wide range including the center point. Good. When there is a high possibility that the distribution state of the low frequency component will fluctuate due to uneven density due to the influence of moire fringes or variations in pattern width, it is desirable to carry out both of the above two processes.

  A moiré removal apparatus to which the present invention is applied extracts a set of low-frequency components distributed in a central portion of a spectral image, and a first conversion means for generating a spectral image by performing a two-dimensional Fourier transform on the input pattern image A set of frequency components that satisfy a predetermined criterion and are extracted as noise from a region outside the set of low-frequency components extracted by the low-frequency component extractor of the spectrum image Noise extracting means; noise removing means for removing a set of noise frequency components from a spectral image including a set of low frequency components in the center; and a second one for performing two-dimensional inverse Fourier transform on the spectrum image after the noise removal. Conversion means.

  The moire removal apparatus is a computer that operates based on a program in which the processing of each means of the first conversion means, the low-frequency component extraction means, the noise extraction means, the noise removal means, and the second conversion means is described. When this program is incorporated in an information terminal device having a photographing function, moire removal processing can be performed in the information terminal device on the pattern image generated by imaging with the information terminal device.

  The above program can also be incorporated in an external device (server device) having an image input means for receiving a pattern image output from an information terminal device that has captured a pattern by communication. In this case, in addition to the process of removing moire fringes from the pattern image, the external device may have a function of performing recognition processing using the pattern image after removal and transmitting the recognition result to the information terminal device. it can. However, the recognition processing using the pattern image after removing the moire and the processing for transmitting the recognition result to the mobile terminal device may be performed in a second external device different from the above external device.

  According to the present invention, moire fringes generated in a pattern image can be stably removed without being affected by the direction, period, or regularity. Therefore, even when pattern recognition is performed on an image obtained by capturing a pattern displayed on the display screen, a highly accurate result can be obtained by recognition processing on an image from which the influence of moire fringes is excluded. it can. In addition, the processing efficiency can be improved.

It is a figure which shows the structural example of the recognition processing system to which this invention was applied. It is a figure which shows the example of the image of the state in which the moire fringe produced, and the correction image after a noise removal process. It is a flowchart which shows the procedure of a noise removal process. It is a figure which shows the example of the multi-value spectrum image and the binary spectrum image after an expansion process. It is a figure which shows the example which extracted the low frequency component from the binary spectrum image, and set the mask. It is a figure which shows the example which deleted the noise from the multi-value spectrum image based on the extraction result of the frequency component of noise.

  FIG. 1 shows a system including a mobile terminal device 1 having a color image photographing function, a communication server 2 and an analysis server 3 as a configuration example of a recognition processing system to which the present invention is applied. Although FIG. 1 shows one smartphone as the mobile terminal device 1, there are actually many different types of mobile terminal devices 1 that are permitted to connect to the communication server 2. To do.

  The mobile terminal device 1 incorporates an application for OCR. When the user activates this application and captures a character string, an image file including a color image generated by the capture is automatically transmitted from the mobile terminal device 1 to the communication server 2. The image file is transferred from the communication server 2 to the analysis server 3, and the analysis server 3 performs a series of processes for recognizing a character string in the image. The result of the recognition process (recognized character string) is transmitted from the analysis server 3 to the mobile terminal device 1 that is the transmission source of the image file via the communication server 2.

  After transmitting the image file, the OCR application of the mobile terminal device 1 waits for reception of the recognized character string and displays the received recognized character string on the screen. Since an image generated by shooting, information on the progress of communication with the communication server 2, and the like are not displayed at all, the user of the mobile terminal device 1 is not aware of a series of processes after performing a shooting operation. Only the recognition result can be confirmed. Since the recognized character string is transmitted as text data, when the user determines that the recognized character string is correct, the user transfers the recognized character string to another application by a predetermined operation, or stores it in a character input dictionary or the like. A recognition character string can be registered. If there is an error in a part of the recognized character string, it can be corrected manually.

The analysis server 3 of this embodiment includes an image input unit 31, a pre-processing unit 32, a noise removal processing unit 33, a recognition processing unit 34, a recognition result output unit 35, etc., as shown in the dotted frame in FIG. Function is set.
The image input unit 31 inputs an image file (a color image including a character string to be recognized) received from the mobile terminal device 1 via the communication server 2. The preprocessing unit 32 converts the input color image into a gray scale image and passes it to the noise removal processing unit 33. The noise removal processing unit 33 removes noise such as moire fringes from the gray scale image, and this function is set by the program according to the present invention.

  The recognition processing unit 34 includes a dictionary file (not shown) in which various characters are registered, cuts out the characters in the image one by one from the grayscale image after noise removal, and maps each cut out character to the dictionary. Recognize each character by matching with the registered characters in the file. Further, the recognition processing unit 34 generates a recognized character string by arranging the recognized characters in the original arrangement. This recognized character string is transmitted from the recognition result output unit 35 to the communication server 2.

  FIG. 2 (1) schematically shows a grayscale image A1 to be processed by the noise removal processing unit 33, and FIG. 2 (2) is obtained by performing noise removal processing on the grayscale image A1. An example of the corrected image A2 is shown. For convenience of illustration, in the image A1, the length direction of the moire fringes is aligned in the direction from the upper left to the lower right, and the arrangement direction and intervals of the moire fringes are also unified. A certain amount of moire fringe remnants (represented by a halftone dot pattern) is also observed in the corrected image A2, but does not include significant moire fringes as seen in the original image A1.

  In an image generated by photographing such as a PC screen or a television screen, there is a high possibility that the direction and period of the moire fringes vary in various ways, not the highly regular moire fringes as seen in the image A1 of FIG. However, according to the noise removal processing of this embodiment, the moire fringes can be removed to such an extent that the recognition processing is not hindered without being influenced by the state of the moire fringes.

  FIG. 3 shows the procedure of the noise removal process. 4 to 6 show an outline of processing at a specific step in FIG. 3 using work images B1 to B5 generated in the course of this processing. Hereinafter, the noise removal processing in this embodiment will be described along the procedure of FIG. 3 with reference to FIGS.

  When the grayscale image A1 to be processed is acquired in the first step S1, in step S2, the grayscale image A1 is subjected to a two-dimensional fast Fourier transform, and a multi-valued spectral image B1 (FIG. 4) showing the conversion result. Top). This spectrum image B1 shows the distribution state of frequency components in each direction with the center point as the frequency zero, and becomes closer to white as the intensity increases.

The spectrum image B1 shown in FIG. 4 is generated by performing a fast Fourier transform on the grayscale image A1 shown in FIG. In the spectrum image B1 in this example, strong frequency components are connected along the respective directions from the center to the top, bottom, left, and right, and a dense distribution of strong frequency components is generated around these connected portions. Furthermore, strong frequency components are gathered at several locations away from the center. In particular, in the spectral image B1 of this embodiment, a plurality of sets are arranged along the direction from the lower left to the upper right corresponding to the moire fringes, by the frequency component reflecting the moire fringes.
The set of low-frequency components close to the center in the spectrum image B1 includes frequency components that reflect portions with small changes in density, such as images inside characters and images between moire fringes in the background portion. The set of high frequency components includes a frequency component reflecting a portion having a large density change, such as a contour portion of a character, in addition to a frequency component reflecting a moire fringe.

  In this embodiment, in order to clarify and easily extract a set of various frequency components, the processes of steps S3 to S5 described below are performed. Since the first spectral image B1 is finally used again, it is held in the buffer memory.

  First, in step S3, a filter such as a Gaussian filter is applied to the spectrum image B1 to smooth the contour line in the image. In step S4, the smoothed spectral image is binarized with a predetermined threshold value. By this binarization, a binary spectrum image is generated in which a frequency component having an intensity exceeding the threshold is a white pixel and a frequency component having an intensity less than the threshold is a black pixel.

  In step S5, while paying attention to each pixel of the binary spectral image in turn, if there is a white pixel in any of the pixels near 8 around the pixel of interest, the pixel of interest is set as a white pixel, and any pixel in the vicinity of 8 If it is a black pixel, the white pixel group is expanded by a method in which the pixel of interest is a black pixel. Furthermore, by tracking the connection relationship of white pixels in the vicinity of the surrounding 8 and labeling each connected white pixel group, a set of white pixels in the spectrum image is individually recognized.

  The lower image B2 in FIG. 4 is a binary spectrum image corresponding to the spectrum image B1, and shows a state in which the processes in steps S3, S4, and S5 have been completed. A portion where strong frequency components are gathered in the multi-value spectrum image B1 is converted into a connected body of white pixels (hereinafter referred to as “white pixel region”) that extends over a wider range.

In step S6, a mask is set for the low-frequency component distributed in the center based on the result of the labeling process for the binary spectral image B2. Specifically, in this embodiment, as shown in the upper part of FIG. 5, a rectangular area having a predetermined size is set at the center of the spectrum image B2, and a mask is set for all white pixels included in the square area. In addition, white pixels that are outside the rectangular area and have the same label as the center point of the image (that is, pixels included in a connected body of white pixels that are continuous from the center point) are extracted, and a mask is applied to them. Set.
The rectangular area is set based on setting data determined based on a result of specifying the distribution range of the low frequency component using a sample image without moire fringes in advance.

  An image B3 in the lower part of FIG. 5 shows a spectrum image after the mask is set, and the masked portion is indicated by a hatched line together with the relationship with the rectangular area. On the upper left side of FIG. 6, the spectral image B3 after the same mask setting is represented in a format in which the background portion is replaced with white and the white pixel region is represented by a contour line (white pixels in which the solid line region is not masked). This is a low frequency component area in which the area and the dotted line area are masked.)

  In step S6, a mask is not set for pixels that are outside the central rectangular area and have the same label as the central point but whose distance from the rectangular area exceeds a predetermined allowable value. It is desirable to make it.

  In step S7, a white pixel region having an area equal to or larger than a reference value registered in advance is extracted from the spectral image B3 after the mask setting, and a noise flag is set for each pixel in the extracted region. The image B4 on the right side of the upper stage in FIG. 6 represents the noise flag setting result as an image.

  In step S8, the first generated multi-valued spectral image B1 is read from the buffer memory, and the frequency component corresponding to noise is removed from the spectral image B1 based on the setting of the noise flag. Specifically, the intensity corresponding to the pixel for which the noise flag is set in the spectrum image B1 is changed to zero.

  In the lower part of FIG. 6, a multi-valued spectral image B1 before noise removal (the same as that shown in FIG. 4) is shown on the left, and a multi-valued spectral image B5 from which noise has been removed is shown on the right. The spectral image B5 is obtained by replacing the frequency of the pixel corresponding to the noise shown in the image B4 in the original multi-valued spectral image B1 with zero, and a set of significant frequency components other than the central portion is removed. Yes. On the other hand, the characteristics of the initial spectral image B1 are reflected in the low frequency band at the center of the image B5.

  In step S9, a two-dimensional fast inverse Fourier transform is performed on the multi-valued spectral image B5 after the above noise removal. By this process, the corrected image A2 shown in FIG. 2 is generated.

  In the above embodiment, for the sake of simplification, an image in which moire fringes with high periodicity are generated is shown. Even in the spectral image, the distribution of strong frequency components becomes irregular. However, according to the procedure shown in FIG. 3, moire fringes in any state can be stably removed.

On the other hand, the high-frequency component set as noise by the processing in step S7 may include a frequency component reflecting the outline portion of the character in addition to the frequency component reflecting the moire fringes. Since the noise flag is set to the frequency component of the contour portion of the character and the frequency component is removed from the multi-valued spectral image B1, the character is blurred in the corrected image A2 by the two-dimensional fast inverse Fourier transform. End up.
However, since most of the moire fringes that adversely affect recognition are removed and most of the low-frequency components reflecting the image inside the characters are maintained, even if the characters are somewhat blurred, there is no particular problem in recognition. Therefore, recognition accuracy can be ensured.

  Therefore, even when an image obtained by photographing a character string displayed on the display screen is processed, a highly accurate recognition process can be performed without being affected by moire fringes. In addition, since it is possible to prevent the moire fringes from being handled as character candidates, the processing efficiency can be improved.

Hereinafter, possible modifications will be described.
First, in the above embodiment, in order to improve the detection accuracy of a set of low frequency components and a set of frequency components of noise, the multi-value spectrum image B1 by the two-dimensional fast Fourier transform is binarized and subjected to expansion processing or the like. Not limited to this, each set may be detected using a multi-valued spectral image. For example, the maximum value is obtained from the intensity distribution in the multi-value spectrum image, the peak portion including each maximum value is specified as the range where the strong frequency component is distributed, the mask is set for the peak distributed in the center, and the other Mountains can be regarded as noise and excluded.

  Even when the spectrum image is binarized, the process can be advanced by a method other than the expansion process or the labeling process. For example, by applying clustering processing to a binarized white pixel group, areas where white pixels are connected or densely distributed are extracted, and mask setting targets and noise flag setting targets are selected from these areas. Good.

  Next, in the above embodiment, a character string is a recognition target. However, the same applies to a case where an optical information code such as a two-dimensional code is read or a recognition process such as pattern matching is performed on a specific mark. With this method, the recognition process can be performed after removing the moire fringes from the image to be processed.

  With respect to the system configuration of FIG. 1, it has been described that the recognition character string is returned as the recognition result to the mobile terminal device 1 that has transmitted the recognition target image, but another process using the recognition character string is further performed, The result may be sent. For example, a web search using a recognized character string can be performed, and the search result can be associated with the recognized character string and transmitted.

  If the memory capacity of the mobile terminal device 1 is sufficient, a gray scale conversion process or noise removal process is performed in the mobile terminal device 1 after shooting, and the image after the noise is removed is transmitted to an external device. Recognition processing may be performed, or may be performed in the mobile terminal device 1 including recognition processing.

DESCRIPTION OF SYMBOLS 1 Mobile terminal device 3 Analysis server 31 Image input part 32 Pre-processing part 33 Noise removal processing part 34 Recognition processing part 35 Recognition result output part A1 Input image A2 Correction image B1, B5 Multi-value spectrum image)
B2 Binary spectral image after expansion processing B3 Binary spectral image after mask setting B4 Binary spectral image after noise flag setting

Claims (6)

Prior to performing pattern recognition processing using a pattern image generated by photographing a predetermined pattern, a method of removing moire fringes generated in the pattern image,
Applying a two-dimensional Fourier transform to the pattern image to generate a spectral image;
Extracting a set of low frequency components distributed in the center of the spectral image;
Extracting, as noise, a set of frequency components that satisfy a predetermined criterion and are distributed from a region outside the set of low-frequency components extracted at the center of the spectrum image;
Removing the set of noise frequency components from a spectral image including a set of low frequency components in the center;
Applying a two-dimensional inverse Fourier transform to the spectral image after removal of the noise,
A moiré removal method for pattern recognition characterized by being implemented. A step of binarizing the spectrum image generated by the two-dimensional Fourier transform with a predetermined intensity and a step of performing an expansion process on the binarized spectrum image;
The step of extracting the set of low frequency components at the center and the step of extracting the set of frequency components of noise are performed on the binary spectrum image after the expansion processing, and the set of noise frequency components is removed. The moire removal method for pattern recognition according to claim 1, wherein the step of performing is performed on a multi-valued spectral image before binarization.   In the step of extracting the set of low-frequency components in the central portion, a region of a predetermined size is set based on predetermined setting data in the central portion of the spectral image to be processed, and the frequency components included in the region are extracted. The moiré removal method for pattern recognition according to claim 1 or 2, wherein at least one of a process and a process of extracting a connected body of frequency components connected in series including a center point of a spectrum image is executed. An apparatus for inputting a pattern image generated by photographing a predetermined pattern and removing moire fringes generated in the pattern image,
First conversion means for performing a two-dimensional Fourier transform on the pattern image to generate a spectral image;
Low-frequency component extraction means for extracting a set of low-frequency components distributed in the center of the spectrum image;
Noise extraction means for extracting, as noise, a set of frequency components that satisfy a predetermined criterion and are distributed from a region outside the set of low frequency components extracted by the low frequency component extraction means of the spectral image;
Noise removing means for removing a set of frequency components of the noise from a spectral image including a set of low-frequency components in the center;
Second transforming means for performing a two-dimensional inverse Fourier transform on the spectrum image after removing the noise;
A moiré removal apparatus comprising: An apparatus as claimed in claim 4, comprising:
An image that receives a pattern image generated by photographing the predetermined pattern by an information terminal device having a photographing function and output from the information terminal device through communication, and supplies the pattern image to the first conversion means A moire removing device further comprising an input means. A program for causing a computer to function as a moire removing device for removing moire fringes generated in a pattern image generated by photographing a predetermined pattern,
First conversion means for generating a spectral image by performing a two-dimensional Fourier transform on the pattern image;
Low-frequency component extraction means for extracting a set of low-frequency components distributed in the center of the spectrum image;
Noise extraction means for extracting, as noise, a set of frequency components that satisfy a predetermined criterion and are distributed from a region outside the set of low frequency components extracted by the low frequency component extraction means of the spectrum image;
Noise removing means for removing a set of frequency components of the noise from a spectral image including a set of low-frequency components in the center;
A second transforming means for performing a two-dimensional inverse Fourier transform on the spectral image after removing the noise;
A program for moire removal processing, wherein the function of each means is set in the computer.