JP2005352705A - Device and method for pattern recognition, and character recognizing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To precisely recognize a character string even hen a frame line whose position is indefinite is present in the circumference of a pattern (e.g. character string) to be recognized. <P>SOLUTION: A readout range conforming with a specified shape is cut out of an image of the same subject which is continuous in time series, an image of an object pattern recognition line is taken out of the cut image, and while a pattern recognition candidate included in the image of the object pattern recognition line is extracted, a non-pattern-recognition candidate is removed from the pattern recognition candidate, so that the pattern recognition candidate from which the non-pattern-recognition candidate is removed is outputted as a determined pattern. When a character string encircled with frame lines is recognized as patterns and a part of a frame line is closed to the character string and misrecognized as the character string (for example, a longitudinal bar of a frame line is misrecognized as "I" or "1".), the misrecognized character is removed as a non-pattern-recognition candidate. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a pattern recognition apparatus, a pattern recognition method, and a character recognition method, for example, recognition of a registration number (so-called vehicle number) of a vehicle such as an automobile, and a character string (eg, telephone number, homepage) posted on an advertisement signboard. The present invention relates to a pattern recognition apparatus, a pattern recognition method, and a character recognition method suitable for use in recognition of addresses, e-mail addresses, and the like.

  Pattern recognition is the action that humans recognize the outside world through the visual system, that is, the similarity between an unknown input pattern and a standard pattern stored in advance is evaluated, and the standard pattern with the highest similarity is input. This refers to having a computer act as a series of actions that are recognized as patterns.

  Here, the type of standard pattern is called “class”. For example, in the case of numbers, there are 10 classes from 0 to 9. Typical target examples of pattern recognition are numbers, characters, symbols, and the like (hereinafter referred to as “characters”). This is because there is an upper limit in the class of characters and the like, and it is easy to handle. For example, numbers and alphabets have only a total of 128 types of 7-bit classes called ASCII (American Standard Code for Information Interchange) codes (94 types excluding blank characters and control symbols).

  In addition, an extended ASCII character set (8-bit ASCII characters) is also defined which can express accented characters, Cyrillic characters, Arabic characters, etc. used in other European languages. In addition, there are 63 different standards for 1-byte kana characters (JIS7 / JIS8) and multilingual character set standards that handle major characters (Japanese, Chinese, Korean, etc.) all over the world ( So-called Unicode) is also defined.

  As described above, among the pattern recognitions, those that recognize characters and the like, for example, when the target is numerals, alphabets, and some symbols, it is sufficient to evaluate at most 94 types of classes. This is practical because it does not cause any pressure on the computer, and the processing load on the computer is light. However, the idea of the present invention is not limited to such limited pattern recognition. It may be anything that can be defined as a class, not limited to characters and the like, and may recognize a figure designed arbitrarily. In the following description, the pattern to be recognized is described as “characters”, but this is for convenience of explanation.

  The prior art of the pattern recognition apparatus will be described. For example, Patent Document 1 describes a technique relating to “a method and apparatus for reading a wide range of documents from a time-series document image with a small calculation cost and memory amount”.

  According to this prior art, “time-series document image” means time-series image acquisition means, that is, a device such as a video camera that can acquire and output an image every moment, or a document moving image that has already been recorded. It is assumed that the image is acquired by a video device or the like that can reproduce the video.

  In general, a video camera generates and outputs a still image of several tens of frames per second (for example, 30 frames / second), and a video device records (records) an output image of the video camera and reproduces and outputs it as necessary. Therefore, the above “time-series document image” is a moving image obtained by taking a document (a character string written or printed on paper) as a subject and photographing the subject with a video camera for a predetermined time. It can be said that there is. The “time series” in this case is understood to mean that still images are continuous at a constant period, for example, an interval of about 30 frames / second.

  In this prior art, character recognition results extracted from images (that is, time-series document images) acquired with a small time difference have a large number of duplicates. In addition to showing the basic principle that results can be synthesized, if the number of characters in a time-series document image is small, this indicates a technical problem that character extraction is likely to be erroneous. By matching (DP matching) considering cut-out errors, and using the cumulative distance in the partial section of dynamic programming, the character code that solves the above technical problem, that is, easily recognizes the recognition error. (For example, “BA” and “PA”) can be correctly identified.

  Here, when DP matching is outlined, there is a case where the length of the standard pattern and the unknown input pattern (number of data of feature vectors) is usually different even in the same pattern figure. Also, even when the length of the same pattern is used, it may match very well when locally expanded or contracted. A process that is effective in such a case is a process of “corresponding (alignment) between elements of two patterns using dynamic programming, and thereby calculating a similarity”. This process is called DP matching after the acronym of Dynamic Programming. When you want to solve a certain problem, dynamic programming uses a solution of “a group of problems of the same type and smaller size”, and it can be obtained with a small amount of computation and by repeating the same procedure. It is a problem-solving technique that can be done, especially a computer-oriented technique.

Japanese Patent No. 2858560

  However, the above prior art is useful in that DP matching is applied so that character codes that are likely to cause recognition errors can be correctly distinguished. For example, recognition of a character string surrounded by an indefinite frame is recognized. There is a problem that the accuracy is not sufficient.

  FIG. 20 is a diagram illustrating an example of a character string image surrounded by a frame line. In this figure, a character string 1 of “123456” shown for convenience is surrounded by a horizontally long rectangular frame 2.

  When the character string 1 is cut out and recognized from the image 3 including the character string 1 and the frame line 2, a part of the frame line 2 may be mistaken as a character. For example, when the distance L between the character string 1 and the frame line 2 is extremely close, a part of the frame line 2 positioned before and after the character string 1 may be mistaken for “I” or “1”. . This misidentification cannot be avoided even if DP matching is applied. This is because the frame 2 is also reflected in each of the images acquired with a minute time difference. If the position of the frame line 2 is fixed (for example, if the distance L from the character string 1 is large and known), an unnecessary frame line 2 is set by setting a clipping window for only the character string 1 or the like. Although it is considered that the capturing can be suppressed and the misidentification can be avoided, when the position of the frame line 2 is indefinite, it is difficult to set an appropriate cutout window, and the misidentification of the character string 1 cannot be denied.

  Such a misidentification case may occur, for example, when a license plate of a vehicle with a decorative frame is recognized. In particular, when the decorative frame protrudes from the edge of the license plate to the inside, the decorative frame approaches the beginning or tail of the character string (vehicle number), or the decorative frame overlaps part of the character string. This can happen if you are. In addition, because the decorative frame of the license plate has various designs, the position of the frame line formed by the decorative frame is also undefined.

  The “decorative frame” refers to a frame that is attached as desired so as to surround the license plate in order to reinforce the license plate or enhance the appearance. Also called license plate frame. A wide variety of products are used, from simple designs to elaborate colors and shapes, or decorated with letters and graphics.

  Alternatively, when the character string 4 is surrounded by a horizontally long elliptical frame 5, when the character string 4 is cut out and recognized from the image 6 including the character string 4 and the frame 5 as described above. Again, a part of the frame line 5 may be misidentified as a character (a part of the frame line 5 positioned before and after the character string 4 may be “(”, “)”, etc.).

  Such a misidentification case may occur, for example, when a character string such as an advertising billboard is recognized. When the character string is surrounded by an elliptical frame to make it stand out, there is a case where a sufficient free space cannot be secured between the front and rear of the character string and the frame due to the convenience of the design.

  Therefore, an object of the present invention is to provide a pattern recognition device, a pattern recognition method, and a character recognition that can accurately recognize a character string even when a frame with an indefinite position exists around the recognition target pattern (for example, a character string). To provide a method.

The present invention provides an imaging unit that captures images of the same subject at a constant or indefinite period and outputs them in time series, a clipping unit that clips a reading range that matches a predetermined shape from the image, and the clipping Pattern recognition line extraction means for extracting an image of a target pattern recognition line from the cut image cut out by the means, extraction means for extracting a pattern recognition candidate included in the image of the pattern recognition line, and the pattern recognition candidate Removing means for removing non-pattern recognition candidates from the pattern, and output means for outputting the pattern recognition candidates from which the non-pattern recognition candidates have been removed as a definite pattern.
Preferably, the removing unit removes non-pattern recognition candidates from the pattern recognition candidates by DP matching.
Preferably, the predetermined shape is a shape corresponding to an outer shape of a license plate of a vehicle, and the pattern recognition line is a character string line on the license plate.
Here, the image pickup unit may take an image of the subject in real time, but is not limited thereto. For example, the image pickup unit may take a picture or record in advance like a video camera or a video device. It is also possible to reproduce and output the image of the subject that has been placed.

In the present invention, a reading range that matches a predetermined shape is cut out from images of the same subject that are continuous in time series, an image of a target pattern recognition line is taken out from the cut-out image, and becomes the target. Since the pattern recognition candidate included in the image of the pattern recognition line is extracted, the non-pattern recognition candidate is removed from the pattern recognition candidate, and the pattern recognition candidate from which the non-pattern recognition candidate is removed is output as a confirmed pattern. For example, in the case of pattern recognition of a character string surrounded by a frame line, a part of the frame line approaches the character string and is mistaken as a character string (the vertical bar of the frame line is “I” or “1”. In such a case, these misidentified characters are removed as non-pattern recognition candidates. Even if the border is present, the pattern recognition apparatus of the character string can be recognized accurately, it is possible to provide a pattern recognition method and a character recognition method.
In addition, a frame peripheral image including an image around the frame-shaped object or the frame-shaped graphic from among a plurality of images obtained by capturing the same subject including the frame-shaped object or the frame-shaped graphic in time series And character recognition is performed on each of the frame surrounding images corresponding to the plurality of images to calculate a plurality of character recognition candidate characters, and a subset of the character recognition candidate character strings is matched. If a character string with a high degree is output as a character recognition result, it is a continuous moving image in which the position of the frame cannot be specified, and there are many types of character digits and character arrangements. However, the character image is not erroneously recognized as a non-character. This is because the result of recognition for an image other than a character is determined as an indefinite character or is not stably recognized as the same character.
Further, the frame surrounding image is searched up and down to detect a portion that does not include a character region, a portion that does not include the character region is deleted from the frame surrounding image, and a portion that does not include the character region is deleted. If character recognition is performed on the surrounding image, it is possible to suppress a segmentation error in which characters are erroneously excluded due to interference with the frame.

  In the following, embodiments of the present invention will be described with reference to the drawings, taking application to a “vehicle license plate reading device” as an example, although not particularly limited. It should be noted that the specific details or examples in the following description and illustrations of numerical values, character strings, and other symbols are for reference only to clarify the idea of the present invention, and all or some of them may be used as a reference. Obviously, the idea of the invention is not limited. In addition, a well-known technique, a well-known procedure, a well-known architecture, a well-known circuit configuration, and the like (hereinafter, “well-known matter”) are not described in detail, but this is also to simplify the description. Not all or part of the matter is intentionally excluded. Such well-known matters are known to those skilled in the art at the time of filing of the present invention, and are naturally included in the following description.

  FIG. 1 is a system configuration diagram of a vehicle license plate reader. In this figure, a vehicle license plate reading device 11 (pattern recognition device) mounted on a vehicle such as an automobile (hereinafter referred to as “own vehicle”) 10 includes a television camera 12 (imaging means), a control unit 13, and a display unit. 14.

  The TV camera 12 is attached to the vehicle body with the photographing lens 15 having a predetermined angle of view facing an arbitrary direction (here, the front of the host vehicle 10), and the front image of the host vehicle 10 is displayed in a predetermined short period (for example, , About 30 frames per second) to generate a moving image that is a collection of still images that are continuous in time series, and outputs the generated moving image to the control unit 13. Here, the television camera 15 may be any device provided with an imaging device capable of generating the above-described moving image. In principle, for example, a television tube imaging tube may be used. In view of weight and weight, it is desirable to use a semiconductor imaging device such as a CCD (Charge Coupled Devices) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. Furthermore, considering a shooting environment of all brightness outdoors, it is desirable to use a camera (for example, a CMOS camera) having a particularly wide dynamic range. However, the specific configuration of the television camera 15 is not limited in the concept of the invention. A front image of the host vehicle 10 is shot at a predetermined short cycle (a constant cycle is desirable, but an indefinite cycle may be used as long as it is not extremely different), and a moving image that is a collection of continuous still images in time series is taken. Anything that can be generated and output is acceptable.

  The control unit 13 uses the moving image from the TV camera 15 to set the character string (vehicle number) of the license plate 17 of the vehicle to be recognized (here, “preceding vehicle 16” positioned in front of the host vehicle 10). In addition, the display unit 14 displays a result of character recognition and notifies a passenger (such as a driver) of the vehicle 10.

  FIG. 2 is a conceptual functional block diagram of the vehicle license plate reader 11. The vehicle license plate reader 11 includes an image input unit 18 (imaging unit), a position detection unit 19 (cutout unit), a vertical background removal unit 20 (pattern recognition line extraction unit), and a character recognition unit 21 (extraction unit) for each function. ), Correction processing unit 22 (removal unit), recognition result output unit 23 (output unit), and constraint condition storage unit 24.

  The image input unit 18 captures a front image of the host vehicle 10 at a predetermined short cycle, generates a moving image that is a collection of still images continuous in time series, and outputs the moving image. Corresponds to the television camera 15 described above. Note that the “input part” of the image is not limited to the output image of the television camera 15, but may mean, for example, a playback image of a video device.

  Each unit other than the image input unit 18, that is, the position detection unit 19, the vertical background removal unit 20, the character recognition unit 21, the correction processing unit 22, the recognition result output unit 23, and the constraint condition storage unit 24 are included in the control unit 13. For example, when the control unit 13 is composed of a computer and its peripheral circuits, hardware resources such as those computers and peripheral circuits, basic programs of the computer, various application programs, etc. However, it is needless to say that all or part of these functions may be configured by hard logic.

  FIG. 3 is a diagram showing an operation flow of the entire system of the vehicle license plate reader 11. In this figure, step S1 is an operation process (imaging process) of the image input unit 18, step S2 is an operation process (cutout process) of the position detection unit 19, and step S3 is an operation process (pattern recognition line) of the vertical direction background removal unit 20. Step S4 is an operation process (extraction process) of the character recognition unit 21, step S5 is an operation process (removal process) of the correction processing unit 22, and step S6 is an operation process (output process) of the recognition result output unit 23. is there. The vehicle license plate reader 11 recognizes the character string (vehicle number) of the license plate 17 of the preceding vehicle 16 while sequentially executing these steps S1 to S6, and displays the recognition result on the display unit 14. This operation is repeatedly executed while the host vehicle 10 is traveling.

  Here, the recognition result of the vehicle number of the preceding vehicle 16 is simply displayed on the display unit 14, but the present invention is not limited to this. For example, a vehicle number registration database may be provided separately, and the result of recognition of the vehicle number of the preceding vehicle 16 and the registration information in the database may be collated to display a match / mismatch on the display unit 14. If it does in this way, collation work of a stolen vehicle can be automated, for example.

  First, in step S1, moving images including the preceding vehicle 16 are continuously captured in time series, and in subsequent steps S2 to S6, each still image constituting the moving image (referred to as “one image” in the flowchart). )) To perform the required processing. That is, in step S2, an image of the license plate 17 (an image including at least a plate frame and a character string) is cut out from the still image, and in step S3, an image of a line (characters) in which a character string exists from the cut-out image of the license plate 17 An image including a column and a plate frame positioned before and after the character string is taken out. Next, in step S4, the upper and lower backgrounds excluding the line are removed from the image of the line in which the character string exists, the character included in the image from which the background is removed is recognized, and the recognized character string is temporarily recognized. Output as a string. Next, in step S5, DP matching processing is performed to determine and output a character string having a high degree of matching of a subset of the temporarily recognized character string as a corrected recognized character string. Finally, the recognition result is determined based on the corrected recognition character string, and the determination result is output to the display unit 14. The constraint condition storage unit 24 in FIG. 2 will be described later.

The operation of each part will be described in detail.
FIG. 4 is a diagram illustrating a specific operation flow of the position detection unit 19.
First, in step S2a, one frame image (still image) is extracted from the moving image input by the image input unit 18.

  FIG. 5 is a diagram illustrating the moving image 25 input by the image input unit 18. The moving image 25 is a moving image that is a collection of still images that are continuous in time series, and each still image is indicated by reference signs F1, F2, F3,.

Next, in step S2b, edge extraction of one frame image (still image) is performed.
FIG. 6 is a diagram showing the edge extraction image 26. Edge extraction refers to processing that makes a contour line in an image stand out. In the edge extraction image 26 shown in the drawing, contour lines of the rear image of the preceding vehicle 16 such as a body outer shape, a rear window, a lighting tool such as a tail lamp, a rear bumper, and a tail gate opening / closing knob are highlighted (white level emphasis in the figure). In addition, the outer frame of the license plate and the outline of the characters in the plate are also highlighted.

  Next, in step S2c, the size of the edge extraction image 26 is reduced. Here, for the sake of convenience, it is assumed that the size of the edge extraction image 26 is 193 × 96 pixels (pixels) and is compressed to 32 × 16 pixels.

  Next, in step S2d, the center of gravity of the attention area (here, the license plate 17) is obtained from the edge-extracted image 26 that has been subjected to size compression.

  FIG. 7 is a conceptual diagram used when obtaining the center of gravity of the region of interest (here, the license plate 17). Here, a neural network (hereinafter abbreviated as “NN”) is taken as an example. The illustrated NN has a three-layer structure of an input layer 27, an intermediate layer 28, and an output layer 29, and the center-of-gravity coordinates of the license plate are learned in the intermediate layer 28. Further, other than the license plate, for example, lighting learning such as a rear window and a tail lamp, a rear bumper, and a tail gate opening / closing knob are subjected to suppression learning. When the edge extraction image 26 whose size is compressed is given to the input layer 27, the position of the center of gravity of the license plate is extracted from the output layer 29.

  Finally, in step S2e, the center of gravity of the license plate is projected and added to the original image size to determine the cutout position of the partial image (image including the license plate area), and in step S2f, the license plate image and its position are output. . In addition, an approximate continuous distribution can be created by applying a Gaussian filter during projection addition, and the accuracy of position detection can be improved.

  FIG. 8 is a diagram illustrating an example of a license plate image. In FIG. 1A, a license plate image 31 (cut-out image) is a vehicle number character string (in the figure, US number “4UMV844” is shown as an example) and a horizontally long rectangular shape surrounding the character string. These are all highlighted with white edge lines. FIG. (B) is a diagram showing a license plate image 32 compressed by reducing the number of pixels.

FIG. 9 is a diagram showing a specific operation flow of the vertical direction background removal unit 20.
First, in step S3a, a license plate image 32 is input, and then in step S3b, a vertical edge image is projected in the horizontal direction to create a histogram.

  FIG. 10 is a license plate image 32 and its histogram distribution diagram. In FIG. 5A, the black part around the license plate image 32 is a decorative frame of the license plate, and the character string (“4WPD 602”) of the white part surrounded by the decorative frame is the vehicle number. In addition, an unreadable character string is recognized slightly above the vehicle number, and approximately two unreadable character strings are also recognized inside the decorative frame.

  FIG. 2B is an image 34 in which only the edges in the vertical direction are emphasized. In addition to the character string (vehicle number) 34a to be read, the vertical edge-enhanced image 34 includes several character strings 34b, 34c, 34d corresponding to the above-described unreadable character strings, and decorative frames of license plates. Unnecessary portions such as the vertical edge portions 34e and 34f are included.

  Now, when a histogram is created by projecting the vertical edge image of such a license plate image 32 in the horizontal direction, as shown in FIG. 4B, a large mountain 35a near the center and two small peaks positioned above and below it. A histogram 35 having peaks 35b and 35c is obtained. The large peak 35a represents the position of the character string (vehicle number) 34a to be read, and the two small peaks 35b and 35c represent the positions of the character strings 34b, 34c, and 34d that are not required to be read. The large mountain 35a and the two small peaks 35b and 35c are clearly different. In particular, the hem width A of the large mountain 35a is much larger than that of the two small peaks 35b and 35c, so The line position of the character string (vehicle number) 34a can be reliably specified.

  Therefore, in step S3c, a character string row is searched in the vertical direction from the center of the image 34 based on the distribution of the histogram 35. In step S3d, the width and upper / lower end information of the character string row are calculated. In step S3e, The binarization is performed on the image 34 including the plate area by the discriminant analysis method. In step S3f, the background of upper and lower portions not including characters is removed from the binarized image based on the upper and lower end information. Thereby, it is possible to extract an image of only the row of the character string (vehicle number) 34a to be read.

  FIG. 11 is a diagram showing an image 36 (image of a pattern recognition line) of only the line of the character string (vehicle number) 34a to be read extracted in this way. Compared with the original image 34 of FIG. 10, the upper and lower excess portions of the line of the character string (vehicle number) 34a to be read are removed. However, in this image 36, since the vertical edge portions 34e and 34f of the decorative frame of the license plate remain before and after the character string (vehicle number) 34a to be read (the left-right direction in the drawing), this unnecessary portion is represented by characters. May be misidentified (in the example shown, it is a vertical bar, so it is misidentified as “I” or “1”). This unnecessary portion is removed by the next character recognition unit 21 and correction processing unit 22.

FIG. 12 is a diagram illustrating a specific operation flow of the character recognition unit 21.
In the character recognition unit 21, first, in step S4a, the size of the labeled character is normalized to 12 × 24 pixels, and then in step S4b, the character is input to the three-layer alphanumeric character recognition NN.

  FIG. 13 is a conceptual structural diagram of a three-layer alphanumeric recognition NN. In this figure, 37 alphanumeric characters “0 to 9”, “A to Z” and “others” are identified for convenience of explanation. The three-layer alphanumeric character recognition NN includes an input layer 37, an intermediate layer 38, and an output layer 39. When the labeled character image 40 is given to the input layer 37, the output value is the highest among the 37 characters in each digit. The object is output as the character recognition result of the character image 40.

FIG. 14 is a diagram illustrating a specific operation flow of the correction processing unit 22.
In this correction processing unit 22, first, in step S5a, the character recognition candidate output by the character recognition unit 21 is input, and in step S5b, the distance between the character recognition candidates is the shortest and the degree of subset matching by DP matching. The position where is the highest is calculated. Then, the character string from the position with the highest degree of coincidence is output as the final corrected character recognition result.

  Here, the license plate character extraction accuracy for each frame in the moving image is not stable because it is affected by environmental changes, vehicle body colors, and the like. In addition, image blurring occurs while the vehicle is running, so that frequent clipping errors often occur and the number of digits in the character string is often incorrect. For example, a phenomenon in which a bumper or a plate boundary is cut out and erroneously determined as a character is a typical example.

  The number of digits is known in Japan's license plate, and land codes and vehicle type codes are composed of numbers only, except for the small letters. Therefore, as long as the Japanese license plate is targeted, it is possible to make corrections using the known position information of the vehicle number, land code, and vehicle type code, but the number of digits is not constant in overseas license plates. Some of them are not only combinations of alphanumeric characters but also symbols and marks, and are especially prone to misclassification when targeting foreign license plates.

  Therefore, in the present embodiment, a method of correcting the character cutout result of the current frame using the recognition result of the previous frame is used. In this method, the number of digits of the character string to be recognized may be unknown. Corresponding points between the character string clipped in the previous frame and the character string of the current frame are obtained, and the error is corrected by absorbing the shift of the cutout position.

DP matching is used to calculate the corresponding points.
FIG. 15 is a conceptual diagram of DP matching in the correction processing unit 22. Further, the following expression (1) is a DP matching calculation expression, and using this expression (1), the shortest distance between the recognized character strings of the previous frame and the current frame is calculated, and the position is calculated for the current frame. The correction result. In addition, the character string after the position correction process is voted on the character string stack as time-series data, and the character having the highest appearance frequency for each digit position is set as the final recognition result.

  Here, i and j are digit positions of recognized characters in the current frame and the previous frame, respectively. g (i, j) is the cumulative distance from the beginning of the recognized character string to the character (i, j). The position where this distance is the shortest is set as the correction position of the current frame.

FIG. 16 is a diagram illustrating a specific operation flow of the recognition result output unit 23.
In step S6a, the recognition result output unit 23 determines whether there is an input of a corrected character recognition result. If there is an input of a corrected character recognition result, next, in step S6b, a predetermined constraint condition is satisfied. It is determined whether or not. If the predetermined constraint condition is satisfied, it is determined in step S6c whether or not the corrected character recognition results are continuously the same. If the corrected character recognition results are the same continuously, in step S6d. The corrected character recognition result is output to the display unit 14.

Note that the predetermined constraint condition is information stored in the constraint condition storage unit 24 of FIG.
FIG. 17 is a diagram illustrating an example of information stored in the constraint condition storage unit 24. In the illustrated example, four constraint conditions are stored. The first constraint condition 41 is “when the number of digits is less than the minimum number of characters or when the number of characters is greater than or equal to the maximum number of characters”, and the second constraint condition 42 is “the recognition candidate character string starts. The third constraint 43 starts with a character string registered in the character string and does not output as a recognition result when the number of characters is less than or greater than the minimum number of characters. When there is a character string registered in the column and the number is less than the minimum number of characters or greater than the maximum number of characters, no recognition result is output.

Next, an evaluation experiment of the vehicle license plate reader 11 according to this embodiment will be described.
As the image data used in the evaluation experiment, a large number of monochrome images (images cut out with a useful range of 8 bits) taken with a CCD camera in the field of view in front of the vehicle 10 while driving on a highway in California were used. The image size is 640 × 480 pixels. Also, a CCD camera with a wide dynamic range was used in order to cope with a large luminance change. The image database stored 46 types of scenes with license plates. Note that scenes whose character strings cannot be read visually are removed from the database. Table 1 shows the number of image data.

  The image data is roughly classified into those having a decorative frame (number plate decorative frame) and those having no decorative frame. A 7-digit alphanumeric string is used as a standard layout, and other layouts are used as a special layout.

  First, the position detection process will be described. In the evaluation experiment, as the learning data for the position detection NN, 53 images that were randomly extracted from all 16392 images stored in the image database so that the license plates were all different were used. In addition, 17 images including portions where the NN is likely to react erroneously, such as a tail lamp portion and an emblem of a car, were used for the suppression learning.

  In the position detection process, when the center of gravity position of the license plate is known in the previous frame, it is not necessary to search the entire image when searching for the current frame. In addition, when the edge is unclear due to image blurring or the like, the position detection NN may erroneously react to the background or the like. Therefore, when the position of the license plate is known by the processing of the previous frame, we decided to search only the vicinity of the coordinates in the next frame. As a result, the calculation amount can be minimized and the processing speed can be improved.

  FIG. 18 is a distribution diagram of the inter-frame movement distance, and FIG. 19A is a diagram showing the effect of limiting the search area. The movement distance between license plate frames in the moving image data used for the evaluation is 35 pixels in the x-axis direction and 25 pixels in the y-axis direction, as shown in the distribution diagrams 45a and 45b of the movement distance between frames in FIG. is there. Effect of Limiting Search Area in FIG. 19A As shown in FIG. 46, compared to the method of searching all the frames (bar graph 46a), the method of limiting the search area of the next frame from the position detection coordinates of the current frame ( In the bar graph 46b), the position detection accuracy is increased from 87.1% (14283/16392) to 94.2% (15446/16392), and the average frame rate is greatly improved from 7.8 frames per second to 13.8 frames per second. . Furthermore, in the character recognition process, even when one alphanumeric character cannot be recognized, the recognition rate is 95.1% (bar graph 46c) by determining that it is a false detection of a non-existing frame or license plate centroid position (bar graph 46c). The average frame rate was improved to 14.3 frames per second.

  Alphanumeric recognition was performed using a license plate image without a decorative frame (10673 sheets) and an image with a decorative frame (4909 sheets) cut out by the position detection process. By estimating the upper and lower ends of the character string using the edge information in the vertical direction, it was possible to effectively remove the background area such as a decorative frame in contact with the lower end of the character string. The threshold for determining the upper and lower ends of the characters was set to “frequency × 0.085”, which is the maximum of the histogram, by preliminary experiments. By this method, 83.5% (4099/4909) of alphanumeric characters with a decorative frame could be cut out in addition to 89.1% (9511/10673) of license plate alphanumeric characters without a decorative frame.

  Using the character string information obtained by the character recognition process, the corresponding point with the character string of the previous frame was obtained by DP matching, and the digit position correction process was performed. The recognition rate in the case of only the alphanumeric recognition NN is 72.5%, but by adding character string correction processing using DP matching, misrecognition due to digit position shifts or missing characters is corrected, and the recognition rate 97.8% was obtained.

  FIG. 19B is a diagram illustrating an example of recognition improvement by DP matching. In this figure, when there is no correction processing, erroneous recognition frames exist intermittently until the end of the scene, whereas when correction processing using DP matching is added, the recognition result is obtained in the first 25 frames. Confirmed. This confirmed that the misrecognized frame was corrected. The recognition result was corrected in the same way when there was a misrecognition due to a segmentation error at the beginning of the character string or a drop error due to a luminance change.

  As described above, the vehicle-mounted license plate recognition system and its evaluation experiment have been described. By adding time-series information as moving images to the conventional position detection processing and character recognition processing by the NN method, It was confirmed that the vehicle number can be recognized with high accuracy even in a scene where the vehicle changes in real time. In addition, it was confirmed that the character strings were adaptively separated even when the decorative frame of the license plate was adjacent to the character strings, which was difficult with the conventional method.

  As described above, according to the present embodiment, among a plurality of still images constituting a moving image obtained by photographing the same object (subject including the license plate 17 of the preceding vehicle 16) in time series. Even if there are many types of character digits and character arrangements from multiple images in which the position of the frame containing the characters cannot be specified, the characters in the image of the character part in the frame with few decoration restrictions It is possible to obtain an excellent effect that character recognition can be performed correctly.

  Also, if you can accurately recognize only alphanumeric characters or only limited characters such as alphanumeric characters and katakana without separating all characters and pictures written in the frame and recognizing them, the following It can be used for various purposes as shown in

  For example, it is possible to recognize and use characters written on a license plate of a preceding vehicle, a road sign, or a signboard using an in-vehicle camera. There are various ways of use such as tracking a vehicle, identifying a stolen vehicle, and acquiring environmental information and principality information. It should be noted that the registration plate may be read with the license plate.

  Moreover, you may apply to the mobile phone with a moving image camera. Characters written on posters, menus, signboards, etc. can be recognized and used by photographing with a camera of a mobile phone. Examples of usage include recognizing the phone number or email address shown to send an email or making a call, or recognizing the store name shown on the signboard to find a phone number or making a phone call. Various things can be considered.

  Also, images taken with a video camera are input to an image processing device such as a personal computer, and a display board, poster, signboard, sign, price tag, number tag, name tag surrounded by a frame that was captured in the input image. The characters written on the license plate may be recognized and used. The usage example can be used as a title for image editing, as a keyword for scene search, or used to specify a shooting target.

  The image features common to these applications are that images of the same frame appear repeatedly in images taken in time series, the position of the frame in the image cannot be known in advance, and character images are recognized for each image. (In particular, the position of the frame in the image may move depending on the movement of the camera or the movement of the target. The illuminance of the frame or text varies due to environmental changes such as natural light, lighting, or shadows. The frame and text may be temporarily hidden by the influence of the object between the camera and the objects that make up the frame, and images that are similar in size and shape to characters other than text are included in the frame ), Etc.

  In addition, images included in frames used for specific purposes, such as license plates, road signs, and display boards, are also non-character images that have restrictions on the type, size, and position of characters displayed in the frames. There are features such as restrictions on notation. By utilizing these features, the recognition rate can be further improved.

  In the present invention, the feature relating to the image of the frame included in the image repeatedly photographed (time-series image obtained by photographing the same object a plurality of times) in order to enable the application described above is utilized to the maximum extent. To make it possible to accurately recognize limited characters among the characters written in the box.

  For time-series images obtained by shooting the same object multiple times, the frame position may move within the image under conditions constrained by the relationship between the camera and the frame. Characters at the same position include an image that is easy to recognize and an image that is difficult to recognize, and there are features such that a correct recognition result may be included when repeatedly taken and recognized.

  There is a correspondence relationship between the characters in the preceding and following images, but there is a possibility of misrecognition, so they do not always match completely. For this reason, if it is attempted to improve the recognition accuracy by using the characteristics of an image taken a plurality of times, it is necessary to solve the following problems that do not occur when character recognition is performed from one image.

  That is, it is necessary to improve the recognition accuracy by using the image of the same frame so that it can cope with the fluctuation of the position in the image of the frame. In addition, the number of characters in the written character string is not known, and all characters recognized in the frame may be correct, but may include misrecognized characters. In order to calculate and correct the degree of coincidence based on the above, it is necessary to recognize that there is no omission in the part before and after the character string in the frame. Even if the image is taken a plurality of times, if a character in the frame is erroneously recognized as being outside the frame or the frame, the character is not output as a recognized character candidate and cannot be corrected. In addition, in order to be able to calculate the degree of match, even when unrecognizable characters are included, so that character omission is not lost during character recognition so that information on the correspondence between character strings is not lost, It is necessary to output the recognition result of recognized characters.

The above-described problem in improving the recognition accuracy using images taken a plurality of times is solved as follows.
(B) When a character string is recognized in the frame by character recognition performed on the previous image, the search range of the next frame is limited based on the position information, and the previous frame Avoid cropping different images.
(B) Recognition is performed with the maximum number of characters predicted for an image including an image around the frame in the length direction of the character string, and recognition is performed including image portions other than the characters included therein, and character strings are missing. Recognize that there is no.
(C) Character recognition is performed so that other characters and images other than characters are not mistakenly recognized as limited characters, and unrecognizable characters are set as indefinite characters so that the correspondence between characters in the recognition candidate character strings does not shift. The degree of coincidence is calculated.

  In character recognition, the method that uses the character with the closest similarity as the recognition result misrecognizes a non-alphanumeric character as an alphanumeric character, so the similarity is high with any of the alphanumeric characters to avoid it. It is sometimes desirable to output it as a recognition candidate, and to output an indefinite character as a recognition candidate if the similarity of any alphanumeric characters is low.

  In this way, there are many types of character digits and character arrangements from consecutive moving images where the position of the license plate cannot be specified. Misrecognition is suppressed.

  In addition, the result of recognition for an image other than a character is determined as an indefinite character by the character recognition unit, or is not stably recognized as the same character, and thus is excluded by the correction process. Further, a segmentation error in which characters are erroneously excluded due to interference with a frame or decoration is suppressed. Further, when only a part of the license plate is included in the image, when the number of digits is small, only the characters that are in the shooting range are recognized, and the recognition result that the characters outside the shooting range are dropped is not output. Further, when the number plate part moves from outside the shooting range to within the shooting range, there is no erroneous recognition that some characters are dropped. Further, when the number plate part moves from the shooting range to the outside of the shooting range, there is no erroneous recognition that some characters are dropped.

It is a system block diagram of a vehicle license plate reader. 2 is a conceptual functional block diagram of a vehicle license plate reading device 11. FIG. It is a figure which shows the operation | movement flow of the whole system of the vehicle license plate reader. It is a figure which shows the specific operation | movement flow of the position detection part 19. FIG. It is a figure which shows the moving image 25 input by the image input part 18. FIG. It is a figure which shows the edge extraction image 26. FIG. It is a conceptual diagram used when calculating | requiring the gravity center of attention area (here number plate 17). It is a figure which shows the example of a license plate image. It is a figure which shows the specific operation | movement flow of the up-down direction background removal part. It is a license plate image 32 and its histogram distribution map. It is a figure which shows the image 36 of only the line of the character string (vehicle number) 34a of the reading object. It is a figure which shows the specific operation | movement flow of the character recognition part. It is a conceptual structure figure of NN for 3 layer alphanumeric character recognition. It is a figure which shows the specific operation | movement flow of the correction process part. It is a conceptual diagram of DP matching in the correction processing unit 22. It is a figure which shows the specific operation | movement flow of the recognition result output part. 6 is a diagram illustrating an example of information stored in a constraint condition storage unit 24. FIG. It is a distribution map of the movement distance between frames. It is a figure which shows the effect by search area limitation, and the figure which shows the example of recognition improvement by DP matching. It is a figure which shows an example of the character string image enclosed with the frame line.

Explanation of symbols

S1 step (imaging process)
S2 step (cutout process)
S3 step (pattern recognition line extraction process)
S4 step (extraction process)
S5 step (removal process)
S6 step (output process)
11 Vehicle license plate reader (pattern recognition device)
12 TV camera (imaging means)
18 Image input unit (imaging means)
19 Position detector (cutout means)
20 Vertical background removal unit (pattern recognition line extraction means)
21 Character recognition part (extraction means)
22 Correction processing unit (removal means)
23. Recognition result output unit (output means)
31 License plate image (cutout image)
36 images (images of pattern recognition lines)

Claims (8)

Imaging means for capturing images of the same subject at a constant or indefinite period and outputting them in time series;
Clipping means for cutting out a reading range that matches a predetermined shape from the image;
Pattern recognition line extraction means for extracting an image of a target pattern recognition line from the cutout image cut out by the cutout means;
Extraction means for extracting pattern recognition candidates included in the image of the pattern recognition line;
Removing means for removing non-pattern recognition candidates from the pattern recognition candidates;
An output means for outputting the pattern recognition candidate from which the non-pattern recognition candidate has been removed as a definite pattern. The pattern recognition apparatus according to claim 1, wherein the removing unit removes a non-pattern recognition candidate from the pattern recognition candidates by DP matching. 2. The pattern recognition apparatus according to claim 1, wherein the predetermined shape is a shape corresponding to an outer shape of a license plate of a vehicle, and the pattern recognition line is a character string line on the license plate. . An imaging process for capturing images of the same subject at a constant or indefinite period and outputting them in time series;
A cutout step of cutting out a reading range that matches a predetermined shape from the image;
A pattern recognition row extraction step for extracting an image of a target pattern recognition row from the cutout image cut out by the cutout step;
An extraction step of extracting pattern recognition candidates included in the image of the pattern recognition line;
Removing the non-pattern recognition candidate from the pattern recognition candidates;
An output step of outputting the pattern recognition candidate from which the non-pattern recognition candidate is removed as a definite pattern. 5. The pattern recognition method according to claim 4, wherein the removing step removes non-pattern recognition candidates from the pattern recognition candidates by DP matching. 5. The pattern recognition method according to claim 4, wherein the predetermined shape is a shape corresponding to an outer shape of a license plate of a vehicle, and the pattern recognition line is a character string line on the license plate. . A frame peripheral image including an image around the frame-shaped object or the frame-shaped figure is cut out from a plurality of images obtained by chronologically capturing the same subject including the frame-shaped object or the frame-shaped graphic. , Character recognition is performed on each of the frame surrounding images corresponding to the plurality of images to calculate a plurality of character recognition candidate characters, and the matching degree of the subset of the character recognition candidate character strings is A character recognition method that outputs a high character string as a character recognition result. A frame surrounding image in which the frame surrounding image is searched up and down to detect a portion that does not include a character region, the portion that does not include the character region is deleted from the frame surrounding image, and the portion that does not include the character region is deleted The character recognition method according to claim 7, wherein character recognition is performed on the character.

Images