JP2021005164A - Character recognition device, imaging device, character recognition method, and character recognition program - Google Patents

Abstract

To accurately read characters even when the characters are not unified in an image.SOLUTION: A character recognition device (1) comprises: a first model being a convolutional neural network which outputs a feature quantity of a character image; a second model being a recurrent neural network which receives the feature quantity outputted from the first model once or more and outputs character information; and a character processing unit which uses the models to output character information indicative of a character included in the character image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a character recognition device, a photographing device, a character recognition method, and a character recognition program.

Conventionally, a technique of reading characters from an image using AI (Artificial Intelligence) is known. For example, in Non-Patent Document 1, characters are recognized while cutting out a feature matrix (feature map) obtained by inputting an image containing characters arranged in one line into a machine learning model one column at a time in the horizontal direction. The technology is disclosed.

Palaiahnakote Shivakumara, 5 outsiders, "CNN-RNN based method for license plate recognition", [online], CAAI Trans. Intell. Technol., 2018, Vol. 3, Iss. 3, pp. 169-175, [Reiwa Search on June 5, 1], Internet <URL: https://core.ac.uk/download/pdf/161769815.pdf>

However, the above-mentioned prior art has a problem that the reading accuracy is lowered when the character arrangement, the character size, and the character attributes such as the font are not unified. For example, when trying to process an image containing a plurality of lines of characters, the feature amount of one column cut out from the feature amount matrix includes the feature amount of a plurality of characters arranged in the vertical direction, so that the feature amount of one column is included. There is a problem that even if a quantity is used, it cannot be recognized as one character.

One aspect of the present invention is to provide a technique capable of accurately reading characters even if the arrangement and attributes of the characters are not unified in the image.

In order to solve the above problems, the character recognition device according to one aspect of the present invention is a first convolutional neural network in which a character image containing one or more characters is input and a feature amount of the character image is output. The model, the second model which is a recursive neural network in which the feature amount output by the first model is input one or more times and the character information indicating the character included in the character image is output, and the first model. The model and the second model are used to provide a character processing unit that outputs character information indicating characters included in the character image.

The character recognition method according to one aspect of the present invention is a character recognition method using a first model and a second model, in which the first model inputs a character image containing one or more characters. It is a convolutional neural network that outputs the feature amount of the character image, and the second model is input with the feature amount output by the first model one or more times, and the character included in the character image is input. It is a recursive neural network that outputs the indicated character information, and includes a character processing step that outputs character information indicating characters included in the character image using the first model and the second model.

According to one aspect of the present invention, it is possible to provide a technique capable of accurately reading characters even if the arrangement and attributes of the characters are not unified in the image.

It is a block diagram which shows the structure of the character recognition apparatus which concerns on one Embodiment of this invention. It is a figure which shows the example of the image processed by the character recognition apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the processing of the character recognition apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the process which outputs the character from the character partial image which concerns on one Embodiment of this invention. It is a figure which shows the example of the license plate which concerns on one Embodiment of this invention. It is a figure which shows the input / output of the model which concerns on one Embodiment of this invention. It is a block diagram which shows the hardware structure of the character recognition apparatus which concerns on one Embodiment of this invention.

[Embodiment]
Hereinafter, one embodiment of the present invention will be described in detail.

(Configuration of character recognition device 1)
FIG. 1 is a block diagram showing a configuration of a character recognition device 1 according to the present embodiment. The character recognition device 1 includes an input / output unit 11, a character recognition unit 12, and a learning unit 13. The character recognition unit 12 includes a character processing unit 14, a third model MA, a first model MB, and a second model MC.

A camera (photographing device) 2 and a display device 3 are connected to the character recognition device 1. The camera 2 captures a predetermined scene and outputs a captured image, which is the captured image, to the character recognition device 1. The type of camera 2 includes, for example, both a camera used for public purposes (for example, a traffic safety monitoring system, etc.) and a camera used for individuals. The display device 3 displays the character recognition result of the captured image output from the character recognition device 1.

In the character recognition unit 12, the third model MA is a convolutional neural network (CNN) in which an image is input and region information indicating a region where characters exist in the image is output. In the present specification, the wording "character" includes symbols, numbers, and characters in various languages. In addition, the area information includes the coordinates of the four corners of the area in which the character exists in the image. The third model MA is trained using an image including a region in which characters exist and teacher data including the coordinates of the four corners of the region in which the characters exist in the image.

The first model MB is a convolutional neural network in which a character image containing one or more characters is input and the feature amount of the character image is output. The second model MC is a recurrent neural network (RNN) in which the feature amount output by the first model is input one or more times and character information indicating the characters included in the character image is output. The recurrent neural network may be a long short-term memory network (LSTM). The first model MB and the second model MC are trained using teacher data including a character image including one or more characters and character information indicating characters included in the character image. ..

The input / output unit 11 extracts an image including a plurality of characters from the image captured by the camera 2 and outputs the image to the character processing unit 14. Further, the input / output unit 11 causes the display device 3 to display the character information output from the character processing unit 14. The character information output by the input / output unit 11 may be provided to another information processing device in addition to or in place of the display device 3. In such other information processing devices, the above character information is used for comparison with other character information stored in a database, or is compared with character information based on an image captured by another camera. It is also used for.

The character processing unit 14 uses the third model MA, the first model MB, and the second model MC to output character information indicating characters included in the character image. The character processing unit 14 inputs the image acquired from the input / output unit 11 into the third model MA, cuts out the area indicated by the area information output by the third model MA from the image, and sets the cut out area as a character image. Is input to the first model MB. However, the character processing unit 14 may perform processing that does not use the third model MA, and in that case, the character processing unit 14 uses the image acquired from the input / output unit 11 as a character image in the first model MB. input.

Then, the character processing unit 14 inputs the feature amount output by the first model MB into the second model MC at least once, and concatenates the character information for one character output by the second model MC. , Get the characters contained in the character image. In one aspect, the character processing unit 14 inputs the feature amount of the character image and the character information indicating the character output by the second model MC in the previous time to the second model MC.

The learning unit 13 trains the first model MB and the second model MC using the teacher data. Specifically, the learning unit 13 trains the first model MB and the second model MC, respectively, using the loss function calculated from the output of the second model MC. In addition, the learning unit 13 trains the third model MA using the teacher data.

(Example of image)
FIG. 2 is a diagram showing an example of an image processed by the character recognition device 1 according to the present embodiment. Reference numeral 201 denotes an example of a photographed image of a road including a vehicle body. 202 is an example of a vehicle body image (hereinafter referred to as a vehicle body image). 203 is an example of a license plate image (hereinafter referred to as a license plate image).

(Processing of character recognition device 1)
FIG. 3 is a flowchart showing the processing of the character recognition device 1 according to the present embodiment. Hereinafter, a process in which the character recognition device 1 reads (recognizes) the characters on the license plate from the image including the vehicle body (that is, the character image is a photographed image including the license plate or a partial image of the photographed image) will be described. This process is performed in real time. The character recognition device 1 may acquire an image from a recording medium provided in its own device, or may acquire an image via a local or global network, instead of acquiring an image from the camera 2. ..

(Step S301)
In the character recognition device 1, the input / output unit 11 acquires a photographed image of the road including the vehicle body (corresponding to 201 in FIG. 2) from the camera 2.

(Step S302)
Next, the input / output unit 11 detects the vehicle body from the captured image acquired in step S301, and cuts out the vehicle body image (corresponding to 202 in FIG. 2). An arbitrary object detection algorithm can be used for the process of detecting the vehicle body from the captured image. For example, "SqueezeDet" or the like, which is relatively lighter than other methods, is used. As a result, high-speed processing becomes possible, and real-time processing can be realized.

(Step S303)
Then, the input / output unit 11 detects the license plate from the vehicle body image cut out in step S302, and cuts out the license plate image (corresponding to 203 in FIG. 2). An arbitrary object detection algorithm can be used for the process of detecting the license plate from the vehicle body image, and for example, SqueezeDet or the like is used as in step S302. As a result, high-speed processing becomes possible, and real-time processing can be realized. Further, by cutting out the license plate image from the vehicle body image, the license plate image and the vehicle body image can be managed in association with each other.

(Step S304)
Subsequently, the input / output unit 11 inputs the license plate image cut out in step S303 into the third model MA, and cuts out a character partial image from the license plate image. As will be described later, by performing character recognition using the character partial image, it is possible to improve the identification accuracy as compared with the case of performing character recognition using the license plate image.

The learning unit 13 causes the third model MA to learn the character portion (for example, a rectangular area) included in the license plate image in advance. In that case, as the teacher data, a license plate image to which the coordinates of each vertex (four corners) of the rectangular region including characters are added is used.

The process of step S304 is not essential and may be omitted.

(Step S305)
Further, the character processing unit 14 acquires a character partial image from the third model MA, inputs the character partial image to the first model MB, and causes the second model MC to output characters.

FIG. 5 is a diagram showing an example of a license plate according to the present embodiment. For example, when the license plate is 503 in FIG. 5, the second model MC outputs "TN77J8285". It is also possible to train the first model MB and the second model MC so as to output "-". The details of the process of step S305 will be described with reference to FIG.

(How to create teacher data)
An example of how to create teacher data for models MA, MB and MC is shown below.
i) Obtain a large number of license plate images.
ii) Annotate the coordinates of the four corners by hand for each license plate image.
iii) For each license plate image, read the character string by hand. At this time, the reading direction of multi-stage characters and the presence or absence of reading "-" are unified.
iv) The license plate image of i) and the annotation data of the coordinates of the four corners of ii) are used as the teacher data of the third model MA for cutting out the character partial image.
v) Generate a character partial image from the license plate image of i) and the annotation data of the coordinates of the four corners of ii).
The character partial image of vi) v) and the character string of iii) are used as teacher data of the first model MB and the second model MC.

After learning has progressed to some extent, the character recognition device 1 itself performs vehicle body detection, license plate detection, license plate cutting, and character reading for new data. Finally, the annotation data can be semi-automatically generated by correcting the part where the human has made a mistake in reading the character.

(Process to output characters from the character partial image)
FIG. 4 is a flowchart showing a process of outputting characters from the character partial image according to the present embodiment. This process is a detailed version of the process of step S305 in FIG. FIG. 6 is a diagram showing input / output of the first model MB and the second model MC according to the present embodiment.

Here, since the first model MB is, for example, a convolutional neural network having 10 or less layers, it has real-time responsiveness. As an example, the first model MB is a convolutional neural network consisting of 6 layers. In one aspect, in the first model MB, the character partial image of the license plate is input, and the feature matrix of the character partial image is output.

Further, the second model MC is a recurrent neural network, in which the feature matrix of the character partial image is input one or more times, and the character information indicating the characters of the license plate is output one character at each time.

(Step S401)
In the character recognition device 1, the character processing unit 14 inputs the character partial image acquired from the third model MA into the first model MB, and outputs a feature matrix. This process corresponds to the input / output of the first model MB in (first step) of FIG.

(Step S402: Character processing step)
Next, the character processing unit 14 inputs the feature matrix output by the first model MB and the special character indicating the beginning into the second model MC, and outputs one character. Here, the character processing unit 14 inputs the feature matrix output by the first model MB as it is into the second model MC. This process corresponds to the input / output of the second model MC in (first step) of FIG.

(Step S403)
Here, the character processing unit 14 determines whether or not one character output by the second model MC is a special character indicating the end. When one character is a special character indicating the end (YES in step S403), the character processing unit 14 ends the processing. This is the post-processing of the input and output of the second model MC in (the last step) of FIG. Here, for example, the character recognition device 1 may concatenate and output the characters output in step S402.

On the other hand, when one character is not a special character indicating the end (NO in step S403), the character processing unit 14 executes the process of step S404.

(Step S404: Character processing step)
The character processing unit 14 inputs the one character first output by the second model MC and the feature matrix output by the first model MB into the second model MC, and outputs the next one character. In the second model MC, the one character output first by the second model MC is interpreted to be used to determine where to look next in the character partial image. This process corresponds to the input / output of the second model MC in (the i-th step) of FIG.

In one aspect, in step S404, the character processing unit 14 may be configured not to input one character previously output by the second model MC into the second model MC. In this case, the internal state of the second model MC is interpreted to be used to determine where to look next in the text partial image. However, in step S404, it is preferable that the character processing unit 14 inputs one character previously output by the second model MC to the second model MC.

(Effect of this embodiment)
In the configuration of Non-Patent Document 1 in which the feature amount matrix of an image containing characters is divided one column at a time and the feature amounts of the divided one column are sequentially input to the RNN, an image in which characters are arranged in multiple stages is processed. In some cases, the feature amount of one column may include the feature amount of a plurality of characters, so that the identification accuracy is lowered.

The second model MC according to the present embodiment has a configuration in which the entire feature matrix of the character image is input and character information for one character is output. Therefore, by learning the image in which the characters are arranged in multiple stages, it is possible to prevent the identification accuracy from being lowered even in the image in which the characters are arranged in multiple stages. As described above, according to the present embodiment, it is possible to provide a technique capable of accurately reading characters even if the arrangement and attributes of the characters are not unified in the image.

[Additional notes]
In the above embodiment, the characters included in the license plate of the vehicle body have been described, but the present invention can be applied to other uses. In the following, other uses will be described.

(1) The character recognition device 1 may perform real-time character recognition from an arbitrary image (for example, a business card, a signboard, a leaflet, etc.) other than a license plate whose character arrangement, size, and font are not unified. ..

(2) The character recognition device 1 may read a multi-stage character. That is, the character image may include a plurality of characters arranged in a plurality of stages.

(3) The character recognition device 1 may perform super-resolution processing, blur correction processing using a plurality of frames, and the like immediately before inputting an image into the model. Further, the character recognition device 1 may perform a process of correcting the inclination of the character image (license plate image).

(4) The captured image of the camera 2 may be an infrared image.

(5) The second model MC may output two or more characters instead of one character at a time. The number of characters output by the second model MC at one time can be adjusted by the teacher data at the time of learning. Further, instead of the character itself, the character replaced with a number (character information indicating the character) may be output.

(6) When applied to Japanese license plates, it is advisable to prepare a dictionary of place names.

(7) The camera 2 may be configured to include at least a part of the character recognition device 1. In this case, the camera 2 includes a shooting unit and a character recognition device 1, and the character processing unit 14 outputs character information by using the captured image captured by the photographing unit or a partial image of the captured image as a character image. May be good. Further, the camera 2 includes a photographing unit, an input / output unit 11, and a character recognition unit 12, and the character processing unit 14 uses a photographed image photographed by the photographing unit or a partial image of the photographed image as a character image and characters. Information may be output. In this case, the function of the learning unit 13 may be provided by a server connected to the camera 2 via the network.

[Example of realization by software]
Even if the control block of the character recognition device 1 (in particular, the input / output unit 11, the character recognition unit 12, and the learning unit 13) is realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. It may be realized by software.

In the latter case, the character recognition device 1 includes a computer that executes instructions of a program that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. Then, in the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, in addition to a "non-temporary tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the above program may be further provided. Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

FIG. 7 is a block diagram showing a specific example of the hardware configuration of the character recognition device 1 according to the present embodiment. The character recognition device 1 includes an input / output unit 11, an arithmetic unit 15, a main storage device 16, and an auxiliary storage device 17, which are connected to each other via a bus 18. A camera 2 and a display device 3 are connected to the input / output unit 11. The arithmetic unit 15, the main storage device 16, and the auxiliary storage device 17 may be, for example, a processor (for example, CPU: Central Processing Unit, GPU: Graphics Processing Unit, etc.), RAM (random access memory), or a hard disk drive, respectively. .. The arithmetic unit 15 includes the learning unit 13 and the character processing unit 14 shown in FIG. 1 as an example. As an example, the main storage device 16 and the auxiliary storage device 17 store the third model MA, the first model MB, and the second model MC shown in FIG.

(Summary)
The character recognition device according to the first aspect of the present invention includes a first model which is a convolutional neural network in which a character image containing one or more characters is input and outputs a feature amount of the character image, and the first model. A second model, which is a recursive neural network in which the feature amount to be output is input one or more times and character information indicating a character included in the character image is output, and the first model and the second model are used. It is provided with a character processing unit that outputs character information indicating characters included in the character image.

In the character recognition device according to the second aspect of the present invention, in the first aspect, the character processing unit outputs the characters output by the second model to the second model together with the feature amount in the previous time. You may enter the character information to indicate.

In the character recognition device according to the third aspect of the present invention, in the first or second aspect, the first model and the second model display a character image containing one or more characters and characters included in the character image. It may be that the training is performed using the teacher data including the indicated character information.

In the third aspect, the character recognition device according to the fourth aspect of the present invention may further include a learning unit for learning the first model and the second model using the teacher data.

The character recognition device according to the fifth aspect of the present invention further comprises a third model, which is a convolutional neural network in which an image is input and area information indicating an area in which a character exists in the image is output in the first to fourth aspects. The character processing unit may cut out a region indicated by the area information output by the third model from the image, and input the cut out area as the character image into the first model.

In the character recognition device according to the sixth aspect of the present invention, in the fifth aspect, the third model includes a teacher data including an image including a region where characters exist and the coordinates of four corners of the region where characters exist in the image. It may be that it was trained using.

In the character recognition device according to the seventh aspect of the present invention, in the first to sixth aspects, the character image may include a plurality of characters arranged in a plurality of stages.

In the character recognition device according to the eighth aspect of the present invention, in the first to seventh aspects, the character image may be a photographed image including a license plate or a partial image of the photographed image.

The photographing device according to the ninth aspect of the present invention includes a photographing unit and a character recognition device according to any one of the first to eighth aspects, and the character processing unit of the character recognition device is a photographed image taken by the photographing unit. Alternatively, the character information is output by using the partial image of the captured image as the character image.

The character recognition device according to the tenth aspect of the present invention is a character recognition method using a first model and a second model, and the first model inputs a character image containing one or more characters. It is a convolutional neural network that outputs the feature amount of the character image, and the second model is input with the feature amount output by the first model one or more times, and the character included in the character image is input. It is a recursive neural network that outputs the indicated character information, and includes a character processing step that outputs character information indicating characters included in the character image using the first model and the second model.

The character recognition device according to each aspect of the present invention may be realized by a computer. In this case, the character recognition device is made into a computer by operating the computer as each part (software element) included in the character recognition device. The character recognition program of the character recognition device and the computer-readable recording medium on which the character recognition device is recorded also fall within the scope of the present invention.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1 character recognition device 2 camera (shooting device)
11 Input / output unit 12 Character recognition unit 13 Learning unit 14 Character processing unit MA 3rd model MB 1st model MC 2nd model

Claims (11)

A first model, which is a convolutional neural network in which a character image containing one or more characters is input and the feature amount of the character image is output,
A second model, which is a recurrent neural network in which the feature amount output by the first model is input one or more times and character information indicating characters included in the character image is output,
Using the first model and the second model, a character processing unit that outputs character information indicating characters included in the character image, and a character processing unit.
A character recognition device characterized by being equipped with. The first aspect of claim 1, wherein the character processing unit inputs character information indicating characters output by the second model in the previous time together with the feature amount to the second model. Character recognition device. It is understood that the first model and the second model are trained by using teacher data including a character image including one or more characters and character information indicating characters included in the character image. The character recognition device according to claim 1 or 2. The character recognition device according to claim 3, further comprising a learning unit for learning the first model and the second model using the teacher data. It further comprises a third model, which is a convolutional neural network into which an image is input and outputs area information indicating the area in which characters exist in the image.
The character processing unit cuts out a region indicated by the area information output by the third model from the image, and inputs the cut out region as the character image into the first model. The character recognition device according to any one of 1 to 4. 5. The third model is characterized in that it is trained using an image including a region in which characters exist and teacher data including the coordinates of the four corners of the region in which the characters exist in the image. The character recognition device described in. The character recognition device according to any one of claims 1 to 6, wherein the character image includes a plurality of characters arranged in a plurality of stages. The character recognition device according to any one of claims 1 to 7, wherein the character image is a photographed image including a license plate or a partial image of the photographed image. With the shooting department
The character recognition device according to any one of claims 1 to 8 is provided.
The character processing unit of the character recognition device is a photographing device that outputs the character information by using a photographed image taken by the photographing unit or a partial image of the photographed image as the character image. A character recognition method using the first model and the second model.
The first model is a convolutional neural network in which a character image containing one or more characters is input and the feature amount of the character image is output.
The second model is a recurrent neural network in which the feature amount output by the first model is input one or more times and character information indicating characters included in the character image is output.
A character recognition method comprising the character processing step of outputting character information indicating a character included in the character image by using the first model and the second model. The character recognition program for operating a computer as the character recognition device according to claim 1, wherein the first model, the second model, and character recognition for operating the computer as the character processing unit. program.

Images