CN114596571A - Intelligent lens-free character recognition system - Google Patents

Abstract

The invention discloses an intelligent lensless character recognition system. The system comprises an optical module and a calculation imaging and intelligent character positioning recognition module, wherein the optical module consists of a mask plate with adjustable amplitude and a sensor, and the transmission light amplitude distribution of the mask plate is modeled into a two-dimensional convolution layer and can be optimized as a parameter; the calculation imaging and character positioning identification module comprises a calculation imaging model, a character positioning model and a character identification model, input data is original data obtained on a sensor after passing through the optical module, the original data is output to be a text form of predicted characters, and meanwhile, the light transmission amplitude distribution of a mask plate in the optical module and the parameters of a calculation imaging network are optimized through result feedback. The invention realizes the software and hardware integrated lens-free imaging and character recognition deep learning model optimization, improves the accuracy of character positioning and character recognition under the condition of no lens, and each module of the system has universality and strong practical applicability.

Description

An Intelligent Lensless Character Recognition System

technical field

The invention belongs to the field of lensless imaging, in particular to an intelligent lensless character recognition system.

Background technique

With the rapid development and application of vision tasks, cameras are integrated on various hardware devices. Some application scenarios have strict requirements on the size of the camera, and a lensless camera is an imaging system that uses a thin reticle to replace the lens, so the size of the camera can be greatly reduced.

Compared with a camera with a lens, a lensless camera needs to perform computational imaging on the data collected on the sensor to restore the image, but the image reconstructed based on the lensless has the shortcomings of blur and resolution, which makes it unable to perform many visual tasks. Research on non-single-character text detection and recognition based on lensless.

Therefore, a lensless character recognition system is required.

SUMMARY OF THE INVENTION

Aiming at the situation that the current lensless imaging technology is not applied to character location and recognition of non-single letters due to poor imaging quality, the present invention provides a lensless-based character location and recognition system. The recognition accuracy is high and the system method has generality.

The technical scheme adopted in the present invention is as follows:

The intelligent lensless character recognition system of the present invention includes an optical module and a computational imaging and character positioning recognition module. The optical module is mainly composed of an adjustable amplitude mask and an optical sensor placed in parallel. The target to be recognized is placed in front of the optical module, and the target to be recognized After the light emitted by the target is scattered by the adjustable amplitude mask, it is projected on the plane of the optical sensor to form a projected image (raw data), and the optical sensor transmits the projected image to the computational imaging and text positioning and recognition module;

The computational imaging and text recognition module includes a computational imaging model, a text positioning model and a text recognition model, and the three models are connected in series; the input of the computational imaging and text recognition module is the projection image obtained on the sensor after the optical module, and the output is the projection image The textual form of the text on the image.

The modulated amplitude mask is a binarized mask composed of k*k cells, and the value of each cell is 1 or 0, 1 means that light can pass, and 0 means that light cannot pass.

The projected image outputs the predicted reconstructed image through the computational imaging model; the text localization model processes the input reconstructed image to output the position of the text in the image; after the output result of the text localization model is input into the text recognition model, the text recognition result of the image is output;

During the training process of the computational imaging and text recognition module, only the computational imaging model participates in the training, and the parameters need to be updated. The text positioning model and text recognition model do not participate in the training.

The computational imaging model is a neural network of the encoder-decoder system, specifically U-NET; the text localization model adopts an arbitrary text localization model structure, specifically CTPN; the text recognition model adopts an arbitrary text recognition model structure, specifically CRNN.

The pattern on the modulated amplitude mask is displayed on a liquid crystal display, and the pattern on the mask is randomly generated or determined after optimization by training; the method for determining the pattern on the mask after optimization by training includes the following steps:

1) Model the imaging process of the target to be recognized and the optical module as a two-dimensional convolution layer, specifically:

m=w*o

Among them, w represents the amplitude distribution on the reticle, that is, the value distribution of the cells on the reticle; the coordinate system is constructed with the center point of the reticle as the origin, (i, j) is the coordinate of the center point of the cell on the reticle, w _{i, j} represents the value of the cell whose coordinates are (i, j) on the mask;

o represents the image scaled on the sensor plane when the target to be recognized does not pass through the reticle (that is, o represents the image scaled on the sensor plane when the target to be recognized passes through the aperture); the coordinate system is constructed with the center point of the sensor plane as the origin, ( x, y) represents the coordinate value of the pixel of the projected image on the sensor plane, o _{x, y} represents the pixel value at (x, y) of the sensor plane when the target to be recognized does not pass through the reticle; o _{x+i, y+j} represents the pixel value at (x+i, y+j) on the sensor plane;

m represents the image projected on the sensor plane after the target to be recognized passes through the reticle; m _{x, y} represents the pixel value at (x, y) of the sensor plane after the target to be recognized passes through the reticle;

k represents the number of rows or columns of cells on the reticle, i∈[1,k];

2) Binarize the two-dimensional convolutional layer to obtain a two-dimensional convolutional layer of a binary neural network. The results are as follows:

in,

Among them, w ^b represents the result of binarizing w;

Since the mask has only 0 and 1 values, we use a binary neural network for training, which uses the sign function to map two consecutive values to -1 or +1, then add 1 and divide by 2;

3) The parameter w ^b of the two-dimensional convolution layer of the binary neural network is used as a model parameter to be trained and optimized together with the computational imaging and text location recognition modules;

3.1) During the training process, the pattern of the mask is randomly initialized through circuit adjustment, and the result of random initialization is used as the initial parameter of the convolutional layer of the binary neural network;

3.2) Training of the forward propagation process of the system: fix the target to be recognized, measure the projected image of the target to be recognized on the plane of the optical sensor after passing through the mask in the real physical scene, and use it as computational imaging and text positioning recognition module input;

Training of the backpropagation process: Calculate the loss function Loss of the predicted image output by the imaging and text location recognition module and the real image label, backpropagate the loss function Loss to the convolution layer of the binary neural network, and update the convolution of the binary neural network. layer parameter w ^b , and modulates the adjustable mask according to the updated parameter w ^b , and the modulation result is used as the mask pattern in the forward propagation process of the model in the next round of training;

3.3) The mask pattern obtained after training is the optimized result.

The cell size of the modulated reticle is the same as the pixel size on the sensor plane; the distance d1 between the target to be identified and the modulated amplitude reticle is much greater than the distance between the modulated amplitude reticle and the optical sensor. The distance d2, d1>100d2; therefore, the amplitude distribution on the reticle is approximately equal to the projection of the amplitude distribution on the reticle on the sensor plane.

The loss function Loss of the computational imaging and text location recognition module in the training process is:

Loss=a×Loss1+b×Loss2;

Among them, Loss1 is the error between the predicted image output by the computational imaging model and the real image label (target image to be recognized); Loss2 is the predicted text finally output by the computational imaging and text location recognition module and the real text label of the target to be recognized (to be recognized). Recognize the error between the text information on the target image); a and b are weights.

Effective benefits of the present invention:

The lensless character recognition system of the present invention can reduce the size limitation brought by the lens, so that it is more convenient for the camera to be integrated on other devices.

The invention realizes the optimization of the deep learning model of lensless imaging and character recognition integrated with software and hardware, improves the accuracy of character positioning and character recognition without lens, and each module of the system has universality and universality , has strong practical application.

Description of drawings

Figure 1 is the overall data flow of the present invention.

FIG. 2 is a schematic diagram of an optical module in the present invention.

FIG. 3 is a schematic diagram of a computational imaging and character location recognition module in the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

As shown in FIG. 1 , the lensless character recognition system of the present invention includes an optical module, a computational imaging and character location recognition module. The target to be recognized obtains the original data through the optical module, and the original data obtains the text form of the text through the computational imaging and text positioning and recognition module.

The optical module consists of a modulated amplitude mask and a sensor, wherein the modulated amplitude mask is a binarized mask, which is divided into k*k cells, and the value of the cell is 1 or 0, 1 It means that the light can pass through, and 0 means that the light cannot pass through. It is placed in front of the optical sensor, and the two are placed in parallel. The object is placed in front of the mask. After the light emitted by the object is scattered by the adjustable amplitude mask, a specific projection image is displayed. The (raw data) is projected on the plane where the sensor is located, recorded by the sensor and then transmitted to the computational imaging and text positioning and recognition module. The pattern of the modulated amplitude mask can be controlled in real time through the circuit, and can be displayed on a liquid crystal display. The pattern on the mask can be randomly generated, or the mask can be updated through training optimization; the mask obtained through training optimization The version can make the recognition effect of the computational imaging and text positioning recognition module better. When the reticle pattern in the optical module is fixed, without optimizing the reticle, the fixed reticle pattern can still be used to obtain the text through the original data obtained from the sensor, through the computational imaging and text positioning recognition module. location and identification results.

As shown in Figure 2, the training method for optimizing the reticle pattern is as follows:

1) Model the process of the imaging target interacting with the reticle as a 2D convolutional layer

m=w*o

Among them, w represents the amplitude distribution on the reticle, that is, the value distribution of the cells on the reticle; the coordinate system is constructed with the center point of the reticle as the origin, (i, j) is the coordinate of the center point of the cell on the reticle, w _{i, j} represents the value of the cell whose coordinates are (i, j) on the mask;

o represents the image scaled on the sensor plane when the modulated amplitude mask is fully transparent and the target to be recognized does not pass through the mask; the coordinate system is constructed with the center point of the sensor plane as the origin, and (x, y) represents the image on the sensor plane The coordinate value of the pixel point, o _{x, y} represents the pixel value at (x, y) of the sensor plane when the target to be recognized does not pass through the reticle;

m represents the image projected on the sensor plane after the target to be recognized passes through the reticle; m _{x, y} represents the pixel value at (x, y) of the sensor plane after the target to be recognized passes through the reticle;

k represents the number of rows or columns of cells on the reticle, i∈[1,k].

Since the reticle cell pixel size is the same as the sensor pixel size, and d1>>d2 (d1 is greater than 100 times d2), w can be approximately equal to the projection of the amplitude distribution on the reticle on the sensor plane.

2) The parameter w of the two-dimensional convolution layer is used as a model parameter to train and optimize together with the subsequent computational imaging and text location recognition modules.

2.1) Binarize the two-dimensional convolutional layer to obtain a binary neural network:

Among them, w ^b represents the result of binarizing w.

Since the reticle has only 0 and 1 values, we train it using a binary neural network that maps w to -1 or +1 using the sign function, then adds 1 and divides by 2.

2.2) During the training process, the mask pattern is randomly initialized by circuit adjustment, and this value is used as the initial parameter of the convolutional layer of the binary neural network.

In the process of forward propagation of the system: fix the target to be measured, measure the raw data obtained on the sensor after passing through the mask in the real physical scene, and use it as the input of the computational imaging and text positioning and recognition module,

After calculating the imaging and text positioning recognition module, the error with the real label is obtained,

In the process of gradient backpropagation: calculate the error between the output of the imaging and text location recognition module and the real label, calculate the gradient and backpropagate, backpropagate the error to the convolutional layer of the binary neural network, and update the volume of the binary neural network Layer, and modulate the adjustable mask according to the updated weight results, as the mask pattern in the forward propagation process of the model in the next round of training.

After training, fix the reticle pattern.

As shown in FIG. 3 , the computational imaging and character recognition module includes a computational imaging model, a character positioning model and a character recognition model. During use, the module is serialized by three models, the input data is the original data obtained on the sensor after passing through the optical module, and the output is the text form of the predicted text. During the training process, only the calculation of the imaging model needs to update the parameters, and the text localization model and the text recognition model do not participate in the training.

A computational imaging model is a deep learning-based imaging method that computes a predicted reconstructed image (model output) from raw data obtained on the sensor (model input). The computational imaging model is a neural network with an encoder-decoder structure, and specifically a network with a U-NET structure can be used. In the training process, the image containing letters and numbers is used as the target to be recognized, and the raw data obtained on the sensor after passing through the optical module is used as the model input. The training loss function consists of two parts: 1) The target image to be recognized (real image label) ) and the error Loss1 between the predicted image output by the model; 2) input the predicted reconstructed image into the text localization model and the text recognition model followed by, obtain the predicted text, calculate the predicted text and the text label of the target to be recognized. Error Loss2. Loss function Loss=a×Loss1+b×Loss2.

Gradient back-propagation: The loss function calculates the gradient back to update the computational imaging model and the two-dimensional convolutional layer described above, and finally obtains the corresponding trained computational imaging model.

The text localization model is a neural network model. The input data is an image, and the output data is where the text is located. The model can use any text positioning model structure. The specific structure can be used such as CTPN: the image passes through the VGG16 network, and a 3*3 sliding window is calculated in each line of the last convolutional layer of the network, and the results are connected through the BLSTM structure. Finally, a fully connected layer is input, and the output is the predicted coordinates and confidence score. And the model is trained and does not participate in the back propagation of the loss function to update the parameters in the system.

The text recognition model is a neural network model. The input data is an alphanumeric image and the output is the result of word recognition. The model can use any text recognition model structure, such as CRNN: the image first goes through several convolutional layers, extracts the feature sequence, enters the BLSTM, and finally outputs the score of the predicted letter. And the model is trained, and does not participate in the back propagation of the loss function to update the parameters in the system.

Claims (7)

1. An intelligent lens-free character recognition system, characterized in that it comprises an optical module and a computational imaging and character positioning recognition module, the optical module is mainly composed of a modulated amplitude mask plate and an optical sensor placed in parallel, and the target to be identified is placed in the In front of the optical module, after the light emitted by the target to be identified is scattered by the adjustable amplitude mask, a projected image is projected on the plane of the optical sensor, and the optical sensor transmits the projected image to the computational imaging and text positioning and recognition module; The pattern on the adjustable amplitude mask is displayed on the liquid crystal display, and the pattern on the mask is randomly generated or determined after training and optimization; The computational imaging and text recognition module includes a computational imaging model, a text positioning model and a text recognition model, and the three models are connected in series; the input of the computational imaging and text recognition module is the projection image obtained on the sensor after the optical module, and the output is the projection image The textual form of the text on the image. 2. An intelligent lensless character recognition system according to claim 1, wherein the modulated amplitude mask is a binarized mask consisting of k*k cells, each The value of the cell is 1 or 0, 1 means light can pass through, 0 means light can't pass through. 3. An intelligent lensless character recognition system according to claim 1, wherein the projected image outputs a predicted reconstructed image through a computational imaging model; the character localization model processes the input reconstructed image, and outputs a position; after inputting the output result of the text positioning model into the text recognition model, the text recognition result of the image is output; During the training process of the computational imaging and text recognition module, only the computational imaging model participates in the training, and the text localization model and text recognition model do not participate in the training. 4. a kind of intelligent lensless character recognition system according to claim 3, is characterized in that, the computational imaging model is the neural network of the encoder-decoder system, specifically adopts U-NET; Character location model adopts arbitrary character location model Structure, specifically using CTPN; text recognition model using arbitrary text recognition model structure, specifically using CRNN. 5. A kind of intelligent lensless character recognition system according to claim 1, is characterized in that, the method for determining reticle pattern after optimization by training comprises the following steps: 1) Model the imaging process of the target to be recognized and the optical module as a two-dimensional convolution layer, specifically: m=w*o

Among them, w represents the amplitude distribution on the reticle, that is, the value distribution of the cells on the reticle; the coordinate system is constructed with the center point of the reticle as the origin, (i, j) is the coordinate of the center point of the cell on the reticle, w _{i, j} represents the value of the cell whose coordinates are (i, j) on the mask; o represents the image scaled on the sensor plane when the target to be recognized does not pass through the reticle; the coordinate system is constructed with the center point of the sensor plane as the origin, (x, y) represents the coordinate value of the pixel of the projected image on the sensor plane, o _{x, y} represents the pixel value at (x, y) on the sensor plane when the target to be recognized does not pass through the reticle; o _{x+i, y+j} represents the pixel value at (x+i, y+j) on the sensor plane Pixel values; m represents the image projected on the sensor plane after the target to be recognized passes through the reticle; m _{x, y} represents the pixel value at (x, y) of the sensor plane after the target to be recognized passes through the reticle; k represents the number of rows or columns of cells on the reticle, i∈[1,k]; 2) Binarize the two-dimensional convolutional layer to obtain a two-dimensional convolutional layer of a binary neural network. The results are as follows:

in,

Among them, w ^b represents the result of binarizing w; 3) The parameter w ^b of the two-dimensional convolution layer of the binary neural network is used as a model parameter to be trained and optimized together with the computational imaging and text location recognition modules; 3.1) During the training process, the pattern of the mask is randomly initialized through circuit adjustment, and the result of random initialization is used as the initial parameter of the convolutional layer of the binary neural network; 3.2) Training of the forward propagation process of the system: fix the target to be recognized, measure the projected image of the target to be recognized on the plane of the optical sensor after passing through the mask in the real physical scene, and use it as computational imaging and text positioning recognition module input; Training of the backpropagation process: Calculate the loss function Loss of the predicted image output by the imaging and text location recognition module and the real image label, backpropagate the loss function Loss to the convolution layer of the binary neural network, and update the convolution of the binary neural network. layer parameter w ^b , and modulates the adjustable mask according to the updated parameter w ^b , and the modulation result is used as the mask pattern in the forward propagation process of the model in the next round of training; 3.3) The mask pattern obtained after training is the optimized result. 6. An intelligent lensless character recognition system according to claim 5, wherein the cell size of the modulated reticle is the same as the pixel size on the sensor plane; The distance d1 between the modulated amplitude masks is much larger than the distance d2 between the modulated amplitude masks and the optical sensor, d1>100d2; therefore, the amplitude distribution on the reticle is approximately equal to the amplitude distribution on the reticle on the sensor plane. projection on. 7. a kind of intelligent lensless character recognition system according to any one of claim 3 and 5, is characterized in that, the loss function Loss of described computational imaging and character location recognition module in training process is: Loss=a×Loss1+b×Loss2; Among them, Loss1 is the error between the predicted image output by the computational imaging model and the real image label; Loss2 is the error between the predicted text finally output by the computational imaging and text positioning recognition module and the real text label of the target to be recognized; a and b for weight.

Images