RU2792743C1 - Identification of writing systems used in documents - Google Patents

Abstract

FIELD: computer systems.

SUBSTANCE: present invention relates primarily to computer systems and, in particular, to systems and methods for identifying writing systems used in documents. The technical result consists in increasing the efficiency of optical character recognition by applying a neural network to image fragments, reducing each image fragment to the size of the input layer of the network, in order to reduce the requirements for computing resources. The technical result is achieved due to the following. The method includes: obtaining an image of a document; splitting the image into fragments; generating probability vectors by the means of a neural network containing a plurality of numerical elements, and each numerical element reflects the probability of an image fragment containing text associated with the corresponding writing system; calculating an aggregated probability vector, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system; and given the determination that the maximum numerical element of the aggregated probability vector exceeds a certain threshold value, it is concluded that the document image contains one or more characters associated with the corresponding writing system.

EFFECT: increase in the efficiency of optical character recognition by applying a neural network to image fragments.

20 cl, 5 dwg

Description

FIELD OF TECHNOLOGY

[0001] The present invention relates primarily to computer systems and, in particular, to systems and methods for identifying writing systems used in documents.

BACKGROUND OF THE INVENTION

[0002] A writing system is a method of visualizing verbal communication based on a script and a set of rules governing its use. Writing systems can be divided into broad categories such as alphabets, syllabics, or logographies, although some writing systems may have attributes from more than one category. In alphabets, each character represents the corresponding speech sounds. Abjads do not include vowels. In abjads or alphasyllabic alphabets, each character represents a pair of vowel and consonant. In syllabaries, each character represents a syllable or mora. In logographs, each character represents a semantic unit, such as a morpheme. Some writing systems also include a special set of characters known as punctuation, which is used to make it easier to interpret and express the nuances of a message's meaning.

DISCLOSURE OF THE INVENTION

[0003] In accordance with one or more aspects of the present invention, an exemplary method for identifying a writing system used in a document includes: obtaining by a computer system an image of the document; splitting the document image into a plurality of image fragments; generating a plurality of probability vectors by means of a neural network processing a plurality of image fragments, wherein each probability vector of the plurality of probability vectors is created by processing the corresponding fragments of the image and contains a plurality of numerical elements, and each numerical element of the plurality of numerical elements reflects the probability of an image fragment containing text, associated with a writing system that is identified by the index of the numeric element in the corresponding probability vector; calculating an aggregated probability vector by aggregating a plurality of probability vectors, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system, which is identified by an index of the numeric element within the aggregated probability vector; and given the determination that the maximum numeric element of the aggregated probability vector exceeds a predetermined threshold value, it is concluded that the document image contains one or more characters associated with the writing system identified by the index of the maximum numeric element within the aggregated probability vector .

[0004] In accordance with one or more aspects of the present invention, the exemplary system comprises a storage device and a processor coupled to the storage device. The processor is configured to: receive an image of the document; splitting the document image into a plurality of image fragments; generating a plurality of probability vectors by means of a neural network processing a plurality of image tiles, each probability vector of the plurality of probability vectors being generated by processing the respective image fragments and containing a plurality of numeric elements, each numeric element of the plurality of numeric elements reflecting the probability of the image fragment containing the text associated with a writing system that is identified by the index of the numeric element in the corresponding probability vector; calculating an aggregated probability vector by aggregating a plurality of probability vectors, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system, which is identified by an index of the numeric element within the aggregated probability vector; and given the determination that the maximum numeric element of the aggregated probability vector exceeds a predetermined threshold, it is concluded that the document image contains one or more characters associated with the writing system identified by the index of the maximum numeric element within the aggregated probability vector .

[0005] In accordance with one or more aspects of the present invention, the exemplary computer-readable non-volatile storage medium contains executable instructions that, when executed by a computer system, cause the computer system to perform the following operations: obtaining an image of a document; splitting the document image into a plurality of image fragments; generating a plurality of probability vectors by means of a neural network processing a plurality of image fragments, wherein each probability vector of the plurality of probability vectors is created by processing the corresponding fragments of the image and contains a plurality of numerical elements, and each numerical element of the plurality of numerical elements reflects the probability of an image fragment containing text, associated with a writing system that is identified by the index of the numeric element in the corresponding probability vector; calculating an aggregated probability vector by aggregating a plurality of probability vectors, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system, which is identified by an index of the numeric element within the aggregated probability vector; and given the determination that the maximum numeric element of the aggregated probability vector exceeds a predetermined threshold value, it is concluded that the document image contains one or more characters associated with the writing system identified by the index of the maximum numeric element within the aggregated probability vector .

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present invention is illustrated by way of examples and not by way of limitation, and can be more fully understood by referring to the following detailed description when considered in connection with the figures, in which:

[0007] FIG. 1 is a flow chart of an exemplary method for identifying a writing system used in a document, in accordance with one or more aspects of the present invention;

[0008] FIG. 2 is a flow chart of another exemplary method for identifying a writing system used in a document, in accordance with one or more aspects of the present invention;

[0009] FIG. 3 is a flow diagram of yet another exemplary method for identifying a writing system used in a document, in accordance with one or more aspects of the present invention;

[00010] FIG. 4 schematically depicts an example of a neural network architecture that can be used by systems and methods within the scope of the present invention;

[00011] FIG. 5 depicts a diagram of the components of an exemplary computer system that may be used to implement the methods described herein.

DETAILED DESCRIPTION

[00012] This document describes methods and systems for identifying writing systems used in document images. The systems and methods within the scope of the present invention process indexed images on various media (eg, printed or handwritten paper documents, banners, posters, signs, billboards, and/or other physical objects containing visible graphemes on one or more surfaces). "Grapheme" here refers to the elementary unit of the alphabet. A grapheme can be represented, for example, by a logogram meaning a word or a morpheme, a syllabic symbol meaning a syllable, or letter symbols meaning a phoneme.

[00013] The systems and methods of the present invention can recognize multiple writing systems, including, for example, Latin/Cyrillic alphabets, Korean alphabet, Chinese/Japanese logography, and/or Arabic Abjad. In some embodiments, identifying the writing system used in a document is a prerequisite for performing optical character recognition (OCR) on an image of the document. In an illustrative example, the systems and methods described herein may be used to determine values for one or more OCR mobile application parameters, including writing system and/or image orientation.

[00014] The method of identifying the writing system used for the input image in the example involves dividing the original image into a predetermined number of rectangular fragments (for example, into a regular 3 x 3 grid consisting of nine rectangular fragments, a regular 4 x 4 grid consisting of 16 rectangular tiles, a regular 5 x 5 grid of 25 rectangular tiles, etc.). After being compressed to a predetermined size (e.g. 224 x 224 pixels), the rectangular tiles are passed to a neural network that applies a set of functional transformations to a set of input data (e.g. image pixels) and then uses the transformed data for pattern recognition. The neural network creates a numeric vector that includes N+1 values (where N is the number of writing systems recognized by the neural network), so that the i-th elements of the vector reflect the probability of the image fragment that contains the characters of the i-th writing system, and the last element of the vector reflects the probability that there are no recognizable characters on the image fragment. All vectors created by the neural network for the set of input image fragments are then added together, and then the resulting vector is normalized. If the maximum value among all elements of the resulting normalized vector exceeds a predetermined threshold value, it is concluded that the image contains the characters of the writing system identified by the index of the vector's maximum element. Otherwise, the two writing systems with the two highest scores are returned.

[00015] Thus, the system and methods of the present invention improve the performance of optical character recognition by applying a neural network to image fragments rather than the entire image, and reducing each image fragment to the size of the input layer of the network, resulting in a significant reduction in computational resource requirements. used by the neural network compared to the basic scenario of passing the entire image to the neural network.

[00016] The systems and methods described herein may be implemented using hardware (e.g., general purpose and/or special processing devices and/or other devices and associated circuitry), software (e.g., instructions executed by the processing device) or their combinations. Various aspects of the above methods and systems are described in detail herein below by way of example and not limitation.

[00017] FIG. 1 depicts a flow diagram of an exemplary method 100 for identifying a writing system used in a document, in accordance with one or more aspects of the present invention. Method 100 and/or each of its individual functions, programs, subroutines, or operations may be executed by one or more processors of a computer system (in particular, computer system 700 depicted in FIG. 5) implementing the method. In some embodiments, method 100 may be performed with a single processing thread. In an alternative embodiment, method 100 may be executed by two or more processing threads, each of which performs one or more separate functions, programs, subroutines, or operations related to the method. In an illustrative example, the processing threads implementing method 100 may be synchronized (eg, using semaphores, critical sections, and/or other thread synchronization mechanisms). In an alternative embodiment, the processing threads implementing method 100 may execute asynchronously with respect to each other. Therefore, although in FIG. 1 and in its associated descriptions, the operations of method 100 are performed in a specific order, in various embodiments of the method, at least some of the described operations may be performed in parallel and/or in an arbitrary order.

[00018] In step 110, the computer system implementing the method obtains an input document image. In some embodiments, prior to transmission to method 100, the input document image may be pre-processed, such as by cropping the original image, scaling the original image, and/or converting the original image to a grayscale or black and white image.

[00019] In step 120, the computer system partitions the original image into a predetermined number of rectangular tiles using a rectangular grid. The required number of fragments can be calculated tentatively to provide the desired balance between computational complexity and accuracy of the result. In an illustrative example, the original image may be divided into a regular 3 x 3 grid consisting of nine rectangular tiles. In other examples, various other grids may be used. The resulting rectangular tiles are compressed to a predetermined size (eg, 224 x 224 pixels) that can be predicted to provide the desired balance between computational complexity and result precision. In some embodiments, the fragments may be further pre-processed, such as by normalizing the brightness of a pixel, to bring it to a predetermined range, such as (-1, 1).

[00020] At step 130, the computer system sequentially feeds the pre-processed image fragments to the neural network, which creates a numeric vector including N+1 values (where N is the number of writing systems recognized by the neural network), so that each element of the vector reflects the probability that the image fragment reflects the characters of the writing system identified by the vector element index (i.e., the i-th elements of the vector reflect the probability of the image fragment that represents the characters of the i-th writing system). The last element of the vector reflects the probability that there are no recognizable characters on the image fragment. Thus, for each fragment of the input image, the neural network will create the corresponding probability vector of the fragment of the image containing the characters of the corresponding writing system.

[00021] In some embodiments, the implementation of the neural network may be represented by a convolutional neural network corresponding to a certain architecture, as described in more detail by reference to FIG. 4.

[00022] In some embodiments, a second level classifier may be introduced that receives the spatial feature map generated by the neural network in step 130 and learns to identify the writing system based on the spatial feature map.

[00023] In some embodiments, the neural network at step 130 and/or the second level classifier can be trained to perform a multi-feature classification that will produce two or more writing systems with corresponding characters present in the input document image.

[00024] In step 140, the computer system aggregates the vectors generated by the neural network for the set of input image tiles such that the resulting vector has dimension N+1 and the i-th component of the resulting vector is the sum of the i-th components of all vectors generated by the neural network for a set of input image fragments:

[00025]

- j-ый компонент вектора суммы, а[00026] where

is the j-th component of the sum vector, and

- j-ый компонент вектора вероятности, созданного нейронной сетью для i-го фрагмента изображения.[00027]

- j-th component of the probability vector created by the neural network for the i-th fragment of the image.

[00028] The resulting vector can then be normalized, for example, by dividing each element of the vector by the square root of the L2 norm of the vector:

[00029]

- i-ый компонент нормализованного получившегося вектора,[00030] where

- i-th component of the normalized resulting vector,

- i-ый элемент вектора, а[00031]

- i-th element of the vector, and

- норма L2 вектора.[00032]

is the L2 norm of the vector.

(т. е. максимальное значение среди всех элементов получившегося нормализованного вектора) превышает заранее определенное пороговое значение, компьютерная система на этапе операции 160 делает вывод о том, что изображение содержит символы системы письма, идентифицированной индексом максимального (max) элемента вектора. В противном случае, если максимальное значение ниже или равно заранее определенному пороговому значению, заранее определенное количество K (например, две) систем письма-кандидатов, набравших K наибольших значений, возвращается на этап операции 170, и способ завершается. Предварительно определенное пороговое значение может быть вычислено предположительно в качестве обеспечения желаемой точности распознавания.[00033] Given the determination at operation 150 that

(i.e., the maximum value among all elements of the resulting normalized vector) exceeds a predetermined threshold value, the computer system concludes at step 160 that the image contains the characters of the writing system identified by the index of the maximum (max) element of the vector. Otherwise, if the maximum value is less than or equal to the predetermined threshold value, the predetermined number K (eg, two) of the highest K writing system candidates returns to step 170 and the method ends. A predetermined threshold value may be calculated as a guess as to provide the desired recognition accuracy.

[00034] As noted above in this document, the number of fragments into which the original image is divided may be arbitrary and may depend on the type and / or other known parameters of the input document. In some embodiments, method 100 can be adapted to process input documents that contain text chunks in different font sizes (eg, headings, plain text, subscripts, superscripts, etc.). In order for the network to give a reliable answer for a given image fragment, this fragment must contain several lines of text (i.e. contain a sufficient number of graphemes), while reducing the fragment to the size of the network input layer (for example, 224 x 224), the text must remain large enough, so that the writing system can be identified. This consideration can be used to determine the range of font sizes that will be used to train the neural network.

[00035] Training the neural network may include activating the neural network for each input value in the training data set. The value of the loss function can be computed based on the observed output of a certain layer of the neural network and the desired output given in the training dataset. The error can be passed back to the previous layers of the neural network, in which the edge weights and/or other network parameters can be adjusted accordingly. This process may be repeated until the value of the loss function stabilizes around a certain value or falls below a predetermined threshold.

[00036] In order for the trained network to produce reliable results for arbitrary font sizes, including font sizes outside the font size range on which the neural network was trained, method 100 can be modified to use scaled image tiles. To be able to handle a wide range of font sizes, the neural network can be trained to produce a new category of images corresponding to text that is too small to accurately recognize the writing system.

[00037] Thus, after processing the original image, reduced to a size corresponding to the size of the input layer of the network, the neural network can receive several fragments of the image, which can be created, for example, by applying a predefined grid (for example, a regular 2 x 2 grid, consisting of four rectangular fragments) to the original image. The grid can be recursively applied to image fragments that have been characterized by the network as containing text that is too small to provide accurate recognition, until all fragments contain characters large enough to allow the neural network to recognize the writing system, or until until a predetermined minimum image fragment size is reached.

[00038] FIG. 2 depicts a flowchart of an exemplary method 200 for identifying a writing system used in a document, in accordance with one or more aspects of the present invention. The method 200 and/or each of its individual functions, programs, subroutines, or operations may be executed by one or more processors of the computer system (in particular, computer system 700 depicted in FIG. 5) implementing the method. In some embodiments, method 200 may be performed with a single processing thread. In an alternative embodiment, method 200 may be executed by two or more processing threads, each of which performs one or more separate functions, programs, subroutines, or operations related to the method. In an illustrative example, the processing threads implementing method 200 may be synchronized (eg, using semaphores, critical sections, and/or other thread synchronization mechanisms). In an alternative embodiment, the processing threads implementing method 200 may execute asynchronously with respect to each other. Therefore, although in FIG. 2 and associated descriptions, the steps of method 200 are performed in a specific order, in various embodiments of the method, at least some of the described steps may be performed in parallel and/or in an arbitrary order.

[00039] In step 210, the computer system implementing the method obtains an input document image. In some embodiments, prior to transmission to method 200, the input document image may be pre-processed, such as by cropping the original image, scaling the original image, and/or converting the original image to a grayscale or black and white image.

[00040] In step 220, the computer system partitions the original image into a predetermined number of rectangular tiles using a rectangular grid, as described in more detail hereinbefore. The resulting rectangular tiles are compressed to a predetermined size (eg, 224 x 224 pixels) that can be predicted to provide the desired balance between computational complexity and result precision. In some embodiments, the fragments may be further pre-processed, such as by normalizing the brightness of a pixel, to bring it to a predetermined range, such as (-1, 1).

[00041] At step 230, the computer system sequentially feeds the pre-processed sub-images to the neural network, which creates, for each input image sub-tile, a corresponding sub-image probability vector containing the characters of the corresponding writing system, as described in more detail hereinbefore. In some embodiments, the implementation of the neural network may be represented by a convolutional neural network corresponding to a certain architecture, as described in more detail by reference to FIG. 4.

[00042] Given the determination at step 240 that the neural network has not classified image tiles as containing text that is too small to provide accurate recognition, or that a predetermined minimum image tiles size has been reached, processing continues at block 260; otherwise, the method proceeds to operation 250.

[00043] At step 250, the computer system further sub-segments the image fragments that the network has characterized as containing text that is too small to provide accurate recognition, and the method returns to block 230.

[00044] In step 260, the computer system aggregates the vectors generated by the neural network for the set of input image tiles such that the resulting vector has dimension N+1 and the i-th component of the resulting vector is the sum of the i-th components of all vectors generated by the neural network for a set of input image fragments. The resulting vector can then be normalized, as described in more detail above in this document.

[00045] Given the determination at step 270 that the maximum value among all elements of the normalized resulting vector exceeds a predetermined threshold value, the computer system at step 280 concludes that the image contains characters of the writing system identified by the index of the maximum element of the vector . Otherwise, if the maximum value is less than or equal to the predetermined threshold value, the predetermined number K of candidate writing systems that scored the highest K values returns to step 290 and the method ends.

[00046] In some embodiments, in order to reduce processing time, only a subset of all image fragments (eg, "regions of interest" (ROI)) may be fed into the neural network for further processing. In order to increase the overall efficiency of the method, the selected sub-images may not necessarily cover the entire image and/or may be selected in a predetermined order, such as a checkerboard pattern. In another illustrative example, a neural network for identifying a writing system may operate on a complete set of image tiles.

[00047] The training dataset used to train the area of interest identification neural network may subsequently consist of at least a subset of all image tiles and their respective inference accuracy values exhibited by the writing system identification neural network. The image fragments can then be sorted in the reverse order of the accuracy of the output performed by the writing system's identification neural network. A predetermined number of image tiles can then be selected, corresponding to the highest precision of the inference performed on those tiles by the writing system identification neural network, and used to train the area of interest identification neural network.

[00048] In another illustrative example, a planned reference data approach can be implemented, including storing in a segmentation data structure (eg, a heat map) the respective probabilities for each segment that were obtained by the classifier. The segmentation data structure can then be used to train a neural network to identify the writing system by selecting a predetermined number (eg, N) of the best tiles (eg, by selecting M maximum size superpixels from a heatmap) to run the classifier. To improve efficiency, the heatmap can be sorted by counting, which exhibits a computational complexity of O(N), where N is the number of superpixels in the heatmap. The classifier can then be retrained to match exactly those selected chunks, which can be achieved by weighting the loss function for each chunk. After selecting a predetermined number of fragments (areas of interest), a retrained classifier may be run on them to obtain a predetermined number of probabilities which may be aggregated as described herein above.

[00049] In some embodiments, an area of interest identification neural network can be trained to handle a wide range of font sizes. A writing system identification neural network can be trained to identify a writing system for documents using a predetermined font size range (eg, font sizes from X to 2X). The writing system identification neural network can then be run on image fragments of different scales. The training dataset used to train the region-of-interest identification neural network may subsequently consist of a subset of all image tiles and their respective inference accuracy values exhibited by the writing system identification neural network.

[00050] FIG. 3 depicts a flow chart of an exemplary method 300 for identifying a writing system used in a document, in accordance with one or more aspects of the present invention. The method 300 and/or each of its individual functions, programs, subroutines, or operations may be executed by one or more processors of the computer system (in particular, computer system 700 depicted in FIG. 5) implementing the method. In some embodiments, method 300 may be performed with a single processing thread. In an alternative embodiment, method 300 may be executed by two or more processing threads, each of which performs one or more separate functions, programs, subroutines, or operations related to the method. In an illustrative example, the processing threads implementing method 300 may be synchronized (eg, using semaphores, critical sections, and/or other thread synchronization mechanisms). In an alternative embodiment, the processing threads implementing method 300 may execute asynchronously with respect to each other. Therefore, although in FIG. 3 and in its associated descriptions, the operations of method 300 are performed in a specific order, in various embodiments of the method, at least some of the described operations may be performed in parallel and/or in an arbitrary order.

[00051] At step 310, the computer system implementing the method obtains an input document image. In some embodiments, the image of the input document may be pre-processed before being transmitted to the implementation of method 300, as described in more detail hereinbefore.

[00052] At step 320, the computer system partitions the original image into a predetermined number of rectangular tiles using a rectangular grid, as described in more detail hereinbefore.

[00053] At step 330, the computer system identifies regions of interest (ROI) among all image fragments. In some embodiments, region of interest identification may be performed by a dedicated region of interest identification neural network that may identify a subset including a predetermined number of image tiles, as described in more detail hereinbefore.

[00054] At step 340, the computer system sequentially feeds the pre-processed sub-images to the neural network, which creates, for each pre-processed input image sub-tile, a corresponding sub-image probability vector containing the characters of the corresponding writing system, as described in more detail hereinbefore. In some embodiments, the implementation of the neural network may be represented by a convolutional neural network corresponding to a certain architecture, as described in more detail by reference to FIG. 4.

[00055] In step 350, the computer system aggregates the vectors generated by the neural network for the set of input image tiles such that the resulting vector has dimension N+1 and the i-th component of the resulting vector is the sum of the i-th components of all vectors generated by the neural network for a set of input image fragments. The resulting vector can then be normalized, as described in more detail above in this document.

[00056] Given the determination at step 360 that the maximum value among all elements of the normalized resulting vector exceeds a predetermined threshold value, the computer system at step 370 concludes that the image contains characters of the writing system identified by the index of the maximum element of the vector . Otherwise, if the maximum value is less than or equal to the predetermined threshold value, the predetermined number K of candidate writing systems that scored the highest K values returns to step 380 and the method ends.

[00057] In addition, in some embodiments, a writing system identification neural network may be used to identify both the writing system and the spatial orientation of an input image. Accordingly, the neural network can be modified to include additional output that will produce a value describing the spatial orientation of the input image. In an illustrative example, a value can reflect one of four possible orientations: normal, rotated 90 degrees clockwise or counterclockwise, rotated 180 degrees, rotated 270 degrees clockwise or counterclockwise, and so on. Similar to identifying a writing system, spatial orientation can be determined with respect to a set of input image tiles and then aggregated over all tiles to determine the spatial orientation of the input image. In some embodiments, the spatial orientation of an image can be identified by a second level classifier that receives the spatial feature map generated by the neural network in step 130 and learns to identify the spatial orientation of the input image.

[00058] Similar to the data sets that are used to train writing system identification neural networks, the data set for training the network to recognize image orientation may consist of real and/or synthesized images. Such a training data set may include images that have different spatial orientations such that the image orientation is distributed either uniformly or based on the frequency of occurrence of document images having different orientations in a given volume of documents.

[00059] As noted above herein, a neural network implemented in accordance with aspects of the present invention may include multiple layers of different types. In an illustrative example, an input image may be acquired by an input layer and subsequently processed by a series of layers, including convolutional layers, merging layers, linear rectification unit (ReLU) layers, and/or fully connected layers, each of which may perform a specific OCR operation on the input image. . The output of a layer can be passed as input to one or more subsequent layers. Processing the original image by a convolutional neural network can iteratively apply each successive layer until each layer has performed its respective operation.

[00060] In some embodiments, a convolutional neural network may include interleaved convolutional and pooling layers. Each convolution layer can perform a convolution operation, which involves processing each pixel of the input image fragment with one or more filters (convolution matrices) and writing the result to the corresponding position in the output array. One or more convolution filters can be designed to detect a particular feature of an image by processing the input image and producing a corresponding feature map.

[00061] The output of the convolutional layer may be passed to the ReLU layer, which may apply a non-linear transformation (eg, an activation function that replaces negative numbers with zero) to process the output of the convolutional layer. The output of the ReLU layer may be passed to the pooling layer, which may perform a downsampling operation to reduce the resolution and size of the feature map. The output of the pooling layer can be passed to the next convolutional layer.

[00062] In some embodiments, writing system identification neural networks implemented in accordance with aspects of the present invention may be compatible with the MobileNet architecture, which is a family of general purpose machine vision neural networks designed for mobile devices to perform image classification, detection and/or other similar tasks. In an illustrative example, a neural network for writing system identification may follow the MobileNetv2 architecture, as schematically shown in FIG. 4. As shown in FIG. 4, an example neural network 400 may include two kinds of blocks. Block 402 is a one-step residual block. Block 404 is a step two block.

[00063] Each of blocks 402, 404 includes three layers: a point convolution layer 412A-412B, which is responsible for creating new features by computing linear combinations of input channels; a deep convolutional layer 414A-414B that performs lightweight filtering by applying one convolutional filter per input channel; and a second convolutional layer 416A-416B with no non-linearity. When the start and end feature maps are the same size (when the depth convolution step is one and the input and output channels are equal), a residual connection 418 is added to promote gradient flow during backpropagation.

[00064] In another illustrative example, a neural network for identifying a writing system may follow the MobileNetv3 architecture, all blocks of which are bottlenecks with compression and excitation mechanisms. A neural network can have four general layers for feature extraction, followed by three branches: a writing system identification branch, a spatial orientation identification branch, and a text size clustering branch. The latter can be used for post-processing and selection of connections to perform further classification of scaled text. Each branch may include a predetermined number of blocks (for example, three or four).

[00065] Although FIG. 4 and the accompanying description explain the specific number and types of layers of an exemplary convolutional neural network architecture, the convolutional neural networks used in various alternative embodiments may include any suitable number of convolutional layers, ReLU layers, pooling layers, and/or any other layers. The order of the layers, the number of layers, the number of filters, and/or any other parameter of the convolutional neural networks can be adjusted (eg, based on empirical data).

[00066] Neural networks used by systems and methods within the framework of the present invention can be trained based on training data sets, including real and/or synthesized images of text. Various image enhancement methods can be applied to synthesized images to achieve "photorealistic" image quality. Each image can include multiple lines of text in a particular language, which are displayed using a given font size. The language used in the image determines the writing system that is used as the label associated with the image fragment when training the neural network.

[00067] FIG. 5 depicts a diagram of the components of an exemplary computer system that may be used to implement the methods described herein. Computer system 700 may be connected to another computer system on a local area network, an intranet, an extranet, or the Internet. Computer system 700 may operate as a server or client computer system in a client-server network environment, or as a peer-to-peer computer system in a peer-to-peer (or distributed) network environment. Computer system 700 may be a personal computer (PC), a tablet, a set top box (STB), a personal digital assistant (PDA), a mobile phone, or any computer system capable of executing a set of instructions (serial or otherwise) that specifies operations to be performed. performed by such a computer system. Furthermore, while illustrative of only one computer system, the term "computer system" should also include any collection of computer systems that individually or collectively execute a set (or multiple sets) of instructions to implement any one or more of the methods described herein.

[00068] An exemplary computer system 700 includes a processor 702, main memory (memory) 704 (e.g., read only memory (ROM), dynamic random access memory (DRAM)) and storage device 718 that interact with each other. with a friend on the 730 bus.

[00069] The processor 702 may be one or more general purpose processing devices such as a microprocessor, a central processing unit, and the like. More specifically, the processor 702 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing microprocessor (RISC), a microprocessor with a very long instruction word (VLIW), a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processor 702 may also be one or more special purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, and the like. The processor 702 is configured to executing commands 726 to implement the methods described herein.

[00070] The computer system 700 may further include a network interface device 722, a visual display device (video display) 710, an alphanumeric input device 712 (e.g., a keyboard), and a touch input device 714.

[00071] Storage device 718 may include a computer-readable storage medium 724 that stores one or more sets of commands/instructions 726 that implement any one or more of the methods or functions described herein. Instructions 726 may also reside, in whole or at least in part, in main storage 704 and/or processor 702 while they are executed by computer system 700, main storage 704, and processor 702, also a computer-readable storage medium. Commands 726 may optionally be transmitted or received over network 716 via network interface device 722.

[00072] In an illustrative example, instructions 726 may include instructions for methods 100, 200, and/or 300 for identifying a writing system used in a document, implemented in accordance with one or more aspects of the present invention. Although the computer readable medium 724 is shown in the example of FIG. 5 as a single medium, the term "machine-readable storage medium" should be understood to include one or more media (eg, a centralized or distributed database and/or associated caches and servers) that store one or more instruction sets. The term "machine-readable medium" should also include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present invention. The term "computer-readable storage medium" accordingly includes, but is not limited to, solid-state storage devices, optical and magnetic media.

[00073] The methods, components, and characteristics described herein may be implemented as discrete hardware components, or may be integrated into the functionality of other hardware components such as ASICs, FPGAs, DSPs, or similar devices. In addition, the methods, components, and characteristics may be implemented as firmware modules or functional circuits in hardware devices. Also, the methods, components, and characteristics may be implemented in any combination of hardware devices and software components, or only in software.

[00074] Numerous details have been set forth in the above description. However, a person skilled in the art, who will be able to enjoy the benefits of the present invention, it will be obvious that aspects of the present invention can be implemented without such specific details. In some instances, well-known structures and devices are shown in block diagram form without detail so as not to obscure the subject matter of the present invention.

[00075] Some portions of the detailed description are presented as algorithms and symbolic representations of operations on data bits in computer storage. Such algorithmic descriptions and representations are tools used by data scientists to most effectively convey the content of their work to others in the field. Both within the framework of this document and in general, an algorithm is understood as a self-consistent sequence of operations leading to the desired result. These operations are stages that require physical manipulations with physical quantities. Typically, though not necessarily, such quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, or otherwise used. Sometimes, mainly for reasons of common practice, it is convenient to refer to such signals as bits, values, elements, graphemes, signs, terms, numbers, etc.

[00076] However, it should be remembered that all such and similar terms are to be associated with the corresponding physical quantities and are only convenient designations applied to such quantities. Unless specifically stated otherwise, as is evident from the following discussion, it should be understood that throughout the description of the invention, discussions using terms such as "definition", "calculation", "calculation", "counting", "obtaining", "identifying" , "modification", etc., refers to the actions and processes of a computer system or similar electronic computing device that uses and converts data represented in the form of physical (for example, electronic) quantities in the registers and memory of a computer system into other data, similarly represented in the form of physical quantities in the memory or registers of a computer system or other similar devices for storing, transmitting or displaying information.

[00077] The present invention also relates to a device for performing the operations described herein. This device may be specially designed for the required purpose, or it may consist of a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium such as any type of disk, including floppy disks, optical disks, compact disks and magnetic optical disks, read-only memory (ROM), random access memory (RAM), PROM, EEPROM, magnetic or optical boards or any type of media suitable for storing electronic commands.

[00078] It should be understood that the above description is for illustrative purposes and is not restrictive. Many other embodiments will be apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of the invention must be determined with reference to the appended claims along with the full scope of the equivalents to which such claims apply.

Claims (52)

1. A method for identifying a writing system, including: obtaining an image of the document by the computer system; splitting the document image into a plurality of image fragments; generating a plurality of probability vectors by means of a neural network processing a plurality of image fragments, wherein each probability vector of the plurality of probability vectors is created by processing the corresponding fragments of the image and contains a plurality of numeric elements, wherein each numeric element of the plurality of numeric elements reflects the probability of an image fragment containing text, associated with the writing system, which is identified by the index of the numeric element in the corresponding probability vector; calculating an aggregated probability vector by aggregating a plurality of probability vectors, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system, which is identified by an index of the numeric element within the aggregated probability vector; And given the determination that the maximum numeric element of the aggregated probability vector exceeds a predetermined threshold value, it is concluded that the document image contains one or more characters associated with the writing system, which is identified by the index of the maximum numeric element within the aggregated probability vector. 2. The method of claim 1, further comprising: given the determination that the maximum numerical element of the aggregated probability vector is less than or equal to a predetermined threshold value, it is concluded that the document image contains one or more characters associated with one of the following systems: the first writing system, which is identified by the first index the maximum numeric element within the aggregated probability vector, or a second writing system that is identified by the second index of the next largest numeric element within the aggregated probability vector. 3. The method of claim 1, further comprising: identifying a plurality of regions of interest (ROI) among the plurality of image fragments. 4. The method of claim 1, further comprising: normalization of the aggregated probability vector. 5. The method of claim. 1, wherein each sub-image of the plurality of sub-images is a rectangular sub-image of a predetermined size. 6. The method of claim 1, further comprising: recursively splitting into corresponding image sub-fragments of one or more image fragments, which are characterized by the neural network as containing text having a text size less than the minimum threshold size. 7. The method according to p. 1, further comprising: pre-processing of the document image. 8. The method of claim. 1, wherein splitting the document image into a plurality of image fragments further includes: converting each sub-image of the plurality of sub-images to a predetermined size. 9. The method of claim 1, further comprising: determining the spatial orientation of the document image using a neural network that processes a plurality of image fragments. 10. The method according to p. 1, further comprising: identifying, based on a predetermined order, a subset of the plurality of image fragments to be transmitted to the neural network. 11. A system for identifying a writing system, including: Memory device; a processor coupled to the storage device, the processor being configured to perform actions such as: getting an image of the document; splitting the document image into a plurality of image fragments; generating a plurality of probability vectors by means of a neural network processing a plurality of image fragments, wherein each probability vector of the plurality of probability vectors is created by processing the corresponding fragments of the image and contains a plurality of numeric elements, wherein each numeric element of the plurality of numeric elements reflects the probability of an image fragment containing text, associated with the writing system, which is identified by the index of the numeric element in the corresponding probability vector; calculating an aggregated probability vector by aggregating a plurality of probability vectors, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system, which is identified by an index of the numeric element within the aggregated probability vector; And given the determination that the maximum numeric element of the aggregated probability vector exceeds a predetermined threshold value, it is concluded that the document image contains one or more characters associated with the writing system, which is identified by the index of the maximum numeric element within the aggregated probability vector. 12. The system of claim 11, wherein the processor is further configured to: given the determination that the maximum numerical element of the aggregated probability vector is less than or equal to a predetermined threshold value, it is concluded that the document image contains one or more characters associated with one of the following systems: the first writing system, which is identified by the first index the maximum numeric element within the aggregated probability vector, or a second writing system that is identified by the second index of the next largest numeric element within the aggregated probability vector. 13. The system of claim. 11, wherein each sub-image of the plurality of sub-images is a rectangular sub-image of a predetermined size. 14. The system of claim 11, wherein the processor is further configured to perform: recursively splitting into corresponding image sub-fragments of one or more image fragments, which are characterized by the neural network as containing text having a text size less than the minimum threshold size. 15. The system of claim 11, wherein splitting the document image into a plurality of image fragments further includes: converting each sub-image of the plurality of sub-images to a predetermined size. 16. The system of claim 11, wherein the processor is further configured to perform: determining the spatial orientation of the document image using a neural network that processes a plurality of image fragments. 17. A machine-readable non-volatile data carrier, including executable instructions that, when executed by a computing system, ensure that the computing system performs the following operations: getting an image of the document; splitting the document image into a plurality of image fragments; generating a plurality of probability vectors by means of a neural network processing a plurality of image fragments, wherein each probability vector of the plurality of probability vectors is created by processing the corresponding fragments of the image and contains a plurality of numeric elements, wherein each numeric element of the plurality of numeric elements reflects the probability of an image fragment containing text, associated with the writing system, which is identified by the index of the numeric element in the corresponding probability vector; calculating an aggregated probability vector by aggregating a plurality of probability vectors, each numeric element of the aggregated probability vector reflecting a probability of an image containing text associated with a writing system, which is identified by an index of the numeric element within the aggregated probability vector; And given the determination that the maximum numeric element of the aggregated probability vector exceeds a predetermined threshold value, it is concluded that the document image contains one or more characters associated with the writing system, which is identified by the index of the maximum numeric element within the aggregated probability vector. 18. The storage medium of claim 17, further including executable instructions that, when executed by the computing system, perform the following operations: given the determination that the maximum numerical element of the aggregated probability vector is less than or equal to a predetermined threshold value, it is concluded that the document image contains one or more characters associated with one of the following systems: the first writing system, which is identified by the first index the maximum numeric element within the aggregated probability vector, or a second writing system that is identified by the second index of the next largest numeric element within the aggregated probability vector. 19. The storage medium of claim 17, wherein splitting the document image into a plurality of image fragments further includes: converting each sub-image of the plurality of sub-images to a predetermined size. 20. The storage medium of claim 17, further including executable instructions that, when executed by the computing system, perform the following actions: determining the spatial orientation of the document image using a neural network that processes a plurality of image fragments.

Images