RU2787136C2 - Traceless image capture, using mobile device - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: group of inventions relates to image processing technologies and more specifically to systems and methods for image capture, using a mobile device. A computer-implemented method for image capture with a mobile device is proposed. The method contains a stage, at which an application for video capture, launched on the mobile device, receives a video stream from a camera of the mobile device. Next, a specific frame of the video frame is identified. A set of hypotheses determining image borders in the specific frame is generated. A trained neural network selects a specific hypothesis among the set of hypotheses. A value of a quality indicator of a candidate image is determined. In this case, if the value of the quality indicator of the selected frame is lower than a value of the quality indicator of a previously processed image, the selected frame is rejected, and a new frame is identified. A frame with the highest quality indicator is automatically recorded.

EFFECT: provision of automatic capture of the best quality of a video frame of a video stream with a mobile device, using an application not requiring installation on the mobile device.

20 cl, 6 dwg

Description

FIELD OF TECHNOLOGY

[0001] Embodiments of the invention generally relate to image processing, and more specifically to systems and methods for capturing an image using a mobile device.

BACKGROUND OF THE INVENTION

[0002] Currently, mobile devices can be used to capture documents. However, various methods for capturing documents by mobile devices involve user actions such as manually adjusting the lighting, stabilizing the mobile device, and so on. The present invention describes the automated capture of images of a mobile device.

DISCLOSURE OF THE INVENTION

[0003] Implementations of the present invention describe mechanisms for capturing an image by a mobile device, comprising: receiving, by a video capture application running on a mobile device, a video stream from a camera of the mobile device; identification of a particular frame of the video stream; generation of a set of hypotheses that define the boundaries of the image in a particular frame; selection by the neural network of a specific hypothesis among the set of hypotheses; generating a candidate image by applying a particular hypothesis to a particular frame; determining a quality score value of the candidate image; determining that the quality score of the candidate image is greater than one or more quality scores of one or more previously processed images extracted from the video stream; however, the image capture application does not require installation. In some embodiments, the method further comprises performing OCR on the document depicted in the candidate image, wherein OCR on the document is performed on the mobile device; cropping a candidate image; displaying the candidate image on the screen of the mobile device; wherein the video stream is received in response to receiving the command via the web browser control; wherein the reception of the video stream is performed in response to the receipt of the command via the control user interface of the mobile device.

[0004] The non-volatile computer-readable storage medium of the invention includes instructions that, when accessed by a processing device, cause the processing device to: receive, with a video capture application running on a mobile device, a video stream from a camera of the mobile device; identify a specific frame of the video stream; generate a set of hypotheses that determine the boundaries of the image in a particular frame; using a neural network to select a specific hypothesis from a set of hypotheses; create a candidate image by applying a specific hypothesis to a specific frame; determine the value of the quality indicator of the candidate image; determine that the value of the quality index of the candidate image exceeds one or more values of the quality index of one or more previously processed images extracted from the video stream; however, the image capture application does not require installation.

In some embodiments, the instructions of the non-volatile computer-readable storage medium further cause the processing device to perform OCR on the document depicted in the candidate image, wherein the OCR of the document is performed on the mobile device; crop the candidate image; display the candidate image on the screen of the mobile device; wherein the video stream is received in response to receiving the command via the web browser control; wherein the reception of the video stream is performed in response to the receipt of the command via the control user interface of the mobile device.

[0005] The system of the invention includes a memory and a processing device operatively associated with the memory, a processing device for receiving, by a video capture application running on a mobile device, a video stream from a camera of the mobile device; identification of a particular frame of the video stream; generation of a set of hypotheses that define the boundaries of the image in a particular frame; selecting, using a neural network, a specific hypothesis from a set of hypotheses; generating a candidate image by applying a particular hypothesis to a particular frame; determining a quality score value of the candidate image; determining that the quality score of the candidate image is greater than one or more quality scores of one or more previously processed images extracted from the video stream; however, the image capture application does not require installation.

In some implementations, the system further comprises performing OCR on the document depicted in the candidate image, wherein the OCR of the document is performed on the mobile device; cropping the candidate image; displaying the candidate image on the screen of the mobile device; wherein the video stream is received in response to receiving the command via the web browser control; wherein the reception of the video stream is performed in response to the receipt of the command via the control user interface of the mobile device.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The invention may be better understood by the detailed description given below and by the accompanying illustrations showing various applications of the invention. The illustrations, however, should not be taken as the only possible applications of the invention, but are provided only for explanation and understanding.

[0007] FIG. 1 is a schematic representation of a mobile device in accordance with one or more embodiments of the present invention.

[0008] FIG. 2 is a flowchart illustrating one exemplary image capture method in accordance with some embodiments of the present invention.

[0009] FIG. 3 is a flowchart illustrating an exemplary method for identifying image boundaries in accordance with some embodiments of the present invention.

[0010] FIG. 4 is a schematic representation of the structure of a neural network operating in accordance with one or more embodiments of the present invention.

[0011] FIG. 5 is a block diagram of an exemplary computer system in which the present invention may be applied.

[0012] FIG. 6 is a block diagram of a computer system in accordance with some embodiments of the present invention.

DESCRIPTION OF THE PREFERRED IMPLEMENTATION OPTIONS

[0013] Mobile image capture systems and methods are described. Various ways of capturing an image with a mobile device include important user actions such as manually adjusting the lighting, waiting for the mobile device to stabilize to avoid blur and glare in the captured image. Careful positioning of the mobile device in relation to the image being captured can also be critical to successful image capture. All parts of the image must be visible to the camera of the mobile device. The mobile device should be positioned close to the center of the image and preferably parallel to the image surface to avoid significant distortion. Often the user needs to take multiple shots to select the most appropriate shot with the least number of issues.

[0014] In most cases, in order to capture an image with a mobile device, a user needs to install a specialized application for capturing images on the mobile device. However, the resulting images are often of poor quality. In addition, many users do not want to install another application on their devices.

[0015] Embodiments of the invention address the above and other disadvantages by providing mechanisms for automatic image capture. According to the present invention, image capture is performed by analyzing a plurality of frames in a video stream captured by a camera and selecting a frame that is suitable for image processing according to a set of user-configurable or automatically determined requirements.

[0016] One aspect of the present invention is the automation of image capture. As used in the present invention, image capturing is automated, where a sequence of video frames captured by a camera is analyzed in real time without recording a video stream. If the analysis shows that the video frame is of acceptable quality, such a video frame is automatically recorded. Subsequently, the recorded image can be cropped and further processed to eliminate distortion and other imperfections.

[0017] Another embodiment of the present invention is that all interactions with the camera of the mobile device, as well as analysis of the video stream, identification of a suitable video frame, and transmission of the captured image to another device, are performed by an application that does not require installation. A non-installation application is an application that is not pre-installed on a mobile device, but instead is downloaded by the mobile device only in response to a user action in a browser window; upon completion, the application is automatically deleted from the mobile device. For example, such an application may be loaded by a mobile device when a user activates a particular graphical user interface (GUI) control on a browser page. Once image capture is complete, this zero-memory app is removed from the mobile device.

[0018] The term "frame" as used herein may refer to an image captured by the camera of a mobile device in real time, whether the captured video stream, which includes a sequence of frames, is recorded in real time or asynchronous to capture.

[0019] As used herein, "mobile device" may refer to a portable computer such as a mobile phone, smartphone, phablet, tablet computer, netbook, digital media player, handheld game console, smart watch, personal digital assistant device, enterprise digital assistant, laptop , digital video camera, digital camera, digital video camera. Some mobile devices may have a touch screen that provides a touch screen interface with number buttons and a numeric keypad. Some mobile devices may have physical buttons along with a physical keyboard. Some mobile devices may be able to connect to transfer digital data through one or more input/output (I/O) interfaces (for example, one or more network wireless and/or wired network interfaces that can be used to establish communication sessions with one or more servers and/or peer devices over one or more networks containing local and/or wide area networks such as the Internet).

[0020] the Described methods allow automatic image capture using artificial intelligence. These methods may include training a neural network to determine the appropriate edge of an image in a video frame. The neural network may include a plurality of neurons that are associated with trainable weights and biases. Neurons can be organized into layers. A neural network can be trained on a training set of image data with correctly identified image boundaries. For example, the training dataset may include sample images and suitable bounds for those images as learning outcomes.

[0021] The neural network may generate an output image for each original training image. The output image of the neural network can be compared to the target output associated with the original training image, as set by the training dataset, and the error can be propagated to previous layers of the neural network, whose parameters (such as neuron weights and biases) can be changed accordingly. During training of the neural network, the parameters of the neural network can be changed to optimize the prediction accuracy. Once trained, the neural network can be used to automatically capture an image.

[0022] FIG. 1 is an exemplary illustration of a mobile device 102 having a screen 104 and a camera control 106 for triggering video streaming and recording by the camera of the mobile device 102. In some embodiments of the present invention, the camera 106 may be controlled by a virtual button on the touch screen of the mobile device 102. In other implementations, the camera control 106 may be a physical button. Screen 104 may display images (eg, video frames) that are captured by the camera of the mobile device.

[0023] FIG. 2 is a flow diagram illustrating one exemplary image capture method 200 in accordance with some embodiments of the present invention. Method 200 may be performed by processing logic, which may include hardware (eg, circuitry, custom logic, programmable logic, microcode, and so on), software (eg, instructions executed on a processing device), firmware, or a combination thereof. In one application, method 200 may be performed by a processor (eg, processor 602 in FIG. 6) of computing device 510 and/or server 550, as described in connection with FIG. 5. In some applications, method 200 may be performed by a single processing thread. Alternatively, method 200 may be executed by two or more processing threads, each of which performs one or more separate functions, procedures, subroutines, or method steps. For example, the processing threads implementing method 200 may be synchronized (eg, using semaphores, critical sections, and/or other thread synchronization mechanisms). In another application, the processing threads that implement method 200 may execute asynchronously with respect to each other. Therefore, if in FIG. 2 and its associated description lists the steps of the method 200 in a specific order, various applications of the method may perform at least some of the described steps in parallel and/or in an arbitrary order.

[0024] In step 201, the user of the mobile device 102 opens the web page in the browser of the mobile device. The web page contains a control (such as a visual control or voice control) that allows the user to launch an application that has camera control capabilities. In accordance with the present invention, this application does not require installation. Accordingly, when the web page is eventually closed by the user or otherwise terminated, the application is automatically removed from the mobile device 102.

[0025] In step 202, the user activates a control on the web page. In response to activation of the control, the camera of the mobile device 102 is activated 203 by an application that does not require installation. The user points the mobile device's camera lens at the object to be captured, and when the camera captures a sequence of video frames, the no-install application analyzes the video stream. In some implementations, when the camera of the mobile device 102 is active, the no-installation application automatically starts analyzing the video stream when an image of a document appears in the camera lens.

[0026] At step 204, the system of the present invention identifies a particular frame of the video stream (eg, the last captured frame at the current point in time). In some embodiments of the present invention, the system converts the identified frame to HTML video and displays it to the user in real time using screen 104.

[0027] In some embodiments of the present invention, the received video frames are analyzed in real time without first recording the video stream or frame. Accordingly, the processed video frames can be discarded (for example, using a ring buffer in the memory of the mobile computing device to store the received video frames), so that the (N + 1)th frame replaces the previously recorded first frame, provided that the buffer capacity allows to store N frames, where N is a predefined integer value.

[0028] In other embodiments of the present invention, the video stream is recorded, that is, stored in volatile and/or non-volatile memory, and then may be processed asynchronously with respect to the recording. The recorded video stream may be obtained from the local memory of the mobile computing device, or from a remote server or peer-to-peer mobile computing device via a wired or wireless network connection.

[0029] Once a particular frame is identified (204), analysis of the frame begins. The frame is analyzed to determine the image quality of the document contained in the frame and to determine the boundaries of the document image.

[0030] "Image borders" are outlines that define the edges of an area in a frame that will be further processed by the application (eg, by performing OCR on the text contained in the imaged document). If the image is a document page, the image borders form a rectangle. Image borders are formed by "main lines". In the case of a rectangular image, the number of main lines is four.

[0031] In step 205, a specific frame is analyzed by a shift detector. If the shift of a particular frame exceeds a predetermined threshold, the particular frame is discarded and another frame is identified similarly to step 204.

[0032] At step 206, a particular frame is analyzed to identify an image boundary within the frame. The specific steps of the boundary identification process 206 are described in more detail below with reference to FIG. 3.

[0033] To determine the border of the image in the frame, the system converts the frame to grayscale (301).

[0034] Color images are often stored as three separate image matrices, one storing the amount of red (R) in each pixel, another the amount of green (G), and a third the amount of blue (B), the so-called RGB format. When converting an RGB image to grayscale, the RGB values for each pixel are converted to a single value representing the brightness of that pixel. In some implementations, this transform takes the average of each channel's contribution: (R+G+B)/3. However, since perceived luminance is often dominated by a green component, another more "human-oriented" implementation would take a weighted average, such as: 0.3R+0.59G+0.11 V or another weighted average.

[0035] In step 302, the system according to the present invention compresses the selected frame in grayscale. In some embodiments, compression may be performed using bilinear interpolation. The compression is performed to a level where small non-essential details (such as scratches, noise, smudges) are erased, but the essential details of the image are still clearly visible. In some implementations, the frame is compressed to a predetermined size, such as 1-3 megapixels. In some implementations, the frame is compressed to a size of 256 × 256 pixels, that is, to 0.07 megapixels.

[0036] In some embodiments, the frame is compressed to the size of a square whose side is a power of 2. Such compressions are especially useful if the Hough transform is used in further processing of a particular frame. Compression of the selected frame significantly reduces the time required for frame analysis and, as a result, allows real-time image capture from the video stream.

[0037] In some embodiments of the present invention, other pre-processing operations, such as a 3×3 median filter or a Canny edge detector, may be applied to a particular frame.

[0038] In step 303, the system according to the present invention detects the boundaries of an image in a particular frame. In some implementations, the Fast Hough Transform is used to detect boundary lines. In some implementations, the fast Hough transform is applied to a particular frame at least twice to detect horizontal and vertical lines. The resulting image is scanned to detect local maxima. Each detected maximum defines a potential edge of the image.

[0039] In some embodiments of the present invention, the system classifies the detected peaks to identify peaks corresponding to image boundaries. This classification is performed by a linear classifier which determines the level of confidence for each detected maximum that corresponds to an image boundary. Peaks with confidence levels exceeding a predetermined threshold are identified, and the lines corresponding to them are sorted into groups according to the potential edge of the image. For example, for a rectangle, the identified lines are sorted into four groups: top border group, bottom border group, left border group, and right border group.

[0040] In some embodiments of the present invention, the resulting line groups are filtered to remove duplicates (lines that are too similar to each other). Thanks to the Hough transform line identification method, one line generates a set of local maxima such as (x, y), (x-1, y+2), (x-2, y+4), …. The lines resulting from these local maxima intersect the upper and lower boundaries of the image in the coordinates [xTop, xBottom], [xTop+1, xBottom+1], [xTop+1, xBottom-1], ... respectively. Since these lines are very similar, they will all receive the same levels of confidence from the classifier.

[0041] In some embodiments of the present invention, to determine if two lines are too similar, the following heuristic test is applied: for each pair of lines, the system determines the distance between the points where these lines intersect opposite image boundaries. If these distances are below a given threshold, the lines are considered too similar and one of them is excluded from the group of potential boundary lines.

[0042] Based on these potential boundary lines, the system may generate boundary rectangle hypotheses. For each generated hypothesis, a confidence level can be calculated corresponding to the confidence levels of the lines that form the rectangle according to that hypothesis. In some implementations, the hypotheses are ranked by confidence levels.

[0043] The set of hypotheses can be filtered to exclude sets of lines that may not form a suitable rectangle. In some implementations, the system also filters out rectangle hypotheses with unlikely geometry. The system can use the following parameters to determine inappropriate rectangles: vertex angles, area of the resulting rectangle, side dimensions, and other geometric parameters. These parameters can be analyzed to identify implausible hypotheses. In some implementations, the system evaluates such geometric parameters against predetermined threshold parameters.

[0044] In step 304, the system may further filter the set of hypotheses based on geometric correlations and weights of points located on the rectangle lines. In some implementations, filtering is performed by a linear classifier. The system may further filter the remaining hypotheses to exclude substantially similar hypotheses.

[0045] In step 305, the system may further evaluate the remaining hypotheses using the neural network. In some embodiments of the present invention, the system may further evaluate the corners of the rectangles using a neural network.

[0046] FIG. 4 schematically shows the structure of a neural network operating in accordance with one or more embodiments of the present invention. As shown in FIG. 4, neural network 400 may be represented by a non-periodic feed-forward neural network including an input layer 410, an output layer 420, and one or more hidden layers 430 connecting the input layer 410 and the output layer 420. The output layer 420 may have the same number nodes as input layer 410, so that network 400 can be trained by an unsupervised learning process to recover its own input data.

[0047] The neural network may include a plurality of neurons that are associated with trainable weights and biases. Neurons can be organized into layers. A neural network can be trained on a training data set of boundary hypotheses.

[0048] The neural network may generate an output image for each original training image. During training of the neural network, the parameters of the neural network can be changed to optimize the prediction accuracy. Training a neural network may include processing the boundary hypotheses by the neural network so that the network determines a quality metric (i.e., the observed outcome) for that hypothesis, and comparing the determined quality metric with a known quality metric (i.e., the target output corresponding to the training input , as specified in the training dataset). The observed output of the neural network can be compared to the target output, and the error can be fed back to the previous layers of the neural network whose parameters (e.g. neuron weights and biases) can be adjusted accordingly to minimize the loss function (i.e. the difference between the observed output and learning outcomes).

[0049] After training, the neural network can be used to automatically determine the quality score for the boundary hypotheses. In some embodiments, a neural network may be used to automatically determine the quality scores for the vertices of the rectangles. The mechanisms for determining quality scores described herein can improve the quality of an image capture process by performing a quality score determination using a trained neural network in a manner that considers the most relevant attributes.

[0050] In some embodiments, a hypothesis quality score may be determined based on the rectangle vertex quality scores. Vertex quality metrics can be computed by a neural network processing vertex images. Such images are usually relatively small and easy to handle, which shortens the processing time for image capture in accordance with the present invention.

[0051] In step 306, the system may perform a second level classification of the rectangle hypotheses. In some implementations, a gradient boosting classifier is used to perform the second level classification. The second level classification can use the same parameters as the first classification. In some embodiments, the rectangle vertex quality scores generated by the neural network may also be used as second-level classification parameters.

[0052] As a result of the second level classification, the best boundary hypothesis can be selected.

[0053] After selecting the best boundary hypothesis in step 306, returning to FIG. 2, in step 207, the system determines the quality score of the selected frame. Only that part of the selected frame that falls within the selected boundary is taken into account. In some embodiments of the present invention, the system may apply noise reduction techniques to an image in a frame within a selected boundary to determine its quality score.

[0054] In step 208, the system may compare the quality metric of the selected frame (as determined in step 207) with a previously calculated quality metric of the previous frame. If the quality score of the selected frame is worse than the quality score of the previous frame, the selected frame is discarded and a new frame 204 is identified.

[0055] In some embodiments of the present invention, the system identifies a particular frame in a video stream at a predetermined rate. In some embodiments of the present invention, the system identifies a particular frame in the video stream at predetermined time intervals. In some embodiments of the present invention, the next frame is identified after parsing of the previous identified frame is complete. In some implementations, the system analyzes a predetermined number of specific frames and proceeds to the next step if the best frame is not identified among a predetermined number of frames or within a predetermined period of time.

[0056] In step 209, the system evaluates the suitability of the frame for optical character recognition (OCR). In some implementations, the OCR suitability evaluation is performed locally on the mobile device. Alternatively, OCR validation may be performed on the server after the selected frame has been captured and transmitted to the server.

[0057] In step 210, the system performs post-processing of the selected frame. Post-processing may include optical character recognition of the image in the selected frame, machine translation of text in the selected frame, or other document processing or image processing operations.

[0058] FIG. 5 is a block diagram of an exemplary computer system 500 in which embodiments of the invention may be applied. As depicted, system 500 may include a computing device 510, storage 520, and a server 550 connected to network 530. Network 530 may be a public network (eg, the Internet), a private network (eg, a local area network (LAN), or a wide area network). computer network (GWS)) or their combination.

[0059] Computing device 510 may be a desktop computer, laptop, smartphone, tablet, server, scanner, or any suitable computing device capable of performing the methods described herein. In some applications, the computing device 510 may be (and/or include) one or more of the computing devices 600 in FIG. 6.

[0060] Frame 540 may be received by computing device 510. Frame 540 may be received in any suitable manner. Furthermore, in cases where computing device 510 is a server, a client device connected to the server via network 530 may upload frame 540 to the server. In cases where computing device 510 is a client device connected to a server via network 530, the client device may download frame 540 from the server or from storage 520.

[0061] Frame 540 may be used to train a set of machine learning models, or may be a new frame for which a quality score determination is desired.

[0062] In one implementation, the computing device 510 may include a quality score engine 511. The quality metric determination mechanism 511 may include instructions stored on one or more tangible, computer-readable storage media of the computing device 510 and executed by one or more processing devices of the computing device 510.

[0063] In one implementation, the quality score engine 511 may use a set of trained machine learning models 314 to determine one or more frame quality scores 540. The frame library may reside in a storage 520. The machine learning models 514 are trained and used to determine the quality scores.

[0064] The quality score engine 511 may be a client application or a combination of client and server components. In some implementations, the quality score engine 511 may run entirely on a client computing device, such as a server computer, desktop computer, tablet computer, smartphone, laptop computer, camera, camcorder, or the like. Alternatively, the client component of the quality score engine 511 running on the client computing device may receive the frame and transmit it to the server component of the score engine 511 running on the server device that performs the quality score determination. The server component of the quality score engine 511 may then return the determined quality scores to the client component of the quality score engine 511 running on the client computing device for storage. Alternatively, the server component of the quality score engine 511 may provide the identification result to another application. In other implementations, the quality score engine 511 may run on a server device as an Internet-accessible application accessible through a browser interface. The server device may be one or more computer systems, such as one or more server machines, workstations, system units, personal computers (PCs), and so on.

[0065] Server machine 550 may be and/or include a rack server, router, personal computer, portable assistant, mobile phone, laptop, tablet, camera, camcorder, netbook, desktop computer, media center, or any combination of the above. The server machine 550 may include a learning engine 551. The learning engine 551 may build a machine learning model(s) 514 to determine quality metrics. The machine learning model(s) 514 as shown in FIG. 5 can be trained by the learning engine 551 using training data that includes training inputs and associated training outputs (correct responses to corresponding training inputs). The learning engine 551 can find patterns in the training data by associating the input training data with the output training data (intended response) and create machine learning models 514 that capture these patterns. The set of machine learning models 514 may consist of, for example, a single layer of linear or non-linear operations (such as a support vector machine), or may be a deep neural network such as a machine learning model that consists of multiple layers of non-linear operations. Examples of deep neural networks are neural networks including convolutional neural networks, recurrent neural networks (RNNs) with one or more hidden layers, and fully connected neural networks.

[0066] Machine learning models 514 may be trained to determine quality metrics for frame 340. The training data may reside in store 520 and may include one or more sets of input training data 522 and one or more sets of output training data 524. The training data may also include map data 526 that links input training data 522 to output training data 524. During training, learning engine 551 can find patterns in training data 526 that can be used to link input training data to output training data. The patterns can then be used by the machine learning model(s) 514 for future predictions. For example, upon receiving input from an unknown frame, the trained machine learning model(s) 514 may predict quality scores for that frame and provide those quality scores as output.

[0067] Store 520 may be a persistent store capable of storing structures for performing similarity measure determination in accordance with embodiments of the present invention. Storage 520 may be located on one or more storage devices such as main memory, magnetic or optical disks, tapes or hard drives, NAS, SAN, and so on. Although the image shows computing device 510 separate from storage 520, in application, storage 520 may be part of computing device 510. In other uses, storage 520 may be a network file server, while in other applications, storage 520 may be different. type of data storage, such as an object-bound database, a relational database, and so on, which may reside on a server machine or on one or more different machines connected through network 530.

[0068] FIG. 6 shows an example computer system 600 that may perform one or more of the methods described herein. The computing system may be connected (eg, on a network) to other computing systems using a LAN, intranet, extranet, or the Internet. The computing system may operate as a server in a client-server network environment. The computing system may be a personal computer (PC), a tablet, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a camera, a camcorder, or any device capable of executing a set of commands (sequential or otherwise) specifying actions to be performed. this device. In addition, although only one computing system is illustrated, the term "computer" should also be understood to include any collection of computers that individually or collectively execute a set (or sets) of instructions to perform one or more of the methods discussed herein.

[0069] An example computer system 600 includes a processor 602, main memory 604 (i.e., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), static a memory 606 (i.e., flash memory, static random access memory (SRAM)) and a storage device 616 that communicate with each other over a bus 608.

[0070] Processor 602 represents one or more general purpose processing devices such as a microprocessor, a central processing unit, and the like. In particular, processor 602 may be a full instruction set (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor 602 may also be one or more special purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, and the like. Processor 602 is configured to execute instructions 626 to implement the quality score engine 511 and/or the learning engine 551 of FIG. 5 and performing the operations and steps described herein (eg, method 200 of FIG. 2).

[0071] The computer system 600 may further include a network interface device 622. The computer system 600 may also include a visual display device 610 (for example, a liquid crystal display (LCD) or cathode ray tube (CRT)), an alphanumeric input device 612 (for example , keyboard), a cursor control device 614 (eg, a mouse), and a signal generating device 620 (eg, a speaker). In one of the examples shown, the display device 610, the alphanumeric input device 612, and the cursor control device 614 may be combined into a single component or device (eg, an LCD touch screen).

[0072] Storage devices 616 include a computer-readable storage medium 624 that stores instructions 626 describing one or more of the methodologies or functions described herein. Instructions 626 may also refer in whole or at least in part to main memory 604 and/or processor 602 at the time they are executed by computer system 600, main memory 604, and processor 602, which are also referred to as computer readable media. In some applications, instructions 626 may additionally be transmitted or received over the network via network interface device 622.

[0073] While the computer-readable storage medium 624 is shown in the illustrative examples as the only medium, the term "machine-readable storage medium" should be understood to include one medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) , which store one or more command sets. The term "machine-readable storage medium" should also include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a machine and that causes the machine to execute one or more of the methodologies of the present invention. The term "computer-readable storage medium" should be appropriately understood to include solid-state storage devices, optical media, and magnetic media.

[0074] Although the steps of the methods herein are shown and described in a specific order, the order of steps of each method can be changed so that certain steps can be performed in reverse order, or so that a certain step can be performed at least in part simultaneously with other steps. . In some embodiments, commands or sub-operations of individual operations may be discontinuous and/or interleaved.

[0075] It should be understood that the above description is for illustration and not limitation. Many other implementation options will be apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of the present invention is to be determined by reference to the appended claims, together with the full scope of the equivalents to which such claims are entitled.

[0076] Numerous details have been set forth in the above description. However, it will be apparent to those skilled in the art that aspects of the present invention may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than in detail so as not to obscure the present invention.

[0077] Some parts of the detailed descriptions above are presented in terms of algorithms and symbolic representations of operations on data bits in computer memory. These algorithmic descriptions and representations are the means used by data scientists to most effectively convey the essence of their work to other specialists in the art. The algorithm here and in general is conceived as a self-consistent sequence of steps leading to the desired result. These steps require physical manipulations of physical quantities. Typically, though not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise used. It is sometimes convenient, mainly for reasons of common usage, to refer to these signals as bits, values, elements, characters, symbols, terms, numbers, and the like.

[0078] However, it should be borne in mind that all these and similar terms should be associated with the corresponding physical quantities and are merely convenient designations applied to these quantities. Unless specifically stated otherwise, as evident from the following discussion, it should be understood that throughout the description of the discussion using terms such as "obtaining", "determining", "selection", "storage", "analysis", etc. , refer to the operation and processes of a computing system or similar electronic computing device that processes and converts data represented as physical (electronic) quantities in the registers and memory of the computing system into other data similarly represented as physical quantities in the memory or registers of the computing system. systems or other similar devices for storing, transmitting or displaying information.

[0079] The present invention also relates to a device for performing operations in this document. This device may be specially designed for the desired purpose or may include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium such as, but not limited to, any type of disk, including floppy disks, optical disks, compact disks and magneto-optical disks, Read Only Memory (ROM), Random Access Memory (RAM), EEPROM, EEPROM, magnetic or optical boards or any type of media suitable for storing electronic instructions, each connected to the computer's system bus.

[0080] The algorithms and displays presented herein are not inherently associated with any particular computer or other device. Various general purpose systems may be used with programs in accordance with the ideas outlined, or it may be convenient to construct a more specialized device to carry out the required methodological steps. The required structure for many of these systems will be displayed as indicated in the description. In addition, embodiments of the present invention are not described by reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present invention as described herein.

[0081] Embodiments of the present invention may be provided in the form of a computer program product or software that may include a computer-readable medium for storing instructions used to program a computer system (or other electronic devices) to perform a process in accordance with the present invention. . A computer-readable medium includes any mechanism for storing or transmitting information in a form that is readable by a machine (eg, computer). For example, computer-readable (i.e., computer-readable) media includes computer-readable (i.e., computer-readable) storage media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash drives, -memory, etc.).

[0082] The words "example" or "exemplary" are used here to refer to examples, cases, or illustrations. Any implementation or design described herein as an "example" should not necessarily be construed as preferred or advantageous over other implementations or designs. Rather, the use of the words "example" or "exemplary" is intended to represent specific concepts. As used in this application, the term "or" is intended to mean an inclusive "or" and not an exclusive "or". That is, unless otherwise indicated or clear from the context, "X includes A or B" means any of the natural inclusive permutations. That is, if X includes A, X includes B, or X includes both A and B, then the proposition "X includes A or B" is true in any of the above cases. In addition, the indefinite articles "a" and "an" used in the English version of this application and the appended claims should generally mean "one or more", unless otherwise indicated or the context does not imply that this refers to singular form. Moreover, the use of the terms "use", "single use", "implementation", or "single implementation" is not intended to refer to the same application or implementation unless they are described as such. In addition, as used herein, the terms "first", "second", "third", "fourth", etc. denote labels to distinguish between different elements, and may optionally have an ordinal value according to their numerical designation.

[0083] Since many changes and modifications to the invention will no doubt be apparent to those skilled in the art upon reading the foregoing description, it should be understood that any particular use shown and described by means of the illustrations should not be considered limiting in any way. Therefore, references to the details of various applications should not limit the scope of requests, which in themselves contain only those features that are considered as the invention.

Claims (52)

1. A computer-implemented method for capturing an image by a mobile device, including: receiving by the video capture application running on the mobile device a video stream from the camera of the mobile device; identification of a particular frame of the video stream; by processing a frame of the video stream, generating a set of hypotheses that define the boundaries of the image in a particular frame; selection of a specific hypothesis trained by the neural network among a plurality of hypotheses; generating a candidate image by applying a particular hypothesis to a particular frame; determining a quality score value of the candidate image; determining that the value of the quality index of the candidate image exceeds one or more values of the quality index of one or more previously processed images extracted from the video stream, while if the value of the quality index of the selected frame is lower than the value of the quality index of the previously processed image, the selected frame is rejected and a new frame is identified; automatic frame recording with the highest quality index; however, the video capture application is an application that does not require installation. 2. The method of claim. 1, which additionally performs optical character recognition of the document contained in the candidate image. 3. The method of claim 2, wherein OCR of the document is performed on the mobile device. 4. The method of claim 1, further comprising cropping the candidate image. 5. The method of claim 1, further comprising displaying the candidate image on the screen of the mobile device. 6. The method of claim 1, wherein the video stream is received in response to receiving a command via a web browser control. 7. The method of claim. 1, in which the reception of the video stream is performed in response to the receipt of the command through the user control interface of the mobile device. 8. A system for capturing an image by a mobile device, comprising: Memory device; and a processor associated with the storage device, configured to: receiving, by the video capture application running on the mobile device, a video stream from the camera of the mobile device; identification of a specific frame of the video stream; by processing the frame of the video stream, generating a set of hypotheses that determine the boundaries of the image in a particular frame; selecting a particular hypothesis by the trained neural network among the plurality of hypotheses; generating a candidate image by applying a particular hypothesis to a particular frame; determining a quality score value of the candidate image; determining that the value of the quality index of the candidate image exceeds one or more values of the quality index of one or more previously processed images extracted from the video stream, while if the value of the quality index of the selected frame is lower than the value of the quality index of the previously processed image, the selected frame is rejected and a new frame is identified; automatic frame recording with the highest quality index; however, the video capture application is an application that does not require installation. 9. The system of claim. 8, which additionally performs optical character recognition of the document contained in the candidate image. 10. The system of claim. 9, wherein the optical character recognition of the document is performed on a mobile device. 11. The system of claim 8, further comprising cropping the candidate image. 12. The system of claim 8, further comprising displaying the candidate image on the screen of the mobile device. 13. The system of claim 8, wherein the video stream is received in response to receiving a command via a web browser control. 14. The system of claim. 8, where the reception of the video stream is performed in response to receiving a command through the user control interface of the mobile device. 15. A non-volatile computer-readable storage medium including executable instructions that, when executed by a computer system, cause the computer system to: receiving by the video capture application running on the mobile device a video stream from the camera of the mobile device; identification of a particular frame of the video stream; by processing a frame of the video stream, generating a set of hypotheses that define the boundaries of the image in a particular frame; selection of a specific hypothesis trained by the neural network among a plurality of hypotheses; generating a candidate image by applying a particular hypothesis to a particular frame; determining a quality score value of the candidate image; determining that the value of the quality index of the candidate image exceeds one or more values of the quality index of one or more previously processed images extracted from the video stream, while if the value of the quality index of the selected frame is lower than the value of the quality index of the previously processed image, the selected frame is rejected and a new frame is identified; automatic frame recording with the highest quality index; however, the video capture application is an application that does not require installation. 16. The storage medium of claim 15, further comprising performing OCR on the document contained in the candidate image. 17. The storage medium of claim 16, wherein the OCR of the document is performed on the mobile device. 18. The storage medium of claim 15, further comprising cropping the candidate image. 19. The storage medium of claim 15, further comprising displaying an image of the candidate on the screen of the mobile device. 20. The storage medium of claim 15, wherein the video stream is received in response to receiving a command via a web browser control.

Images