RU2775808C2 - Neural network systems and computer-implemented methods of identifying and/or evaluating one or more food products in visual input information - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to a method, a system and a data medium for processing a video stream for identifying and evaluating the quality of pizza production. Method comprises receiving, by a processor, a continuous video stream from at least one camera position above a table configured to receive ready pizzas; collecting, using the processor, a plurality of video frames of the pizza containing video frames of the selected pizza from the video stream; using the processor to use a first convolutional neural network (CNN) to select a set of video frames of the selected pizza with the highest rating from the plurality of video frames of the pizza; using the processor first CNN to identify the image of the selected pizza with the highest rating from the set of video frames of the highest rated pizza; using the processor first CNN to localize at least one portion of the selected pizza in the identified image with the highest rating pizza; using the processor of the first CNN to determine the type of the selected pizza from the identified image with the highest rating pizza; using the processor to use a second convolutional neural network (CNN) to set a map of components of the selected pizza as a result of automatically performing segmentation of the image of the portion of pizza based on at least the type of pizza; and applying, by means of a processor, a second CNN for automatically scoring a selected pizza based on a given pizza component map, comprising: dividing, using a processor, a portion of pizza from the identified best image into a plurality of slices; calculating, using the processor, gradients of one of the plurality of slices of the selected pizza; repeating, by the processor, the step for calculating gradients of the remaining slices from the plurality of slices; and determining, using the processor, the final score of the selected pizza based on the calculation of gradients of the plurality of slices.

EFFECT: high accuracy of recognizing objects in a video stream.

30 cl, 23 dwg, 1 tbl

Description

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

The present invention generally relates to food production processes, and in particular to the use of one or more machine learning techniques (e.g., a neural network) to identify and/or evaluate one or more food products in visual input information generated in the process. production.

BACKGROUND OF THE INVENTION

The rapidly growing demand for pizza creates incentives for pizza producers to improve the quality and efficiency of pizza production. The level of technology, including developments devoted to the problem of optimizing production and improving the quality of pizza produced, can be represented by the following technical solutions - RU 2654199 C1 (Samsung Electronics Co., Ltd.), 05/16/2018; CN 106529564 A (Universal Zhejiang Technology), 03/22/2017; CN 108335300 A (Universal Beijing Technology & Business), 07/27/2018; CN 107563439 A (Hunan Luchuan Information Technology Co. Ltd.), 01/09/2018; US 2018/0122063 A1 (Roche Diabetes Care, Inc.), 05/03/2018: WO 2018/148740 A1 (Iceberg Luxembourg), 08/16/2018; US 9977980 B2 (Microsoft Technology Licensing, LLC), 05/22/2018; US 9659225 B2 (Microsoft Corporation), May 23, 2017; CN 1147723 C (Huazhong Science & Engineering), 04/28/2004; WO 2016/077934 A1 (Le Henaff, Guy), 05/26/2016. However, most pizza manufacturers still use manual monitoring of the pizza production process, which causes changes in the quality of the pizza and the cost of time and human labor.

SUMMARY OF THE INVENTION

In some embodiments, the present invention relates to exemplary pizza retrieval and scoring computer systems and methods supported by video cameras and implemented with advanced convolutional neural network (CNN) algorithms that provide the benefits of accurate pizza scoring and improve accuracy, efficiency, and the quality of pizza production processes.

In some embodiments, the present invention relates to an exemplary convolutional neural network-based, computer-implemented method for identifying and evaluating food production, such as pizza production, comprising: receiving, by means of a processor, a continuous video stream from at least one camera positions above a table configured to receive ready-made pizzas; collecting by the processor a plurality of pizza video frames containing video frames of the selected pizza from the video stream; applying, by the processor, the first CNN to select a set of video frames of the selected pizza with the highest rated pizza from the plurality of pizza video frames; applying by the processor the first CNN to identify the image of the selected pizza with the highest rated pizza; using the processor on the first CNN to localize at least one serving of the selected pizza in the identified image with the highest rated pizza; applying by the processor the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; applying, by the processor, a second CNN to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and using the second CNN by the processor to automatically score the selected pizza based on the predetermined pizza ingredients map.

In some embodiments, the processor's application of a second CNN to automatically score a selected pizza based on a predetermined pizza ingredient map includes: dividing the pizza slice from the identified best image into a plurality of slices by the processor; calculating with the gradient processor one of the plurality of slices of the selected pizza; repeating by the processor the step of calculating the gradients of the remaining slices from the plurality of slices; and determining, with the processor, a final score of the selected pizza based on the calculation of the gradients of the plurality of slices.

In some embodiments, video frames of a video stream are categorized in cases including:

i) first case for images without pizza;

ii) second case for non-centered pizza images;

iii) the third case for centered pizza images and a resolution quality of X;

iv) the fourth case for centered pizza images and a resolution quality of Y, where Υ is higher than X;

v) the fifth case for centered pizza images of the first type and the required resolution quality; and

vi) the sixth case for centered pizza images of the second type and the required resolution quality.

In some embodiments, the application of the first CNN to select the highest rated set of pizza video frames from the pizza video frames and identify the best image is performed by a graphics processing unit (GPU).

In some embodiments, applying the first CNN to select the highest rated set of pizza video frames from the pizza video frames is performed with a high speed imprecise detector.

In some embodiments, selecting the highest rated set of pizza video frames includes discounting each pizza video frame with motion blur and/or defocus blur.

In some embodiments, the application of the first CNN to identify the best image is performed with an accurate low speed detector.

In some embodiments, the number of elements of the slice set is 8.

In some embodiments, the second CNN has a compression path and an extension path.

In some embodiments, the compression path includes a plurality of convolution and activation layers.

In some embodiments, the compression path further includes a downsampling and batch normalization layer following the first convolution and activation layer.

In some embodiments, the compression path further includes a Rectified Linear Units (ReLU) layer and a subsampling layer following each convolution and activation layer before moving on to the next convolution and activation layer.

In some embodiments, the extension path includes a sequence of upconvolutions and concatenations configured to combine spatial feature information with high resolution features from the compression path.

In some embodiments, applying the first CNN to localize at least one serving of a particular pizza in the identified highest rated pizza image includes: defining a bounding box and using one or more predefined binary masks.

In some embodiments, the present invention relates to an exemplary convolutional neural network-based, computer-implemented system for identifying and evaluating food production, such as pizza production, comprising: at least one image capture device; long-term storage device; one or more processors; and computer program code stored in the long-term memory, causing, when executed by one or more processors, the one or more processors to: receive a continuous video stream from at least one camera position above a table configured to receive finished pizzas; collecting a plurality of pizza video frames containing video frames of the selected pizza from the video stream; applying the first CNN to select a set of video frames of the selected pizza pizza with the highest rating from the plurality of pizza video frames; applying the first CNN to identify an image of the selected pizza with the highest rated pizza from the set of video frames of the highest rated pizza; applying the first CNN to localize at least one portion of the selected pizza in the identified image with the highest rated pizza; applying the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; applying the second CNN to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and applying the second CNN to automatically score the selected pizza based on the given pizza ingredients map.

In some embodiments, the computer program code includes instructions for: applying a second CNN to automatically score a selected pizza based on a given map of pizza ingredients, including: dividing the pizza portion in the identified best image into a plurality of slices; calculating gradients of one of the plurality of selected pizza slices; repeating the gradient calculation step to calculate gradients of the remaining slices from the plurality of slices; and determining the final score of the selected pizza based on the calculation of the gradients of the plurality of slices.

In some embodiments, the computer program code includes instructions for applying the first CNN on the GPU to select the highest rated set of pizza video frames from the pizza video frames and identify the best image.

In some embodiments, the computer program code includes instructions for using the first CNN to select the highest rated set of pizza video frames from the pizza video frames performed by the high speed inaccurate detector.

In some embodiments, the computer program code includes instructions for using the first CNN to identify the best image, performed with an accurate low speed detector.

In some embodiments, the present invention relates to an exemplary convolutional neural network-based, computer-realizable, durable, computer-readable storage medium for identifying and evaluating food production, such as pizza production, comprising processor-executable instructions for: receiving a continuous a video stream from at least one camera position above a table configured to receive finished pizzas; collecting a plurality of pizza video frames containing the selected pizza video frames from the video stream; selecting a set of video frames of the selected pizza pizza with the highest rating from the plurality of pizza video frames; applying the first CNN to identify an image of the selected top rated pizza pizza from the set of top rated pizza video frames; applying the first CNN to localize at least one portion of the selected pizza in the identified image with the highest rated pizza; applying the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; applying a second CNN to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and applying the second CNN to automatically score the selected pizza based on the given pizza ingredients map.

In some embodiments, the non-volatile computer-readable storage medium includes processor-executable instructions for applying a second CNN to automatically score a selected pizza based on a predetermined map of pizza ingredients, including: dividing the pizza portion in the identified best image into a plurality of slices; calculating gradients of one of the plurality of selected pizza slices; repeating the gradient calculation step to calculate gradients of the remaining slices from the plurality of slices; and determining the final score of the selected pizza based on the calculation of the gradients of the plurality of slices.

BRIEF DESCRIPTION OF THE DRAWINGS

Further explanation of the present invention can be made with reference to the accompanying drawings, in which the same structures are indicated by the same numbers in all individual views. The drawings shown are not necessarily drawn to scale, with emphasis generally on illustrating the principles of the present invention. Therefore, the selected structural and functional details disclosed herein should not be interpreted as limiting, but solely as a representative basis for teaching those skilled in the art the various uses of the present invention.

In FIG. 1-9 show block diagrams and/or certain computer architectures that are representative of some exemplary aspects of the present invention, in accordance with at least some principles in at least some embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Among the advantages and improvements that have been disclosed, other objects and advantages of this invention may become apparent from the description below, accompanied by the accompanying figures. This document discloses detailed embodiments of the present invention; however, it should be understood that the disclosed embodiments are merely illustrative of the invention, which may be embodied in various forms. In addition, each of the examples given in the various embodiments of the present invention is for illustrative purposes only and is not restrictive. For example, numerous embodiments and examples of the present invention relate to the scoring of pizzas, however, it is understood that many modifications may become apparent to those skilled in the art, including the application of the principles of the present invention to scoring other foods that may be subjected to visual grading. and scoring (eg cakes, breads and other baked goods; various cooked foods (eg fried chicken), etc.).

Throughout the description, each of the following terms has the same meaning in this document, unless the context clearly states otherwise. The phrases "in one embodiment" and "in some embodiments" as used herein do not necessarily refer to the same embodiment(s), but this is not excluded. In addition, the phrases "in another embodiment" and "in some other embodiments" in this document do not necessarily refer to different embodiments, but this is not excluded. Thus, as described below, various embodiments of the invention can be easily combined without departing from the scope or essence of the invention.

In addition, the term "based on" is non-exclusive and includes the possibility of taking into account additional factors not described, unless the context clearly indicates otherwise. In addition, throughout the description, the meanings of nouns with the indefinite and definite articles include the plural, and the meaning of "in" includes "in" and "on".

It should be understood that at least one object/functionality of the various embodiments described herein may be performed in real time and/or dynamically. As used herein, the term "real time" refers to an event/action that may occur instantly or almost instantly at the time of another event. For example, the phrases "real-time processing", "real-time computing", and "real-time execution" refer to the performance of computing during the actual time of the associated physical processing (for example, during user interaction with an application on a mobile device), to enable the results of the calculation to be used to control this physical processing.

As used herein, the term "runtime" refers to any action that is dynamically determined during the execution of a software application, or at least a portion of a software application.

The term "dynamic (in dynamics)" means the possibility of initiating and/or occurrence of events and/or actions without any human intervention. In some embodiments, events and/or actions in accordance with the present invention may occur in real time and/or at predetermined intervals of at least: nanoseconds, multiple nanoseconds, milliseconds, multiple milliseconds, seconds, multiple seconds, minutes and/or several minutes, hourly, several hours apart, daily, several days apart, weekly and/or monthly, etc.

In some embodiments, the claimed electronic systems are in communication with the electronic mobile devices (e.g., smartphones, etc.) of users and the server(s) in a distributed network environment, allowing data to be exchanged over a suitable data network (e.g., the Internet, etc.). .) using at least one suitable communication protocol (eg IPX/SPX, X.25, AX.25, AppleTalk(TM), TCP/IP (eg HTTP), etc.). In some embodiments, multiple concurrent users (e.g., pizza lines/station) are allowed, including at least 2 (e.g., including 2-10), at least 10 (e.g., including including 10-100), at least 100 (for example, including 100-1000), at least 1000 (for example, including 1000-10000), etc.

In some embodiments, the claimed specially programmed computing systems with associated devices are configured to operate in a distributed network environment that allows data to be exchanged over a suitable data network (for example, the Internet, etc.) using at least one suitable data transfer protocol. (e.g. IO/SPX, X.25, AX25, AppleTalk(TM), TCP/IP (e.g. HTTP), etc.). It should be noted that the embodiments described herein can, of course, be implemented using any suitable hardware and/or computer software languages. However, those skilled in the art are well versed in the type of computer hardware used, the type of computer programming techniques that can be used (e.g., object-oriented programming), and the type of computer programming languages that can be used (e.g., C++, Objective-C). , Swift, Java, Javascript, Python, Perl). Of course, the examples given are illustrative and not restrictive.

The terms "image(s)" and "image data" are used interchangeably herein and serve to identify data representing visual content, which includes, but is not limited to, images encoded in various computer formats (e.g., ".jpg", ".bmp", etc.), video streaming based on various protocols (such as Real Time Streaming Protocol (RTSP), Real Time Transfer Protocol (RTP), Real Time Transmission Control Protocol (RTCP), etc.) .d.), recorded/generated non-streaming video in various formats (e.g. ".mov", ".mpg", ".wmv", ".avi", ".flv" etc.), and visual images in real-time received through the camera applications on a mobile device.

The subject matter disclosed herein may be implemented in software or firmware, or a combination thereof, or as instructions stored on a computer-readable medium that can be read and executed by one or more processors. A computer-readable medium may include any medium and/or mechanism for storing or transmitting information in a machine-readable (eg, computing device) form. For example, the computer-readable medium may include Read Only Memory (ROM); random access memory (RAM); magnetic disk drives; optical media; flash memory devices; electrical, optical, acoustic or other forms of propagated signals (eg carrier waves, infrared signals, digital signals, etc.) and others.

In any of the examples mentioned above, or in other examples, a non-transitory computer-readable medium can be used as another form of computer-readable medium. The exception is when using a transient signal as such. As such a carrier, elements other than elements using the signal as such, with the ability to temporarily store data "for a short time", such as RAM, etc., can be used.

The terms "computer engine" and "engine" identify herein at least one software component and/or a combination of at least one software component and at least one hardware component that is designed/programmed/configured to management/control of other software and/or hardware components (such as libraries, software development kits (SDKs), objects, etc.).

Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), logic gates, registers, semiconductors, chips, microchips, chipsets, etc. In some embodiments, one or more of the processors may be implemented as full instruction set (CISC) or reduced instruction set (RISC) processors, x86 compatible processors, multi-core, or any other microprocessor, central processing unit (CPU), or graphics processor ( GPU). In various embodiments, the one or more processors may be dual core processor(s), dual core mobile processor(s), and so on.

Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, program modules, routines, routines, functions , methods, procedures, programming interfaces, application program interfaces (APIs), instruction sets, computational code, machine code, code segments, machine code segments, words, values, symbols, or any combination thereof. The decision to implement an implementation using hardware and/or software elements may vary depending on a number of factors such as required computational speed, power levels, thermal tolerances, processing cycle budget, input data rates, output data rates, resources. memory, data bus speed, and other design or performance limitations.

In one embodiment, a multiprocessor system may include a plurality of processor chips, each of which includes at least one input/output component that is designed to be directly connected to photonic components for connection to at least an input/output device. output. In some embodiments, the I/O device may be a standard interface, such as a Peripheral Components Defined Interface (PCIe), Universal Serial Bus (USB), Ethernet, Infiniband, and the like. In some embodiments, the implementation of the input/output device may include a storage device.

In one embodiment, a multiprocessor system may include a plurality of photonic components and off-chip memory. Off-chip memory can be shared by more than one of the processor chips. Off-chip memory can be directly coupled to one processor chip and shared with other processor chips using a global memory architecture implemented using a processor-to-processor approach. The multiprocessor system may also include a cache memory and a plurality of processor chips, each of which includes at least one input/output component for direct communication with photonic components for communication with one or more other processor chips. At least one I/O component of at least one of the processor chips may be configured to use a cache coherence protocol. In some embodiments, a cache memory of at least one of the processor chips may be configured to store directory information. In some embodiments, off-chip memory may include DRAM (Dynamic Random Access Memory). In some embodiments, the directory information may be stored in an off-chip memory and an on-chip cache of at least one of the processor chips. In some embodiments, the multiprocessor system may further include a directory subsystem configured with some portion of the subsystem implemented on a high performance chip that is part of the 3D DRAM memory stack. In some embodiments, the multiprocessor system may further include a directory subsystem configured to support a different number of delimiters per memory block. In some embodiments, the multiprocessor system may further include a directory subsystem configured to support a different number of delimiters per memory block using hashing for memory entries for a different number of delimiter pointers. In some embodiments, the multiprocessor system may further include a directory subsystem configured to use hashing to reduce the amount of memory allocated to zero delimiter memory blocks.

One or more objects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium representing various logic in a processor, which, when read by a machine, cause that machine to generate logic for executing the technical solutions described herein. Such representations, known as "IP cores", may be stored on a tangible computer-readable medium and presented to various clients or manufacturing entities for download to manufacturing machines that actually create the logic or processor.

Illustrative examples of claimed, based on CNN,

computer-implemented methods for searching and counting pizza points in a video stream

Illustrative Examples of the General Stated Architecture

In some embodiments, the present invention provides a generic, claimed, CNN-based, computer-implemented system configured to identify and evaluate food production, such as pizza production, based on visual input (eg, video stream, etc.). Figure 1 shows an exemplary general architecture 100 for implementing an exemplary claimed CNN-based computer-implemented system for identifying and evaluating food production, such as pizza production. In some embodiments, the architecture stores the video stream using the Smart Video Saver 1 module described herein with reference to Figures 2A-B and 3A-B. The saved video stream contains one or more video frames Video 1, 2, 3, … N.

In some embodiment, Video 1, 2, 3, ... N frames of a video stream are provided as input to a generic claimed Pizza Detector 2 module, in which the video frames are one or more of the Video 1, 2, 3, ... N video frames that do not have motion blur. due to motion and/or out of focus are processed by the pizza detector 2 to identify one or more of the best potential video frames. In some embodiment, the best potential video frames are then processed by the pizza detector 2 by applying a generic trainable claimed convolutional neural network to identify the best video frame of the best resolution pizza.

In some embodiments, the exemplary claimed Pizza Detector 2 module is configured to classify Video 1, 2, 3, ... N video frames in cases that can be defined as, including:

case 1: no pizza image;

case 2: there is a pizza image, but it is not located in the center of the frame;

case 3: the pizza image is available, located in the center of the frame and has a resolution quality of X;

case 4: there is a pizza image, located in the center of the frame, and has a resolution quality of Y, where Υ is higher than X;

case 5: a pizza image is present, located in the center of the frame, has the required resolution quality, and the pizza is a normal type of pizza;

case 6: a pizza image is present, centered in the frame, has the required resolution quality, and the pizza is a white (ricotta) type of pizza; … etc.

In some embodiments, a typical claimed Pizza Detector 2 module uses a generic claimed CNN trained to classify Video 1, 2, 3, ... N video frames based on predefined cases to identify at least one frame with the best image of a certain type of pizza (e.g. , the image is centered, has the required resolution quality, and the pizza is a white (ricotta) type of pizza). In some embodiments, in the case of pizza video frames for position identification (localization), a typical claimed CNN may be trained at least in part based on the definition of a bounding box 102 and one or more binary masks enabling impairment, for example, in that number, background pixels in the frame.

In some embodiments, one or more binary masks represent data in a binary form used for bit operations, in particular in a bit field. Using a mask allows you to set multiple bits in a byte, nibble, word, and so on. to turn on, turn off, or invert from turn on to turn off (or vice versa) in a single bit operation. In accordance with one embodiment of an exemplary claimed CNN-based computer-implemented system, one or more binary masks are composed of zeros and digits, where each digit indicates that pixel corresponds to a pizza.

In some embodiments, the exemplary claimed Pizza Detector 2 module is configured to use the exemplary claimed CNN to extract the best images (e.g., 10 best images) from Video 1, 2, 3, ... N video frames. In some embodiments, best image extraction is performed on the GPU.

In some embodiments, a typical claimed Pizza Detector 2 module contains two detectors: a typical claimed high speed inaccurate detector and a typical claimed slow slow detector. In some embodiments, a generic high-speed inaccurate detector uses a generic claimed CNN trained to apply a generic claimed support vector machines (SVMs as well as support vector networks) approach and a generic advertised directional gradients (HOG) histogram and configured to extract the best images (e.g., , 10 best images) from Video 1, 2, 3, … N.

In machine learning, a generic claimed SVM uses a generic claimed CNN trained by supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. Given a set of training examples, each marked as belonging to one or the other of two categories, the generic claimed SVM learning algorithm creates a model that assigns new examples to one or the other category, and makes it a non-variant binary linear classifier (despite the existence of methods , such as scaling Plat, t to use SVM in probabilistic classification tuning). The generic claimed SVM model is a representation of examples as points in space, with a mapping to ensure that examples of individual categories are separated by as wide spaces as possible. New examples are then displayed in this space and are assumed to be in a category determined by the side of the space on which the examples fall.

In some embodiments, as used in the present invention, when applying the exemplary claimed CNN, the model is trained on a data set obtained from one or more cameras, such as IP camera 202 in Figure 2 described herein. In some embodiments, the models are trained using SVM.

In some embodiments, the dataset contains approximately 9000 images with markup bounding boxes 102 . In some embodiments, for each pizza, there is a type label to be identified by the Pizza Classifier 3 type claimed module detailed herein. One or more generic declared binary pizza masks are then created for approximately 90 images (approximately 1%) of the 9000 images of the dataset.

In some embodiments, the exemplary claimed HOG is a directional gradient histogram descriptor used in computer vision and image processing for object detection. The Generic Claimed HOG uses a Generic Claimed CNN trained to count gradient orientation occurrences in localized image regions. In some embodiments, a generic claimed HOG uses a generic claimed CNN trained to describe the appearance and shape of a local object in an image using intensity gradient distributions or edge directions. The image is divided into small connected regions called cells, and a histogram of gradient directions is plotted for the pixels in each cell. The descriptor is a concatenation of these histograms. To improve accuracy, local histograms can be contrast-normalized by computing a measure of intensity over a larger region of the image, called a block, and then using that value to normalize all cells in the block. This normalization results in better invariance to lighting and shading changes.

In some embodiments, the exemplary claimed CNN has been trained to apply to the best images to perform fine localization. The exemplary claimed CNN has been further trained to check for blurry or non-blurry images prior to feeding those images to detectors, in accordance with some embodiments.

In some embodiments, the generic claimed low speed fine detector is based on a generic claimed CNN that has been trained to apply to the best images to perform fine localization. The exemplary low speed claimed accurate detector further utilizes a exemplary claimed CNN that has been trained to test whether or not these images are blurred before feeding those images to detectors, in accordance with some embodiments.

In some embodiment, the selected best frame is sent to a generic claimed Pizza Classifier 3 module using a CNN to determine: pizza type and/or crust type, such as, but not limited to, thin crust, traditional crust, and thick crust. Some generic pizzas that can be identified using the generic claimed CNN-based computer-implemented system Pizza Classifier 3 include, but are not limited to, pepperoni, sausage, mushroom, green pepper, with tomatoes, with Canadian bacon, with pepperoni / sausage (only with quality pepperoni), with pepperoni / tomatoes (only with quality pepperoni), with pepperoni / mushrooms (only with quality pepperoni), with pepperoni / bacon (only with quality pepperoni) , with pepperoni/onions (only with quality pepperoni), with pepperoni/Canadian bacon (only with quality pepperoni), with pepperoni/green peppers (only with quality pepperoni), with tomatoes/sausage, with tomatoes/mushrooms, with tomatoes/onions (only with quality tomatoes), with tomatoes/bacon (only with quality tomatoes), with tomatoes/Canadian bacon, with mushrooms/sausage, with mushrooms/bacon (only with quality mushrooms), with mushrooms/Canadian Bacon, Green Peppers/Sausage, Green Peppers/Tomatoes, Green Peppers/Bacon (only with quality green peppers), Green Peppers/Onions (only with quality green peppers), Green Peppers/Canadian Bacon, Canadian Bacon /onions (only with quality Canadian bacon), with Canadian bacon/bacon (only with quality Canadian bacon), with Canadian bacon/sausage.

In some embodiments, the generic claimed Pizza Classifier 3 module uses a generic advertised CNN trained to identify up to 21 types of pizza with a predetermined accuracy. In some embodiments, the accuracy may be at least 80%. In some embodiments, the implementation of the accuracy can reach at least 85%. In some embodiments, the accuracy may be at least 90%. In some embodiments, the accuracy may be at least 95%. In some embodiments, the accuracy can be as high as 80-100%. After determining the type of pizza, the selected best frame is sent to the input of the next CNN of the type claimed module Scorer 4, which uses the advanced CNN architecture. The generic claimed Scorer 4 module uses a generic claimed CNN trained to represent the dough, cheese, and other ingredients map to perform a pizza quality check and scoring based on the dough, cheese, and other ingredients map illustrated herein in Figures 4-5.

Search for pizza in the video stream

In some embodiments, the exemplary claimed CNN-based computer-implemented system may be implemented on one or more cameras installed in the vicinity of the pizza to be searched. FIG. 2A shows an exemplary apparatus 200 for implementing a CNN-based, computer-implemented system for identifying and evaluating food production, such as pizza production.

In some embodiments, the exemplary claimed CNN-based computer-implemented system utilizes an Internet protocol camera or IP camera 202, which is a digital video camera used for surveillance that can send and receive data over a computer network and the internet. Such an IP camera is either centralized (requires a central network video recorder (NVR) to control video recording and playback and alarms) according to some embodiments, or decentralized (no need for an NVR since the camera can record to any local or remote media ) according to some other embodiments. In some embodiments, the camera used has a resolution of 4 megapixels - 1920x1080 (Full High-Definition (HD)) and up to 4 streams H.264/MJPEG, 25 fps. In some embodiments, the camera's viewing angle is at least 32-87° (horizontally) and 18-46° (vertically). In some embodiments, the minimum distance between the camera and an object (such as a pizza) is approximately 1 m.

The above configuration allows a typical claimed CNN-based computer-implemented system to access a web interface and camera settings, access video streaming (on at least one channel) and pizza images at a sufficiently high resolution, in accordance with some embodiments. implementation.

In some embodiments, one or more video cameras 202 are mounted above the cutting table 204 to record activity on the cutting table. In some embodiments, the camera is mounted on the ceiling panel 206 directly above the cutting table 204 with two self-tapping screws. All wiring can be hidden inside the ceiling. The socket for connecting the camera can be located inside the ceiling. The camera 202 is mounted on the ceiling 206 and is directed towards the middle of the cutting board 208 mounted on the cutting table 204. In some embodiments, the camera is mounted at a slight inclination of 2-5 cm.

Figure 2B shows a cutting board 208 mounted on a cutting table 204 in an exemplary claimed apparatus as shown in Figure 2A, in accordance with some embodiments. In some embodiments, the cutting board 208 is secured to the table 204 by aluminum rods 210a-c, which may be hidden within the cutting board. The rods 210a-c can be glued using, among other things, a secure one. The black circle 212 on the cutting board 208 in Figure 2A represents a pizza. Camera 202 is configured to capture pizza placed anywhere on cutting board 208.

In some embodiments, it is possible to change the network settings according to the selected network configurations or the selected external address of each restaurant.

In the case where the duration of the capture activity exceeds a predetermined period of time, such as 3 seconds, such a video frame is uploaded to the server and then processed by the search algorithm. In some embodiments, each camera can store up to 1-1500 short video frames per day. In some embodiments, each camera can store at least 500 short video frames per day. In some embodiments, each camera can store at least 1000 short video frames per day.

In some embodiments, the search algorithm is performed by the generic claimed Pizza Detector 2 module in Figure 1 using the generic claimed CNN. All video frames without motion blur and/or out of focus are processed by the generic claimed CNN for the decision procedure.

In some embodiments, the search algorithm is performed using a typical claimed high speed inaccurate detector as part of a typical claimed module Pizza Detector 2 in figure 1. All video frames without motion blur and/or out of focus are processed by a high speed detector based on typical claimed HOG descriptors with using the generic claimed linear SVM as the decision procedure.

The best video frame set N is selected by a confidence value where N is an integer other than 0 (eg, 3, 5, 10, 15, etc.). In some embodiment, N can be any integer in the range 1-20. In some embodiment, N may be at least 5. In some embodiment, N may be at least 10. In some embodiment, the confidence is 95%. In some embodiment, the confidence may be changed after retraining the algorithm.

In some embodiments, the N best video frame set is processed and selected using the exemplary claimed Pizza Detector 2 module in Figure 1 using CNN. The best video frame is selected from a set with a confidence greater than a predetermined confidence value, such as 95%. If there is no video frame with a certainty greater than this predetermined certainty value, the algorithm terminates.

In some embodiments, the N best video frame set is processed and selected using a typical claimed low speed accurate detector within the generic claimed Pizza Detector 2 module in Figure 1. The best video frame is selected from the set with a confidence greater than a predetermined confidence value, such as 95%. If there is no video frame with a certainty greater than this predetermined certainty value, the algorithm terminates.

Pizza scoring

Figure 3A shows an exemplary selected best frame of a pizza image 300, which may be further tested by a CNN-based, computer-implemented system, in accordance with some embodiments. In some embodiments, the type and size of the pizza is determined from the selected frame 300 using an exemplary claimed CNN-based computer-implemented system. In some embodiments, the size is determined by the diameter of the pizza in the selected frame in pixels.

In some embodiments, the implementation of the Pizza Classifier 3 module in figure 1 uses a generic claimed CNN trained to determine the type of pizza, as noted above with reference to figure 1. If it is not possible to reliably recognize the type of pizza, the algorithm terminates.

The selected frame is then routed to the input of the next CNN, driven by the Scorer 4 module in Figure 1, using the generic advertised CNN trained to perform pizza image segmentation, and thus output dough/crust, cheese, and toppings maps. The dough/crust, toppings, cheese, and doneness of the pizza are tested and each of these components is scored as described in this document. The overall score, which is a combination of the dough/crust, toppings, cheese, and doneness scores, forms an overall pizza quality score. In some embodiments, the Scorer 4 module in Figure 1 uses a generic advertised CNN trained to score the quality of a pizza in terms of an ingredient map.

A typical method for processing a portion of pizza in the image contains: 1) determining the crust; 2) determining the flow of cheese, as described in this document; 3) determination of the amount of fillers in the cheese edge; 4) assessment of the pizza dough as a result of the pizza inspection and determination of the color of the dough; and 5) scoring each pizza slice in the image to determine the height of the rise of the crust and its size.

In accordance with some embodiments of the exemplary CNN-based computer-implemented system, during the scoring or checking process, the pizza circle representing the pizza slice in the best image identified by the exemplary CNN claim is defined by the outer circle 302. The cheese edge is defined by the circles 304 and 306 with a tolerance of about ±3 mm. The portion of pizza on the images is divided into 8 equal slices/sectors 1-8, as shown in figure 3A. In some embodiments, the pizza circle is divided into N equal slices/sectors, where N is a non-zero integer. In some embodiments, each sector is evaluated independently of the others.

According to the pizza ingredients map, a typical pizza outline can be defined in more detail by the lines shown in Figure 3B. As shown in figure 3B, NTZ shows a zone without fillers. Line 1 defines the maximum diameter circle outside the pizza circle. Regions 2 define the inner margin and outer margin of the pizza portion, in accordance with one embodiment. Plot 3 defines the outer edge of the pre-baked pizza. In some embodiments, this outer edge has some tolerance. In some embodiment, this tolerance may be, including, ±3 mm. Plot 4 defines the boundary of the pizza sauce. In some embodiments, boundary 4 may have a tolerance on the order of ±3 mm. Line 5 shows the standard outer circumference of a pizza circle. Plot 6 shows the allowable standard pizza edge radius. In some embodiment, the allowable standard pizza edge radius is 5 cm. Region 7 defines the cheese edge line. In some embodiments, the implementation of the cheese edge 7 has some tolerance. In some embodiment, this tolerance is ±3 mm. The threshold value of the allowable tolerance of the order of ±3 mm, provide the possibility of practical determination of the size. In particular, the radius of the concentric circles varies within an ideal radius of ±3 mm.

According to one embodiment, the pizza is initially scored at 10 points, and then the score is reduced due to penalty points awarded during the evaluation process of each slice of the pizza.

In some embodiments, an exemplary advertised CNN has been trained to perform a pizza crust test to determine the rise height of the crust and its size. The crust check includes, among other things, a size check and a check for the cleanliness of the crust.

In accordance with some embodiments, a generic claimed CNN has been trained to perform a size check against a pizza ingredients map containing images with one or more binary masks. For example, the center of pizza is defined as the center of mass of the pizza components map. For example, the size of a pizza (for example, 23 cm, 30 cm, 35 cm, 40 cm, etc.) is determined by the number of pixels of the pizza image in the frame.

Figure 4A shows images of a generic pizza from which a generic claimed CNN was trained to estimate based on the outer circumference of the pizza circle along line 5, with a tolerance of ±Z mm (e.g., 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1-15), in accordance with some embodiments. In the case when more than Υ% (for example, 5%, 10%, 15%, 20%, 25%, etc.) of the contour of a certain sector is located outside the boundaries of concentric circles, a penalty of X points is charged for this sector (for example , 0.1).

Some of the scoring standards applied by the generic claimed CNN are shown in Figure 4B, in which the top slices have an acceptable crust size, the bottom left slices illustrate a small crust size, and the bottom right slices illustrate typical large crust sizes.

In some embodiments, the exemplary claimed CNN has been trained to perform a crust cleanliness check. The position of the crust is determined by the center of the pizza and its radius, in accordance with some embodiment. In the case when more than 50% of the crust area of a sector is covered with cheese or filler, a penalty of 0.1 points is charged for this sector. Some scoring standards were applied using the generic claimed CNN shown in Figure 4C, in which slices on top have acceptable crust clarity, slices on the bottom left illustrate typical soiled crusts, and slices on the bottom right illustrate typical clean crusts.

In some embodiments, the exemplary claimed CNN has been trained to perform a pizza topping test to determine if the toppings and cheese are distributed evenly or unevenly.

Some typical distributions of toppings on pizza slices are shown in Figure 4D, in which the top slices have an even distribution of toppings and the bottom slices have an uneven distribution of toppings.

In some embodiment, during the filling process, each of these eight sectors is divided by a circle into two parts: an inner field and an outer field, shown as sections 2 in figure 3B, as noted herein. The number of pixels for each type of filler is calculated for each inner margin and outer margin separately. In the event that the number of some fillers on the inner field or on the outer field is less than specified in the rules, a penalty of 0.25 points is charged for this section of the sector. The minimum number of toppings for each type and size of pizza is set in the process of learning the algorithm (in accordance with the conditions for minimizing errors on the training set).

In some embodiments, an exemplary claimed CNN has been trained to perform a pizza cheese test to determine whether cheese is evenly or unevenly distributed when there is enough cheese, ie, no red edge. Some typical cheese distributions on pizza slices are shown in Figure 4E, in which the top slices have the desired distribution. The slice in the middle has an invalid distribution, and the slice at the bottom does not have the required distribution, but has a valid distribution.

In some embodiment, a typical claimed CNN has been trained to check the border between the crust and the cheese: at least 75% of the cheese must be located inside the fill zone of the smaller concentric circle circle or on the inside field (and not exceed the border by more than 3 mm, for example in otherwise), a penalty of 0.1 points is charged for this sector, which can be up to 0.8 points for the entire pizza).

In the event that less than 75% of the unfilled area in each segment of each inner field and outer field is covered with cheese (excessive crust), a penalty of 0.1 points is awarded, which can be up to 1.6 points for the entire pizza. Burnt/half-baked/cheese and perforated cheese sectors also incur a penalty of 0.1 points, which can be as high as 0.8 for the whole pizza.

In some embodiments, the exemplary claimed CNN has been trained to perform a pizza crust readiness test to determine whether the crust is done to the extent that the cheese is completely melted, no burnt toppings are present, and the crust is golden brown.

Some exemplary pizza slices in varying degrees of doneness are shown in Figure 4F, in which the top left slices are burnt, the top right slices are normal/good, and the bottom slices are uncooked.

In some embodiments, the pizza dough/crust is judged by the color of both the top of the pizza and the bottom of the pizza. Figure 4G shows a generic color map that is the standard for making decisions and calculating pizza doneness gradients. The color chart consists of an upper table and a lower table showing different colors that indicate the different degrees of doneness of the top and bottom of the pizza, respectively, and the corresponding scores for each range of colors.

Figure 4H shows a typical perfectly made pizza with the outer circumference at the standard outer edge (thin [red] line) and toppings and cheese uniformly distributed over 16 areas defined by this outer edge and the cheese edge (thick [red] line) .

Figure 5 is an exemplary table 500 showing the calculation of a typical pizza's overall score from the scores computed from a crust check, toppings check, cheese check, and crust readiness check performed with a typical claimed CNN in accordance with some embodiments.

In the example described herein in Figure 5, each of the Total Crust Scores, Total Fillers Scores, Total Cheese Scores, and Total Crust Readiness Scores is calculated according to the rules as outlined above with reference to Figure 4.

The overall pizza score is the sum of the scores obtained from the crust test, toppings test, cheese test, and crust readiness test. In the example described herein in Figure 5, total score (6.65) = total crust score (1.4) + total filler score (1.75) + total cheese score (2.3) + total crust readiness score (1,2).

Figure 6A shows a typical claimed scoring architecture 600 that uses a generic claimed CNN trained to implement the methods described above, in accordance with some embodiments. At block 602, the selected best pizza image is provided as input to the exemplary claimed scoring architecture. Here, the scoring procedure comprises two main steps: a segmentation operation step 604 and a gradient calculation/scoring step 606 (both magnified in FIG. 6B).

Step 604 is to segment the pizza image. In some embodiment, a generic advertised U-net CNN has been trained to perform this step. In accordance with an embodiment, in sub-step 604a, the generic claimed scoring architecture 600 uses a generic claimed U-net CNN trained to segment a pizza image into N slices, where N is a non-zero integer. In some embodiment, N = 8, and these eight slices represent a specific layer for further processing to determine the appropriate index for each segmented slice.

In some embodiment, a typical claimed U-net CNN has been trained to implement a network consisting of a compression path and an extension path that give the network a U-shaped architecture. Along the typical claimed U-net CNN compression path, there is a reduction in spatial information and an increase in feature information.

In particular, the exemplary claimed compression path is a convolutional network, which consists of reapplying convolution and activation layers (sub-steps 604b, 604d and 604e). In some embodiments, after the first sub-step 604a of the convolution and activation layers, sub-step 604 c of the downsampling and batch normalization layers is applied.

In some embodiments, each Type Notified Convolution and Activation layer is followed by a Type Notified Rectified Linear Units (ReLU) layer and a Type Notified subsampling layer before proceeding to the next convolution and activation layer. The generic claimed ReLU layer uses a generic claimed U-net CNN trained to apply a non-saturable activation function f(x) = max(0, x). This enhances the non-linear properties of the decision function and the overall network without affecting the receptive fields of the typically claimed convolution and activation layer.

In some embodiments, a generic claimed subsampling layer uses a generic claimed U-net CNN trained to represent a non-linear subsampling shape by combining the outputs of neural clusters in one layer into a single neuron in the next layer. In some embodiments, a typical claimed U-net CNN can be trained to implement a generic claimed maximization subsampling of dividing the input image into a set of non-overlapping rectangles and outputting, for each such subregion, the maximum value from each cluster of neurons in the previous layer. The exemplary claimed subsampling layer serves to gradually reduce the spatial size of the representation, reduce the number of parameters and the amount of computation in the network, and hence also to control overfitting.

After the generic advertised compression path, the generic advertised CNN U-net proceeds to the generic advertised extension path and presents a fully connected layer in substep 604f, in accordance with some embodiment. In the expansion process, a generic claimed U-net CNN was trained to combine the feature and spatial information through a sequence of upconvolutions and concatenations with high resolution features from the compression path.

In some embodiments, fully connected layers consist of neurons in a typical fully connected layer claimed and have connections to all activation layers in previous layers. In accordance with an embodiment, the neurons of the generic claimed fully connected layers correspond to a map of pizza ingredients based on the classification of each image pixel performed by the generic claimed U-net CNN. The pizza ingredient list includes crust, cheese, and toppings such as pepperoni and ham.

Table 1 shows an exemplary declared data schema for an exemplary segmentation model implemented at 604, as described herein.

Using the ingredient map obtained in step 604, in step 606, a generic claimed CNN U-net was trained to apply the generic scoring rules as detailed herein to calculate pizza gradients, resulting in a single slice eight slice scoring column. pizza. The output results of computing gradients from a typical reported CNN U-net for one slice of pizza are further illustrated with magnification in Figure 6C.

After computing the gradients of all slices, a typical claimed CNN U-net outputs a matrix with eight columns, each corresponding to each of the eight slices, at step 608 (expanded and enlarged in Figure 6D). The claimed CNN U-net then post-processes this matrix using statistical models to improve the scoring at step 610 (expanded in Figure 6E). At step 612 (expanded in Figure 6F), a typical claimed CNN U-net outputs an improved scoring matrix in binary form consisting of only 0s and 1s. From this improved scoring matrix, a final score is determined at step 614.

Figure 7 is a diagrammatic snapshot of a workflow 700 for identifying and evaluating food production, such as pizza production, using the exemplary claimed CNN described herein, in accordance with some embodiments. This process begins at block 702, where a video stream containing at least a video frame of a pizza image is stored using the Smart Video Saver 1 module, as noted with reference to Figure 1. At block 704, it is determined whether each of the video frames contains a video stream image of pizza or not. If there is no pizza image in the video frame, at block 706, the video frame is discarded. If there is a pizza image in the video frame, at step 708, pizza video frames are collected and used to find the best images. At step 710, a set of video frames with the best pizza image is created, selected from all pizza video frames. In some embodiment, the creation of the highest rated pizza video frame set is performed using the exemplary claimed high speed inaccurate detector within the exemplary claimed Pizza Detector 2 module, as noted with reference to Figure 1. In an embodiment, the creation of the highest rated pizza video frame set , includes, among other things, determining the blurring of video frames due to movement or defocus. In step 712, the set of selected best pizza video frames is processed based on the first trained generic claimed CNN to identify the best pizza image from the highest rated pizza video frames, locate the pizza within the best pizza image boundaries, and determine the type of pizza from the selected best image. In some embodiment, identifying the best pizza image is performed using a typical low speed claimed accurate detector as part of a generic claimed Pizza Detector 2 module, as noted with reference to Figure 1. In some embodiment, pizza localization includes defining a bounding box and one or more binary masks on the image.

After determining the type of pizza, the process then performs automatic segmentation of the pizza image using the second trained Generic Claimed CNN at 714. In some embodiments, this segmentation is performed using a Generic Claimed Scorer 4 module, as noted with reference to Figure 1. In some embodiments, this segmentation defines a map of pizza ingredients such as dough, cheeses, and toppings. Then, at step 716, automatic scoring is performed on this pizza ingredients map. In some embodiments, scoring is performed using the exemplary claimed Scorer 4 module, as noted with reference to Figure 1.

Figure 8 is a diagrammatic snapshot of an automatic scoring workflow 800 with an implementation of the second trained generic claimed CNN described herein, in accordance with some embodiments. At block 802, the selected best pizza image is provided as input to the scoring process. Next, at step 804, a transition is made to dividing this pizza image into N slices, where N is an integer other than 0. In an embodiment, N = 8. At step 806, one of the N slices is processed to define a pizza component map by applying a generic claimed compression path and a generic claimed expansion path of a second trained generic claimed CNN, in accordance with some embodiments.

In some embodiment, a typical claimed compression path consists of repeated applications of convolution and activation layers. In some embodiments, a downsampling and batch normalization layer may be applied following the first convolution and activation layer. In some embodiments, each Type Notified Convolution and Activation layer is followed by a Type Notified Rectified Linear Units (ReLU) layer and a Type Notified subsampling layer before proceeding to the next convolution and activation layer.

In some embodiment, over a typical claimed expansion path, the process combines the feature and spatial information through a sequence of upconvolutions and concatenations with high resolution features from a typical claimed compression path.

Using the pizza ingredients map from block 806, at block 808, the exemplary claimed process may apply one or more of the pizza scoring rules detailed herein to compute slice gradients. Steps 806-808 are then repeated at step 810 for computing the gradients of the remaining pizza slices. Finally, at step 812, a final score is determined from the gradients of all pizza slices.

Figure 9 shows an exemplary claimed CNN-based computer-implemented system 900 trained to search and score a food item 902, such as pizzas made by restaurants. Video frames Video 1, 2, 3, ... N of the video stream are captured by one or more cameras: video camera A 904a, video camera B 904b ... video camera N 904n (for example, IP camera 202 in figure 2A), as noted in this document in accordance with some options implementation. The video stream is presented as input 906 and transmitted through the network 908 for further search and scoring processes. In some embodiments, video frames Video 1, 2, 3, ... N of the video stream are stored in storage device 910 for later retrieval. The storage device 910 may include, but is not limited to, a local storage device or a remote storage medium and/or a database. In some embodiments, input 906 is transmitted via network 1008 to server 912 for processing. The server 912 selects the best pizza image from the frames of the video stream and locates the pizza slice in the best image. The best image is then processed by the scoring server 912, the results of which are transmitted as output information 914. In some embodiment, it is possible to store the output information 914 in the memory device 910 for later retrieval to control and/or improve food production processes.

In some embodiments, the present invention relates to exemplary pizza retrieval and scoring computer systems and methods supported by video cameras and implemented with advanced convolutional neural network (CNN) algorithms that provide the benefits of accurate pizza scoring and thus improved accuracy. , efficiency and quality of pizza production processes.

In some embodiments, the present invention relates to an exemplary convolutional neural network-based computer-implemented method for identifying and evaluating food production, such as pizza production, comprising: receiving, by means of a processor, a continuous video stream from at least one camera positions above a table configured to receive ready-made pizzas; collecting by the processor a plurality of pizza video frames containing video frames of the selected pizza from the video stream; applying, by the processor, the first CNN to select a set of video frames of the selected pizza pizza with the highest rating from the plurality of pizza video frames; applying, by the processor, the first CNN to identify an image of the selected highest rated pizza pizza from the set of highest rated pizza video frames; using the processor on the first CNN to localize at least one portion of the selected pizza in the identified image with the highest rated pizza; using the first CNN with the processor to determine the type of pizza selected from the identified image with the highest rated pizza; applying, by the processor, a second CNN to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and using the second CNN by the processor to automatically score the selected pizza based on the predetermined pizza ingredients map.

In some embodiments, the processor's application of a second CNN to automatically score a selected pizza based on a given pizza component map includes: dividing the pizza slice from the identified best image into a plurality of slices by the processor; calculating with the gradient processor one of the plurality of slices of the selected pizza; repeating by the processor the step of calculating the gradients of the remaining slices from the plurality of slices; and determining, with the processor, a final score of the selected pizza based on the calculation of the gradients of the plurality of slices.

In some embodiments, video frames of a video stream are categorized in cases including:

i) first case for images without pizza;

ii) second case for non-centered pizza images;

iii) the third case for centered pizza images and a resolution quality of X;

iv) the fourth case for centered pizza images and a resolution quality of Y, where Υ is higher than X;

v) the fifth case for centered pizza images of the first type and the required resolution quality; and

vi) the sixth case for centered pizza images of the second type and the required resolution quality.

In some embodiments, the application of the first CNN to select the highest rated set of pizza video frames from the pizza video frames and identify the best image is performed by a graphics processing unit (GPU).

In some embodiments, applying the first CNN to select the highest rated set of pizza video frames from the pizza video frames is performed with a high speed imprecise detector.

In some embodiments, selecting the highest rated set of pizza video frames includes discounting each pizza video frame with motion blur and/or defocus blur.

In some embodiments, the application of the first CNN to identify the best image is performed with an accurate low speed detector.

In some embodiments, the number of elements of the slice set is 8.

In some embodiments, the second CNN has a compression path and an extension path.

In some embodiments, the compression path includes a plurality of convolution and activation layers.

In some embodiments, the compression path further includes a downsampling and batch normalization layer following the first convolution and activation layer.

In some embodiments, the compression path further includes a Rectified Linear Units (ReLU) layer and a subsampling layer following each convolution and activation layer before moving on to the next convolution and activation layer.

In some embodiments, the extension path includes a sequence of upconvolutions and concatenations configured to combine spatial feature information with high resolution features from the compression path.

In some embodiments, applying the first CNN to localize at least one serving of a particular pizza in the identified highest rated pizza image includes: defining a bounding box; and using one or more predefined binary masks.

In some embodiments, the present invention relates to an exemplary convolutional neural network-based, computer-implemented system for identifying and evaluating food production, such as pizza production, comprising: at least one image capture device; long-term storage device; one or more processors; and computer program code stored in the long-term memory, causing, when executed by one or more processors, the one or more processors to: receive a continuous video stream from at least one camera position above a table configured to receive finished pizzas; collecting a plurality of pizza video frames containing video frames of the selected pizza from the video stream; applying the first CNN to select a set of video frames of the selected pizza pizza with the highest rating from the plurality of pizza video frames; applying the first CNN to identify an image of the selected pizza with the highest rated pizza from the set of video frames of the highest rated pizza; applying the first CNN to localize at least one portion of the selected pizza in the identified image with the highest rated pizza; applying the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; applying the second CNN to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and applying the second CNN to automatically score the selected pizza based on the given pizza ingredients map.

In some embodiments, the computer program code includes instructions for: applying a second CNN to automatically score a selected pizza based on a given map of pizza ingredients, including: dividing the pizza portion in the identified best image into a plurality of slices; calculating gradients of one of the plurality of selected pizza slices; repeating the gradient calculation step to calculate gradients of the remaining slices from the plurality of slices; and determining the final score of the selected pizza based on the calculation of the gradients of the plurality of slices.

In some embodiments, the computer program code includes instructions for applying the first CNN on the GPU to select the highest rated set of pizza video frames from the pizza video frames and identify the best image.

In some embodiments, the computer program code includes instructions for using the first CNN to select the highest rated set of pizza video frames from the pizza video frames performed by the high speed inaccurate detector.

In some embodiments, the computer program code includes instructions for using the first CNN to identify the best image, performed with an accurate low speed detector.

In some embodiments, the present invention relates to an exemplary convolutional neural network-based, computer-realizable, durable, computer-readable storage medium for identifying and evaluating food production, such as pizza production, comprising processor-executable instructions for: receiving a continuous a video stream from at least one camera position above a table configured to receive finished pizzas; collecting a plurality of pizza video frames containing video frames of the selected pizza from the video stream; selecting a set of video frames of the selected pizza with the highest rated pizza from the plurality of pizza video frames; applying the first CNN to identify an image of the selected pizza with the highest rated pizza from the set of video frames of the highest rated pizza; applying the first CNN to localize at least one serving of the selected pizza in the identified image with the highest rated pizza; applying the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; applying a second CNN to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and applying the second CNN to automatically score the selected pizza based on the given pizza ingredients map.

In some embodiments, the non-volatile computer-readable storage medium includes processor-executable instructions for applying a second CNN to automatically score a selected pizza based on a predetermined map of pizza ingredients, including: dividing the pizza portion in the identified best image into a plurality of slices; calculating gradients of one of the plurality of selected pizza slices; repeating the gradient calculation step to calculate gradients of the remaining slices from the plurality of slices; and determining the final score of the selected pizza based on the calculation of the gradients of the plurality of slices.

Publications cited in this document are incorporated herein by reference in their entirety. The various aspects of the invention have been illustrated above with reference to examples and embodiments. However, it is clear that the scope of the invention is not defined by the above description, but by the following claims, properly construed in accordance with the principles of patent law. In addition, many modifications may become apparent to those skilled in the art, including the possibility of using various embodiments of the claimed methodologies of the claimed systems and claimed devices described herein in any combination with each other. In addition, it is possible to perform the various steps in any desired order (and add any desired steps and/or eliminate any undesirable steps in the chosen embodiment).

Claims (94)

1. A method for processing a video stream for identifying and assessing the quality of pizza production, comprising: receiving by the processor a continuous video stream from at least one camera position above a table configured to receive finished pizzas; collecting by the processor a plurality of pizza video frames containing video frames of the selected pizza from the video stream; using the processor with a first convolutional neural network (CNN) to select a set of video frames of the selected pizza with the highest rated pizza from the plurality of pizza video frames; applying, by the processor, the first CNN to identify an image of the selected pizza with the highest rated pizza from the set of video frames of the highest rated pizza; using the processor on the first CNN to localize at least one portion of the selected pizza in the identified image with the highest rated pizza; using the first CNN with the processor to determine the type of pizza selected from the identified image with the highest rated pizza; using the processor with a second convolutional neural network (CNN) to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and applying, by means of a processor, a second CNN to automatically score a selected pizza based on a given map of pizza ingredients, comprising: dividing, by the processor, the pizza portion from the identified best image into a plurality of slices; calculating with the gradient processor one of the plurality of slices of the selected pizza; repeating by the processor the step of calculating the gradients of the remaining slices from the plurality of slices; and determining, by means of the processor, a final score of the selected pizza based on the calculation of the gradients of the plurality of slices. 2. The method according to claim 1, characterized in that the video frames of the video stream are categorized in cases containing: i) first case for images without pizza; ii) second case for non-centered pizza images; iii) the third case for centered pizza images and with a resolution quality of X; iv) the fourth case for centered pizza images and with a resolution quality of Y, where Υ is higher than X; v) the fifth case for centered pizza images of the first type and with the required resolution quality; and vi) the sixth case for centered pizza images of the second type and with the required resolution quality. 3. The method of claim 1, wherein selecting the highest rated set of pizza video frames further comprises discounting each pizza video frame with motion blur and/or defocus blur. 4. The method according to claim 1, characterized in that the number of elements of the set of slices is 8. 5. The method of claim 1, wherein the second CNN has a compression path and an expansion path. 6. The method of claim 5, wherein the compression path contains a plurality of convolution and activation layers. 7. The method of claim 6, wherein the compression path further comprises a subsampling and batch normalization layer following the first convolution and activation layer. 8. The method of claim 6, wherein the compression path further comprises a Rectified Linear Units (ReLU) layer and a subsampling layer following each convolution and activation layer before proceeding to the next convolution and activation layer. 9. The method of claim 5, wherein the extension path comprises a sequence of upconvolutions and concatenations configured to combine spatial feature information with predetermined resolution features from the compression path. 10. The method of claim 1, wherein applying the first CNN to localize at least one serving of the selected pizza in the identified best pizza image further comprises: defining a bounding box; and using one or more predefined binary masks. 11. A video stream processing system for identifying and evaluating the quality of pizza production, comprising: at least one image capture device; long-term storage device; one or more processors; and computer program code stored in this long-term storage device, which, when executed by one or more processors, causes one or more processors to execute: receiving by the processor a continuous video stream from at least one camera position above a table configured to receive finished pizzas; collecting by the processor a plurality of pizza video frames containing video frames of the selected pizza from the video stream; using a first convolutional neural network (CNN) processor to select a set of video frames of the selected pizza pizza with the highest rating from the plurality of pizza video frames; using the processor of the first CNN to identify an image of the selected pizza with the highest rated pizza from the set of video frames of the highest rated pizza; using the processor on the first CNN to localize at least one portion of the selected pizza in the identified image with the highest rated pizza; applying by the processor the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; using a second convolutional neural network (CNN) processor to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and application by the processor of the second CNN to automatically calculate the points of the selected pizza based on the given map of pizza ingredients, containing: dividing, by the processor, the pizza portion from the identified best image into a plurality of slices; calculating with the gradient processor one of the plurality of slices of the selected pizza; repeating by the processor the step of calculating the gradients of the remaining slices from the plurality of slices; and determining, by means of the processor, a final score of the selected pizza based on the calculation of the gradients of the plurality of slices. 12. The system according to claim 11, characterized in that the video frames of the video stream are categorized in cases containing: i) first case for images without pizza; ii) second case for non-centered pizza images; iii) the third case for centered pizza images and with a resolution quality of X; iv) the fourth case for centered pizza images and with a resolution quality of Y, where Υ is higher than X; v) the fifth case for centered pizza images of the first type and with the required resolution quality; and vi) the sixth case for centered pizza images of the second type and with the required resolution quality. 13. The system of claim 11, wherein the instructions for selecting the highest rated set of pizza video frames further comprise discounting each pizza video frame with motion blur and/or defocus blur. 14. The system according to claim 11, characterized in that the number of elements of the set of slices is 8. 15. The system of claim 11, wherein the second CNN has a compression path and an expansion path. 16. The system according to claim 15, characterized in that the compression path contains a plurality of convolution and activation layers. 17. The system of claim 16, wherein the compression path further comprises a subsampling and batch normalization layer following the first convolution and activation layer. 18. The system of claim 16, wherein the compression path further comprises a Rectified Linear Units (ReLU) layer and a subsampling layer following each convolution and activation layer before proceeding to the next convolution and activation layer. 19. The system of claim 15, wherein the extension path comprises a sequence of upconvolutions and concatenations configured to combine spatial feature information with predetermined resolution features from the compression path. 20. The system of claim. 11, characterized in that the use of the first CNN to localize at least one serving of the selected pizza in the identified best pizza image further comprises: defining a bounding box; and using one or more predefined binary masks. 21. A long-term computer-readable storage medium containing instructions executable by the processor for: receiving by the processor a continuous video stream from at least one camera position above a table configured to receive finished pizzas; collecting, by the processor, a plurality of pizza video frames containing video frames of the selected pizza from the video stream; using a first convolutional neural network (CNN) processor to select a set of video frames of the selected pizza with the highest rated pizza from the plurality of pizza video frames; applying by the processor the first CNN to identify an image of the selected pizza with the highest rated pizza from the set of video frames of the highest rated pizza; using the processor on the first CNN to locate at least one portion of the selected pizza in the identified image with the highest rated pizza; applying by the processor the first CNN to determine the type of pizza selected from the identified image with the highest rated pizza; applying, by means of the processor, a second convolutional neural network (CNN) to define a component map of the selected pizza by automatically performing image segmentation of the pizza portion based on at least the type of pizza; and application by the processor of the second CNN to automatically calculate the points of the selected pizza based on the given map of pizza components, containing: dividing, by the processor, the pizza portion from the identified best image into a plurality of slices; calculating with the gradient processor one of the plurality of slices of the selected pizza; repeating by the processor the step of calculating the gradients of the remaining slices from the plurality of slices; and determining, by means of the processor, a final score of the selected pizza based on the calculation of the gradients of the plurality of slices. 22. A long-term computer-readable storage medium according to claim 21, characterized in that the video frames of the video stream are categorized in cases containing: i) first case for images without pizza; ii) second case for non-centered pizza images; iii) the third case for centered pizza images and with a resolution quality of X; iv) the fourth case for centered pizza images and with a resolution quality of Y, where Υ is higher than X; v) the fifth case for centered pizza images of the first type and with the required resolution quality; and vi) the sixth case for centered pizza images of the second type and with the required resolution quality. 23. The durable computer-readable storage medium of claim 21, wherein the processor-executable instructions for selecting the highest rated set of pizza video frames further comprise devaluing each pizza video frame with motion blur and/or defocus blur. 24. A long-term computer-readable storage medium according to claim 21, characterized in that the number of elements of the plurality of slices is 8. 25. The durable computer-readable storage medium of claim 21, wherein the second CNN has a compression path and an expansion path. 26. The durable computer-readable storage medium of claim 25, wherein the compression path comprises a plurality of convolution and activation layers. 27. The durable computer-readable storage medium of claim 26, wherein the compression path further comprises a subsampling and batch normalization layer following the first convolution and activation layer. 28. The durable computer-readable storage medium of claim 26, wherein the compression path further comprises a Rectified Linear Units (ReLU) layer and a subsampling layer following each convolution and activation layer before moving on to the next convolution and activation layer. 29. The durable computer-readable storage medium of claim 25, wherein the extension path comprises a sequence of upconvolutions and concatenations configured to combine spatial feature information with predetermined resolution features from the compression path. 30. The durable computer-readable storage medium of claim 21, wherein the processor-executable instructions for using the first CNN to localize at least one serving of the selected pizza in the identified best pizza image further comprise: defining a bounding box; and using one or more predefined binary masks.

Images