RU2819619C2 - Systems and methods of training generative-adversarial networks, as well as use of trained generative-adversarial networks - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to computer systems for training generative adversarial networks. System for training a generative adversarial network using medical images including representations of a feature of interest, comprising: at least one memory unit and at least one processor, capable of performing operations, including: providing a first plurality of images and using the first plurality of images and indicators of the feature of interest, training the object detection network to detect the feature of interest, providing a second plurality of images, which include representations of the feature of interest, wherein the second plurality of images comprises a greater number of images than the first plurality of images, involvement of the network for detecting the trained objects to the second plurality of images, providing manual installation of checks for true and false positives with respect to the first plurality of detections, training the generative adversarial network and retraining the generative adversarial network using at least one additional set of images and detections of the feature of interest.

EFFECT: high accuracy of analysing medical images.

20 cl, 10 dwg

Description

TECHNICAL FIELD

[01] The present invention generally relates to the field of neural networks and the use of such networks for image analysis and object detection. More specifically and without limitation, this invention relates to computer systems and methods for training and using generative adversarial networks. The systems and methods described herein, as well as trained neural networks, can be used in a variety of vision applications and systems, such as medical image analysis and systems that benefit from accurate object detection capabilities.

BACKGROUND OF THE ART

[02] In many object detection systems, an object is detected in an image. The object of interest can be a person, place or thing. In some applications, such as medical image analysis and diagnostics, object location is also important. However, computer systems using image classifiers typically cannot identify or provide the location of a detected object. Accordingly, existing systems that only use image classifiers are not very useful.

[03] Additionally, learning methods for object detection can rely on manually annotated training sets. Such annotations take a long time when the detection network being trained is based on bounding box, such as You Only Look Once (YOLO) architecture, Single Shot Detector (SSD) architecture, etc. Accordingly, large datasets are difficult to annotate for training, which often results in the neural network being trained on a smaller dataset, which reduces accuracy.

[04] For computer systems, existing medical images are typically built on a single network of detectors. Accordingly, once detection is completed, the network simply transmits the detection data to, for example, a physician or other healthcare professional. However, such a detection may be false positive, for example, the absence of polyps during endoscopy, etc. Such systems do not provide a separate network for distinguishing false positives from true positives.

[05] In addition, neural network-based object detectors typically pass features identified by the neural network to the detector, which may contain a second neural network. However, such networks are often inaccurate because feature detection is performed by a generalized network, with only part of the detector being specialized.

[06] Finally, many existing object detectors operate with a delay. For example, medical images can be captured and stored before analysis. However, some medical procedures, such as endoscopy, are performed in real time. Consequently, these systems are usually difficult to apply in real time.

DISCLOSURE OF INVENTION

[07] In view of the above, embodiments of the present invention provide computer-implemented systems and methods for training a generative adversarial network and using them for applications such as medical image analysis. The systems and methods of the present invention provide advantages over existing systems and methods, including improved object detection and location information.

[08] According to some embodiments, a computer system is provided comprising an object detection network that identifies features of interest (i.e., anomalies or objects of interest) as well as their locations, and an adversarial network that distinguishes true positives from false positives. Moreover, embodiments of the present invention provide a two-loop technique for training an object detection network. This training process uses detection view-based annotations, so manual annotation can be done much faster and therefore with a larger dataset. Moreover, this process can be used to train a generative adversarial network to distinguish false positives from true positives.

[09] In addition, disclosed systems are proposed that combine an object detection network with a generative adversarial network. By combining such networks, it is possible to distinguish false positives from true positives, thereby providing more accurate results. By reducing the number of false positives, a physician or other healthcare professional can pay increased attention to the network output, resulting in significantly increased accuracy.

[010] Additionally, embodiments of the present invention include neural networks that do not use common feature identification by a single neural network in combination with a dedicated detector. Rather, a single, seamless neural network is trained for the object detector portion, resulting in greater specialization as well as increased accuracy and efficiency.

[011] Finally, embodiments of the present invention are configured to display real-time video (eg, endoscopy video or other medical images) along with object detections on a single display. Accordingly, embodiments of the present invention provide video bypass to minimize potential problems due to errors and other deficiencies associated with the object detector. Moreover, detected objects can be displayed in a special way to better attract the attention of a doctor or other healthcare professional.

[012] In one embodiment, a system for training a generative adversarial network using images containing representations of a feature of interest may comprise at least one memory unit configured to store instructions and at least one processor configured to calling instructions intended to perform operations. Operations may include: providing a first set of images containing representations of an object of interest, indicators of the location of the feature of interest in the images of the first set of images, and using the first set of images, as well as indicators of the feature of interest, training an object detection network to identify the feature of interest. The operations may further include: providing a second set of images that contains representations of the feature of interest, and applying a trained object detection network to the second set of images to generate a first set of detections of the feature of interest. The second set of images may contain more images than those contained in the first set of images. The operations may further include: providing manual checks for true positives and false positives against the first set of detections; using true positive and false positive tests against the first set of detections, training a generative adversarial network; and retraining the generative adversarial network using at least one additional set of images and detections of the feature of interest, together with additional manually set true positive and false positive tests against further detections of the feature of interest.

[013] In some embodiments, the at least one processor may be further configured to retrain the generative adversarial network by providing false negative checking for missed detections of a feature of interest in two or more images.

[014] In any embodiment, the object detection network may be a convolutional neural network.

[015] In any embodiment, the number of images in the second set of images may be at least 100 times greater than the number of images contained in the first set of images.

[016] In any embodiment, the first set of images and the second set of images may comprise medical images. For example, medical images may contain images of the gastrointestinal tract.

[017] In any of the embodiments, at least one of the first set of images and the second set of images comprises images obtained from the endoscopic device. Additionally or alternatively, at least one of the first set of images and the second set of images may comprise images obtained from an imaging device used during at least one of the following procedures: gastroscopy, colonoscopy, enteroscopy, or upper gastrointestinal endoscopy, such like endoscopy of the esophagus.

[018] In any of the embodiments, the feature of interest may be an anomaly. For example, an abnormality may represent a change in human tissue where one type of cell changes into another type of cell. Additionally or alternatively, the anomaly may represent the absence of human tissue at the site where it is suspected. Additionally or alternatively, the abnormality may be an entity within or outside of human tissue.

[019] In any embodiment, the abnormality may be a lesion site. Specifically, the lesion may be a polypoid lesion or a non-polypoid lesion.

[020] In one embodiment, a method for training a neural network-based human organ image anomaly detection system may include a plurality of videos stored in a database, including representations of the anomalies; sampling the first subset from the plurality of videos; and applying a perception branch of the object detection network to frames of a first subset of the plurality of videos to generate a first plurality of detected anomalies. The method may further include selecting a second subset of the plurality of videos; and using the first set of detections and frames from a second subset of the plurality of videos, training the generative network to generate a plurality of artificial anomaly representations. Many artificial representations can be created through residual learning. The method may further include training an adversarial branch of the discriminator network to distinguish between artificial anomaly representations and true anomaly representations; applying an adversarial branch of the discriminator network to the plurality of artificial representations to create indicators of difference between the artificial representations of anomalies and the true representations of anomalies contained in frames of a second subset of the plurality of videos; applying the perception branch of the discriminator network to the artificial representations to obtain a second set of anomaly detections; and retraining the perception branch based on the difference indicators and the second set of detections. These steps may be performed by at least one processor.

[021] In some embodiments, the abnormality may be a change in human tissue from one cell type to another cell type. Additionally or alternatively, the anomaly may be the absence of human tissue at the site where it is suspected. Additionally or alternatively, the abnormality may be a mass within or outside of human tissue.

[022] In any of the embodiments, the abnormality may be a lesion site. For example, the lesion may be a polypoid lesion or a non-polypoid lesion.

[023] In any embodiment, each artificial representation may provide a false representation of the anomaly that is very similar to the true representation of the anomaly.

[024] In any embodiment, the generator network may comprise a generative adversarial network.

[025] In any embodiment, the discriminator network may comprise a convolutional neural network.

[026] In one embodiment, a system for detecting a feature of interest in images of a human organ may comprise at least one memory storage instruction and at least one processor configured to invoke instructions for performing operations. Operations may include: sampling frames from a video of a human organ; applying the trained neural network system to the frames to perform at least one detection of a feature of interest; generating a location indicator of at least one detection on one of the frames; transcoding frames into video; and output of recoded video with indicator. The neural network system may be trained according to any of the embodiments set forth herein.

[027] Additional objects and advantages of the present invention will be set forth in part in the detailed description that follows, and in part will be apparent from the description or may be learned by practice of the present invention. The objects and advantages of the present invention will be realized and achieved by the elements and combinations specifically set forth in the appended claims.

[028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and are not limiting of the proposed embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

[029] The accompanying drawings, which form a part of this specification, illustrate several embodiments and, together with the description, serve to explain the principles and features of the proposed embodiments. On the drawings:

[030] In FIG. 1 illustrates a schematic diagram of an exemplary computer-implemented system for overlaying detected objects on a video stream in accordance with embodiments of the present invention.

[031] In FIG. 2 illustrates an exemplary two-phase training loop for an object detection network according to embodiments of the present invention.

[032] In FIG. 3 illustrates a flowchart of an exemplary method for training an object detection network according to embodiments of the present invention.

[033] In FIG. 4 illustrates a schematic diagram of an exemplary object detector with a discriminator network and a generative network according to embodiments of the present invention.

[034] In FIG. 5 illustrates a flow diagram of an exemplary method for detecting a feature of interest using a discriminator network and a generator network, according to embodiments of the present invention.

[035] In FIG. 6 illustrates a schematic diagram of a computer-implemented system using an object detection network according to embodiments of the present invention.

[036] In FIG. 7 illustrates a flowchart of an exemplary method for overlaying object indicators on a video stream using an object detection network according to embodiments of the present invention.

[037] In FIG. 8A illustrates an example of an overlay display for object detection in a video according to embodiments of the present invention.

[038] In FIG. 8B illustrates another example of an overlay display for object detection in a video according to embodiments of the present invention.

[039] In FIG. 8C illustrates an example of an overlay display for object detection in a video according to embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[040] The disclosed embodiments relate to computer-implementable systems and methods for training and using generative adversarial networks. Advantageously, exemplary embodiments can provide improved trained networks and fast and efficient object detection. Embodiments of the present invention may also provide improved object detection for medical image analysis with a reduced number of false positives.

[041] Embodiments of the present invention can be implemented and used in a variety of vision applications and systems. For example, embodiments of the present invention may be implemented for medical image analysis systems and other types of systems that benefit from object detection, where the objects may be true positives or false positives. Although embodiments of the present invention are described herein generally with reference to medical image analysis and endoscopy, it should be understood that embodiments may be applied to other medical imaging procedures such as gastroscopy, colonoscopy, enteroscopy and upper gastrointestinal endoscopy, for example endoscopy of the esophagus. Moreover, embodiments of the present invention are not limited to other vision environments and systems, such as laser rangefinder, surveillance, autopilot, and other imaging systems or systems containing them.

[042] According to one aspect of the present invention, a computer-implemented system is provided for training a generative adversarial network using images containing representations of an object of interest. The system may include at least one memory unit configured to store instructions and at least one processor configured to recall instructions (as illustrated, for example, in FIGS. 1 and 6). The at least one processor may provide the first plurality of images. For example, the at least one processor may retrieve the first plurality of images from one or more databases. Additionally or alternatively, the first plurality of images may comprise a plurality of frames extracted from one or more videos.

[043] As used herein, the term "image" refers to any digital representation of a scene or field of view. The digital representation may be encoded in any suitable format, such as Joint Photographic Experts Group (JPEG), Graphics Interchange Format (GIF), bitmap format, Scalable Vector Graphics (SVG) format, Encapsulated PostScript (EPS) format. and so on. Similarly, the term "video" refers to any digital representation of a scene or region of interest consisting of many sequential images. The digital representation may be encoded in any suitable format, such as Moving Picture Experts Group (MPEG) format, Flash video format, Audio Video Interleaved (AVI) format, and the like. In some embodiments, the sequence of images may be combined with sound.

[044] The first set of images may include representations of a feature of interest (ie, an anomaly or object of interest) and indicators of locations of the feature of interest in images of the first set of images. For example, the feature of interest may be an abnormality in human tissue or outside of tissue. In some embodiments, the feature of interest may comprise an object, such as a vehicle, person, or other object.

[045] In accordance with the present invention, an "abnormality" may include formations formed in or outside the human tissue, a change in the human tissue from one cell type to another, and/or the absence of human tissue where it is expected. For example, a tumor or other tissue growth may contain an abnormality because more cells are present than expected. Likewise, a bruise or other change in cell type may contain an abnormality because blood cells are present where they should not be (that is, outside the capillaries). Similarly, a depression in human tissue may represent an anomaly because in this case the cells are missing from the expected location, resulting in the appearance of a depression.

[046] In some embodiments, the abnormality may be a lesion. The lesions may represent lesions of the gastrointestinal mucosa. Lesions may be classified histologically (eg, Vienna classification), morphologically (eg, Paris classification), and/or classified structurally (eg, serrated or non-serrated). The Paris classification describes polypoid and non-polypoid lesions. Polypoid lesions may be raised, pedunculated, or sessile lesions. Nonpolypoid lesions may present as superficial raised, flat, superficial, shallowly depressed, or indented lesions.

[047] Regarding the abnormalities found, serrated lesions may be sessile serrated adenomas (SSAs); traditional serrated adenomas (TSA); hyperplastic polyps (HP); fibroblastic polyps (FP); or mixed polyps (MP). In accordance with the Vienna classification, anomalies are divided into the following five categories: (category 1) - negative for neoplasia/dysplasia; (category 2) indefinite in terms of neoplasia/dysplasia; (category 3) low-grade noninvasive neoplasia (low-grade adenoma/dysplasia); (category 4) high-grade mucosal neoplasia, such as high-grade adenoma/dysplasia, noninvasive carcinoma (carcinoma in situ), or suspected invasive carcinoma; and (category 5) invasive neoplasia, intramucosal carcinoma, submucosal carcinoma, etc.

[048] The location indicators of an anomaly or feature of interest may comprise points (eg, coordinates) or areas (eg, rectangle, square, oval, or any other regular or irregular shape). Indicators can contain manual annotations on or to images. In some embodiments, the first set of images may comprise medical images, such as images of the gastrointestinal tract or other organ or area of human tissue. The images may be created by a medical imaging device such as those used during an endoscopy, gastroscopy, colonoscopy, enteroscopy, or an upper gastrointestinal endoscopy procedure such as an esophageal endoscopy procedure. In such embodiments, if the feature of interest is a lesion or other abnormality, a physician or other healthcare professional may annotate the images to place indicators of the abnormality on the images.

[049] The system processor(s) may use the first plurality of images and indicators of the feature of interest to train an object detection network to detect the feature of interest. For example, an object detection network may comprise a neural network with one or more layers configured to receive an image as input and to output an indicator of the location of an object of interest. In some embodiments, the object detection network may comprise a convolutional network.

[050] Training an object detection network may include adjusting the weights of one or more network nodes and/or adjusting the activation (or transmission) functions of one or more network nodes. For example, the weights of an object detection network can be adjusted to minimize the loss function associated with the network. In some embodiments, the loss function may comprise a quadratic loss function, a piecewise linear loss function, a logistic loss function, a cross-entropy loss function, or any other suitable loss function or combination of loss functions. In some embodiments, the activation (or transmission) functions of the object detection network may be modified to improve the correspondence between one or more node(s) models and the input to the node(s). For example, the processor(s) may increase or decrease the power of the polynomial function associated with the node(s), may change the associated function from one type to another (for example, from a polynomial to an exponential function, from a logarithmic polynomial function, or the like), or perform any different configuration of the node(s) model(s).

[051] The system processor(s) may further provide a second set of images that contains representations of the feature of interest. For example, the processor(s) may retrieve the first set of images from one or more databases, whether the same database that stored the first set of images or one or more different databases. Additionally or alternatively, the second set of images may comprise a plurality of frames extracted from one or more videos, regardless of whether the same video was used to extract the first set of images or one or more different videos.

[052] In some embodiments, the second set of images may comprise medical images, such as images from an endoscopic device. In such embodiments, the feature of interest may be a lesion or other abnormality.

[053] In some embodiments, the second set of images may contain a greater number of images than those found in the first set of images. For example, the second set of images may contain at least one hundred times more images than the first set of images. In some embodiments, the second set of images may contain the first set, at least in part, or may be images different from the first set. In embodiments where the second set of images is extracted, at least in part, from one or more videos from which at least a portion of the first set of images was extracted, the second set of images may comprise frames different from the first set from the same video.

[054] The system processor(s) may apply the trained object detection network to the second set of images to produce a first set of detections of the feature of interest. For example, in embodiments where the trained object detection network comprises a neural network, the at least one processor may add a second set of images to the network and receive detection results. The detections may include indicators of the location of an object of interest in the second set of images. If the second set of images does not contain the feature of interest, the indicator may include a null indicator or other indicator of the absence of the feature of interest.

[055] The system processor(s) may further provide manually set true positive and false positive checks against the first set of detections. For example, confirmations may be retrieved from one or more databases or received as input. In embodiments, when the feature of interest is a lesion or other abnormality, confirmation may be entered by a physician or other health care professional. For example, the processor(s) may output detection results for display to a physician or other healthcare professional and receive acknowledgments in response to the displayed detections.

[056] The system processor(s) may use true positive and false positive tests against the first set of detections to train the generative adversarial network. For example, the generative branch of the network can be trained to generate artificial representations of a feature of interest. Accordingly, the generative branch may contain a convolutional neural network.

[057] Similar to an object detection network, training a generative branch may involve adjusting the weights of one or more network nodes and/or adjusting the activation (or transfer) functions of one or more network nodes. For example, as discussed in this document, the weights of the parent branch can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) functions of the generative branch can be modified to improve the correspondence between one or more models of the node(s) and the input to the node(s).

[058] Moreover, the adversarial branch of the network can be trained to distinguish false positives from true positives using manual checks. For example, the adversarial branch may contain a neural network that takes an image and one or more corresponding detections as input, and will perform verification as output. In some embodiments, the processor(s) may further retrain the generative network by providing false negative checks for missed detections of a feature of interest in two or more images. By providing artificial representations from the parent branch as input to the adversarial branch and recursively using the outputs from the adversarial branch, the adversarial branch and the parent branch can perform unsupervised learning.

[059] Similar to the generative branch, training the adversarial branch may involve adjusting the weights of one or more network nodes and/or adjusting the activation (or transfer) functions of one or more network nodes. For example, as discussed herein, the weights of the adversarial branch can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) functions of the adversarial branch can be modified to improve the correspondence between one or more models of the node(s) and the input to the node(s).

[060] Accordingly, in embodiments where the feature of interest includes a lesion or other anomaly, the generative branch can be trained to generate representations of non-anomalies that appear similar to the anomalies, and the adversarial branch can be trained to distinguish artificial non-anomalies from anomalies in the second set images.

[061] The system processor(s) may retrain the generative adversarial network using at least one additional set of images and detections of the feature of interest, along with additional manually set true positive and false positive checks on the additional detections of the feature of interest. For example, the processor(s) may retrieve an additional set of images from one or more databases, whether the same database that stores the first set of images and/or the second set of images, or one or more other databases. Additionally or alternatively, the additional set of images may comprise a plurality of frames extracted from one or more videos, regardless of whether the same video was used to extract the first set of images and/or the second set of images, or one or more other videos. Similar to training, retraining an adversarial branch may involve additional adjustments to the weights of one or more network nodes and/or additional adjustments to the activation (or transfer) functions of one or more network nodes.

[062] In accordance with another aspect of the present invention, a computer-implemented method is provided for training a neural network system to detect anomalies in images of a human organ. The method may be implemented by at least one processor (see, for example, processor 607 illustrated in FIG. 6).

[063] According to the exemplary method, the processor(s) may store a plurality of videos, including anomaly views, in a database. For example, the videos may be endoscopy videos. The videos may be encoded in one or more formats, such as the Moving Picture Experts Group (MPEG) format, Flash video format, Audio Video Interleaved (AVI) format, and the like.

[064] The method may further include sampling, by the processor(s), a first subset of the plurality of videos. For example, the processor(s) may randomly sample the first subset. Alternatively, the processor(s) may use one or more database indexes to fetch the first subset. For example, the processor(s) may sample the first subset as a video indexed as containing anomaly representations.

[065] The method may further include engaging, in conjunction with the processor(s), a perception branch of the object detection network to frames of a first subset of the plurality of videos to produce a first plurality of anomaly detections. For example, an object detection network may comprise a neural network trained to take images as input and output a first set of detections. The first detection set may include indicators of locations of anomalies in the frames, such as a point or area of the detected anomaly. The absence of an anomaly may result in a null indicator or other indicator of the absence of an anomaly. The perception branch may comprise a neural network (eg, a convolutional neural network) configured to detect anomalies and output indicators of the locations of any detected anomalies.

[066] The method may further include sampling, by the processor(s), a second subset of the plurality of videos. In some embodiments, the second subset may comprise, at least in part, the first subset or may be videos different from the first subset.

[067] The method may further include using the first set of detections and frames from a second subset of the plurality of videos to train the generator network to generate a plurality of artificial anomaly representations. For example, the generator network may comprise a neural network configured to create artificial representations. In some embodiments, the generator network may comprise a convolutional neural network. Many artificial representations can be created through residual learning.

[068] As mentioned above, training a generative network may include adjusting the weights of one or more network nodes and/or adjusting the activation (or transfer) functions of one or more network nodes. For example, as mentioned above, the weights of the generative network can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) functions of the generative network can be modified to improve the correspondence between one or more models of the node(s) and the input to the node(s).

[069] The method may further include training, by the processor(s), an adversarial leg of the discriminator network to distinguish between artificial anomaly representations and true anomaly representations. For example, the adversarial branch may contain a neural network that takes representations as input and outputs indicators of whether the input representation is artificial or true. In some embodiments, the neural network may comprise a convolutional neural network.

[070] Similar to the generative branch, training the adversarial branch of the discriminator network may include adjusting the weights of one or more network nodes and/or adjusting the activation (or transfer) functions of one or more network nodes. For example, as mentioned above, the weights of the adversarial branch of the discriminator network can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) functions of the adversarial leg of the discriminator network can be modified to improve the correspondence between one or more node(s) models and the input to the node(s).

[071] The method may further include applying, in conjunction with the processor(s), an adversarial branch of the discriminator network to the plurality of artificial representations to create indicators of the difference between the artificial anomaly representations and the true anomaly representations contained in frames of a second subset of the plurality of videos. For example, artificial representations may contain representations of non-anomalous states that appear similar to the anomalies. Accordingly, each artificial representation can provide a false representation of the anomaly that is very similar to the true representation of the anomaly. The adversarial branch can learn to distinguish between non-anomalies (false beliefs) and anomalies (true beliefs), especially between non-anomalies that are similar to anomalies.

[072] The method may further include applying, in conjunction with the processor(s), the perception branch of the discriminator network to the artificial representations to produce a second set of anomaly detections. Similar to the first detection set, the second detection set may contain indicators of anomaly locations in artificial representations, such as a point or area of the detected anomaly. The absence of an anomaly may result in a null indicator or other indicator of the absence of an anomaly.

[073] The method may further include retraining the perception branch based on the difference indicators and the second set of detections. For example, retraining a perceptual branch may include adjusting the weights of one or more network nodes and/or adjusting the activation (or transfer) functions of one or more network nodes. For example, as mentioned above, the weights of the perception branch can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) functions of the perception branch can be modified to improve the correspondence between one or more node(s) models, difference indicators, and the second set of detections.

[074] An example of a learning method discussed in this document may create a trained neural network system. The trained neural network system may be part of a system used to detect a feature of interest in images of a human organ (eg, the neural network system may be implemented as part of the overlay device 105 illustrated in FIG. 1). For example, such a system may include at least a single memory unit configured to store instructions and at least one processor configured to execute the instructions. The at least one processor may sample frames from the video of the human organ. In particular, the video may be an endoscopy video.

[075] The system processor(s) may apply a trained neural network system to the frames to make at least one detection of a feature of interest. In some embodiments, the feature of interest may be an anomaly. The at least one detection may include an indicator of the location of a feature of interest. For example, the location may contain a point or area that includes a detected object of interest. A neural network system can be trained to detect anomalies as mentioned above.

[076] In some embodiments, the system processor(s) may further apply one or more additional classifiers and/or neural networks to the detected feature of interest. For example, if the feature of interest includes a lesion, the at least one processor may classify the lesion according to one or more types (eg, malignant or non-malignant or the like). Additionally or alternatively, the neural network system may further infer whether the detected feature of interest is a false positive or a true positive.

[077] The system processor(s) may generate a location indicator of at least one detection located in one of the frames. For example, the location of an object of interest can be determined based on an indicator and a graphical location indicator placed on the frame. In embodiments where the location contains a point, the graphical indicator may include a circle, a star, or any other shape placed at the point. In embodiments, when the location contains an area, the graphical indicator may include a border around the area. In some embodiments, the shape or border may be animated; accordingly, a shape or boundary can be generated for multiple frames so that it tracks the location of an object of interest across the frames and also appears animated when the frames are displayed sequentially. As further mentioned below, the graphical indicator may be combined with other indicators, such as an audible and/or vibration indicator.

[078] Any aspect of the indicator may depend on the classification of the property of interest, such as corresponding to one or more types or being a false positive or true positive feature. Accordingly, the color, shape, pattern, or other aspect of the graphical indicator may depend on the classification. In embodiments that also use a sound and/or vibration indicator, the duration, frequency and/or amplitude of the sound and/or vibration may depend on the classification available.

[079] The system processor(s) can transcode the frames into video. Accordingly, after generating a (graphic) indicator and overlaying it on the frame(s), the frames can be reassembled into a video sequence. In this way, the system processor(s) can output the re-encoded video along with the indicator.

[080] In accordance with another aspect of the present invention, a computer-implemented system (see, for example, illustrated in FIGS. 1 and 6) for real-time video processing is described. The system may include an input port for receiving live video. For example, the input port may include a video graphics array (VGA) port, a high-definition multimedia interface (HDMI) port, a digital visual interface (DVI) port, a serial digital interface (SDI), and the like. The live video may contain medical video. For example, the system can receive real-time video from an endoscopy device.

[081] The system may further comprise a first bus for transmitting received video in real time. For example, the first bus may comprise a parallel connection or a series connection and may be connected in a multidrop or daisy chain topology. The first bus may include a PCI Express (Peripheral Component Express) bus, a Universal Serial Bus (USB), an IEEE 1394 (FireWire) interface, etc.

[082] The system may include at least one processor configured to receive real-time video from the first bus, perform object detection on frames of the received real-time video, and overlay a boundary indicating the location of the at least one detected object within the frames. The processor(s) may perform object detection using a neural network system trained to perform at least one object detection. In some embodiments, the at least one object may include a lesion or other abnormality. Accordingly, a neural network system can be trained to detect anomalies, as discussed in this document.

[083] The processor(s) may span the boundary as discussed herein. For example, a boundary may surround an area containing an object, which area is received with at least one detection by the processor(s).

[084] The system may further comprise a second bus for receiving edge overlay video. For example, like the first bus, the second bus may contain a parallel connection or a series connection and may be connected in a multidrop or daisy chain topology. Accordingly, like the first bus, the second bus may contain a PCI Express (Peripheral Component Interconnect Express) bus, a universal serial bus (USB), an IEEE 1394 (FireWire) interface, etc. The second bus may comprise a bus of the same type as the first bus, or may comprise a different type of bus.

[085] The system may further include an output port for outputting edge occluded video from the second bus to an external display. The output port may be a VGA port, an HDMI port, a DVI port, an SDI port, etc. Accordingly, the output port may be the same type as the input port, or may be a different type of port.

[086] The system may include a third bus for directly transmitting received video in real time to an output port. The third bus can transmit real-time video from the input port to the output port passively, so that it operates even when the entire system is turned off. In some embodiments, the third bus may be a default bus that is active when the entire system is turned off. In such embodiments, the first and second buses may be activated when the entire system is activated, and the third bus may be deactivated accordingly. The third bus can be re-enabled when the entire system is shut down or after receiving an error signal from the processor(s). For example, if object detection performed by the processor fails, the processor(s) may activate a third bus, thereby allowing the real-time video stream to continue to be output without interruption due to the failure.

[087] In some embodiments, the imposed boundary may be varied on a frame-by-frame basis. For example, the overlay border may comprise a two-dimensional shape that is displayed around an area of the image containing at least one detected object, the border being the first color. After a certain period of time, the processor(s) may change the boundary to a second color if at least one detected object is a true positive, and to a third color if at least one detected object is a false positive. Additionally or alternatively, the processor(s) may change the boundary based on the classification of the detected object. For example, if an object contains a lesion or other abnormality, the modification may be based on whether the lesion or lesion is malignant or abnormal in some other sense.

[088] In any of the embodiments described herein, the overlay indicator may be paired with one or more additional indicators. For example, the processor(s) may command one or more speakers to play a sound when at least one object is detected. In embodiments, when the boundary is changed, the processor(s) may issue a command to change the boundary. In such embodiments, at least one of the sound duration, tone, frequency, and amplitude may depend on whether the at least one detected object is a true positive or a false positive. Additionally or alternatively, at least one of the duration, tone, frequency and amplitude of the sound may depend on the classification of the detected object.

[089] Additionally or alternatively, the processor(s) may command the at least one wearable device to vibrate when the at least one object is detected. In embodiments when the boundary is changed, the processor(s) may issue a command if the boundary changes. In such embodiments, at least one of the vibration duration, frequency, and amplitude may depend on whether the at least one detected object is a true positive or a false positive. Additionally or alternatively, at least one of the duration, tone, frequency and amplitude of the vibration may depend on the classification of the detected object.

[090] According to another aspect of the present invention, a system for processing video in real time is described. Similar to the processing system described in this document, the system may include an input port for receiving real-time video; at least one processor configured to receive real-time video from an input port, perform object detection by applying a trained neural network to frames of the received real-time video, and applying a boundary indicating the location of the at least one detected object in the frames; and an output port for outputting edge-overlaid video from the processor to an external display.

[091] The system may further include an input device for receiving a sensitivity setting from a user. For example, the input device may include a switch, one or more buttons, or any other device suitable for receiving one command to increase a setting value and another command to decrease a setting value.

[092] The system processor(s) may adjust at least one parameter of the trained neural network in response to the sensitivity adjustment. For example, the processor(s) may adjust one or more weights of one or more network nodes to increase or decrease the number of detections produced by the network, depending on the sensitivity setting. Additionally or alternatively, one or more network output layer thresholds and/or applied to detections received from the network output layer may be increased or decreased in response to the sensitivity setting. Accordingly, if the sensitivity setting is increased, the processor(s) may decrease the threshold(s) so that the number of detections produced by the network increases. Likewise, if the sensitivity setting is decreased, the processor(s) may increase the threshold(s) so that the number of detections made by the network decreases.

[093] In FIG. 1 illustrates a schematic diagram of an exemplary system 100 comprising a pipeline for overlaying detected objects onto a video stream, in accordance with embodiments of the present invention. As shown in the example illustrated in FIG. 1, system 100 includes an operator 101 who operates an imaging device 103. In embodiments where the video stream is a medical video, the operator 101 may be a physician or other healthcare professional. The imaging device 103 may include a medical imaging device, such as an X-ray machine, a computed tomography (CT) machine, a magnetic resonance imaging (MRI) machine, an endoscopic machine, or other medical imaging device that provides video or one or more images of the human body or its parts. The operator 101 may control the imaging device 103 by changing the grip speed of the device 103 and/or moving the device 103, for example, through or relative to the human body. In some embodiments, imaging device 103 may comprise a Pill-Cam™ device or other form of capsule endoscopic device, rather than an external imaging device, such as an X-ray machine or an imaging device inserted through a human body cavity, such as an endoscopic device.

[094] As further illustrated in FIG. 1, renderer 103 may transmit captured video or images to overlay device 105. Overlay device 105 may include one or more processors for processing video, as described herein. Additionally, in some embodiments, operator 101 may control overlay device 105 in addition to imaging device 103, for example, by controlling the sensitivity of an object detector (not illustrated) of overlay device 105.

[095] As illustrated in FIG. 1, overlay device 105 may expand the video received from renderer 103 and then transmit the expanded video to display 107. In some embodiments, the expansion may include an overlay described herein. As further illustrated by FIG. 1, overlay device 105 may also be configured to perform video relay from renderer 103 directly to display 107. For example, overlay device 105 may perform live relay under predetermined conditions, such as if an object detector (not shown) contained in the device overlay 105, faulty. Additionally or alternatively, overlay device 105 may perform direct relay if operator 101 enters a command into overlay device 105 to perform this operation. The command may be received through one or more buttons located in the overlay device 105 and/or through an input device such as a keyboard or the like.

[096] In FIG. 2 illustrates a schematic representation of a two-phase training loop 200 for an object detection network in accordance with embodiments of the present invention. Loop 200 may be implemented by one or more processors. As illustrated in FIG. 2, phase I of loop 200 may use the image database 201, including the item of interest. In embodiments where the images comprise medical images, the element of interest may be an abnormality, such as a lesion.

[097] As mentioned above, the database 201 may store individual images and/or one or more videos, each video containing a plurality of frames. During Phase I of cycle 200, one or more processors may retrieve a subset 203 of images and/or frames from the database 201. One or more processors may sample the subset 203 randomly or at least in part using one or more templates. For example, if database 201 stores video, one or more processors may select at most one, two, or a similar number of frames from each video subset 203.

[098] As further illustrated in FIG. 2, feature indicators 205 may include annotations about a subset 203. For example, the annotations may include a point or region containing an object of interest. In some embodiments, an operator may view video or images and manually enter annotations via an input device (eg, any combination of keyboard, mouse, touch screen and display) into the processor(s). Annotations can be stored as a data structure separate from the image, in formats such as JSON, XML, text, etc. For example, in embodiments where the images are medical images, the operator may be a physician or other health care professional. Although shown as being added to subset 203 after retrieval, subset 203 could have been annotated before being stored in database 201 or at some earlier time. In such embodiments, one or more processors may sample a subset 203 by selecting images in a database 201 having feature indicators 205.

[099] Subset 203, along with feature indicators 205, contains a training set 207. One or more processors may train the discriminator network 209 using the training set 207. For example, the discriminator network 209 may comprise an object detection network, as described above. As mentioned above, training the discriminator network may include adjusting the weights of one or more network nodes and/or adjusting the activation (or transfer) functions of one or more network nodes. For example, the weights of an object detection network can be adjusted to minimize the loss function associated with the network. In another example, the activation (or transfer) functions of an object detection network may be modified to improve the correspondence between one or more models of the node(s) and the input to the node(s).

[0100] As illustrated in FIG. 2, during phase II of cycle 200, one or more processors may retrieve a subset 211 of images (and/or frames) from database 201. Subset 211 may contain, at least in part, some or all of the images from subset 203 or may contain another subset . In embodiments where subset 203 contains multiple frames from one or more videos, subset 211 may contain adjacent or other frames from one or more of the same videos. Subset 211 may contain more images than subset 203, for example, at least 100 times more images.

[0101] One or more processors may apply the discriminator network 209' (which represents the discriminator network 209 after completion of training phase I) to the subset 211 to create a plurality of feature indicators 213. For example, the feature indicators 213 may include a point or region containing an object of interest, detected by the discriminator network 209'.

[0102] As further illustrated in FIG. 2, tests 215 may include annotations for feature indicators 213. For example, the annotations may include an indicator that indicates whether each feature indicator is a true positive or a false positive. An image in which no object of interest was detected, but which contains an object of interest, may be annotated as a false negative.

[0103] The subset 211 along with the feature indicators 213 and tests 215 form a training set 217. One or more processors may train the generative adversarial network 219 using the training set 217. For example, the generative adversarial network 219 may comprise a generative network and an adversarial network 219 , as described above. Training the generative adversarial network may include training the generative network to create artificial representations of a feature of interest or a false feature of interest that is similar to the true feature of interest, and training the adversarial network to recognize the artificial representations from real representations, such as those contained in subset 211.

[0104] Although not illustrated in FIG. 2, tests 213 may further be used to retrain the discriminator network 209'. For example, the weights and/or activation (or transfer) functions of the discriminator network 209' may be adjusted to eliminate detections in images annotated as false positives and/or adjusted to generate detections in images annotated as false negatives.

[0105] In FIG. 3 illustrates a flow diagram of an exemplary method 300 for training an object detection network. Method 300 may be performed by one or more processors. At step 301, illustrated in FIG. 3, the at least one processor may provide a first plurality of images that include representations of a feature of interest and indicators of the location of the feature of interest among images of the first plurality of images. Indicators may include manually set indicators. Manually set indicators can be retrieved from a database or received as operator input.

[0106] At step 303, the at least one processor may, using the first set of images and indicators of the feature of interest, train an object detection network to identify the feature of interest. In particular, an object detection network can be trained as mentioned above.

[0107] At step 305, the at least one processor may provide a second set of images that contain representations of an object of interest, the second set of images containing a greater number of images than those contained in the first set of images. In some embodiments, the second set of images may overlap, at least partially, with the first set of images. Alternatively, the second set of images may consist of images different from those belonging to the first set.

[0108] At step 307, the at least one processor may apply the trained object detection network to the second set of images to produce a first set of detections of the feature of interest. As mentioned above, in some embodiments, detections may include indicators of the location of detected features of interest. For example, an object detection network may comprise a convolutional neural network outputting one or more matrices, each of which defines the coordinates and/or regions of any detected features of interest, optionally with one or more associated confidence scores for each detection.

[0109] At step 309, the at least one processor may provide manually set true positive and false positive checks against the first set of detections. For example, the at least one processor may retrieve manually installed checks from a database or receive them as input from an operator.

[0110] At step 311, the at least one processor may, using tests of true positives and false positives relative to the first set of detections, train a generative adversarial network. In particular, a generative adversarial network can be trained as described in this document.

[0111] At step 313, the at least one processor may retrain the generative adversarial network using at least one additional set of images and detections of the feature of interest, along with additional manual setting of true positive and false positive checks regarding further detections of the feature of interest. . In some embodiments, the additional set of images may overlap, at least partially, with the first set of images and/or the second set of images. Alternatively, the additional set of images may consist of images different from the first set of images and the second set of images. Thus, step 313 may include applying the trained object detection network to an additional set of images to obtain additional detections of the feature of interest, providing manually set true positive and false positive checks against further detections, and retraining the generative adversarial network using the checks against further detections.

[0112] In accordance with the present invention, the exemplary method 300 may include additional steps. For example, in some embodiments, method 300 may include retraining a generative adversarial network by providing false negative checks for missed detections of an object of interest in two or more images. Accordingly, manually installed checks, extracted from a database or received as input, may include checks for false negatives, as well as checks for true positives and false positives. False negatives can be used to retrain the generative adversarial network. Additionally or alternatively, false negatives can be used to retrain an object detection network.

[0113] In FIG. 4 illustrates a schematic representation of an object detector 400. The object detector 400 may be implemented using one or more processors. As illustrated in FIG. 4, the object detector may 400 use a video database 401 containing the feature of interest. In embodiments where the images comprise a medical image, the element of interest may be an abnormality such as a lesion. In the example illustrated in FIG. 4, database 401 contains a database of endoscopic videos.

[0114] As further illustrated in FIG. 4, detector 400 may retrieve a subset 403 of videos from database 401. As discussed above with reference to FIG. 2, subset 403 may be selected randomly and/or using one or more templates. The detector 400 may operate a perception branch 407 of the discriminator network 405 with respect to frames of the subset 403. The perception branch 407 may comprise an object detection network as described above. The perception branch 407 may be trained to detect a feature of interest and identify a location (e.g., a point or region) associated with the detected feature of interest. For example, the perception branch 407 may detect anomalies and output bounds, including detected anomalies.

[0115] As illustrated in FIG. 4, perception branch 407 may output detections 413. As mentioned above, detections 413 may include points or regions identifying locations of detected features of interest in subset 403. As further illustrated in FIG. 4, detector 400 may retrieve a subset 411 of videos from database 401. For example, subset 411 may overlap at least partially with subset 403 or consist of different videos. Subset 411 may contain more videos than subset 403, for example, at least 100 times more videos. The detector 400 may use the subset 411 and detectors 413 to train the generative network 415. The generative network 415 may be trained to create artificial representations 417 of features of interest, such as anomalies. Artificial representations 417 may contain false representations of a feature of interest that appear similar to true representations of the feature of interest. Accordingly, the generative network 415 can be trained to trick the perception branch 407 into detecting false positives.

[0116] As further illustrated in FIG. 4, the generative network 415, after training, can create artificial representations 417. The detector 400 can use the artificial representations 417 to train an adversarial branch 409 of the discriminator network 405. As described above, the adversarial branch 409 can be trained to distinguish the artificial representations 417 from the subset 411. Accordingly, the adversarial branch 409 may define difference indicators 419. Distinction indicators 419 may represent any feature vectors or other aspects of the image that are present in the artificial representations 417 but not in the subset 411, are present in the subset 411 but not in the artificial representations 417, or subtractive vectors or others aspects representing differences between feature vectors or other aspects of artificial representations 417 and those of the subset 411.

[0117] As illustrated in FIG. 4, the detector 400 may retrain the perception branch 407 using the difference indicators 419. For example, in embodiments where the artificial representations 417 contain false representations of the feature of interest, the detector 400 may retrain the perception branch 407 such that the false representations do not result in detection of the true one. representations in subset 411.

[0118] Although not illustrated in FIG. 4, detector 400 may further use recursive learning to improve generative network 415, perception branch 407, and/or adversarial branch 409. For example, detector 400 may retrain generative network 415 using difference indicators 419. Accordingly, the output of adversarial branch 409 may be used to retrain the generative network 415 so that the artificial representations look even more similar to the true representations. In addition, the retrained generative network 415 may create a new set of artificial representations used to retrain the adversarial branch 409. Accordingly, the adversarial branch 409 and the generative network 415 may engage in unsupervised learning where the output of each network is used to retrain the other network in a recursive manner. This recursive training may be repeated until a threshold number of cycles is reached and/or until the loss function associated with the generative network 415 and/or the loss function associated with the adversarial branch 409 reaches the threshold. Moreover, during this recursive training, the perception branch 407 can also be retrained using each new output of the difference indicators, so that a new subset with new detections can be used to further retrain the generative network 415.

[0119] In FIG. 5 illustrates a flow diagram of an exemplary method 500 for detecting a feature of interest using a discriminator and a generative network. Method 500 may be performed by one or more processors.

[0120] As illustrated in FIG. 5, at block 501, the at least one processor may store a plurality of videos in a database, including representations of a feature of interest, such as anomalies. For example, the video could have been taken during endoscopic procedures. As part of step 501, the at least one processor may further sample a first subset of the plurality of videos. As mentioned above, the at least one processor may sample randomly and/or using one or more patterns.

[0121] At step 503, the at least one processor may apply a perception branch of the object detection network to frames of a first subset of the plurality of videos to produce a first plurality of anomaly detections. In some embodiments, as mentioned above, the detections may be indicators of locations of detected anomalies. Additionally, in some embodiments, the perception branch may comprise a convolutional neural network, as mentioned above.

[0122] At step 505, the at least one processor may select a second subset of the plurality of videos. As mentioned above, the at least one processor may sample randomly and/or using one or more patterns. Using the first set of detections and frames from a second subset of the set of videos, the at least one processor may further train the generative network to generate a plurality of artificial anomaly representations, the plurality of artificial representations being generated through residual learning. As mentioned above, each artificial representation provides a false representation of the anomaly that is very similar to the true representation of the anomaly.

[0123] At step 507, the at least one processor may train an adversarial branch of the discriminator network to distinguish between artificial anomaly representations and true anomaly representations. For example, as mentioned above, the adversarial branch can be trained to identify differences between artificial representations and ground truth representations in frames. In some embodiments, the adversarial branch may be a convolutional neural network, as mentioned above.

[0124] At step 509, the at least one processor may implement an adversarial branch of the discriminator network to the plurality of artificial representations to create indicators of difference between the artificial anomaly representations and the true anomaly representations contained in frames of a second subset of the plurality of videos. For example, as mentioned above, difference indicators may represent any feature vectors or other aspects of an image that are present in artificial representations but not in frames, are present in frames but not in artificial representations, or are subtractive vectors or other aspects representing differences between feature vectors or other aspects of artificial representations and frame representations.

[0125] At step 511, the at least one processor may operate the perception branch of the discriminator network in relation to the artificial representations to produce a second set of anomaly detections. Similar to the first set of detections, the detections may contain indicators of locations of detected anomalies in the artificial representations.

[0126] At step 513, the at least one processor may retrain the perception branch based on the difference indicators and the second set of detections. For example, in embodiments where each artificial representation provides a false anomaly representation that is very similar to a true anomaly representation, the at least one processor may retrain the perception branch to reduce the number of detections returned from the artificial representations and, accordingly, to increase the number of null ones indicators or other indicators of the absence of an anomaly returned from artificial representations.

[0127] In accordance with the present invention, the exemplary method 500 may include additional steps. For example, in some embodiments, method 500 may include retraining a generative network based on difference indicators. In such embodiments, method 500 may further include employing a generative network to generate an additional set of artificial anomaly representations and retraining an adversarial branch based on the additional set of artificial anomaly representations. Such retraining steps can be recursive. For example, method 500 may include applying a retrained adversarial branch to an additional set of artificial representations to create additional indicators of the difference between the additional artificial anomaly representations and the true anomaly representations found in frames of a second subset of the plurality of videos, and retraining the generative network based on further indicators of difference. As mentioned above, this recursive retraining may be repeated until a threshold number of cycles is reached and/or until the loss function associated with the generative network and/or the loss function associated with the adversarial network reaches the threshold.

[0128] In FIG. 6 illustrates a schematic diagram of a system 600 comprising a video streaming hardware configuration in accordance with embodiments of the present invention. As illustrated in FIG. 6, system 600 may be communicatively coupled to an imaging device 601, such as a camera or other device that outputs a video stream. For example, imaging device 601 may include a medical imaging device such as a CT scanner, MRI machine, endoscopic device, or the like. System 600 may further be communicatively coupled to a display 615 or other video display or storage device. In particular, display 615 may be a monitor, screen, or other device for displaying images to a user. In some embodiments, display 615 may be replaced or supplemented by a storage device (not illustrated) or network interface controller (NIC) connected to a cloud storage system (also not illustrated) with communication capability.

[0129] As further illustrated in FIG. 6, system 600 may include an input port 603 for receiving a video stream from camera 601, as well as an output port 611 for outputting video to display 615. As discussed herein, input port 603 and output port 611 may include VGA ports, HDMI ports, DVI ports, etc.

[0130] System 600 further includes a first bus 605 and a second bus 613. As illustrated in FIG. 6, first bus 605 may transmit video received through input port 603 through at least one processor 607. For example, processor(s) 607 may implement any of the object detection and/or discriminator networks discussed herein. Accordingly, processor(s) 607 may overlay one or more indicators, such as the exemplary graphical indicator that is illustrated in FIG. 8, in video obtained via the first bus 602, for example, using the exemplary method 700 illustrated in FIG. 7. Processor 607 can then transmit the overlay video via third bus 609 to output port 611.

[0131] Under certain circumstances, the object detector implemented by processor(s) 607 may not operate correctly. For example, software implementing an object detector may crash or otherwise stop working properly. Additionally or alternatively, processor(s) 607 may be commanded to stop video overlay (eg, from an operator of system 600). In response to a fault and/or command, processor(s) 607 may activate second bus 613. For example, processor(s) 607 may send a command or other signal, as illustrated in FIG. 6 to activate the second bus 613.

[0132] As illustrated in FIG. 6, a second bus 613 may transmit received video directly from an input port 603 to an output port 611, thereby allowing the system 600 to function as a pass-through channel to the renderer 601. The second bus 613 may provide a seamless presentation of video from the renderer 601, even if software the software implemented by the processor 607 malfunctions or in the event that the hardware overlay operator 600 decides to stop the overlay in the middle of the video stream.

[0133] In FIG. 7 illustrates a flow diagram of an exemplary method 700 of overlaying object indicators on a video stream using an object detection network, in accordance with embodiments of the present invention. Method 700 may be performed by one or more processors. At step 701, illustrated in FIG. 7, the at least one processor may provide at least one image. For example, the at least one image may be retrieved from a database or obtained from an imaging device. In some embodiments, the at least one image may comprise a frame from the video stream.

[0134] At step 703, the at least one processor may impose a boundary containing a two-dimensional shape around an image region detected to contain an element of interest, the boundary being rendered in a first color. At step 705, after the elapsed period of time, the at least one processor may change the boundary to be displayed in a second color if the feature of interest is a true positive, and to be displayed in a third color if the feature of interest exhibits a false positive. The elapsed time period may be a predetermined period (eg, a threshold number of frames and/or seconds) and/or may be the time elapsed between detection of the feature of interest and its classification as a true positive or false positive.

[0135] Additionally or alternatively, the at least one processor may change the boundary to a second color if the object of interest is classified as a first category, and change the boundary to a third color if the object of interest is classified as a second category. For example, if the feature of interest is a lesion, the first category may contain cancerous lesions and the second category may contain benign lesions.

[0136] In accordance with the present invention, the exemplary method 700 may include additional steps. For example, in some embodiments, method 700 may include commanding one or more speakers to play a sound when a boundary changes and/or commanding at least one wearable device to vibrate when a boundary changes. In such embodiments, at least one of the duration, tone, frequency, and amplitude of the sound and/or vibration may depend on whether the at least one detected object is a true positive or a false positive.

[0137] In FIG. 8A illustrates an example of an overlay 801 for detecting an object in a video in accordance with embodiments of the present invention. In the example illustrated in FIG. 8A and also in FIG. 8B and 8C, illustrated sample videos 800a and 800b are taken from the results of a colonoscopy procedure. It will be appreciated from the present invention that video taken from the results of other imaging procedures and devices may be used in implementing embodiments of the present invention. Thus, video samples 800a and 800b are non-limiting examples of the present invention. Moreover, the video display example illustrated in FIG. 8A-8C may be presented on a display device such as the display 107 illustrated in FIG. 1, or display 615 illustrated in FIG. 6.

[0138] Overlay 801 represents one example of a graphical boundary used as an indicator for a detected anomaly or feature of interest in a video. As illustrated in FIG. 8A, images 800a and 800b contain video frames containing the detected feature of interest. Image 800b contains a graphic overlay 801 and corresponds to a frame that is further down the sequence or later in time than image 800a.

[0139] As illustrated in FIG. 8A, images 800a and 800b contain video frames from a colonoscopy and the object of interest is a lesion or polyp. In other embodiments, as mentioned above, images from other medical procedures such as gastroscopy, enteroscopy, upper gastrointestinal endoscopy such as esophageal endoscopy, etc. can be used and overlaid on a graphical indicator such as overlay 801. In some embodiments, indicator 801 may be overlaid after an anomaly is detected and time has elapsed (eg, a certain number of frames and/or seconds between image 800a and image 800b). In the example illustrated in FIG. 8A, overlay 801 includes an indicator in the form of a rectangular border with a specified pattern (ie, solid corners). In other embodiments, the overlay 801 may have a different shape (regular or irregular). In addition, the overlay 801 may be displayed in a predetermined color or transition from a first color to another color.

[0140] In the example illustrated in FIG. 8A, overlay 801 includes an indicator with solid corners surrounding the detected location of an object of interest in a video frame. Overlay 801 appears in video frame 800b, which may sequentially follow video frame 800a.

[0141] In FIG. 8B illustrates another example of an overlay display for object detection in a video according to embodiments of the present invention. In FIG. 8B illustrates an image 810a (similar to image 800a) and a later image 810b (similar to image 800b) on which indicator 811 is superimposed. In the example illustrated in FIG. 8B, overlay 811 includes a rectangular frame with solid lines on all sides. In other embodiments, the overlay 811 may have a first color and/or a different shape (regular or irregular). In addition, the overlay 811 may be displayed in a predetermined color or transition from a first color to another color. As illustrated in FIG. 8B, an overlay 811 is placed over a detected anomaly or feature of interest in the video. Overlay 811 appears in video frame 810b, which may sequentially follow video frame 810a.

[0142] In FIG. 8C illustrates another example of an overlay image for object detection in a video according to embodiments of the present invention. In FIG. 8C illustrates an image 820a (similar to image 800a) and a later image 820b (similar to image 800b) on which indicator 821 is superimposed. In the example illustrated in FIG. 8C, overlay 821 includes a rectangular border with dotted lines on all sides. In other embodiments, the indicator 821 may have a different shape (regular or irregular). In addition, the overlay 821 may be displayed in a predetermined color or transition from a first color to another color. As illustrated in FIG. 8C, an overlay 821 is placed over a detected anomaly or feature of interest in the video. Overlay 821 appears in video frame 820b, which may sequentially follow video frame 820a.

[0143] In some embodiments, the graphical indicator (ie, overlay 801, 811, or 821) may change pattern and/or color. For example, the pattern and/or color of the pattern border may vary depending on the period of time (eg, a certain number of frames and/or seconds between image 800a and image 800b, image 810a and image 810b, or image 820a and image 820b). Additionally or alternatively, the pattern and/or color of the indicator may be changed according to the particular classification of the feature of interest (eg, if the feature of interest is a polyp, classification of the polyp as malignant or non-cancerous, etc.). Moreover, the pattern and/or color of the indicator may depend on the classification of the feature of interest. Accordingly, the indicator may have a first pattern or color if the feature of interest is classified in a first category, a second pattern or color if the feature of interest is classified in a second category, and the like. Alternatively, the pattern and/or color of the indicator may depend on whether the feature of interest is identified as a true positive or a false positive. For example, a feature of interest may be detected by an object detection network (or the perception branch of a discriminatory network) as mentioned above, resulting in an indicator, but is then detected as a false positive by the adversarial branch or network, as mentioned above, resulting in the indicator being the first pattern or color. The indicator may appear in a second pattern or color if the feature of interest is instead determined to be a true positive by the adversarial branch or network.

[0144] The above description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations of the embodiments will be apparent from consideration of the specification and practice of the disclosed embodiments. For example, the described embodiments include hardware, however, systems and methods in accordance with the present invention can be implemented using both hardware and software. In addition, although some components are described as being associated with each other, such components may be either integrated with each other or distributed in any suitable manner.

[0145] Additionally, while illustrative embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., aspects in different embodiments), adaptations, and/or changes based on of the present invention. Elements in the claims should be interpreted broadly based on the language used in the claims and not be limited to the examples described herein or at the time of examination of the application, which examples should be considered non-exclusive. In addition, the steps of the disclosed methods may be modified in any way, including rearrangement steps and/or insertion or deletion steps.

[0146] The features and advantages of the invention will be apparent from the detailed description, and it is therefore intended that the appended claims cover all systems and methods falling within the true spirit and scope of the invention. As used herein, singular terms mean “one or more.” Likewise, the use of a term in the plural does not necessarily mean plurality unless it is unambiguous in the context. Words such as “and” or “or” mean “and/or” unless specifically stated otherwise. In addition, since numerous modifications and variations will readily occur upon examination of the present invention, it is not desirable to limit the disclosure to the precise structures and operations illustrated and described and, accordingly, all suitable modifications and equivalents may be made within the scope of the invention.

[0147] Other embodiments will be apparent from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the description and examples be construed as exemplary only, with the true scope and spirit of the disclosed embodiments set forth in the following claims.

Claims (46)

1. A system for training a generative adversarial network using medical images, including representations of the feature of interest, containing: at least one memory unit configured to store instructions; And at least one processor configured to execute instructions to perform operations including: providing a first plurality of images that include representations of the feature of interest and indicators of the location of the feature of interest in the images of the first plurality of images; using the first set of images and indicators of the feature of interest, training the object detection network to detect the feature of interest; providing a second plurality of images that include representations of the feature of interest, the second plurality of images comprising a greater number of images than those included in the first plurality of images; applying the trained object detection network to the second set of images to produce a first set of detections of the feature of interest; providing manual installation of true positive and false positive checks against the first set of detections; using true positive and false positive testing against the first set of detections, training a generative adversarial network; And retraining the generative adversarial network using at least one additional set of images and detections of the feature of interest, together with additionally manually setting true positive and false positive checks against further detections of the feature of interest. 2. The system according to claim 1, characterized in that the object detection network is a convolutional neural network. 3. The system of claim 1, wherein the at least one processor is further configured to retrain the generative adversarial network by providing false negative testing for missed detections of an object of interest in two or more images. 4. A system as claimed in any one of the preceding claims, wherein the number of images in the second set of images is at least 100 times greater than the number of images in the first set of images. 5. The system as claimed in any one of the preceding claims, wherein the first set of images and the second set of images comprise medical images. 6. The system according to any one of paragraphs. 1-4, characterized in that the first set of images and the second set of images contain medical images, and the medical images are images of the gastrointestinal tract. 7. A system as claimed in any one of the preceding claims, wherein at least one of the first plurality of images and the second plurality of images comprise images from an imaging device used during at least one of gastroscopy, colonoscopy, enteroscopy or endoscopy upper gastrointestinal tract. 8. The system according to any one of paragraphs. 1-6, characterized in that at least one of the first set of images and the second set of images contain images from an imaging device used during at least one of gastroscopy, colonoscopy, enteroscopy or upper gastrointestinal endoscopy tract, including an endoscopy device. 9. A system according to any of the preceding claims, characterized in that the feature of interest is an anomaly. 10. The system according to any one of paragraphs. 1-8, characterized in that the feature of interest is an abnormality that includes a change in human tissue from one cell type to another cell type, the absence of human tissue in a specific location where it is expected, and/or a formation located within the human tissue or outside it. 11. The system according to claim 9 or 10, characterized in that the anomaly is a lesion. 12. The system according to claim 9 or 10, characterized in that the anomaly is a lesion including a polypoid lesion or a non-polypoid lesion. 13. A method for training a neural network system to detect anomalies in medical images, including the following steps performed by at least one processor: storing multiple videos in the database, including images of anomalies; selecting a first subset of the plurality of videos; applying the perception branch of the object detection network to frames of a first subset of the plurality of videos to generate a first plurality of anomaly detections; selecting a second subset from the plurality of videos; using the first set of detections and frames from a second subset of the plurality of videos, training a generative network to generate a plurality of artificial polyp representations, wherein the plurality of artificial representations are generated through residual learning; training an adversarial branch of the discriminator network to distinguish between artificial anomaly representations and true anomaly representations; applying an adversarial branch of the discriminator network to the plurality of artificial representations to create indicators of difference between the artificial anomaly representations and the true anomaly representations contained in frames of a second subset of the plurality of videos; applying the perception branch of the discriminator network to the artificial representations to obtain a second set of anomaly detections; And retraining the perception branch based on the difference indicators and the second set of detections. 14. The method according to claim 13, characterized in that the generative network contains a generative adversarial network. 15. The method according to claim 13, characterized in that the discriminator network is a convolutional neural network. 16. Method according to any one of paragraphs. 13-15, characterized in that the anomaly is a change in human tissue from one cell type to another cell type, the absence of human tissue in the place where it is expected, and/or the presence of a formation in the area of human tissue or outside of this area. 17. The method according to claim 16, characterized in that the anomaly is a lesion. 18. The method according to claim 16, wherein the abnormality is a lesion including a polypoid lesion or a non-polypoid lesion. 19. Method according to any one of paragraphs. 13-18, characterized in that each artificial representation provides a false representation of the anomaly that is very similar to the true representation of the anomaly. 20. A system for detecting anomalies in medical images, comprising: at least one memory storage instruction; And at least one processor configured to execute instructions for processing operations containing: a selection of frames from a video of a human organ; applying the trained neural network system to the frames to perform at least one anomaly detection; generating a location indicator of at least one detection on one of the frames; transcoding frames into video; And output of recoded video with indicator, wherein the neural network system is trained in accordance with the method according to any one of claims. 13-19.