WO2025106970A1

WO2025106970A1 - Systems and methods for smoke opacity monitoring

Info

Publication number: WO2025106970A1
Application number: PCT/US2024/056380
Authority: WO
Inventors: Timothy James Robertson; Arsineh HECOBIAN; Nathaniel BLANCHARD; Daniel ZIMMERLE; Sonu DILEEP; Timothy L. VAUGHN
Original assignee: Chevron Australia Pty Ltd; Chevron USA Inc; Colorado State University Research Foundation
Current assignee: Chevron Australia Pty Ltd; Chevron USA Inc; Colorado State University Research Foundation
Priority date: 2023-11-17
Filing date: 2024-11-18
Publication date: 2025-05-22
Anticipated expiration: 2026-05-17

Abstract

Images of smoke captured over a time duration are obtained, and temporal and spatial features of the images are extracted. The temporal features are representative of changes across the images and the spatial features are representative of texture and color in the images. The temporal and spatial features are combined, with the combined temporal and spatial features being representative of both static appearance and dynamic motion of smoke in the images. The combined temporal and spatial features are used to determine the opacity of smoke in the images.

Description

SYSTEMS AND METHODS FOR SMOKE OPACITY MONITORING CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of (1) United States Provisional Application Number 63/600,594, entitled “AN ARTIFICIAL INTELLIGENCE FOR AUTOMATED SMOKE OPACITY ESTIMATION,” which was filed on November 17, 2023, and (2) United States Provisional Application Number 63/563,301 , entitled “SMOKE OPACITY ESTIMATION SYSTEM AND METHOD,” which was filed on March 8, 2024, the entirety of each of which is hereby incorporated herein by reference.

FIELD

[0002] The present disclosure relates generally to the field of smoke opacity monitoring using temporal and spatial features of images depicting smoke.

BACKGROUND

[0003] Monitoring smoke emissions via smoke opacity is an important task in various environments. Traditional methods of measuring smoke opacity, such as infrared cameras or trained human observers, can be costly and/or lead to subjective measurements.

SUMMARY

[0004] This disclosure relates to monitoring opacity of smoke. Images captured over a duration of time may be obtained. The images may depict smoke emitted from a source. Temporal features of the images may be extracted. The temporal features may be representative of changes across the images. Spatial features of the images may be extracted. The spatial features may be representative of texture and color in the images. The temporal features of the images and the spatial features of the images may be combined. The combined temporal and spatial features of the images may be representative of both static appearance of the smoke and dynamic motion of the smoke. A smoke opacity value for the smoke depicted within the images may be determined based on the combined temporal and spatial features of the images and/or other information. Smoke opacity monitoring may be facilitated based on the smoke opacity value and/or other information.

[0005] A system for monitoring opacity of smoke may include one or more electronic storage, one or more processors and/or other components. The electronic storage may store information relating to smoke, information relating to images, information relating to temporal features, information relating to spatial features, information relating to combined features, information relating to smoke opacity value, information relating to smoke opacity monitoring, and/or other information.

[0006] The processor(s) may be configured by machine-readable instructions.

Executing the machine-readable instructions may cause the processor(s) to facilitate monitoring opacity of smoke. The machine-readable instructions may include one or more computer program components. The computer program components may include one or more of an image component, an extraction component, a combine component, a smoke opacity component, a monitoring component, and/or other computer program components.

[0007] The image component may be configured to obtain images captured over one or more durations of time. The images may depict smoke emitted from one or more sources. In some implementations, a source may include a flare stack.

[0008] The extraction component may be configured to extract different features of the images. The extraction component may be configured to extract temporal features of the images, spatial features of the images, and/or other features of the images. The temporal features may be representative of changes across the images. The spatial features may be representative of texture and color in the images.

[0009] In some implementations, the temporal features of the images may be extracted based on calculation of structural similarity index measure scores for the images and/or other information. In some implementations, the spatial features of the images may be extracted using one or more parts of an Inception-ResNet and one or more parts of an EfficientNet.

[0010] The combine component may be configured to combine different features of the images. The combine component may be configured to combine the temporal features of the images, the spatial features of the images, and/or other features of the images. The combined temporal and spatial features of the images may be representative of both static appearance of the smoke and dynamic motion of the smoke. [0011] The smoke opacity component may be configured to determine one or more smoke opacity values for the smoke depicted within the images based on the combined temporal and spatial features of the images and/or other information. In some implementations, a smoke opacity value may include a Ringelmann number.

[0012] In some implementations, the dimensionality of the combined temporal and spatial features of the images may be reduced. The combined temporal and spatial features of the images with reduced dimensionality may be re-combined with the temporal features of the images to emphasize temporal variations in determination of the smoke opacity value.

[0013] In some implementations, extraction of the temporal features of the images, extraction of the spatial features of the images, combination of the temporal features of the images and the spatial features of the images, and/or determination of the smoke opacity value(s) may be performed by one or more deep learning models.

[0014] In some implementations, a deep learning model may be pre-trained on a binary classification task to distinguish between smoke and non-smoke. A pre-trained deep learning model may be subsequently trained to estimate the smoke opacity value.

[0015] The monitoring component may be configured to facilitate smoke opacity monitoring based on the smoke opacity value(s) and/or other information. In some implementations, facilitation of the smoke opacity monitoring based on the smoke opacity value(s) may include control of gas combustion at a flare stack based on the smoke opacity value(s).

[0016] These and other objects, features, and characteristics of the system and/or method disclosed herein, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. As used in the specification and in the claims, the singular form of “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 illustrates an example system for monitoring opacity of smoke. [0018] FIG. 2 illustrates an example method for monitoring opacity of smoke. [0019] FIG. 3 illustrates an example flow diagram for monitoring opacity of smoke. [0020] FIG. 4A illustrates an example flow diagram for monitoring opacity of smoke. [0021] FIG. 4B illustrates an example flow diagram for monitoring opacity of smoke. [0022] FIG. 4C illustrates an example flow diagram for monitoring opacity of smoke.

DETAILED DESCRIPTION

[0023] The present disclosure relates to monitoring opacity of smoke. Images of smoke captured over a time duration are obtained, and temporal and spatial features of the images are extracted. The temporal features are representative of changes across the images and the spatial features are representative of texture and color in the images. The temporal and spatial features are combined, with the combined temporal and spatial features being representative of both static appearance and dynamic motion of smoke in the images. The combined temporal and spatial features are used to determine the opacity of smoke in the images.

[0024] The methods and systems of the present disclosure may be implemented by a system and/or in a system, such as a system 10 shown in FIG. 1 . The system 10 may include one or more of a processor 11 , an interface 12 (e.g., bus, wireless interface), an electronic storage 13, an electronic display 14, and/or other components. Images captured over a duration of time may be obtained by the processor 11. The images may depict smoke emitted from a source. Temporal features of the images may be extracted by the processor 11 . The temporal features may be representative of changes across the images. Spatial features of the images may be extracted by the processor 11 . The spatial features may be representative of texture and color in the images. The temporal features of the images and the spatial features of the images may be combined by the processor 11 . The combined temporal and spatial features of the images may be representative of both static appearance of the smoke and dynamic motion of the smoke. A smoke opacity value for the smoke depicted within the images may be determined by the processor 11 based on the combined temporal and spatial features of the images and/or other information. Smoke opacity monitoring may be facilitated by the processor 11 based on the smoke opacity value and/or other information.

[0025] The electronic storage 13 may be configured to include one or more electronic storage media that electronically stores information. The electronic storage 13 may store software algorithms, information determined by the processor 11 , information received remotely, and/or other information that enables the system 10 to function properly. For example, the electronic storage 13 may store information relating to smoke, information relating to images, information relating to temporal features, information relating to spatial features, information relating to combined features, information relating to smoke opacity value, information relating to smoke opacity monitoring, and/or other information.

[0026] The electronic display 14 may refer to an electronic device that provides visual presentation of information. The electronic display 14 may include a color display and/or a non-color display. The electronic display 14 may be configured to visually present information. The electronic display 14 may present information using/within one or more graphical user interfaces. For example, the electronic display 14 may present information relating to smoke, information relating to images, information relating to temporal features, information relating to spatial features, information relating to combined features, information relating to smoke opacity value, information relating to smoke opacity monitoring, and/or other information.

[0027] Monitoring smoke emissions via smoke opacity is an important task in various environments, such as industrial, urban, rural, domestic, commercial agricultural, fixed, and mobile environments. Traditional methods of measuring smoke opacity, such as infrared cameras or trained human observers, can be costly and/or lead to subjective measurements.

[0028] Detecting and characterizing smoke presents a unique set of challenges. Smoke is an insubstantial object, lacking a constant shape or structure due to its physical nature, and behavior associated with various environmental factors. Its form is highly transient and can change rapidly, making it difficult to track and identify consistently.

Moreover, smoke does not maintain a consistent shade. Depending on the combustion process, the shade of smoke can range from light (white) to dark (black) and can vary in color and opacity. Environmental factors, such as sunlight, can also affect the perceived color of smoke. This amount of variability poses significant challenges to the development of robust and reliable models that seek to detect or characterize smoke. [0029]Another challenge arises from the similarity between smoke and clouds, which both share similar textures. In outdoor settings, the presence and variability of clouds can make it significantly more difficult to differentiate between the target object and the background field. Furthermore, classifying the opacity of smoke requires the ability to not only detect smoke and distinguish it from surrounding elements but also the ability to discern fine-grained characteristics of the smoke and ascertain how much of the background is being obscured by it. Additionally, as density of smoke decreases, it becomes harder to detect and predict the opacity of smoke. This is particularly problematic as lower density smoke can have environmental and amenity impacts. [0030] Traditional computer vision methods for detecting objects, such as convolutional neural networks, tend to pick out the general shape and structure of an object in an image. Such detection techniques are useful when detecting objects with definite shape and/or texture, such as vehicles or buildings. However, smoke is transient and constantly morphing, with its opacity is defined by the penetration of light, rather than texture. With constant changes in shape and texture from frame to frame, smoke detection is an inherently difficult problem from a computer vision standpoint.

[0031] The present disclosure provides a smoke monitoring tool to determine opacity of smoke (e.g., Ringelmann number) accurately and efficiently. The tool is capable of modeling the complicated movements and dynamics of smoke. The tool includes a deep learning model that incorporates both temporal and spatial characteristics of smoke into its analysis. The tool may centrally integrate temporal and spatial features of images for robust smoke pattern detection and tracking.

[0032] The layers/blocks of the deep learning model capture useful features of smoke from both the spatial and temporal dimensions. The tool may be trained using a combination of synthetic and real-world data to handle a wide range of scenarios. The tool not only addresses the texture similarity issue with background objects such as clouds and heat haze effects on background objects, but also considers other factors such as varying lighting conditions, weather, and the transient nature of smoke itself. By capturing the subtleties of smoke behavior, the tool effectively differentiates between smoke and other environmental elements, resulting in more precise monitoring outcomes. [0033] The tool also takes advantage of a weighted loss function that accounts for the fact that smoke with close opacity values tend to have visual similarities. In other words, the loss function does not punish the model as much if it guesses close to the correct opacity class/value.

[0034] Deep learning models may use a substantial number of parameters to make predictions, leading to challenges in terms of computational efficiency. The architecture of the tool presented herein is designed to operate with fewer parameters, thereby contributing to reduction of hardware costs, and reduced computational effort. The efficient architecture of the present disclosure enables smoke opacity predictions to be performed in real time.

[0035] FIG. 3 illustrates an example flow diagram for monitoring opacity of smoke. FIG. 3 illustrates a general view of a deep learning model architecture for monitoring opacity of smoke. Images 302 may have been captured over a duration of time (T). The images 302 may depict smoke emitted from a source, such as a flare stack. The images 302 may have a resolution defined by width (W) and height (H). Pixels of the images 302 may have color values (C). Temporal features 304 and spatial features 306 may be extracted from the images 302. The temporal features 304 may be representative of changes across the images 302 while the spatial features 306 may be representative of texture and color in the images 302. The temporal features 304 and the spatial features 306 may capture both static appearance of smoke and dynamic motion of smoke across the images 302. The temporal features 304 and the spatial features 306 may be processed through a convolutional neural network (CNN) block 308, a transformer block 310, and a long short-term memory (LSTM) + CNNs + fully connected layer block 312, which may change the dimensions of information being passed. The blocks 308, 310, 312 may be used to determine, given the temporal features 304 and the spatial features 306 and the training data, what features are meaningful for smoke opacity prediction.

[0036] Both the temporal features 304 and the spatial features 306 may be processed through the CNN block 308. The temporal features 304 and the spatial features 306 may be combined (concatenated) before being processed through the CNN block 308. The output of the CNN block 308 and the temporal features 304 may be processed through the transformer block 310. The output of the transformer block 310 may be processed through the LSTM + CNNs + FC block 312. The output of the LSTM + CNNs + FC block 312 may include prediction of smoke opacity value 314 of the smoke depicted within the images 302.

[0037] FIG. 4A illustrates an example flow diagram for monitoring opacity of smoke. FIG. 4A illustrates a detailed view of a deep learning model architecture for monitoring opacity of smoke. The deep learning model architecture may include average pooling between different blocks. The deep learning model architecture may capture the complex nature of smoke through a combination of spatial feature extraction and temporal sequence processing blocks/layers. Spatial and temporal features of images may be combined to extract the spatio-temporal features from the images (input data), followed by CNN blocks and transformer blocks which learn how to effectively fuse the features together for smoke opacity value determination. The architecture’s focus on temporal aspects may ensure a nuanced interpretation of dynamic smoke behavior. [0038] Images 402 may be captured over a duration of time. The images 402 may include video frames of a video. The images 402 may include sequential and/or nonsequential video frames of a video. The images 402 may be processed through a deep learning model. The images 402 may include a number of frames captured over a duration of time. For example, in FIG. 4A, the images 402 may include six frames captured over a duration of time. Use of other numbers of images is contemplated. The images may have a resolution and color values. For example, in FIG. 4A, the images 402 may have a resolution of 224x224 and color values in three color channels (e.g., R channel, G channel, B channel). Use of other resolutions and other color spaces is contemplated. The dimensionality of the images 402 may be defined by the number of images, the resolution, and the color channels. As the images 402 are processed through the deep learning model, the dimensionality of the images/information from images may be changed, such as shown in FIG. 4A. Other dimensions and other changes in dimensions are contemplated.

[0039] Temporal features 404 and spatial features 406 may be extracted from the images 402. The temporal features 404 and the spatial features 406 may be concatenated. Concatenation of the temporal features 404 and the spatial features 406 may result in a comprehensive representation that encapsulates both the static appearance and dynamic motion of smoke, enabling a robust detection and tracking of smoke patterns within the sequences of images (e.g., video frames). The concatenated features may be passed through a convolutional block 408 (e.g., Conv2D) for dimensionality reduction, resulting a more compact and efficient representation. The output (output vector) of the convolutional block 408 may be re-concatenated with the temporal features 404 for added emphasis on temporal variations, and the combined feature set may be further processed by a transformer block 410.

[0040] The transformer block 410 may include three layers and two heads, where each feature point attends to all pixels in the current frame. Instead of utilizing conventional window or patch-based attention mechanisms, every feature point may attend to every other pixel within the current frame. The output of the transformer block 410 may be fed into a convolutional LSTM block 412 (e.g., ConvLSTM2D) for frame-to-frame transition learning (enabling the model to learn transitions between frames), followed by additional convolutional block 414 (e.g., Conv2D) and a fully connected layer 416. The convolutional block 414 may include a series of 2D convolutional filters that reduces the spatial dimension of the processed data while learning contextual information. A softmax function at the end of the deep learning model may output prediction on opacity value of smoke depicted within the images 402 (e.g., classify the images 402 into opacity levels, output Ringelmann number). The deliberate emphasis on temporal characteristics by the model ensures that the dynamic behavior of smoke is not overshadowed by its visual appearance, leading to a detailed and precise interpretation of smoke opacity.

[0041] FIG. 4B illustrates an example flow diagram for monitoring opacity of smoke. The flow diagram in FIG. 4B may show extraction of temporal features 456 from images 450. Structural similarity index measure (SSIM) scores may be calculated for pixels of the images 450, enabling the model to learn the movement of smoke and focus on regions of interest within the images 450. The SSIM may gauge the visual impact of luminance, contrast, and structure within the images 450. The SSIM scores may be used to distinguish smoke from other similar phenomena, such as clouds, by tracing the motion exhibited in the smoke pixels. The SSIM scores may be calculated for individual pixels between adjacent ones of the images 450 (e.g., adjacent video frames). SSIM scores may serve as an efficient tool for pinpointing differences between the images 450 that might indicate smoke’s characteristic motion. One or more sizes of window may be used to perform the SSIM calculation and encompass a localized context around individual pixels (e.g., a window size of seven). The computed SSIM scores may be combined (e.g., concatenated) together and then processed through a Convolutional Long Short-Term Memory (ConvLSTM2D) layer to model temporal features in the images 450.

[0042] Negative values within the SSIM computation may be set to zero to ensure that the range of scores represents meaningful variations. An SSIM score of one may signify consistency across the images 450, reflecting little to no change in pixel values. Conversely, a decrease in SSIM score may correspond to significant variations indicative of motion or changes between the images 450. Individual ones of the images 450 may be divided into smaller windows, and the SSIM scores may be calculated to generate an SSIM map. The SSIM map may aid in detecting changes from adjacent ones of the images 450. Local motion of pixels across the image 450, as is prevalent in smoke dispersion, may lead to reduced SSIM scores. Locations of low SSIM scores in the SSIM map may become the regions of interest.

[0043] FIG. 4C illustrates an example flow diagram for monitoring opacity of smoke. The flow diagram in FIG. 4C may show extraction of spatial features 466 from the images 460. The spatial features 466 of the images 460 may be extracted using one or more part of a part of an EfficientNet 462 and one or more parts of an Inception-ResNet 464. The output of the Inception-ResNet 464 may be resized 466 before being combined (e.g., concatenated) with the output of the EfficientNet 462 to generate the spatial features 466.

[0044] For example, the images 460 may be processed through the first tens layers of lnceptionResnetV2 model and the first eleven layers of the EfficientNetB6 model. These specific layers may effectively capture the texture and color of smoke while not filtering out necessary features of the smoke. The lnceptionResnetV2 model may be adept at capturing the texture of smoke while the EfficientNetB6 model may aid in denoising noisy input data. The spatial feature extraction may encapsulate the essence of smoke’s appearance while discarding unnecessary complexities. Use of other models/parts of other models is contemplated.

[0045] The deep learning model may be trained in multiple stages. For example, the deep learning model may be pre-trained on a binary classification task to distinguish between smoke and non-smoke. The pre-trained deep learning model may be subsequently trained to estimate the smoke opacity value (e.g., Ringelmann number). Pre-training the deep learning model on the binary classification task to distinguish between smoke and non-smoke may include training the deep learning model to io associate features with smoke depicted within the images input into the deep learning model. After being trained on which features are meaningfully associated with smoke, the deep learning model may be then trained to associate these features with smoke opacity value and thereby make prediction on smoke opacity value of smoke depicted within the images.

[0046] Different loss functions may be used to different parts/phases of the training. For pre-training on the binary classification task to distinguish between smoke and nonsmoke, a simple binary cross entropy loss function may be used. The goal of using this loss function may be to minimize the difference between the predicted probabilities and the actual class labels. This loss function may be represented as shown below, where N is the number of samples, y_t is the actual class label, and y_t is the predicted probability:

[0047] For subsequent training on estimation of smoke opacity value, a weighted loss function that considers the difference between the integer-encoded predictions and the original label may be used. This may encourage the model to predict opacity numbers that are closer to each other, even in the event of misclassification. The weight w, for each sample may be calculated as the absolute difference between the integer-encoded label and the original label, as shown below, where y, is the original label and y/ is the integer-encoded predictions:

[0048] Incorporating the above into the loss function results in the below weighted loss function, which encourages the model to make predictions that are closer to the actual smoke opacity value (e.g., Ringelmann number):

[0049] The training data for the deep learning model may include images depicting smoke. The training data may include single images and/or video frames of one or more videos captured by one or more cameras. The training data may include color images. Use of greyscale images is contemplated. The training data may include images captured from a single perspective or multiple perspectives. For example, smoke may be captured within images (e.g., video frames) by a single camera with the smoke/source within its field of view. As another example, smoke may be captured within images by multiple cameras, with different cameras capturing different perspectives of smoke (e.g., six cameras positioned equidistantly around the source to capture six divergent viewpoints). Multiple cameras may be positioned to provide stereo-vision of smoke (e.g., pairs of cameras positioned around the source; six cameras positioned in pairs to form a triangular configuration around the source). The model may be trained to receive as input images from a single camera/single perspective and/or images from multiple cameras/multiple perspectives. Use of multiple perspectives may enable three-dimensional modeling of smoke.

[0050] The training data may include real data (e.g., images depicting real smoke) and/or virtual data (e.g., images depicting virtual smoke). The training data may include images depicting smoke emission in a variety of scenarios, such as in various deployment locations, backgrounds (including sky, buildings, and structures), atmospheric conditions, camera angles, types of smoke, smoke patterns, times of day, temperatures (e.g., different heat haze effects on background objects), weather conditions, and lighting conditions (e.g., cloudy days, hazy or dusty conditions, clear days, light rain/snow, medium rain/snow, heavy rain/show). The training data may be annotated with labels (e.g., smoke, no smoke, smoke opacity value, Ringelmann number, scenario description). Data augmentation (e.g., rotation, scaling, flipping, brightness adjustment) may be performed on the training data to improve the generalization of the model. The training data may include downsized (e.g., scaled- down) images. The color channels of the images may be normalized. [0051] Referring back to FIG. 1, the processor 11 may be configured to provide information processing capabilities in the system 10. As such, the processor 11 may comprise one or more of a digital processor, an analog processor, a digital circuit designed to process information, a central processing unit, a graphics processing unit, a microcontroller, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. The processor 11 may be configured to execute one or more machine-readable instructions 100 to facilitate monitoring opacity of smoke. The machine-readable instructions 100 may include one or more computer program components. The machine-readable instructions 100 may include one or more of an image component 102, an extraction component 104, a combine component 106, a smoke opacity component 108, a monitoring component 110, and/or other computer program components.

[0052] The image component 102 may be configured to obtain images. Obtaining an image may include accessing, acquiring, analyzing, determining, examining, generating, identifying, loading, locating, measuring, opening, preparing, receiving, retrieving, reviewing, selecting, storing, and/or otherwise obtaining the image. Obtaining an image may include obtaining information (e.g., images information defining an image, video information defining a video) that characterizes, conveys, defines, describes, identifies, quantifies, and/or reflects the images. For example, obtaining an image may include obtaining an image file or a video file. The image component 102 may obtain an images from a storage location, such as the electronic storage 13, electronic storage of a device accessible via a network, and/or other locations. The image component 102 may obtain an image from one or more hardware components (e.g., a computing device, a sensor) and/or one or more software components (e.g., software running on a computing device). The image component 102 may obtain an image based on user interaction with the system 10 (e.g., a user identifying, selecting, and/or uploading images/video(s) from which smoke opacity values are to be determined).

[0053] The images may have been captured over one or more durations of time. The images may have been captured by one or more cameras. The images may include color images. The images may provide a time series data for smoke, having dimensions of time, width, height, and colors. The images may have been captured using the same frame rate or different frame rates. The image may include some or all of the video frames captured during the duration(s) of time. [0054] The images may depict smoke emitted from one or more sources. Smoke may refer to gaseous or solid (e.g. colloidal solutions of solid in gas) product of burning materials (e.g., combustion of materials), emission of dust/particulates (e.g., evolved from unvegetated ground, or initiated by crushing, grading, earthworks), and/or other suspension of airborne particulates and/or gasses. The images may provide a single or multiple perspectives of the smoke emitted from the source(s). A source may refer to an object, an area, a piece of equipment, a thing, and/or a structure/facility from which smoke is emitted. A source may continuously, regularly, or intermittently emit smoke. A source may include a fixed source or a mobile source. For example, a source may include fixed equipment, such as a flare stack (e.g., oil and gas flare stack), or a mobile equipment, such as a vehicle (e.g., tail pipe). A source may be located in an urban setting, a rural setting, or other settings. A source may include one or more activities/processes, such as an industrial activity/process, an agricultural activity/process, or a civil/construction activity/process. Other types of smoke and other types of sources are contemplated.

[0055] The extraction component 104 may be configured to extract different features of the images. Extracting a feature of an image may include calculating, computing, determining, identifying, processing, quantifying, and/or otherwise extracting the feature of the image. Extracting a feature of an image refer to a process/analysis that involves identifying and determining/identifying relevant features from the image. Feature extraction may include one or more processes of transforming data in the images into features that can be processed. A feature may refer to an individual measurable property or characteristic. A feature may include one or more representations (e.g., vector representation) of information. One or more features of the images may be extracted by one or more machine learning models and/or one or more parts of machine learning model(s). The images may be input into/processed through one or more blocks/layers of the machine learning model(s), with the output including the extracted features. Other feature extractions are contemplated.

[0056] The extraction component 104 may be configured to extract temporal features of the images, spatial features of the images, and/or other features of the images. For example, the temporal features of the image may be extracted as described with respect to the temporal features 304 in FIG. 3, the temporal features 404 in FIG. 4A, and/or the temporal features 456 in FIG. 4B. For example, the spatial features of the image may be extracted as described with respect to the spatial features 306 in FIG. 3, the spatial features 406 in FIG. 4A, and/or the spatial features 466 in FIG. 4C. Other extractions of the temporal features of the images and the spatial features of the images are contemplated.

[0057] The temporal features may be representative of changes across the images. The temporal features may be representative of changes across a set of images processed through the deep learning machine learning model. For example, six images (video frames) may be processed through the deep learning machine learning model at a time, and the temporal features may be representative of changes across the six images. Use of other numbers of images is contemplated. The temporal features may include vector representation of changes across the images.

[0058] In some implementations, the temporal features of the images may be extracted based on calculation of structural similarity index measure (SSIM) scores for the images and/or other information. For example, SSIM scores may be calculated as described with respect to FIG. 4B. One or more SSIM maps may be generated for the images to detect changes between adjacent images and identify regions of interest (e.g., regions depicting smoke within the images). Use of other machine learning models/tools is contemplated.

[0059] The spatial features may be representative of texture and color in the images. The spatial features may be representative of texture and color in a set of images processed through the deep learning machine learning model. For example, six images (video frames) may be processed through the deep learning machine learning model at a time, and the spatial features may be representative of texture and color in the six images. Use of other numbers of images is contemplated. The spatial features may include vector representation of texture and color in the images.

[0060] In some implementations, the spatial features of the images may be extracted using one or more parts of an Inception-ResNet and one or more parts of an EfficientNet. The Inception-ResNet and the EfficientNet may capture different aspects of texture and color in the processed images. For example, the spatial features of the images may be extracted as described with respect to FIG. 4C. Use of other machine learning models/tools is contemplated.

[0061] The combine component 106 may be configured to combine different features of the images. Combining different features of the images may include grouping, joining, linking, merging, uniting, and/or otherwise combining the different features of the images. For example, one or more of the features of the images may be resized and concatenated with other feature(s) of the images. The combine component 106 may be configured to combine the temporal features of the images, the spatial features of the images, and/or other features of the images. The temporal features of the images and the spatial features of the images may be combined by one or more parts of a deep learning model. The temporal features of the images and the spatial features of the images may be combined for processing through the rest of the deep learning model. Combining other features of the images is contemplated.

[0062] The combined temporal and spatial features of the images may be representative of both static appearance of the smoke and dynamic motion of the smoke. The combined temporal and spatial features of the images may provide information on both (1) color and texture in the images and (2) where relevant movement (movement associated with smoke) is depicted within the images.

[0063] The smoke opacity component 108 may be configured to determine one or more smoke opacity values for the smoke depicted within the images. Determining a smoke opacity value for the smoke depicted within the images may include ascertaining, approximating, calculating, classifying, establishing, estimating, finding, identifying, obtaining, quantifying, and/or otherwise determining the smoke opacity value for the smoke depicted within the images. A smoke opacity value may refer to a value that indicates and/or reflects the opacity (lacking transparency or translucence) of smoke. [0064] In some implementations, a smoke opacity value may include a Ringelmann number. The Ringelmann number may be a standardized scale used to measure the opacity of smoke (e.g., emitted from industrial activities). The scale may range from 0 to 5, with each increment representing a 20% increase in opacity. A Ringelmann number of 0 may correspond to 0% opacity, while a Ringelmann number of 5 may signify 100% opaque smoke.

[0065] The smoke opacity value(s) may be determined based on the combined temporal and spatial features of the images and/or other information. A smoke opacity value for a set of images may be determined based on the combined temporal and spatial features of the set of images and/or other information. A set of images may cover a duration of time (e.g., fraction of a second, second(s), minute(s), hour(s)). For example, six images (video frames) may be processed through the deep learning machine learning model at a time, and the combined temporal and spatial features of the six images may be used to determine the smoke opacity value of smoke depicted within the six images. Use of other numbers of images is contemplated.

[0066] In some implementations, extraction of the temporal features of the images, extraction of the spatial features of the images, combination of the temporal features of the images and the spatial features of the images, and/or determination of the smoke opacity value(s) may be performed by one or more deep learning models. A deep learning model may refer to a computer model (computer process/file/tool) that uses neural networks/machine learning. A deep learning model may refer to a machine learning model. A deep learning model may use multilayered neural networks to process information. For example, extraction of the temporal features of the images, extraction of the spatial features of the images, combination of the temporal features of the images and the spatial features of the images, and/or determination of the smoke opacity value(s) may be performed by one or more deep learning models with architecture shown in FIGS. 3, 4A, 4B, and/or 4C.

[0067] In some implementations, a deep learning model may be pre-trained on a binary classification task to distinguish between smoke and non-smoke. That is, a deep learning model may be initially trained to distinguish between depiction of smoke and depiction of no smoke within images before being trained to output smoke opacity value(s) of smoke depicted within images. A pre-trained deep learning model may be subsequently trained to estimate the smoke opacity value from the input images. A deep learning model may be trained using training data that include images depicting smoke emission in a variety of scenarios.

[0068] In some implementations, the dimensionality of the combined temporal and spatial features of the images may be reduced. For example, as shown in FIGS. 3 and 4A, the dimensionality of the combined temporal and spatial features of the images may be reduced via processing through convolutional block (the CNN block 308, the convolutional block 408) and/or other blocks, resulting a more compact and efficient representation of the temporal and spatial features. Other dimension reductions are contemplated.

[0069] In some implementations, the combined temporal and spatial features of the images with reduced dimensionality may be re-combined with the temporal features of the images to emphasize temporal variations in determination of the smoke opacity value(s). For example, as shown in FIGS. 3 and 4A, the output of the convolutional block may be combined/concatenated with the temporal features of the images for processing through a transformer block (the transformer block 310, the transformer block 410).

[0070] The monitoring component 110 may be configured to facilitate smoke opacity monitoring. Facilitating smoke opacity monitoring may include assisting, automating, carrying out, controlling, designing, enabling, implementing, initiating, performing, planning, scheduling, setting up, and/or otherwise facilitating the smoke opacity monitoring. Smoke opacity monitoring may refer to detecting, observing, tracking, checking, maintaining surveillance over, and/or otherwise monitoring the opacity of smoke. Smoke opacity monitoring may include tracking of smoke opacity value(s) (e.g., tracking smoke opacity values at different moments in time). Smoke opacity monitoring may include determination of when smoke opacity value(s) satisfies/does not satisfy one or more criteria. For example, smoke opacity monitoring may include determination of the duration of time over which smoke opacity value(s) is a certain value (e.g., a certain Ringelmann number). As another example, smoke opacity monitoring may include determination of the duration of time over which smoke opacity value(s) is greater than a threshold value (e.g., a threshold Ringelmann number). Other types of smoke opacity monitoring is contemplated.

[0071] Smoke opacity monitoring may be facilitated based on the smoke opacity value(s) and/or other information. Smoke opacity monitoring may be facilitated based on information relating to and/or determined from the smoke opacity value(s). For example, smoke opacity monitoring may include (1) presenting the smoke opacity value(s) on the electronic display 14, (2) presenting information relating to and/or determined from the smoke opacity value(s) on the electronic display 14, (3) presenting results of smoke opacity monitoring on the electronic display 14, and/or (4) providing information relating to smoke opacity value(s) to one or more users and/or one or more computing devices.

[0072] In some implementations, facilitation of the smoke opacity monitoring based on the smoke opacity value(s) may include control of one or more operations based on the smoke opacity value(s) and/or other information. Operation(s) relating to the source of the smoke and/or operation(s) of an object, a piece of equipment, a thing, and/or a structure/facility connected to the source of the smoke may be facilitated (e.g., designed, selected, scheduled, controlled, initiated, carried out, performed) based on the smoke opacity value(s). For example, the source may include a flare stack, and gas combustion at the flare stack may be controlled based on the smoke opacity value(s). Gas combustion at the flare stack may be changed based on the smoke opacity value(s) (e.g., adjust temperature of gas flowing to the flare stack, adjust amount of airflow/blow speed to the flare stack, adjust/vary process conditions to optimize combustion at the flare tip and reduce/minimize smoking). In some implementations, adjustment to the operation of the flare stack may be recommended to one or more users for use. In some implementations, adjustment to the operation of the flare stack may be automatically performed. One or more alerts may be generated based on the smoke opacity value(s), such as based on the smoke opacity value(s) being greater than a smoke opacity threshold.

[0073] In some implementations, the smoke opacity values may facilitate regulatory compliance of smoke emission from the source. Regulations may require smoke emissions to adhere to permissible opacity limits set by authorities. Different authorities may set different permissible opacity limits. Regulations may require reporting of smoke emissions based on smoke opacity value. For example, smoke emission having smoke opacity of Ringelmann 3 for a continuous 20-minute period may be a reportable event. As another example, smoke emission having smoke opacity of Ringelmann 2 for a continuous 40-minute period may be a reportable event. Other reporting requirements are contemplated. The smoke opacity values may be tracked to identify emissions that need to be reported. Reports may be automatically generated/prepared based on the tracked smoke opacity values. Smoke emissions from the source may be managed based on the smoke opacity value(s). Continuous monitoring of smoke opacity values may be used to detect changes in smoke opacity, which in turn may be used to perform corrective actions to meet regulatory requirements.

[0074] Implementations of the disclosure may be made in hardware, firmware, software, or any suitable combination thereof. Aspects of the disclosure may be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a non-transitory, tangible computer-readable storage medium may include read-only memory, random access memory, magnetic disk storage media, optical storage media, flash memory devices, and others, and a machine-readable transmission media may include forms of propagated signals, such as carrier waves, infrared signals, digital signals, and others. Firmware, software, routines, or instructions may be described herein in terms of specific exemplary aspects and implementations of the disclosure, and performing certain actions.

[0075]As used herein, the phrase “configured to” is intended to be interpreted broadly, as “being capable of or suitable for performing” some function or feature, without requiring any adaptations to provide said function or feature.

[0076] In some implementations, some or all of the functionalities attributed herein to the system 10 may be provided by external resources not included in the system 10. External resources may include hosts/sources of information, computing, and/or processing and/or other providers of information, computing, and/or processing outside of the system 10.

[0077] Although the processor 11 , the electronic storage 13, and the electronic display 14 are shown to be connected to the interface 12 in FIG. 1 , any communication medium may be used to facilitate interaction between any components of the system 10. One or more components of the system 10 may communicate with each other through hardwired communication, wireless communication, or both. For example, one or more components of the system 10 may communicate with each other through a network. For example, the processor 11 may wirelessly communicate with the electronic storage 13. By way of non-limiting example, wireless communication may include one or more of radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, or other wireless communication. Other types of communications are contemplated by the present disclosure.

[0078] Although the processor 11 , the electronic storage 13, and the electronic display 14 are shown in FIG. 1 as single entities, this is for illustrative purposes only. One or more of the components of the system 10 may be contained within a single device or across multiple devices. For instance, the processor 11 may comprise a plurality of processing units. These processing units may be physically located within the same device, or the processor 11 may represent processing functionality of a plurality of devices operating in coordination. The processor 11 may be separate from and/or be part of one or more components of the system 10. The processor 11 may be configured to execute one or more components by software; hardware; firmware; some combination of software, hardware, and/or firmware; and/or other mechanisms for configuring processing capabilities on the processor 11.

[0079] It should be appreciated that although computer program components are illustrated in FIG. 1 as being co-located within a single processing unit, one or more of computer program components may be located remotely from the other computer program components. While computer program components are described as performing or being configured to perform operations, computer program components may comprise instructions which may program processor 11 and/or system 10 to perform the operation.

[0080] While computer program components are described herein as being implemented via processor 11 through machine-readable instructions 100, this is merely for ease of reference and is not meant to be limiting. In some implementations, one or more functions of computer program components described herein may be implemented via hardware (e.g., dedicated chip, field-programmable gate array) rather than software. One or more functions of computer program components described herein may be software-implemented, hardware-implemented, or software and hardware-implemented.

[0081] The description of the functionality provided by the different computer program components described herein is for illustrative purposes, and is not intended to be limiting, as any of computer program components may provide more or less functionality than is described. For example, one or more of computer program components may be eliminated, and some or all of its functionality may be provided by other computer program components. As another example, processor 11 may be configured to execute one or more additional computer program components that may perform some or all of the functionality attributed to one or more of computer program components described herein.

[0082] The electronic storage media of the electronic storage 13 may be provided integrally (/.e., substantially non-removable) with one or more components of the system 10 and/or as removable storage that is connectable to one or more components of the system 10 via, for example, a port (e.g., a USB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storage 13 may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media e.g., EPROM, EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc. , and/or other electronically readable storage media. The electronic storage 13 may be a separate component within the system 10, or the electronic storage 13 may be provided integrally with one or more other components of the system 10 (e.g., the processor 11). Although the electronic storage 13 is shown in FIG. 1 as a single entity, this is for illustrative purposes only. In some implementations, the electronic storage 13 may comprise a plurality of storage units. These storage units may be physically located within the same device, or the electronic storage 13 may represent storage functionality of a plurality of devices operating in coordination.

[0083] FIG. 2 illustrates a method 200 for monitoring opacity of smoke. The operations of method 200 presented below are intended to be illustrative. In some implementations, method 200 may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. In some implementations, two or more of the operations may occur substantially simultaneously.

[0084] In some implementations, method 200 may be implemented in one or more processing devices (e.g., a digital processor, an analog processor, a digital circuit designed to process information, a central processing unit, a graphics processing unit, a microcontroller, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information). The one or more processing devices may include one or more devices executing some or all of the operations of method 200 in response to instructions stored electronically on one or more electronic storage media. The one or more processing devices may include one or more devices configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of method 200. [0085] Referring to FIG. 2 and method 200, at operation 202, images captured over a duration of time may be obtained. The images may depict smoke emitted from a source. In some implementations, operation 202 may be performed by a processor component the same as or similar to the image component 102 (Shown in FIG. 1 and described herein).

[0086] At operation 204, temporal features of the images may be extracted. The temporal features may be representative of changes across the images. In some implementations, operation 204 may be performed by a processor component the same as or similar to the extraction component 104 (Shown in FIG. 1 and described herein). [0087] At operation 206, spatial features of the images may be extracted. The spatial features may be representative of texture and color in the images. In some implementations, operation 206 may be performed by a processor component the same as or similar to the extraction component 104 (Shown in FIG. 1 and described herein). [0088] At operation 208, the temporal features of the images and the spatial features of the images may be combined. The combined temporal and spatial features of the images may be representative of both static appearance of the smoke and dynamic motion of the smoke. In some implementations, operation 208 may be performed by a processor component the same as or similar to the combine component 106 (Shown in FIG. 1 and described herein).

[0089] At operation 210, a smoke opacity value for the smoke depicted within the images may be determined based on the combined temporal and spatial features of the images and/or other information. In some implementations, operation 210 may be performed by a processor component the same as or similar to the smoke opacity component 108 (Shown in FIG. 1 and described herein).

[0090] At operation 212, smoke opacity monitoring may be facilitated based on the smoke opacity value and/or other information. In some implementations, operation 212 may be performed by a processor component the same as or similar to the monitoring component 110 (Shown in FIG. 1 and described herein).

[0091] Although the system(s) and/or method(s) of this disclosure have been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed implementations, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.

Claims

What is claimed is:

1 . A system for monitoring opacity of smoke, the system comprising: one or more physical processors configured by machine-readable instructions to: obtain images captured over a duration of time, the images depicting smoke emitted from a source; extract temporal features of the images, wherein the temporal features are representative of changes across the images; extract spatial features of the images, wherein the spatial features are representative of texture and color in the images; combine the temporal features of the images and the spatial features of the images, wherein the combined temporal and spatial features of the images are representative of both static appearance of the smoke and dynamic motion of the smoke; determine a smoke opacity value for the smoke depicted within the images based on the combined temporal and spatial features of the images; and facilitate smoke opacity monitoring based on the smoke opacity value.

2. The system of claim 1 , wherein the source includes a flare stack.

3. The system of claim 3, wherein facilitation of the smoke opacity monitoring based on the smoke opacity value includes control of gas combustion at the flare stack based on the smoke opacity value.

4. The system of claim 1 , wherein the smoke opacity value includes a Ringelmann number.

5. The system of claim 1, wherein: dimensionality of the combined temporal and spatial features of the images is reduced; and the combined temporal and spatial features of the images with reduced dimensionality are re-combined with the temporal features of the images to emphasize temporal variations in determination of the smoke opacity value.

6. The system of claim 1 , wherein extraction of the temporal features of the images, extraction of the spatial features of the images, combination of the temporal features of the images and the spatial features of the images, and determination of the smoke opacity value are performed by a deep learning model.

7. The system of claim 6, wherein the deep learning model is pre-trained on a binary classification task to distinguish between smoke and non-smoke.

8. The system of claim 7, wherein the pre-trained deep learning model is subsequently trained to estimate the smoke opacity value.

9. The system of claim 1 , wherein the temporal features of the images are extracted based on calculation of structural similarity index measure scores for the images.

10. The system of claim 1 , wherein the spatial features of the images are extracted using a part of an Inception-ResNet and a part of an EfficientNet.

11. A method for monitoring opacity of smoke, the method comprising: obtaining images captured over a duration of time, the images depicting smoke emitted from a source; extracting temporal features of the images, wherein the temporal features are representative of changes across the images; extracting spatial features of the images, wherein the spatial features are representative of texture and color in the images; combining the temporal features of the images and the spatial features of the images, wherein the combined temporal and spatial features of the images are representative of both static appearance of the smoke and dynamic motion of the smoke; determining a smoke opacity value for the smoke depicted within the images based on the combined temporal and spatial features of the images; and facilitating smoke opacity monitoring based on the smoke opacity value.

12. The method of claim 11, wherein the source includes a flare stack.

13. The method of claim 13, wherein facilitating the smoke opacity monitoring based on the smoke opacity value includes controlling gas combustion at the flare stack based on the smoke opacity value.

14. The method of claim 11, wherein the smoke opacity value includes a Ringelmann number.

15. The method of claim 11 , wherein: dimensionality of the combined temporal and spatial features of the images is reduced; and the combined temporal and spatial features of the images with reduced dimensionality are re-combined with the temporal features of the images to emphasize temporal variations in determination of the smoke opacity value.

16. The method of claim 11 , wherein extracting the temporal features of the images, extracting the spatial features of the images, combining the temporal features of the images and the spatial features of the images, and determining the smoke opacity value are performed by a deep learning model.

17. The method of claim 16, wherein the deep learning model is pre-trained on a binary classification task to distinguish between smoke and non-smoke.

18. The method of claim 17, wherein the pre-trained deep learning model is subsequently trained to estimate the smoke opacity value.

19. The method of claim 11 , wherein the temporal features of the images are extracted based on calculation of structural similarity index measure scores for the images.

20. The method of claim 11 , wherein the spatial features of the images are extracted using a part of an Inception-ResNet and a part of an EfficientNet.

T1