WO2025035229A1

WO2025035229A1 - Validating a machine learning model for generating synthetic rock files

Info

Publication number: WO2025035229A1
Application number: PCT/CN2023/112425
Authority: WO
Inventors: Yan Song HUANG; Olga DOMANOVA
Original assignee: Schlumberger Canada Ltd; Services Petroliers Schlumberger SA; Geoquest Systems BV; Schlumberger Technology Corp
Current assignee: Schlumberger Canada Ltd; Services Petroliers Schlumberger SA; Geoquest Systems BV; Schlumberger Technology Corp
Priority date: 2023-08-11
Filing date: 2023-08-11
Publication date: 2025-02-20
Anticipated expiration: 2026-02-11
Also published as: CN121753040A

Abstract

A ML validation and verification system validates an ML model using cross validation, consistency check, and physical modeling. Cross validation includes splitting a set of original rock files into a training subset and a validation subset, and generating a synthetic rock file to be validated on the validation subset. Consistency check includes generating multiple synthetic rock files using different realization of the ML model and comparing them for consistency. The validation engine generates a synthetic physical simulation and an original physical simulation using the physical model and compares the two simulations.

Description

VALIDATING A MACHINE LEARNING MODEL FOR GENERATING SYNTHETIC ROCK FILES

BACKGROUND OF THE DISCLOSURE

Downhole drilling systems use cutting structures, such as bits, to advance a depth of a wellbore. The bit includes multiple cutting elements. Cutting elements are in different geometries and shapes. A rock file used is to describe the cutting behavior of a particular cutting element (in a particular shape and a particular orientation or cutter-rock interaction status) against a particular rock type in a particular confining pressure. A rock file gives cutting element forces when cutting the rock in a set of orientations and a set of depth of cut. During bit design, the bit may be modeled using rock files of the cutting element. Conventionally, the rock files are generated using experimental data, which may be limited, based on experimental and other limitations.

SUMMARY

In some embodiments, the techniques described herein relate to a method. The method includes receiving a target configuration of a cutting element. The target configuration is related to an original configuration of an original rock file. The method includes applying a machine learning (ML) model to the target configuration to generate a synthetic rock file for the cutting element. The synthetic rock file includes cutting forces associated with the target configuration. The ML validation and verification system validates the ML model by generating a consistency ratio for the synthetic rock file based on validation synthetic rock files generated by the ML model. The validation synthetic rock files are generated by different realizations of the ML model using the target configuration. The ML model is further validated by performing a physical simulation using the synthetic rock file and the original physical simulation using the original rock file.

In some embodiments, the techniques described herein relate to a method. The method may include training a first realization of a machine learning (ML) model for a cutting element using a first training subset of a plurality of original rock files. The method may also include applying the first realization of the ML model to a target configuration for the cutting element to generate a first synthetic rock file. The target configuration may match an original configuration of a first validation subset of the plurality of original rock files. The ML validation and verification system may train a second realization of the ML model using a second training subset of the plurality of original rock files and may apply the second realization of the ML model to the target configuration for the cutting element to generate a second synthetic rock file. The second target configuration may match the original configuration of a second validation subset of the plurality of original rock files. The ML validation and verification system may validate the ML model by tuning a hyperparameter of the ML model based on a first cross validation ratio between the first synthetic rock file and the first validation subset and a second cross validation ratio between the second synthetic rock file and the second validation subset. The validation engine of the ML validation and verification system may generate a consistency ratio for the ML model between the first synthetic rock file and the second synthetic rock file. Using a physical model, the validation engine may perform, a first synthetic physical simulation for the first synthetic rock file and a second synthetic physical simulation for the second synthetic rock file. The validation engine may also perform a first original physical simulation for the first validation subset and a second original physical simulation for the second validation subset. The validation engine may generate a simulation comparison ratio using the first synthetic physical simulation, the second synthetic physical simulation, the first original physical simulation, and the second original physical simulation.

This summary is provided to introduce a selection of concepts that are further described in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter. Additional features and aspects of embodiments of the disclosure will be set forth herein, and in part will be obvious from the description, or may be learned by the practice of such embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and other features of the disclosure may be obtained, a more particular description will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. For better understanding, the like elements have been designated by like reference numbers throughout the various accompanying figures. While some of the drawings may be schematic or exaggerated representations of concepts, at least some of the drawings may be drawn to scale. Understanding that the drawings depict some example embodiments, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

FIG. 1 is a representation of a drilling system for drilling an earth formation 101 to form a wellbore, according to at least one embodiment of the present disclosure;

FIG. 2 is a representation of a ML validation and verification 212, according to at least one embodiment of the present disclosure;

FIG. 3 is a representation of a ML validation and verification system, according to at least one embodiment of the present disclosure;

FIG. 4 is a representation of a cross validation system, according to at least one embodiment of the present disclosure;

FIG. 5 is a representation of a consistency check validation system, according to at least one embodiment of the present disclosure;

FIG. 6 is a representation of a physical comparison validation system, according to at least one embodiment of the present disclosure;

FIG. 7 is a representation of a radar plot generated from a physical simulation comparison of a bit using the synthetic rock file and the original rock file to simulate the cutting elements in the bit arrangement, according to at least one embodiment of the present disclosure;

FIG. 8 is a representation of a box-and-whisker plot generated from a physical simulation comparison of a bit indicating variation in different applications of the physical model, according to at least one embodiment of the present disclosure;

FIG. 9 is a representation of rock file plots, according to at least one embodiment of the present disclosure;

FIG. 10 is a flowchart of a method for validating an ML model, according to at least one embodiment of the present disclosure;

FIG. 11 is a flowchart of a method for validating an ML model, according to at least one embodiment of the present disclosure; and

FIG. 12 is a representation of a computing system, according to at least one embodiment of the present disclosure.

DETAILED DESCRIPTION

This disclosure generally relates to devices, systems, and methods for training and validating a machine learning (ML) model used to generate synthetic rock files for cutting elements utilized on a bit. The ML model may be trained on original rock files of cutting element geometries, orientations, confining pressure, and rock types. The original rock files may be experimentally generated. In some situations, the amount of information available from the original rock files may be limited. The ML model may be trained to generate a large number of synthetic rock files based on a relatively small dataset of the original rock files. In some situations, the ML model may be trained to generate synthetic rock files for cutting elements having different geometries than the geometries used in the original rock files.

In accordance with at least one embodiment of the present disclosure, the ML model may be validated using numerical error minimization through the hyperparameter tuning of the ML model. For example, the ML model may be validated through cross-validation, such as through 90-10 cross validation. In some embodiments, the ML model may be validated through a consistency check across multiple different realizations of the ML model. For example, synthetic rock files generated using different realizations of the ML model may be compared to determine a deviation between the synthetic rock files. In some embodiments, the ML model may be validated by simulating the operation of a bit using a physical model of the bit utilizing the synthetic rock files. The results of the physical model may be compared to the results of the physical model implemented using related original rock files.

Utilizing cross-validation, consistency checks, and physical model evaluations may help to improve the validation and verification of the ML model. For example, based on the results of at least one of the cross-validation, consistency checks, and physical model evaluations, the ML model may be retrained. This may help to improve the accuracy and/or relevance of the synthetic rock files generated by the ML model. This may help a drilling operator to generate a library of synthetic rock files for cutting element configurations for which there are no original rock files. A validated ML model may allow for improved bit (and other cutting structure) design using synthetic rock files. This may improve the drilling efficiency of drilling systems utilizing the bit designed with synthetic rock files.

As illustrated by the foregoing discussion, the present disclosure utilizes a variety of terms to describe features and advantages of the synthetic rock file generation system. Additional detail is now provided regarding the meaning of such terms. For example, as used herein, the term “rock file” refers to a set of numbers describing the physical forces generated while cutting element is interacting with the rock.. In particular, the term “rock file” may include information about the shape of cutting element, such as 3-dimentional shape, size, thickness, orientation (e.g., backrake, siderake) , bevel, any other information, and combinations thereof. In some embodiments, the term “rock file” may include information about the formation, such as formation name, rock type, rock hardness, rock unconfined compressive strength, confining pressure, confining medium, fracture and joint information, any other rock information, and combinations thereof. In some embodiments, the term “rock file” may include cutting forces for the cutting element shape and orientation and the formation information in a set of depth of cut, such as vertical force (e.g., F_v) , side force (F_side) , the cutting force (F_cut) , any other forces, and combinations thereof. A rock file may be determined using experimental data. Such a rock file may be an “original rock file. ” In some embodiments, a rock file may be determined using a model or algorithm, such as a physics model, an analytical model, a numerical simulation model, a machine learning model, any other model, and combinations thereof. Such a model may be a “synthetic rock file. ”

As a specific, non-limiting example, an original rock file may include cutting forces derived experimentally for 3,000 psi confining pressure and 9,000 psi confining pressure. A synthetic rock model may include cutting forces for 6,000 psi confining pressure. In some examples, an original rock file may include cutting forces derived experimentally for cutting element sizes including 9 mm, 13 mm, and 19 mm. The synthetic rock file may include cutting forces for cutting element sizes including 11 mm and 16 mm. In some examples, an original rock file may include cutting forces for a planar shape and the synthetic rock file may include cutting forces for a wedge-shaped shape of the cutting element. In some examples, the original rock file may include cutting forces associated with a backrake of 5° and the synthetic rock file may include cutting forces for backrakes of 0°, 10°, 15°, 20°, and so forth. In some examples, the original rock file may include cutting forces associated with a siderake of 5° and the synthetic rock file may include cutting forces for siderakes of 0°, 10°, and so forth.

As used herein, a “machine learning model” or “ML model” refers to a computer algorithm or model (e.g., a classification model, a regression model, a language model, an object detection model) that may be tuned (e.g., trained) based on an input or input set (e.g., training input) to develop output for unknown information. For example, a machine learning model may refer to a neural network (e.g., a convolutional neural network (CNN) , deep neural network (DNN) , recurrent neural network (RNN) ) , or other machine learning algorithm or architecture that learns and approximates complex functions and generates outputs based on a plurality of inputs provided to the machine learning model.

As used herein, a “physical simulation” or a “physical model” refers to a physical model of a drilling system. The physical simulation may receive as inputs a rock file of one or more cutting elements. Using the rock file (s) , the physical simulation may analyze the impact of the cutting elements in a drilling system. For example, the physical simulation may simulate the operation of a bit in a downhole drilling system. The bit may include multiple cutting elements in a bit design. Each cutting element may have an associated configuration (e.g., shape, orientation, rock type) . The physical simulation may simulate the forces on the bit design using the rock files for the cutting elements in a particular formation and/or a particular rock type. This may result in one or more of an imbalance force, a radial imbalance force, a circumferential imbalance force, a side rake imbalance force, a weight-on-bit (WOB) , a bit torque, any other bit forces, and combinations thereof. While embodiments of the present disclosure may discuss a bit or a bit configuration, the physical simulation may simulate any other cutting device or combination of cutting elements, such as a reamer, a casing cutter, any other cutting device, and combinations thereof. In some embodiments, the physical simulation may utilize any simulation methodology, such as finite element analysis.

As used herein, the term “validation” refers to verification of an ML model. Validating the ML model may include determining whether the ML model produces outputs that are representative of the real-world elements the outputs are simulating. For example, validating the ML models of the present disclosure may include determining whether the cutting forces in the synthetic rock files are representative of the forces a physical cutting element having the properties in the synthetic rock file would exhibit. As will be discussed in further detail herein, validation may occur using one or more mechanisms, such as cross-validation, consistency checks, physical simulations, any other validation mechanism, and combinations thereof.

FIG. 1 shows one example of a drilling system 100 for drilling an earth formation 101 to form a wellbore 102. The drilling system 100 includes a drill rig 103 used to turn a drilling tool assembly 104 which extends downward into the wellbore 102. The drilling tool assembly 104 may include a drill string 105, a bottomhole assembly ( “BHA” ) 106, and a bit 110, attached to the downhole end of drill string 105.

The drill string 105 may include several joints of drill pipe 108 connected end-to-end through tool joints 109. The drill string 105 transmits drilling fluid through a central bore and transmits rotational power from the drill rig 103 to the BHA 106. In some embodiments, the drill string 105 may further include additional components such as subs, pup joints, etc. The drill pipe 108 provides a hydraulic passage through which drilling fluid is pumped from the surface. The drilling fluid discharges through selected-size nozzles, jets, or other orifices in the bit 110 for the purposes of cooling the bit 110 and cutting structures thereon, and for lifting cuttings out of the wellbore 102 as it is being drilled.

The BHA 106 may include the bit 110 or other components. An example BHA 106 may include additional or other components (e.g., coupled between to the drill string 105 and the bit 110) . Examples of additional BHA components include drill collars, stabilizers, measurement-while-drilling ( “MWD” ) tools, logging-while-drilling ( “LWD” ) tools, downhole motors, underreamers, section mills, hydraulic disconnects, jars, vibration or dampening tools, other components, or combinations of the foregoing. The BHA 106 may further include a rotary steerable system (RSS) . The RSS may include directional drilling tools that change a direction of the bit 110, and thereby the trajectory of the wellbore. At least a portion of the RSS may maintain a geostationary position relative to an absolute reference frame, such as gravity, magnetic north, and/or true north. Using measurements obtained with the geostationary position, the RSS may locate the bit 110, change the course of the bit 110, and direct the directional drilling tools on a projected trajectory.

In general, the drilling system 100 may include other drilling components and accessories, such as special valves (e.g., kelly cocks, blowout preventers, and safety valves) . Additional components included in the drilling system 100 may be considered a part of the drilling tool assembly 104, the drill string 105, or a part of the BHA 106 depending on their locations in the drilling system 100.

The bit 110 in the BHA 106 may be any type of bit suitable for degrading downhole materials. For instance, the bit 110 may be a drill bit suitable for drilling the earth formation 101. Example types of drill bits used for drilling earth formations are fixed-cutter or drag bits. In other embodiments, the bit 110 may be a mill used for removing metal, composite, elastomer, other materials downhole, or combinations thereof. For instance, the bit 110 may be used with a whipstock to mill into casing 107 lining the wellbore 102. The bit 110 may also be a junk mill used to mill away tools, plugs, cement, other materials within the wellbore 102, or combinations thereof. Swarf or other cuttings formed by use of a mill may be lifted to surface, or may be allowed to fall downhole.

In accordance with at least one embodiment of the present disclosure, the cutting elements on the bit 110 or other cutting structure may be described by a rock file for the particular cutting element in a particular configuration. As discussed herein, one or more of the rock files may be an original rock file, developed at least partially using experimental data. For example, the cutting forces of the original rock file may be directly measured in an experimental setup by cutting a formation with a cutting element having a particular configuration and measuring the resulting forces on the cutting element. In accordance with at least one embodiment of the present disclosure, the rock files used to describe the cutting element on the bit 110 may be a synthetic rock file. The synthetic rock file may include information about the cutting element that is not generated experimentally. In some embodiments, at least a portion of the cutting forces for the cutting element is generated experimentally and a portion is generated using one or more models (e.g., an analytical model and/or a ML model discussed herein) . In some embodiments, none of the information in the synthetic rock file may be generated experimentally.

The synthetic rock files used to design the bit 110 may be determined with an ML model. For example, the ML model may be applied to a target configuration of the cutting element and used to develop the cutting forces associated with the target configuration. In some embodiments, the target configuration may be the same as, similar to, and/or related to an original configuration of an original rock file. In some embodiments, the target configuration may be different than an original configuration of an original rock file. For example, as discussed herein, the ML model may be validated against the original rock files by generating synthetic rock files related to the original configuration of the original rock files.

In accordance with at least one embodiment of the present disclosure, the ML model may be validated to improve the accuracy and/or representativeness of the resulting synthetic rock files to physical conditions. The ML model may be validated using one or more validation mechanisms. For example, as discussed in further detail herein, the ML model may be validated using a cross-validation mechanism, a consistency check, and a physical simulation of the drilling system in which the cutting element described by the synthetic rock file may be used.

In some embodiments, the physical simulation may simulate the operation of the BHA 106 and/or individual elements of the BHA 106. This may allow the drill designer and/or drill planner to determine how the BHA 106 may behave in a particular set of drilling conditions, such as the earth formation 101, the depth of the BHA 106, the confining pressure of the earth formation 101 at the BHA 106, the rock type of the earth formation 101 at the BHA 106, the configuration of the BHA 106, any other elements of the drilling system 100, and combinations thereof.

FIG. 2 is a representation of a ML validation and verification system 212, according to at least one embodiment of the present disclosure. The ML validation and verification system 212 includes a ML model 214. The ML model 214 may be trained using original rock files 216. The ML model 214 may be trained to output one or more synthetic rock files 218.

The ML model 214 may be validated using a validation engine 220. The validation engine 220 may determine whether the synthetic rock files 218 generated by the ML model 214 are representative of the actual cutting forces experienced by the associated physical cutting element. In some embodiments, the ML validation and verification system 212 may utilize cross-validation. For example, to validate the ML model 214, ML validation and verification system 212 may split the original rock files 216 into a training subset 222 and a validation subset 224 and one or more test subsets. The training subset 222 may be used to train the ML model 214. For example, the training subset 222 may be used to train the ML model 214 to generate the synthetic rock files 218 to be representative of the portion of the original rock files 216 in the training subset 222. The test subsets may be used to test the trained ML model 214.

The validation subset 224 may be used by the validation engine 220 to validate the ML model 214. For example, the ML model 214 may generate the synthetic rock files 218 using known cutting element configurations, such as the cutting element configurations in the validation subset 224. The validation engine 220 may compare the synthetic rock files 218 to the validation subset 224 to determine whether the synthetic rock files 218 match the validation subset 224.

The ML validation and verification system 212 may determine 226 whether the synthetic rock file 218 is validated, or whether the synthetic rock file 218 matches or approximately matches the validation subset 224. For example, the ML validation and verification system 212 may determine 226 how different the synthetic rock file 218 is from the validation subset 224. If the synthetic rock file 218 matches the validation subset 224, or exceeds a threshold match, then an ML tuner 228 may re-train the ML model 214 using the synthetic rock file 218.

In some embodiments, the ML tuner 228 may tune one or more hyperparameters of the ML model 214. The hyperparameters of the ML model 214 may be the parameters in the ML model 214 that control the learning process. Other parameters of the ML model 214 may be adjusted or derived during training of the ML model 214. For example, the ML tuner 228 may identify which portions of the synthetic rock files 218 do not match the validation subset 224 of original rock files. The ML tuner 228 may determine which hyperparameter or hyperparameters may adjust the outputted synthetic rock files 218. Adjusting or tuning the hyperparameters may adjust the outputted the outputted synthetic rock files 218.

In some embodiments, if the ML validation and verification system 212 determines 226 that the synthetic rock files 218 are validated by the validation subset 224, the ML validation and verification system 212 may output one or more validated synthetic rock files 230. The validated synthetic rock files 230 may be representative of the physical conditions experienced by the cutting element.

In some embodiments, the validation engine 220 may implement cross-validation with any proportion of the original rock files 216. For example, the validation engine 220 may implement 90-10 cross validation. In 90-10 cross validation the validation engine 220 may split the original rock files 216 into 10 sections, resulting in 10 different training subsets 222 and 10 different validation subsets 224. The ML validation and verification system 212 may generate a synthetic rock file 218 for each of the 10 sections. Each synthetic rock file 218 may be validated with the associated validation subset 224. In some embodiments, each synthetic rock files 218 may be generated using the configuration or one of the configurations of the associated validation subset 224.

Each of the synthetic rock files 218 may be validated by the validation engine 220 and the ML validation and verification system 212 may determine 226 whether the synthetic rock files 218 match the validation subset 224. The ML tuner 228 may tune one or more hyperparameters of the ML model 214 based on the synthetic rock files 218 generated using the 90-10 cross validation. In some embodiments, the ML tuner 228 may tune the ML model 214 using each of the synthetic rock files 218 in the 90-10 cross validation. In some embodiments, the ML tuner 228 may tune the ML model 214 using the synthetic rock files 218 that did not match the associated validation subsets 224. In some embodiments, the ML tuner 228 may tune the ML model 214 using one or more synthetic rock files 218 that do match the associated validation subsets 224. For example, the ML tuner 228 may tune the ML model 214 using synthetic rock files 218 that match the validation subset 224 if one or more of the synthetic rock files 218 do not match the associated validation subset 224.

In some embodiments, the validation engine 220 may validate the ML model 214 using a consistency check of synthetic rock files 218 generated by different realizations of the ML model 214. For example, the ML model 214 may include different realizations trained by different training subset 222. The resulting synthetic rock files 218 may be generated using the same cutting element configuration. The validation engine 220 may compare the cutting forces of the synthetic rock files 218 generated by different realizations of the ML model 214. The validation engine 220 may generate a consistency ratio based on the differences between the various synthetic rock files 218. If the consistency ratio is different than a consistency ratio threshold, then the ML validation and verification system 212 may determine 226 that the ML model 214 is not validated, and the ML model 214 may be retrained using the differences between the synthetic rock files 218. This may help to improve the consistency of the ML model 214.

In some embodiments, the validation engine 220 may perform a physical simulation of a bit or other cutting structure having cutting elements simulated by the synthetic rock files 218 (e.g., a synthetic physical simulation) . In some embodiments, the validation engine 220 may perform a simulation of a bit or other cutting structure having cutting elements simulated by the validation subset 224 (e.g., an original physical simulation) , with the synthetic rock files 218 and the validation subset 224 having the same configuration.

The validation engine 220 may compare the synthetic physical simulation using the synthetic rock files 218 and the original physical simulation using the validation subset 224 of the original rock files 216. For example, the validation engine 220 may generate a simulation comparison ratio of one or more bit performances generated by the synthetic physical simulation and the original physical simulation. If the simulation comparison ratio is above a simulation comparison ratio threshold, the validation engine 220 may validate the ML model 214. If the simulation comparison ratio is below the simulation comparison ratio threshold, the validation engine 220 may not validate the ML model 214 and the validation engine 220 may re-train the ML model 214 based on the synthetic physical simulations and the original physical simulations.

In some embodiments, the validation engine 220 may validate the ML model 214 using each of the validation mechanisms simultaneously. For example, the validation engine 220 may validate the ML model 214 using all three of cross-validation, consistency check, and physical simulation. This multi-factor cross-validation may help to improve the validation and verification of the ML model 214. In some embodiments, the validation engine 220 may validate the ML model 214 using only one or two of the validation mechanisms.

In accordance with at least one embodiment of the present disclosure, the ML model 214 may be trained using a relatively small dataset. For example, as discussed herein, the quantity of original rock files 216 may be limited, such as by constraints in the availability of experimental data used to generate the original rock files 216.

In some embodiments, the ML model 214 may be relatively small. For example, the ML model 214 may include less than 5 hidden layers. In some examples, the ML model 214 may include between 50 and 100 parameters. In one particular, non-limiting example, the ML model 214 have one hidden layer and 66 parameters. The small size of the ML model 214 may allow the ML model 214 to be trained on a small dataset. As discussed herein, the validation engine 220 may validate and re-train the ML model 214 if the resulting synthetic rock files 218 are not validated using cross-validation, consistency check, and physical simulation. The ML model 214 may be re-trained until the synthetic rock files 218 are validated. When the synthetic rock files 218 are validated, the ML model 214 may be validated and the ML model 214 may be used to generate additional synthetic rock files 218.

In some embodiments, the validated ML model 214 may be used to generate synthetic rock files 218 for configurations of cutting elements that are not included in the original rock files 216. To generate the synthetic rock files 218, the ML model 214 may receive a target configuration for the cutting element, the target configuration including a shape of the cutting element, an orientation of the cutting element (e.g., back rake, side rake) , depth of cut, a rock type for the cutting element, any other portion of the configuration, and combinations thereof. In some examples, the target configuration may be the same as, similar to, or related to an original configuration of an original rock file. In some examples, the new synthetic rock files 218 may be generated for target orientations having new geometries, orientations, shapes, or other configurations of the synthetic rock files 218 that are not included in the original rock files 216. For example, new synthetic rock files 218 may be generated for new configurations that are different than the original configuration. In this manner, the ML validation and verification system 212 may generate the synthetic rock files 218 for cutting elements and/or configurations of cutting elements that have not been physically tested. This may help a drilling operator to design and build a bit and understand the forces and associated cutting action. This may help to improve the drilling efficiency of the drill bit and drilling system.

FIG. 3 is a representation of a ML validation and verification system 312, according to at least one embodiment of the present disclosure. Each of the components of the ML validation and verification system 312 may include software, hardware, or both. For example, the components may include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the ML validation and verification system 312 may cause the computing device (s) to perform the methods described herein. Alternatively, the components may include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components of the ML validation and verification system 312 may include a combination of computer-executable instructions and hardware.

Furthermore, the components of the ML validation and verification system 312 may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components may be implemented as one or more web-based applications hosted on a remote server. The components may also be implemented in a suite of mobile device applications or “apps. ”

The ML validation and verification system 312 includes a ML model 314. The ML model 314 may be trained to generate synthetic rock files when applied to a target configuration of a cutting element. As discussed herein, the target configuration may be the same as, similar to, or related to an original configuration of an original rock file. In some examples, the target configuration may be different than the original configuration. The ML model 314 may be trained in any manner. For example, the ML model 314 may be trained using a deep learning training system 332. The deep learning training system 332 may train the ML model 314 by receiving original rock files as input and identifying connections and/or comparisons between the elements of the original rock files. In this manner, the deep learning training system 332 may train the ML model 314 to generate synthetic rock files using the identified connections between the original rock files.

In some embodiments, the ML model 314 may be trained using a transfer learning training system 334. The transfer learning training system 334 may train the ML model 314 to generate synthetic rock files using dissimilar data. Transfer learning may include transferring at least a portion of a first model trained on a first set of data. The transferred portion may be used as the starting point for the ML model 314. The ML model 314 may then be trained for a second set of data that is dissimilar to the data originally used to train the first model. This may allow the ML model 314 to generate synthetic rock files for cutting elements that have dissimilar geometries and/or dissimilar shapes to the cutting elements used to generate the original rock files.

As a non-limiting example, a first ML model may be trained using original rock files generated with a cylindrical cutting element. The first ML model, or at least a portion of the first ML model, may be used as the starting point for the ML model 314. The transfer learning training system 334 may train the ML model 314 to generate synthetic rock files for cutting elements having a different shape, such as a non-cylindrical shape, including conical, wedge, offset wedge, offset conical, convex, concave, any other shape, and combinations thereof. In some embodiments, the ML model 314 may be re-trained using the validation engine 320 or other retraining mechanism to further refine and validate the ML model 314.

The ML validation and verification system 312 includes a validation engine 320. The validation engine 320 may validate the ML model 314 to help the generated synthetic rock files to be representative of the actual forces experienced by the physical cutting element. This may help to improve the accuracy and/or reliability of the ML model 314.

The validation engine 320 may validate the ML model 314 in any manner. For example, as discussed herein, the validation engine 320 may validate the ML model 314 using cross validation 336. As discussed herein, cross validation 336 may include generating multiple subsets of training original rock files and validation original rock files. The ML model 314 may be trained on each of the training original rock files and generate an associated synthetic rock file. The synthetic rock files may be compared to the validation original rock files to determine how similar the synthetic rock files are to the validation original rock files. If the synthetic rock files are not within a cross validation threshold ratio, then the ML model 314 may be retrained using the results from the cross validation 336.

In some embodiments, the cross validation 336 may include a 90-10 cross validation, with 10%of the original rock files reserved for validation and 90%of the original rock files used to train the ML model 314. This may allow for 10 separate instances or realizations of the ML model 314. In some embodiments, the cross validation 336 may include any ratio of cross validation, such as 50-50, 60-40, 70-30, 80-20, 90-10, 95-5, 99-1, or any ratio therebetween.

In some embodiments, the validation engine 320 may validate the ML model 314 using a consistency check 338. The consistency check 338 may include generating a synthetic rock file for multiple realizations of the ML model 314. A realization of the ML model 314 may be a version of the ML model 314 that is trained using a separate training subset of the original rock files. The different realizations of the ML model 314 may generate a synthetic rock file for the same target configuration of a cutting element. The consistency check 338 of the validation engine 320 may compare the generated synthetic rock files and generate a consistency ratio to determine the variation in the synthetic rock files. If the consistency ratio is below a consistency ratio threshold, the consistency check 338 of the validation engine 320 may cause the ML model 314 to be retrained to reduce the variation in the synthetic rock files. If the consistency ratio is above the consistency ratio threshold, the consistency check 338 of the validation engine 320 may determine that the ML model 314 is at least partially validated.

The validation engine 320 may further validate the ML model 314 using a physical simulation 340. The physical simulation 340 may simulate a bit or other cutting structure using the cutting information for synthetic rock files for cutting elements that may be used on the bit (e.g., a synthetic physical simulation) . The physical simulation 340 may result in bit performances representative of forces that may be experienced by the bit. In some embodiments, the physical simulation 340 may be performed on both the synthetic rock files and original rock files having the same configuration of cutting elements (e.g., an original physical simulation) . This may allow the validation engine 320 to compare the bit performances between the synthetic and original rock files. The validation engine 320 may generate a simulation comparison ratio between the synthetic and original simulations. If the simulation comparison ratio is below a simulation comparison ratio threshold, then the validation engine 320 may cause the ML model 314 to be retrained to reach the simulation comparison ratio threshold. If the simulation comparison ratio is above the simulation comparison ratio threshold, then the physical simulation 340 of the validation engine 320 may determine that the ML model 314 is at least partially validated. In some embodiments, the validation engine 320 may generate a performance comparison ratio for each individual bit performance.

In some embodiments, the validation engine 320 may perform the physical simulation 340 (either or both of the synthetic physical simulation or the original physical simulation) using multiple ROPs in ROP control mode. For example, the validation engine 320 may perform the physical simulation 340 using ROPs including 1 foot per hour (ft/hr. ) , 5 ft/hr., 10 ft/hr., 25 ft/hr., 50 ft/hr., 100 ft/hr., 250 ft/hr., 500 ft/hr., or any value therebetween. The validation engine 320 may compare the bit performances for the different ROPs to identify the performance of the synthetic rock files at various ROPs. In some embodiments, the validation engine 320 may compare the bit performances for the different ROPs to determine in which areas the cutter forces of the synthetic rock files may be improved. In this manner, the validation engine 320 may retrain the ML model 314 to further improve the synthetic rock files.

In some embodiments, the validation engine 320 may perform the physical simulation 340 using different arrangements of configurations of cutting elements, including different configurations at different ROPs. In this manner, the ML model 314 may help to identify bit configurations that are stable and/or efficient.

In some embodiments, the validation engine 320 may perform the physical simulation 340 using the synthetic rock files to perform a bit sensitivity analysis. For example, the validation engine 320 may perform the physical simulation 340 using random variation or noise in the synthetic rock files. In this manner, the ML validation and verification system 312 may identify the sensitivity of the bit to changes in the synthetic rock files. In some embodiments, the ML validation and verification system 312 may identify the sensitivity of the synthetic rock files to changes in configuration.

FIG. 4 is a representation of a cross validation system 442, according to at least one embodiment of the present disclosure. The cross validation system 442 includes a ML model 414 that is applied to a target configuration of a cutting element. The ML model 414 may generate one or more synthetic rock files 418 for the target configuration. The synthetic rock files 418 may be analyzed by a rock file manager 444. The rock file manager 444 may receive validation original rock files 446. The validation original rock files 446 may be original rock files having the same configuration as the synthetic rock files 418 (e.g., the target configuration) .

The rock file manager 444 may compare the synthetic rock files 418 to the validation original rock files 446 to determine a cross validation ratio of one or more cutting forces of the synthetic rock file 418 (e.g., forces in the synthetic rock file 418 divided by the respective forces in validation original rock file 446) . The cross validation ratio may include any other comparison on the forces or other values of the synthetic rock file 418, such as the mechanical specific energy (MSE) or other comparison. If the cross validation ratio is not within a cross validation ratio threshold range, then a ML tuner 428 may tune (e.g., re-train) the ML model 414 using the differences between the synthetic rock file 418 and the validation original rock file 446. In some embodiments, a lower boundary for the cross-validation threshold range may be in a range having an upper value, a lower value, or upper and lower values including any of 80%, 90%, 95%, 97%, 98%, 99%, 99.5%, or any value therebetween. In some embodiments, an upper boundary for the cross-validation threshold range may be in a range having an upper value, a lower value, or upper and lower values including any of 100.5%, 101%, 102%, 103%, 105%, 110%, 120%, or any value therebetween. In some embodiments, the cross validation threshold range may be any combination of lower boundary and upper boundary. For example, the cross validation threshold range may be between 99.5%and 100.5%. In some examples, the cross validation threshold range may be between 99%, and 100%. In some examples, the cross validation threshold range may be between 98%and 102%. In some examples, the cross validation threshold range may be any combination of boundaries discussed herein.

In some embodiments, if the rock file manager 444 determines that the synthetic rock files 418, or the cutting forces of the synthetic rock files 418, are within the cross validation threshold range, then the rock file manager 444 may validate or at least partially validate the ML model 414. In this manner, the cross validation system 442 may help to validate the ML model 414 using the validation original rock files 446 having the same configuration as the synthetic rock files 418. This may help to improve the accuracy and/or representation of the ML model 414 to the physical cutting elements.

FIG. 5 is a representation of a consistency check validation system 548, according to at least one embodiment of the present disclosure. The consistency check validation system 548 includes a ML model 514. The ML model 514 may include multiple ML model realizations (collectively 550) . The different ML model realizations 550 may be generated using different training data sets.

In some embodiments, the ML model realizations 550 may generate a set of validation synthetic rock files (collectively 552) . The validation synthetic rock files 552 may all be generated using the same cutting element configuration. A consistency manager 554 may review the validation synthetic rock files to determine a consistency ratio between two or more of the validation synthetic rock files 552.

For example, a first ML model realization 550-1 may generate a first validation synthetic rock file 552-1. A second model realization 550-2 may generate a second validation synthetic rock file 552-2. The consistency check validation system 548 may include n-number of ML model realizations 550, with an nth ML model realization 550-n generating an nth validation synthetic rock file 552-n.

The consistency manager 554 may generate a consistency ratio between the validation synthetic rock files 552. For example, the consistency manager 554 may generate a consistency ratio between the first validation synthetic rock file 552-1 and the second validation synthetic rock file 552-2. In some examples, the consistency manager 554 may generate a consistency ratio between the first validation synthetic rock file 552-1 and the nth validation synthetic rock file 552-n. In some embodiments, the consistency manager 554 may a consistency ratio for every combination of validation synthetic rock files 552.

The consistency manager 554 may determine whether the consistency ratio is greater than a consistency ratio threshold. In some embodiments, the consistency ratio threshold may be in a range having an upper value, a lower value, or upper and lower values including any of 80%, 90%, 95%, 97%, 98%, 99%, 99.5%, 100.5%, 101%, 102%, 103%, 105%, 110%, 120%, or any value therebetween. For example, the consistency ratio threshold may be greater than 80%. In another example, the consistency ratio threshold may be less than 120%. In yet other examples, the consistency ratio threshold may be any value in a range between 80%and 120%. In some embodiments, it may be critical that the consistency ratio threshold is greater than 98%to improve the validation of the ML model 514.

If the consistency ratio is less than the consistency ratio threshold, then the consistency manager 554 may cause the ML model 514 to be retrained to improve the consistency of the synthetic rock files. For example, the consistency manager 554 may identify which portion of the synthetic rock files are causing the difference in the consistency ratio. The consistency manager 554 may direct the training of the ML model 514 to improve the consistency of the ML model 514.

FIG. 6 is a representation of a physical comparison validation system 656, according to at least one embodiment of the present disclosure. The physical comparison validation system 656 includes an ML model 614. The ML model 614 may be applied to a target configuration for a cutting element to generate one or more synthetic rock files 618.

A synthetic rock file parser 658 may receive the synthetic rock files 618 and apply the more synthetic rock files 618 to a physical model 640. The synthetic rock file parser 658 may receive bit designs for the physical model 640 from a physical model library 660. For example, the physical model library 660 may include one or more bit or other cutting structure designs. A particular bit design may include a combination of cutting element configurations. The physical model 640 may analyze the bit performances or forces experienced by the bit having the cutting elements arranged in the bit design. The cutting element configurations may be modeled by the synthetic rock files to generate the bit performances.

In some embodiments, the physical comparison validation system 656 may generate the bit performances using original rock files 616 having the same configuration as the synthetic rock files. A physical comparison manager 662 may compare the original bit performances from the original rock files 616 and the synthetic bit performances from the synthetic rock files 618 to generate a physical comparison ratio. If the physical comparison ratio is within a physical comparison ratio threshold range, the physical comparison manager 662 may determine that the ML model 614 is at least partially validated. If the physical comparison ratio is not within the physical comparison ratio threshold range, the physical comparison manager 662 may cause the ML model 614 to be further trained or retrained to improve how representative the synthetic rock files 618 are to the physical cutting elements.

In some embodiments, a lower boundary for the physical comparison ratio threshold range may be in a range having an upper value, a lower value, or upper and lower values including any of 80%, 90%, 95%, 97%, 98%, 99%, 99.5%, or any value therebetween. In some embodiments, an upper boundary for the physical comparison ratio threshold range may be in a range having an upper value, a lower value, or upper and lower values including any of 100.5%, 101%, 102%, 103%, 105%, 110%, 120%, or any value therebetween. In some embodiments, the physical comparison ratio threshold range may be any combination of lower boundary and upper boundary. For example, the physical comparison ratio threshold range may be between 99.5%and 100.5%. In some examples, the physical comparison ratio threshold range may be between 99%, and 100%. In some examples, the physical comparison ratio threshold range may be between 98%and 102%. In some examples, the physical comparison ratio threshold range may be any combination of boundaries discussed herein.

In some embodiments, the physical model 640 may generate bit performances including one or more of an imbalance force, a radial imbalance force, a circumferential imbalance force, a side rake imbalance force, a WOB, a bit torque, any other bit forces, and combinations thereof. The physical comparison manager 662 may generate the physical comparison ratio for each of the bit performances. If any of the bit performances are outside of the physical comparison ratio threshold range, the physical comparison manager 662 may cause the ML model 614 to be retrained based on that bit performance. In some embodiments, the WOB may be more related to the F_v, and if the WOB physical comparison ratio is outside of the associated threshold range, then the physical comparison manager 662 may cause the ML model 614 to be retrained to improve the F_v. The imbalance force (s) may be more related to the F_side, and if the imbalance physical comparison ratio is outside of the associated threshold range, then the physical comparison manager 662 may cause the ML model 614 to be retrained to improve the F_side. The bit torque force may be more related to the F_cut, and if the bit torque physical comparison ratio is outside of the associated threshold range, then the physical comparison manager 662 may cause the ML model 614 to be retrained to improve the F_cut.

In some embodiments, as discussed herein, the physical comparison validation system 656 may perform multiple physical models 640 in different configurations. For example, the physical comparison validation system 656 may perform multiple physical models 640 for different ROPs in ROP control mode. This may allow the physical comparison manager 662 to determine the impact of synthetic rock files on the bit performances at various ROPs, and retrain the ML model 614 accordingly. In some examples, the physical comparison validation system 656 may perform multiple physical models 640 for statistical variations in the configuration of the synthetic rock files 618. This may help to determine how sensitive the bit and/or the bit performances are to noise in the synthetic rock files 618.

FIG. 7 is a representation of a radar plot 764 generated from a physical simulation comparison of a bit using the synthetic rock file and original rock file to simulate the cutting elements in the bit arrangement, according to at least one embodiment of the present disclosure. The radar plot 764 identifies six bit performances comparison generated based on the physical simulation results, including imbalance force 766, radial imbalance force 768, circumferential imbalance force 770, side rake imbalance force 772, WOB 774, and bit torque 776.

The radar plot 764 may visually identify a ratio for each of the bit performances, with halfway between the center and the perimeter of the radar plot 764 indicating a ratio of 1: 1, or 100%, indicating that the compared bit performances are the same. The radar plot 764 may include multiple lines on the radar plot 764, with each line indicating a different application of the physical model. As discussed herein, different applications of the physical model may have different parameters. For example, different applications of the physical model may include a synthetic physical model applied to a bit modeled with synthetic rock files and an original physical model applied to a bit modeled with original rock files having the same configuration. This may allow the drill operator and/or drill planner to determine the difference between the synthetic rock files and the original rock files. The ML model may be refined using the bit performances and/or other information identified or inferred from the radar plot 764.

In some examples, different lines on the radar plot 764 may indicate the bit modeled at different ROPs in ROP control mode. This may provide the drill operator and/or drill planner with information regarding the bit stability at different ROPs and/or information regarding the synthetic rock files representation of the physical cutting elements at the various ROPs. In some examples, different lines on the radar plot 764 may indicate the bit modeled with noise introduced into the synthetic rock files. This may provide the drill operator and/to drill planner with information regarding the bit sensitivity to various parameters.

FIG. 8 is a representation of a box-and-whisker plot 878 generated from a physical simulation comparison of a bit indicating variation in different applications of the physical model, according to at least one embodiment of the present disclosure. The box-and-whisker plot 878 identifies six bit performances modeled by the physical model, including imbalance force 866, radial imbalance force 868, circumferential imbalance force 870, side rake imbalance force 872, WOB 874, and bit torque 876.

The box-and-whisker plot 878 may identify the statistical variation in the different applications of the physical model, including an indication of the mean, standard deviation, and outliers. A drill operator and/or drill planner may review the box-and-whisker plot 878 to determine the variation in the various bits modeled using the synthetic rock files. In some embodiments, the box-and-whisker plot 878 may be generated using a ratio between two applications, such as a ratio between a bit generated using synthetic rock files and original rock files. In some embodiments, the box-and-whisker plot 878 may be generated using variation in the input synthetic rock files to identify the sensitivity of the bit to changes in the synthetic rock files.

FIG. 9 is a representation of rock file plots 980 for an original rock file plot 980-1 and a synthetic rock file plot 980-2, according to at least one embodiment of the present disclosure. The rock file plots 980 include cutting lines 982 of different sizes of cutting elements having the same configuration (e.g., back rake, side rake, and confining pressure) , with the lines 982 plotted with depth of cut 984 on the horizontal axis (x-axis) and cutting force 986 on the vertical axis (y-axis) .

As may be seen, original cutting lines 982-1 of the original rock file plot 980-1 and synthetic cutting lines 982-2 of the synthetic rock file plot 980-2 may be similar. This may indicate that the synthetic rock file is representative of the physical cutting element represented by the original rock file.

In some embodiments, the synthetic rock file may “correct” non-representative data in the original rock file. For example, due to experimental error, the original rock file for one or more diameters of cutting elements may include errors. The synthetic rock file generated from a validated ML model may generate the synthetic cutting lines 982-2 that are more representative of the actual physical state of the cutting element. As a specific non-limiting example, in the original rock file plot 980-1 an original 9 mm cutting element line 987-1 may have a higher force 986 than an original 11 mm cutting element line 988-1, even though larger cutting elements have higher forces based on the larger size of the cutting element. In the synthetic rock file plot 980-2, a synthetic 9 mm cutting element line 987-2 may have a lower force than a synthetic 11 mm cutting element line 988-2. This may indicate that the synthetic rock file has “corrected” the original rock file, or has generated forces that do not include any experimental error.

In accordance with at least one embodiment of the present disclosure, a drill operators’ library of rock files may be supplemented and/or at least partially replaced with synthetic rock files. This may result in a rock file library that has a lower impact from experimental error, thereby improving the quality and/or completeness of the rock file library. In some embodiments, the rock file library may be reliable, thereby improving the simulation results used based on the rock file library.

FIG. 10 and FIG. 11, the corresponding text, and the examples provide a number of different methods, systems, devices, and computer-readable media of the ML validation and verification system. In addition to the foregoing, one or more embodiments may also be described in terms of flowcharts comprising acts for accomplishing a particular result, as shown in FIG. 10 and FIG. 11. FIG. 10 and FIG. 11 may be performed with more or fewer acts. Further, the acts may be performed in differing orders. Additionally, the acts described herein may be repeated or performed in parallel with one another or parallel with different instances of the same or similar acts.

As mentioned, FIG. 10 illustrates a flowchart of a series of acts 1089 for validating and verifying a ML model in accordance with one or more embodiments. While FIG. 10 illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown in FIG. 10. The acts of FIG. 10 may be performed as part of a method. Alternatively, a computer-readable medium may comprise instructions that, when executed by one or more processors, cause a computing device to perform the acts of FIG. 10. In some embodiments, a system may perform the acts of FIG. 10.

The ML validation and verification system may receive a target configuration of a cutting element at 1090. The target configuration may include an orientation of the cutting element, a shape of the cutting element, and a rock type. The target configuration may be related to an original configuration of an original rock file. The ML validation and verification system may apply a ML model to the target configuration to generate a synthetic rock file for the cutting element at 1091. The synthetic rock file may include cutting forces associated with the target configuration.

The ML validation and verification system may validate the ML model at 1092. To validate the ML model, a validation engine may generate a consistency ratio for the synthetic rock file at 1093. The consistency ratio may be based on validation synthetic rock files generated by the ML model that are generated by different realizations of the ML model using the target configuration. The validation engine may perform a physical simulation using the synthetic rock file at 1094. In some embodiments, performing the physical simulation may include performing a synthetic physical simulation. In some embodiments, performing the physical simulation may include performing an original physical simulation. The validation engine may generate a simulation comparison ratio using the synthetic physical simulation and the original physical simulation.

In some embodiments, when at least one of the simulation comparison ratio is greater than a simulation comparison ratio threshold or the consistency ratio threshold is greater than a consistency ratio threshold, the ML model may generate a new synthetic rock file for a new configuration. The new configuration may be different than the target configuration and the original configuration. In some embodiments, when at least one of the simulation comparison ratio is less than a simulation comparison ratio threshold or the consistency ratio threshold is less than a consistency ratio threshold, the ML model may be trained based on the synthetic rock file. In some embodiments, retraining the ML model may include retraining the ML model until the simulation comparison ratio and/or the consistency ratio are greater than their respective thresholds.

As mentioned, FIG. 11 illustrates a flowchart of a series of acts 1100 for validating and verifying a ML model in accordance with one or more embodiments. While FIG. 11 illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown in FIG. 11. The acts of FIG. 11 may be performed as part of a method. Alternatively, a computer-readable medium may comprise instructions that, when executed by one or more processors, cause a computing device to perform the acts of FIG. 11. In some embodiments, a system may perform the acts of FIG. 11.

A ML validation and verification system may train a first realization of a ML model for a cutting element at 1101. The ML model may be trained using a first training subset of a plurality of original rock files. The ML validation and verification system may apply the first realization of the ML model to a target configuration for the cutting element to generate a first synthetic rock file at 1102. The target configuration may match an original configuration of a first validation subset of the plurality of original rock files.

The ML validation and verification system may train a second realization of the ML model using a second training subset of the plurality of original rock files at 1103. The ML validation and verification system may apply the second realization of the ML model to the target configuration for the cutting element to generate a second synthetic rock file at 1104. The second target configuration may match the original configuration of a second validation subset of the plurality of original rock files.

The ML validation and verification system may validate the ML model at 1105. To validate the ML model, a validation engine may tune a hyperparameter of the ML model at 1106. The tuning may be based on a first cross validation ratio between the first synthetic rock file and the first validation subset and a second cross validation ratio between the second synthetic rock file and the second validation subset. The validation engine may generate a consistency ratio for the ML model between the first synthetic rock file and the second synthetic rock file at 1107.

The validation engine may perform, using a physical model, a first synthetic physical simulation for the first synthetic rock file and a second synthetic physical simulation for the second synthetic rock file at 1108. The validation engine may perform, using the physical model, a first original physical simulation for the first validation subset and a second original physical simulation for the second validation subset at 1109. The validation engine may generate a simulation comparison ratio using the first synthetic physical simulation, the second synthetic physical simulation, the first original physical simulation, and the second original physical simulation at 1110.

FIG. 12 illustrates certain components that may be included within a computer system 1200. One or more computer systems 1200 may be used to implement the various devices, components, and systems described herein.

The computer system 1200 includes a processor 1201. The processor 1201 may be a general-purpose single or multi-chip microprocessor (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM) ) , a special purpose microprocessor (e.g., a digital signal processor (DSP) ) , a microcontroller, a programmable gate array, etc. The processor 1201 may be referred to as a central processing unit (CPU) . Although just a single processor 1201 is shown in the computer system 1200 of FIG. 12, in an alternative configuration, a combination of processors (e.g., an ARM and DSP) could be used.

The computer system 1200 also includes memory 1203 in electronic communication with the processor 1201. The memory 1203 may be any electronic component capable of storing electronic information. For example, the memory 1203 may be embodied as random access memory (RAM) , read-only memory (ROM) , magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included with the processor, erasable programmable read-only memory (EPROM) , electrically erasable programmable read-only memory (EEPROM) memory, registers, and so forth, including combinations thereof.

Instructions 1205 and data 1207 may be stored in the memory 1203. The instructions 1205 may be executable by the processor 1201 to implement some or all of the functionality disclosed herein. Executing the instructions 1205 may involve the use of the data 1207 that is stored in the memory 1203. Any of the various examples of modules and components described herein may be implemented, partially or wholly, as instructions 1205 stored in memory 1203 and executed by the processor 1201. Any of the various examples of data described herein may be among the data 1207 that is stored in memory 1203 and used during execution of the instructions 1205 by the processor 1201.

A computer system 1200 may also include one or more communication interfaces 1209 for communicating with other electronic devices. The communication interface (s) 1209 may be based on wired communication technology, wireless communication technology, or both. Some examples of communication interfaces 1209 include a Universal Serial Bus (USB) , an Ethernet adapter, a wireless adapter that operates in accordance with an Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, a wireless communication adapter, and an infrared (IR) communication port.

A computer system 1200 may also include one or more input devices 1211 and one or more output devices 1213. Some examples of input devices 1211 include a keyboard, mouse, microphone, remote control device, button, joystick, trackball, touchpad, and lightpen. Some examples of output devices 1213 include a speaker and a printer. One specific type of output device that is typically included in a computer system 1200 is a display device 1215. Display devices 1215 used with embodiments disclosed herein may utilize any suitable image projection technology, such as liquid crystal display (LCD) , light-emitting diode (LED) , gas plasma, electroluminescence, or the like. A display controller 1217 may also be provided, for converting data 1207 stored in the memory 1203 into text, graphics, and/or moving images (as appropriate) shown on the display device 1215.

The various components of the computer system 1200 may be coupled together by one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For the sake of clarity, the various buses are illustrated in FIG. 12 as a bus system 1219.

The embodiments of the ML validation and verification system have been primarily described with reference to wellbore drilling operations. However, it is to be appreciated that the ML validation and verification systems described herein may be used in applications other than the drilling of a wellbore. In other embodiments, ML validation and verification systems according to the present disclosure may be used outside a wellbore or other downhole environment used for the exploration or production of natural resources. For instance, ML validation and verification systems of the present disclosure may be used in a borehole used for placement of utility lines. Accordingly, the terms “wellbore, ” “borehole” and the like should not be interpreted to limit tools, systems, assemblies, or methods of the present disclosure to any particular industry, field, or environment.

One or more specific embodiments of the present disclosure are described herein. These described embodiments are examples of the presently disclosed techniques. Additionally, in an effort to provide a concise description of these embodiments, not all features of an actual embodiment may be described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous embodiment-specific decisions will be made to achieve the developers’ specific goals, such as compliance with system-related and business-related constraints, which may vary from one embodiment to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

Additionally, it should be understood that references to “one embodiment” or “an embodiment” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. For example, any element described in relation to an embodiment herein may be combinable with any element of any other embodiment described herein. Numbers, percentages, ratios, or other values stated herein are intended to include that value, and also other values that are “about” or “approximately” the stated value, as would be appreciated by one of ordinary skill in the art encompassed by embodiments of the present disclosure. A stated value should therefore be interpreted broadly enough to encompass values that are at least close enough to the stated value to perform a desired function or achieve a desired result. The stated values include at least the variation to be expected in a suitable manufacturing or production process, and may include values that are within 5%, within 1%, within 0.1%, or within 0.01%of a stated value.

A person having ordinary skill in the art should realize in view of the present disclosure that equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations may be made to embodiments disclosed herein without departing from the spirit and scope of the present disclosure. Equivalent constructions, including functional “means-plus-function” clauses are intended to cover the structures described herein as performing the recited function, including both structural equivalents that operate in the same manner, and equivalent structures that provide the same function. It is the express intention of the applicant not to invoke means-plus-function or other functional claiming for any claim except for those in which the words ‘means for’ appear together with an associated function. Each addition, deletion, and modification to the embodiments that falls within the meaning and scope of the claims is to be embraced by the claims.

The terms “approximately, ” “about, ” and “substantially” as used herein represent an amount close to the stated amount that is within standard manufacturing or process tolerances, or which still performs a desired function or achieves a desired result. For example, the terms “approximately, ” “about, ” and “substantially” may refer to an amount that is within less than 5%of, within less than 1%of, within less than 0.1%of, and within less than 0.01%of a stated amount. Further, it should be understood that any directions or reference frames in the preceding description are merely relative directions or movements. For example, any references to “up” and “down” or “above” or “below” are merely descriptive of the relative position or movement of the related elements.

The present disclosure may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are to be considered as illustrative and not restrictive. The scope of the disclosure is, therefore, indicated by the appended claims rather than by the foregoing description. Changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

A method, comprising:

receiving a target configuration of a cutting element, the target configuration related to an original configuration of an original rock file;

applying a machine learning (ML) model to the target configuration to generate a synthetic rock file for the cutting element, the synthetic rock file including cutting forces associated with the target configuration; and

validating the ML model by:

generating a consistency ratio for the synthetic rock file based on validation synthetic rock files generated by the ML model, the validation synthetic rock files generated by different realizations of the ML model using the target configuration; and

performing a physical simulation using the synthetic rock file and the original rock file.
The method of claim 1, wherein performing the physical simulation includes performing a synthetic physical simulation using the synthetic rock file, and performing an original physical simulation using the original rock file, and further comprising generating a simulation comparison ratio using the synthetic physical simulation and the original physical simulation.
The method of claim 2, further comprising, when at least one of the simulation comparison ratio is greater than a simulation comparison ratio threshold or the consistency ratio is greater than a consistency ratio threshold, applying the ML model to a new configuration to generate a new synthetic rock file, the new configuration being different from the target configuration and the original configuration by at least one of orientation, shape, or rock type.
The method of claim 2, further comprising, when at least one of the simulation comparison ratio is less than a simulation comparison ratio threshold or the consistency ratio is less than a consistency ratio threshold, training the ML model based on the synthetic rock file, wherein performing the synthetic physical simulation includes performing the synthetic physical simulation for the synthetic rock file arranged in a bit configuration, and performing the original physical simulation includes performing the original simulation for the original rock file arranged in the bit configuration.
The method of claim 4, wherein training the ML model includes training the ML model until the at least one of the simulation comparison ratio is greater than a validation threshold or the consistency ratio is less than the consistency ratio threshold.
The method of claim 4, wherein performing the synthetic physical simulation includes generating synthetic bit performances for the bit configuration and performing the original physical simulation includes generating original bit performances for the bit configuration, and generating the simulation comparison ratio includes generating a performance ratio for each of the synthetic bit performances based on the original bit performances.
The method of claim 6, wherein performing the synthetic physical simulation includes performing the synthetic physical simulation for a plurality of rates of penetration (ROPs) and generating the synthetic bit performances for each of the plurality of ROPs, wherein performing the original physical simulation includes performing the original physical simulation for the plurality of ROPs and generating the original bit performances using the original bit performances for each of the plurality of ROPs.
A method, comprising:

training a first realization of a machine learning (ML) model for a cutting element using a first training subset of a plurality of original rock files;

applying the first realization of the ML model to a target configuration for the cutting element to generate a first synthetic rock file, the target configuration matching an original configuration of a first validation subset of the plurality of original rock files;

training a second realization of the ML model using a second training subset of the plurality of original rock files;

applying the second realization of the ML model to the target configuration for the cutting element to generate a second synthetic rock file, the second target configuration matching the original configuration of a second validation subset of the plurality of original rock files; and

validating the ML model, including:

tuning a hyperparameter of the ML model based on a first cross validation ratio between the first synthetic rock file and the first validation subset and a second cross validation ratio between the second synthetic rock file and the second validation subset;

generating a consistency ratio for the ML model between the first synthetic rock file and the second synthetic rock file;

performing, using a physical model, a first synthetic physical simulation for the first synthetic rock file and a second synthetic physical simulation for the second synthetic rock file;

performing, using the physical model, a first original physical simulation for the first validation subset and a second original physical simulation for the second validation subset; and

generating a simulation comparison ratio using the first synthetic physical simulation, the second synthetic physical simulation, the first original physical simulation, and the second original physical simulation.
The method of claim 8, further comprising, when the consistency ratio is above a consistency ratio threshold and the simulation comparison ratio is above a simulation comparison ratio threshold, applying the ML model to a new configuration for the cutting element.
The method of claim 8, wherein training the first realization of the ML model and the second realization of the ML model includes training using at least one of: transfer learning and deep learning.