AU2019277193A1

AU2019277193A1 - Method and apparatus for controlling and manipulation of multi-phase flow in microfluidics using artificial intelligence

Info

Publication number: AU2019277193A1
Application number: AU2019277193A
Authority: AU
Inventors: Yongsheng Gao; Nam-Trung Nguyen; Say Hwa TAN; Jun Zhou
Original assignee: Ai Fluidics Pty Ltd
Current assignee: Ai Fluidics Pty Ltd
Priority date: 2018-05-28
Filing date: 2019-03-12
Publication date: 2020-10-29
Also published as: WO2019227126A9; AU2019101835A4; CN112566721A; WO2019227126A1

Abstract

The present invention relates to a system for controlling microfluidic droplets or bubbles. The system includes a microfluidic generator for generating the micro-droplets or bubbles. A feedback sensor is provided for sensing one or more feedback parameters of the generated micro-droplets or bubbles. A controller is provided for controlling the microfluidic generator using the sensed feedback parameters.

Description

METHOD AND APPRATUS FOR CONTROLLING AND MANIPULATION OF MULTIPHASE FLOW IN MICROFLUIDICS USING ARTIFICIAL INTELLIGENCE

TECHNICAL FIELD

[0001] The present invention generally relates to micro-droplets or bubbles having sub-millimeter scale.

[0002] The present invention has particular, although not exclusive application in lap-on-chip platforms, biochemical and chemical analyses, bio-chemical assays, particle and material analyses, pharmaceutical and cosmetic products, particle and cell sorting systems, cell encapsulation or material synthesis.

BACKGROUND

[0003] The reference to any prior art in this specification is not, and should not be taken as an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge.

[0004] Micro-droplets having desired parameters are desirable for many chemical, biochemical, material, cosmetic, pharmaceutical, and biological processes.

[0005] In biochemical applications, droplets with discrete volumes of fluids can be individually handled and manipulated. Each droplet can act as an independent micro reactor in which reactions can be processed at a high throughput rate (e.g. several droplets per millisecond). As another advantage, the high surface area to volume ratio in droplets can be used to enhance reaction rates, or to favor heat or material exchange.

[0006] In pharmaceutical applications, droplets of desired parameters are highly favorable as medical drugs (usually in the form of droplets) whether in the form of designer drugs, personalized medicine or various medical imaging technologies. In particular, a wearable chip device can be incorporated onto a human to provide customized dosages of medicine. In medical imaging, droplets encapsulated with various cells can be imaged, detected, isolated and counted. [0007] In biological applications, droplets can contain cells and serve as bioreactors. Cells can assemble and be cultured into tissues or organoids. Cells such as sperms and eggs can be sorted to initiate artificial reproduction including artificial insemination, in vitro fertilization, cloning and embryonic splitting, or cleavage.

[0008] It is known that droplets can be generated by flowing immiscible fluids into microchannels of a microfluidic generator, which are joined at a channel junction. The fluids flow to the junction, where the droplets of one of the fluids (dispersed phase fluid) are produced in the other fluid (continuous phase fluid). The droplet diameter and frequency of the droplet generation are determined mainly by the geometry of the channels, the flow rates of the fluids and fluid properties like viscosity and surface tension. Active droplet generation methods involve using electric field, thermal energy, pneumatic, acoustic and magnetic fields.

[0009] It is possible to tune the droplet diameter by adapting flow rates or imposed pressure to the channels. Typically, droplet parameters such as droplet sizes are measured after generation, and trial and error approach is then used in controlling droplets. Often, droplet parameters cannot be sensed at all in near real-time.

[00010] The preferred embodiment provides a system for more effectively controlling micro droplets than using trial and error. The preferred embodiment also compensates for parameters which cannot be reliably sensed.

[00011] Droplet-based microfluidics focuses on generation and manipulation of discrete droplets for chemical, biological and material processes in a microfluidic system. Individual droplets can serve as micro-reactors or compartments offering numerous advantages like isolation or separation of different samples for analysis, reduction in reagents consumption and reaction times, and the ability to upscale experiments in a single device. These advantages have led to its use in many areas ranging from music to single cell genomics. Due to the small volume and high speed of droplets generated, a quick method for sorting droplets is needed to obtain the desired number of particles or cells for analysis.

[00012] Droplet sorting is an important process for transporting and distributing droplets into different channels downstream of a generation device. One established droplet sorting method is fluorescence activated droplet sorting (FADS) (J.-C. Baret et al. in“Lab on a Chip”, volume 9, 2009, p. 1850-1858) which can process approximately 2000 fluorescence activated droplets per second with an electrical field. To date, electric actuation is the most preferred choice as being the most robust, highly accurate and quick response time which is necessary for sorting at high speeds.

[00013] A general disadvantage of conventional droplet sorting systems based on electric actuation is that the biological samples have to be labelled with fluorescence dyes. The labelling practice can be time consuming and not very desirable as dyes are not compatible with all cells types and cell behaviour may be affected during live cell imaging. In certain cases, the labelling of cells with fluorescence can only be done with dead cells.

[00014] To observe sample in their natural state, a label-free sorting system is preferred. This is achievable with the fact that most microscopes have a camera attached to them, but would require new design of image acquisition devices and advanced processing algorithms. Early attempt detects Actinobacteria cultivated in picolitre droplets by a photodiode (E. Zang et al. in“Lab on a Chip”, volume 13, 2013, p. 3707-3713). The resulting signal is transformed into a TTL-signal that actuates the camera, following by frame grabbing. The captured frames are processed with the content in the droplets classified to trigger a function generator via an A/D-converter (ADC) for sorting. The drawback of this system is that the true positive sorting rate is only 70% at a sorting frequency of 100Hz. This rate is low when compared to the FADS system due to the adoption of simple rule-based droplet classification approach.

[00015] Another design uses a low speed camera to capture droplet images, and multiple template matching algorithm to detect cells or objects encapsulated in droplets (M. Girault et al.“Scientific reports”, volume 7, 2017, 40072). This system can successfully identify and discriminate droplets containing D. tertiolecta and P.

tricornutum with 91 ±4.5% and 90±3.8% accuracy, respectively. However, the speed of processing is only at a rate of 10Hz. This shows that better imaging and processing systems have to be developed in order to be comparable with FADS.

[00016] From the effective and efficient droplet recognition point of view, although great advances have been achieved in computer vision during the past years, real-time microdroplet analysis is still a challenging problem. Firstly, microdroplets move very fast in a microfluidics system. High-throughput cameras need to capture images at hundreds or thousands of frames per second in order to be able to track such movement. This has imposed high requirement on the efficiency of image processing and object recognition approaches that can be adopted. Secondly, illumination changes

dramatically given the setting of focus of microscope or brightness of lighting source. This requires image processing methods to be adaptive to such situation. Thirdly, the size of the detection targets, such as particles or cells, varies significantly during the movement, which makes recognition a difficult problem.

[00017] The preferred embodiment seeks to minimise the foregoing droplet sorting disadvantages, or at least provides a useful commercial choice.

SUMMARY OF THE INVENTION

[00018] According to one aspect of the present invention, there is provided a system for controlling micro droplets or bubbles including:

a microfluidic generator for generating the micro droplets or bubbles;

a feedback sensor for sensing one or more feedback parameters of the generated micro droplets or bubbles; and

a controller for controlling the microfluidic generator using the sensed feedback parameters.

[00019] Advantageously, the controller uses the sensed feedback parameters in controlling the microfluidic device which is more effective than trial and error.

[00020] The feedback sensor may include a predictor for predicting parameters, which may be either not sensed or unreliable, and further use these predicted parameters for controlling the microfluidic generator.

[00021] The controller may build a model using training data from the feedback sensor, and test the generated droplets or bubbles to determine accuracy of the model.

[00022] The controller may include an artificial intelligence controller incorporating, for example, a machine learning classifier. The controller may include a pressure-driven flow controller or a syringe pump. Preferably, the controller is configured to control the microfluidic device so that the microfluidic device generates a mono-disperse emulsion of the micro droplets or bubbles. [00023] The sensor may include an optical system. The optical system may acquire information such as sizes, shapes, colours of a gas, liquid or solid phase in a fluid flow of the micro droplets or bubbles. The parameters may include any one or more of:

droplet area, diameter, shape, frequency, separation distance and speed.

[00024] According to another aspect of the present invention, there is provided a method for controlling micro droplets or bubbles including:

generating the micro droplets or bubbles;

sensing one or more feedback parameters of the generated micro droplets or bubbles; and

controlling the microfluidic generator using the sensed feedback parameters.

[00025] The step of controlling may include a training phase involving building a model using training data from a feedback sensor. The training phase may involve determining a training region on a pressure characteristic of a generator performing the step of generating. The training region is between parallel flow and stable region transition lines. The training phase may involve querying in a serpentine manner against a training boundary.

[00026] The model may be a regression model. The regression model may involve determining optimal regression parameters by minimizing a loss on the training data.

[00027] The step of controlling may further include a testing phase involving testing the generated droplets or bubbles to determine accuracy of the model. The testing may involve inputting an expected droplet area and shape into the model. The accuracy may involve an error of less than 2.5% when comparing the input expected droplet area and shape with the generated droplets.

[00028] According to another aspect of the present invention, there is provided a system for sorting micro droplets or bubbles, the system including:

a sensor for sensing one or more parameters of the micro droplets or bubbles, or their contents;

a sorter for sorting the micro droplets or bubbles; and

a controller for controlling the sorter using the sensed parameters. [00029] Preferably, the sorter sorts the droplets or bubbles depending upon whether or not they contain particles and without the need for droplet labelling. Disadvantages of conventional label-free droplet sorting techniques are avoided and an enlarged range of applications are enabled. The sorting may depend upon whether the sensed

parameters relate to the droplet containing no particle, a single particle, multiple particles, or particles of different shapes and forms. The particles may be biological cells such as blood cells, cancer cells, sperms or eggs.

[00030] The system may further include a microfluidic generator for generating the micro droplets or bubbles. The system may further include a spacer (or separator) for spacing the generated droplets or bubbles. The generator, spacer and sorter may be integrated together as a single microfluidic device. The microfluidic device may be formed using lithographic techniques.

[00031] The sensor may include an imaging device. The imaging device may include a microscope, a camera for capturing image frames enlarged by the microscope, and a frame grabber for grabbing frames from the camera.

[00032] The controller may include a machine learning model. The controller may include a pressure controller or a syringe pump.

[00033] Sorter may operate according to any one or more of: hydrodynamic, pneumatic, electro-kinetic, dielectrophoretic, magnetic and thermal principles.

[00034] According to another aspect of the present invention, there is provided a method for sorting micro droplets or bubbles, the method including:

sensing one or more parameters of the micro droplets, bubbles or their contents; sorting the micro droplets or bubbles; and

controlling the sorter using the sensed parameters.

[00035] Preferably, the sensing involves using computer vision, the controlling involves using machine learning or the sorting is in real time.

[00036] The sensing may involve grabbing image frames of the droplets. The method may involve determining a region of interest including a droplet in each frame. The method may involve droplet tracking or droplet classification. The droplet tracking may involve using a Hough transform or other boundary detection method to detect the circular boundary of droplets. The droplet classification may involve voting.

[00037] The sorting may involve accepting droplets of bubbles containing single particle, and rejecting other droplets or bubbles in a flow. The sorting may be enabled through electrical actuation. The sorting may involve synchonising droplet generation and recognition modules. The sorting may be based upon droplet categories

determined based on contained contents.

[00038] The controlling may involve machine learning involving offline training and online sorting using the results of the offline training.

[00039] The method may further involve generating the micro-droplets or bubbles. The controlling may involve analysing both droplet generation and the sorting. The method may further involve spacing or separating the generated droplets or bubbles in a flow.

[00040] Any of the features described herein can be combined in any combination with any one or more of the other features described herein within the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[00041] Preferred features, embodiments and variations of the invention may be discerned from the following Detailed Description which provides sufficient information for those skilled in the art to perform the invention. The Detailed Description is not to be regarded as limiting the scope of the preceding Summary of the Invention in any way. The Detailed Description will make reference to a number of drawings as follows:

[00042] Figure 1 a is a schematic diagram of a system for controlling micro droplets in accordance with a first embodiment of the present invention;

[00043] Figure 1 b is a flow diagram showing training and testing phases of the system of Figure 1 a; [00044] Figure 2 is a flow diagram showing the different flow regimes observed in the system of Figure 1 a;

[00045] Figure 3 is a flow chart showing a procedure for capturing training data;

[00046] Figure 4a is a 3D diagram illustrating the droplet areas and shapes at different P1 and P2 obtained from training data with the points A, B, C and D showing training boundaries of the system;

[00047] Figure 4b is a 2D plot of the same data of Figure 4a with droplet area against oil pressure P1 ;

[00048] Figure 4c is a 2D plot of the same data of Figure 4a with droplet area against water pressure P2;

[00049] Figure 5 is a user interface of the system of Figure 1 a showing real time droplet analysis;

[00050] Figure 6 shows a comparison between detected droplet area against user input expected droplet area, where two separate experiments (a & b) were carried out to compare the results with separate training sessions and parameters optimization;

[00051] Figure 7 is a schematic diagram of a system for sorting micro droplets in accordance with a second embodiment of the present invention;

[00052] Figure 8a shows the design detail of a microfluidic device of the system of Figure 7;

[00053] Figure 8b shows dimensions of a microfluidic generator of the microfluidic device of Figure 8a;

[00054] Figure 8c shows dimensions of a spacer of the microfluidic device of Figure 8a;

[00055] Figure 8d shows dimensions of a sorter of the microfluidic device of Figure 8a;

[00056] Figure 9 shows processing of a region of interest (ROI) in a captured video frame; [00057] Figure 10 is a flow diagram for droplet recognition undertaken within the ROI of Figure 9;

[00058] Figure 1 1 is a flow diagram showing machine learning methods of the system of Figure 7 involving offline training and online sorting;

[00059] Figure 12 is a flow regime diagram plotted with water pressure against oil pressure for generating droplets in the system of Figure 7;

[00060] Figure 13 shows a plot of spacer oil pressure against droplet frequency observed at the droplet sorting junction in a sorter of the system of Figure 7;

[00061] Figure 14 shows experimental classification, voting and sorting accuracies in different droplet frequencies; and

[00062] Figure 15 shows experimental classification, voting and sorting accuracies for different voting options.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[00063] According to an embodiment of the present invention, there is provided a system 100 shown in Figure 1 a for controlling micro droplets of a dispersed phase fluid in a continuous phase fluid.

[00064] The system 100 includes a microfluidic generator 102 for generating the micro droplets 104. A feedback sensor 106 senses feedback parameters of the generated micro droplets 104. The feedback sensor 106 also includes a predictor 107 for predicting parameters, which may be either not sensed or unreliable. The system 100 also includes a controller 108 for controlling the microfluidic generator 102 using the sensed and predicted feedback parameters. Advantageously, the controller 108 uses the feedback parameters in automatically controlling the microfluidic generator 102 in real-time which is more effective than trial and error. A detailed description of the system 100 is provided below.

[00065] The microfluidic generator 102 is a microfluidic flow-focusing device fabricated in polydimethylsiloxane (PDMS, Dow Corning) using standard photolithography and soft lithography procedures. The controller 108 is configured to control the microfluidic device 102 so that the microfluidic device 102 generates a mono-disperse emulsion of the microfluidic droplets 104.

[00066] The feedback sensor 106 includes an optical system which can acquire information such as sizes, shapes, colours of a gas, liquid or solid phase in a fluid flow of the microfluidic droplets 104. The sensor 106 also includes an analyser for analysing the acquired information to determine the droplet parameters.

[00067] In order to facilitate real-time imaging and data transfer for real-time measurements, a high performance camera (CP90, Optronis) and a customized computer is provided. The computer is equipped with a Core i7 6850K central processing unit (CPU), 128GB Random-Access Memory (RAM), Direct Memory Access (DMA) controller and frame grabber (VQ8-CXP6D, Silicon Software mE5 Ironman). The camera is equipped with a 4 megapixel monochrome image sensor and at full resolution (2304 x 1720) captures images at a rate of 500 frames per second. The captured image is then delivered from the camera via a coaxpress interface (CX-304-1 -304-05,

Optronis) in a near real-time manner. The coaxpress interface is a high-speed multiple (4 cores) coaxial cable and is directly connected to the frame grabber. The images take an average of about several pico-seconds to transfer to the frame grabber. After the frame grabber receives the images from the camera, they are transferred line by line to the computer RAM using the DMA controller. The DMA controller sends data directly to the main memory and bypasses the computer CPU to speed up the memory storing operation. The pressure controller 108 is integrated with the predictor 107 with a (OB1 MK3, Elveflow) software development kit (SDK).

[00068] The predictor 107 is implemented using QT (Qt, Qt Project) which is a cross platform development tool. Several libraries are used, including armadillo library and OpenCV library, as well as application programming interface (API) for the pressure controller 108, frame grabber and camera. Five threads are used in order to obtain high efficiency of the program. The threads respectively take care of the tasks for frame capture and memory buffering, calculation of droplet parameters, displaying droplet parameters and videos and system parameters on a graphical interface, saving the calculated results, and interaction with the pressure controller 108. When a desired droplet size is input into the system 100 via the graphical interface, the regression model calculates the corresponding pressure values instantly and pass them to the pressure controller 108. The pressure controller 108 sets the designated oil and water pressure values, which will immediately change droplet areas. This influence is captured by the thread for droplet parameter measurement, which is used to determine whether the desired size has been reached. In Video, the convergence only requires several frames and can be reached in 1 -2 seconds approximately, enabling the predictor 107 to perform near real-time tasks.

[00069] Figure 1 b illustrates the framework for the prediction. The prediction involves a training phase 1 10 for building a model using training data from the feedback sensor 106, and a testing phase 1 12 for testing the generated droplets to determine accuracy of the model.

[00070] The system 100 first undergoes the training phase 1 10 to collect the data required for building the predetermination model. During this phase 1 10, real-time measurements of various droplet parameters such as droplet separation distance and area are also collected and verified using manual image analysis (ImageJ, NIH).

[00071] After building the model, the system 100 is then tested to determine the accuracy and reliability of the model. The model is validated by setting the criteria that generated droplets must not deviate from corresponding set values by more than 5%. In this embodiment, focus is only on the droplet area (assuming droplets are spherical) for the generation of the model as the corresponding droplet volume is of greater importance in most applications. However, other droplet parameters (droplet diameter, shape, frequency, separation distance and speed) can also be incorporated into the built model in other embodiments.

[00072] Turning to Figure 2, before initiating the training, the operator manually identifies an appropriate droplet generation regime to reduce the down-time due to flow destabilization that may occur when the regime changes to either parallel flow or backflow. In practice, this step can also be automated with time and cost further reduced. The error bars are obtained from 5 repeated experiments. P1 is the oil pressure and P2 is the water pressure. Pressure values for the stable interface (red line) and parallel flow regime (black line) are given with the droplet generation regime observed in between. Backflow is observed above the red line. For the model generation, the training requires uninterrupted droplet generation. Scale bars in the insets in Figure 2 represent 100um.

[00073] Subsequently, the limits of the training region 200 are demarked using four boundary points, A, B, C and D. These points are selected close to the stable interface and parallel flow transition lines to ensure that the maximum training region 200 was used and no regime changes are encountered during the training phase. The points are also outside the error bars of the two transition lines to ensure that the system always remains in the stable droplet generation regime.

[00074] Figure 3 illustrates the procedure 300 to capture the required training data. In short, the starting pressure values (P1 and P2) are initially set at point A. The system 100 then measures the generated droplet areas to determine the stability of the system 100. Stabilization is achieved when 10 consecutive droplet areas are within 2% of the average droplet area. After stabilization, the system moves to the next set of pressure values in a serpentine motion using intervals of 5 mBar. The training process is completed when the system reaches point D. The stabilization timings at each set of pressure values typically ranges from 1 to 36 seconds depending on the flow conditions.

[00075] In the present embodiment, the longest timing for stabilization is observed when P1 is at 285 mBar and P2 at 155 mBar. At this point, the separation distance between two droplets is about 544 urn. This implies that more frames are required to capture 10 conservative droplets using the current magnification and image resolution.

In total, the training phase collects and processes about 650,000 sets of data using 2,600 sets of different pressure values. At each pressure value, an approximate 250 sets of data are captured. The pressure-driven flow controller 108 used to generate the droplets 104 can include a set of syringe pumps. Flowever, taking into account that the time required for stabilization may be significantly longer (usually in the order of minutes), the syringe pumps may not be used. The separation distance of droplets 104 is also measured and used for model building.

[00076] The method 300 in Figure 3 enables the system 100 to query the different pressure values in a serpentine motion. At each pressure value, the droplet area is captured for use in the subsequent building of the M-SVR model. A close-up view 302 of the initial training pressure values shows the serpentine motion 304 across the training region 200. When the pressure values are at the training boundary, the method 300 correspondingly adjusts the pressure value and changes the query direction.

[00077] In order for the system 100 be able to predetermine the pressure values required for a fixed droplet area, a machine learning approach is used to build a regression model to describe the relationship between the relevant variables. This approach is termed as multivariable support vector regression (M-SVR), which is used in multiple-input multiple-output (MIMO) systems. This machine learning technique enables the prediction of target variables through the generation of a model based on initial training data. Other machine learning techniques may be used to build the regression model.

[00078] Given a training dataset of n samples (ci,3¾), C ,>¾_.), where ¾ = (¾. %), i = is a vector containing the droplet area % and the separation distance ¾ of two neighbouring droplets, ¾ = ¾,¾} contains the oil pressure _t and the water pressure ¾ The goal of model building is to define y =< w_s x > -ife, such that the set of multidimensional regression parameters

can be estimated from the training data. Since the model takes two input variables and generates two output variables, 0=2, W \s a 2x2 matrix, and b is a 2x1 vector. Note this regression model can be extended to more input and output variables with the values of Q increased.

[00079] In order to obtain the optimal regression parameters, the following loss function is minimized:

is a normalization loss function with a Gaussian kernel. The first term in the above equation is a regularization term that controls the complexity of the learned model. C is used to balance the contribution of the loss and the complexity. Based on these equations, a multidimensional hyperplane model is generated. Optimization can be done by solving the quadratic approximation of the loss function in the equation, thereby minimizing the loss on the training data.

[00080] Results

[00081] In the training stage, several variables need to be tuned. These include the ratio between two inputs and M-SVR parameters (for example parameter C and other relevant parameters in the loss function). The tuning of these parameters is based on feedback from the droplet area prediction process. Given a desired droplet area value, the predicted pressured values by the learned regression model are used to generate droplets. The areas of the droplets are then automatically calculated and compared with the desired value. When a set of model parameters produce errors lower than a deviation criterion, they are selected for the final system. During the parameter tuning step, a greedy strategy is followed, i.e., the parameters are tuned individually by fixing all other parameters. Once an optimal value is reached, it is fixed in tuning the next parameter..

[00082] Another factor that influences the performance of the predictor 107 is the selection of kernels. In the system 100, a nonlinear Gaussian kernel is used. This is selected after comparison with both linear kernel and nonlinear kernels including polynomial and Gaussian kernels. Experimental results show that the Gaussian kernel is the best one out of all options. After the M-SVR model has been trained, predictions can be made by applying the model to unseen testing samples and calculating the required oil and water pressures given desired droplet size with the separation distance fixed. Figure 4 shows the 3D diagram 400 illustrating the droplet areas at different P1 and P2 obtained from the training data.

[00083] On average, the training phase takes about 20 minutes including the time for data collection and parameter tuning. Given a set of collected data, model building takes about 10 seconds. In the prediction procedure, once a desired droplet size is set, the system 100 automatically predicts the best pressures which are input to the pressure controller 108. This prediction process is very fast and any timing bottleneck mainly comes from the training and parameter tuning stage. However, this stage only needs to be done once for a device of fixed geometry and a fixed set of fluids. Furthermore, the training time can be reduced by decreasing the number of data points required to be collected in the training region 200 of Figure 2, or relaxing the deviation criteria during the parameter tuning.

[00084] Given the above mentioned system setup, several experiments have been undertaken to evaluate the effectiveness and efficiency of the prediction. The goal of the first experiment is to assess the accuracy of the real-time automatic droplet parameter measurements step. This step provides input to the regression model, including droplet separation distance and area of droplets. System measured value was compared against manual data made with ImageJ on ten different droplets. For each droplet, five repeated manual measurements were made with the mean and standard deviation calculated. This comparison also serves as an illustration where uncertainty or errors in human measurements may arise. The results are summarized in Table below. It can be seen from the table that the automatic droplet parameter measurement has produced close results to those from the human operator. The highest differences on droplet separation and area are 2.65% and 1.67%, respectively. On one hand, this ascertained the high accuracy of the droplet parameter estimation in system 100. On the other hand, this also clearly illustrates that manual human measurements generates different values in each time unless great effort is devoted to ensure the precision. Flere, it is

emphasized that automatic measurement produces objective and consistent results on the same droplet or image. This property is highly desired and will be useful in many droplet-based microfluidics applications.

[00085] Table of Comparison between manual measurement and the system measurement. The manual measurements were conducted using ImageJ and repeated five times.

Droplet Separation Droplet Area

DropletMeasured System Measured System

Error Error

No. Manually Value Manually Value

um um % um² um² % 1 256.88 + 0.38 256.29 -0.23 11086.87 + 42.16 11047 -0.36

2 288.47 + 0.4 289.9 0.5 10270.31 + 39.83 10222.1 -0.47

3 307.83 + 0.25 305.71 -0.69 9579.12 ± 11.88 9527.19 -0.54

4 333.61 + 0.29 338.48 1.46 8926.26 + 41.37 8973.24 0.53

5 83.16 + 0.25 80.95 -2.65 8390.75 + 52.31 8313.9 -0.92

6 111.81 ± 0.25 111.33 -0.43 7450.11 + 33.72 7389.52 -0.81

7 187.42 + 0.53 188.57 0.61 7356.06 + 22.69 7355.19 -0.01

8 125.65 + 0.63 126.19 0.43 10514.68 + 16.77 10499.86 -0.14

9 91.33 + 0.31 90.67 -0.73 11434.1 ± 41.87 11328.95 -0.92

10 110.83 + 0.29 109.62 -1.09 9197.8 + 69.67 9044.29 -1.67

[00086] In a second experiment, the accuracy of the system 100 was evaluated based on the discrepancy between desired droplet size and the actual measured droplet size with the predicted pressure setting. To do so, the system 100 was tested by entering droplet area values ranging from 7.0x103 um2 to 11.0x103 um2 on the user input panel 500 shown in Figure 5. Two rounds of experiments were run at different times, but using the same device 102 and operating steps. In the first round, parameter tuning follows a relaxed version with upper bound of error to be 5%. In the second round, this upper bound is reduced to 2.5%. The experiments were carried out using a single device 102 to minimize the differences due to fabrication.

[00087] For the initial training process, a user manually sets the pressure values of channels 1 and 2 at the user input panel 502 to determine the four points A, B, C and D. The“Start Initialization” button 503 is used to start the training. After the model is generated, the user can subsequently set the expected droplet area. The system 100 determines the most appropriate point and displays the P1 and P2 values beside the textbox. The bottom panel 504 displays the readout from the system analysis.

[00088] The results are shown in Figures 6a and 6b for the first and second rounds, respectively. This result shows that the errors between expected and actual droplet areas can be tuned to a very low level for a specified range of droplet areas. Note that this area range is chosen as it has to be within the measurements obtained from the training data (Fig. 3). The errors obtained in Figure 6 may be caused by several factors. First, there are small systematic pressure fluctuations induced by the piezoelectric regulators inbuilt within the pressure controller 108. These fluctuations translate to small pressure variations which result in the observed difference. Second, the regression model is not 100% accurate since only the relationship between droplet size, separation distance, and pressures was modelled. Other factors such as droplet speed and frequency, device stabilities and dimensions deviation due to long operation time may also affect the relationship between the parameters. These factors can be integrated into the model.

[00089] Conclusion

[00090] The foregoing demonstrates using system 100 with machine learning approach, for real-time automated droplet analysis and size predetermination. Droplet parameters like droplet area and separation distance are evaluated using image-based techniques during training data collection. The system 100 develops a model using the M-SVR machine learning technique for the regression of these parameters with fluidic pressures.

[00091] The model is then used for the prediction of pressure values upon user input of expected droplet areas. Experimental results have shown an error lower than 2.5% for the generated droplets against the user input of the expected droplet area. This system 100 allows accurate droplet generation without repeated manual adjustments. This system also proves to be a cost-effective solution for various biomedical

applications requiring accurate droplet sizing.

[00092] The artificial intelligence (Al) controller 108, including machine learning, provides learning schemes such as supervised, unsupervised, semi-supervised and deep learning which use seen data to predict unseen data from the model. For example, the Al controller 108 can learn the relationship between active controls and then build a model to predict the desired droplet parameters. The use of active droplet generation provides an additional level of control and freedom to manipulate the droplets. For example, these active controls allow droplets to be formed on demand.

[00093] An Artificial Intelligence System for Real-time, Label Free Microdroplet Sorting

[00094] Another embodiment of the invention described in detail below relates to an intelligent droplet system 700 for real-time label free sorting of micro-droplets 104 that contain particles. This embodiment has particular, although not exclusive application to lap-on-chip platforms, biochemical and chemical analyses, bio-chemical assays, particle and material analyses, pharmaceutical and cosmetic products, particle and cell sorting systems, cell encapsulation or material synthesis.

[00095] According to this embodiment, the system 700 for sorting microfluidic droplets 104 includes an optical sensor 702 for sensing parameters of the microfluidic droplets 104. The sensor 702 includes an imaging device 703 incorporating a

microscope, a camera, and a frame grabber. A high speed imaging and data transfer device 703 enables video sequences of microdroplets 104 to be captured at 1000+ frame rate, and then be transferred to the memory of a computer system for further processing.

[00096] A microfluidic device 704 includes a sorter 706 for sorting the microfluidic droplets 104. For example, useful droplets 104 containing a single particle can be sorted from others which are waste. The microfluidic device 704 further includes a microfluidic generator 705 which is controlled by a controller 708 to generate the microfluidic droplets 104. The microfluidic device 704 further includes a spacer (or separator) 709 for spacing the generated droplets 104.

[00097] The system 700 further includes the controller 708 for controlling the sorter 706 through an electrical actuator 71 1 and using the sensed parameters. The sorter 706 sorts the droplets 104 depending upon whether they contain particles and without the need for droplet labelling. Disadvantages of conventional label-free droplet sorting techniques are avoided and an enlarged range of applications are enabled. The controller 708 includes a classifier 710 and a pressure controller 712.

[00098] The microfluidic device 704 is a polydimethylsiloxane (PDMS, Dow Corning) droplet sorting microfluidic device. The device 704 is fabricated using traditional soft and photo-lithography methods. This PDMS layer consists of both fluidic and electrode channels which is plasma bonded on to a glass slide (7101 , Sail brand). Fluidic channels are then further hydrophobized with Aquapel (PPG industries) before experimentation. The device 704 includes the three key modules 705, 709 and 706 defining droplet generation, spacer oil junction and a sorting region. [00099] Generation of droplets 104 by the generator 705 is achieved using flow focusing module. HFE7500 oil (Novec, Sigma Aldrich) with 5% (w/w) ammonium salt of Krytox acting as a surfactant serves as both the continuous phase and spacer oil while deionised water with 5% wt 7-micron polymer beads (78462, Sigma Aldrich) is used for the dispersed phase. All fluids are actuated using a pressure controller (OB1 -MK3, Elveflow) with flow pressure ranging from 0 to 1000 mbar. In the middle portion of the device, a space oil junction provided by the spacer (or separator) 709 spaces the droplets 104 apart. Pressure flow is adjusted accordingly to prevent droplets 104 generated from flowing into the spacer oil channel.

[000100] Droplet generation and droplet separation are controlled by the pressure controller 712 via adjusting fluidic pressure across the inlets. Upon attaining a stable generation of droplets 104, images of droplets 104 are captured by the high-speed camera 703. The system is then initiated to detect, track, and count droplets 104 and classify them into containing no, single, or multiple particles using the classifier 710. Single particle droplets are subsequently sorted through electrical actuation of the actuator 71 1. The red inset box in Figure 7 shows the magnified view of the sorting region where single particle droplets are pulled into a sorting channel while the other droplets go to a waste channel.

[000101] Sorting of droplets 104 using the actuator 71 1 is performed using a DC electrical field. Electrode channels are filled with indium alloy (Indium Corporation,

USA) for the generation of DC electric field. Generation of an electric field is done using a DAQ card (National Instruments Corporation) which sends a square wave at 5V at a frequency of 25000Flz to the sorting electrodes after being amplified by a factor of 1 ,000 by a high-voltage amplifier (623B Trek). The whole sorting process takes roughly <1 ms.

[000102] Image processing

[000103] Figure 8 shows the dimensions of the fluidic channels readers of microfluidic device 704. The overall height of the device channel is approximately 52um.

[000104] The device 704 observed by the sensor 702 under an inverted microscope (Nikon Ti-E, Japan) equipped with a high-speed camera (CP90-4-M-500, Optronis) and a customised computer (Intel i7 6850K CPU, 128GB RAM). Observations of droplets 104 are performed using a 10X magnification objective lens (Plan Flour Nikon). The images are captured at 1000 frames per second at a resolution of 2304^*452 pixels. The captured images are sent from the camera via a coaxpress connection (CX-304-1 -SOS OS, Optronis) to the frame grabber (VQ8-CXP6D, Silicon Software ME5 Ironman) which is connected to a computer. After the frame grabber captures the images from the camera, it transfers images line by line to the computer RAM by a Direct Memory Access (DMA) controller. This process takes approximately 1 ms for each image to ensure enough time for image processing. Once the image processing and droplet classification are completed by the controller 708, the controller computer sends a signal to the DAQ which controls the droplet sorting behaviour.

[000105] The system 700 contains key artificial intelligence functions to enable automated real-time droplet image processing by the classifier 710. The software was developed in C++ using a computer vision library OpenCV. The software contains a framework that employs computer vision, machine learning, and multithreading approaches to process captured video sequence. This framework includes three key components, namely: droplet recognition, droplet sorting, and synchronisation.

[000106] Turning to Figure 9, for each captured video frame, the system 700 first pre- processes the binarised image 900 by extracting a region of interest (ROI) 902, which defines a rectangular window that contains only droplets within a channel boundary and with the noisy background removed. To determine the width of the ROI window 902, the inflection point is located within the channel. To estimate the locations for the top and bottom boundaries of the ROI 902, the image is projected in the horizontal direction. Then the local maximum points of the projection histogram are extracted as the locations of channel boundaries. The parameters 904 of droplets 104 are also displayed.

[000107] Turning to Figure 10, droplet recognition 1000 is undertaken within the ROI 902 and involves two tasks: namely droplet tracking 1002 and droplet classification 1004.

[000108] Since the droplets 104 are in round shape, the system 700 uses the Flough transform to detect the circular boundary of droplets during droplet tracking 1002. The Flough transform is a feature extraction method in computer vision. The Flough transform finds a certain class of shapes by a voting procedure, which is done in the parameter space of the shape. The principle is to fit the following equation to the edge map detected from the image:

(x - xo)² + (y - yo)² = A² where x and y are the coordinates of points on the target circle, (xo, yo) is the circle center, and ris the circle radius.

[000109] The Hough transform tries to find all the possible (xo, yo) by searching the local maxima in the accumulator space. As the radius of microdroplets 104 can be easily determined beforehand, e.g., by controlling the size of microdroplets 104 in the microfluidics system 700, in practice, the system 700 directly designates the radius, which reduces computation cost. Other boundary detection method may also be used to detect microdroplets.

[000110] Each droplet 104 with complete boundary is assigned a unique TrackID (tracking identifier) when it enters the ROI 902. Since the images are captured in high frame rate, the speed of the droplets 104 is controlled with a threshold, so the distance that droplets 104 travel in adjacent frames is smaller than the distance between two neighboring droplets 104. This allows the calculation of the positional relationship between droplets 104 in two adjacent frames in order to track their movements and travel speed. These in turn is used to estimate when a droplet 104 would move to a sorting location.

[000111] With the droplets 104 detected, the next step is to determine whether a droplet contains no particle, a single particle, or multiple particles so they can be properly sorted. To achieve this goal, the droplet regions 902 are cropped from the image 900. The cropped droplet regions 902 are normalised into square image patches of the same size (50x50 pixels). Then each patch is converted into a vector of features. The vectors are used as the input to train a classifier. For feature extraction, the circular boundary of each droplet 104 is first removed because the boundary is common to all droplets and does not contain discriminative information. Then all remaining connected black pixels in the patch are retrieved. These connected pixels correspond to presence of objects such as particles, noises, and incompletely removed boundary. The system 700 measures the area, height, and width of each object. The objects are sorted based on their areas. The measurements of top ten objects are concatenated to form a 30- dimensional feature vector. When there are less than ten objects in a droplet 104, the rest of the entries in the feature vector are set to zero. Such feature vectors can be calculated from a number of droplets 104, forming the training data for learning of a linear Support Vector Machine (SVM) classifier. The trained model is then used to predict the category of new droplets. Note that other types of classifiers can also be trained to perform the prediction task.

[000112] Droplet classification 1004 is a challenging task, mainly due to the fact that particles may touch the boundary of droplets 104 or touch each other when there are multiple particles in a droplet 104. Distinguishing them is sometimes even not an easy task for human vision. The location of particles might change during the movement of droplets 104, and a droplet 104 can appear in multiple frames. Thus, the system 700 adopts a classifier voting scheme based on per frame recognition result. The class of a distinctive droplet 104 is predicted in each frame, then the final decision is made by the majority voting across frames. This voting increases the accuracy of droplet

classification.

[000113] Once a droplet 104 with single particle is determined, the system 700 uses the dielectrophoretic force to pull the droplet 104 into the upstream channel when it reaches the location of sorting at the sorter 706. A key of success for this sorting is system synchronisation. The image acquisition module 702 captures images at a speed of 1000 frames per second. That is, it takes 1 ms to generate a frame. Before the droplet 104 reached the location of sorting at the sorter 706, several frames could be captured for the same droplet 104. The actual number of frames is related to the speed of the droplet 104. Such conditions requires that each frame be processed within 1 ms.

Otherwise, the processing timestamp will have lagged behind the timestamp of the captured frame. As the time goes by, the latency is accumulated, leading to missed frames and eventually the failure of the sorting system 706.

[000114] To solve this problem, this embodiment adopts a framework to speed up frame processing using multithreading technique. The framework processing includes four modules: capture, processing, trigger and control. The capture module is the image acquisition module. The processing module is the machine learning module. The trigger and control modules take care of the droplet sorting. These threads are run in parallel and independently on a multi-core CPU to increase the processing speed. [000115] The system 700 also includes a time buffer between the droplet recognition and sorting, such that the droplet 104 can continue to move with its class label ready before it has to be sorted. Since the speed of the droplet 104 has been calculated, the system can predict when it would reach the sorting location 706. This allows the system 700 to create a queue of droplets 104. Each droplet 104 in the queue has three attributes: the TrackID, the class label, and the delay time. The TrackID identifies different droplets. The class label indicates how the droplets should be sorted. The delay time controls when the voltage force should be triggered. A droplet is removed from the queue once its sorting has been completed.

[000116] Once the system 700 has detected the circular shape of a microdroplet 104, the detected circles are sorted in sequence according to their x-coordinates in the image. As the oldest and newest generated circles are located from the right most part of the frame to the left-most part of frame, the age of a microdroplet 104 can be represented using the x-coordinate of the microdroplet 104. After sorting, the

microdroplets 104 are arranged in their ages. With the sorted x-coordinates

microdroplets at hand, the system 700 can find the relationship of microdroplets 104 between the current frame and the previous frame. Here we make the hypothesis that the speed of droplets 104 is close to constant, i.e., a new microdroplet 104 cannot overpass an old microdroplet 104 in the movement. Therefore, the system 700 can use this hypothesis to determine the TrackID of microdroplet.

[000117] In the control thread, the system 700 uses a DAQ to start the control process. The DAQ accepts a duration time. Because of the time required for activation of electrodes, the actual execution time is about 1 ms longer than the duration time. The longest time allowed for a trigger depends on the distance and speed of two

neighboring droplets 104. It shall ends before the next droplet 104 reaches the sorting location 706. Otherwise, both droplets 104 are influenced by the same trigger action. This implies that there is an upper bound for length of trigger. At the same time, there is a lower bound of the length of the trigger which shall be long enough to pull the current droplet 104 into the downstream channel given the speed of the droplet 104.

The following relationship can be established: are two neighbouring droplets, v_E÷1 is the speed of _Pe÷1, is the minimal pulling time, which can be determined empirically. The limit on the sorting speed of the system is bounded by the sum of T_miJ3 and the time required for image acquisition.

[000118] Figure 1 1 shows that the implementation of the machine learning methods in the system 700 includes two main stages: offline training 1 100 and online sorting 1102. The offline stage 1 100 is where training of the developed classification model is carried out using labelled training samples. The training samples are droplets 104 detected from video sequences of the microfluidics device 704, which contain different number of particles. In general, more labelled training samples are recommended for better classification performance.

[000119] Among these labelled samples, 60% are used for the initial training, and the rest 40% are used to validate the trained classifier model. The parameters of the classifier 710 are tuned till the validation is greater than 99% accuracy in which case the system 700 can be reliably used for the online stage 1102. A number of machine learning methods can be used to develop the classifier, e.g., neural networks, decision tree, Bayesian classifiers, in addition to, or conjuction with the Support Vector Machine.

[000120] During the online sorting stage 1102, the trained model is used to classify the droplets 104 in real time. For accepted droplets 104 containing single particle, an electric field is turned on, pulling the droplet into the required sorting channel. For droplets containing zero or multiple particles, electric field remains off, letting the rejected droplet 104 go into the waste channel.

[000121] Results

[000122] As time is needed for the system 700 to process individual droplet 104 and activate the sorting electrical field, droplets 104 generated must be spaced out. To determine the optimal separation distance, parameters such as droplet size, sorting frequency and distance between individual droplets 104 are taken into account. This process is analysed in two stages: namely, droplets generation and droplets sorting.

[000123] Figure 12 shows a flow regime diagram 1200 obtained to determine the range for droplet generation. The lower blue line 1202 gives the pressure values at stable interface (blue bordered inset 1204) before transitioning into droplet generation (green bordered inset 1206). The red line 1208 depicts the pressure values for onset of parallel flow (red bordered inset 1210). Scale bars in Figure 12 represent dimensions of 100 um. The black squares points are the pressure combinations used for subsequent study of spacer oil influence.

[000124] A replot of these pressure values against pressure ratio PWater/POil is given in the inset 1212 with a representative image of the generated droplet sizes also given. Droplet areas remain constant across same pressure ratio. Scale bar represents dimensions of 50um.

[000125] Experiment on oil pressure was conducted with increased intervals of 10OmBar from 400mBar to 900mBar. Pressure below 400mBar were not used as no droplet 104 was generated when spacer oil is introduced while the limit of the pressure controller is at 900mBar. Stable interface and parallel flow values were gathered against the corresponding water pressure. These values were plotted to obtain a droplet generation region 1214 shown in green in Figure 12. The biggest and smallest droplet sizes were also measured, which are shown in the inset 1212 of Figure 12. This ensures no droplet splitting will occur at the spacer oil junction as no slug will form at the junction. Once the droplet generation region 1214 was determined, readings were taken at ten different points across this region 1214 (dots in black in Fig. 12) to determine the frequency and distance of the droplet.

[000126] Droplet sorting frequency is determined by a combination of the system response time and the rate of droplets 104 entering the sorting region 706. Time taken for system processing and activation of electrodes were approximately 1 ms (measured using the computer clock) and 1 ±0.5ms respectively. An additional 3ms is added for leaving the electrical field on to pull up a single particle droplet 104 into the right channel. Using the multithreading technique adopted in the system 700, system processing can be run in parallel as sorting. Therefore, the theoretical maximum sorting frequency is purely dependent on the sum of the time for activation of electrodes and pulling, which is around 286 Hz. The rate of droplets entering the electric field region during sorting is influenced by the spacer oil injection prior to the sorting stage. Varied spacer oil pressures ranging from 100mBar to 1000mBar were applied to the ten points that were chosen. For certain points, the spacer oil pressure was stopped before it reached l OOOmBar as the water pressure was unable to overcome the Laplace pressure threshold at the liquid-oil interface for droplet generation. [000127] Figure 13 shows a plot 1300 of spacer oil pressure against droplet frequency observed at the droplet sorting junction in the sorter 706. A region of unstable droplet motion 1302 is shown in pink, where the droplet 104 enters the spacer oil channel or the sorting channel. The points within the blue region 1304 give stable droplet motion. The points in this region 1304 are plotted against droplet separation to form the plot 1306 in the inset showing that each pressure combination can be used for the subsequent droplet sorting experiment when above the minimum separation of 1 mm.

[000128] Based on observations, a region of instability 1302 was detected as shown in pink in Figure 13. In this region 1302, droplets were observed either entering the spacer oil channel or sorting channels due to low spacer oil pressure or droplets being too close. This results in the generation of false positives and inconsistent droplet separation distances, making it unusable for characterization of droplet sorting. In the blue region 1304, droplets generated are monodispersed with regular droplet separation. These droplets are thus suitable for use in droplet sorting by the sorter 706. Since the system limitation is set at 286Flz, the bright blue region 1308 showed the setting for the pressure controller 712. The inset 1306 of Figure 13 highlights different droplet separations for data points in the bright blue region 1308. Droplet separation is equally critical as the droplets 104 need to be of a minimal distance apart. This is to prevent the sorting mechanism 706 from affecting subsequent droplets 104 upon actuation, resulting in false positives. This was capped within the inset 1306 of Figure 13. As a result of these limitations, the bright blue region 1308 in the inset 1306 is highlighted to show the list of applicable points for use in the sorting experiment below.

[000129] In the first experiment, the classification, voting, and sorting accuracies were calculated when the droplet frequency changes with the classification. The results of 3 frames were used for voting. The droplet frequency is the number of droplets generated per second. In the experiments, each time, 10000 frames were run at 1000 FPS, and the central 1000 frames used for testing since they are the most stable frames. Then the means and standard deviations of each accuracy were calculated. The results are shown in Figure 14.

[000130] The system 700 can process the droplet 104 at a maximum frequency of 270Flz. In some cases, the classification accuracy is not high. This is mainly because the particles within the droplets 104 touched the boundary of droplets 104, making it difficult to distinguish zero/single or single/multiple particles. This problem is well solved by the voting strategy which improves the classification results. The final sorting accuracy is related to the voting outcome but witnessed dropping in performance.

Overall, the voting accuracy is higher than the classification accuracy, and the sorting accuracy sits between the classification and voting results. At the 260Hz frequency, the system achieved the best results of 98.85% accuracy.

[000131] A second experiment was conducted to analyse the influence of voting strategy. Given a fixed droplet frequency at 220Hz, the system 700 was tested by voting from 3, 5, and 7 frames respectively. The results are shown in Figure 15. No matter which voting option was adopted, the voting accuracy was higher than the classification accuracy. This again proves the effectiveness of classifier voting. In the experiments, voting using 7 frames led to the best sorting result of 94.53% accuracy. Given this set of results, the system in this invention has proven to be robust, allowing high efficiency sorting for use in single particle applications.

[000132] Throughout the experiments, droplet rupturing was not observed during the electric field actuation, and this showed that the electrical energy applied does not affect the droplet 104. Further improvements to provide optimal response speeds for high frequency droplet sorting, or high throughput droplet sorting, can be readily made by increasing computer processing speeds or through integration of field-programmable gate arrays (FPGA) in the system 700, or neural networks for sorting of multiplex droplet libraries. The DC voltage can also be replaced by an AC voltage. An FPGA integrated circuit can also be used to provide the electrical energy or actuation.

[000133] Conclusion

[000134] The system 700 provides real-time droplet sorting based on machine learning. Further, the system 700 provides a label-free, image-based approach that makes use of computer vision and machine learning for real time analysis and sorting. Droplets 104 are sorted automatically without the need for labelling or the use of additives that may interfere with systems and experiments downstream. A droplet generation regime has been determined with experiments showing the influence of various fluidic pressures on droplet sorting frequency. Following that, offline training and validation of the classifier model has been performed before online experiments. A high accuracy of 98.85% is achieved in sorting droplets at 260Flz from a continuous stream of 1000 frames. The ability to sort these droplets efficiently is of high value for future biomedical applications, particularly single cell analysis.

[000135] A person skilled in the art will appreciate that many embodiments and variations can be made without departing from the ambit of the present invention.

[000136] The preferred embodiment related to microfluidic droplets 104 but is also applicable to micro bubbles.

[000137] The sorter 706 of the preferred embodiment used electric actuation. In other embodiments, the sorter may operate according to any one or more of: hydrodynamic, pneumatic, electro-kinetic, dielectrophoretic, magnetic and thermal principles.

[000138] In compliance with the statute, the invention has been described in language more or less specific to structural or methodical features. It is to be understood that the invention is not limited to specific features shown or described since the means herein described comprises preferred forms of putting the invention into effect.

[000139] Reference throughout this specification to‘one embodiment’ or‘an embodiment’ means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearance of the phrases‘in one embodiment’ or‘in an embodiment’ in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more combinations.

Claims

The claims defining the invention are as follows:

1. A system for controlling micro droplets or bubbles including:

a microfluidic generator for generating the micro droplets or bubbles;

2. A system as claimed in claim 1 , wherein the feedback sensor includes a predictor for predicting parameters, which may be either not sensed or unreliable, and further use these predicted parameters for controlling the microfluidic generator.

3. A system as claimed in claim 1 , wherein the controller builds a model using training data from the feedback sensor, and tests the generated droplets or bubbles to determine accuracy of the model.

4. A system as claimed in claim 1 , wherein the controller includes an artificial intelligence controller incorporating, for example, a machine learning classifier.

5. A system as claimed in claim 1 , wherein the controller includes a pressure-driven flow controller or a syringe pump.

6. A system as claimed in claim 1 , wherein the controller is configured to control the microfluidic generator so that the microfluidic generator generates a mono-disperse emulsion of the micro droplets or bubbles.

7. A system as claimed in claim 1 , wherein the sensor includes an optical system.

8. A system as claimed in claim 7, wherein the optical system acquires information such as sizes, shapes, colours of a gas, liquid or solid phase in a fluid flow of the micro droplets or bubbles.

9. A system as claimed in claim 7, wherein the optical system determines parameters including any one or more of: droplet area, diameter, shape, frequency, separation distance and speed.

10. A method for controlling micro droplets or bubbles including:

generating the micro droplets or bubbles;

controlling the microfluidic generator using the sensed feedback parameters.

1 1. A method as claimed in claim 10, wherein the step of controlling includes a training phase involving building a model using training data from a feedback sensor.

12. A method as claimed in claim 1 1 , wherein the training phase involves

determining a training region on a pressure characteristic of a generator performing the step of generating.

13. A method as claimed in claim 12, wherein the training region is between parallel flow and stable region transition lines.

14. A method as claimed in claim 1 1 , wherein the training phase involve querying in a serpentine manner against a training boundary.

15. A method as claimed in claim 1 1 , wherein the model is a regression model.

16. A method as claimed in claim 15, wherein the regression model involves determining optimal regression parameters by minimizing a loss on the training data.

17. A method as claimed in claim 1 1 , wherein the step of controlling further includes a testing phase involving testing the generated droplets or bubbles to determine accuracy of the model.

18. A method as claimed in claim 17, wherein the testing involves inputting an expected droplet area and shape into the model.

19. A method as claimed in claim 17, wherein the accuracy involves an error of less than 2.5% when comparing the input expected droplet area and shape with the generated droplets.

20. A system for sorting micro droplets or bubbles, the system including:

a sorter for sorting the micro droplets or bubbles; and

a controller for controlling the sorter using the sensed parameters.

21. A system as claimed in claim 20, wherein the sorter sorts the droplets or bubbles depending upon whether or not they contain particles and without the need for droplet labelling.

22. A system as claimed in claim 21 , wherein the particles include biological cells such as blood cells, cancer cells, sperms or eggs.

23. A system as claimed in claim 20, wherein the sorting depends upon whether the sensed parameters relate to the droplet containing no particle, a single particle, multiple particles, or particles of different shapes and forms.

23. A system as claimed in claim 20, further including a microfluidic generator for generating the micro droplets or bubbles.

24. A system as claimed in claim 20, further including a spacer, or separator, for spacing the generated droplets or bubbles.

25. A system as claimed in claim 24, wherein the generator, spacer and sorter are integrated together as a single microfluidic device.

26. A system as claimed in claim 25, wherein the microfluidic device is formed using lithographic techniques.

27. A system as claimed in claim 20, wherein the sensor includes an imaging device.

28. A system as claimed in claim 27, wherein the imaging device includes a microscope, a camera for capturing image frames enlarged by the microscope, and/or a frame grabber for grabbing frames from the camera.

29. A system as claimed in claim 20, wherein the controller includes a machine learning model.

30. A system as claimed in claim 20, wherein the controller includes a pressure controller or a syringe pump.

31. A system as claimed in claim 20, wherein the sorter operates according to any one or more of: hydrodynamic, pneumatic, electro-kinetic, dielectrophoretic, magnetic and thermal principles.

32. A method for sorting micro droplets or bubbles, the method including:

controlling the sorter using the sensed parameters.

33. A method as claimed in claim 32, wherein the sensing involves using computer vision, the controlling involves using machine learning and/or the sorting is in real time.

34. A method as claimed in claim 32, wherein the sensing involves grabbing image frames of the droplets.

35. A method as claimed in claim 34, involving determining a region of interest including a droplet in each frame.

36. A method as claimed in claim 32, involving droplet tracking or droplet

classification.

37. A method as claimed in claim 36, wherein the droplet tracking involves using a Hough transform or other boundary detection method to detect the circular boundary of droplets.

38. A method as claimed in claim 36, wherein the droplet classification involves voting.

39. A method as claimed in claim 32, wherein the sorting involves accepting droplets or bubbles containing single particle, and rejecting other droplets or bubbles in a flow.

40. A method as claimed in claim 32, wherein the sorting is enabled through electrical actuation, preferably involving synchonising droplet generation and recognition modules and based upon droplet categories determined based on contained contents.

41. A method as claimed in claim 32, wherein the controlling involves machine learning involving offline training and online sorting using the results of the offline training.

42. A method as claimed in claim 32, further involving generating the micro-droplets or bubbles, and further involving spacing or separating the generated droplets or bubbles in a flow.