KR102587495B1 - Apparatus and method for providing microorganism information - Google Patents

Abstract

One embodiment of the present invention includes a receiver that receives a plurality of images taken in time series of the waves emitted from the sample, and a device that extracts features of changes over time from the plurality of images taken in time series. It includes a detection unit, a learning unit that machine learns classification criteria based on the extracted features, and a determination unit that classifies the type or concentration of microorganisms contained in the sample based on the classification criteria, and each of the plurality of images is transmitted to the sample. A microorganism information providing device including speckle information generated by multiple scattering by the microorganism due to an incident wave is provided.

Description

Apparatus and method for providing microbial information {APPARATUS AND METHOD FOR PROVIDING MICROORGANISM INFORMATION}

Embodiments of the present invention relate to an apparatus and method for providing microbial information.

Various pathogenic microorganisms such as bacteria, fungi, and viruses appear and begin to colonize in the blood, body fluids, and tissues of the human body, causing infectious diseases. Recently, nosocomial infections have been gaining social prominence. The frequency is gradually increasing, and nosocomial infections have begun to become a legal issue in the medical community. Since nosocomial treatment can result in loss of life, early diagnosis and prompt treatment without complications are essential. It is important. Therefore, the development of accurate and rapid diagnostic methods for these pathogenic microorganisms is a medical technology required by the times.

Moreover, in recent years, the bacterial culture rate has decreased due to the misuse of antibiotics, and the increase in the use of immunosuppressants due to transplantation, the increase in drug administration due to anticancer treatment, and the increase in AIDS have led to a diversification of causative strains, so culture tests are required. Diagnostic methods for diagnosing existing infectious diseases such as are increasingly facing difficulties.

In order to solve the above problems and/or limitations, the purpose is to provide a device and method that provides microbial information by classifying the type and concentration of microorganisms in a sample using machine learning.

One embodiment of the present invention includes a receiver that receives a plurality of images taken in time series of the waves emitted from the sample, and a device that extracts features of changes over time from the plurality of images taken in time series. It includes a detection unit, a learning unit that machine learns classification criteria based on the extracted features, and a determination unit that classifies the type or concentration of microorganisms contained in the sample based on the classification criteria, and each of the plurality of images is transmitted to the sample. A microorganism information providing device including speckle information generated by multiple scattering by the microorganism due to an incident wave is provided.

In one embodiment of the present invention, the learning unit may learn the classification criteria using a convolution neural network (CNN).

In one embodiment of the present invention, the learning unit may perform a convolution operation using a convolution kernel with a size smaller than the size of one speckle.

In one embodiment of the present invention, when the size of one speckle corresponds to m pixels, the learning unit performs a convolution operation using a convolution kernel of n x n size smaller than m. It can be done.

In one embodiment of the present invention, the stride of the convolution operation may correspond to the n value.

In one embodiment of the present invention, the convolution kernel may include kernel feature maps that correspond to the number of images or are larger than the number of images.

In one embodiment of the present invention, the learning unit may perform a convolution operation using a kernel set including a plurality of convolution kernels corresponding to the number of output channels.

In one embodiment of the present invention, the number of output channels by the convolution operation may correspond to the number of the plurality of images.

In one embodiment of the present invention, the learning unit may learn the classification criteria based on temporal correlation of the plurality of images.

In one embodiment of the present invention, the classification criteria include a change in the shape of the speckle pattern among the features, a temporal correlation coefficient calculated based on the light intensity of the speckle pattern, and the light intensity of the speckle pattern. It can be learned using one of the changes in the standard deviation value of (intensity).

In one embodiment of the present invention, the standard deviation value and the concentration of the microorganism may have a linear relationship.

One embodiment of the present invention includes the steps of irradiating waves to a sample whose type or concentration of microorganisms is known in advance and receiving a plurality of learning images in which the emitted waves are previously photographed in time series, A step of machine learning classification criteria based on the features of changes over time from a plurality of learning images, a step of irradiating waves to a new sample and receiving a plurality of images taken in time series of the emitted waves, and A step of classifying the type or concentration of microorganisms contained in the new sample based on the plurality of images and the classification criteria, wherein each of the plurality of learning images or each of the plurality of images is applied to the wave incident on the sample. As a result, a method of providing microorganism information is provided, including speckle information generated by multiple scattering by the microorganism.

In one embodiment of the present invention, in the step of machine learning the classification criteria, the classification criteria may be learned using a convolution neural network (CNN).

In one embodiment of the present invention, the step of machine learning the classification criteria may be performed by performing a convolution operation using a convolution kernel with a size smaller than the size of one speckle.

In one embodiment of the present invention, the step of machine learning the classification criteria is: when the size of one speckle corresponds to m pixels, a convolution kernel of n x n size smaller than m is used. You can perform a convolution operation using .

In one embodiment of the present invention, the stride of the convolution operation may correspond to the n value.

In one embodiment of the present invention, the convolution kernel may include kernel feature maps that correspond to the number of images or are larger than the number of images.

In one embodiment of the present invention, the step of machine learning the classification criteria may be performed by performing a convolution operation using a kernel set including a plurality of convolution kernels corresponding to the number of output channels.

In one embodiment of the present invention, the number of output channels by the convolution operation may correspond to the number of the plurality of images.

In one embodiment of the present invention, the learning unit may learn the classification criteria based on temporal correlation of the plurality of images.

In one embodiment of the present invention, the classification criteria include a change in the shape of the speckle pattern among the features, a temporal correlation coefficient calculated based on the light intensity of the speckle pattern, and the light intensity of the speckle pattern. It can be learned using one of the changes in the standard deviation value of (intensity).

In one embodiment of the present invention, the standard deviation value and the concentration of the microorganism may have a linear relationship.

One embodiment of the present invention includes a sample unit that accommodates a sample, a wave source that irradiates waves toward the sample, an image sensor that acquires a plurality of images by photographing waves emitted from the sample in time series, and the time series. A microorganism information providing device that provides microorganism information including the type or concentration of microorganisms using a plurality of images taken in sequential order, the microorganism information providing device comprising: a receiving unit that receives the plurality of images; the time series order; A detection unit that extracts features of changes over time from a plurality of images taken with a camera, a learning unit that machine learns classification criteria based on the extracted features, and the types of microorganisms included in the sample based on the classification criteria. or a determination unit that distinguishes concentrations, wherein each of the plurality of images includes speckle information generated by multiple scattering by the microorganisms due to a wave incident on the sample, microbial information. Provides a delivery system.

Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention.

The device and method for providing microorganism information according to embodiments of the present invention extract the characteristics of changes in speckle over time and learn them to obtain classification criteria for classifying the type or concentration of microorganisms, thereby providing a separate chemical method. It is possible to quickly and accurately distinguish the type or concentration of microorganisms in a sample without relying on the method.

Figure 1 is a diagram schematically showing a microorganism information provision system according to an embodiment of the present invention.
Figure 2 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
Figure 3 is a block diagram of a device for providing microorganism information according to an embodiment of the present invention.
Figure 4 is a diagram for explaining the basic principle by which a microorganism information providing device according to an embodiment of the present invention confirms the presence of microorganisms in a sample.
Figures 5a and 5b show in time series a method for providing microbial information according to an embodiment of the present invention.
Figure 6 is a diagram for explaining a method of analyzing the temporal correlation of speckles in a learning unit according to an embodiment of the present invention.
Figure 7 is a diagram showing the standard deviation distribution of the light intensity of the speckle measured over time.
Figure 8 is an exemplary diagram of a convolutional neural network according to an embodiment of the present invention.
FIGS. 9 and 10 are diagrams for explaining the convolution operation of FIG. 8.
Figure 11 is a graph comparing expected microbial information and actual microbial information obtained using a method for providing microbial information according to an embodiment of the present invention.

Hereinafter, the following embodiments will be described in detail with reference to the accompanying drawings. When describing with reference to the drawings, identical or corresponding components will be assigned the same reference numerals and duplicate descriptions thereof will be omitted.

Since various transformations can be made to these embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. The effects and features of the present embodiments and methods for achieving them will become clear by referring to the detailed description below along with the drawings. However, the present embodiments are not limited to the embodiments disclosed below and may be implemented in various forms.

In the following embodiments, terms such as first and second are used not in a limiting sense but for the purpose of distinguishing one component from another component.

In the following examples, singular expressions include plural expressions, unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean the presence of features or components described in the specification, and do not exclude in advance the possibility of adding one or more other features or components.

In the following embodiments, when a part of a unit, area, component, etc. is said to be on or on another part, it is not only the case where it is directly on top of the other part, but also when other units, areas, components, etc. are interposed between them. Also includes cases.

In the following embodiments, terms such as connect or combine do not necessarily mean a direct and/or fixed connection or combination of two members, unless the context clearly indicates otherwise, and do not mean that another member is interposed between the two members. It's not exclusion.

This means that the features or components described in the specification exist, and does not preclude the possibility of adding one or more other features or components.

In the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, so the following embodiments are not necessarily limited to what is shown.

Figure 1 is a diagram schematically showing a microorganism information providing system 1 according to an embodiment of the present invention.

Referring to Figure 1, the microbial information providing system 1 according to an embodiment of the present invention includes a wave source 100, a sample unit 200, an image sensor 300, and a microbial information providing device 400. .

The wave source 100 may irradiate the wave L1 toward the sample 201. The wave source 100 can be any type of source device that can generate waves, for example, it can be a laser that can irradiate light in a specific wavelength band. The present invention is not limited to the type of wave source, but for convenience of explanation, the description below will focus on the case of a laser.

For example, in order to form speckles in a sample, a laser with good coherence can be used as the wave source 100. At this time, the shorter the spectral bandwidth of the wave source, which determines the coherence of the laser wave source, the shorter the measurement accuracy can be. That is, the longer the coherence length, the greater the measurement accuracy. Accordingly, laser light whose spectral bandwidth of the wave source is less than a predefined reference bandwidth can be used as the wave source 100, and as it is shorter than the reference bandwidth, measurement accuracy can increase. For example, the spectral bandwidth of the wave source can be set so that the condition of Equation 1 below is maintained.

[Equation 1]

According to Equation 1, the spectral bandwidth of the wave source 100 can be maintained below 5 nm when light is irradiated into the sample 201 at each reference time to measure the change in the pattern of the laser speckle.

As an example, the wave source 100 may be disposed outside the sample unit 200 and irradiate the wave L1 toward the sample unit 200, as shown. At this time, the wave source 100 may be placed adjacent to the sample unit 200 and irradiate the wave L1 directly toward the sample unit 200. However, it may be placed at a distance from the sample unit 200 and use an optical fiber (optical fiber). The wave (L1) may be irradiated to the sample unit 200 through the fiber (105).

Since the optical fiber 105 performs the function of transmitting only specific wavelengths, it can remove noise of the wave L1 transmitted from the wave source 100 and expand the beam size when emitting the wave L1. It may also be provided as a collimator (107).

The sample unit 200 can accommodate the sample 201 to be measured. The sample 201 may be accommodated through a sample placement means such as a container or pipe, and may be accommodated in a static state. As an example, as shown in the drawing, the sample unit 200 can accommodate a sample 201 that is stabilized due to lack of fluidity using a container. As another example, although not shown, the sample unit 200 may accommodate the fluid sample 201 using a pipe. At this time, when the sample unit 200 uses a pipe, the sample 201 may be a liquid, and the sample unit 200 circulates the sample 201 at least once along the entire flow path including the pipe to dissolve the sample 201 in the pipe. A stable state of the sample 201 can be formed.

The sample unit 200 may further include a multiple scattering amplifier unit. The multiple scattering amplifier may reflect at least a portion of the output wave L2 emitted from the sample 201 to the sample 201 to amplify the number of multiple scatterings in the sample 201. The multiple scattering amplifier may include multiple scattering materials. For example, the multiple scattering material includes titanium oxide (TiO2), and the multiple scattering amplifier may reflect at least a portion of the wave L1 incident on the multiple scattering amplifier.

The multiple scattering amplifier is disposed adjacent to the sample 201, so that the output wave (L2), which is multiply scattered and emitted from the sample 201, travels through the space between the sample 201 and the multiple scattering amplifier at least once. You can. The multiple scattering amplifier may be placed on the path of the wave, and may be placed on the path of the incident wave (L1) and the path of the exit wave (L2), respectively.

In another embodiment, the sample unit 200 does not include a multiple scattering amplification unit as a separate component, but rather coats the surface of the main body of the sample unit 200, which accommodates the sample 201, using a multiple scattering amplification material, It may be composed of multiple scattering amplification areas. Alternatively, the sample unit 200 may be configured to include multiple scattering substances in the sample 201. The multiple scattering amplification area may scatter at least a portion of the wave incident on the internal space of the sample unit 200 and emitted after passing through the sample 201 back into the sample 201. The waves scattered in this way are emitted to the other side through the sample 201 and are scattered, and through this process, the number of multiple scatterings within the sample 201 can be increased. The multiple scattering amplification area may be formed in at least a portion of the path along which the wave passes, and may be disposed in the entire area except for some areas where the wave is emitted and some areas where the wave is incident.

The image sensor 300 is disposed on the path along which the output wave (L2) passes, and can acquire a plurality of images by photographing the output wave (L2) emitted from the sample 201 in time series. The image sensor 300 may include sensing means corresponding to the type of wave source 100. For example, when using a light source in the visible light wavelength band, a CCD camera, a photographing device that captures images, is used. can be used.

Here, each of the plurality of images may include speckle information generated by multiple scattering by microorganisms due to the wave L1 incident on the sample 201. In other words, the image sensor 300 can detect laser speckles generated by multiple scattering of the irradiated wave L1 within the sample 201 at a preset time point. Here, time refers to a moment in the continuous flow of time, and times may be set in advance at the same time interval, but are not necessarily limited to this, and are set in advance at arbitrary time intervals. It could be.

The image sensor 300 detects a first image including first speckle information at least from a first viewpoint, and captures a second image including second speckle information from a second viewpoint to provide microorganism information. It can be provided as (400). Meanwhile, the first viewpoint and the second viewpoint are only examples selected for convenience of explanation, and the image sensor 300 may capture a plurality of images from a plurality of viewpoints more than the first viewpoint and the second viewpoint. The image sensor 300 is provided with a polarizer 305 in the path along which the output wave L2 passes, thereby maximizing interference efficiency for forming speckles and removing unnecessary external reflected light.

The image sensor 300 may be placed at a certain distance from the sample unit 200 so that the size d of one pixel of the image sensor is smaller than or equal to the grain size of the speckle pattern. For example, the image sensor 300 may be placed on the path along which the output wave L2 passes so as to satisfy the conditions of Equation 2 below.

[Equation 2]

As shown in Equation 2, the size d of one pixel of the image sensor 300 must be less than the grain size of the speckle pattern, but if the pixel size is too small, undersampling occurs. There may be difficulties utilizing pixel resolution. Accordingly, in order to achieve an effective signal to noise ratio (SNR), the image sensor 300 may be arranged so that up to 5 pixels or less are located in the speckle grain size.

Meanwhile, the microbial information providing device 400 may receive a plurality of images from the image sensor 300, perform machine learning using them, and then perform a function of distinguishing the type or concentration of microorganisms in the sample 201. . Hereinafter, with reference to FIGS. 2 and 3, the microorganism information providing device 400 of the present invention will be described in more detail.

Figure 2 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.

The network environment in FIG. 2 shows an example including a plurality of user terminals 401, 402, 403, and 404, a server 405, and a network 406. Here, the microorganism information providing device 400 may be a server or a user terminal. Figure 2 is an example for explaining the invention, and the number of user terminals or servers is not limited as in Figure 2.

The plurality of user terminals 401, 402, 403, and 404 may be fixed terminals implemented as computer devices or mobile terminals. When the microorganism information providing device 400 is a server 405, the plurality of user terminals 401, 402, 403, and 404 may be terminals of an administrator who controls the server. Examples of the plurality of user terminals 401, 402, 403, and 404 include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Portable Multimedia Players). ), tablet PC, etc. For example, user terminal 1 (401) may communicate with other user terminals (402, 403, 404) and/or the server 405 through the network 406 using wireless or wired communication methods. In another embodiment, a plurality of user terminals 401, 402, 403, and 404 may be equipped with the above-described image sensor 300 and transmit a plurality of acquired images to the server 405 through the network 406. there is.

The communication method is not limited, and may include not only a communication method utilizing a communication network that the network 406 may include (for example, a mobile communication network, wired Internet, wireless Internet, and a broadcast network), but also short-range wireless communication between devices. For example, the network 406 may be a personal area network (PAN), a local area network (LAN), a capus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), or a metropolitan area network (MAN). ), WAN (wide area network), BBN (broadband network), Internet, etc. may include one or more arbitrary networks. Additionally, network 406 may include any one or more of network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, etc. Not limited.

The server 405 is a computer device or a plurality of computer devices that communicate with a plurality of user terminals 401, 402, 403, 404 and a network 406 to provide commands, codes, files, content, services, etc. It can be implemented.

For example, the server 405 may provide a file for installing an application to user terminal 1 (401) connected through the network 406. In this case, user terminal 1 (401) can install the application using a file provided from the server. In addition, user terminal 1 (401) connects to the server 405 under the control of an operating system (OS) and at least one program (for example, a browser or an installed application) and services provided by the server 405. I can receive content. As another example, the server 405 may establish a communication session for data transmission and reception, and route data transmission and reception between a plurality of user terminals 401, 402, 403, and 404 through the established communication session.

Figure 3 is a block diagram of a microorganism information providing device 400 according to an embodiment of the present invention.

Referring to FIG. 3, the microorganism information providing device 400 according to an embodiment of the present invention may correspond to at least one processor or include at least one processor. Accordingly, the microbial information providing device 400 may be driven as included in a hardware device such as a microprocessor or general-purpose computer system. Here, 'processor' may mean, for example, a data processing device built into hardware that has a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of data processing devices built into hardware include microprocessor, central processing unit (CPU), processor core, multiprocessor, and application-specific integrated (ASIC). Circuit), FPGA (Field Programmable Gate Array), etc., but the scope of the present invention is not limited thereto. The microorganism information providing device 400 may be mounted on at least one of a plurality of user terminals 401, 402, 403, and 404, or may be provided on the server 405. As another embodiment, the microorganism information providing device 400 may be provided in two servers 405. At this time, one server is used to learn the criteria for microbial classification, which will be described later, and the other server can determine the type or concentration of microorganisms in the measured sample based on the learned algorithm.

The microorganism information providing device 400 shown in FIG. 3 shows only components related to this embodiment in order to prevent the features of this embodiment from being obscured. Accordingly, those skilled in the art can understand that other general-purpose components may be included in addition to the components shown in FIG. 3.

Referring to FIG. 3, the microorganism information providing device 400 according to an embodiment of the present invention may include a receiving unit 410, a processor 420, a memory 430, and an input/output interface 440.

The receiving unit 410 may receive a plurality of images taken in time series of the output wave L2 emitted from the sample 201 described above. As an embodiment, when the microbial information providing device 400 is mounted on a user terminal (401, 402, 403, 440) equipped with an image sensor 300, the receiver 410 is connected to the image sensor 300 by wire. You can receive multiple connected images.

As another embodiment, when the microbial information providing device 400 is provided in a server 405 that is provided separately from the image sensor 300, the receiving unit 410 functions as a communication module using wired or wireless communication, Videos can be provided. At this time, the receiving unit 410 can provide a function for user terminal 1 (401) and the server 405 to communicate through the network 406, and can communicate with another user terminal (for example, user terminal 2 (402)) or another user terminal. A function for communicating with a server (for example, server 405) may be provided.

For example, a request generated by the processor of user terminal 1 (401) according to a program code stored in a recording device such as a memory may be transmitted to the server 405 through the network 406 under the control of the communication module. Conversely, control signals, commands, content, files, etc. provided under the control of the processor 420 of the server 405 are transmitted to the communication module of user terminal 1 (401) through the receiver 410 and the network 406. It can be received by user terminal 1 (401). For example, control signals or commands of the server 405 received through the communication module may be transmitted to the processor or memory.

The processor 420 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 420 by the memory 430 or the receiver 410. For example, processor 420 may be configured to execute received instructions according to program code stored in a recording device such as memory 430. The processor 420 may include a learning unit 421, a detection unit 422, and a determination unit 423.

The memory 430 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Additionally, the memory 430 may store an operating system and at least one program code (for example, code for a browser installed and running on user terminal 1 (401) or the above-described application, etc.). These software components may be loaded from a computer-readable recording medium separate from the memory 430 using a drive mechanism. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 430 through the receiver 410 rather than a computer-readable recording medium. For example, at least one program is a program (for example, installed by files provided through the network 406 by developers or a file distribution system that distributes the installation file of the application (for example, the server 405 described above) It may be loaded into the memory 430 based on the above-described application).

The input/output interface 440 may be a means for interfacing with an input/output device. For example, an input device may include a device such as a keyboard or mouse, and an output device may include a device such as a display for displaying a communication session of an application. As another example, the input/output interface 440 may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen. As a more specific example, the processor 420 of the server 405 processes the commands of the computer program loaded in the memory 430, and inputs and outputs service screens or content constructed using data provided by user terminal 2 (402). It may be displayed on the display through the interface 440.

Hereinafter, with reference to FIG. 4, the principle of the microorganism information providing device 400 of the present invention will be described.

FIG. 4 is a diagram illustrating the basic principle by which the microorganism information providing device 400 confirms the presence of microorganisms in the sample 201 according to an embodiment of the present invention.

In the case of materials with a homogeneous internal refractive index, such as glass, refraction occurs in a certain direction when light is irradiated. However, when coherent light such as a laser is irradiated to an object with a non-homogeneous internal refractive index, very complex multiple scattering occurs inside the material.

Referring to FIG. 4, among the light or wave (hereinafter referred to as wave for simplicity) emitted from the wave source, a portion of the wave scattered in a complex path through multiple scattering passes through the inspection target surface. Waves passing through various points on the surface to be inspected cause constructive interference or destructive interference with each other, and the constructive/destructive interference of these waves generates grain-shaped patterns (speckles). do.

In this specification, waves scattered through such complex paths are called “chaotic waves,” and chaotic waves can be detected through speckle information.

Again, the left drawing of FIG. 4 is a drawing showing when a stable medium is irradiated with a laser. When a stable medium in which there is no movement of internal constituent materials is irradiated with interference light (for example, a laser), a stable speckle pattern without change can be observed.

However, as shown in the right drawing of FIG. 4, if it contains an unstable medium that moves among the internal components, such as bacteria, the speckle pattern changes.

In other words, the light path can change in real time due to the microscopic life activities of organisms (e.g., movement within cells, movement of microorganisms, movement of mites, etc.). Since the speckle pattern is a phenomenon that occurs due to the interference of waves, a slight change in the optical path can cause a change in the speckle pattern. In particular, minute changes in the optical path are expressed as a high signal-to-noise ratio due to the characteristics of the data measured by the embodiments of the present invention, which is an image interfered with by a narrow bandwidth light source, and the signal is multiplied by microorganisms due to multiple scattering. This is because it is influenced by Accordingly, by measuring temporal changes in the speckle pattern, the movement of the organism can be quickly measured. In this way, when measuring changes in the speckle pattern over time, the presence and concentration of organisms can be known, and furthermore, the type of organism can also be known.

Figures 5a and 5b show in time series a method for providing microbial information according to an embodiment of the present invention. Specifically, Figure 5a shows in time series a method for learning classification criteria among the methods for providing microorganism information according to an embodiment of the present invention, and Figure 5b shows classification standards among the method for providing microorganism information according to an embodiment of the present invention. It is a time series representation of the method for classifying the concentration or type of microorganisms based on .

First, referring to FIG. 5A, the microorganism information providing system 1 according to an embodiment of the present invention prepares a sample of which the type or concentration of microorganisms is known in advance, and photographs the sample to obtain a plurality of learning images. .

The process of receiving multiple learning images from the microorganism information provision system 1 is the same as the process of receiving multiple images of a new sample, which will be described later. In other words, the microbial information provision system 1 uses the wave source 100 to irradiate the wave L1 to the sample 201 accommodated in the sample unit 200. At this time, the sample 201 may be a sample whose concentration or type of microorganisms is known in advance.

Next, the image sensor 300 acquires a plurality of images by capturing the output wave L2 emitted from the sample 201 in time series. In other words, the image sensor 300 can acquire a plurality of images by photographing the sample 201 at a preset time point or at each time point, and at this time, each of the plurality of images is a wave (L1) incident on the sample 201. ) may include speckle information generated by multiple scattering by microorganisms. The image sensor 300 can acquire a plurality of images at a speed sufficient to detect the movement of microorganisms. For example, it can photograph the sample 201 at a rate of 25 to 30 frames per second.

Next, the microorganism information providing device 400 receives a plurality of learning images acquired from the image sensor 300 through the receiving unit 410.

Next, the detector 422 extracts features of changes over time from a plurality of learning images (S12). Here, the plurality of learning images are images captured continuously at time intervals, and time information is included between the plurality of learning images captured in time series. The detector 422 can extract features of changes over time from these plurality of learning images.

Next, the learning unit 421 machine learns classification criteria based on the extracted features (S13). The learning unit 421 learns classification criteria based on deep learning, and deep learning creates high-level abstractions (key information in large amounts of data or complex data) through a combination of various non-linear transformation techniques. It is defined as a set of machine learning algorithms that attempt to summarize content or function. The learning unit 421 uses deep learning models such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), and deep belief neural networks (DNN). Networks, DBN) may be used.

As an example, the learning unit 421 may machine learn classification criteria based on the temporal correlation of the plurality of received learning images.

As described above, the plurality of learning images may include speckle information generated by multiple scattering from the sample 201. As previously explained with reference to FIG. 4, when microorganisms exist in the sample 201, the speckle may change over time due to the biological activity of the microorganisms. In addition, since the change in speckle over time varies depending on the type or concentration of microorganisms, the learning unit 421 can learn classification criteria for classifying the type or concentration of microorganisms using the change in speckle over time. You can.

As an embodiment, the learning unit 421 machine learns the classification standard using changes in the speckle pattern, which is speckle information included in each of the plurality of learning images, or the temporal correlation of the plurality of received learning images. Classification criteria can be machine learned based on temporal correlation. At this time, the learning unit 421 can learn the classification criteria for microorganisms using the speckle pattern detected in each of the plurality of learning images among the features.

Meanwhile, Figure 5a shows a step of extracting features of changes over time in the detection unit 422 (S12) and a step of machine learning classification criteria based on the features extracted by the learning unit 421 (S13). However, the present invention is not necessarily limited thereto, and of course, each step does not have a priority relationship and may be performed simultaneously.

FIG. 6 is a diagram illustrating a method of analyzing the temporal correlation of speckles in the learning unit 421 according to an embodiment of the present invention.

Referring to FIG. 6, the learning unit 421 classifies using the difference between the first image information of the speckle detected at the first viewpoint and the second image information of the speckle detected at a second viewpoint different from the first viewpoint. Standards can be learned. Here, the first image information and the second image information may be at least one of speckle pattern information or wave intensity information. Meanwhile, an embodiment of the present invention does not only use the difference between the first image information at the first viewpoint and the second image information at the second viewpoint, but extends it to include the difference in the plurality of images detected at the plurality of viewpoints. Speckle information can be used. The learning unit 421 can calculate a temporal correlation coefficient between images using image information of speckles generated at a plurality of preset viewpoints, and learn classification criteria based on the temporal correlation coefficient.

For example, the time correlation coefficient calculated based on the light intensity of the speckle pattern can be calculated using Equation 3 below.

[Equation 3]

In equation 3: is the temporal correlation coefficient, is the normalized light intensity, (x,y) are the pixel coordinates of the camera, t is the measured time, T is the total measurement time, represents time lag.

The time correlation coefficient can be calculated according to Equation 3. In one embodiment, through analysis that the time correlation coefficient falls below a preset standard value, the learning unit 421 classifies the type or concentration of microorganisms. Learn the standards. Specifically, it can be analyzed that microorganisms exist when the time correlation coefficient exceeds a preset error range and falls below a reference value, and these reference values may have different values depending on the type of microorganism. The learning unit 421 can learn a classification standard for distinguishing types of microorganisms using the reference value analysis of the time correlation coefficient described above.

In addition, as shown in the graph of FIG. 6, as the concentration of microorganisms increases, the time for the time correlation coefficient to fall below the reference value becomes shorter. Using this, the concentration of microorganisms is determined through the slope value of the graph representing the time correlation coefficient. can be analyzed. The learning unit 421 can learn classification criteria for classifying the concentration of microorganisms using slope value analysis of the time correlation coefficient.

Figure 7 is a diagram showing the standard deviation distribution of the light intensity of the speckle measured over time.

Referring to FIG. 7, the learning unit 421 can calculate the standard deviation of the light intensity of the speckle pattern using a plurality of learning images measured at each reference time. As bacteria and microorganisms present in a sample continuously move, constructive interference and destructive interference may change in response to the movement. At this time, as constructive interference and destructive interference change, the degree of light intensity may change. The learning unit 421 can calculate the standard deviation indicating the degree of change in light intensity, analyze where bacteria and microorganisms are located in the sample, and learn the distribution map of the bacteria and microorganisms.

For example, the learning unit 421 may calculate the standard deviation of the light intensity over time of the speckle pattern detected in each of the plurality of learning images. The standard deviation of the light intensity over time of the speckle can be calculated based on Equation 4 below.

[Equation 4]

In Equation 4, S: standard deviation, (x,y): camera pixel coordinates, T: total measurement time, t: measurement time, It: light intensity measured at time t, : Can indicate average light intensity over time.

Since the constructive and destructive interference patterns change depending on the movement of bacteria and microorganisms, and the standard deviation value calculated based on Equation 4 changes, the concentration of bacteria and microorganisms can be measured based on this. The learning unit 421 may learn classification criteria based on the linear relationship between the size of the standard deviation value of the light intensity of the speckle pattern and the concentration of bacteria and microorganisms.

Hereinafter, the description will focus on the case where the learning unit 421 learns classification criteria using a convolutional neural network (CNN).

Here, a convolutional neural network (CNN) is a type of multilayer perceptrons designed to use minimal preprocessing. A convolutional neural network (CNN) includes a convolution layer that performs convolution on input data, and further includes a subsampling layer that performs subsampling on the image to extract a feature map from the data. can do. Here, the subsampling layer is a layer that increases the contrast between neighboring data and reduces the amount of data to be processed, and max pooling, average pooling, etc. may be used.

Figure 8 is an example diagram of a convolutional neural network according to an embodiment of the present invention, and Figures 9 and 10 are diagrams for explaining the convolution operation of Figure 8.

Referring to FIGS. 8 to 10 , the convolutional neural network (CNN) 510 used by the learning unit 421 according to an embodiment of the present invention may include a plurality of convolutional layers (A1). The learning unit 421 may generate an output by performing a convolution operation between the kernels of the convolution layers and the inputs.

The input of the convolution layer is data adopted as input to the corresponding convolution layer and includes at least one input feature map corresponding to the initial input data or the output generated by the previous layer. For example, the input of convolutional layer 1 (A11) shown in FIG. 8 is a plurality of images 501, which are the first inputs of the convolutional neural network 510, and the input of convolutional layer 2 (A12) is the convolutional neural network 510. It may be the output 511 of layer 1 (A11).

The input 501 of the convolutional neural network 510 is a plurality of C images 501 received from the receiver 410, and a temporal correlation may be established between the images, which are input feature maps. Each image, that is, each input feature map, may have a plurality of pixels with a preset width (W) and height (H). Since there are C input feature maps, the size of the input 501 can be expressed as W x H x C. The learning unit 421 may perform a convolution operation corresponding to the input 501 using one kernel corresponding to the convolution layer A1.

At least one kernel of the convolution layer (A1) is data employed for a convolution operation corresponding to the corresponding convolution layer (A1), and may be defined, for example, based on the input and output of the corresponding convolution layer. . At least one kernel may be designed for each convolutional layer (A1) constituting the convolutional neural network 510, and at least one kernel corresponding to each convolutional layer may be referred to as a kernel set (S600).

Here, the kernel set (S600) may include kernels corresponding to the output channels (D). For example, in order to obtain the desired output of the convolution layer A1, a kernel set S600 of the convolution layer may be defined so that a convolution operation is performed with the input of the convolution layer A1. The output of the convolution layer (A1) is data resulting from a convolution operation between the input of the corresponding convolution layer and the kernel set, includes at least one output feature map, and can be adopted as the input of the next layer.

The learning unit 421 may generate an output 511 by performing a convolution operation between the kernel set S600 corresponding to the convolution layer A1 and the input 501. The output 511 of the convolutional layer A1 includes output feature maps 5111 corresponding to D output channels, and the size of each output feature map 5111 may be W x H. Here, the width, height, and number (depth) of the output 511 are W, H, and D, respectively, and the size of the output 511 can be expressed as W x H x D.

For example, the kernel set S600 corresponding to the convolution layer A1 may include convolution kernels corresponding to the number of D output channels. The learning unit 421 may generate output feature maps 5111 corresponding to the D output channels based on the results of operations between the input 501 and kernels corresponding to the D output channels.

The learning unit 421 includes a plurality of convolutional layers (A1) and, in one embodiment, may include 4 to 7 convolutional layers (A1). For example, as shown in the figure, it may include five convolutional layers (A11, A12, A13, A14, and A15). Through these plurality of convolutional layers (A11, A12, A13, A14, and A15), the learning unit 421 can increase the learning capacity that can imply complex non-linear relationships. However, the present invention is not limited to this, and of course learning can be done using more convolutional layers (A1).

Meanwhile, each convolutional layer (A1) may include an activation function. The activation function can be applied to each layer to perform a function that allows each input to have a complex non-linear relationship. Activation functions can be used such as Sigmoid, tanh, Rectified Linear Unit (ReLU), and Leacky ReLU, which can convert input to normalized output. .

The learning unit 421 according to an embodiment of the present invention completely initializes the kernels using values from -1 to 1 when first training using the convolutional neural network 510. Data with negative values When using an activation function that outputs all values as 0, output values containing valid information cannot be passed on to the next convolution layer, which may reduce learning efficiency. Therefore, in one embodiment of the present invention, the learning unit 421 adds a Leacky ReLU layer behind the convolution layer A1, outputs positive values as is, but applies a constant slope to input data of negative values. It can be output to have . Through this, the learning unit 421 can prevent slow convergence speed and local minimization problems during learning.

The input 501 may be a set of input feature maps with padding applied, and padding refers to a technique of filling some areas of the input with a specific value. Specifically, applying padding to the input with the pad size set to 1 means filling in a specific value at the edge of the input feature map, and zero padding means setting that specific value to 0. it means. For example, if zero padding with a pad size of 1 is applied to an input of size X x Y x Z, the padded input has an edge of 0 and a size of (X+1) As (Y+1) x Z data, it may include (X+1) x (Y+1) x Z input elements.

Meanwhile, the learning unit 421 performs a convolution operation using a plurality of images including speckle information as input, and learns using the temporal correlation of the plurality of images. At this time, each image includes It may contain multiple speckles, which are grain-shaped patterns. At this time, the learning unit 421 can learn the classification criteria based on the temporal correlation of each speckle. In other words, the learning unit 421 can learn classification criteria based on the temporal correlation of each speckle. In other words, the learning unit 421 includes time information rather than two-dimensional information of the image. Classification criteria are learned using three-dimensional information.

The learning unit 421 must focus on the information about one speckle, and in order to accurately obtain three-dimensional information about one speckle, it must distinguish between the one speckle and surrounding speckles to set a classification standard. It must be learned. Therefore, the learning unit 421 can perform a convolution operation using a convolution kernel with a size smaller than the size of one speckle. That is, when the size of one speckle corresponds to m pixels, the learning unit 421 performs a convolution operation using a convolution kernel of size n x n smaller than m. As an embodiment, as described above, the image sensor 300 is arranged so that a maximum of 5 pixels are located in the speckle grain size, so m may be 5 and n is 1. It can have a value. That is, the learning unit 421 can perform a convolution operation using a 1 x 1 convolution kernel. However, the technical idea of the present invention is to perform a convolution operation using a kernel whose size is smaller than the size of the speckle, and is not limited to this.

The kernel set S600 includes convolution kernels corresponding to D output channels, and each convolution kernel may include kernel feature maps corresponding to a plurality of images. Since the size of each kernel feature map is n x n, the kernel set (S600) includes n x n x C x D kernel elements. Here, the size of the convolution kernel is n x n x C, and C can be the number of multiple images, that is, the number of image frames. The number of convolution kernels may be equal to the number (C) of a plurality of images, and in this case, the output result of each layer can be used for Fourier transform under the same conditions or analysis using the same.

As shown in FIG. 9, the learning unit 421 performs an operation between the convolution kernel corresponding to the first output channel of the kernel set (S600) and the input 501 to generate an output feature map ( 5111) can be generated. In this way, the learning unit 421 can generate output feature maps 5111 corresponding to the D output channels by performing operations between the D kernels in the kernel set (S600) and the input 501. And, an output 511 including the generated output feature maps 5111 can be generated.

For example, the learning unit 421 performs an operation between the convolution kernel 600 corresponding to the Dth output channel of n x n x C size and the input 501 of size W x H x C to obtain W x H An output feature map 5111 with a size of can be generated, and the generated output feature map 5111 corresponds to the Dth output channel. Specifically, the convolution kernel 600 corresponding to the D output channel includes C channel feature maps, and the size of each kernel feature map is n x n. The learning unit 421 slides each kernel feature map with a size of n x n on each input feature map with a size of W x H included in the input 501 with a specific stride, and creates a An output feature map 5111, which is the result of an operation between the convolution kernel 600 and the input 501, can be generated.

Here, stride refers to the interval of sliding the kernel feature map during convolution operation. As described above, in order to focus on information about one speckle, the learning unit 421 must learn a classification standard by distinguishing between the one speckle and surrounding speckles, so the sliding interval stride (s) ) may have a value corresponding to the size of the convolution kernel so that each convolution kernel and the corresponding region do not overlap. In other words, when using a convolution kernel of size n x n, stride (s) can have n values. For example, in the case of a 1 x 1 convolution kernel, stride (s) may be 1. Through this, the learning unit 421 can make one speckle non-overlap with other surrounding speckles and learn a classification standard using only the time information of one speckle.

As an embodiment, the learning unit 421 performs a convolution operation to have the number of output channels (D) equal to the number of images (C) when learning classification criteria using the convolutional neural network 510. You can. In other words, the kernel set S600 may include D convolution kernels including C kernel feature maps, and in this case, C and D may be the same.

Meanwhile, the learning unit 421 may reduce the size of the output 515 by using a pooling operation after performing a convolution operation using the convolution layers (A1). For example, the learning unit 421 may reduce the size of the output 515 using sub sampling (A2). For example, subsampling may be global average pooling, which is an operation that sets the average value within a range of a certain size as a representative of that range. However, the present invention is not limited to this, and of course max pooling, min pooling, etc. can be used.

Afterwards, the learning unit 421 uses a fully connected layer to apply weights along with a predetermined operation to the feature maps 521 extracted by passing the subsampling (A2) filter, and then outputs the final output. You can obtain (541). For example, the learning unit 421 further applies Leaky ReLU to the fully connected layer after performing subsampling (A2) (A3), and then to the fully connected layer (A3). The final output can be obtained by applying a softmax layer to the layer (A4). Here, the final output 541 may be a classification standard for distinguishing types or concentrations of microorganisms based on the temporal correlation of speckles.

Next, referring to FIG. 5B, the method for providing microorganism information according to an embodiment of the present invention radiates waves to a new path (S21) and receives a plurality of images taken in time series of the emitted waves (S22). . Thereafter, in the method of providing microorganism information, the determination unit 423 can distinguish the type or concentration of microorganisms included in the new sample 201 based on the obtained classification criteria (S23). The output data obtained through the above-described process can be provided back to the learning unit 421 and used as learning data.

Figure 11 is a graph comparing expected microbial information (prediction) and actual microbial information (ground truth) obtained using a method for providing microbial information according to an embodiment of the present invention.

Referring to FIG. 11, it can be seen that the matching rate between expected microbial information and actual microbial information obtained through the method for providing microbial information according to an embodiment of the present invention is very high. Through the results of FIG. 11, the method for providing microorganism information not only distinguishes the types of microorganisms (B.subtilis, E.Coli, P.aeruginosa, S.aureus) contained in the sample 201, but also determines the concentration of each. You can confirm that they are separated.

As described above, the apparatus and method for providing microorganism information according to embodiments of the present invention obtain classification criteria for classifying the type or concentration of microorganisms using changes in the temporal correlation of speckles, thereby providing a separate chemical method. It is possible to quickly and accurately distinguish the type or concentration of microorganisms in a sample without relying on the method. Through this, it is possible to quickly and effectively prescribe medical treatments such as antibiotics to patients with infectious diseases, as well as various applications such as checking the subject's health status.

So far, the present invention has been examined with a focus on preferred embodiments. A person skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting perspective. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

1: Microbial information provision system
100: wave source
200: Sample section
300: Image sensor
400: Microbial information provision device
410: Receiving unit
421: Learning Department
422: detection unit
423: Judgment unit

Claims (1)

A receiving unit that receives a plurality of images captured in time series of the waves emitted from the sample;
a detection unit that extracts features of changes over time from a plurality of images captured in the time series;
A learning unit that machine learns classification criteria based on the extracted features; and
It includes a determination unit that classifies the type or concentration of microorganisms contained in the sample based on the classification criteria,
Each of the plurality of images includes speckle information generated by multiple scattering by the microorganisms due to waves incident on the sample,
The learning department
Distinguish between one speckle and other surrounding speckles formed around the one speckle from the plurality of images,
A microorganism information providing device that learns the classification criteria based on the time information of the single differentiated speckle.

Images