CN111443476A - Microscope auto-focusing method, microscope system, medical device, and storage medium - Google Patents
Microscope auto-focusing method, microscope system, medical device, and storage medium Download PDFInfo
- Publication number
- CN111443476A CN111443476A CN202010284514.XA CN202010284514A CN111443476A CN 111443476 A CN111443476 A CN 111443476A CN 202010284514 A CN202010284514 A CN 202010284514A CN 111443476 A CN111443476 A CN 111443476A
- Authority
- CN
- China
- Prior art keywords
- image
- microscope
- camera
- light
- lens
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000003287 optical effect Effects 0.000 claims description 45
- 238000011156 evaluation Methods 0.000 claims description 27
- 238000012545 processing Methods 0.000 claims description 25
- 238000005259 measurement Methods 0.000 claims description 14
- 230000010287 polarization Effects 0.000 claims description 12
- 238000011088 calibration curve Methods 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 16
- 230000008569 process Effects 0.000 description 11
- 230000003190 augmentative effect Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000004422 calculation algorithm Methods 0.000 description 8
- 238000013473 artificial intelligence Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000001575 pathological effect Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 5
- 238000003745 diagnosis Methods 0.000 description 5
- 210000003734 kidney Anatomy 0.000 description 3
- 239000007788 liquid Substances 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000003672 processing method Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004195 computer-aided diagnosis Methods 0.000 description 2
- 239000013256 coordination polymer Substances 0.000 description 2
- 235000019800 disodium phosphate Nutrition 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000003902 lesion Effects 0.000 description 2
- 238000001000 micrograph Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 108700004914 Ac-Nal(1)-Cpa(2)-Pal(3,6)-Arg(5)-Ala(10)- LHRH Proteins 0.000 description 1
- 206010008342 Cervix carcinoma Diseases 0.000 description 1
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 1
- 208000006105 Uterine Cervical Neoplasms Diseases 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 201000010881 cervical cancer Diseases 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 201000005202 lung cancer Diseases 0.000 description 1
- 208000020816 lung neoplasm Diseases 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/28—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising
- G02B27/283—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising used for beam splitting or combining
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/02—Objectives
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/18—Arrangements with more than one light path, e.g. for comparing two specimens
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/24—Base structure
- G02B21/241—Devices for focusing
- G02B21/244—Devices for focusing using image analysis techniques
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/24—Base structure
- G02B21/241—Devices for focusing
- G02B21/245—Devices for focusing using auxiliary sources, detectors
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/24—Base structure
- G02B21/248—Base structure objective (or ocular) turrets
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/361—Optical details, e.g. image relay to the camera or image sensor
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/10—Beam splitting or combining systems
- G02B27/106—Beam splitting or combining systems for splitting or combining a plurality of identical beams or images, e.g. image replication
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/10—Beam splitting or combining systems
- G02B27/14—Beam splitting or combining systems operating by reflection only
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
- G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Microscoopes, Condenser (AREA)
Abstract
The present invention provides a microscope system comprising: the beam splitter component is used for separating and projecting light rays in different light paths respectively; the lens assembly is used for projecting light rays generated by an observed sample during observation after the light rays enter different light paths through the objective lens, and the camera assembly is used for photographing the sample to be observed in the field of view of the microscope so as to realize that the camera assembly shoots clear focused images through the first light path; an auxiliary focusing device for determining a focal length matching the camera assembly; the focusing device is configured to adjust the focal length of the image light entering the camera assembly according to the defocusing amount of the sample image to be measured determined by the auxiliary focusing device. The invention also provides an automatic focusing method of the microscope, medical equipment and a storage medium, and the automatic focusing method can realize automatic focusing of the microscope system, form and output a clear focused image shot through the first light path, save the focusing time of the microscope system and improve the focusing accuracy.
Description
Technical Field
The present invention relates to a medical image processing technology, and more particularly, to an auto-focusing method for a microscope, a microscope system, a medical device, and a storage medium.
Background
As artificial intelligence technology is researched and developed, the artificial intelligence technology is developed and applied in various fields, for example, augmented reality technology and artificial intelligence have been proposed in recent years for use in conventional optical microscope systems. The method comprises the steps of acquiring an image of a sample to be observed by using a camera on a traditional optical microscope, and analyzing the real-time image by combining a machine learning algorithm.
The fact that the camera can acquire high-quality images is the guarantee of the accuracy of the algorithm of the augmented reality microscope. The defocused image of the sample loses much important optical information, so that it is important to ensure that the camera can acquire an image of the sample which is accurately focused. The microscope image with inaccurate focus affects the output effect of the model.
Disclosure of Invention
In view of this, an embodiment of the present invention provides an auto-focusing method for a microscope, a microscope system, a medical device, and a storage medium, and the technical solution of the embodiment of the present invention is implemented as follows:
an embodiment of the present invention provides a microscope system, including:
the objective lens is used for acquiring light rays of a sample to be observed, entering the first light path, and converging the light rays with the light rays generated by the image projection module at the beam splitter after entering the first light path through the lens component;
the beam splitter component comprises at least one beam splitter and is used for separating and projecting light rays in different light paths respectively;
the lens assembly comprises at least one lens and is used for projecting light rays generated by an observed sample during observation after the light rays enter different optical paths through the objective lens so as to realize propagation of the light rays along different optical paths;
the image projection component is arranged in a corresponding light path of the light projected by the lens component and is used for carrying out image enhancement processing on the image of the sample to be observed;
the camera assembly is arranged in the first optical path and comprises a camera for photographing a sample to be observed in the field of view of the microscope so as to form and output a clearly focused image photographed through the first optical path;
the auxiliary focusing device comprises an auxiliary focusing light source and an auxiliary focusing camera, is arranged in the second light path and is used for determining a focal length matched with the camera component;
a focusing device configured to adjust a focal length of image light entering the camera assembly according to the defocus amount of the sample image to be measured determined by the auxiliary focusing device.
In the above scheme, the microscope system further includes:
the eyepiece is sleeved with the trinocular lens barrel and used for observing a sample to be observed through the objective lens;
the eyepiece barrel is arranged at one end, far away from the objective, of the beam splitter and comprises at least two channels and a tube lens, wherein the channels are located at one end, far away from the beam splitter, of the channels, one of the channels is communicated with the eyepiece, and the tube lens is located at one end, close to the beam splitter.
In the above scheme, the focusing device includes a moving driving component and a zoom lens, so as to photograph a sample to be observed in the field of view of the microscope at different focal lengths.
In the above-mentioned scheme, the first step of the method,
the beam splitter assembly is respectively communicated with the objective lens and the tube lens of the lens barrel, and the camera assembly is arranged in one channel of the lens barrel;
the beam splitter assembly comprises a beam splitter, and the lens assembly comprises a lens and is arranged between the beam splitter and the image projection assembly;
the focusing device is positioned between the beam splitter and the camera assembly and used for adjusting the focal length of the image light rays entering the camera assembly according to the defocusing amount of the sample image to be detected determined by the auxiliary focusing device.
In the above-mentioned scheme, the first step of the method,
the image projection assembly further comprises a first polarizer, wherein the first polarizer is positioned between the lens assembly and the beam splitter and is used for carrying out polarization processing on the corresponding light rays in the first light path;
the camera assembly further includes a second polarizer positioned between the focusing device and the beam splitter for polarizing the respective light rays collected by the camera assembly.
In the above-mentioned scheme, the first step of the method,
the auxiliary focusing light source is arranged in a Fourier back focal plane corresponding to a condenser lens assembly of the microscope system and used for emitting auxiliary focusing light rays to form the second light path;
the beam splitter assembly comprises a beam splitter arranged between the focusing device and the camera assembly and used for reflecting the light rays in the second light path to the auxiliary focusing camera;
the auxiliary focusing camera is arranged at an axial offset position of a conjugate plane of the camera assembly and is used for shooting an overlapped image matched with a sample to be observed in the microscope visual field based on light rays in the second light path.
In the above-mentioned scheme, the first step of the method,
the auxiliary focusing light source is arranged in a Fourier back focal plane corresponding to a condenser lens assembly of the microscope system and used for emitting auxiliary focusing light rays to form the second light path;
the auxiliary focusing camera and the image projection assembly are arranged oppositely along the beam splitter assembly and used for shooting an overlapped image matched with a sample to be observed in the microscope visual field based on light rays in the second light path.
In the above scheme, the image projection module and the camera module operate using a time division multiplexing scheme.
In the above scheme, the microscope system further includes:
at least one output interface device coupled to the data processing unit of the microscope system to output a sharply focused image taken via a first optical path and an image of the sample to be observed subjected to image enhancement processing,
in the above solution, the objective lens includes at least one of:
an achromatic objective lens, a field semi-apochromatic objective lens, or a field apochromatic objective lens;
the beam splitter comprises at least one of:
cube beam splitters, plate beam splitters, or pellicle beam splitters.
The embodiment of the invention also provides an automatic focusing method of the microscope, which comprises the following steps:
acquiring a measurement sample shot by an auxiliary focusing camera in a second light path of the microscope;
calculating corresponding image evaluation parameters according to the measurement sample shot by the focusing-assisted camera and the corresponding image evaluation standard;
according to the image evaluation parameters, the relation between the image evaluation parameters and the defocusing amount is searched in a pre-stored calibration curve, and then the required defocusing amount is determined
And adjusting the focal length of the image light rays entering the camera assembly according to the determined defocus amount to realize that the camera assembly shoots a clearly focused image through the first optical path.
In the foregoing scheme, the acquiring a measurement sample taken by an auxiliary focusing camera in a second optical path of a microscope includes:
collecting light rays in the second light path through the auxiliary focusing camera;
and processing the collected light rays in the second light path based on the type of the auxiliary focusing camera so as to realize the shooting of the overlapped image matched with the sample to be observed in the microscope visual field.
In the above scheme, the method further comprises:
and based on the result of the focal length adjustment, photographing the sample to be observed in the field of view of the microscope through light rays in the first light path, and forming and outputting a clearly focused image photographed through the first light path.
An embodiment of the present invention further provides a medical device, including:
comprising a microscope system, a memory and a processor, the microscope system being the microscope system provided in the previous embodiment, the processor performing the steps of:
a memory for storing executable instructions;
and the processor is used for realizing the prior microscope automatic focusing method when the executable instructions stored in the memory are operated.
The embodiment of the invention also provides a computer readable storage medium, which stores executable instructions, and the executable instructions are executed by a processor to realize the microscope automatic focusing method.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention is used for acquiring light of a sample to be observed, entering a first light path through an objective lens, and converging the light with the light generated by an image projection module at a beam splitter after entering the first light path through a lens component; the beam splitter component comprises at least one beam splitter and is used for separating and projecting light rays in different light paths respectively; the lens assembly comprises at least one lens and is used for projecting light rays generated by an observed sample during observation after the light rays enter different optical paths through the objective lens so as to realize propagation of the light rays along different optical paths; the image projection component is arranged in a corresponding light path of the light projected by the lens component and is used for carrying out image enhancement processing on the image of the sample to be observed; the camera assembly is arranged in the first optical path and comprises a camera for photographing a sample to be observed in the field of view of the microscope so as to form and output a clearly focused image photographed through the first optical path; the auxiliary focusing device comprises an auxiliary focusing light source and an auxiliary focusing camera, is arranged in the second light path and is used for determining a focal length matched with the camera component; a focusing device configured to adjust a focal length of image light entering the camera assembly according to the defocus amount of the sample image to be measured determined by the auxiliary focusing device. Therefore, the focusing device can automatically focus the camera assembly of the microscope system, and a clearly focused image shot through the first light path is formed and output, so that the focusing time of the microscope system is saved, and the focusing accuracy is improved.
Drawings
FIG. 1 is a schematic diagram of an environment for using an auto-focusing method for a microscope according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a component structure of a medical device according to an embodiment of the present invention;
FIG. 3 is an alternative configuration of a microscope system in accordance with embodiments of the present invention;
FIG. 4 is a schematic process diagram of an alternative microscope auto-focusing method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an alternative configuration of a microscope system provided by an embodiment of the present invention;
FIG. 6 is a schematic diagram of an alternative configuration of a microscope system provided by an embodiment of the present invention;
FIG. 7 is a schematic diagram of an alternative configuration of a microscope system provided by an embodiment of the present invention;
FIG. 8 is a diagram illustrating the relationship between defocus and distance between ghosts in an embodiment of the present invention;
FIG. 9 is a diagram illustrating the relationship between defocus and inter-ghost distance.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail with reference to the accompanying drawings, the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.
Before further detailed description of the embodiments of the present invention, terms and expressions mentioned in the embodiments of the present invention are explained, and the terms and expressions mentioned in the embodiments of the present invention are applied to the following explanations.
1) In response to the condition or state on which the performed operation depends, one or more of the performed operations may be in real-time or may have a set delay when the dependent condition or state is satisfied; there is no restriction on the order of execution of the operations performed unless otherwise specified.
2) Terminals, including but not limited to: the system comprises a common terminal and a special terminal, wherein the common terminal is in long connection and/or short connection with a sending channel, and the special terminal is in long connection with the sending channel.
3) The client, the carrier in the terminal implementing the specific function, such as mobile client (APP), is the carrier of the specific function in the mobile terminal, such as executing the function of paying for consumption or purchasing a financial product.
4) The lens assembly, the arrangement of at least one lens combination, may be provided with a lens barrel for viewing a magnified optical image of an object such as a cell.
5) A field of view, the range that can be observed when viewing a magnified image of the cells in the smear through the lens assembly.
6) In Computer Aided Diagnosis (AD Computer Aided Diagnosis), CAD is used to assist in finding lesions and improving the accuracy of Diagnosis by imaging, medical image processing techniques, and other possible physiological and biochemical means, combined with Computer analysis and calculation.
Referring to fig. 1, fig. 1 is a schematic view of a use scenario of the microscope auto-focusing method provided by the embodiment of the present invention, referring to fig. 1, a terminal (including a terminal 10-1 and a terminal 10-2) is provided with corresponding clients capable of performing different functions, where the clients are terminals (including a terminal 10-1 and a terminal 10-2) that acquire different slice images from a corresponding server 200 through a network 300 for browsing, the terminal is connected to the server 200 through the network 300, the network 300 may be a wide area network or a local area network, or a combination thereof, and data transmission is achieved by using a wireless link, where types of the slice images acquired by the terminals (including the terminal 10-1 and the terminal 10-2) from the corresponding server 200 through the network 300 may be the same or may be the same In contrast, for example: the terminals (including the terminal 10-1 and the terminal 10-2) can acquire pathological images or pathological videos matched with the target object from the corresponding server 200 through the network 300, and can also acquire pathological sections only matched with the current target from the corresponding server 200 through the network 300 for browsing. The server 200 may store slice images corresponding to different target objects, or may store auxiliary analysis information corresponding to the slice images of the target objects.
The neural network model in the artificial intelligence field deployed by the server can acquire images of a sample to be observed by using a camera on a traditional optical microscope, and the images are analyzed in real time by combining a machine learning algorithm. Artificial Intelligence (AI intellectual Intelligence) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results.
In particular, artificial intelligence is an integrated technique in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
It should be noted that the lesion of the patient viewed under the microscope system (the medical device in contact with the pathological cell section of the target object) may include a plurality of different application scenarios, such as screening different cell sections for lung cancer cells, early screening for cervical cancer, and the like. The microscope system image processing method based on the embodiment can be deployed to various application scenes, so that remote reference and use of doctors are facilitated.
The server 200 transmits the pathological information of the same target object to the terminal (terminal 10-1 and/or terminal 10-2) through the network 300 to enable the user of the terminal (terminal 10-1 and/or terminal 10-2) to analyze the pathological information of the target object, thereby. As an example, the server 200 deploys a corresponding neural network model for analyzing the clear image information output by the microscope system, wherein the image acquisition by the microscope system can be realized by: acquiring a measurement sample shot by an auxiliary focusing camera in a second light path of the microscope; calculating corresponding image evaluation parameters according to the measurement sample shot by the focusing-assisted camera and the corresponding image evaluation standard; according to the image evaluation parameters, searching the relation between the image evaluation parameters and the defocusing amount in a pre-stored calibration curve, and further determining the required defocusing amount; and adjusting the focal length of the image light rays entering the camera assembly according to the determined defocus amount to realize that the camera assembly shoots a clearly focused image through the first optical path.
And photographing a sample to be observed in the microscope visual field based on the result of the focal length adjustment, and forming and outputting a clearly focused image photographed through a first light path.
As will be described in detail below, the medical device according to the embodiment of the present invention may be implemented in various forms, such as a dedicated terminal with a microscope system image processing function, or a medical device with a microscope system image processing function or a cloud server, for example, the server 200 in fig. 1. Fig. 2 is a schematic diagram of a component structure of a medical device according to an embodiment of the present invention, and it is to be understood that fig. 2 only shows an exemplary structure of the medical device and not a whole structure thereof, and a part of or the whole structure shown in fig. 2 may be implemented as needed.
The medical equipment provided by the embodiment of the invention comprises: at least one processor 201, memory 202, user interface 203, and at least one network interface 204. The various components in the medical device 20 are coupled together by a bus system 205. It will be appreciated that the bus system 205 is used to enable communications among the components. The bus system 205 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 205 in fig. 2.
The user interface 203 may include, among other things, a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touch pad, or a touch screen.
It will be appreciated that the memory 202 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The memory 202 in embodiments of the present invention is capable of storing data to support operation of the terminal (e.g., 10-1). Examples of such data include: any computer program, such as an operating system and application programs, for operating on a terminal (e.g., 10-1). The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic services and processing hardware-based tasks. The application program may include various application programs.
For example, the processor in the form of a hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable logic devices (P L D, Programmable L analog devices), Complex Programmable logic devices (CP L D, Complex Pr analog L analog devices), Field Programmable Gate Arrays (FPGAs), or other electronic components.
As an example of the microscope system implemented by combining software and hardware, the microscope system provided by the embodiment of the present invention may be directly embodied as a combination of software modules executed by the processor 201, where the software modules may be located in a storage medium located in the memory 202, and the processor 201 reads executable instructions included in the software modules in the memory 202, and completes the image processing method of the microscope system provided by the embodiment of the present invention in combination with necessary hardware (for example, including the processor 201 and other components connected to the bus 205).
By way of example, the Processor 201 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor or the like.
As an example of the microscope system provided by the embodiment of the present invention implemented by hardware, the apparatus provided by the embodiment of the present invention may be implemented by directly using the processor 201 in the form of a hardware decoding processor, for example, by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable logic devices (P L D, Programmable L) complex Programmable logic devices (CP L D, complex Programmable L) Field Programmable Gate Arrays (FPGAs), or other electronic components, to implement the image processing method of the microscope system provided by the embodiment of the present invention.
In other embodiments, the microscope system provided by the embodiment of the present invention may be implemented in software, and fig. 2 shows the microscope system 2020 stored in the memory 202, which may be software in the form of programs and plug-ins, and includes a series of modules, and as examples of the programs stored in the memory 202, the microscope system 2020 may be included, and the microscope system 2020 includes the following software modules:
the information processing module 2081 is used for acquiring a measurement sample shot by an auxiliary focusing camera in a second optical path of the microscope; calculating corresponding image evaluation parameters according to the measurement sample shot by the focusing-assisted camera and the corresponding image evaluation standard; according to the image evaluation parameters, searching the relation between the image evaluation parameters and the defocusing amount in a pre-stored calibration curve, and further determining the required defocusing amount; and adjusting the focal length of the image light rays entering the camera assembly according to the determined defocus amount to realize that the camera assembly shoots a clearly focused image through the first optical path.
Before describing the microscope auto-focusing method provided by the present invention, a microscope focusing process in the related art will be described, wherein, referring to fig. 3, fig. 3 is an alternative structure of a microscope system in the related art of the present invention, and in the related embodiment, a microscope 300 is provided, the microscope 300 has a microscope body 301, a microscope body stage focusing knob 302, a microscope body stage 303, a sample 304 to be observed by the microscope, a body objective 305, a eyepiece barrel 306, a camera 307, and an eyepiece 308. Wherein, microscope body 301 top is provided with microscope objective table 303, places on the microscope body objective table 303 and remains to observe sample 304, microscope body 301 both sides are equipped with microscope body objective table focusing knob 302, microscope body objective 305 is located the top of microscope body objective table 303, still is equipped with trinocular lens section 306 in the top of microscope body objective 305, and trinocular lens 306 head is connected with camera 307 and eyepiece 308 respectively. Adjusting the microscope body stage focusing knob 302 can adjust the microscope body stage 303 to be raised or lowered in the vertical direction, thereby changing the distance between the microscope body stage 303 and the microscope body objective lens 305 to achieve focusing. Of course, it is also possible to move the microscope body objective lens 305 so as to change the distance between the microscope body stage 303 and the microscope body objective lens 305 to achieve focusing.
The focusing of the microscope 300 is assumed to be in alignment with the end of the eyepiece 308 and the end of the camera 307 of the eyepiece barrel 306 of the microscope 300. However, the image of the camera 307 is out of focus with the image of the eyepiece 308 end due to the limitations of the focusing technique. For example: the parfocal of the objective lens of different multiples is not well adjusted, the diopters of the eyes of different microscope 300 users are different, and when the microscope 300 users are exchanged, a new user does not have the awareness of adjusting the diopter knob of the eyepiece 308 and directly adjusts the objective table to refocus the sample. These reasons can cause the images at the eyepiece 308 end and the camera 307 end of the microscope 300 to be out of focus, so that when the human eye sees a clear image, the image acquired by the camera 307 is out of focus, and the correctness of the analysis result of the image algorithm cannot be ensured. In the field of automatic analysis of microscope images, a camera can acquire high-quality images, and the accuracy of the algorithm of the augmented reality microscope is guaranteed. Many important optical information can be lost in the defocused image of the sample, and even a later algorithm with strong computing power cannot compensate the important optical information. Therefore, it is important to ensure that the camera can acquire an accurately focused image of the sample.
In order to overcome the above-mentioned drawbacks, referring to fig. 4, fig. 4 is an alternative process diagram of an automatic microscope focusing method according to an embodiment of the present invention, where the automatic microscope focusing method includes the following steps:
step 401: acquiring a measurement sample shot by an auxiliary focusing camera in a second light path of the microscope;
step 402: and calculating corresponding image evaluation parameters according to the measurement sample shot by the focusing-assisted camera and the corresponding image evaluation standard.
The image evaluation criterion may be an offset of a relative pixel of an image ghost acquired by the focusing-assisted camera.
Step 403: according to the image evaluation parameters, searching the relation between the image evaluation parameters and the defocusing amount in a pre-stored calibration curve, and further determining the required defocusing amount;
the calibration curve can be pre-stored in a corresponding storage medium, so that the auto-focusing function of the augmented reality microscope can be realized by calling the calibration curve, wherein the pre-stored calibration curve is a curve determined according to pre-acquired images corresponding to different defocus amounts and corresponding image evaluation criteria (such as offset of relative pixels of image ghosts acquired by an auxiliary focusing camera), so that the relationship between the defocus amount and the image evaluation parameters of different defocus degrees can be determined by searching the calibration curve.
Step 404: and adjusting the focal length of the image light rays entering the camera assembly according to the determined defocus amount so as to realize that the camera assembly shoots a clearly focused image through the first optical path.
According to the automatic focusing method provided by the implementation, when the focal length of the image light entering the camera assembly is adjusted according to the determined defocusing amount, no matter whether the image to be observed presented in the eyepiece is clear or not, the camera assembly can shoot the clear focused image through the first light path, and the defects of low manual focusing speed and poor precision in the prior art are overcome.
The Microscope system with different forms is different from the Microscope shown in fig. 3 in structure, for example, an Augmented Reality Microscope (ARM advanced Reality Microscope) can conveniently and accurately observe a sample to be observed under the Microscope and simultaneously acquire other Augmented information, thereby helping an observer to quickly locate and quantify an interested feature. Only taking the application to a medical diagnosis scene as an example, when a doctor observes a slice by using an augmented reality microscope, the doctor can simultaneously obtain a diagnosis result based on the slice, namely, the augmented reality microscope can superpose the diagnosis result on the slice as augmented reality information, so that the doctor can conveniently read a conclusion in a field of view in real time, and meanwhile, a neural network model running in a server can make auxiliary diagnosis and treatment judgment on a diseased area so as to help the doctor make correct judgment on pathological information of a focus.
Continuing with the description of the structure of the microscope system provided in the present application with reference to fig. 5, fig. 5 is an optional structural schematic diagram of the microscope system provided in the embodiment of the present invention, wherein the microscope system 1100 specifically includes an objective lens 115, a beam splitter 1112, an image projection assembly 1111, a camera assembly 117, and a eyepiece barrel 116, the objective lens 115 has a first end 10a and a second end 10b opposite to each other, the first end 10a faces a sample to be observed, the beam splitter 1112 is disposed at the second end 10b, the beam splitter 1112 is respectively communicated with the objective lens 115 and a tube lens 1118 of the eyepiece barrel, the camera assembly 117 is disposed in one of the passages of the eyepiece barrel, wherein the image projection assembly 1111 projects an image in a corresponding field of view through light transmitted in a lens 1115, the camera assembly 117 receives light transmitted by the tube lens 1118, the camera assembly 117 includes a camera and a corresponding image output device, and is used for transmitting the shot image in the corresponding field of view to a server for processing or recognizing the image, the three-eye lens barrel 116 is disposed at one end of the beam splitter 1112 far away from the objective lens 10, the three-eye lens barrel 116 includes at least two channels and a tube lens 1118, the channels are located at one end far away from the beam splitter 1112, the tube lens 1118 is located at one end close to the beam splitter 1112, and the camera assembly 117 receives light output by the beam splitter 1112 through the tube lens 1118 to complete the collection of the image in the corresponding field of view.
In this process, since the camera module 117 and the image projection module 1111 are disposed at different positions, in order to avoid the influence of the light during the propagation process, the image projection module 1111 further includes a first polarizer 1116, and the first polarizer 1116 is located between the tube mirror 1118 and the beam splitter 1112, and is used for performing polarization processing on the corresponding light in the first light path; the camera assembly also includes a second polarizer 1117, and the second polarizer 1117 is positioned between the tube mirror 1118 and the camera assembly 117 for polarizing the corresponding light collected by the camera assembly 117.
Further, the optical path of the microscope system 1100 is: the light from the objective 115 is transmitted to the beam splitter 1112, the beam splitter 1112 reflects a portion of the light to the tube lens 1118 and onto the photosensitive chip of the camera module 117 through the first polarizer 1116, at the same time, the beam splitter 1112 transmits a portion of the light to the tube lens 1118, passes through the tube lens 1118 and reflects the transmitted light through the tube lens 1118 to the lens barrel 116, the lens barrel 116 transmits the light to the eyepiece 118, an image of the sample 114 to be observed can be observed through the eyepiece 118, and at the same time, the light generated by the image projection module 1111 is polarized along the lens 1115 by the second polarizer 1117 and cannot reach the camera module 117 through the beam splitter 1112 and the first polarizer 1116, and the shooting of the camera module 117 is not affected.
However, in this process, the microscope observers with different diopters need to repeat the tedious three-eye parfocal adjustment each time when the microscope system is used interchangeably. The camera cannot autonomously complete automatic focusing and cannot acquire clear images. Meanwhile, when a microscope user sees a clear image through the eyepiece, the camera acquires an out-of-focus image, so that the correctness of an image algorithm analysis result executed in the server cannot be ensured.
To solve the above problem, further, referring to fig. 6, fig. 6 is an alternative structural schematic diagram of a microscope system provided by an embodiment of the present invention, wherein the microscope system 600 specifically includes an objective lens 115, a beam splitter 1112, an image projection assembly 1111, a camera assembly 117, and a eyepiece barrel 116, the objective lens 115 has a first end 10a and a second end 10b opposite to each other, the first end 10a faces a sample to be observed, the beam splitter 1112 is disposed at the second end 10b, the beam splitter 1112 is respectively communicated with the objective lens 115 and a tube 1118 of the eyepiece barrel, the camera assembly 117 is disposed in one of channels of the eyepiece barrel, wherein the image projection assembly 1111 projects an image in a corresponding field of view through light transmitted in a lens 1115, the camera assembly 117 receives light transmitted by the tube 1118, the camera assembly 117 includes a camera and a corresponding image output device, and is used for transmitting the captured image in the corresponding field of view to a server for processing or recognizing the image, the trinocular lens 116 is disposed at an end of the beam splitter 1112 far away from the objective lens 10, the trinocular lens 116 includes at least two channels and a tube lens 1118, the channels are located at an end far away from the beam splitter 1112, the tube lens 1118 is located at an end near the beam splitter 1112, and the camera assembly 117 receives the light output by the beam splitter 1112 through the tube lens 1118 to complete the collection of the image in the corresponding field of view. Since the camera module 117 is disposed at a different position from the image projection module 1111 to avoid the influence of the light during the propagation process, the image projection module 1111 further comprises a second polarizer 1117, and the second polarizer 1117 is disposed between the lens 1115 and the beam splitter 1112 for polarization processing of the corresponding light in the first optical path; the camera module further includes a first polarizer 1116 between the tube lens 1118 and the camera module 117 for polarizing the corresponding light collected by the camera module 117, and specifically, since the naked eye of the microscope operator desires to observe both the image of the sample 114 to be observed and the image outputted from the image projection module 1111 through the eyepiece 118, the camera module 117 desires to photograph only the image of the sample 114 to be observed while ignoring the image outputted from the image projection module 1111, the light outputted from the image projection module 1111 may be changed into polarized light by the second polarizer 1117 and may directly reach the human eye through the eyepiece 118 for observation, but since the polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other, the polarized light outputted from the second polarizer 1117 may be eliminated by the first polarizer 1116, so that the camera assembly 117 can only take images of the sample 114 to be observed.
In some embodiments of the invention, the objective lens comprises at least one of:
an achromatic objective, a field semi-apochromatic objective, or a field apochromatic objective, said beam splitter comprising at least one of: cube beam splitters, plate beam splitters, or pellicle beam splitters. In particular, in view of the fact that there may be different magnification requirements when performing object observations, for example different magnification requirements for the same observation, such as the contour and kernel of a cell, or different magnification requirements for different size observations, combinations of objective lenses with different magnification may also be provided for selection by the user. For example, combinations of objective lenses with magnifications of 4.0X, 10.0X, 20.0X, 60.0X and 100.0X may be provided for user selection. Meanwhile, the cube beam splitter, the flat plate beam splitter or the film beam splitter can be selectively adapted according to the type of the augmented reality microscope so as to adapt to different use environments.
Meanwhile, in order to realize automatic focusing, the microscope system 1100 further includes an auxiliary focusing light source 1140 disposed in the corresponding fourier back focal plane of the condenser lens assembly 1141 for emitting auxiliary focusing light, wherein the auxiliary focusing light source 1140 can be two identical infrared L ED light emitters, and the auxiliary focusing camera 1143 is disposed at an axially offset position of the conjugate plane of the camera assembly 117, and a second light path formed by the light generated by the auxiliary focusing light source 1140 is as shown in 1144.
Further, the microscope system 1100 includes an optical path: the microscope comprises a first light path and a second light path, wherein the first light path is used for projecting light generated by an observed sample when the observed sample is observed after the light enters the light path through the objective lens, so that the camera component photographs the sample to be observed in the field of the microscope to form and output a clearly focused image photographed through the first light path, and meanwhile, the image projection component can perform image enhancement processing on the image of the sample to be observed by utilizing the light in the second light path.
Wherein, first light path includes: the light from objective lens 115 is transmitted to beam splitter 1112, and beam splitter 1112 reflects a portion of the light to tube lens 1118 and passes through first lens 1116 and onto the photo-sensing chip of camera assembly 117, at the same time, beam splitter 1112 transmits a portion of the light to tube lens 1118, passes through tube lens 1118 and reflects the transmitted light through tube lens 1118 to lens barrel 116, lens barrel 116 transmits the light to eyepiece 118, an image of sample 114 to be observed can be viewed through eyepiece 118, meanwhile, the light generated from the image projection unit 1111 passes through the polarization process of the second polarizer 1117 along the lens 1115, and through the action of the beam splitter 1112, and the first polarizer 1116, does not reach the camera assembly 117, so that the camera assembly 117 can only take an image of the sample 114 to be observed, without affecting the taking of the camera assembly 117, wherein the polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other. Before the camera assembly 117 captures an image in the corresponding field of view, the focal length of the camera assembly 117 needs to be adjusted first, and specifically, the corresponding defocus parameter can be determined from the image in the second optical path.
The second optical path includes the light from the secondary focusing light source 1140 in the fourier back focal plane passing through the object 1185 to the beam splitter 1112, the beam splitter 1112 transmits the light to the tube mirror 1118, and the infrared light finally emitted by the infrared L ED as the secondary focusing light source is transmitted to the secondary focusing camera 1143 through the tube mirror 1118 and then imaged at the secondary focusing camera 1143 (partially overlapped image).
Further, in some embodiments of the present invention, a focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117, and is configured to drive the first lens for focus adjustment based on a focus determined by the defocus amount of the overlapped image, so as to form a new focus. In particular, the focal device 1121 may be an electric motor such as an ultrasonic drive motor or other mechanical motor that may be used to drive the lens group accordingly; the liquid zoom lens can also be a liquid zoom lens which performs liquid zooming independently of the lens group so as to adapt to different use environments.
In some embodiments of the present invention, a focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117 for performing a focal length adjustment through a variable focus lens based on a focal length determined by the defocus amount of the overlapped image, so as to form the new focal length; and the camera component 117 is used for photographing a sample to be observed in the microscope visual field based on the new focal length to form and output a clearly focused image shot through the first optical path.
In some embodiments of the present invention, in consideration of the fact that the camera interface may not be uniform, in order to be compatible with multiple cameras, or in order to expand or reduce the field of view, the camera may be used in cooperation with a camera adapter, and the camera based on the photosensitive chip is connected to the camera interface at the top end of the binocular viewing tube through the camera adapter, so as to achieve connection between the camera and the binocular viewing tube. The camera adapter can be further internally embedded with a polaroid, and the embedded polaroid can filter light rays with polarization states perpendicular to the polaroid and avoid interference with imaging.
Referring to fig. 7, fig. 7 is an alternative structural schematic diagram of a microscope system provided by an embodiment of the present invention, wherein the microscope system 700 specifically includes an objective lens 115, a beam splitter 1112, an image projection assembly 1111, a camera assembly 117, and a eyepiece barrel 116, the objective lens 115 has a first end 10a and a second end 10b oppositely disposed, the first end 10a faces a sample to be observed, the beam splitter 1112 is disposed at the second end 10b, the beam splitter 1112 is respectively communicated with the objective lens 115 and a tube lens 1118 of the eyepiece barrel, the camera assembly 117 is disposed in one of channels of the eyepiece barrel, wherein the image projection assembly 1111 projects an image in a corresponding field of view through light transmitted in a lens 1115, the camera assembly 117 receives light transmitted by the tube lens 1118, the camera assembly 117 includes a camera and a corresponding image output device, for transmitting the captured images in the corresponding fields of view to a server for processing or recognizing the images, the lens barrel 116 is disposed at an end of the beam splitter 1112 far away from the objective lens 10, the lens barrel 116 includes at least two channels and a tube lens 1118, the channels are located at an end far away from the beam splitter 1112, the tube lens 1118 is located at an end near the beam splitter 1112, and the camera assembly 117 receives the light output by the beam splitter 1112 through the tube lens 1118 to complete the capturing of the images in the corresponding fields of view. Since the camera module 117 is disposed at a different position from the image projection module 1111 to avoid the influence of the light during the propagation process, the image projection module 1111 further comprises a second polarizer 1117, and the second polarizer 1117 is disposed between the lens 1115 and the beam splitter 1112 for polarization processing of the corresponding light in the first optical path; the camera assembly also includes a first polarizer 1116, which is positioned between the tube lens 1118 and the camera assembly 117 for polarizing the corresponding light collected by the camera assembly 117. Specifically, since the naked eye of the microscope operator desires to observe both the image of the sample 114 to be observed and the image outputted from the image projection unit 1111 through the eyepiece 118, and at the same time, the camera unit 117 desires to photograph only the image of the sample 114 to be observed while ignoring the image outputted from the image projection unit 1111, the light outputted from the image projection unit 1111 can be converted into polarized light by the second polarizing plate 1117 and can be directly transmitted to the human eye through the eyepiece 118 for observation, but since the polarization directions of the first polarizing plate 1116 and the second polarizing plate 1117 are perpendicular to each other, the polarized light outputted from the second polarizing plate 1117 can be eliminated by the first polarizing plate 1116, so that the camera unit 117 can photograph only the image of the sample 114 to be observed.
Meanwhile, in order to realize automatic focusing, the microscope system 1100 further includes an auxiliary focusing light source 1140 disposed in the corresponding fourier back focal plane of the condenser lens assembly 1141 for emitting auxiliary focusing light, wherein the auxiliary focusing light source 1140 can be two identical infrared L ED light emitters, the auxiliary focusing camera 1143 is disposed at a horizontally symmetrical position of the image projection assembly 1111, and receives, through the lens 1119, light rays in the second light path refracted by the beam splitter 112, and a second light path formed by the light rays generated by the auxiliary focusing light source 1140 is as shown in 1144.
Further, the microscope system 1100 includes an optical path: a first optical path and a second optical path, wherein the first optical path comprises: the light from the objective 115 is transmitted to the beam splitter 1112, the beam splitter 1112 reflects a portion of the light to the tube lens 1118, passes through the first lens 1116, and transmits to the photosensitive chip of the camera module 117, and at the same time, the beam splitter 1112 transmits a portion of the light to the tube lens 1118, passes through the tube lens 1118, reflects the transmitted light through the tube lens 1118 to the lens barrel 116, the lens barrel 116 transmits the light to the eyepiece 118, an image of the sample 114 to be observed can be observed through the eyepiece 118, and at the same time, the light generated by the image projection module 1111 is along the lens 1115, wherein the polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other, the light cannot reach the camera module 117 through the polarization process of the second polarizer 1117, and the beam splitter 1112 and the first polarizer 1116, so that the camera module 117 can only photograph the image of the sample 114 to be observed, and the light output by the image projection module 1111 does not affect the photographing of the camera module 117, at the same time, the naked eye of the microscope operator desires to observe both the image of the sample 114 to be observed and the image outputted in the image projection unit 1111 through the eyepiece 118. Before the camera assembly 57 captures an image in the corresponding field of view, the focal length of the camera assembly 117 needs to be adjusted, and the corresponding defocus parameter can be determined from the image in the second optical path.
The second optical path includes the object 1185 through which the light of the secondary focusing light source 1140 in the fourier back focal plane reaches the beam splitter 1112, and the beam splitter 1112 transmits the light to the lens 1119, and the infrared light finally emitted by the infrared L ED as the secondary focusing light source is transmitted to the secondary focusing camera 1143 through the lens 1119 and then imaged at the secondary focusing camera 1143 (partially overlapped image).
Further, in some embodiments of the present invention, a focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117, and is configured to drive the first lens for focus adjustment based on a focus determined by the defocus amount of the overlapped image, so as to form a new focus.
In some embodiments of the present invention, a focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117 for performing a focal length adjustment through a variable focus lens based on a focal length determined by the defocus amount of the overlapped image, so as to form the new focal length; and the camera component 117 is used for photographing a sample to be observed in the microscope visual field based on the new focal length to form and output a clearly focused image shot through the first optical path.
The microscope auto-focusing method provided by the present invention is further described with reference to the method shown in fig. 3 and the different microscope system configurations shown in fig. 5 to 7, wherein, with reference to fig. 8, fig. 8 is a schematic diagram illustrating the relationship between the defocus amount and the distance between the ghost images in the embodiment of the present invention, wherein the infrared light emitted by the infrared L E D as the auxiliary focusing light source is transmitted to the auxiliary focusing camera 1143 and then imaged (partially overlapped images) at the auxiliary focusing camera 1143, wherein the auxiliary focusing camera may be a general industrial camera with the infrared filter removed, or a special infrared camera.
Under the condition of different defocus amounts, the peak position of an autocorrelation result obtained after autocorrelation operation is performed on an image acquired by the auxiliary focusing camera changes.
Can be derived from the theory shown below:
suppose that the image collected by the auxiliary focusing camera is z [ x ]]＝s[x]+s[x-x0]Wherein s [ x ]]And s [ x-x ]0]Is spaced apart by a distance x0Two ghosts of (a). z [ x ]]It can also be expressed in this form: z [ x ]]＝s[x]*h[x]. Where '. prime' represents the convolution symbol, h [ x ]]＝[x]+[x-x0]。
By pairing z [ x ]]Performing autocorrelation to obtain R (z [ x ]])＝R(s[x])*R(h[x])＝R(s[x])*(2[x]+[x-x0]+[x+x0]). Wherein 'R ()' represents an autocorrelation operation symbol. 2[ x ]]+[x-x0]+[x+x0]Three functions are represented. R (s [ x ]])*(2[x]+[x-x0]+[x+x0]) Represents R (s [ x ]]) And the convolution of three functions. That means that R (z [ x ]]) Three spikes are formed by arithmetic operation in the result of (1)Peak(s). One peak is positioned at the highest position in the middle, and the other two peaks are positioned at two sides of the highest peak and are respectively away from the peak by a distance x0. This also means that x can be obtained if the distance between any two of the three peaks formed by the algorithm can be determined0I.e. the distance between two ghosts acquired by the camera.
The microscope autofocus method provided by the present invention is further illustrated by taking the mouse kidney slice observed through the microscope as an example, wherein fig. 5 shows a sample ghost formed by two infrared rays L ED40 on the surface of the focusing-assisted camera (a 20-fold objective lens of the mouse kidney slice is shown in the figure).
Continuing to refer to FIG. 8, wherein: (a1) the (b1) and (c1) show in-focus images in the corresponding visual fields acquired by the auxiliary focus camera when the mouse kidney slices are at different defocusing amounts. (a2) The results of the autocorrelation operations (a1), (b1), and (c1) are shown as (b2) and (c2), respectively.
Fig. 8 (a1) - (c1) respectively show two ghost-containing infrared images captured by the focusing-assist camera in the microscope system according to the previous embodiment. When the sample is at different defocus amounts, the distance between the two ghosts will be different. The relationship curve of the sample defocusing amount and the distance between two ghosts in the image is obtained by performing some operations (including but not limited to autocorrelation operations) on the image acquired by the auxiliary focusing camera. So that the fitted curve is used as a reference table for the focusing process to realize automatic focusing.
With continuing reference to fig. 9, fig. 9 is a schematic diagram of the relationship between defocus and inter-ghost distances in the present invention, wherein a curve fitted according to the relationship between defocus and inter-ghost distances is a schematic diagram of fig. 9, and a curve is fitted according to the relationship between sample defocus and ghost of an image acquired by an auxiliary focus camera.
The reason why the curves in fig. 9 show a monotonically increasing trend is that the auxiliary focus camera is set with a certain offset with respect to the camera assembly. The offset in this figure is 60 microns, as is the sixth most central point in the figure. Fig. 9 shows a curve fitted from 11 ghost images acquired from a sample at-30 microns (30 microns in fig. 9, i.e., the first point) to +30 microns (90 microns in fig. 7, i.e., the 11 th point). The reason for setting the bias is because the fitted curve theoretically approaches a "V" shape if no bias is set. However, near sample focus, the distance of the two ghosts can be very close. According to the autocorrelation calculation method described above, the three peaks are also very close in distance, resulting in the values of the peaks being overwhelmed or new unexpected uncorrelated peaks. This is not conducive to finding the peak location that needs to be determined. The bias is introduced here in order to pull apart the distance between the three peaks.
It should be noted that different index information values are fit to a curve, where the vertex of the curve is the corresponding position when the defocus amount is zero. The closer the defocus is to zero, the sharper the image is, i.e. the position to which the focusing device needs to be adjusted to drive the camera.
The beneficial technical effects are as follows:
according to the embodiment of the invention, a measurement sample shot by an auxiliary focusing camera in a second optical path of a microscope is obtained; calculating corresponding image evaluation parameters according to the measurement sample shot by the focusing-assisted camera and the corresponding image evaluation standard; according to the image evaluation parameters, searching the relation between the image evaluation parameters and the defocusing amount in a pre-stored calibration curve, and further determining the required defocusing amount; and adjusting the focal length of the image light rays entering the camera assembly according to the determined defocus amount to realize that the camera assembly shoots a clearly focused image through the first optical path. Therefore, the focusing device can automatically focus the camera assembly of the microscope system, and a clearly focused image shot through the first light path is formed and output, so that the focusing time of the microscope system is saved, and the focusing accuracy is improved.
The above description is only exemplary of the present invention and should not be taken as limiting the scope of the present invention, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (15)
1. A microscope system, characterized in that the microscope system comprises:
the objective lens is used for acquiring light rays of a sample to be observed, entering a first light path, and converging the light rays with the light rays generated by the image projection module at the beam splitter after entering the first light path through the lens component;
the beam splitter component comprises at least one beam splitter and is used for separating and projecting light rays in different light paths respectively;
a camera assembly disposed in the first optical path, the camera assembly including a camera for taking a picture of a sample to be observed in the field of view of the microscope to form and output a sharply focused image taken through the first optical path;
the auxiliary focusing device comprises an auxiliary focusing light source and an auxiliary focusing camera, is arranged in the second light path and is used for determining a focal length matched with the camera component;
and the focusing device is configured to adjust the focal length of the image light entering the camera assembly according to the defocusing amount of the sample image to be detected determined by the auxiliary focusing device.
2. The microscope system of claim 1, further comprising:
the lens assembly comprises at least one lens and is used for projecting light rays generated by an observed sample during observation after the light rays enter different optical paths through the objective lens so as to realize propagation of the light rays along different optical paths;
the eyepiece is sleeved with the trinocular lens barrel and used for observing a sample to be observed through the objective lens;
the image projection component is arranged in a corresponding light path of the light projected by the lens component and is used for carrying out image enhancement processing on the image of the sample to be observed;
the eyepiece barrel is arranged at one end, far away from the objective, of the beam splitter and comprises at least two channels and a tube lens, wherein the channels are located at one end, far away from the beam splitter, of the channels, one of the channels is communicated with the eyepiece, and the tube lens is located at one end, close to the beam splitter.
3. The microscope system of claim 1, wherein the focusing device comprises a moving drive assembly and a variable focus lens to enable a sample to be observed in the field of view of the microscope to be photographed at different focal lengths.
4. The microscope system of claim 2,
the beam splitter assembly is respectively communicated with the objective lens and the tube lens of the lens barrel, and the camera assembly is arranged in one channel of the lens barrel;
the beam splitter assembly comprises a beam splitter, and the lens assembly comprises a lens and is arranged between the beam splitter and the image projection assembly;
the focusing device is positioned between the beam splitter and the camera assembly and used for adjusting the focal length of the image light rays entering the camera assembly according to the defocusing amount of the sample image to be detected determined by the auxiliary focusing device.
5. The microscope system of claim 4,
the image projection assembly further comprises a first polarizer, wherein the first polarizer is positioned between the lens assembly and the beam splitter and is used for carrying out polarization processing on the corresponding light rays in the first light path;
the camera assembly further includes a second polarizer positioned between the focusing device and the beam splitter for polarizing the respective light rays collected by the camera assembly.
6. The microscope system of claim 4,
the auxiliary focusing light source is arranged in a Fourier back focal plane corresponding to a condenser lens assembly of the microscope system and used for emitting auxiliary focusing light rays to form the second light path;
the beam splitter assembly comprises a beam splitter arranged between the focusing device and the camera assembly and used for reflecting the light rays in the second light path to the auxiliary focusing camera;
the auxiliary focusing camera is arranged at an axial offset position of a conjugate plane of the camera assembly and is used for shooting an overlapped image matched with a sample to be observed in the microscope visual field based on light rays in the second light path.
7. The microscope system of claim 4,
the auxiliary focusing light source is arranged in a Fourier back focal plane corresponding to a condenser lens assembly of the microscope system and used for emitting auxiliary focusing light rays to form the second light path;
the auxiliary focusing camera and the image projection assembly are arranged oppositely along the beam splitter assembly and used for shooting an overlapped image matched with a sample to be observed in the microscope visual field based on light rays in the second light path.
8. The microscope system of claim 1, wherein the image projection assembly and the camera assembly operate in a time division multiplexed scheme.
9. The microscope system of claim 1, further comprising:
at least one output interface device coupled to the data processing unit of the microscope system to output a sharply focused image taken via the first optical path and an image of the sample to be observed subjected to image enhancement.
10. The microscope system of claim 1, wherein the objective lens comprises at least one of:
an achromatic objective lens, a field semi-apochromatic objective lens, or a field apochromatic objective lens;
the beam splitter comprises at least one of:
cube beam splitters, plate beam splitters, or pellicle beam splitters.
11. A method for automatic focusing of a microscope, the method comprising:
acquiring a measurement sample shot by an auxiliary focusing camera in a second light path of the microscope;
calculating corresponding image evaluation parameters according to the measurement sample shot by the focusing-assisted camera and the corresponding image evaluation standard;
according to the image evaluation parameters, searching the relation between the image evaluation parameters and the defocusing amount in a pre-stored calibration curve, and further determining the required defocusing amount;
and adjusting the focal length of the image light rays entering the camera assembly according to the determined defocus amount to realize that the camera assembly shoots a clearly focused image through the first optical path.
12. The method of claim 11, wherein the obtaining of the measurement sample taken by the focus-assisted camera in the second optical path of the microscope comprises:
collecting light rays in the second light path through the auxiliary focusing camera;
and processing the collected light rays in the second light path based on the type of the auxiliary focusing camera so as to realize the shooting of the overlapped image matched with the sample to be observed in the microscope visual field.
13. The method of claim 11, further comprising:
and based on the result of the focal length adjustment, photographing the sample to be observed in the field of view of the microscope through light rays in the first light path, and forming and outputting a clearly focused image photographed through the first light path.
14. A medical device, characterized in that it comprises:
comprising a microscope system, a memory and a processor, the microscope system being the microscope system of any one of claims 1 to 10, the processor performing the steps of:
a memory for storing executable instructions;
a processor for implementing the microscope autofocus method of any of claims 11 to 13 when executing the executable instructions stored in the memory.
15. A computer readable storage medium storing executable instructions which when executed by a processor implement the method of microscope auto-focus of any of claims 11 to 13.
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010284514.XA CN111443476B (en) | 2020-04-13 | 2020-04-13 | Microscope autofocus method, microscope system, medical device, and storage medium |
| CN202310437825.9A CN116430568A (en) | 2020-04-13 | 2020-04-13 | Microscope system, microscope auto-focusing method, and medical apparatus |
| PCT/CN2021/077828 WO2021208603A1 (en) | 2020-04-13 | 2021-02-25 | Microscope autofocusing method, microscope system, medical device, and storage medium |
| US17/745,571 US20220342195A1 (en) | 2020-04-13 | 2022-05-16 | Microscope automatic focusing method, microscope system, medical device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010284514.XA CN111443476B (en) | 2020-04-13 | 2020-04-13 | Microscope autofocus method, microscope system, medical device, and storage medium |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310437825.9A Division CN116430568A (en) | 2020-04-13 | 2020-04-13 | Microscope system, microscope auto-focusing method, and medical apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111443476A true CN111443476A (en) | 2020-07-24 |
| CN111443476B CN111443476B (en) | 2023-04-14 |
Family
ID=71651731
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010284514.XA Active CN111443476B (en) | 2020-04-13 | 2020-04-13 | Microscope autofocus method, microscope system, medical device, and storage medium |
| CN202310437825.9A Pending CN116430568A (en) | 2020-04-13 | 2020-04-13 | Microscope system, microscope auto-focusing method, and medical apparatus |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310437825.9A Pending CN116430568A (en) | 2020-04-13 | 2020-04-13 | Microscope system, microscope auto-focusing method, and medical apparatus |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US20220342195A1 (en) |
| CN (2) | CN111443476B (en) |
| WO (1) | WO2021208603A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021208603A1 (en) * | 2020-04-13 | 2021-10-21 | 腾讯科技（深圳）有限公司 | Microscope autofocusing method, microscope system, medical device, and storage medium |
| CN116095477A (en) * | 2022-08-16 | 2023-05-09 | 荣耀终端有限公司 | Focusing processing system, method, equipment and storage medium |
| CN116794822A (en) * | 2023-07-05 | 2023-09-22 | 苏州欧米特光电科技有限公司 | Microscope control system and method |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2024000239A (en) * | 2022-06-20 | 2024-01-05 | 株式会社エビデント | microscope system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104932092A (en) * | 2015-06-15 | 2015-09-23 | 上海交通大学 | Automatic focusing microscope based on eccentric beam method and focusing method thereof |
| CN107656364A (en) * | 2017-11-16 | 2018-02-02 | 宁波舜宇仪器有限公司 | A kind of micro imaging system and its real-time focusing method |
| CN108051897A (en) * | 2018-01-17 | 2018-05-18 | 宁波舜宇仪器有限公司 | A kind of micro imaging system and real-time focusing method |
| CN108254853A (en) * | 2018-01-17 | 2018-07-06 | 宁波舜宇仪器有限公司 | A kind of micro imaging system and its real-time focusing method |
| CN108646396A (en) * | 2018-04-27 | 2018-10-12 | 合肥工业大学 | Auto-focusing microscopic system |
| CN110673325A (en) * | 2019-09-25 | 2020-01-10 | 腾讯科技（深圳）有限公司 | Microscope system, smart medical device, auto-focusing method, and storage medium |
| CN110727093A (en) * | 2019-11-21 | 2020-01-24 | 宁波五维检测科技有限公司 | Multispectral microscopic automatic focusing device and method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012181341A (en) * | 2011-03-01 | 2012-09-20 | Olympus Corp | Microscope device |
| JP6119963B2 (en) * | 2012-11-21 | 2017-04-26 | 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． | Automatic focus control device, semiconductor inspection device and microscope |
| CN111443476B (en) * | 2020-04-13 | 2023-04-14 | 腾讯科技（深圳）有限公司 | Microscope autofocus method, microscope system, medical device, and storage medium |
-
2020
- 2020-04-13 CN CN202010284514.XA patent/CN111443476B/en active Active
- 2020-04-13 CN CN202310437825.9A patent/CN116430568A/en active Pending
-
2021
- 2021-02-25 WO PCT/CN2021/077828 patent/WO2021208603A1/en active Application Filing
-
2022
- 2022-05-16 US US17/745,571 patent/US20220342195A1/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104932092A (en) * | 2015-06-15 | 2015-09-23 | 上海交通大学 | Automatic focusing microscope based on eccentric beam method and focusing method thereof |
| CN107656364A (en) * | 2017-11-16 | 2018-02-02 | 宁波舜宇仪器有限公司 | A kind of micro imaging system and its real-time focusing method |
| CN108051897A (en) * | 2018-01-17 | 2018-05-18 | 宁波舜宇仪器有限公司 | A kind of micro imaging system and real-time focusing method |
| CN108254853A (en) * | 2018-01-17 | 2018-07-06 | 宁波舜宇仪器有限公司 | A kind of micro imaging system and its real-time focusing method |
| CN108646396A (en) * | 2018-04-27 | 2018-10-12 | 合肥工业大学 | Auto-focusing microscopic system |
| CN110673325A (en) * | 2019-09-25 | 2020-01-10 | 腾讯科技（深圳）有限公司 | Microscope system, smart medical device, auto-focusing method, and storage medium |
| CN110727093A (en) * | 2019-11-21 | 2020-01-24 | 宁波五维检测科技有限公司 | Multispectral microscopic automatic focusing device and method |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021208603A1 (en) * | 2020-04-13 | 2021-10-21 | 腾讯科技（深圳）有限公司 | Microscope autofocusing method, microscope system, medical device, and storage medium |
| CN116095477A (en) * | 2022-08-16 | 2023-05-09 | 荣耀终端有限公司 | Focusing processing system, method, equipment and storage medium |
| CN116095477B (en) * | 2022-08-16 | 2023-10-20 | 荣耀终端有限公司 | Focusing processing system, method, equipment and storage medium |
| CN116794822A (en) * | 2023-07-05 | 2023-09-22 | 苏州欧米特光电科技有限公司 | Microscope control system and method |
| CN116794822B (en) * | 2023-07-05 | 2024-04-30 | 苏州欧米特光电科技有限公司 | Microscope control system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116430568A (en) | 2023-07-14 |
| WO2021208603A1 (en) | 2021-10-21 |
| US20220342195A1 (en) | 2022-10-27 |
| CN111443476B (en) | 2023-04-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111443476B (en) | Microscope autofocus method, microscope system, medical device, and storage medium | |
| US10330906B2 (en) | Imaging assemblies with rapid sample auto-focusing | |
| KR101632578B1 (en) | Image capturing apparatus, image processing apparatus, and image processing method for generating auxiliary information for captured image | |
| WO2010016625A1 (en) | Image photographing device, distance computing method for the device, and focused image acquiring method | |
| JP6862569B2 (en) | Virtual ray tracing method and dynamic refocus display system for light field | |
| US10146039B2 (en) | Image capture method for a microscope system, and corresponding microscope system | |
| WO2021057422A1 (en) | Microscope system, smart medical device, automatic focusing method and storage medium | |
| CN111308690B (en) | Optical field electronic endoscopic equipment and imaging method thereof | |
| CN113568153B (en) | Microscopic imaging equipment and nanoscale three-dimensional shape measurement system | |
| CN115830675B (en) | Gaze point tracking method and device, intelligent glasses and storage medium | |
| Kagawa et al. | A three‐dimensional multifunctional compound‐eye endoscopic system with extended depth of field | |
| CN110519493A (en) | Imaging device and imaging method | |
| US8508589B2 (en) | Imaging systems and associated methods thereof | |
| CN108937909B (en) | Layer-selecting blood flow speckle imaging device and method based on lamellar light | |
| CN111443477B (en) | Microscope auto-focusing method, microscope system, medical device, and storage medium | |
| US8593508B2 (en) | Method for composing three dimensional image with long focal length and three dimensional imaging system | |
| US20040169922A1 (en) | Stereo microscopy | |
| JP6069324B2 (en) | Single-axis stereoscopic imaging device with dual sampling lens | |
| US20180017776A1 (en) | Variable working distance microscope | |
| Koh et al. | Autofocus method using dual aperture and color filters | |
| JPH08206081A (en) | Photographing device | |
| JP2013174709A (en) | Microscope device and virtual microscope device | |
| CN107249102B (en) | A kind of method and apparatus for eliminating defocus | |
| Alonso | Multi-focus computational optical Imaging in Fourier Domain | |
| JP2024005590A (en) | Image processing device, imaging device, image processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40026357 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |