CN110196642B - Navigation type virtual microscope based on intention understanding model - Google Patents

Abstract

The invention provides a navigational virtual microscope based on an intention understanding model, which includes a multi-modal input and perception module, a multi-modal information integration module and an interactive application module; the multi-modal input and perception module is used to obtain the user's information through a microphone. Voice information, and obtain the user's operation behavior; the multimodal information integration module is used to process the voice information through the visual channel information and the operation behavior through the tactile channel information, and then process the processed voice information and operation behavior through the multi-channel information. Integration, completes the interaction between the microscope and the user. The invention acquires and integrates multi-modal information, utilizes simple sensing elements, adds multi-modal signal input and intelligent sensing technology, and on the basis of ensuring the advantages of digital microscopes, allows the majority of ordinary and poor middle school students to also Ability to learn microscopes, increase cognitive experience of the microscopic world, and experience intelligent microscopes.

Description

A Navigated Virtual Microscope Based on Intent Understanding Model

technical field

The invention belongs to the technical field of intelligent microscope design, and particularly relates to a navigational virtual microscope based on an intention understanding model.

Background technique

The use of microscopes is a key content in the study of biological sciences in middle school, and it is also a content that students often make mistakes during the examination process. In the process of learning, mastering the rules of using microscopes is of great benefit to learning theoretical knowledge and exploring the microscopic world. . However, the microscopes often used in middle schools and introduced in the textbooks are very traditional monocular microscopes. Students often spend a lot of time adjusting the focus and proper brightness in the process of using the microscope, and the imaging effect is not good. One experiment After class, students' mood often changes from initial excitement to exhaustion. In traditional teaching experimental classes, teachers usually demonstrate on the podium and students observe below. Even if the students' operation process is relatively standardized, they may not get good results. Teachers often run between different students, resulting in insufficient communication between teachers and students. Effective, causing students to have biases in cognitive biology at the perceptual level. Therefore, how to enable students to correctly observe the actual biological structures and phenomena becomes particularly important. What is more limited is that each microscope can only be observed by one person at the same time. If the teacher wants to see the experimental results of the students, he must check the images under the microscopes of the students one by one, and the purpose of sharing the experimental results between different students cannot be achieved. Therefore, , if successful experimental results and wrong case results can be displayed to students and teachers, it will have a good guiding effect on students' learning and motivation, and it will also bring great convenience for teachers to evaluate students' experimental conditions.

What is currently gaining popularity is the digital microscope. The so-called digital microscope can be connected to a computer, and the structures or phenomena observed in the microscope field of view are displayed on the computer display screen through software, so that the observed structures or phenomena are clearer and can be viewed by multiple people at the same time. The digital microscope has the following advantages: first, the object image is displayed on the computer screen, which reduces the difficulty of finding the object image; second, the teacher can monitor the student's computer screen in real time, quickly judge the students' operation problems, and make adjustments in time; third, if When students fail in experiments, teachers can share screenshots of other groups' successful images on the computer, urging students to analyze the reasons for failure. Using a digital microscope can simplify the evaluation process of the experiment, improve the feasibility and expand the coverage. However, the cost of digital microscopes is still relatively expensive. The price of the most common microscopes is about 2,000 yuan. For ordinary or poor middle schools, it becomes difficult to purchase microscope equipment in large quantities.

SUMMARY OF THE INVENTION

The invention proposes a navigational virtual microscope based on an intention understanding model, so that ordinary and poor middle school students can also have the conditions to study microscopes, increase their cognition and experience of the microscopic world, and experience intelligent microscopes.

In order to achieve the above purpose, a navigational virtual microscope based on an intention understanding model proposed by the present invention includes a multimodal input and perception module, a multimodal information integration module and an interactive application module;

The multimodal input and perception module is used to obtain the user's voice information through the microphone, and obtain the user's operation behavior; the operation behavior includes obtaining the direction of the user's adjustment of the microscope thickness and quasi-focus spiral by rotating the sensor and identifying it through the image acquisition device. The sample to be observed and the image information to detect the movement of the sample to be observed;

The multimodal information integration module is used to process the voice information through the visual channel information and the operation behavior through the tactile channel information, and then integrate the processed voice information and operation behavior through the multi-channel information to complete the communication between the microscope and the user. interaction;

The interactive application module predicts the user's intention and provides operation guidance through visual display and auditory guidance.

Further, the method for identifying the sample to be observed by the image acquisition device is that the upper surface of the glass slide is provided with a two-dimensional code picture representing the sample to be observed, and the image acquisition device recognizes different two-dimensional codes, and then retrieves the data from the database. to retrieve a sample image.

Further, the method for processing the visual channel information is: identifying the original RGB image of the sample, where D is a collection of sample images; observing the change of the sample, the image of the currently observed sample is P, and the moving transformation function is PW() , the newly generated image is P', then the displacement transformation process formula is P'(X, Y)=PM(X-△X, Y-△Y);

The P∈D, X, Y)∈P, ΔX and ΔY are the offsets of the pixels in the image P, respectively.

Further, the method for detecting the image information of the movement of the sample to be observed is that the lower surface of the slide glass is provided with a circular surface with detection marks, the position of the sample to be detected moves during the operation process, and the circular surface with detection marks is used. According to the formula of the surface and the displacement transformation process, the sample position movement is calculated, and the sample image is observed according to the offsets ΔX and ΔY.

所以根据算术均值滤波函数可以得到新生成的图像P″Further, the method for processing the tactile channel information is: adjusting the image sharpness by adjusting the coarse focal-focus screw or the fine focal-focus screw, wherein the rotation change function of the coarse quasi-focus screw or the fine quasi-focus screw is PV( ), and the adjustment Coarse quasi-focus spiral, get discrete values t∈T{11,12,13,14,15,16}, S _xy means the center point is at (x, y), the size of the filter window is mxn, and the mean filter function is PV ,in addition

Therefore, the newly generated image P" can be obtained according to the arithmetic mean filter function.

Adjust the fine quasi-focus spiral to obtain discrete values s, s∈S{1,2,3,4,5,6}, S _xy indicates that the center point is at (x, y), the size of the filter window is mxn, and the mean filter The function is PV, and in addition

Therefore, the newly generated image P" can be obtained according to the arithmetic mean filter function.

The P(x, y) is the current image that has not been adjusted by the coarse focal-focus screw or not adjusted by the fine-focus screw.

Further, the multi-channel information integration includes dual-channel information integration and three-channel information integration;

The two-channel information integration is used to adjust either the coarse focal spiral or the fine focal spiral, move the sample and observe the data integration of the sample; the multi-channel integration function is Muf(), the unadjusted image is P, and the moving The transformation function is PM(), and the rotation transformation function is PV(). Therefore, the multi-channel integration function can be expressed as:

Muf(P)=ɑPM(P)+(1-ɑ)PV(P);

The ɑ is the selection parameter, the ɑ=0, 1, the rotation sensor rotates, when one adjustment is completed, P″′=Muf(P); P″′ can be adjusted as the current observation image;

The three-channel information integration is used to input image information, operation behavior and voice information in the integration strategy. Under the current state conditions, the system enters different system states according to different user behaviors.

Further, the operation guide is that when the user uses the navigation virtual microscope, the voice prompts the operation steps or the operation method so that the sample to be tested is located in the field of view of the microscope.

Further, the image is saved as when the coarse focus screw or the fine focus screw is rotated, when the clarity of the sample in the field of view reaches the set threshold, a voice prompt will be given to save the image, and after the user confirms, the virtual microscope system will save the current field of view. The images in it are saved to the specified folder.

The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

The embodiment of the present invention proposes a navigable virtual microscope based on an intention understanding model, which includes a multimodal input and perception module, a multimodal information integration module and an interactive application module; the multimodal input and perception module is used to obtain information through a microphone. The user's voice information, and the user's operation behavior; the operation behavior includes obtaining the direction of the user's adjustment of the microscope thickness and quasi-focus spiral by rotating the sensor, and identifying the sample to be observed and detecting the image information of the movement of the sample to be observed through the image acquisition device; multi-mode The state information integration module is used to process the speech information through the visual channel information and the operation behavior through the tactile channel information, and then integrate the processed speech information and operation behavior through the multi-channel information to complete the interaction between the microscope and the user. The interactive application module predicts the user's intention and gives operation guidance through visual display and auditory guidance. Through the acquisition and integration of multimodal information, the present invention adjusts the thickness of the quasi-focus spiral through the acquisition of voice information and operation behaviors, moves the slide, and switches between different eyepieces and objective lenses to observe samples under different magnifications during the adjustment process of the microscope. image. For students, speech and visual presentation are natural interaction methods. Deep learning technology makes speech recognition, speech-to-text information and speech synthesis technology more accurate, and also makes the human-computer interaction process more flexible and intelligent. In terms of image processing, the intelligent microscope can accurately identify different sample labels with the help of image processing technology, calculate the position of the tracking point in real time, and control the movement of the sample image according to certain rules. The invention utilizes a simple sensing element, adds a variety of modal signal input and intelligent sensing technology, on the basis of ensuring the advantages of the digital microscope, so that the majority of ordinary and poor middle school students can also have the conditions to study the microscope, increasing the Cognitive feelings of the microscopic world, and experience the intelligent microscope.

Description of drawings

1 is an overall frame diagram of a multi-modal interaction of a navigation virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention;

2 is a diagram of a three-channel information integration strategy in a navigational virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention;

3 is a schematic structural diagram of the realization of a navigation virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention;

4 is a schematic diagram of the eyepiece structure of a navigation virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention;

Among them: 1- light hole; 2- stage; 3- camera; 4- lower surface of glass slide; 5- upper surface of glass slide 1; 6- upper surface of glass slide 2.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the description of the present invention, it should be understood that the terms "portrait", "horizontal", "upper", "lower", "front", "rear", "left", "right", "vertical", The orientation or positional relationship indicated by "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention, rather than indicating or It is implied that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and therefore should not be construed as limiting the invention.

Example 1

Embodiment 1 of the present invention provides a navigational virtual microscope based on an intention understanding model, including a multimodal input and perception module, a multimodal information integration module, and an interactive application module;

The multi-modal input and perception module is used to obtain the user's voice information through the microphone, and obtain the user's operation behavior; the operation behavior includes obtaining the direction of the user's adjustment of the microscope's thickness and quasi-focus spiral by rotating the sensor, and identifying the sample to be observed through the image acquisition device. Detect the image information of the movement of the sample to be observed;

The multimodal information integration module is used to process the speech information through the visual channel information and the operation behavior through the tactile channel information, and then integrate the processed speech information and operation behavior through the multi-channel information to complete the interaction between the microscope and the user ;

The interactive application module predicts the user's intention and gives operation guidance through visual display and auditory guidance.

Figure 1 shows an overall frame diagram of a multi-modal interaction of a navigation virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention;

The multi-modal input and perception module includes multi-modal input and interactive devices, and uses a microphone to obtain the user's voice information. The microphone uses a microphone, and the MIC is connected to the voice recognition device. The user's operation is conveyed through touch and vision, that is, from rotation Sensor and image acquisition device, the rotation sensor is used to capture the direction of the user to adjust the thickness of the quasi-focus screw, which is used to adjust the clarity of the sample, and the image acquisition device is used to identify the sample and detect the movement of the sample. The image acquisition device uses a camera.

The method of identifying the sample to be observed by the image acquisition device is as follows: the upper surface of the glass slide is provided with a two-dimensional code picture representing the sample to be observed, the image acquisition device recognizes different two-dimensional codes, and then retrieves the sample image from the database .

The multimodal information integration module is used for intention understanding, processing the speech information through the visual channel information and the operation behavior through the tactile channel information, and then integrating the processed speech information and operation behavior through the multi-channel information. The method of processing the speech information through the visual channel information is: identify the original RGB image of the sample, where D is the sample image collection; observe the change of the sample, the image of the currently observed sample is P, and the moving transformation function is PW(), The newly generated image is P', then the displacement transformation process formula is P'(X,Y)=PM(X-△X,Y-△Y);

Where P∈D, (X,Y)∈P, △X, △Y are the offsets of the pixels in the image P, respectively.

所以根据算术均值滤波函数可以得到新生成的图像P″The method of processing the operation behavior through the tactile channel information is to adjust the image sharpness by adjusting the coarse focal-focus screw or the fine focal-focus screw, wherein the rotation change function of the coarse focal-focus screw or the fine-focus screw is PV(), and the adjustment of the coarse focal-focus screw or the fine-focus screw is PV(). Quasi-focal spiral, the discrete value t∈T{11,12,13,14,15,16} is obtained, S _xy indicates that the center point is at (x, y), the size of the filter window is mxn, and the mean filter function is PV, in addition

Therefore, the newly generated image P" can be obtained according to the arithmetic mean filter function.

Adjust the fine quasi-focus spiral to obtain discrete values s, s∈S{1,2,3,4,5,6}, S _xy indicates that the center point is at (x, y), the size of the filter window is mxn, and the mean filter The function is PV, and in addition

Therefore, the newly generated image P" can be obtained according to the arithmetic mean filter function.

Among them, P(X,Y) is the current image that has not been adjusted by the coarse focal-focus screw or not adjusted by the fine-focus screw.

The processed voice information and operation behavior are integrated through multi-channel information, wherein the multi-channel information integration includes dual-channel information integration and three-channel information integration; dual-channel information integration is used to adjust any of the coarse focal spiral or fine focal spiral. One, move the sample and observe the data integration of the sample; the multi-channel integration function is Muf(), the unadjusted image is P, the moving transformation function is PM(), and the rotation transformation function is PV(). Therefore, the multi-channel integration function It can be expressed as:

Muf(P)=ɑPM(P)+(1-ɑ)PV(P);

Among them, ɑ is the selection parameter, ɑ=0, 1, the rotation sensor rotates, when one adjustment is completed, P″′=Muf(P); P″′ can continue to be adjusted as the current observation image;

FIG. 2 shows a three-channel information integration strategy diagram in a navigation virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention. The circle represents the current state of execution, the directed edge points to a possible transition state, and the directed edge has conditions that need to be met for the transition to occur. The input in the integration strategy includes image information P ₀ , P ₁ , operation behavior (obtained by the rotation sensor), voice information, the output content is P, P ₁ , P ₂ , and voice prompts. Among them, P ₀ is the sample label map, P ₁ is the label map that controls the movement of the sample, and P, P ₁ , P ₂ are the observed sample images. In the system, under the current state conditions, the system enters different system states according to different user behaviors. In Figure 3, for example, after the sample label P ₀ is successfully identified, the blurred sample P is displayed, and the user can choose to move the sample to observe, or choose to adjust the current sample to be clear, such as ɑ=0, the system enters the sample moving state , the camera detects the movement of the current sample, calculates the offset (△X, △Y) in real time, and displays the moved sample P ₂ . In this state, it can continue to “interact” with the Picture Move state, and an alternating State switching can also transition to the Recognition adjustment state according to user behavior. C is the intention predictor variable, which is not the user input, but the control condition generated by the system according to the current definition of P _{2. When P 2 meets} the definition requirement, it is considered that the user has obtained the sample image that he most wants to observe, namely It is inferred that the user will stay at this moment to observe the sample, so the system guides the user to save the current image for later observation. Therefore, the system will enter the voice interaction state, prompting the user whether to save the current image, if the user chooses to save the image, the system will enter the save state, otherwise, return to the previous state. If the image is successfully saved, it will go to the image display state. If it is unsuccessful, it needs to enter the voice input state. According to the confidence of the voice recognition, it is considered that the reason for the failure is that the voice input does not meet the requirements, so the user needs to re-enter the voice interaction state.

Different from the traditional microscope, the navigation virtual microscope based on the intention understanding model proposed by the present invention is no longer constructed with optical elements, but uses a variety of sensors and communication modules. The advantage of doing identification is that it avoids students from putting too much energy on sample production, and also avoids the occurrence of difficult observation due to improper sample production, and also solves the problem that some samples are difficult to obtain and cannot be reused. . On the basis of the traditional microscope function, the invention adds an interactive application module, wherein the interactive application module completes the prediction of the user's intention and gives the operation guidance through visual display and auditory guidance.

When users use a traditional microscope, when moving the sample up, down, left and right, it is easy to move the sample cells to be observed out of the field of view. At the same time, when using a smart microscope, the detection mark under the slide may also move out of the camera's field of view. Therefore, a hint is given for this situation. For example, the user keeps moving the sample to the left, and the image in the field of view keeps moving to the right. When the sample image is about to move out of the field of view, the user is voiced: "The image is about to exceed the left area, please move to the right".

The image is saved through the dialogue experiment between man and machine. The user rotates the coarse focus screw or the fine focus screw under a certain magnification to adjust the clarity of the sample in the field of view. When the clarity of the sample in the field of view reaches the set point The system considers that the user has obtained the sample he wants to observe, so the user is prompted by voice: "The sample image is very clear, you can choose to save the image". At this point, the user can say "save", "save this image", etc., and the system will intelligently save the image in the current field of view to the specified folder. You can also say "no need" or continue to adjust the thickness of the focus screw, and continue to adjust the observed sample.

FIG. 3 is a schematic structural diagram of the realization of a navigation virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention. On smart microscopes, the mirror and stage are replaced by square cubes and the slides are treated specially. A tiny camera is placed inside the cube, and the camera is facing the light hole, which is used to observe the slide sample. The upper surface of the slide is a QR code, which is different for different samples, and the lower surface is a circular surface with marks. Image acquisition equipment is used to identify the sample to be observed. Each slide has a QR code image representing the sample. Different two dimensions are identified according to the image, and then the sample image is retrieved from the database. Initializing the sample image, the blurred sample P can be observed.

During the operation process, the position of the sample to be detected moves, and the sample position is calculated by using the circular surface with the detection mark and the formula P'(X,Y)=PM(X-△X,Y-△Y) of the displacement transformation process Moving, the sample image P ₂ is observed according to the offset amounts ΔX and ΔY.

First, the QR code on the upper surface of the glass slide faces the light hole, the camera recognizes the sample label, and the sample picture is seen under the microscope. Detect the position of the circular surface of the marker and control the sample under the microscope in the opposite direction.

或者

以及Coarse focusing spiral or fine focusing spiral are composed of two concentric cylinders of different sizes, one end is sealed, and the other end is designed with a rotating shaft that can block light. At one end of the rotating shaft, six photosensitive sensors are evenly installed inside the inner cylinder. , rotate the shaft, after processing, the discrete values t, s will be obtained in turn, as the measurement of the user's rotation scale. According to the formula

or

as well as

Adjust the sharpness of the sample image _P3 .

Figure 4 shows a schematic diagram of the eyepiece structure of a navigational virtual microscope based on an intention understanding model proposed in Embodiment 1 of the present invention. For the eyepiece of the microscope, the top is an intelligent display, and the effect of sample adjustment will be displayed on the screen in real time. Three sensing buttons are designed on the barrel, which represent 5 times, 10 times and 40 times magnification respectively. Selecting different buttons can observe that the current sample image is magnified.

The above content is only an example and description of the structure of the present invention. Those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar methods to replace them, as long as they do not deviate from the structure of the invention. Or beyond the scope defined by the claims, all belong to the protection scope of the present invention.

Claims (7)

1. A navigation type virtual microscope based on an intention understanding model is characterized by comprising a multi-modal input and perception module, a multi-modal information integration module and an interactive application module;

the multi-mode input and perception module is used for acquiring voice information of a user through a microphone and acquiring operation behaviors of the user; the operation behavior comprises the steps of obtaining the direction of a user adjusting the thickness and the focus of the microscope through a rotation sensor, identifying a sample to be observed through image acquisition equipment and detecting image information of the movement of the sample to be observed;

the multi-mode information integration module is used for processing the voice information through a visual channel and processing the operation behavior through a tactile channel, and then integrating the processed voice information and the operation behavior through multi-channel information to finish the interaction between the microscope and a user; the multi-channel information integration comprises two-channel information integration and three-channel information integration;

the two-channel information integration is used for adjusting any one of a coarse quasi-focal spiral or a fine quasi-focal spiral, moving a sample and observing data integration of the sample; the multi-channel integration function is Muf (), the unadjusted image is P, the shift transformation function is PM (), and the rotation transformation function is PV (), so the multi-channel integration function can be expressed as:

Muf(P)＝ɑPM(P)+(1-ɑ)PV(P)；

the alpha is a selection parameter, the alpha is 0, 1, the rotation sensor rotates, and after one adjustment is completed, P' is Muf (P); p' "as the current viewed image can continue to be adjusted;

the three-channel information integration is used for inputting image information, operation behaviors and voice information in an integration strategy, and under the condition of the current state, the system enters different system states according to different user behaviors;

the interactive application module carries out prediction on user intention and gives operation guidance through visual display and auditory guidance.

2. The virtual microscope of claim 1, wherein the identification of the sample to be observed by the image capturing device is performed by arranging a two-dimensional code picture representing the sample to be observed on the upper surface of the slide, and the image capturing device identifies different two-dimensional codes to retrieve the sample image from the database.

3. The virtual microscope of claim 1, wherein the visual channel information is processed by: identifying an original RGB image of a sample, wherein D is a sample image collection; observing the change of the sample, wherein the currently observed image of the sample is P, the motion transformation function is PW (), the newly generated image is P ', and the displacement transformation process formula is P' (X, Y) ═ PM (X-delta X, Y-delta Y);

and P belongs to D, X and Y) belongs to P, and delta X and delta Y are respectively the offset of the pixel points in the image P.

4. The virtual microscope of claim 1, wherein the method of detecting the image information of the movement of the sample to be observed is that the lower surface of the slide glass is provided with a circular surface with a detection mark, the sample to be observed is moved in position during the operation, the movement of the sample position is calculated by using the circular surface with the detection mark and the formula of the displacement transformation process, and the image of the sample is observed according to the offsets Δ X and Δ Y.

5. The virtual microscope of claim 1, wherein the haptic channel information processing method is as follows: adjusting the image definition by adjusting a coarse quasi-focus spiral or a fine quasi-focus spiral, wherein the rotation change function of the coarse quasi-focus spiral or the fine quasi-focus spiral is PV (), adjusting the coarse quasi-focus spiral to obtain a discrete value T e to T {11,12,13,14,15,16},S_xyDenotes the center point at (x, y), the size of the filter window is mxn, the mean filter function is PV, and

so that a newly generated image P' can be obtained from the arithmetic mean filtering function

Adjusting the fine focusing screw to obtain a discrete value S, which belongs to S {1,2,3,4,5,6}, S_xyDenotes the center point at (x, y), the size of the filter window is mxn, the mean filter function is PV, and

so that a newly generated image P' can be obtained from the arithmetic mean filtering function

And P (x, y) is an image which is not subjected to coarse focusing screw adjustment or fine focusing screw adjustment currently.

6. The virtual microscope of claim 1, wherein the manipulation is directed to voice prompt manipulation steps or methods to position the sample in the field of view of the microscope when the user uses the virtual microscope.

7. The virtual microscope of claim 1, wherein the image is saved as a coarse quasi-focal spiral or a fine quasi-focal spiral, when the sample definition in the field of view reaches a set threshold, the image is saved by voice prompt, and after confirmation by the user, the virtual microscope system saves the image in the current field of view into a designated folder.

Images