CN104537705A - Augmented reality based mobile platform three-dimensional biomolecule display system and method - Google Patents

Abstract

The invention relates to an augmented reality based mobile platform three-dimensional biomolecule display system and an augmented reality based mobile platform three-dimensional biomolecule display method. The identification of the traditional paper print biomolecule two-dimensional picture and the three-dimensional real-time display of the corresponding molecule are realized on a mobile platform. The image feature point detection and matching method is improved, so that the display is collaboratively completed by a mobile client and a cloud server, the flexibility is improved, and the client resources are reduced. Three-dimensional molecular modeling data is derived from a PDB (Protein Data Bank), so that the accuracy and the authority are ensured. The augmented reality technology is expanded, a typical application is provided in biochemistry and molecular biology education learning aspects, and the technical means and the framework can be applied to any education and teaching fields; because of the development and popularization of a mobile terminal, and the portable characteristic of the mobile terminal, the system can be used by users at any time and any place, so that the teaching efficiency and the interest in learning are greatly improved.

Description

Based on the mobile platform three dimensional biological molecular display system and method for augmented reality

Technical field

The present invention is a kind of three-dimensional protein, nucleic acid molecules real-time rendering, mutual and tracer technique of realizing on a mobile platform, relates to the application of augmented reality.

Background technology

For a long time, preclinical medicine science, during the such subject of especially similar Biochemistry and Molecular Biology is teaching and research, a maximum difficult problem is exactly microcosmic, the invisibility of research object.Along with the arrival in genome times afterwards comprehensively, Leaf proteins epoch, this contradiction is more and more outstanding.The fast development of subject requires to grasp new theory and new technology as early as possible from, practitioner, and traditional teaching using paper media as carrier, research means do not catch up with requirement.

Augmented reality (AR, augmented reality) technology is that a kind of computer technology that utilizes carries out the technology of scene expansion to the real world that user sees.AR comes from virtual reality (Virtual Reality, VR) technology, and the latter is the technology utilizing the virtual environment of Practical computer teaching to simulate the behaviors such as the vision of people in physical environment, the sense of hearing, motion realistically.With it unlike, reality scene superposes after analog simulation by AR again, the figure that it produces computing machine, word are fused in real scene by the position of specifying and attitude, user is immersed in true and virtual hybird environment, produces a kind of novel man-machine interaction mode.Augmented reality has become a key areas in virtual reality research, and it has a wide range of applications in industries such as military affairs, safety, industry, medical treatment, education and amusements.Augmented reality contains multimedia, three-dimensional modeling, real-time video display and control, sensor, real-time follow-up and new technology and the new tool such as registration, scene fusion.Augmented reality provides in the ordinary course of things, is different from the appreciable information of the mankind.

Present invention utilizes that mobile platform is easy to carry, feature easy to use and high in the clouds data retrieval.Achieve AR technology mobile terminal is complete, and made expansion in conjunction with actual conditions and improved.AR technology and biomolecule three-dimensional display combine by the present invention first, three-dimensional model and animation is made as three-dimensional molecular modeling foundation, for AR technology proposes a kind of typical case realize in micro molecule teaching, simulation field using RCSB protein and nucleic acid structure information database.

Summary of the invention

The technical matters that the present invention solves is: a kind of mobile platform three dimensional biological molecule virtual display technology based on augmented reality.Comprise that picture feature point high in the clouds stores, recognition detection and Model Matching, to be identified as basic cloud server-Protein Data Bank data retrieval and transmission, the real-time rendering of mobile terminal biomolecule three-dimensional model, mobile terminal camera location position and scene drawing with picture feature point.

Instant invention overcomes the limitation of traditionally on paper media to three-dimensional molecular structure and function demonstration, propose a kind of brand-new three-dimensional molecular real-time exhibition technology and teaching means, overcome efficiency and speed issue that complex polygon shows at cell phone platform, realize for AR technology proposes a typical case in the application of micro molecule teaching, simulation field.

The technical solution used in the present invention is: a kind of mobile platform three dimensional biological molecule virtual display technology based on augmented reality, comprises following five steps:

Step (1) picture feature point extracts: take pictures to the plane picture on teaching material or scan, extract minutiae data are also stored in service end.

Step (2) PDB file acquisition and process: No. ID, its PDB is specified to each pictures, server is retrieved at PDB with No. ID, PDB, obtain the laggard row relax of pdb file, and be stored in server, Image ID is associated with pdb file and picture relevant information.

Step (3) picture recognition: picture feature dot matrix is extracted in mobile terminal, and upload to service end with certain hour interval, the characteristic point matrix in service end ergodic data storehouse carries out similarity-rough set.Two width images match are thought when two width Image Feature Point Matching quantity are greater than the threshold value of setting.

The drafting of step (4) three-dimensional separation flow: in mobile terminal according to the three-dimensional atomic point coordinate in the PDB file obtained, real-time rendering molecule C atomic framework ribbon model and surface three dimension model.

The Tracing Registration of step (5) view-based access control model mark: the true picture that the model generated in step (4) and camera are taken is carried out Overlapping display by mobile client end system.Carry out camera when client camera position changes again to demarcate, through coordinate transform real-time update visual angle, make observer can from arbitrarily angled and distance observing and nursing.

The present invention's advantage is compared with prior art:

1. recognition efficiency and accuracy: Feature point recognition method conventional in AR technology is Corner Detection, as Roberts, Sobel, Prewitt, Canny operator detection method etc.And in the books illustration of reality, some picture feature points must be had less, adopt these methods cannot extract effective unique point.Cause the decline of discrimination.In the present invention, the identification coupling of picture feature point uses Surf algorithm.Surf algorithm based on Sift algorithm, but does not need to carry out double sampling to image when building metric space, thus improves algorithm performance.Its maximum feature is that recognition speed is fast.Under moderate condition, the coupling completing two width objects in images achieves real-time process substantially.Relatively other algorithm, can effective recognition feature point, and speed and the accuracy rate of identification have greatly improved.

2. the identification of plurality of pictures, Network Recognition: traditional AR technology is that identification picture is retained in this locality as template, take up room large and dumb.And several specific pictures can only be identified, be of limited application.In the present invention, identify that picture is retained in cloud server as template, reduce client volume and take, in theory any plurality of pictures of identifiable design; Picture recognition depends on network, and the actual of user end to server transmission is the characteristic point data extracted, and its data volume is very little, can be understood as simple text, can reduce network traffics, shorten the transmission time.

3. three-dimensional separation flow real-time rendering: traditional AR technology is generally first make three-dimensional model to be shown, carries out when needing showing and hiding.The present invention, in mobile client, utilizes OpenGL ES to complete the real-time rendering of three-dimensional separation flow.Corresponding PDB file processes by server, only only sends a small amount of information (three-dimensional molecular volume coordinate) to client, further increases the real-time of identification; When needing to change model, only need transmission of coordinate file again, alleviate mobile end system burden.

4. the present invention first by AR Technology application in the three-dimensional real-time exhibition with biomolecule.

5. the present invention utilizes the feature of mobile terminal portability, can use whenever and wherever possible, overcome the limitation utilizing desktop computer etc. to realize AR technology.

Accompanying drawing explanation

Fig. 1 is system principle of the present invention and implementation schematic diagram;

Fig. 2 is that server processes retrieving the pdb file obtained;

Fig. 3 is according to Surf method, the picture feature Point matching schematic diagram that service end picture and client photograph;

Fig. 4 camera model adopts pin-hole model approximate representation, the projected position p of the center of gravity P of mark on image in true environment, the line OP put for optical centre O and P and the intersection point of the plane of delineation;

Fig. 5 is net result (human hemoglobin, PDB ID:2dhb).

Embodiment

Below in conjunction with accompanying drawing, the specific embodiment of the invention is described:

Step:

1. picture feature point extracts

1.1 image acquisition:

Illustration in book is obtained by scanner or the form of taking pictures.

1.2 picture pre-service:

S1: gamma correction: the rgb value of each pixel is added simultaneously or deduct a specific value.

S2:: gray scale transformation: by gray-scale map Nogata graphing, calculate the number of pixels of each Pixel-level; Form accumulation histogram, then carry out histogram equalization; Set up new brightness look-up table.

S3: smoothing processing, to image length and width respectively × 2, and Gauss filters smoothing.

1.3Surf algorithm realization and unique point generate:

This part is mainly divided into 5 steps:

S1: build Hessian matrix, replace functional value f (x with image pixel L (x, y), y), select second order standard gaussian function as wave filter, by the convolutional calculation second-order partial differential coefficient between particular core, so just, H matrix can be calculated, in formula, Lxx (x, σ), Lxy (x, σ), the second derivative of Lyy (x, σ) for X point place Gaussian function and the convolution of image.

$H(x,\mathrm{\σ})=\left[\begin{array}{c}\mathrm{Lxx}(x,\mathrm{\σ})\mathrm{Lxy}(x,\mathrm{\σ})\\ \mathrm{Lxy}(x,\mathrm{\σ})\mathrm{Lyy}(x,\mathrm{\σ})\end{array}\right]$

S2: build metric space: in Sift algorithm, every tomographic image depends on last tomographic image, and image needs to reset size, therefore, this computing method operand is comparatively large, and Surf algorithm is different from Sift in use pyramid.Algorithm allows metric space multi-layer image to be processed simultaneously, does not need to carry out secondary sample to image, thus improves performance.

S3: location feature point: determine a suitable default extreme value, uses and detects with the wave filter of the corresponding size of S2 mesoscale tomographic image resolution, if the eigenwert of pixel is greater than surrounding pixel, can determine that this point is the unique point in this region.All values being less than default extreme value are all dropped.

S4: principal direction is determined: centered by unique point, calculating radius be 6s (S is the scale-value at unique point place) neighborhood in the Haar small echo (the Haar small echo length of side get 4s) of point in z, y direction respond, and compose Gauss's weight coefficient to these responses, make the response contribution near unique point large, and it is little away from the response contribution of unique point, secondly the response within the scope of 60 ° is added to form new vector, travel through whole border circular areas, the direction selecting most long vector is the principal direction of this unique point.Like this, calculated one by one by unique point, obtain the principal direction of each unique point.

S5: generating feature point descriptor: get the window of 8 × 8 centered by key point.Try to achieve gradient magnitude and the gradient direction of each pixel, compute weighted with Gauss's window to it, the corresponding vector of each pixel, then calculates the gradient orientation histogram in 8 directions, draws the accumulated value of each gradient direction on the fritter of every 4 × 4.

The preservation of 1.4 picture feature vectors

Above-mentioned unique point vector information is kept at home server, in a database by Image ID associated.

2.PDB file acquisition and process

2.1PDB file acquisition: when server background uploads every pictures, specify No. ID, the PDB of every pictures to manually, server is retrieved in RCSB PDB data base manipulation RCSB-PDB FETCH services service according to this No. ID, because No. ID unique, so the pdb file obtained also is unique.

2.2PDB file processing: original pdb file is general comparatively large, includes a lot of redundant information, for reducing server and client side's network data transmission amount, server carries out pre-service to the pdb file retrieved.The row comprising atom point coordinate in Pdb file starts with ATOM or HETATM.Find with the row that ATOM or HETATM. first time occurs in file, from this journey, row is before deleted, only retain the file of three-dimensional atomic point coordinate, as shown in Figure 2.

3. picture recognition and template matches

Picture is aimed at by camera in mobile terminal, catches scene, carries out size and unitizes and primary compression, to continue every the speed of 10fps the picture captured to be carried out extract minutiae process, the same step of disposal route (1) described method.Mail to service end after extracting image characteristic point, in service end, for a unique point on certain width image, utilized as K-D tree algorithm finds with the proper vector of point to be matched apart from minimum and secondary minimum unique point in another piece image.If minor increment and time minor increment ratio are less than certain threshold value, then think point to be matched with apart from minimum Point matching.Otherwise mate unsuccessful.As shown in Figure 2, left figure is service end picture, and right figure is the picture that mobile terminal captured in real-time arrives.Both carry out the quantity of Feature Points Matching (line) much larger than the threshold value of setting, think and have captured picture.

4. view-based access control model mark Tracing Registration, three-dimensional separation flow draw

S1: camera position is demarcated:

This part mainly realizes based on Qualcomm Vuforia FastCV storehouse of increasing income, and existing is illustrated principle:

Camera calibration technology is relevant with four coordinate systems, is world coordinate system respectively, imaging plane coordinate system, camera coordinates system and image displaing coordinate system.The three-dimensional body of real world becomes the process of the two-dimensional points of the plane of delineation by camera imaging principle, need the coordinate transform between four coordinate systems, camera calibration technology needs to determine the transformation relation between these coordinate transforms, then utilizes this relation to calculate this position in true three-dimension space according to the two-dimensional points on image manifesting planes.

First 5 inner parameters and 6 external parameters of video camera are calculated.

Camera model adopts pin-hole model approximate representation, the projected position p of the center of gravity P of mark on image in true environment, the line OP put for optical centre O and P and the intersection point of the plane of delineation, as shown in Figure 4.

Color sign point three dimensional space coordinate in world coordinate system known Xwi, Ywi, Zwi.Its coordinate in camera coordinate system and the position in image slices areal coordinate system can represent with following formula.Wherein R, T are respectively rotation matrix and translation vector.F is the focal length of video camera, and u0, v0 are image plane center coordinate, and △ x, △ y is the spacing of neighbor pixel in horizontal and vertical direction.

$\left[\begin{array}{c}\mathrm{Xc}\\ \mathrm{Yc}\\ \mathrm{Zc}\end{array}\right]=R\left[\begin{array}{c}\mathrm{Xw}\\ \mathrm{Yw}\\ \mathrm{Zw}\end{array}\right]+T$

$\mathrm{Zc}\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}\mathrm{Xc}\\ \mathrm{Yc}\\ \mathrm{Zc}\\ 1\end{array}\right]$

$u=\frac{x}{\mathrm{\Δx}}+u_0,v=\frac{y}{\mathrm{\Δy}}+v_0$

Above formula comprises 11 unknown parameters altogether, 5 intrinsic parameters of the camera f, u0, v0, △ x, △ y; 6 external parameter R (Rx, Ry, Rz) and T (Tx, Ty, Tz).More than 6 the picture point coordinate (u as long as therefore have living space _i, v _i) just can obtain whole inside and outside parameter.

S2: model, scene drawing

Protein structure has multiple drawing notation, modal have wire-frame model (wireframe), rod model (Bonds), ball-and-stick model (CpK), VDW model, the belt pattern that wafts (Ribbon) and molecular surface model etc.

Wire-frame model (Wireframe): the straight line of the chemical bond between atom represents.

Rod model (Bond): the right cylinder of the chemical bond between atom represents.

Ball-and-stick model (CPK): an atom spheroid represents, a chemical bond right cylinder represents.

Van der Waals model (VDW): an atom spheroid represents, the radius of a ball is van der Waals radius.

Ribbon model (Ribbon): represented by the band shape evenly expanded to both sides centered by the B-spline curves of all C atoms.

Molecular surface model (MolecularSurface): comprise two kinds of definition: Model Molecule surface and by solvent molecule when molecular surface moves the surface that formed of the track of process.

Ribbon model is one of visual most important model of biomacromolecule, and this is mainly because it effectively can reflect the secondary structure information of biomolecule.Molecular surface model shows biomacromolecule outward appearance on the whole, and making has visual understanding to it.Meanwhile, the limitation of mobile terminal GPU performance, it is not suitable for the performance very many ball-and-stick models of gore number or Van der Waals model, and the quantity of information that wire-frame model can show very little.In sum, we adopt the method at client real-time rendering ribbon model and translucent molecular surface model, and the latter is additional on the former.This method can reasonable display molecule structure and three-dimensional appearance.

Ribbon modeling rendering:

The ribbon model geometric of protein is former molecular by main chain, and according to backbone atoms, we can construct the contour curve of ribbon model.Contour curve can have multiple method for drafting.The key point that Carson method and protein chemistry are associated is the basis that planar unit of peptide is used as geometric modeling, and the key point that this method and computer graphics are associated is that B-spline is used as the smooth regular curve producing ribbon model.The method is widely used in the drafting of protein ribbon model.Specific algorithm is as follows:

Definition planar unit of peptide: only need the alpha carbon atom and the carboxyl oxygen atom coordinate that input the good protein molecule of a series of sequence.Below the method (C α represents alpha-carbon atom) defining planar unit of peptide:

1. compute vector A=C α (i+l)-C α (i)

2. compute vector B=O (i)-C α (i)

3. multiplication cross obtains vectorial C=A × B

4. multiplication cross obtains vectorial D=C × A

5. couple vectorial C and D carries out unitization

Vector C is by C α (i), O (i), and the normal direction of the former molecular planar unit of peptide of C α (i+l).Vector D is parallel to planar unit of peptide and perpendicular to vectorial A.

Axial coordinate in generation: the input of this step is the interpolation point number of simulation required for ribbon, adds the assignment of the amino acid whose width for each secondary structure.Below the step defining axial coordinate in B-spline:

Find the mid point P of Ca (i) and Ca (i+l).If this planar unit of peptide is a part for alpha spiral, along the direction translation point P of vectorial C, the width moving half ribbon to the direction away from screw axis obtains a P (-)=P-D and P (+)=P+D respectively.

Just define line segment P (-) → P (+) after step when above completes, its width is the width of the ribbon that we want and is parallel to planar unit of peptide.These points with regard to uniform distribution on this line segment.

There is above-mentioned point coordinate, just can call OpenGL ES storehouse and carry out ribbon modeling rendering.

Molecular surface model is drawn:

The drafting of three-dimensional molecular surface model, most critical be how to define molecular surface.Atom due to molecular surface border determines shape and the major function of molecule, therefore, effectively judges that the atom on molecule border just becomes especially important.This one of them method is exactly the complete smooth surface of of calculating on protein molecule surface atom.This curved surface is just referred to as molecular surface.

Solvent outer curved surface (solvent excluded surface, SES) method for drafting uses detection ball that a radius is R to carry out the reach distance of rolling calculation molecular surface on molecule Van der Waals surface, and the surface model degree of accuracy that it calculates is the highest.But calculated amount is larger.Real-time rendering cannot be accomplished.We adopt the Coarse method for drafting rendered surface model based on Gaussian Blur, are reducing under the prerequisite of drawing precision, achieve three-dimensional molecular surface real-time rendering in mobile terminal.

All reality tests of the present invention are all carried out on pc simulator with the mobile phone of operation Android 4.3 system, and program can smoothly be run.

The technology contents that the present invention does not elaborate belongs to the known technology of those skilled in the art.

Although be described the illustrative embodiment of the present invention above; so that those skilled in the art understand the present invention; but should be clear; the invention is not restricted to the scope of embodiment; to those skilled in the art; as long as various change to limit and in the spirit and scope of the present invention determined, these changes are apparent, and all innovation and creation utilizing the present invention to conceive are all at the row of protection in appended claim.

Claims (7)

1. the mobile platform three dimensional biological molecular display system based on augmented reality, it is characterized in that: comprise mobile terminal (101), wireless network communication (102), cloud server (103), protein structural information database (104), mobile terminal (101) carries out communication by wireless network communication (102) and cloud server (103), cloud server (103) and protein structural information database (104) carry out communication, and mobile terminal (101) is as final display device.

2., based on a mobile platform three dimensional biological molecular display method for augmented reality, its feature comprises the following steps:

Step (1), picture feature point extract: take pictures to the plane picture on teaching material or scan, after extract minutiae, characteristic point data being stored in service end;

Step (2), PDB file acquisition and process: to each pictures, manually determine No. ID, PDB in the protein structural information database of its correspondence, server is retrieved at protein structural information database with No. ID, this PDB, obtain the laggard row relax of pdb file and local storage, Image ID is associated with pdb file and picture relevant information;

Step (3), picture recognition: mobile terminal use camera catches the plane picture on teaching material, after extract minutiae, the unique point extracted is uploaded onto the server by the time at certain intervals, carry out similarity-rough set with the unique point of each pictures in step (1), if do not mate, abandon this figure; If coupling, thinks and has recognized a mark; The PDB file of correspondence and relevant information are mail to client;

The drafting of step (4), three-dimensional separation flow: in mobile terminal according to the three-dimensional atomic point coordinate in the PDB file obtained, utilize OpenGL ES storehouse real-time rendering biomolecule C atomic framework ribbon model, molecular surface three-dimensional model;

The Tracing Registration of step (5), view-based access control model mark: the real background that the model generated in step (4) and camera are taken is carried out Overlapping display by mobile client end system; What when the displacement of client camera space or rotation change, system was real-time redraws window, through coordinate transform real-time update visual angle, make observer can from arbitrarily angled and distance observing and nursing.

3. the mobile platform three dimensional biological molecular display method based on augmented reality according to claim 2, it is characterized in that: step (1) picture feature point extracting method, by Surf (the Speeded Up Robust Features) algorithm based on Sift (Scale-Invariant Feature Transform) algorithm improvement, find image characteristic point, be kept at server.

4. the mobile platform three dimensional biological molecular display method based on augmented reality according to claim 2, it is characterized in that: the PDB file acquisition method described in step (2), to each pictures in step (1), artificial regulation one No. PDBID, utilize RCSB-PDB web services to serve from PDB server retrieves, simplify processes and local storage are carried out to the pdb file obtained.

5. the mobile platform three dimensional biological molecular display method based on augmented reality according to claim 2, it is characterized in that: the image identification method described in step (3), picture feature dot matrix is extracted with algorithm similar in step (1) in mobile terminal, upload to service end with certain hour interval, the characteristic point matrix in service end ergodic data storehouse carries out similarity-rough set; Two width images match are thought when two width Image Feature Point Matching quantity are greater than the threshold value of setting.

6. the mobile platform three dimensional biological molecular display method based on augmented reality according to claim 2, it is characterized in that: the method for drafting of the three-dimensional separation flow described in step (4), use OpenGL ES storehouse, the three-dimensional atomic coordinate information of the PDB file utilizing server to send, draws ribbon model (Ribbon Model) and surface three dimension model.

7. the mobile platform three dimensional biological molecular display method based on augmented reality according to claim 2, it is characterized in that: the Tracing Registration method of the view-based access control model mark described in step (5), four limits of the convex quadrangle illustration main body obtained according to image recognition are with original AR mark, base determines that image affine coordinate system is set up in direction, thus demarcate the spatial information of mobile terminal camera and agent localization mark, comprise position and attitude; The three-dimensional separation flow made in plot step (4) on view plane according to affine transformation matrix or interactive scene, real-time pictures that is last and real scene merges to be played up, and is presented on the screen of mobile terminal; If there is picture additional information, then it is shown together.