CN109410680A - A kind of virtual operation training method and system based on mixed reality - Google Patents
A kind of virtual operation training method and system based on mixed reality Download PDFInfo
- Publication number
- CN109410680A CN109410680A CN201811378127.1A CN201811378127A CN109410680A CN 109410680 A CN109410680 A CN 109410680A CN 201811378127 A CN201811378127 A CN 201811378127A CN 109410680 A CN109410680 A CN 109410680A
- Authority
- CN
- China
- Prior art keywords
- digital
- model
- operating platform
- dimemsional
- dimemsional model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B9/00—Simulators for teaching or training purposes
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
- G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
Abstract
The present invention relates to a kind of virtual operation training method and system based on mixed reality, the present invention establishes the digital three-dimemsional model including bony structure and soft tissue according to the whole body volume data of real human body, so that model data is accurate and can clearly reflect the information such as human skeleton, muscle, blood vessel, nerve, dynamic teaching, dynamic demonstration, real-time communication can be realized and mostly the functions such as holds a consultation in conjunction with mixed reality head is aobvious.The coordinate of digital three-dimemsional model is transformed into the world coordinate system established on the basis of the operating platform, and is matched to operating platform plane, achievees the effect that dummy object is placed in plane naturally, to build operation atmosphere true to nature.According to the relative positional relationship respectively organized in surgical instrument and threedimensional model to the corresponding force feedback of hand application, it can allow operator's sense of reality operating force, know from experience hand tactile response of the surgical instrument when touching different tissues organ, improves operative training validity.
Description
Technical field
The present invention relates to biomedical and information technology cross discipline technical fields, and in particular to one kind is based on mixed reality
Virtual operation training method and system.
Background technique
Biomedical engineering (Biomedical Engineering, abbreviation BME) is that a Men Youli, work, doctor combine
Frontier branch of science is product of the various engineering subject to biomedicine infiltration.It is with modern science and engineering technology
Principle and method disclose its life from the angle of engineering science in structure, function and its correlation of multi-level upper research human body
Phenomenon is ordered, is diseases prevention, curing the disease provides comprehensive, high-tech a subject of new technological means.Traditional operation teaching is done
Method is imparted knowledge to students using simulation rubber people, and the demonstration of operation video plays, and organ material object sample explanation, human cadaver dissects religion
Learn, the forms such as living animal anatomic teaching and expert's surgery observation, for surgical training, operation preview, clinical diagnosis,
The assisted medical teachings link such as remote operation, rehabilitation.Teaching is generally required to be carried out in special laboratory or operating room.
This teaching method is insufficient in the presence of there are many, such as:
1, model person is imparted knowledge to students, and model person generally uses resin or latex material to be made, easy to aging, after using repeatedly
It just needs replacing, the price of model person is relatively high, often replaces teaching aid and considerably increases instruction cost.
2, human anatomic structure is imparted knowledge to students, and under lab by teaching of watching video shows or be autopsied, cannot play makes student
Very intuitive, easily study anatomical structure effect, and a large amount of training expenses, such as teaching material fee are produced, autopsy takes, real
Habit is taken.
3, expert's surgery observation form prevents medical personnel from being immersed in actual scene completely, is touched by audiovisual
Feel etc. that a variety of sense organs understand and learn various operation practical operations, for young doctor, cannot operate in person and and expert
Operating method is compared in time, discovers one's mistakes and insufficient, undoubtedly to train a large amount of expenses of young medical staff waste
With and the time, good trained effect cannot be obtained.
4, doctor can be allowed to learn new operation method and program before the surgery lesion, practices prepared operation
Plan, and treatment results are judged.If unfamiliar medico or trainee is allowed to be operated, greatly increase
The risk of operation can not ensure the life security of patient, increase hospital because of operative failure, the following expenditure cost and society
It can negatively affect.
5, living animal anatomic teaching, one side living animal is sufficiently expensive, and it is more sensitive to be related to animal protection etc.
Problem, the effect ten on the other hand due to the otherness of animals and human beings body, when carrying out operative training, to operation proficiency is improved
Divide limited.
In recent years, with the rapid development of computer graphics, application of the VR in terms of medicine has highly important existing
Sincere justice.It is mainly used for medicine training, medical research, Medical Communication and clinical treatment, in virtual environment, can establish virtual
Manikin, by means of tracking ball, wear-type visual device (HMD), tactilely-perceptible gloves, student can readily understand people
The internal each organ structure in portion, this than it is existing by the way of textbook it is much effective.
Surgeon is before really having an operation, can repeatedly mould over the display by the help of virtual reality technology
Quasi- operation, the intracorporal organ of mobile human find best operation plan and improve proficiency.It is multiple in long-distance remote control surgical operation
The plan arrangement of miscellaneous operation, information guiding of surgical procedure etc., virtual reality technology can play highly important effect.
However the system or method of the class of virtual operation training at present, the target object of training mostly pass through related software
Design, authenticity is not strong, and its training is mostly to be trained with the training form of study course, and theoretical property is stronger, with reality
Clinical operation also there is a big difference.
Summary of the invention
The present invention for the technical problems in the prior art, provides a kind of virtual operation training based on mixed reality
Method and system, authenticity is strong, has height degree of immersing, and can establish more true Training scene for trainee, while can
Think that non-operator constructs more true field observation scene.
The technical scheme to solve the above technical problems is that
On the one hand, the present invention provides a kind of virtual operation training method based on mixed reality, this method comprises:
Collect complete real human body whole body volume data, and utilize the volume data, building include bony structure with
And the digital three-dimemsional model of soft tissue；
The operating platform characteristic in reality scene is acquired, and establishes world coordinates on the basis of the operating platform
System；The coordinate of the digital three-dimemsional model is transformed into world coordinate system, and the digital three-dimemsional model is matched to
The operating platform plane；
Main perspective position coordinates are obtained in real time, and calculate spin matrix peace of the main perspective relative to world coordinate system origin
The amount of shifting to, and display angle of the digital three-dimemsional model on main perspective wear-type visual device projection screen is updated with this
Degree；
Real-time tracing hand motion obtains surgical instrument position delta data, and according to surgical instrument and Digital Three-Dimensional
The relative positional relationship respectively organized in model applies corresponding force feedback to hand by tactilely-perceptible gloves.
The beneficial effects of the present invention are: being established according to real human body whole body volume data includes bony structure and soft tissue
Digital three-dimemsional model so that human 3d model is accurate and can clearly reflect human skeleton, muscle, blood vessel, nerve
Etc. information, dynamic teaching, dynamic demonstration, real-time communication can be realized and mostly the functions such as hold a consultation in conjunction with mixed reality head is aobvious.It will
The coordinate of digital three-dimemsional model is transformed into the world coordinate system established on the basis of the operating platform, and is matched to
The operating platform plane, achievees the effect that dummy object is placed in plane naturally.When this planar structure is detached from sight, number
Word threedimensional model will also disappear in the visual field, when again identifying that planar structure, it will generate Digital Three-Dimensional mould again
Type, the atmosphere so that construction true to nature is performed the operation.Pass through touching according to the relative positional relationship respectively organized in surgical instrument and threedimensional model
Feel that perception gloves apply corresponding force feedback to hand, can allow operator's sense of reality operating force, knows from experience surgical instrument
Hand tactile response when touching different tissues organ improves operative training validity.
Further, real human body whole body volume data acquisition from one of nuclear magnetic resonance, CT and ultrasonic examination or
It is a variety of.
It is existing it is most of to establish the data source of threedimensional model be mostly data mapping, can also although CT interference is less
To reflect the solid shape of bone from multiple planar observation institutional frameworks, but its radiation is larger, from top to bottom to the same person
Scanning obtains whole body data can cause irreversible injury to current human.Nuclear magnetic resonance can arbitrarily do cutting for any angle
Layer, and it is radiationless.It is better than the inspection of the soft tissues such as CT, especially joint, muscle, central nervous system to the susceptibility of pathological tissues
The structure that comes to an end is apparent.But to the performance of the details of lung, skeletal structure, fracture etc., the display effect of nuclear magnetic resonance is not so good as
CT.Ultrasonic examination (B ultrasound) is low with intensity, frequency is high, to human zero damage, no pain, display methods multiplicity and it is famous, it is especially right
The observation of the haemodynamics of the detection and cardiovascular internal organs of human body soft tissue has its distinctive feature, such as: superficial lump, blood vessel,
Needle localization etc., but the penetration power of B ultrasound is weaker, is difficult to detect to gassiness sexual organ, such as lung, intestines, lesion is smaller or acoustic resistance
When robust is little, it is also difficult to be shown on ultrasonogram, such as: the tumor tissues of 1cm or so are also not easy to detect, and have certain limitation
Property.
Therefore the present invention combines a variety of data sources, constructs human 3d model jointly, learns from other's strong points to offset one's weaknesses, model is true to nature.
Further, the complete real human body whole body volume data of the collection, and the volume data is utilized, building packet
Include the digital three-dimemsional model of bony structure and soft tissue, comprising the following steps:
The setting flag object on living body utilizes one of nuclear magnetic resonance, CT and ultrasonic examination or multiple means acquisition packet
Multiple groups two-dimensional image sequence containing the marker；
Three-dimensional reconstruction is carried out respectively according to the multiple groups two-dimensional image sequence；
Data fusion and registration are carried out according to the position of marker, building includes the digitlization of bony structure and soft tissue
Threedimensional model.
The material of the setting flag object on living body, the marker can be different, but total carrying out nuclear-magnetism respectively
When vibration, CT and ultrasonic examination, the position of same marker setting must be identical, to carry out data fusion and to match punctual, energy
Enough there is specific object of reference, so that data fusion operation can be gone on smoothly.
Further, after the digital three-dimemsional model is matched to the operating platform plane, further includes:
At least one is obtained in real time from view position coordinate；
According to described from view position coordinate, the spin matrix from visual angle relative to world coordinate system origin and translation are calculated
Vector, and the digital three-dimemsional model is updated from the angles of display on the wear-type visual device projection screen of visual angle with this.
Multiple participants can be set in the present invention, i.e., by from visual angle wear-type visual device field observation with main perspective
Carry out the operating process of the operator of operative training.In the digitlization shown from the wear-type visual device projection screen of visual angle
Threedimensional model is bound with operating platform Plane association, to reach the effect that dummy object is placed in naturally in plane
Fruit.It obtains from view position coordinate, that is, is calculated according to from the video camera content of shooting of visual angle wear-type visual device in real time
Then camera position coordinate in world coordinate system obtains the view position coordinate from view person according to coordinate transform, and
According to the spin matrix and translation vector from view position coordinate relative to world coordinate system origin, the digitlization is updated
Threedimensional model is from the angles of display on the wear-type visual device projection screen of visual angle.It in this way can simultaneously be the operation of main perspective
Personnel and a true surgical environments are built jointly from the participant at visual angle.Operative training tends to really, so that training
Effect is more preferable.
Further, the digital three-dimemsional model is matched to the operating platform plane, comprising:
The digital three-dimemsional model is rotated, so that the human body central axes of digital three-dimemsional model and operating platform central axes
In parallel；
Obtain after rotation the smallest coordinate points of y value in digital three-dimemsional model, and by operating platform described in its place plane
Plane is registrated, to realize the operation that the digital three-dimemsional model is matched to the operating platform plane.
On the other hand, the present invention also provides a kind of virtual operation training system based on mixed reality, comprising:
Data acquisition unit couples with the data processing unit, for collecting complete living body whole body volume data, and
The living body whole body volume data is uploaded to data processing unit；
Data processing unit, for carrying out three-dimensional reconstruction using the volume data, building includes bony structure and soft
The digital three-dimemsional model of tissue；It is for acquiring the operating platform characteristic in reality scene, and with the operating platform
Benchmark establishes world coordinate system；For the coordinate of the digital three-dimemsional model to be transformed into world coordinate system, and will be described
Digital three-dimemsional model is matched to the operating platform plane；
Main perspective wear-type visual device with video camera, couples with the data processing unit, for shooting camera shooting
Video pictures in the machine visual field, and the video pictures are sent to the data processing unit；For number described in real-time display
The digital three-dimemsional model rebuild according to processing unit；
Data processing unit, the video pictures for being shot according to the main perspective wear-type visual device calculate main perspective
Position is used for basis for calculating spin matrix and translation vector of the main view Angle Position relative to world coordinate system origin
The spin matrix and translation vector update the digital three-dimemsional model on main perspective wear-type visual device projection screen
Angles of display；
Tactilely-perceptible gloves couple with the data processing unit, for acquiring hand motion, obtain surgical instrument position
Delta data simultaneously uploads, for executing the control instruction of data processing unit.
Data processing unit, for according to acquisition hand motion, surgical instrument position delta data calculate surgical instrument with
The relative positional relationship respectively organized in threedimensional model, and the tactilely-perceptible gloves are controlled with this to hand to apply corresponding power anti-
Feedback.
Further, real human body whole body volume data acquisition from one of nuclear magnetic resonance, CT and ultrasonic examination or
It is a variety of.
Further, which further includes, at least one with video camera from visual angle wear-type visual device, with the number
Couple according to processing unit, is sent to the data for shooting the video pictures in camera coverage, and by the video pictures
Processing unit；The digital three-dimemsional model rebuild for data processing unit described in real-time display.
Further, the data processing unit is also used to according to the video shot from visual angle wear-type visual device
Picture is calculated from view position, for calculating the spin matrix from view position relative to world coordinate system origin and translation
Vector is visually set for updating the digital three-dimemsional model according to the spin matrix and translation vector from visual angle wear-type
Angles of display on standby projection screen.
Detailed description of the invention
Fig. 1 is the method for the present invention flow chart；
Fig. 2 is present system structure chart.
Specific embodiment
The principle and features of the present invention will be described below with reference to the accompanying drawings, and the given examples are served only to explain the present invention, and
It is non-to be used to limit the scope of the invention.
On the one hand, the present invention provides a kind of virtual operation training method based on mixed reality, as shown in Figure 1, this method
Include:
Using one of nuclear magnetic resonance, CT and ultrasonic examination or a variety of methods, collects complete real human body whole body and hold
Volume data, and the volume data is utilized, building includes the digital three-dimemsional model of bony structure and soft tissue；
The operating platform characteristic in reality scene is acquired, and establishes world coordinates on the basis of the operating platform
System；The coordinate of the digital three-dimemsional model is transformed into world coordinate system, the digital three-dimemsional model is rotated, makes total
The human body central axes of word threedimensional model and operating platform axis parallel；Obtain after rotation in digital three-dimemsional model y value most
Small coordinate points, and operating platform plane described in plane where it is registrated, to realize the digital three-dimemsional model
It is matched to the operation of the operating platform plane.
Main perspective position coordinates are obtained in real time, and calculate spin matrix peace of the main perspective relative to world coordinate system origin
The amount of shifting to, and angles of display of the digital three-dimemsional model on main perspective HMD projection screen is updated with this；
Real-time tracking hand motion obtains surgical instrument position delta data, and according in surgical instrument and threedimensional model
The relative positional relationship respectively organized applies corresponding force feedback to hand by tactilely-perceptible gloves.
The digital three-dimemsional model including bony structure and soft tissue is established according to living body whole body volume data, so that people
Body three-dimensional models are accurate and can clearly reflect the information such as human skeleton, muscle, blood vessel, nerve, aobvious in conjunction with mixed reality head
Dynamic teaching, dynamic demonstration, real-time communication and the functions such as mostly hold a consultation can be realized.The coordinate of digital three-dimemsional model is converted
Into the world coordinate system established on the basis of the operating platform, and it is matched to the operating platform plane, reaches empty
Quasi- object is placed in the effect in plane naturally.When this planar structure is detached from sight, digital three-dimemsional model also will be in the visual field
In disappear, when again identifying that planar structure, it will generate digital three-dimemsional model again, the atmosphere so that construction true to nature is performed the operation
It encloses.According to the relative positional relationship respectively organized in surgical instrument and threedimensional model to the corresponding force feedback of hand application, Neng Gourang
Operator's sense of reality operating force is known from experience hand tactile response of the surgical instrument when touching different tissues organ, is mentioned
High operative training validity.
It is existing it is most of to establish the data source of threedimensional model be mostly data mapping, can also although CT interference is less
To reflect the solid shape of bone from multiple planar observation institutional frameworks, but its radiation is larger, from top to bottom to the same person
Scanning obtains whole body data can cause irreversible injury to current human.Nuclear magnetic resonance can arbitrarily do cutting for any angle
Layer, and it is radiationless.It is better than the inspection of the soft tissues such as CT, especially joint, muscle, central nervous system to the susceptibility of pathological tissues
The structure that comes to an end is apparent.But to the performance of the details of lung, skeletal structure, fracture etc., the display effect of nuclear magnetic resonance is not so good as
CT.Ultrasonic examination (B ultrasound) is low with intensity, frequency is high, to human zero damage, no pain, display methods multiplicity and it is famous, it is especially right
The observation of the haemodynamics of the detection and cardiovascular internal organs of human body soft tissue has its distinctive feature, such as: superficial lump, blood vessel,
Needle localization etc., but the penetration power of B ultrasound is weaker, is difficult to detect to gassiness sexual organ, such as lung, intestines, lesion is smaller or acoustic resistance
When robust is little, it is also difficult to be shown on ultrasonogram, such as: the tumor tissues of 1cm or so are also not easy to detect, and have certain limitation
Property.
Therefore the present invention combines a variety of data sources, constructs human 3d model jointly, learns from other's strong points to offset one's weaknesses, model is true to nature.
As a preferred embodiment, further, the complete real human body whole body volume data of the collection, and utilize
The volume data, building include the digital three-dimemsional model of bony structure and soft tissue, comprising the following steps:
The setting flag object on real human body is adopted using one of nuclear magnetic resonance, CT and ultrasonic examination or multiple means
Collection includes the multiple groups two-dimensional image sequence of the marker；
Three-dimensional reconstruction is carried out respectively according to the multiple groups two-dimensional image sequence；
Data fusion and registration are carried out according to the position of marker, building includes the digitlization of bony structure and soft tissue
Threedimensional model.
The material of the setting flag object on real human body, the marker can be different, but is carrying out core respectively
When magnetic resonance, CT and ultrasonic examination, the position of same marker setting must be identical, to carry out data fusion and registration
When, there can be specific object of reference, so that data fusion operation can be gone on smoothly.
On the basis of the above embodiments, further, the method for the present invention is matched to by the digital three-dimemsional model
After the operating platform plane, further includes:
At least one is obtained in real time from view position coordinate；
According to described from view position coordinate, the spin matrix from visual angle relative to world coordinate system origin and translation are calculated
Vector, and the digital three-dimemsional model is updated from the angles of display on the wear-type visual device projection screen of visual angle with this.
Multiple participants can be set in the present invention, i.e., by from visual angle wear-type visual device field observation with main perspective
Carry out the operating process of the operator of operative training.In the digitlization shown from the wear-type visual device projection screen of visual angle
Threedimensional model is equally matched with operating platform plane, to achieve the effect that dummy object is placed in plane naturally.It is real
When obtain from view position coordinate, that is, sat according to the world is calculated from the video camera content of shooting of visual angle wear-type visual device
Mark is interior camera position coordinate, then obtains the view position coordinate from view person according to coordinate transform, and according to institute
The spin matrix and translation vector from view position coordinate relative to world coordinate system origin are stated, the Digital Three-Dimensional mould is updated
Type is from the angles of display on the wear-type visual device projection screen of visual angle.In this way can simultaneously for main perspective operator with
And a true surgical environments are built jointly from the participant at visual angle.Operative training tends to really, so that training effect is more
It is good.
On the other hand, the present invention also provides a kind of virtual operation training system based on mixed reality, as shown in Fig. 2, packet
It includes:
Data acquisition unit, for collecting complete real human body whole body volume data；
Data processing unit, for carrying out three-dimensional reconstruction using the volume data, building includes bony structure and soft
The digital three-dimemsional model of tissue；It is for acquiring the operating platform characteristic in reality scene, and with the operating platform
Benchmark establishes world coordinate system；For the coordinate of the digital three-dimemsional model to be transformed into world coordinate system, and will be described
Human 3d model is matched to the operating platform plane；
Main perspective wear-type visual device with video camera, for shooting the video pictures in camera coverage, and will
The video pictures are sent to the data processing unit；The digitlization three rebuild for data processing unit described in real-time display
Dimension module；
Data processing unit, the video pictures for being shot according to the main perspective wear-type visual device calculate main perspective
Position is used for basis for calculating spin matrix and translation vector of the main view Angle Position relative to world coordinate system origin
The spin matrix and translation vector update the digital three-dimemsional model on main perspective wear-type visual device projection screen
Angles of display；
Tactilely-perceptible gloves obtain surgical instrument position delta data for acquiring hand motion；
Data processing unit, for according to acquisition hand motion, surgical instrument position delta data calculate surgical instrument with
The relative positional relationship respectively organized in threedimensional model, and the tactilely-perceptible gloves are controlled with this to hand to apply corresponding power anti-
Feedback.
Further, which further includes, at least one with video camera from visual angle wear-type visual device, for shooting
Video pictures in camera coverage, and the video pictures are sent to the data processing unit；For real-time display institute
State the digital three-dimemsional model of data processing unit reconstruction.
The data processing unit is also used to be calculated according to the video pictures shot from visual angle wear-type visual device
It is used from view position for calculating the spin matrix and translation vector from view position relative to world coordinate system origin
It is projected in updating the digital three-dimemsional model according to the spin matrix and translation vector from visual angle wear-type visual device
Angles of display on screen.
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (9)
1. a kind of virtual operation training method based on mixed reality characterized by comprising
It collects complete real human body whole body volume data, and utilizes the volume data, building includes bony structure and soft
The digital three-dimemsional model of tissue；
The operating platform characteristic in reality scene is acquired, and establishes world coordinate system on the basis of the operating platform；It will
The coordinate of the digital three-dimemsional model is transformed into world coordinate system, and the digital three-dimemsional model is matched to the behaviour
Make platform plane；
Main perspective position coordinates are obtained in real time, and are calculated main perspective relative to the spin matrix of world coordinate system origin and be translated towards
Amount, and angles of display of the digital three-dimemsional model on main perspective wear-type visual device projection screen is updated with this；
Real-time tracing hand motion obtains surgical instrument position delta data, and according to surgical instrument and digital three-dimemsional model
In the relative positional relationship respectively organized by tactilely-perceptible gloves apply corresponding force feedback to hand.
2. a kind of virtual operation training method based on mixed reality according to claim 1, which is characterized in that described true
The acquisition of Whole Body volume data is from one of nuclear magnetic resonance, CT and ultrasonic examination or a variety of.
3. a kind of virtual operation training method based on mixed reality according to claim 2, which is characterized in that the receipts
Collect complete real human body whole body volume data, and utilize the volume data, building includes bony structure and soft tissue
Digital three-dimemsional model, comprising the following steps:
The setting flag object on living body, using one of nuclear magnetic resonance, CT and ultrasonic examination or multiple means acquisition include
The multiple groups two-dimensional image sequence of the marker；
Three-dimensional reconstruction is carried out respectively according to the multiple groups two-dimensional image sequence；
Data fusion and registration are carried out according to the position of marker, building includes the Digital Three-Dimensional of bony structure and soft tissue
Model.
4. a kind of virtual operation training method based on mixed reality according to claim 1, which is characterized in that by the number
Word threedimensional model is matched to the operating platform plane, comprising:
The digital three-dimemsional model is rotated, so that the human body central axes of digital three-dimemsional model and operating platform central axes are flat
Row；
Obtain after rotation the smallest coordinate points of y value in digital three-dimemsional model, and by operating platform plane described in its place plane
It is registrated, to realize the operation that the digital three-dimemsional model is matched to the operating platform plane.
5. a kind of virtual operation training method based on mixed reality according to claim 1, which is characterized in that will be described
After digital three-dimemsional model and the operating platform Plane association, further includes:
At least one is obtained in real time from view position coordinate；
According to described from view position coordinate, calculating relative to the spin matrix of world coordinate system origin and is translated towards from visual angle
Amount, and the digital three-dimemsional model is updated from the angles of display on the wear-type visual device projection screen of visual angle with this.
6. a kind of virtual operation training system based on mixed reality characterized by comprising
Data acquisition unit, for collecting complete real human body whole body volume data；
Data processing unit, for carrying out three-dimensional reconstruction using the volume data, building includes bony structure and soft tissue
Digital three-dimemsional model；For extracting the operating platform characteristic in reality scene, and on the basis of the operating platform
Establish world coordinate system；For the coordinate of the digital three-dimemsional model to be transformed into world coordinate system, and by the number
Change threedimensional model and the operating platform Plane association；
Main perspective wear-type visual device with video camera, couples with the data processing unit, for shooting video camera view
Video pictures in open country, and the video pictures are sent to the data processing unit；At data described in real-time display
Manage the digital three-dimemsional model that unit is rebuild；
Data processing unit, the video pictures for being shot according to the main perspective wear-type visual device calculate main perspective position
It sets, for calculating spin matrix and translation vector of the main view Angle Position relative to world coordinate system origin, for according to institute
It states spin matrix and translation vector updates the digital three-dimemsional model on main perspective wear-type visual device projection screen
Angles of display；
Tactilely-perceptible gloves obtain surgical instrument position delta data for acquiring hand motion；
Data processing unit, for calculating surgical instrument and three-dimensional according to acquisition hand motion, surgical instrument position delta data
The relative positional relationship respectively organized in model, and the tactilely-perceptible gloves are controlled with this and apply corresponding force feedback to hand.
7. a kind of virtual operation training system based on mixed reality according to claim 6, which is characterized in that the living body
The acquisition of whole body volume data is from one of nuclear magnetic resonance, CT and ultrasonic examination or a variety of.
8. a kind of virtual operation training system based on mixed reality according to claim 6, which is characterized in that the system is also
Including, at least one with video camera from visual angle wear-type visual device, couple with the data processing unit, for shooting
Video pictures in camera coverage, and the video pictures are sent to the data processing unit；For real-time display institute
State the digital three-dimemsional model of data processing unit reconstruction.
9. a kind of virtual operation training system based on mixed reality according to claim 8, which is characterized in that the data
Processing unit is also used to be calculated according to the video pictures shot from visual angle wear-type visual device from view position, be used for
The spin matrix and translation vector from view position relative to world coordinate system origin is calculated, for according to the spin moment
Battle array and translation vector update the digital three-dimemsional model from the angles of display on the wear-type visual device projection screen of visual angle.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811378127.1A CN109410680A (en) | 2018-11-19 | 2018-11-19 | A kind of virtual operation training method and system based on mixed reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811378127.1A CN109410680A (en) | 2018-11-19 | 2018-11-19 | A kind of virtual operation training method and system based on mixed reality |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109410680A true CN109410680A (en) | 2019-03-01 |
Family
ID=65474134
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811378127.1A Pending CN109410680A (en) | 2018-11-19 | 2018-11-19 | A kind of virtual operation training method and system based on mixed reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109410680A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109907826A (en) * | 2019-04-12 | 2019-06-21 | 江西省人民医院 | Coronary artery disease Simulated therapy system based on 3D model and VR technology |
| CN110070788A (en) * | 2019-03-18 | 2019-07-30 | 叶哲伟 | A kind of human body 3D meridian point method for visualizing based on mixed reality |
| CN110335516A (en) * | 2019-06-27 | 2019-10-15 | 王寅 | A kind of VR openheart surgery simulation system and its analogy method |
| CN110459083A (en) * | 2019-08-22 | 2019-11-15 | 北京众绘虚拟现实技术研究院有限公司 | A kind of augmented reality operation on oral cavity skill training simulator of visual-tactile fusion |
| CN110689792A (en) * | 2019-11-19 | 2020-01-14 | 南方医科大学深圳医院 | Ultrasonic examination virtual diagnosis training system and method |
| CN110807968A (en) * | 2019-11-28 | 2020-02-18 | 上海褚信医学科技有限公司 | Puncture operation teaching system, realization method, teaching terminal and teaching equipment |
| CN111276032A (en) * | 2020-02-29 | 2020-06-12 | 中山大学中山眼科中心 | Virtual operation training system |
| CN111658142A (en) * | 2019-03-07 | 2020-09-15 | 重庆高新技术产业开发区瑞晟医疗科技有限公司 | MR-based focus holographic navigation method and system |
| CN111768495A (en) * | 2020-06-28 | 2020-10-13 | 上海交通大学医学院附属第九人民医院 | Limbs fracture manual reduction training method |
| CN111882936A (en) * | 2020-07-18 | 2020-11-03 | 纽智医疗科技(苏州)有限公司 | Operation rehearsal and teaching system based on force feedback and virtual reality |
| CN112057139A (en) * | 2020-07-31 | 2020-12-11 | 常州锦瑟医疗信息科技有限公司 | Operation positioning system based on mixed reality technology |
| CN112309215A (en) * | 2020-11-11 | 2021-02-02 | 黑龙江中医药大学 | Demonstration system for clinical medicine internal medicine teaching and control method thereof |
| WO2021047396A1 (en) * | 2019-09-10 | 2021-03-18 | 腾讯科技（深圳）有限公司 | Image processing method and apparatus, electronic device and computer-readable storage medium |
| CN113035038A (en) * | 2021-03-29 | 2021-06-25 | 安徽工业大学 | Virtual orthopedic surgery exercise system and simulation training method |
| CN113971896A (en) * | 2021-11-17 | 2022-01-25 | 苏州大学 | Operation training system and training method |
| CN114913309A (en) * | 2022-05-28 | 2022-08-16 | 深圳大学 | High-simulation surgical operation teaching system and method based on mixed reality |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102592484A (en) * | 2010-11-16 | 2012-07-18 | 韩国电子通信研究院 | Reconfigurable platform management apparatus for virtual reality-based training simulator |
| CN204542390U (en) * | 2015-04-17 | 2015-08-12 | 中国科学院重庆绿色智能技术研究院 | A kind of force feedback surgery operation robot control system based on augmented reality |
| CN105264459A (en) * | 2012-09-27 | 2016-01-20 | 沉浸式触感有限公司 | Haptic augmented and virtual reality system for simulation of surgical procedures |
| WO2017106924A1 (en) * | 2015-12-23 | 2017-06-29 | Deakin University | Tactile medical examination simulator |
| CN107613897A (en) * | 2015-10-14 | 2018-01-19 | 外科手术室公司 | The surgical navigation of augmented reality |
| CN108472095A (en) * | 2015-12-29 | 2018-08-31 | 皇家飞利浦有限公司 | The system of virtual reality device, controller and method are used for robotic surgical |
| CN208031283U (en) * | 2017-03-29 | 2018-11-02 | 上海霖晏医疗科技有限公司 | A kind of operation guiding system |
-
2018
- 2018-11-19 CN CN201811378127.1A patent/CN109410680A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102592484A (en) * | 2010-11-16 | 2012-07-18 | 韩国电子通信研究院 | Reconfigurable platform management apparatus for virtual reality-based training simulator |
| CN105264459A (en) * | 2012-09-27 | 2016-01-20 | 沉浸式触感有限公司 | Haptic augmented and virtual reality system for simulation of surgical procedures |
| CN204542390U (en) * | 2015-04-17 | 2015-08-12 | 中国科学院重庆绿色智能技术研究院 | A kind of force feedback surgery operation robot control system based on augmented reality |
| CN107613897A (en) * | 2015-10-14 | 2018-01-19 | 外科手术室公司 | The surgical navigation of augmented reality |
| WO2017106924A1 (en) * | 2015-12-23 | 2017-06-29 | Deakin University | Tactile medical examination simulator |
| CN108472095A (en) * | 2015-12-29 | 2018-08-31 | 皇家飞利浦有限公司 | The system of virtual reality device, controller and method are used for robotic surgical |
| CN208031283U (en) * | 2017-03-29 | 2018-11-02 | 上海霖晏医疗科技有限公司 | A kind of operation guiding system |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111658142A (en) * | 2019-03-07 | 2020-09-15 | 重庆高新技术产业开发区瑞晟医疗科技有限公司 | MR-based focus holographic navigation method and system |
| CN110070788A (en) * | 2019-03-18 | 2019-07-30 | 叶哲伟 | A kind of human body 3D meridian point method for visualizing based on mixed reality |
| CN109907826A (en) * | 2019-04-12 | 2019-06-21 | 江西省人民医院 | Coronary artery disease Simulated therapy system based on 3D model and VR technology |
| CN110335516A (en) * | 2019-06-27 | 2019-10-15 | 王寅 | A kind of VR openheart surgery simulation system and its analogy method |
| CN110459083A (en) * | 2019-08-22 | 2019-11-15 | 北京众绘虚拟现实技术研究院有限公司 | A kind of augmented reality operation on oral cavity skill training simulator of visual-tactile fusion |
| CN110459083B (en) * | 2019-08-22 | 2020-08-04 | 北京众绘虚拟现实技术研究院有限公司 | Vision-touch fused augmented reality oral surgery skill training simulator |
| US11386799B1 (en) | 2019-08-22 | 2022-07-12 | Beijing Unidraw Vr Technology Researech Institute Co., Ltd | Vision-haptics fused augmented reality simulator for dental surgical skill training |
| US11538229B2 (en) | 2019-09-10 | 2022-12-27 | Tencent Technology (Shenzhen) Company Limited | Image processing method and apparatus, electronic device, and computer-readable storage medium |
| WO2021047396A1 (en) * | 2019-09-10 | 2021-03-18 | 腾讯科技（深圳）有限公司 | Image processing method and apparatus, electronic device and computer-readable storage medium |
| CN110689792A (en) * | 2019-11-19 | 2020-01-14 | 南方医科大学深圳医院 | Ultrasonic examination virtual diagnosis training system and method |
| CN110807968A (en) * | 2019-11-28 | 2020-02-18 | 上海褚信医学科技有限公司 | Puncture operation teaching system, realization method, teaching terminal and teaching equipment |
| CN111276032A (en) * | 2020-02-29 | 2020-06-12 | 中山大学中山眼科中心 | Virtual operation training system |
| CN111768495A (en) * | 2020-06-28 | 2020-10-13 | 上海交通大学医学院附属第九人民医院 | Limbs fracture manual reduction training method |
| CN111882936A (en) * | 2020-07-18 | 2020-11-03 | 纽智医疗科技(苏州)有限公司 | Operation rehearsal and teaching system based on force feedback and virtual reality |
| CN112057139A (en) * | 2020-07-31 | 2020-12-11 | 常州锦瑟医疗信息科技有限公司 | Operation positioning system based on mixed reality technology |
| CN112309215A (en) * | 2020-11-11 | 2021-02-02 | 黑龙江中医药大学 | Demonstration system for clinical medicine internal medicine teaching and control method thereof |
| CN113035038A (en) * | 2021-03-29 | 2021-06-25 | 安徽工业大学 | Virtual orthopedic surgery exercise system and simulation training method |
| CN113971896A (en) * | 2021-11-17 | 2022-01-25 | 苏州大学 | Operation training system and training method |
| CN113971896B (en) * | 2021-11-17 | 2023-11-24 | 苏州大学 | Surgical training system and training method |
| CN114913309A (en) * | 2022-05-28 | 2022-08-16 | 深圳大学 | High-simulation surgical operation teaching system and method based on mixed reality |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109410680A (en) | A kind of virtual operation training method and system based on mixed reality | |
| Spitzer et al. | The Visible Human Dataset: the anatomical platform for human simulation | |
| Ungi et al. | Perk Tutor: an open-source training platform for ultrasound-guided needle insertions | |
| Meller | A typology of simulators for medical education | |
| EP1438703A1 (en) | Medical procedure training system | |
| JP2007528029A (en) | Medical training and evaluation apparatus and method | |
| CN105225562A (en) | A kind of general practitioner's training checking system and method | |
| CN113035038A (en) | Virtual orthopedic surgery exercise system and simulation training method | |
| CN109509555A (en) | A kind of surgical operation preview appraisal procedure and system based on 3-dimensional image | |
| CN115472051A (en) | Medical student operation simulation dummy and use method | |
| Riener et al. | VR for medical training | |
| Seo et al. | Anatomy builder VR: Embodied VR anatomy learning program to promote constructionist learning | |
| RU2687564C1 (en) | System for training and evaluating medical personnel performing injection and surgical minimally invasive procedures | |
| Tahmasebi et al. | A framework for the design of a novel haptic-based medical training simulator | |
| CN113893033B (en) | Pulmonary percutaneous puncture navigation method and system | |
| CN108711313A (en) | Gastroscopy self-aid learning method | |
| Wei et al. | Virtual Reality technology in modern medicine | |
| Lioce et al. | 3D printed anatomical models for training nursing students | |
| RU143299U1 (en) | MEDICAL PROCEDURE MODELING SYSTEM (OPTIONS) | |
| Jang et al. | How direct interaction in a virtual reality program aids in developing an internal representation of a complex 3-D structure | |
| Bichlmeier et al. | The visible korean human phantom: Realistic test & development environments for medical augmented reality | |
| Wasif et al. | Training in temporal bone surgery: A review of current practices | |
| Acosta et al. | Design and Manufacture of a Training System for Ventriculostomy | |
| Klempous et al. | Application of simulation techniques in a virtual laparoscopic laboratory | |
| Heng et al. | Visible Human Based Virtual Medicine. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190301 |
|
| RJ01 | Rejection of invention patent application after publication |