CN101405769A - Systems and methods for collaborative interactive visualization of 3D data sets over a network ('DextroNet') - Google Patents

Systems and methods for collaborative interactive visualization of 3D data sets over a network ('DextroNet') Download PDF

Info

Publication number
CN101405769A
CN101405769A CNA2007800018041A CN200780001804A CN101405769A CN 101405769 A CN101405769 A CN 101405769A CN A2007800018041 A CNA2007800018041 A CN A2007800018041A CN 200780001804 A CN200780001804 A CN 200780001804A CN 101405769 A CN101405769 A CN 101405769A
Authority
CN
China
Prior art keywords
user
data
remote
student
equipment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CNA2007800018041A
Other languages
Chinese (zh)
Inventor
路易斯·塞拉德尔莫利诺
吴令家
周泸萍
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bracco Imaging SpA
Original Assignee
Bracco Imaging SpA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bracco Imaging SpA filed Critical Bracco Imaging SpA
Publication of CN101405769A publication Critical patent/CN101405769A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Processing Or Creating Images (AREA)

Abstract

The present invention relates to the interactive visualisation of 3D data sets and more particularly to the collaborative interactive visualisation of one or more 3D data sets by multiple parties over a network. The suggested system being configured to receive, over a communications link, positional data for one or more remote probes of one or more remote machines, to generate, for display on a local display, a combined 3D scene comprising one remote probe and the 3D image, to manipulate the 3D image by a user local to the apparatus, and send, over a communications link, data regarding said manipulations by said user local to the apparatus to allow the remote machine to display a combined 3D scene comprising an image of the local probe performing manipulations on said 3D image.

Description

Be used for the system and method that the collaborative interactive via the 3-D data set of network (" DextroNet ") develops
Cross reference
The right of priority of the U.S. Provisional Patent Application that comprises by reference below the application requires: (i) title of submitting on Dec 31st, 2005 is the U.S. Provisional Patent Application No.60755658 of " Systems and methods forcollaborative interactive visualization over a network (" DextroNet ") "; (ii) the title of submitting on September 19th, 2006 is the U.S. Provisional Patent Application No.60845654 of " Methods and systems for interacting with a 3D visualization systemusing a 2D interface (" DextroLap ") "; And the title of (iii) submitting on Dec 19th, 2006 is the U.S. Provisional Patent Application of " Systems and methods forcollaborative interactive visualization of 3D data sets over a network (" DextroNet ") " (applicant keep revise this application with correct application number and this are provided right of the submission date of temporary patent application after a while).In addition, also reference of the application: (i) title of submitting on April 27th, 2004 is the common unsettled U.S. utility patented claim No.10/832 of " An augmented realitysurgical navigation system based on a camera mounted on a pointingdevice (" camera probe ") ", 902; And the title of (ii) submitting on July 1st, 2005 is the unsettled U.S. utility patented claim No.11/172 of " system and method forthree-dimensional space management and visualization of ultrasound data (" SonoDex ") ", 729.
Technical field
The present invention relates to the mutual development of three-dimensional (" 3D ") data set, and relate more specifically to develop by the collaborative interactive that uses a plurality of platforms to carry out one or more 3D data sets in many ways via network.
Background technology
Tradition is loaded in special purpose workstation or the computing machine by providing 3D data set (or 3D data set that is generated by a plurality of 2D images), finishes the three-dimensional of 3D data set and develops.Usually, the unique user 3D data set that in single special purpose workstation, alternatively develops.For example, this Dextroscope that can make in Volume Interactions Pte company by Singapore TMOn finish.Dextroscope TMBe to show volume three-dimensionally and allow the user to carry out the high-end true mutual toning system of complete 3D control.The also DEX-Ray that provides by the VolumeInteractions Pte company of Singapore TMSystem is with the special-purpose 3D mutual toning system of real-time video with the 3D scan-data combination of common registration.DEX-Ray TM" back " that allows the surgeon of user-normally-come to be combined into the actual art of combination picture " seeing " open country by the virtual objects that will be partitioned into from scan-data before the art and real-time video.Yet, when using tradition even high-end such as Dextroscope TMOr DEX-Ray TMMutual 3D toning system the time, even a plurality of display systems are arranged, and a lot of personnel can check those demonstrations by standing in the actual user next door, once also can only control development by user only.Thus, in this system, in to the process of handling by the 3D data set of anyone inspection except the personnel that control, not real the participation or cooperation.
For example, DEX-Ray TMSystem can be used for the surgery planning such as the complex operations of neurosurgery.In this surgery planning, as described in the Camera Probe application of quoting before, the team of neurosurgical doctor and Ta can obtain the preceding scan-data of art, is partitioned into interested object from these data, and the layout data of adding the route that will use such as intra-operative.In addition, as further in Camera Probe, describing, can be set to " mark " at a plurality of points that provide the 3D data centralization.Then, run through operation, the tip of hand-held detector of can follow the tracks of and read continuously (via vision or even auditory information prompting) user is with respect to the position of these marks.
In addition, often be desirably in the 3D input that has when operation takes place from operative site.In this scene, for example can follow the tracks of one or more operating theater instruments, and can use develop better mutual between surgeon and the patient of augmented reality by adhering to tracking ball.Therefore, described in Camera Probe, in case the operation beginning, the combination image of can generate and develop real time data and virtual objects.Yet normal conditions are that the surgeon does not dynamically adjust shown virtual objects in his operation (comprise and change the point of being appointed as mark).This is because he concentrates on the over-all properties of optimizing in the development and utilizing the 3D toning system thus seldom if having time simultaneously in operation.
Use DEX-Ray TMDuring the type system, can before operation, specify some interested virtual objects,, and can show those objects at intra-operative such as near the tumour or tumour self crucial neural.The situation of gauge point also is like this.As noted above, Camera Probe describes the gauge point can also how to specify definition quantity, and how to run through surgical procedure and come the most advanced and sophisticated dynamic distance to those objects of tracking detector.Though the surgeon can be at intra-operative aignment mark point in theory, they so do not do usually, because the surgeon is busy with actual operation and has seldom the time putting down work and optimizing the augmented reality parameter.
The surgeon does not also have other reasons at intra-operative and virtual data usually alternately.At first, the feasible on-the-spot mutual hell to pay in most navigational system interfaces.The second, the navigational system interface is non-sterile (non-sterile), so the surgeon will need to carry out adjusting by indication nurse or technician.At DEX-Ray TMIn the system, to develop be feasible (described in Camera Probe) though revise through air by mobile detector only, thereby the surgeon can directly revise display parameter when maintenance is aseptic, but often more advantageously need not to revise in operation the development environment.Under any circumstance, special-purpose expert (dedicated specialist) can assist them to carry out this development, so that these developments seem the most perfect.
Similarly, in addition when the mutual toning system of 3D of use standard such as Dextroscope TMDuring for surgery assistance, guiding or planning, often be difficult in a physics place, coordinate all personnel interested.Sometimes, for example, the surgeon involves in the complicated operation that is difficult to pre-planning fully, such as the conjoined twins' that share certain brain structure separation.In this operation, almost must before the intra-operative of actual (and very long) is continuously with reference to art, scan, and be come advisory group member or even other experts according to what see when the actual exposed brain.Though the mutual 3D development of scan-data often is the best mode that is used for analyzing operation before the art, even all related sides are in the physics place, operation team also is difficult to assemble around the display that is centered around this system.In addition, situation is complicated more, and team is big more at a distance of geographic distance, and then the benefit that advisory data is developed before the art is just difficult more.
Finally, if the use or the operation technique of two mutual 3D data set toning systems of remote parties expectation discussion, when indicating another to carry out this use such as one of them, two sides need side by side to check 3D data set and the other side's manipulation.
Need to allow a plurality of remote parties to come in the mutual toning system of 3D side by side the 3D data set is operated just as these remote parties entities show up (physically present) thus in this area.
Summary of the invention
Exemplary system and method are provided, utilize these exemplary system and method, a plurality of personnel in the long-range physics place can be basically collaborative interactive ground development 3D data set side by side.In illustrative embodiments of the present invention, the main workstation and the one or more remote work station that connect via data network for example can be arranged.Providing main workstation can for example be augmented reality operation guiding system or 3D toning system, and each workstation can load identical 3D data set.In addition, providing workstation can make up real time imagery and the 3D data that obtain before, such as real-time or prerecorded video, or such as the information that provides by the 3D ultrasonic development system that manages.User at the remote work station place can carry out diagnosis or the treatment procedure (procedure) that provides, for example surgical navigational or fluoroscopic examination, maybe can receive instruction from another user of main workstation, at this main workstation place, use the total 3D data set of preserving that lecture notes (lecture) are shown.User at main workstation place can for example see the virtual tool that used by each long-distance user and their motion, and each long-distance user can for example see main user's virtual tool and it the corresponding effect for the data set at remote work station place.For example, it is the mode of the detector that is associated with this remote work station that remote work station can adopt virtual tool just as main workstation, via the virtual control panel of described local machine, show the virtual tool of the main workstation that the 3D data set at described remote work station place is operated.In illustrative embodiments of the present invention, each user's virtual tool can utilize their IP address, different colours and/or other diacritics (designation) to represent.In illustrative embodiments of the present invention, data network can be low bandwidth or high bandwidth.In low bandwidth embodiment, the 3D data set can be pre-loaded on each user's the workstation, and only sends main user's the motion of virtual tool and the manipulation of data set via network.In high bandwidth embodiment, for example, also can send via network such as video, realtime graphic ultrasonic or fluoroscopic image.
Description of drawings
Fig. 1 illustrates the exemplary procedure flow process that is used for illustration teacher type workstation of the illustrative embodiments according to the present invention;
Fig. 2 is the system-level diagram of a plurality of illustration workstations of connecting of the across a network of the illustrative embodiments according to the present invention;
Fig. 3 illustrates the exemplary procedure flow process at the illustration student work station of the illustrative embodiments according to the present invention;
Fig. 4 illustrates the exemplary procedure flow process of illustration surgeon's workstation of the illustrative embodiments according to the present invention;
Fig. 5 illustrates the exemplary procedure flow process of the illustration development assistant workstation of the illustrative embodiments according to the present invention;
Fig. 6 illustrates standard (throwing off (the disengaged)) view of the illustrative surgeon of illustrative embodiments according to the present invention;
Fig. 7 illustrate the illustrative embodiments according to the present invention illustrative surgeon involve in (engaged) view; And
Fig. 8 illustrate according to the present invention illustrative embodiments illustrative development assistant's view;
Fig. 9 illustrates illustrative teacher-student's example framework of the illustrative embodiments according to the present invention;
Figure 10 (a)-(e) illustrates the teacher who connects via illustration DextroNet of according to the present invention illustrative embodiments and two students' illustration view;
Figure 11-13 amplifies and Figure 10 (a)-(c) sequentially is shown;
Figure 14 (a)-(c) illustrates under locking (lock-on) pattern illustration view of the 3D data set of being seen by the illustration teacher of Figure 10 and two students;
Figure 15-17 amplifies and sequentially illustrates the view of Figure 14 (a)-(c);
Figure 18 (a)-(c) illustrates the illustration view of the 3D data set of Figure 14 (a)-(c) respectively, and wherein, the teacher measures, and student 1 has switched to the disengagement view mode;
Figure 19-21 amplifies and sequentially illustrates the view of Figure 18 (a)-(c);
Figure 22 (a)-(c) is illustrated in the illustration view that professor has moved his pen Figure 18 (a)-(c) afterwards respectively;
Figure 23-25 amplifies and sequentially illustrates the view of Figure 22 (a)-(c);
Figure 26-30 illustrates the illustration sequential steps of seeing from teacher's viewpoint of according to the present invention illustrative embodiments, and wherein, two students participate in the networking session;
Figure 31 illustrates two illustration views of the development assistant of the illustrative embodiments according to the present invention;
Figure 32-33 amplifies and sequentially illustrates two views of Figure 31;
Figure 34 illustrates the view corresponding to the illustrative surgeon of the development assistant view of Figure 31 of according to the present invention illustrative embodiments;
Figure 35 illustrates this two side's when development assistant and surgeon locate providing on the illustration incarnation object of according to the present invention illustrative embodiments view respectively;
Figure 36-37 amplifies and sequentially illustrates the view of Figure 35 respectively;
The illustration development assistant that Figure 38 illustrates the illustrative embodiments according to the present invention assists the surgeon to locate point on the illustration object;
Figure 39-40 amplifies and sequentially illustrates the view of Figure 38 respectively;
Figure 41 illustrates the assistant of the point on the co-positioned illustration object of the illustrative embodiments according to the present invention and surgeon's each other illustration view;
Figure 42-43 amplifies and sequentially illustrates the view of Figure 41 respectively;
Figure 44 illustrates the illustration fluoroscopic image;
Figure 45 illustrates illustrative intervention cardiologist's view;
Figure 46 illustrates the view corresponding to the illustration development assistant of Figure 45 of according to the present invention illustrative embodiments;
Figure 47 illustrates the illustration pip image that the illustration development assistant by Figure 46 of according to the present invention illustrative embodiments generates;
Figure 48 illustrates other illustrations 3D view of development assistant of Figure 46-47 of the illustrative embodiments according to the present invention;
Figure 49 illustrates the view of the alternative illustration intervention cardiologist of illustrative embodiments according to the present invention;
Figure 50 illustrates the illustration phase one of illustration role's hand-off process of the illustrative embodiments according to the present invention;
Figure 51 illustrates the illustration subordinate phase of illustration role's hand-off process of the illustrative embodiments according to the present invention;
Figure 52 illustrates the illustration the 3rd and the terminal stage of illustration role's hand-off process of the illustrative embodiments according to the present invention;
Figure 53 (a) and (b) the illustration data transmit queue of illustrative embodiments in illustration teacher (or main user) system according to the present invention be shown;
Figure 54 illustrates the illustration information updating form of the illustrative embodiments according to the present invention;
Figure 55 (a)-(c) illustrates the illustration updating message that is used for illustration control panel, widget (widget) and instrument of the illustration message format of illustrative embodiments use Figure 54 according to the present invention;
Figure 56 illustrates the illustration message about file transfer of the illustration message format of illustrative embodiments use Figure 54 according to the present invention;
Figure 57 illustrates the alternative illustration information updating form of the illustrative embodiments according to the present invention; And
Figure 58 (a) and (b) the illustration file transfer message that according to the present invention illustrative embodiments is used the illustration message format of Figure 54 is shown.
Embodiment
1. general introduction
In illustrative embodiments of the present invention, can connect the mutual toning system of polytype 3D via data network, so that personnel away from each other can substantially side by side operate identical 3D data set for multiple purpose.In illustrative embodiments of the present invention, two or more personnel can be positioned at away from each other and the 3D data set that provides can be loaded on their workstation.Usually, this illustrative embodiments expection " main user " and one or more " long-distance users ".The participant can be the co-worker of surgery planning engineering for example, for example plans the doctor team that conjoined twins separate, or they can be for example teacher or lecturer (lecturer) or one group of student or person on the scene (attendee).In addition, for example, the participant can comprise: (i) use operation guiding system (such as Dex-Ray TMSystem) or some other diagnosis or therapy system (such as Cathlab machine, the ultrasonic system managed etc.) patient carried out or diagnose or surgeon or other clinicians of treatment procedure; And the expert that (ii) develops, along with surgeon or clinician's propelling, dynamically revise and the virtual objects of 3D data centralization of being correlated with that develops from some remote locations, and do not bear the physical restriction of treatment or operating room.Perhaps, in education miscellaneous, specialty, inspection or customer support environment, the participant can comprise teacher and one or more student.Below, the present invention will use a plurality of possible examples or illustration to make to be used for and describe, the present invention can't help that these illustrations use any one limit.Know that function of the present invention can be applied to extensive multiple application, wherein research or polytype cooperation of meticulous (sophisticated) development of expectation or use 3D data set.
For convenience of description, a plurality of illustrative embodiments of the present invention sometimes will be called as " DextroNet ", and " DextroNet " is the expection trade mark that is used by the assignee here in conjunction with these a plurality of embodiment.
A.Dextroscope TMTo Dextroscope TMType is mutual
In illustrative embodiments of the present invention, DextroNet can for example be used for remote client's training or technical support, or the telereference between the manufacturer of the mutual toning system of 3D and its client.For example, DextroNet can be used to provide long-range with selecting (on-demand) technical support to its client by the manufacturer of the mutual toning system of 3D.Whenever in this uses, for example, when problem about the use that provides the mutual toning system of 3D that provided by this manufacturer is provided the client, can be connected with training center.In case connect, then the remote training technician can for example illustrate the instrument of providing that how to use to the client, and this provides instrument such as at Dextroscope TMOn contour editor.For fear of requiring the client to send his whole data set (potential problems of losing time when having patients ' privacy and transmission data), can on standard data set, carry out this illustration training, this standard data set loads when the remote training session begins by both party.Another service that can offer the client of the use of exploring DextroNet can for example be included as the off-line client and prepare (cutting apart) patient data.For example, in medical environment, the data that the client can ftp be associated with one or more interested patients or case, and " radiation service " portion can cut apart these data and it is sent it back the client.Be connected with this service,, can provide and do what or even the online demonstration of the remote training of the actual patient data handled about the client according to illustrative embodiments of the present invention.
In illustrative embodiments of the present invention, can also support telereference.In this embodiment, can all via DextroNet adopt in case, the case multiple be discussed remotely such as surgeon and radiologist, imaging technician, Ref Dr's expert team may approach and similar problem.For example, the radiologist can provide his view, the vascular nerves surgeon can provide his view, and cranium face expert can provide his view again, all with reference to and provide a plurality of volumetric objects of 3D data centralization " sensing " what the scan-data by case generated virtually.
B.Dextrosocpe TMTo DEX-Ray TMType is mutual
As described in the Camera Probe, can use the DEX-Ray system as operation navigation device.This equipment can be connected to the mutual 3D display system of standard via DextroNet.Thus, in illustrative embodiments of the present invention, can " outsourcing " 3D data manipulation performance remove the constraint that aseptic (sterile field) by the surgeon applies.For example, the DEX-Ray in the surgeon has patient's skull TMDuring the type detector, he can not control the interface of operation guiding system usually, unless near some personnel that he begins basic computer shout out order.Certainly, do not expect like that, because this computing machine is busy with control detector and drawing image/video usually fully yet.
Utilize the detector in the patient's skull, the surgeon attempts finding his path (just to point to zones of different around patient's brain, and can not point to the tissue that exceeds that he does not also cut), the surgeon wants help and checks other image modes, other segmenting structures (tumour, vascular, nerve (such as via diffusion tensor imaging)) and not many change position of detector.In this scene, the surgeon can utilize detector to point to, and can talk/listen, and moves into and shift out skull too continually but can not provide.In this case, usefully, help to provide different developments by another surgeon (or assistant of the professional knowledge that possesses skills).These different developments can for example illustrate the surgeon near which vascular, which neural tumour etc. that centers on.This " development assistant " can be connected to surgeon's navigational system via DextroNet.
C.Dextroscope TMTo Cathlab TMType is mutual
Perhaps, in illustrative embodiments of the present invention, be similar to DexRay TMTo Dextroscope TMMutual example can for example use the online information of the form of XRF perspective demonstration.Here, long-range assistant can for example help to be adjusted in real-time substantially x ray view (2D projection) and the 3D view (from the CT or the MR scanning of for example heart) that generates by scanning before the individuality between common registration.Because can not see vascular, require to get involved that the cardiologist comes injection of contrast medium so that opaque in the vascular several seconds with in sight with X ray.For the problematic patient of kidney injection of contrast medium not, and under any circumstance, for patient's (for health and hospital's bill are considered) makes that contrast preparation is minimum.Therefore, the development assistant can use several seconds of contrast agent flow CT before patient's X ray view and the art for example is synchronous.This information can for example present as the enhancing virtual objects above the X ray view, and this information for example can show near another monitor the X ray view.For example, heart can be beated in the X ray view, and CT can beat therewith, or only frozen moment is shown, and this frozen moment can help to guide the cardiologist to utilize conduit along (follow) correct vascular.In illustrative embodiments of the present invention, can suppose that only personnel control mutual (" the main user ") who influences the 3D data, and other participants (" long-distance user ") of connecting via DextroNet of hypothesis can check those influences but can not change data self, rather than only can check they self data viewpoint.In addition, in this illustrative embodiments, DextroNet can for example only send main user's 3D interaction detail via network, make those be performed at each remote work station or system place alternately then.This and traditional system form contrast, wherein, user sends about his mutual data to remote server, and this remote server carries out calculating and sending and sends final computed image back to then, and to remote audience the step of the final image that generates the 3D data or the sequence of operation is not shown.
The solution that these are traditional is such as the VizServer of SGI TMUnilateral network only is provided.In illustrative embodiments of the present invention, DextroNet can be multidirectional network.Such as VizServer TMScheme depend on the user only need be from the hypothesis of the final projection that provides the 3D scene.Iff needs projection (or even such as a pair of stereoprojection), then remote computer can calculate (drafting) image and the pixel that obtains is sent to the client, and the client sees the image that can not differentiate with the image that he obtains by this locality drafting then.The user can cause rotary manipulation by moving cursor, and this order can be sent to remote server, can rotate the 3D model, and this model can be drawn once more, and sends images to client computer for checking.Yet, because do not have, so this traditional scheme does not allow other users that their the 3D interactive device (such as detector, stylus etc.) of self is inserted in the 3D model with respect to the depth information of detector about model.Because all are available is pixel from a user's viewpoint, so this traditional scheme does not also allow client computer to generate separable viewpoint (below will describe).
In addition, this scheme can not accurately illustrate to the long-distance user and adopt what step to obtain final drafting.If he is interested for these interactive command sequences of actual learning, then he can not learn.What form contrast is, in illustrative embodiments of the present invention, can be at each workstation place (such as place, corresponding station in Faculty and Students, or at development assistant and surgeon's place, corresponding station, development assistant and other diagnosticians or carry out the therapist's of operation place, corresponding station for physics patient (physical patient)) obtain the 3D model, this 3D model and user's 3D (just instrument and other objects) can be made up then, and make each station can carry out the drafting of himself then.
In order remotely to influence data, in illustrative embodiments of the present invention, the 3D of main user's stylus can be arranged to alternately such as (long-range) virtual control panel with the long-distance user and carry out alternately.In this illustrative embodiments, this control panel does not need to be converted into the system command on the remote machine alternately, such as " pressing the cutting button " or " the electronic zoom slide block is slided into 2.0 " etc.But, in this illustrative embodiments, such as the image of the instrument by showing main user on long-distance user's machine, in long-distance user's the world, the main user's who provides operation can be used as the result that main user's instrument just handled and is rendered as on long-distance user's machine and just is implemented.In remote user side, when main user's instrument appears at the position of the button top on long-distance user's the virtual control panel and its state for such as " beginning action " (one of four illustration basic status of instrument) time, then can press the button then and need not specific order.By this way, in main user side or remote user side, remote tools can adopt the principle identical with local instrument to come work.Thus, realize that with using the additional system order manipulation that evokes at main user's machine place (or uses traditional VizServer TMScheme) compare, this can obtain more seamless integrated.The reason of using system level order is not, avoids the inconsistent view in remote user side.If for example with remote user side on the stylus of virtual control panel be inconsistent alternately, though then instrument still can be via the system command effect, as the long-distance user in effect or operation " B " if see that instrument can be puzzled in button " A " top when taking place.If the demonstration of tool location etc. can as mentioned above, then need not the additional system order, and instrument will be looked after self by synchronously.
Notice that the calibration of virtual control panel can change between the user on the different platform.This is because a plurality of hardware and softwares configurations of a plurality of 3D toning systems that can connect via illustration DextroNet can be arranged.For example, for a plurality of systems that provided by the assignee Volume InteractionsPte company here, DextroBeam are shown TMDo not have " reflection " calibration because there being mirror, and DextroLAP TMSystem uses mouse and does not use the 3D tracker.Therefore, in illustrative embodiments of the present invention, main user's 3D calibration can be sent to the long-distance user of all connections, and the long-distance user of these all connections can be connected via the mutual toning system platform of the multiple 3D of difference manufacturing and model usually.This calibration sends and can guarantee, for example, the 3D that is carried out by main user strikes on long-distance user's the correct button in orthochronous alternately.Therefore, when being connected to DextroNet, agreement (protocol) can for example cause on control panel and the main user side on the remote user side identical (about position, orientation, size etc.), so that main user can be replicated on the machine a plurality of long-distance users in the operation on his control panel.
When main workstation and remote work station synchronization interface and viewpoint, main workstation can be for example with the parameter of its control panel, and is following described in conjunction with Figure 55 (a), and the position of shown object and orientation, following described in conjunction with Figure 55 (b), send to remote work station.Thus, when remote work station received these parameters, it can for example use this information to upgrade himself control panel and local ground display object.In case finished this processing, then for example can use main user's instrument to handle the Long-distance Control panel, and main user and (a plurality of) long-distance user's initial viewpoint can be identical.
A plurality of example illustrated have been described in this application.Notice that in order to save vocabulary, can describe feature in detail in conjunction with " teacher-student " example, these features also can be applied to " surgeon-development assistant " example, otherwise still, and be not repeated to describe.Know, " surgeon-development assistant " example and " teacher-student " example act in the same manner, the latter with each all by the record of common registration real-time video, real time imagery (for example fluoroscopy, ultrasonic etc.) or in real time instrument positions information add the 3D data centralization to.
D.DextroNet educates (teacher-student) example
In illustrative embodiments of the present invention, can in education, use DextroNet to connect.For example, can use DextroNet to indicate the student to carry out operation and planning, or for example, such as Dextroscope TMThe use of mutual 3D toning system.In addition, can use illustration DextroNet for dissecting indication.For example, can use this to realize allowing the student be familiar with the complicated especially anatomic region of health.In addition, illustration DextroNet can also be used for pathology indication, and can be used as the complementary education instrument thus for postmortem (when the patient may be particularly useful when intra-operative is dead, and the art preplanning and the scan-data that can obtain this patient).
In illustrative embodiments of the present invention, be familiar with Dextroscope TM, DextroBeam TMOr DEX-Ray TMThe lecturer of system or their equivalence (can be such as doctor, surgeon or other experts) can be such as the student who instructs at least one remote location via DextroNet.The student can have the viewpoint in the face of 3D model identical with the lecturer, or for example can have different viewpoints.For example, the student can have a demonstration (or display window) of adopting lecturer's viewpoint, or adopts another demonstration (or display window) of different points of view.Perhaps, for example, student or lecturer can change between different viewpoints.This can be for example just as having picture-in-picture, and wherein, a width of cloth picture is long-range viewpoint, and another picture is local picture, is similar to the mode of video conference operation.In this illustrative embodiments, can all work in this locality station place of two width of cloth pictures generates (width of cloth has local parameter, and another width of cloth has remote parameter), or for example, and long-range picture can be by remote plotting, and to be similar to VizServer TMThe mode of scheme sends (only send final pixel and do not send interactive step).Thus, provide that the user can accurately see the long-distance user and seen, eliminated all feel uncertain such as " what you saw is that I see? "
In order to describe facility, the participant in this education embodiment will be called as " teacher " or " lecturer " (main user) and " student " (long-distance user).Illustrative teacher-student's example then will be described in conjunction with Fig. 1 and Fig. 3.
In illustrative embodiments of the present invention, the teacher may for example can not see student's real-time view image, is just seeing what (it can hint student's area-of-interest) but can be apprised of the student by the direction indication of student's instrument of showing in the teacher world.Though personnel can not come the accurately viewpoint of " knowing " distant station according to the outward appearance of long-range stylus, he can infer certain situation about its position according to the fact known or hypothesis.For example, stylus waits by right-handed personnel are used to refer to forward.
This exemplary scenarios is designed to allow the teacher can instruct a plurality of students.Thus, unless not needing to be interrupted the student, the teacher do not utilize voice or other signals to ask a question.Then, put question to student's demand based on this, illustration teacher can for example rotate and provide object and easily his viewpoint is alignd with student's viewpoint, because he can determine fully how the student checks object from the direction of this student's instrument.Perhaps, their role can put upside down in Faculty and Students, puts upside down the role and can for example click easily and finish by (" role's switching ") pair of buttons as described below in illustrative embodiments of the present invention.Note, in illustrative embodiments of the present invention, in teacher-student's example, the student for example can not local ground manipulating objects except some operation, this some operations do not change voxel or a plurality of voxel that belongs to special object, and these operations are such as comprising translation, rotation and/or point at objects or with the operation of object electronic zoom (changing its enlargement ratio).Instruction that this need avoid a conflict usually or mutual with the 3D data set.
Usually, this locality control that can put into practice of long-distance user (such as student, surgeon or do not take on teacher's development assistant) often is limited.This is because usually, realize that for the teacher's (or development assistant) of data set manipulation as mentioned above, just main user's mutual across a network sends also and realized on each student (or surgeon's) machine locally locally.Main user controls the long-distance user's (this locality) machine thus.In this illustrative embodiments, if allow the student with teacher's manipulation no longer meaningful ways change the local replica of his or her data set, then have confusion.For example, at the Dextroscope of standard TMOn, the user has the control of cutting apart fully for the object of 3D data centralization.He can change this and cut apart by the color lookup table of Change Example such as object.Perhaps, he can be regardless of cutting apart but only changes color or the transparency that is associated with divided object.Though the original contents that operation can not change object is planted in this back, this operation can change its photographic parameter.
Be also noted that, usually at Dextroscope TMHave a lot of instruments and operation in type system and the mutual 3D data set toning system, it is based on supposing to operate about some of the size of the object just handled.For example, have marking tools, this marking tools is operated by ray is projected nearest object surface from tool tip.If the surface of object because the local user has changed cutting apart of object more near this marking tools or further from this marking tools, serve as a teacher the marking tools and the point at objects of picking up him, when realizing on the screen the student, this marking tools may touch diverse object.In fact, on student's machine, can experience the thing on teacher's machine, object surface.Usually, boring bar tool, pick tool, profile extracting tool, contour surface Core Generator and multiple other instruments can depend on the size of the object that they are just operating similarly.Therefore, as noted, in illustrative embodiments of the present invention, student's local control panel can be transferred to the teacher and be controlled.In this illustrative embodiments, the student can do is exactly to throw off color or transparency (rather than size) according to the rotating of his expectation, translation, amplification (electronic zoom) object or controlled plant from teacher's viewpoint.Therefore, can keep teacher and each relation between objects.Notice always have to give the student problem of how much local control.In illustrative embodiments of the present invention, this problem can solve by any local action that allows not change the real data collection usually.
Even so, but, in illustrative embodiments of the present invention, can be by allowing the local user to carry out then that " in fact " with display effect do not influence his local data but the operation that on the local data copy, acts on, main user's operation simultaneously shows in " ghost image (ghosted) " mode, thereby increases complexity.
For example, the teacher can illustrate to one or more students and how use multiple instrument to carry out multiple measurement between the object of 3D data centralization.The teacher can measure, and these measurements will be seen by the student.Yet the teacher also may expect to test the ability that the student carries out similar measurement, and the thing of going to school possibly and using him just to instruct from birth, and requires the student to carry out the other measurement that the teacher does not also make.The student can for example carry out those and measure, and those measurements can for example be displayed on his machine.Then, the teacher can be such as the snapshot of obtaining student's machine and the ability of evaluating student.In a similar manner, can allow the student to handle data set on the display that may be displayed on him, in student's display, " truly " manipulation that can also " ghost image " mode " truly " data set is shown and carries out by the teacher.By this way, all manipulations of teacher can be operated on the ghost image border of object, as as shown on student's machine, and can on student's machine, show that in the entity mode student's local operation is to required degree (just student's local operation intention has influence on the degree of 3D data set).Thus, can allow the student for example to carry out other manipulations except above-mentioned translation, rotation, indication and/or electronic zoom etc., but these other handle all and be treated to cost with big complexity and additional this locality.
Notice that above-mentioned illustrative embodiments requires the student to create the whole data set copy of (comprising 3D object and control panel) when entering this " enhancing " disengaged mode.Under those situations, the student can for example operate local (copy) data set, and the teacher can operate on raw data set, and this raw data set is presented on long-distance user or student's the machine in the ghost image mode.Thus, two independently graphics process can for example on the workstation of identical (student), operate concurrently.If the teacher will see the new route by student exploitation (developed), he can obtain student's view snapshot and with the student newly route discuss.
In illustrative embodiments of the present invention, use the role to switch, wherein, the teacher role can be sent to another participant thus from the participant, and the student can take over control from the teacher.He can for example continue the teacher for the work of object or by the viewpoint that executable operations how or other programs provide him is shown.This allows all participants to cooperate.Because in this illustrative embodiments, only there is a teacher via network in the time that provides, so this cooperation can be serial, and there is not collision problem.
In illustrative embodiments of the present invention, DextroNet can obtain the support of the multiple network of different bandwidth.In an illustrative embodiments, when DextroNet is used to the education indication, can use for example slow network.Each of the mutual toning system of 3D that connects via illustration DextroNet can for example have the imaging that is used for surgery planning, operation or anatomical structure and the local replica of video data.Thus, tracking data, control signal or figure or image parameter data, general size is less, can be sent via network.Usually in fact, in DextroNet, networked information can for example be divided into two types: message and file.Message can comprise that for example tracking data, control signal etc., file can comprise for example figure and view data, and this figure and view data can be compressed before sending.These are described more fully below.
Notice that in illustrative embodiments of the present invention, the action of being introduced by user can be sent at least one other user (student or teacher's) remote location, wherein, second workstation calculates corresponding image locally.Voice communication between the networking participant can be band outer (out of band), such as in slow network embodiment via phone, or such as in illustration express network embodiment by network on voice system, such as by voice protocol on the Internet (" VOIP ").In addition, express network can hold the single transmission as (mono) or three-dimensional video-frequency information that generates by as the photograph detector of describing (Camera Probe) in Camera Probe application.This video information can such as with computer generated image combination, or can independently check on display or the display window.The display video of non-3D (or image snapshot) mostly just can be counted as " background ".This is because image or video (image sequence) are the projections of real world, and does not comprise 3D information.Therefore, visible object can not accurately be located (position) in 3D object front in these images or video.Below this display possibility will be described more fully.
In illustrative embodiments of the present invention, DextroNet for example can support cooperation and work independently.Exemplary system can for example be used as the independent platform that is used for surgery planning and individual training.In this embodiment, for example, can the optimization system configuration of local ground.Thus, the position at the interface in student's the system and the position and the orientation that are orientated the interface in the system that can be different from the lecturer.Yet if activate network savvy, student's system can for example change the system of its system configuration with the coupling lecturer.This can be avoided the mismatch problems that may exist between the coordinate system (coordinate system).
E.DextroNet surgeon-development assistant example
In another illustrative embodiments of the present invention, the lecturer can for example instruct and be positioned at least one student at a distance, and " development assistant " can assist the lecturer by the following: the step that this assistant's finding needs is seen in steers image and suggestion, perhaps, everyone periodically the teacher role is transferred the possession of this assistant, so that can see his viewpoint.For example, the development assistant can steers image with highlight Prwsnt object of interest (such as by changing the color lookup table or the transparency of object); Amplify (electronic zoom) objects; Change orientation or viewpoint; Play, stop, refunding, the video image of F.F. or charged timeout operation; Split screen with different points of view is provided; Video image and computer generated image combination are used for showing; In the conversion (they are video image or computer generated image) that shows between single picture and the stereo-picture; And carry out other associated pictures manipulations and be current task.By also allow the development assistant participate in except lecturer and student, the lecturer for example can concentrate on the instruction of papil rather than notice is used for the manipulation of image.In addition, the adapter system of giving a knee control and when becoming the teacher thus, the student that can assist to be unfamiliar with the operation of the mutual imaging system of 3D obtains viewpoint and the information that they may need.
In illustrative embodiments of the present invention, surgeon or internist and " development assistant " can cooperate during actual operation or other medical treatment via DextroNet.In this embodiment, the development assistant can take on main user substantially, and the internist can take on the long-distance user substantially.Surgeon or internist can be such as operate or carry out some medical treatment, diagnosis or treatment operation over against the patient in operating room or therapeutic room, and the development assistant can be positioned at a distance, for example, be positioned at operative surgery, be positioned at the passage (gallery) of operating room top, be positioned at adjacent chamber, or be arranged in diverse place.This utilizes surgeon-assistant's example to be described in conjunction with Fig. 4-8 below.
Development assistant's task can for example be to carry out voice with the surgeon to contact in surgeon's operation, and dynamically handles the 3D demonstration and better assist the surgeon.Other long-distance users can observe, but long-distance user's manipulation data freely.
For example, suppose that the surgeon uses DEX-Ray in conjunction with neurosurgery ground TMSystem.In this illustration realizes, he main show be limited to from as those parts of the data set that can check of the viewpoint of the video camera of Camera Probe described in applying for.But the development assistant can freely check virtual data from any viewpoint.In illustrative embodiments of the present invention, the development assistant can have two demonstrations, and sees surgeon's viewpoint and his viewpoint.For example, he can check operation from opposite viewpoint (just being fixed to some viewpoints of tumour or other skull inner structures), and sees that the surgeon " enters into " in his viewpoint from " outside ".
For example, suppose that the surgeon operates near optic nerve.As described in the Camera Probe application, one or more points that the user can data centralization are set to mark, and observe the dynamic distance of probe tip to those marks of being read continuously.In addition, the user can utilize one or more tracked operating theater instruments (it can not obtain video image) to carry out identical thing, so that the development expert can for example provide input in surgeon's practical operation.Though the surgeon can not be exclusively used in whole display near the position the optic nerve, the assistant that develops can.
Thus, for example, the development assistant is the electronic zoom surgeon zone of operating dynamically, along near the optic nerve the surgeon five gauge points is set, and the distance that monitors each gauge point, if the surgeon warns the surgeon when coming in certain secure border.In addition, depend on the circumstances, can be on surgeon's system display, see and hear the new mark that is provided with by the development assistant and apart from the dynamic distance of each described mark, because the development assistant, just main user, control 3D data set.
Described in Camera Probe, the user can discern a plurality of virtual architectures that are included in combination (augmented reality) image.This can dynamically finish along with the progress of operation ideally, and in each stage, that can develop and be associated with this stage cuts apart.Yet, in fact normally impossible for the surgeon who works independently, because the surgeon is not the development expert can not freely use system with dynamical fashion a performance.Yet the development assistant is.When the surgeon come provide the zone near the time, the development assistant can add and remove structure with better guided surgery doctor.For example, if certain structure of not cutting apart before operation becomes relevant, then develop the assistant can be impromptu cutting object (on the fly) immediately, this certain structure example is as the part of the brain structure that has not finger piece by the tumour that scans before any art or microgroove (finethread).Thus, in illustrative embodiments of the present invention, any Flame Image Process that remote audience can have more freedom and can carry out thus not finishing during pre-planning is such as regulating colouring (colorization), transparency, segmentation threshold etc.
Use analogy (analogy), the surgeon can be become the football coach by analogy, this football coach allows his assistant (development assistant): get a bird's eye view match from eminence from the press gallery of top, arena, and give a knee when seeing the pattern that other troops dispatch via radio and coach's talk.In illustrative embodiments of the present invention, this consultant can for example see his screen and surgeon's screen, and this consultant can dynamically control thus and be presented at the virtual objects that shows on two screens, thereby in surgeon's operation the surgeon is supported better.
II. teacher-student is mutual
A. exemplary procedure flow process
The program circuit according to illustrative embodiments of the present invention that adopts illustration teacher-student's example will be described below.
Fig. 1 illustrates the illustration teacher's type control desk according to illustrative embodiments of the present invention.With reference to figure 1,101, exemplary system can be checked any student who has connected.Thus, whether 102, judging has new student to add.If, then 103, can be at teacher's control desk and be used for adding recently synchronization interface and data between student's control desk of student.If not, then program circuit can return 101, and 101, system can continue to check any student who has added.
Get back to 103, in case new student has added DextroNet such as the button on the 3D interface of the control desk by clearly clicking him, then the teacher can for example send student's message and comes synchronously their respective interface and object, such as slide block, button etc.These message for example can comprise: the position of teacher's control panel, orientation and size; The state of the virtual window gadget on control panel (" widget ") is such as last knob down, color lookup table setting; The position of the virtual objects of data centralization, orientation and size; And any available video.Perhaps, be different from very much he self data if the teacher finds data on student's side, then the teacher can select to carry out whole data set between he and student's corresponding control desk fully synchronously.This can be for example by by teacher's control desk compressed data set (do not comprise snapshot and be recorded in) and send it to add the DextroNet session recently student and finish.In case interface and by the teacher with add data that the student shares recently by synchronously then is used for the preparation that their collaborative interactive develops and finishes, and program circuit can proceed in the remaining major cycle that comprises Fig. 1, and is as described below.
110, illustration teacher system can inquire and whether receive student's video.This can operate Dex-Ray for example student TMTake place during the type system.If then program circuit can proceed to 111,111, video and the program circuit that can draw described student can proceed to 125.If do not receive student's video 110, then program circuit can directly move to 125.In addition, if teacher's workstation has available video, then this video can read at 105 places and draw at 106 places, and program circuit can move to 120, and 120, whether system can inquire has any such video to use.
Notice,, judge the availability of teacher's video 120.If then can send frame of video, and program circuit can move to 125 126.If not, program circuit can directly move to 125, and 125, teacher 3D equipment can be read.Here, the teacher can be upgraded position, orientation and the state of his virtual tool (incarnation or the awl of the teacher's who concentrates such as stylus, designation data current location) by the local trace system, this local trace system follows the tracks of the motion and the state of three-dimensional control continuously, and the operator utilizes this three-dimensional control to carry out alternately with data set.For example, carrying out the standard Dextroscope that colonoscopy is used TMIn, this 3D control can be for example to be held in stylus and operating rod in every hand in the right hand and left hand by user's (being the teacher) here respectively.Or for example, if at Dextroscope TMLast other application programs of execution, the user can for example replace the 6D controller is held in his left hand.In case read teacher's side 3D equipment 125, then can read network (student) 3D equipment 130,130, the teaching job station can be upgraded position, orientation and the state of student's representative instrument by network via the message that receives from the student work station.
From 130, program circuit can for example move to 135, and 135, the mutual quilt on teacher's side sends to the student via network.Here, the teacher can send position, orientation and the state of any instrument that he is using and his KeyEvent to student, this any instrument such as cutting implement, awl, sections observation device instrument etc.These positions and orientation can be converted into world coordinates, and be following described in the data layout part.From 135, program circuit can for example move to 140,140, can judge that the student is current and whether expect to follow teacher's viewpoint at local student's display place.If, 145, viewpoint synchronously, just, in case the teacher receives the request of student for synchronous viewpoint, then the teacher can send to the student with position, orientation and the size of the virtual objects in its workstation.If for not, and the student selects to follow he self viewpoint thus at 140 places, then program circuit can proceed to 150,150, and 3D view that can the drawing data collection.Because these carry out alternately, be circulated back to 110 and 120 from 150 program circuits, here for example as long as the DextroNet session is activated and just can repeats, and the mutual and manipulation of 3D data set is sent via network continuously.In addition, as described below, the student can and see between the viewpoint of he self (this locality) and changing in the viewpoint of seeing the teacher, and he self viewpoint is not subjected to the restriction of teacher's viewpoint.The student tests and identification in each circulation place by 140 thus for the current state that viewpoint is selected.
Before the program circuit of describing corresponding student's control desk place, the multiple example arrangement of the DextroNet of illustrative embodiments according to the present invention is described below in conjunction with Fig. 2.
The various control platform that Fig. 2 illustrates illustration DextroNet network 250 and is connected to it.Illustrative connection device all is the product or the raw product of the Volume Interactions Pte company of Singapore.For example, DEX-Ray TMType workstation 210 can be connected to DextroNet.As mentioned above, DEX-Ray TMThe type workstation is the workstation that is implemented in technology described in the Camera Probe application.This technology can be used in the multiple environment, and multiple environment comprises such as neurosurgery planning and navigation.At DEX-Ray TMIn the type workstation, can generate brain for example cut apart and handle after art before the assembled view of 3D scan-data or virtual data and real-time video.DEX-Ray TMThe type workstation can be used as teacher's control desk, wherein, the neurosurgical doctor can he carry out surgery planning or even carry out operation in, use DEX-Ray TMThe type workstation illustrates these combination images to other people that connect via network.Or, more expediently, DEX-Ray TMThe type workstation can be used as student's operation, makes the development assistant take on the teacher.
Continuation illustrates the Dextroscope that is connected to DextroNet with reference to figure 2,220 and 230 TMThe type workstation.At standard Dextroscope TMIn the type workstation, the 3D data set can alternatively develop; But, be different from DEX-Ray TMThe type control desk is not caught and integrated live video usually.Yet in the Dextroscope of standard, prerecorded video can be integrated into the 3D data centralization and be handled, such as analyzing and wherein use the prerecorded video that takes place in the neurosurgical examination of DEX-Ray in " postmortem ".
DextroBeam workstation 220 and Dextroscope workstation 230 are different-styles (flavor) of essential identical device, and the main difference part is display interface.DextroBeam replaces having the display as the connection in the standard computer workstation, use projector with its Display projector on for example wall or screen, as shown in 220.What form contrast is that Dextroscope has two displays usually.A display is integrated monitor, and it projects image onto on the mirror, thereby so that the operator can probe into (reach-in) interactive feel by having when catching the 3D controller to operate with hand on the 3D data set that loads below mirror.Another display be show with project to mirror on the standard display monitor of the identical content of content.
Thus, as shown in FIG. 2, use the illustration DextroNet network of illustrative embodiments can obtain the mutual example of multiple cooperative according to the present invention.DEX-Ray TMWorkstation 210 can with Dextroscope TMWorkstation 230 is cooperated or is cooperated with DextroBeam workstation 220.Or for example, its can also with another DEX-Ray TMThe cooperation of workstation 210 (not shown).In addition, DextroBeam workstation 220 can be mutual via network cooperation ground and another DextroBeam workstation 220 or Dextroscope workstation 230.
In addition, though not shown among Fig. 2, the mutual toning system of other 3D can be connected to DextroNet.For example, Volume Interactions company also provides RadioDexter TMVersion, this software is at Dextroscope TMAnd DextroBeam TMMove this Dextroscope in the system TMAnd DextroBeam TMSystem can go up operation at (or other computing machines) on knee, wherein, the 3D manipulation is projected 2D control, such as standard mouse and keyboard.The function of this software is described in detail in the DextroLap application.This software can be called as " DextroLap " thus here.Though not shown DextroLap control desk among Fig. 2, it can also be connected to DextroNet.
In addition, especially in teacher-student's environment, on DextroNet, can have cooperative work station, in teacher-student's environment more than two, teacher can the manipulation data collection, and any amount of student who connects via network 250 can participate in or observe.In this illustrative embodiments, every other student can for example see each student's virtual tool and teacher's virtual tool.The teacher can for example see each student's IP address, and the snapshot or the real-time video of the position of their virtual tool and their demonstration, and is following described in conjunction with Figure 10-30.
Select in the illustrative embodiments of the present invention, control function just is described as teacher's operating function here, can deliver between the participant, and the system that allows any connection is broadcast to other participants alternately with itself and data set.In this illustrative embodiments, icon or message can for example identify the teacher which system is operation then.
Provide this multiple cooperative possibility that DextroNet can provide, will describe the program circuit on the student work station below, know teacher-student can use top any may connection in conjunction with Fig. 2 description (no matter whether shown in Figure 2) alternately.
With reference to figure 3,301, the search of student's control desk is connected to available teacher.This can be for example by at first being finished for the request that adds the DextroNet session in broadcasting on the Internet by student's control desk.If have available teacher, then for example confirmation of receipt of student just can connect then.If do not have the teacher to activate on the Internet (or such as other data networks of the VPN), then the student for example can be connected to by providing the system configuration specified server and waiting for the teacher.
In case be connected to the teacher 301, then program circuit moves to 302,302, the search of student's control desk is to upgrade the relevant interface and the data of 3D data set of just cooperating mutual with it with this student and teacher.Thus, 302, the student can for example be used to upgrade from position, orientation and the dimensional parameters of the teacher's of the teacher's data that send via network interface and data set his control panel.He can also be via the state alignment of the message that receives by network with those widgets in the state of the widget on his control panel and teacher's the system.These message can for example relate to the upper and lower state of the button on the virtual control panel, the selection of the module page (for example is present among the DextroScope, this page representation user is in multiple may module or operator scheme, such as registration, cut apart, development etc.), the currency of scroll bar, the selection of combo box, color lookup table be provided with etc.
In case at 302 interfaces of having upgraded him, then the student can also for example receive data synchronization message, position, orientation, size, transparency and the detailed level of this data synchronization message such as the virtual objects of 3D data centralization (detail level).As noted, if the student is (or in identical robotization is handled, system by the student) judge that the difference between the version of student's data set version and teacher's same data set is too big, the student can also ask the teacher to send data set after the whole compression for analysis.In this case, student's control desk then can decompressed data and is reloaded long-range scene.Perhaps, the teacher can send mutual tabulation and replace whole data set, and allows the student work station come little by little to catch up with teacher's data set version alternately by all that move locally on student's machine that the teacher carried out the data set that begins at some time points.In case upgraded interface and data, make that thus student's control desk and teacher's control desk are synchronous, then program circuit can move to 310 and 320, and is as described below.
310, can judge whether teacher's control desk has sent any video.If then can draw teacher's video, and program circuit can move to 325 314.If 310 is that then program circuit can directly not move to 325.Handle the inverse problem of video problem, just whether student's control desk has the video that send to teacher's control desk, and the decision 320 can judge whether frame of video is available.Before the program circuit at description 320 places, notice that if having student's side-looking frequently, then student's control desk can read this video 305, draw these videos 306, notice that this video can be with also then being sent out 326 320.From 326, program circuit can also move to 325.If, for example, there is not student's video to use, then program circuit can directly move to 325 from 320.
325, student's control desk can read the 3D equipment (for example stylus and controller or their equivalence-just the mutual with it actual physical interface of user) of himself.This can allow student's control desk to calculate position, orientation and the state of its virtual tool, such as given by the actual tracing positional of stylus and 3D controller.If the student is using the DextroLap system, then can read mouse and the keyboard that substitutes 3D equipment 325.330, student's control desk can for example read teacher 3D equipment from the message that receives via DextroNet, and this teacher 3D equipment allows student's control desk to come more position, orientation and state and any KeyEvent of beginning teacher's representative instrument.From 330, program circuit can move to 340,340, can judge that the student selects the viewpoint of control he self or follows teacher's viewpoint.If not, and the student selects to control he self viewpoint, then program circuit can move to 345 then, 345, the student can ignore about the networking message of for example teacher's left hand instrument (in this example, if the left hand instrument be controlled at the 3D data centralization provide the position that the user is in and have current selection be what (a plurality of) object; Be similar to the mouse of moving cursor in two dimension), and the operating rod by for example his this locality or 6D controller directly read position, orientation and the state of he self left hand instrument.If 340 for being, and the student selects to follow teacher's viewpoint thus, and his machine can be asked the current location and the orientation of (a plurality of) virtual objects of teacher at 346 machines that send a message to the teacher.In case he receives answer, he can upgrade he self (a plurality of) object then.
No matter make any selection 340, program circuit can move to 350 from 345 or 346,350, draw the 3D view of student's control desk, then, program circuit can be circulated back to and determine 310 and 320, so that, as mentioned above, DextroNet interactive program flow process can be continuously repeated by session.
B. the mutual illustration feature of teacher-student
Provide above-mentioned basic teacher-student's example, the feature below can in illustrative embodiments of the present invention, realizing.User on the following supposing the system has the virtual tool of both hands, and each instrument has a pushbutton switch, and the user in the system uses the virtual control panel with button and slide block to control the development application program.This exemplary system can be for example to move RadioDexter TMThe Dextroscope of the Volume Interactions Pte company of the Singapore of software TMOr DextroBeam TMSystem.
1. follow the tracks of the minimizing of load
In illustrative embodiments of the present invention, variability for the network data transfer rate, reduce the load of traffic that is applied on the network, and guarantee that the key user who does not omit on the 3D object is mutual, for example can adopt the one of four states of the button switching of virtual tool: check, begin action, move and tenth skill.Under " inspection " state, the user does not click the stylus of control virtual tool, but this stylus that moves (this generates position and orientation data, but does not generate " button is pressed " data).Thus, this virtual tool only is rendered as roaming in virtual world and does not carry out any activation manipulation, can see point at objects, and in fact can trigger some objects and change their state, though not in lasting mode.For example, boring bar tool can be along with the skeleton view that this 3D object is shown alternately of virtual tool and 3D object, unless but can not supress button to object boring.Under " begin action " state, can press the button of stylus, and if keep pressing this button, then can activate " carrying out action " state.In case the user for example discharges the button of stylus, instrument can enter " tenth skill " and can change " inspection " state of getting back to then.Thus, the most time virtual tool is in " inspection " state, should " inspection " state be the actual more unessential state of those states that works of the instrument of comparing.
Thus, under " inspection " state, virtual tool only can be rendered as roaming in virtual world and not carry out any valid function, such as moving to another position from a position.The packet of transmitting the position of instrument under this state and orientation can be considered to the packet of " inessential ".When having congested network, " transmission " buffer of teacher may be full of the grouping that does not send.Under this environment, in illustrative embodiments of the present invention, DextroNet software can be checked " transmission " buffer, and the grouping that those is had " inspection " state is thrown away.Because the most time virtual tool is in " inspection " state, thereby can reduce load of traffic greatly.This can illustrate with reference to Figure 53.For example, suppose that the teacher has one group of message as shown in Figure 53 (a) in his " transmission " buffer.When network connected slowly, he can for example only send the message shown in Figure 53 (b) to the student.In this case, the instrument that the student will see the teacher from the position 1 " jumping " and carry out for example drilling operation then to position N, but any information of not losing actual drilling operation.
In illustrative embodiments of the present invention, the control of DextroNet load of traffic can utilize this fact.Can be to important distribution of messages with transmission priority.Usually, when network speed was very fast, each action of teacher's instrument can for example be transferred to the student.Thus, the student can watch the continuous motion of teacher's instrument.Yet, if detecting networking speed, teacher's system slows down (for example in sending out buffer, having too many message queueing to wait for) by noticing, in illustrative embodiments of the present invention, this teacher's system can abandon the message of " inspection " Status Type.By this way, even under congested network condition, the student also can catch up with teacher's step.Be that under this pattern, the instrument of the teacher in student's local view can be in sight so not level and smooth movably with the compromise of this message " compression " (loss) scheme.However, still, can not lose important manipulation for virtual objects.Notice that this minimizing does not need to take place when networking speed is very fast.Switch can be the queue length in " transmission " buffer of for example teacher.When not having " minimizing ", teacher's instrument can motion smoothly in student's the world.When " minimizing " when taking place really, the motion of teacher's instrument is discontinuous.Yet the teacher can not lose for the operation of any virtual objects, and student's instrument can be caught up with the step of teacher's instrument.By this way, the illustrative embodiments of the present invention instrument that can dynamically be suitable for available networking speed and show the teacher with multiresolution (just coarse and smoothly).
2. viewpoint control
As noted, in illustrative embodiments of the present invention, DextroNet student can control the 3D object that provides, and perhaps follows teacher's viewpoint.If the student selects this locality control of 3D object, then he handles this locality of object, such as the position that does not influence the 3D object in teacher or other students' the world for rotation, translation and the electronic zoom (numeral is amplified) of object.Yet the teacher can see position and the orientation of student's virtual tool with respect to the 3D object, thereby allows the teacher to infer student's viewpoint, as above notices.In this exemplary scenarios, the student can be in his world target rotation to find except when preceding interested other zone or orientation outside selecting by the teacher, and can be for example with voice and/or should be regional in addition or be orientated and be transferred to the teacher by pointing in addition zone or orientation.Then, when serving as a teacher the message that receives the student, can for example forward specific place to, and in teacher's the world, note this position by student's sensing.On the other hand, if the student selects to follow teacher's viewpoint, then all motions in teacher's the world will be shared with student's the local world.
In illustrative embodiments of the present invention, can have specific order, this specific order can be sent between these two states via network and be switched, and reorientates the remote dummy instrument with respect to user's viewpoint.This order can for example drive the judgement at 140 and 340 places in Fig. 1 and 3 respectively.
2. data sync
In illustrative embodiments of the present invention, two synchronous modes can be arranged, simple synchronization and complete synchronous.Illustrative simple synchronization pattern can be for example the only position of communication object, orientation, size, transparency and detailed level.From these parameters can be via the viewpoint of the data network transmission that need not big bandwidth, these parameters can be considered to " light weight (light-weight) ".Thus, in illustrative embodiments of the present invention, teacher's module can for example add network in case detect new student, then realizes this simple synchronization.
In illustrative embodiments of the present invention, for example, only clearly ask complete when synchronous (such as 145 places among Fig. 1), can complete synchronous as the user.This comprises all data (except the data of catching for the Objective of Report, such as snapshot and 3D data) of compression and transmission teacher.In illustrative embodiments of the present invention, this can be optionally synchronously, because this can carry out (take time) calmly synchronously.In this embodiment, the teacher only when the student require complete just can begin when synchronous complete synchronously.If have a plurality of students, for example, only data can be sent to the student who makes this request.In addition, when the departing from of essence takes place between both sides, can use complete synchronously as recovery policy.When not by throughput constraints that available bandwidth caused, can be automatically, periodically or carry out when the incident of some definition takes place that this is complete synchronously.
C.3D interface (control panel) is synchronous
In illustrative embodiments of the present invention, exemplary synchronization can comprise two processes, calibration synchronously and widget synchronous, to describe as following.
1. calibration is synchronous
When using DextroscopeTM or DextroBeamTM type system, for example, virtual control panel needs to utilize (for example being made by acryl resin) physics pedestal accurately to calibrate, the 3D tracker is shelved on this physics pedestal during mutual, so that virtual tool can touch the counterpart of virtual control panel.Thus, calibration and configuration are local, change along with the difference of machine.This can cause in networking mismatch problems simultaneously.For example, teacher's instrument may be because calibration and/or configuration mismatch and can not touch the control panel that provides the student.For fear of this problem, in illustrative embodiments of the present invention, Faculty and Students' control panel can be following synchronous.When activating the networking session, the position of teacher's control panel, orientation and size can be sent to the student, and these parameters can be replaced the parameter of student's self control panel.When the networking session stops, can recover the original configuration of student's machine, thereby allow him to work independently.
2. widget is synchronous
When the networking session began, the original state of the control panel on both sides can be different, for example state of the position of scroll bar, button and label (tab), the tabulation of color lookup table etc.All these parameters need be aligned and be used for networking.
D. the connection between the distinct interaction platform
In illustrative embodiments of the present invention, the connection between dissimilar interaction platforms can for example be supported on Dextroscope TMAnd between the DextroBeam.Thus, the teacher on Dextroscope can indicate a plurality of long-range student who assembles previously at DextroBeam, perhaps for example, and the local student that the teacher can indicate in the DextroBeam front, and at Dextroscope TMOn long-range student can watch.
In illustrative embodiments of the present invention, the connection of another support can be between 3D interaction platform and 2D desktop workstations, such as at Dextrosocpe TMAnd between the DextroLap system, as noted above.This system can allow the desktop workstations that the participant uses does not only have the 3D of stylus and operating rod input equipment, wherein, carries out mutual (described in using at DextroLap) between user and the system via mouse and keyboard.In this scene, the input equipment of Faculty and Students may be different.Because following reason this can be the most useful: the student who lacks 3D stylus and operating rod can still watch the teacher who operates in 3D fully, and communicates by letter with the teacher with keyboard by the mouse that uses them.On desktop workstations, KeyEvent and can be transmitted and insert from the action of tracker.In this illustrative embodiments, key name claims and modifier (modifier) need be wrapped in the internet message.For example, the DextroLAP cursor has the 3D position.This position can be sent to the teacher, and the teacher can see that student's cursor moves in 3D then.
In illustrative embodiments of the present invention, the multiple mutual toning system that moves in Unix and Windows system can all be connected via DextroNet.
E. the teacher detects automatically
In illustrative embodiments of the present invention, serve as a teacher with the student and all be positioned at when providing on the Intranet, DextroNet can automatically detect the teacher.As following more detailed description, if client computer obtains replying from server, then client computer can begin to be connected with the network of server then, and tell server it be teacher or student.Server can for example be preserved the register list (role, IP address, port) of All Clients.If it is the student that server is seen initiate client computer, whether he can check has available teacher also this teacher's IP address and port to be sent to this student.Thus, the student can automatically obtain teacher's information from server.If there is no teacher, server then can for example warn the student to withdraw from networking to connect.Perhaps, the student can wait for up to the teacher and reaching the standard grade.Thus, in this illustrative embodiments, the student does not need to worry the low layer networking task such as configuration server IP address and port.In case teacher is arranged, then student's system just can automatically detect teacher.
III. surgeon-development assistant is mutual
A. general introduction
The illustration collaborative interactive of describing the illustration 3D data set between surgeon and development assistant of illustrative embodiments according to the present invention below with reference to Fig. 4 to 8 develops.Fig. 4 is the program flow diagram of illustration surgeon's control desk.Fig. 5 is the exemplary procedure process flow diagram of development assistant's control desk.Fig. 6 to 8 is illustrated in surgeon in this scene and development assistant's each illustration view.Usually, this example is not restricted to " surgeon " in essence, but can comprise any such scene, in this scene, a participant (" surgeon ") carries out diagnosis or treatment operation for given object, wherein, can obtain before the art of this object imaging data and with the common registration of physical object, and wherein, another participant can be according to the 3D data set development object of being created by imaging data before this art.Thus, described " surgeon " comprises for example ultrasonic examination doctor (described for example in " SonoDex "), use Cathlab TMThe surgeon of the intervention cardiologist of machine, use Medtronic operation guiding system etc.
In described illustration was mutual, the surgeon expected and uses Dex-Ray TMType workstation, and catch position data and real-time video thus, and the development assistant can use the workstation of any kind.With reference to figure 4,401, the surgeon can be connected to network.402, the data on development assistant one side can with the data sync on surgeon's one side.In case finish data sync, then program circuit can for example move from 402 along two parallel routes.At first, 410, the scene that the surgeon can ask to develop the assistant, and if request, then can draw this scene 411.Assistant's the scene if do not ask to develop, then program circuit can be from 410 direct motions to 420, and 420, surgeon's control desk can read the position and the orientation of photograph detector.Here, surgeon's control desk obtains the position of photograph detector and orientation (local) and is converted into coordinate in the virtual world.From 420, program circuit can for example move to 425.
Notice, be parallel to above-mentioned processing, can also have local video to handle.Remember that (recalled) is as DEX-Ray TMThe surgeon's of type workstation control desk also can obtain real-time video.Thus, the video of himself can be read, and the video of himself can be drawn at 406 surgeon's control desk at 405 surgeon's control desk.Program circuit also can move to 425 then.
425, surgeon's control desk can send to the development assistant with its local video that has obtained via network.Therefrom, program circuit can move to 430, and 430, surgeon's control desk sends its photograph detector information.Here, surgeon's control desk sends the position of the photograph detector in the virtual world coordinate and is orientated the development assistant.This only is the transmission of Information that obtains 420.435, surgeon's control desk can upgrade assistant's representative instrument.Here, surgeon's control desk can receive and be updated in the position and the orientation of the representative instrument of the development assistant in surgeon's the virtual world.These data can obtain via the control desk of network from the development assistant.Finally, 450, the 3D view that surgeon's control desk can the drafting 3 D data set, it can comprise video and augmented reality, comprises the position and the orientation of development assistant's representative instrument.Notice that it will be to carry out mutual result by surgeon and development assistant and 3D data set that current 3D draws.Describe the opposite side of surgeon-development assistant example below in conjunction with Fig. 5, concentrate on development assistant one side.
501, the development assistant can be connected to network, and 502, the development assistant can upgrade his or her data.Here, the development assistant need use the surgeon's who obtains via network data to upgrade he or she self virtual objects position, orientation and size.
Program circuit can move to 510 and 520 judgement concurrently from 502.At first, 510, can judge any video that whether receives from surgeon's control desk.If, can draw surgeon's video then 511.If not, then program circuit can move to 530.The second, the scene that whether should send the assistant can be judged by 520 places in judgement.This can be for example in response to inquiry or request from surgeon's workstation.
If the surgeon has asked development assistant's scene, then 525, can send assistant's scene, and the assistant can send the snapshot or the video of his view.These snapshots can be for example three-dimensional or single look like (monoscopic).If in the scene that 520 surgeon did not ask the development assistant, then program circuit also can move to 530, program circuit arrives from 510 and also arrives 530, and 530, development assistant's control desk can read the 3D equipment of himself.Here, the development assistant has the control fully for he self instrument, can utilize position, orientation and the size of the instrument that he self stylus and operating rod control him thus.From here, program circuit can move to 540,540, can upgrade surgeon's representative instrument.Here, development assistant's control desk can receive and be updated in the position and the orientation of the representative instrument of the surgeon in development assistant's the virtual world.
Finally, program circuit moves to 550,550, and the 3D view of development assistant's control desk drafting 3 D data set.This 3D view will comprise the renewal for development assistant's self 3D equipment and surgeon's representative instrument, and can also comprise any video that receives from the surgeon.As noted above, surgeon-development assistant example is assumed to be the DEX-Ray that comprises via DextroNet TMTo Dextroscope TMType is mutual.Aforesaid this scene is the variant of teacher-student's example.In surgeon-development assistant scene, the surgeon takes on students' role usually, and the assistant takes on teacher's role.This be because surgeon (student) because be busy with operating with the ability of data interaction on more restricted; He can relate generally to and watch development assistant (teacher) how to control 3D vision virtual world thus.
B. independent viewpoint
When in operating room, using DEX-Ray TMDuring the type control desk, for example, surgeon's viewpoint is subjected to the restriction of the direction of video detector, as shown in Figure 6.As in the CameraProbe application, describing DEX-Ray more fully in more detail TMType equipment based on the position of video detector and thus the viewpoint that has of the video camera from video detector come the drafting 3 D scene.Thus, for example, use tip that the surgeon of DEX-Ray type control desk can not see video detector how internally just from the viewpoint of target self internal fixation near target, the viewpoint of this target self internal fixation is such as the viewpoint that is fixed to tumour or other skull inner structures.Yet, such as at Dextroscope TMIn the planning system of realizing on the type machine, the assistant has the not limited viewpoint that is used for checking target, as shown in Figure 8.Thus, in illustrative embodiments of the present invention, can use DextroNet with DEX-Ray TMWith Dextroscope TMConnection to be assisting the surgeon or to provide second pair of eye to him, and these second pair of eye has and limited do not move through the 3D data set that is associated over against its actual patient of carrying out operation with the surgeon from origin.Have huge purposes this surgeon during real-time complicated operation, wherein, a plurality of developments can help surgical procedure certainly, but can not be finished by the surgeon who is undergoing surgery in logic.In this scene, the development assistant can significantly contribute to surgery or other (hands-on) treatment (diagnosis) effects with hand.
C. tradable viewpoint
Perhaps, in illustrative embodiments of the present invention, the surgeon can see assistant's viewpoint, and the example is shown in Figure 9.Thus, in illustrative embodiments of the present invention, the surgeon has two types available views.A kind of view is common photograph detector augmented reality view, and just video covers (overlaid) image together with three 2D three plane pictures, and as shown in Figure 6, and another kind of view is all viewpoints from development assistant side as shown in Figure 7.These two kinds of views can show in the window of two separation or display, perhaps can check in the single window that the surgeon can be switched between these two kinds of views.Thus, the surgeon can for example be switched between Fig. 6 and Fig. 7.From development assistant's view (Fig. 7), the surgeon can be from different viewpoints, such as the viewpoint on the destination object that is positioned at tumour for example, the relation of checking virtual objects and his instrument.In addition, in illustrative embodiments of the present invention, the development assistant can also can see surgeon's scene in his display.This is for example shown in Figure 8.In main window is development assistant's view, and this development assistant's view is not limited by the position of the camera of the video detector instrument that the surgeon holds.In addition, have " picture-in-picture " view, in Fig. 8, " picture-in-picture " view is shown as the wicket 810 in Fig. 8 upper left corner, and the view that this wicket 810 illustrates the surgeon is seen just as him.This can be used as frame of video and transmits, and can not allow the development assistant handle by any way thus.Therefore, Fig. 8 illustrates main window, and this main window shows the virtual 3D world of the instrument (as the line that presents) that wherein also can see the surgeon from the top to the bottom.Another illustrates surgeon's real views than wicket 810, and it comprises that field video signal adds the augmented reality of the 3D object that can get on surgeon's scene, and its display parameter are selected and handled by the surgeon separately.Thus, in the 2D video of the picture-in-picture window 810 that the development assistant can be in the 3D world of the main window of Fig. 6 and in Fig. 8 upper left, see surgeon's instrument.
IV. illustration interface alternation
In illustrative embodiments of the present invention, the network savvy of DextroNet can be utilized at main Dextroscope TM" networking " label (tab) in the 3D interface is controlled.
A. connect
In illustrative embodiments of the present invention, can provide the networking module interface to the user.This interface can for example provide three buttons: " teacher ", " student " and " finishing networking ".Via this illustration interface, the user can select thus as teacher or student, or selects to finish the DextroNet session.In illustrative embodiments of the present invention, only when in the network teacher being arranged, the student just can connect.Otherwise, can notify " not having the teacher " message to stop connecting to the student.If the student does not have the 3D input equipment, then he can use as at the mouse that is used to communicate by letter described in the DextroLAP application program.
B. teacher's action
In addition, can show additional interface feature to the user who takes on main user or " teacher ".In this illustrative embodiments, can provide student IP tabulation and " synchronously " button to the teacher.Can be so that this panel only can be used by the teacher.When new student adds fashionablely, for example the snapshot of for example student's initial scene can be sent to the teacher and in student's tabulation, show.In addition, can be in text box middle finger example such as student's IP address.If have the student more than, then the teacher can the scroll list checks other students' IP address and snapshot.In illustrative embodiments of the present invention, can use these snapshots to allow data set that the teacher assesses him and student's data set relative synchronous.For example, there are two kinds when initialization is networked, to finish synchronous method: simple synchronization method and complete method for synchronous.Usually in fact, if Faculty and Students load identical data set, only need simple synchronization.Yet, if their loaded data collection difference, the teacher need start complete synchronously, otherwise have disaster after a while.The teacher's side that impinges upon soon that sends from a plurality of students shows their original state (environment and data) separately.Therefore, the teacher can check whether the student has loaded the data identical with the teacher via snapshot.If not, then he can warn student and complete synchronous.
C. student's action
In illustrative embodiments of the present invention, in case when the user selected as the student, then he lost the control to his virtual control panel.He can check teacher's manipulation and point to his interested part.Yet he is restricted in behavior, because he can not touch his virtual control panel, the teacher controls this virtual control panel.As noted, can be via dedicated button (being similar to those buttons that PC realizes that are used for described in the DextroLap), or via being different from special use " student " virtual control panel of striding " standard " control panel that illustration DextroNet handles by the teacher, come that convenient student can carry out limited one group mutual.
In illustrative embodiments of the present invention, the student has two kinds of methods that can be used for watching operation of teacher.For example, he can (i) follow teacher's viewpoint, or (ii) check data set from the viewpoint of the viewpoint that is different from the teacher.In this illustrative embodiments, he can switch between these two patterns by his stylus of simple click or some other system definition signals.
In illustrative embodiments of the present invention, when the student is in " engage (engaged) " or follows teacher's the pattern of viewpoint, can for example provide the red text demonstration of speech " LockOn (locking) ".Under this pattern, he can not rotate or mobile object.Throw off if he clicks stylus or sends some other signals, then " LockOn " text can for example disappear, and this indicates him to check data set from his viewpoint of self.In this " disengagement " pattern, the student can for example use the left hand instrument to rotate and move shown object.
D. stopping network connecting
Shown in during illustration realizes, because only teacher's instrument can touch virtual control panel, so be responsible for finishing network savvy by clicking " finishing to network " button by the teacher.In case networking finishes, the teacher can for example keep networking during the session, and he or she changes to the institute that data are made.Yet if desired, the student can be for example returns to his scene before the networking, thereby recovers to enter the situation of preserving before the networking session at him.
In illustrative embodiments of the present invention, during networking, may need student's data and teacher's data sync.Therefore, for fear of the local data of upsetting the student, when the student pressed the networking button, for example, he can be copied to some backup directories automatically at the data of self before the actual beginning of networking session.Therefore, when he finished to network session, he can recover he self data by duplicate back data from backup directory.In illustrative embodiments of the present invention, in case press " finish networking " button then can finish described recovery automatically.
E. illustration server
In illustrative embodiments of the present invention, can on the server-client framework, set up DextroNet, as shown in Figure 9.In this illustration structure, teacher 930 and student 920 are client computer.They can for example physically be connected to server 910, can use this server 910 to manage multiple connection, connect registration and connect inquiry.All information from transmitter (client computer) can for example at first be transferred to server 910.This server is the IP address of analysis receiver for example, forwards this information to specific purpose ground then.Thus, in this illustrative embodiments, before activating network savvy on teacher or the student's side, need at first runtime server application program.Illustrative server feature is then described.
1. multiple connection
From the low layer viewpoint, in illustrative embodiments of the present invention, server can use multiplex technique to support multiple connection.This can be an one process parallel server for example, and wherein, the arrival of data triggers to be carried out.To such an extent as to if the too high CPU of load can not handle, then timesharing (time-sharing) can for example be taken over.In addition, from high-rise viewpoint, based on the IP address from transmitter (client computer), server can use ICP/IP protocol to come clean culture, multicast or broadcast to the destination.
2. connect registration
When client computer was connected to server, it is its IP address of registration and networking role (for example teacher or student) on server for example.Can be responsible for for example guaranteeing two standards by server: (1) student has the teacher before adding, and (2) only have a teacher to be connected to server.If do not satisfy standard (1), then for example can warn the student to withdraw from networking, or for example, can advise that the student waits for up to the teacher to connect.If do not satisfy standard (2), then second supposition (putative) teacher can for example be warned and withdraw from network savvy, or for example can inquire whether he expects to connect as the student.
3. connect inquiry
In illustrative embodiments of the present invention, client computer can inquire that server has in the JA(junction ambient) about current being in whom how many (peer) client computer at the same level and these client computer at the same level be.By this way, client computer can know who involves in communication.This is important for the teacher, and the teacher keeps current student's dynamic listing.When server received this " inquiry at the same level " request, it can be for example sends it back the requestor with the IP address and the port of all client computer at the same level.
4. answering server inquiry
In illustrative embodiments of the present invention, can detect server automatically via LAN.For example, when the UDP of server socket (socket) when LAN receives broadcast polling about the client computer of desired service device application program, it can check the title that runs application check desired one whether available.If available, then server can be with himself address (IP: port) send it back the client computer of inquiring.
Thus, in illustrative embodiments of the present invention, the networking role who selects him as the user (for example, when he adds the networking connection, in his development environment, by pressing " teacher " or " student " button on the networking interface), he can comprise the inquiry of server program title via Intranet broadcasting.This information can for example be sent to the designated port on all Intranet machines.If server program moves, it can keep listening attentively to this designated port.In case it receives broadcast from client computer, it can check the title of all working procedures on server machine, checks whether mate.If, then server can be for example with it self address (IP: port) send back to the client computer of inquiry.Simultaneously, client computer can be waited for the answer from server.If do not return after the regular period, then client computer can be reported makeing mistakes of " serverless backup operation ", and can for example use (resume) state that normally works alone again.
5. startup of server (launch)
In illustrative embodiments of the present invention, the DextroNet server can be used as the stand-alone utility that need not to be installed on the development machine and moves.If communication takes place in LAN, Faculty and Students need not clearly to know for example IP address of server.DextroNet software can be for example location-server automatically.If communication is carried out via WAN, then need the IP and the port of server are offered DextroNet.If for example do not detect home server, and do not have remote server, then the teacher can for example automatically start server application on his machine when he attempts beginning network savvy.
Selectively, for example, server capability can be combined with the teacher role.In other words, teacher's machine can become server, and student's machine can remain client computer.Yet, in this illustrative embodiments, because from the development order and the simultaneous communications command of 3D development software, so the burden of teacher's machine is heavier relatively from DextroNet.Thus, for example, can for example pass through DextroScope TMDevelopment cycle slows down the DextroNet communication cycles.This can cause the data that enter or export to be left in the wait buffer more.Therefore, in illustrative embodiments of the present invention,, then can for example the packet of " inessential (insignificant) " be lost if data queue is oversize.That is to say that these packets can for example not treatedly just be removed from formation.The packet of " inessential " in this context, refer to those packets of the output that does not influence program/development, do not carry out those groupings of motion that any important operation (for example bore, shearing etc.) does not influence (Faculty and Students) instrument of data set thus such as transmitting.
In this context, be noted that as described in more detail belowly, have the basic status of four kinds of instruments: " inspection ", " beginning action ", " carrying out action " and " tenth skill ".In " inspection " state, virtual tool only is rendered as roaming in virtual world and does not carry out any effective operation, for example, moves to another position from a position.The position and the orientation of the virtual tool under this state is considered to the packet of " inessential ".When having congested networking situation, " transmission " buffer of teacher will be full of the grouping that does not send.Under this environment, software will be checked " transmission " buffer, lose those and have the grouping of " inspection " state.Because virtual tool is in the time of most of " inspection " state, thereby significantly reduce load of traffic.For example, the teacher has the following message as (a) in his " transmission " buffer.When network was slow, he only sent as the message in (b) to the student.In this case, the instrument that the student will see the teacher is 1 " jumping to " position n from the position, carries out piercing then.
V. illustration realizes
The multiple illustration that Figure 10-49 illustrates DextroNet realizes.This realization is designed to for example be integrated into Dextroscope TMOperation RadioDexter TMIn the software etc., and can be rendered as additional function modules in this system thus.In the example shown, Faculty and Students can communicate with one another by the server of runtime server program.Note during each session, a plurality of students being arranged, but once have only a teacher.
Figure 10-25 illustrates along with the relation between the illustration teacher of time and two students' the view.Figure 10,14,18 and 22 each side by side three width of cloth views are provided, in three accompanying drawings of these accompanying drawings and then, each width of cloth figure is shown then with bigger image.Below these accompanying drawings will be described.Figure 10-25 illustrate the illustration teacher of illustrative embodiments and two students according to the present invention each seen corresponding views and between relation.With reference to Figure 10, the teacher can check whether the student is connected.Teacher's view is shown in Figure 10 (a), and each of Figure 10 (b) and 10 (c) illustrates two illustration students corresponding views of student 1 and student 2 just.In the example shown, Faculty and Students 1 use Dextroscope type system, and student 2 uses DextroLap type system.As described below, Figure 11-13 is shown in further detail each width of cloth of these views.
With reference to Figure 11, the teacher can see he self instrument 1100, student 1 remote tools 1101 and student 2 remote tools 1102.Note,, and therefore not having 3D control completely so student 2 remote tools is actually cursor because student 2 uses DextroLap type system.Also having shown in Figure 11, each student's remote tools is with the IP address of this student's computing machine.In illustrative embodiments of the present invention, can change the title that this IP address shows the student.Selectively show IP address or student's title, or difference shows all.This is shown to the teacher being used for instructing many students' illustrative embodiments of the present invention to be particularly useful simultaneously with the name with each student.By this way, the teacher can know the people of his addressing always.
Figure 11 just can also see the display window 1120 of the snapshot of the demonstration that is used for a plurality of students in teacher's view.Utilize this snapshot window 1120, the teacher can have the ability of the local view of seeing the student.Finally, have three networking instruments: " synchronously " 1150, " long-range snapshot " 1160 and " role's switching " 1170.Synchronous 1150 can be used for synchrodata collection between student and teacher.If obviously the state of their corresponding data set has departed from abundantly when establishment is obscured concerning teacher (or student), then can use this instrument.Long-range snapshot 1160 only obtains the snapshot of the particular student of demonstration in snapshot window 1120.It is convenient that this can for example come by the teacher's control panel with the scroll list that connects the student.Then, the teacher can for example roll by student's tabulation and select a student.Then, serve as a teacher when for example pressing " catching " button, can ask snapshot from student's workstation.Finally, the role is switched the role that Faculty and Students are switched in 1170 permissions.This processing can be restricted to the starting by the teacher, and therefore in greater detail following, this button can for example only can activate on teacher's machine.
Figure 12 illustrates student 1 view.This view illustrates student 1 instrument 1201 and teacher's instrument 1200.Notice that student's 1 its own IP address presents with its instrument.For example, for teacher's view, this feature can be closed, and maybe can substitute with student's name or some other identifiers.The 3D object 1225 that the student sees is identical objects shown in the view with the teacher, and is in identical accurate viewpoint or viewpoint.This is because student 1 view is " locked " to teacher's view.Thus, in the right regions of student 1 screen, show locking symbol (sign) 1290.Can also see teacher's instrument 1200, and three networking instruments: synchronous 1250, long-range snapshot 1260 and role switch 1270.Because student 1 is the student and does not control network savvy, do not show snapshot in snapshot window 1220.In fact, if the student will see teacher's view, he is required, and what do is the view that locks onto the teacher, as shown in Figure 12.Therefore, long-range snapshot 1260 becomes ash (ghosted).Because only the teacher can carry out these functions, so synchronous 1250 become ash, it is 1270 constant that the role is switched.Figure 13 is similar to Figure 12, and student 2 view only is shown.With reference to Figure 13, identical 3D object 1325 is shown.Notice that the student's 1 of 3D object view presents than the view more transparent (transparent) of Faculty and Students 2.Though this is that these workstations may have different configurations because identical program may be moved in all working station.For example, some can be so that some features forbidding, such as, " when touching control panel, making the 3D object become ash " by stylus.This feature can for example be quickened the mutual of stylus and control panel.
This is because (some may use the DextroLAP system that moves on laptop computer, some may have Dextroscope faster according to the user TM) available video card, 3D develop alternately software some more " modification " function may not produce strict identical visual results, but can it seems that performance is identical from the effect on the 3D object (this effect needs strict identical so that keep synchronous view certainly).Such rhetorical function for example, can be seen in Figure 13.In order to quicken to show when user and control panel mutual (and therefore needing to be responsible for accurate button click and slide block) time, can be above the 3D view pellucidly (transparently) catch and paste the drawing image of 3D object, only refresh control panel and stylus.If this does not duplicate at student's workstation place, the result on the application program will be identical, but slack-off alternately, and the vision of object can difference (just opaque).
Refer again to Figure 13, student 2 also is in locking mode, and also shows locking symbol 1390 thus in Figure 13.Student 2 can see he self instrument, is cursor 1302 in this case.The local ip address that also has student 2 that shows, this local ip address can for example be replaced with designator or be strengthened in the illustrative embodiments can selecting, as mentioned above.Be similar to student 1 situation, in snapshot window 1320, do not show snapshot, and networking instrument synchronous 1350 and long-range snapshot 1360 all become ash, and that the role is switched is 1370 constant.
Figure 14-17 is described below, teacher and two students of the Figure 10 when these accompanying drawings illustrate the illustration time point of following shown in Figure 10-13.With reference to Figure 14, the teacher draws straight line between the tip of the cone-shaped body object at the left rear side place of the angle of the cube object at the right lateral side place of 3D object and 3D object.The teacher uses the illustration survey instrument to finish drafting, and measurement frame presents near the end points of measuring thus, and demonstration " 47.68mm ".Two students are in locking mode.The details of this three width of cloth view is described below with reference to Figure 15-17.
Figure 15 illustrates teacher's view.Can see teacher's instrument 1500, use teacher's instrument 1500 to carry out of the measurement of the angle of the cube object from 3D object 1525 to the tip of cone-shaped body object.Be also shown in the IP address of student 1 remote tools 1501, and student 2 remote tools (cursor) 1502 is together with student 2 IP address together with student 1.Can see that student 1 just points to the teacher near the point that begins on the cube to measure with his remote tools from teacher's view of Figure 15.Near the base portion of student's 2 positive sense cone-shaped bodys.
With reference to Figure 16, student 1 view can be seen teacher's instrument 1600.Teacher's instrument is certainly away from student 1.Be also shown in slotted line and measurement frame 1692 that student 1 instrument 1601,3D object 1625 and teacher make.Visible in addition is locking symbol 1690, and indication student 1 is locked into teacher's view.With reference to Figure 17, student 2 view is shown.Student 2 is locked into teacher's view, shows locking symbol 1790 thus.Student 2 remote tools 1702 shows that together with his IP address teacher's instrument 1700 is constant.Visible in addition teacher in 3D object 1725 the cube object and the cone-shaped body object between the slotted line made.Therefore, the similar measurement frame 1792 that illustrates, this measurement frame 1792 illustrates the teacher and makes length measured.
Figure 18-21 illustrates the Faculty and Students of the Figure 10-17 with significant difference.In Figure 18-21, student 1 throws off his view from teacher's view.In detail and with reference to Figure 19, the teacher can see he self instrument 1900 and student's 1 remote tools 1901 and student's 2 each of remote tools 1902.As can seeing in Figure 19, the teacher just utilizes his instrument 1900 to finish measurement from the last left rear corner of cube object to the tip of cone-shaped body object.Similarly,, see that now student 1 no longer locks, in this view, do not occur locking symbol thus with reference to Figure 20.Because student 1 throws off from teacher's view, he can check 3D object 2025 from top or any other his selected viewpoint.In addition, under disengaged mode, for example need not display control panel, as shown here, and in order to realize local interaction, can illustrate as at the one group of control knob of being presented in display one side described in the DextroLap application program to the student, or the local control panel of simplifying for example can be provided.As mentioned above, can for example make this this locality control panel become ash, be not that the teacher controls " truly " control panel down, then can utilize the simplification outward appearance of this this locality control panel and to need not change grey if perhaps can identify easily.With reference to Figure 21, student 2 view is shown.Student 2 still is locked into the teacher, therefore shows locking symbol 2102.Student 2 can see teacher's instrument 2100 and he's self instrument 2102.He can also see the slotted line that 3D object 2125 and teacher make.
Below with reference to Figure 22-25 the similar view with Figure 18 is shown, difference is that the position of teacher's pen changes.With reference to Figure 23, teacher's view notices that this view is identical except move a little the position of teacher's instrument with Figure 19.Thus, teacher's instrument 2300 in fact (essentially) be rotated at the tip of the cone-shaped body object in the 3D object 2325.Also them are seen by the same position place that can occupy in student 1 remote tools 2301 and student's 2 the view of remote tools 2302 at Figure 19.With reference to Figure 24, student 1 disengagement view, unique change be: the orientation of the position of teacher's instrument 2400 and orientation and student's 1 instrument 2401.As can relatively seeing by Figure 20 and 24, student 1 instrument is rotated down around the terminal point of measuring (coming down to the tip of the cone-shaped body in the 3D object 2425), so that compare the littler angle of angle that forms with this slotted line with slotted line formation in Figure 20.Notice that student 1 view does not illustrate student 2 cursor.This is because as mentioned above, only the teacher can see all students, and each student can only see the teacher.Thus, do not know other students' exist (certainly, becoming the teacher unless a student is switched the role, below this process of more detailed description) on each students ' actual situation.
Similarly, Fig. 2 illustrates student 2 view.Figure 25 is identical with Figure 21 except the position of teacher's remote tools 2500 and orientation in fact.Student's 2 self instrument 2502 is not moved, and 3D object 2525 does not move yet.Student 2 still is in the locking mode with respect to teacher's view, and shows locking symbol 2590 thus in this student's 2 view.
Figure 26 to 29 illustrates the replacement illustrative embodiments, and wherein, teacher and two students add the networking session.In Figure 26, the teacher is connected to network.This system's display message " you are the teacher " and can see teacher's virtual tool.Do not connect the student as yet.At Figure 27, first student adds.In data division (section), see his instrument and IP address, and can see the initial snapshot (and time of his adding) of his view.In Figure 28, second student adds, and his instrument and IP address can be obtained by the teacher now.Each student's IP address is rendered as near their text box of virtual tool.In Figure 29 and 30, the teacher is synchronous with each of first student and second student respectively.As mentioned above, who student the teacher can select with synchronous by select the student in the student's tabulation that shows from the left panel of networking control panel.
Figure 31-43 illustrates the sequence of a plurality of " surgeon " and " development assistant " view of the illustrative embodiments according to the present invention.These accompanying drawings illustrate: for example, and when the surgeon uses DEX-Ray TMThe type system undergos surgery, and uses for example Dextroscope TMOr how the illustration cooperation can take place when being connected on surgeon's the machine via network in the development assistant of DextroBeam type system.This example can also for example be applied to so any situation, in this case, personnel are just carrying out diagnosis or therapeutic process and are obtaining real-time information about the receptor thus, and another personnel are receiving this real time data and are using this real time data and produce receptor's development about receptor's 3D data, thereby assist first personnel.This illustration development assistant for example can use the freedom of the viewpoint that he can use, and (just he can be according to expecting rotation freely, translation and being amplified in object in the 3D data, and be not limited to check that with the surgeon position of these objects and orientation check these objects) look at the thing that the surgeon can not see, and thus collaboratively the guided surgery doctor by the patient's of art Yezhong tissue.These accompanying drawings are described below.
Figure 31 illustrates two illustrative development assistants (" VA ") view.Usually, in surgeon/development assistant example, surgeon and VA view are thrown off, unless he determines to lock and check that the optimization that VA has produced develops, this is optimized, and develop usually can be not relevant with him viewpoint and the angle of the route that physics uses in patient or receptor's manipulation.In the environment of Figure 31, and usually when using VA to assist to use operation guiding system such as Dex-Ray TMThe surgeon of type system, the surgeon can take action approx with the student, and as mentioned above, the development assistant then can take action approx with the teacher, as mentioned above.This is because such fact, because the surgeon's who undergos surgery viewpoint is confined to the viewpoint of detector or other apparatuses that he holds, so cause him to have the degree of freedom of less demonstration about the 3D virtual data.For example, use Dex-Ray TMThe viewpoint of the video camera in the system, the viewpoint of watching the 3D data and hand-held detector is identical.On the other hand, because VA only sees the 3D virtual data, so VA can change viewpoint without restriction.Because the common desired display augmented reality of surgeon (or, if use other operation guiding systems, then desired display 3D data are just as from seeing to the viewpoint of his tracked apparatus fully similar viewpoint), surgeon's view is usually and the disengagement of the view of VA.
Therefore, with reference to Figure 31, the development assistant has rotated skull, and interested object in its expression surgeon's operation or the operation is so that can see the crown view on the left side and the sagittal view on the right.In each development assistant's view, can see surgeon's's (taking on the student) remote tools 3100.In sagittal view, see the skull opening easily.Notice that the surgeon's of disengagement viewpoint is different from these each, and will discusses more completely in conjunction with Figure 34 below.As shown in figure 34, surgeon's actual view is more along the axle of surgeon's remote tools 3100, because this is his approach, followingly will know.Figure 32 and 33 is respectively the amplified version of Figure 31.
Figure 34 illustrates the view corresponding to the illustrative surgeon of the development assistant's of Figure 31 view.This view illustrates the actual view that the surgeon has, and this actual view is by for example Dex-Ray that is illustrated in him locally TMIn the type system.Surgeon's viewpoint is corresponding to the viewpoint (and by its restriction) of the video camera in the video camera detector that he held, described in applying at Camera Probe, or, for example, if use another operation guiding system, then corresponding to the physical direction and the path of his route in the patient.In Figure 34, actual video camera detector or navigation detector or apparatus that surgeon's instrument 3400 is held corresponding to him.Be also shown among Figure 34: skull opening 3410; Surgeon's IP address 3416, it can utilize some other identifiers to replace or strengthen as mentioned above; And the visible spheroid 3405 of part only.The development assistant can obtain the better view of this object by the viewpoint of optimizing him, thereby helps surgeon's anchor point in the above.Suppose that in Figure 31 to 43 spheroid 3405 (with reference to Figure 34) is represented interested object, the tumour of handling such as the surgeon.
Corresponding surgical doctor's the disengagement view that Figure 35 illustrates development assistant's view and has the skull of skull opening, as mentioned above.In Figure 35, the development assistant assists surgeon's anchor point on spheroid.The development assistant has cut off skull in his view sidepiece exposes spheroid.On the other hand, as described, surgeon's view, Figure 35 (b) is subjected to the restriction of the following fact: his detector moves in real world, can only move in the true hole in skull or the skull opening.On the contrary, development assistant's view is unrestricted, thereby makes his manipulation data this spheroid that develops best freely.In each view of Figure 35 (a), can see surgeon's instrument 3500.As can seeing in Figure 35 (b), the axle of surgeon's instrument is more or less corresponding to his viewpoint, and in Figure 35 (a), the development assistant checks object from behind with respect to surgeon's true path.Figure 36 and 37 is Figure 35 (a) and corresponding amplification (b).As can in Figure 37, seeing, clearly illustrate surgeon's IP address 3716.
VI. integrated with other operation guiding systems
In illustrative embodiments of the present invention, can will be connected to the mutual toning system of 3D from the equipment with 3D tracking (or 2D follows the tracks of) performance of the manufacturer that changes via DextroNet, such as Dextroscope TMShould will need and to provide some information to this DextroNet in response to the inquiry or the order that send via the DextroNet network by " other places " equipment.For example, the Medtronic of U.S. Colorado state Louisville production is used for multiple navigation (or the image guiding) system of neurosurgery.Such navigational system is TREON for example TMSystem.Medtronic is production application program interface (API) socket software also, and it allows data to flow to external application in real time from this navigational system.
Similarly, another manufacturer of family, the BrainLAB AG of German Munich has similar software product.This product uses the client/server architecture of custom-designed, is called vector vision link (VV link), and it has expanded function from developer tool case (VTK).The VV link makes it possible to carry out in real time bidirectional data transfers, such as image data set, development and tool location.These equipment with DEX-Ray TMThe mode of system similarity provides registration (registration) information and detector coordinates information.Yet, because they are based on strengthening demonstration, so video information is not provided.In illustrative embodiments of the present invention, the modification of DextroNet server can be in conjunction with such as Stealthlink TMOr VV link software, and after connection, wherein can exchange patient information and registration details, DextroNet for example can inquire that these systems obtain their detector coordinates.Thus, in illustrative embodiments of the present invention, these systems can be used as surgeon's workstation, and can provide volume coordinate to arrive teacher (VA) workstation.Note, with above-mentioned DEX-Ray TMRealize differently, can not have the option that shows view at intra-operative from the workstation of VA to the surgeon as these systems of current configuration.In order to do these, the software that then needs them will implement the method for illustrative embodiments of the present invention is attached in their navigational system.
Similarly, notice as above that in illustrative embodiments of the present invention, the machine that stride illustration DextroNet connection can have different configurations from different manufacturers, and has different types.As mentioned above, such as Dex-Ray TMThe operation guiding system of system or Medtronic or BrainLAB system can be striden DextroNet and is connected to such as Dextroscope TM, DextroBeam TMDeng the standard 3D workstation that develops alternately.Be also noted that as top, only send the system of 3D position data,, also can be connected such as the operation guiding system that does not use augmented reality.
Figure 44 that describes below-49 presents another selection of the DextroNet of the illustrative embodiments according to the present invention to be used.This example comprises: illustrative cardiologist produces the fluoroscopic image of 2D, and this 2D fluoroscopic image is sent to mutual 3D toning system via DextroNet, such as Dextroscope TMExample is similar to above-mentioned surgeon and development assistant's example.Yet, in this case, do not have the surgeon and have plenty of the intervention cardiologist.In this illustrative embodiments, because the unrestricted 3D of the CTA that the development assistant obtains before for art scanning handles, the assistant that causes developing can help the develop anatomical structure of his concern of this interventions cardiologist.
Thus, with reference to Figure 44, the illustration fluoroscopic image that obtains from Cathlab is shown.This image obtains by irradiation X ray on patient's chest.In image, artery can be seen, or rather, those parts of the inside of the artery that absorbs the contrast preparation of taking can be seen.
Figure 45 illustrates the view that the illustration standard gets involved the cardiologist.With reference to Figure 45, get involved the 2D projection that the cardiologist only is provided from the viewpoint that is provided by illustration fluoroscopy equipment by the vascular that absorbs the contrast preparation of taking.Image shown in Figure 45 is the simulated projections (not obtaining actual fluoroscopic image and the set of matches of the CTA that is associated) that this tradition gets involved cardiologist's view.By data manipulation obtains simulation (coronary artery of cutting apart CTA (only illustrates this artery thus and its hetero-organization is not shown to CTA, it is the result who flows into artery and act on by the X ray reciprocation of fluoroscopy equipment emission the time when contrast preparation), they dead color is fluoroscopic results just as them, and determine the orientation of the artery that (orient) cut apart and obtain snapshot, obtain snapshot then without the CTA of cutting apart).
Figure 46 illustrates the illustrative development assistant's corresponding with clinician's view of Figure 45 view.This illustrative development assistant can cooperate with the intervention cardiologist of Figure 45.As notice that the unrestricted 3D that this development assistant has for CTA before the art handles.Figure 46 illustrates the zoomed-in view coronarius of VA checking.It is mutual in the illustration that gets involved between cardiologist and the development assistant that Figure 46-48 illustrates according to the present invention illustrative embodiments.They illustrate the manual methods with CTA view and fluoroscopy view registration.Here, for example, VA can obtain fluoroscopic image such as Figure 45 via DextroNet, and then, for example, uses this image can regulate CTA (coronary artery of for example cutting apart) to mate the view of this reception.In case VA orientation (oriented), and know that the cardiologist checks what (has the restriction on the fluoroscopy equipment, it is not tracked usually, but have some normal places that the cardiologist understands very much), unrestricted then 3D handles and allows VA to indicate to the cardiologist what what is.He can for example utilize such as " arteria coroaria sinistra " or " LCA " and similarly for example the note of " arteria coronaria dextra " or " RCA " come those vasculars of mark, perhaps for example, he can point to the stenosis in (3D) vascular, this stenosis (if projection or can obtain this demonstration then three-dimensional ground) in the Cathlab image can be offered and gets involved the cardiologist then.In addition, for example, VA can measure, and if VA can see new images from the fluoroscopy instrument, where and infer the 3D position he can discern conduit, will arrive the Distance Transmission of crucial anatomic landmark then and return the cardiologist.
Figure 47 illustrates and the similar scene of the scene of above-mentioned Fig. 8.The development assistant can see complete 3D rendering and master image, and can see snapshot or " picture-in-picture " image in upper left corner.This pip image is produced by for example illustration fluoroscopy equipment at the Cathlab machine place that gets involved the cardiologist.In fact be exactly above-mentioned image shown in Figure 45.The development assistant can use Dextroscope TMOr equivalent of the apparatus, handling CTA data before the art, with best from fluoroscopy equipment identical viewpoint cut apart (or cutting apart again) and the vascular that develops.He can be for example finishes these by what viewpoint is registered to he sees in picture-in-picture.Illustrative interactive button can also be seen in right side, distant place at the view of VA, these interactive button are similar in the DextroLap realization total those buttons of seeing, illustrate, if environment needs then VA can for example use DextroLap, or for example, he can use complete Dextroscope TM
Figure 48 illustrates that VA can generate can select illustration 3D view in addition.That is to say, as noted, because VA can visit whole CTA data, can for example obtain (bring up) thus and obtain the plane, as shown in the left-side images of Figure 48, or for example can partition data only to represent coronary artery, as shown in the image right of Figure 48.
Finally, Figure 49 illustrates the image side by side that can for example show at the Cathlab equipment place that gets involved the cardiologist.Get involved fluoroscopy view (left side) that the cardiologist can be on identical display obtains him by local machine and by at for example Dextroscope TMOr the view that the development assistant on the DextroLap machine produces compares.This comparison mechanism (facility) can allow the cardiologist to explain fluoroscopic projection better.Use is getting involved communicating by letter between cardiologist and the development assistant, the development assistant can be for example according to the expectation that gets involved the cardiologist and need to he local produce and send to via DextroNet get involved the cardiologist for showing that (as shown in figure 49) and the view that is compared by the intervention cardiologist carry out for example refining (refine), cut apart and optimize.This can finish by for example using such as the feature of Cathlab system, and this Cathlab system has a plurality of monitors that the fluoroscopy program is shown.Adding 3D rendering to another monitor is simple task.The image of these back can for example mate fluoroscopy or the fluoroscopy that do not match.This system also illustrates the fluoroscopy device location, and other patient informations.Or, for example, other displays, the monitor with simple touch-screen that uses down such as aseptic condition can be used to the development of VA is shown back to the clinician.
VII. the role is switched
In illustrative embodiments of the present invention, can support role's handoff functionality.It is feature in illustration DextroNet that the role is switched, and it allows their roles separately of the online exchange of teacher and student (or surgeon and development assistant).In teacher-student's example, the student can not be in 3D data centralization manipulating objects except translation, rotation and point at objects.Utilize the role to switch, in case the student takes over control from the teacher, then he can for example continue the work of teacher for object.This advises collaboration mode on some degree.In addition, because on network, only there is a teacher at every turn,, and there is not collision problem so this cooperation is serial.
In illustrative DextroNet, Faculty and Students can be the low layer users, and it is very natural that this makes the role switch.The role is switched can use current communication session (networking just need not to stop and reconnecting), and can exchange the role of Faculty and Students on high level.By this way, used consuming time by avoiding reconnecting, the role is switched can be very fast.
The role is switched can support a plurality of students.The teacher determines which student he transfers with control.Other students can keep intact, or for example can be apprised of the teacher and be changed.Notice that as above in illustrative embodiments of the present invention, teacher and (several) student can be the client computer on the low layer meaning for example.As shown in Figure 9, in illustrative embodiments of the present invention, DextroNet can use the server-client framework.In this framework, Faculty and Students all are client computer.This server can also for example be used to manage communicating by letter between teacher and a plurality of students.Thus, role's switching can be used current communication session and need not to stop and reconnecting networking.During whole process, need not to take place the change of physical connection.Only need on server, specify and re-register the logic new role.The general procedure of transmission changing role can be for example as described below in All Clients.After student's control right request, the teacher informs that other students and server be ready for the role and switch.Then, for example, Faculty and Students can followingly change the instrument of reseting and upgrading them according to the illustration role:
The machine that will become student's teacher becomes student's instrument into local instrument, and his teacher's instrument is changed into the remote tools (beginning teacher's instrument just) of appointment.From shown instrument, remove other long-range student's instruments.
The machine that will become teacher's student becomes his teacher's instrument into local instrument, and the instrument that adds other students and will become student's teacher is represented (student's instrument).
Other students' machine upgrades their teacher's instrument with the expression beginning teacher.Their local instrument keeps as student's instrument.
Change in case finished these roles, All Clients can re-register their new role on server.Can finish the role then switches.
Because use ongoing communication session, this communication session can be supported a plurality of students, so may should be noted that the management of data stream.Before the illustration role is switched and the data that received afterwards need for example separatedly, and re-registering of the role on server need be for example by synchronously.In order to realize these, then whole process can for example be divided into three phases: for the role is switched ready (Phase I), changes role's (Phase) and re-register new role (Phase I) on server.When each finishes in stage, the state of All Clients synchronously.
This process can be summarized as follows with reference to figure 50-52.
A. Phase I (Figure 50)
(1) all students of teacher's signaling he whom control can be transferred.After he received the confirmation from his all students, his instrument was reset, and notification server he be ready for the role now and switch, and enter snake god (zombie) state (just but machine can only receive can not send the state of data).
(2) when the student receives signal from the teacher, his hardware check he can to become the teacher still be the student of remaining as the same, and confirm to the teacher.Simultaneously, also needing notification server to be ready for the role now after he resets his instrument switches.This is not reseted can influence the change that has taken place on the volumetric objects.Then, he enters passive or " snake god " state, and this state is that described wherein machine only receives message and do not send anything state.
In illustrative embodiments of the present invention, the user need not to worry all notices in role's handoff procedure., the user can automatically finish all role's hand-off process (and in fact all being called the illustration software of realizing these actions about the teacher of above-mentioned " doing some things " or student and ensuing thing thus) in case pressing " role's switching " button.
In illustrative embodiments of the present invention, server can be used to manage the communication between Faculty and Students.It can have store list for example and remember their role.Therefore, when switching the role, need notification server.In addition, between transfer period, when each extreme end, need the state of synchronous Faculty and Students the role.For example, server can also be coordinated this process.For example, when Phase I finished, student/teacher needs notification server, and whether he had been ready for role's switching.After this, he can enter the refreshing state of snake, waits for from he of server changing role's instruction now.How many client computer server can for example be counted and be ready for role's switching.Only after he found that All Clients is ready to, they can enter Phase thereby he can send instructions to everyone.
B. Phase (Figure 51):
(3) be ready for the role when switching when the discovering server All Clients, he send a message to them and with them from the refreshing recovering state of snake.
(4) in a single day recover client computer (teacher/student), client computer changes his role, notification server and enters the refreshing state of snake once more.
C. Phase I (Figure 52):
(5) his client computer obtains the role and switches message in case server is from all, and then he at first recovers the teacher.
(6) teacher re-registers on server and notification server.Then, server recovers the student.
(7) on server, re-register and after the teacher received initial snapshot from the student, teacher's request server recovered next student the student.
(8) for example, can repeating step (7) finish and re-register up to all students.
Additionally, in illustrative embodiments of the present invention, for a plurality of participants, the teacher role can transmit between the participant, and said process is followed in each role's switching.
VIII. data layout
In illustrative embodiments of the present invention, can use following strategy to guarantee suitable demonstration in remote terminal:
A. data:
Every side (teacher/student) keeps the identical copies of data.If the data difference then can synchronous whole data set (compression also sends data file).
B. start:
In illustrative embodiments of the present invention, when starting network savvy, can be on both sides that virtual control panel is synchronous.In Dextroscope/DextroBeam workstation for example, virtual control panel need utilize mutual during physics third rare ester group seat of being shelved of 3D tracker accurately calibrate so that virtual tool can touch the counterpart on virtual control panel.Thus, calibration and configuration are local, and different along with the variation of machine.Mismatch problems when this can for example cause networking: teacher's instrument may for example can not touch student's control panel.For fear of this problem, in illustrative embodiments of the present invention, can the synchro control panel.When activating network savvy, position, orientation and the size of teacher's control panel can be sent to the student, and replace the parameter of student's self control panel.When network savvy finishes, can recover the student before be configured to work alone.
In illustrative embodiments of the present invention, when starting network savvy, the viewpoint on both sides synchronously.For appropriate display, Faculty and Students should share identical eye position, check the position, projection width and height, roll angle (roll angle) etc.All information that belong to viewpoint thus should be by synchronously when the beginning of network savvy.
In illustrative embodiments of the present invention, when starting network savvy, the electronic zoom frame on the both sides synchronously.Thus, when network savvy began, the electronic zoom frame on the both sides need utilize position, orientation, scope (bound), computer screen zone to wait to carry out synchronously.
C. synchronous window widget (widget)
In illustrative embodiments of the present invention, when network savvy began, the original state of the control panel on the both sides can be different: the position of ticker position, button and label, the tabulation of color lookup table etc.Need to proofread and correct all these parameters.
D. communication
In illustrative embodiments of the present invention, can use two types coordinate system: world coordinates and object coordinate.World coordinates is the coordinate that is attached on the virtual world.The object coordinate is the coordinate on each virtual objects that is attached in the virtual world.In illustrative embodiments of the present invention, can in world coordinates, show all virtual tools.Can the communicate by letter type of the coordinate system that they use of each of Faculty and Students.
When the student was in engaged mode (" LockOn "), Faculty and Students all sent the title of his instrument, the state in world coordinates, position, orientation and the size companion to him.
When the student was in disengaged mode (not having " LockOn "), on teacher's one side, if teacher's instrument touch control panel, then the teacher sent title, the state in the world coordinates, position, orientation and the size of instrument to the student.Otherwise state, position, orientation and size that he sends in title, the object coordinate arrive the student.When the student received the relevant information of instrument with the teacher, he can be converted to world coordinates from the object coordinate with the information that receives, and the instrument that shows the teacher then in his world.Therebetween, the student can send position, orientation and size in the title, state, object coordinate of his instrument to the teacher.Then, the teacher can be changed them into world coordinates before the instrument that shows the student.In illustrative embodiments of the present invention, the student can decide according to teacher's position, orientation and the state of virtual tool in his world and take what action.
In illustrative embodiments of the present invention, DextroNet is the viewpoint pattern on Faculty and Students' side synchronously.When the student selected to throw off his viewpoint, he can for example press the button on his stylus.This action can cause sending a message to teacher's machine to reach the effect of " I will throw off ".So far.In fact the student does not for example switch his viewpoint.He continues to use the world coordinates from the teacher.When the machine that serves as a teacher received student's message, this teacher's machine can for example send an acknowledgement to the student then, and changed the object coordinate from then on.In case after student's machine received this affirmation from teacher's machine, then in fact student's machine can change into disengagement then, and can use the object coordinate then.
For example, situation can be identical when the student engages once more.Thus, in illustrative embodiments of the present invention, student's machine can be only changes student's viewpoint after teacher's machine becomes the disengagement decision of knowing the student.By this way, can avoid throw off or engage once more before and send to conflict between student's the coordinate type of machine afterwards by teacher's machine.
E. message format
In illustrative embodiments of the present invention, two kinds of message formats for example can be arranged.A kind ofly for example can be used for updating message, another kind for example can be used for file.
1. the form I that is used for updating message
Figure 54 illustrates the first illustration form that is used for updating message.For example can use following field with following attribute.
Beginning label: the beginning of mark message (signless character);
Data type: it is message or the file that upgrades that content is shown, and this value is 2 (signless integers) for updating message;
IP: sender's IP address (unsigned character);
Object oriented: be appointed as the object that uses this message;
Coordinate system: the coordinate system (unsigned character) that is used for representing position, orientation and the size of message.This can for example be two possible values: for world coordinate system is " wld ", is " app " for the object coordinate system;
Position: the position of the object in the Object Name.The position comprises three values: floating point values x, y and z;
Orientation: the orientation of the object in the Object Name.Orientation is 4 * 4 matrixes.Each element in the matrix is a floating-point;
State: the state of the object in the Object Name when needing (unsigned character).If to liking instrument, this value is one of four kinds of states: MK_CHECK, MK_START_ACTION, MK_DO_ACTION, MK_END_ACTION;
Size: the size of the object in the Object Name.Size comprises three values: floating point values x, y, z; And
End-tag: the end of mark message (unsigned character).
Figure 55 (a) illustrates three examples of the form that uses Figure 54 to 55 (c).
Figure 55 (a) illustrates the updating message that is used for the synchro control panel.
Figure 55 (b) illustrates the updating message that is used for the synchronous window widget, and
Figure 55 (c) illustrates the updating message that is used for synchronous virtual tool.
2. file transfer
In illustrative embodiments of the present invention, long article spare can be divided into piece for transmission.Each message can comprise such piece.In illustrative embodiments of the present invention, before the actual transmissions file, can send updating message and inform that the other side will transfer files.
Form I shown in Figure 54, can be used for such updating message, as long as can revise " size " field so that comprise total block data (signless integer), piece size (signless integer) and last piece size (signless integer).The example of this updating message is provided in Figure 56.Figure 56 illustrates the illustration updating message about the transmission of the illustration file of 991KB (1014921 byte).Thus, provide the data in the size field, the other side knows that file has 248 pieces, and except each piece of last piece has the size of 4096 bytes, and last piece has 3209 bytes.File self can use second form that is used for updating message to send, and this second form is suitable for file transfer.
3. the form II that is used for updating message
Figure 57 illustrates the second illustration form that is used for updating message.In illustrative embodiments of the present invention, can use following field:
Beginning label: the beginning of mark message (unsigned character);
Data type: it is message or the file that upgrades that content is shown, and this value is 1 (signless integer) for file;
Blocks of files: binary blocks of files (unsigned character); And
End-tag: the end of mark message (unsigned character).
In illustrative file transfer, 247 pieces at first, each has 4096 bytes, can for example shown in Figure 58 (a), be sent out, and last piece, having 3209 bytes, can shown in Figure 58 (b), be sent out by using form II.
Though described the present invention with reference to one or more illustrative embodiments of the present invention, but the invention is not restricted to this, and claims intentions be interpreted as not only comprising shown in particular form of the present invention and variant, also comprise form and the variant that can make such as those skilled in the art under the situation that does not depart from true scope of the present invention.

Claims (57)

1. be used for alternatively handling the equipment of 3-D view, described image comprises the volume data that is generated by the imaging data about object, and described equipment is configured to:
Receive the position data of one or more remote probe devices of one or more remote machines via communication link;
The three-dimensional scenic that generates combination is for showing on local display, and the three-dimensional scenic of described combination comprises described at least one remote probe device and described 3-D view;
In response to by for the user of described equipment this locality manipulation, handle described 3-D view to local explorer; And
Send data via described communication link about the described manipulation of being undertaken by described user for described equipment this locality, described data enough are used for allowing described at least one remote machine to show the three-dimensional scenic of combination, and the three-dimensional scenic of described combination comprises the image of described 3-D view being carried out the described local explorer of handling.
2. equipment according to claim 1 also is configured to: in response to network condition, filter the transmission data about the described manipulation of being undertaken by the described user to described equipment this locality.
3. equipment according to claim 3, wherein, described filtration filters the packet of not revising described 3-D view substantially.
4. equipment according to claim 3, wherein, described filtration filters with instrument and moves and one of tool state relevant grouping.
5. equipment according to claim 1 also is configured to: at least one of IP address, user's name and user's designator that shows described one or more remote machines.
6. equipment according to claim 1, wherein, send to the described data about the described manipulation undertaken by described user of described one or more remote machines for described equipment this locality, enough be used for allowing: it similarly is the same mode of detector that is associated with described one or more remote machines that described one or more remote machines adopt described local explorer, undertaken alternately by virtual control panel, be presented at the image of carrying out the described local explorer of handling on the described 3-D view with described local machine.
7. equipment according to claim 1 also is configured to: in response to the order for the described user of described equipment this locality, receive the snapshot of the described demonstration of described one or more remote machines.
8. equipment according to claim 1 also is configured to: in response to order for the described user of described equipment this locality, cause between any remote machine in described equipment and described one or more remote machine 3-D view synchronously.
9. equipment according to claim 8 wherein, causes describedly comprising synchronously: send as described in preserve on the equipment as described in 3-D view the compression copy to as described in one or more remote machines.
10. equipment according to claim 8 wherein, causes describedly comprising synchronously: send the tabulation of interactive command, be at described equipment and carry out described interactive command on the described 3-D view.
11. according to any one described equipment of claim 1-10, described equipment also is configured to: in response to the request by the long-distance user of remote machine, the role of switching described local machine and described remote machine.
12. be used for alternatively handling the equipment of 3-D view, described image comprises the volume data that is generated by the imaging data about object, described equipment is configured to:
Position data via the remote probe device of communication link receiving remote machine;
Receive the position data of local explorer;
The three-dimensional scenic that generates combination is for showing on display, and the three-dimensional scenic of described combination comprises described remote probe device, described local explorer and described 3-D view; And
Be equivalent to basically by for the user of described remote machine this locality via the mode of described local explorer for the manipulation of described 3-D view, in response to the manipulation of described remote probe device, handle described 3-D view.
13. equipment according to claim 12, wherein, be shown image in response to the manipulation for described 3-D view of the manipulation of described remote probe device as described remote probe device, described remote probe device adopt just as described remote probe device be that the mode of the detector that is associated with described equipment and the virtual control panel of described local machine carry out alternately.
14. equipment according to claim 12 also is configured to: at least one of IP address, user's name and user's designator that shows described remote machine.
15. equipment according to claim 12 also is configured to:, send the snapshot of described demonstration in response to the order of described remote machine.
16. equipment according to claim 12 also is configured to: in response to order, use the view data that sends via described communication link from described remote machine, come synchronous described 3-D view from described remote machine.
17. equipment according to claim 12, also be configured to: the viewpoint with one of following viewpoint shows described 3-D view: just as viewpoint for the user's of described remote machine this locality viewpoint, and with the different any viewpoint of viewpoint for the user of described remote machine this locality.
18. equipment according to claim 17 also is configured to show virtual control panel.
19. equipment according to claim 17, also be configured to: when when showing described 3-D view for the different any viewpoint place of the user's of described remote machine this locality viewpoint, in response to order, show special-purpose local control panel for the user of described equipment this locality.
20., also be configured to allow the user that described 3-D view is carried out local operation according to any one described equipment of claim 12-19.
21. equipment according to claim 20, wherein, described local operation comprises does not influence the manipulation which voxel is considered to the part of object.
22. equipment according to claim 20, wherein, described local operation comprises setting one or more of the translation of object and rotation and enlargement ratio and transparency.
23., also be configured to: create the additional local replica of described 3-D view, and handle described additional copies in response to the manipulation that receives from user for described equipment this locality according to claim 12 or 17 described equipment.
24. equipment according to claim 1, described equipment also is configured to: from local for described object and receive additional data about described object for the local remote equipment of one of described remote machine, described additional data is registrated to the described 3-D view of described object jointly.
25. equipment according to claim 24, wherein, about the described additional data acquisition in real time basically of described object.
26. equipment according to claim 24, wherein, be following one or more about the described additional data of described object: real-time video, pre-recorded dvd-video, for the detector of described object this locality or position data, fluoroscopic image, ultrasonoscopy and the multi-modality images of apparatus.
27. any one the described equipment according to claim 1-12 also is configured to: show from local for described object and for the additional data about described object of the local remote equipment of one of described remote machine.
28. equipment according to claim 27, wherein, described additional data is registrated to the described 3-D view of described object jointly.
29. equipment according to claim 27, wherein, described additional data is real-time basically.
30. equipment according to claim 27, wherein, in the described additional data, be following one or more: real-time video, pre-recorded dvd-video, for the detector of described object this locality or position data, fluoroscopic image, ultrasonoscopy and the multi-modality images of apparatus about the described additional data of described object.
31. be used for alternatively handling the system of 3-D view, described image comprises the volume data that is generated by the imaging data about object, described system comprises:
Main workstation comprises any one described equipment of claim 1-11 or 27-30;
One or more distant places workstation comprises any one described equipment of claim 12-26; And
Data network,
Wherein, described main workstation with described one or more distant places workstation each be connected via described data network.
32. be used for alternatively handling the method for 3-D view, described image comprises the volume data that is generated by the imaging data about object, described method comprises:
Any one described first equipment according to claim 1-11 or 27-30 is provided;
Any one described one or more second equipment according to claim 12-26 are provided; And
Data network is provided, and connects described first equipment and described one or more second equipment via described data network,
Wherein, in operation, user and the 3D data set that has in the development of the user collaboration ground at described one or more second equipment places at the described first equipment place.
33. method according to claim 33 wherein, starts back switching role the user at the described first equipment place and the user of user at the described first equipment place at the second equipment place.
34. networked interactive three-dimensional data toning system comprises:
Main system is used to obtain the realtime graphic of object and with the part combination of the 3D volumetric model of the common registration of they and described object;
Display is used for described combination image is shown at least one user;
Detector;
Tracking cell is used to follow the tracks of described position of detector;
Data network; And
One or more remote systems are used for the described combination image that alternatively develops, and are connected to described main system via described data network communications, and each has tracked virtual tool;
Wherein, receive the tracked position and the described combination image of described main system detector by each remote system via described data network, and wherein, each described remote system can alternatively be handled the 3D volumetric model of described combination image or described common registration and the demonstration of the image of described manipulation is transferred to described main system and every other remote system.
35. system according to claim 34, wherein, described combination image is the 2D real-time video that covers on the described object.
36. according to any one described system of claim 34 or claim 35, wherein, described main workstation and described remote work station are suitable for revising computer generated image and simulate the operation of carrying out on described objects.
37. system according to claim 36, wherein, the operation of described simulation comprises the part of the described volumetric model that removes described object.
38. system according to claim 34, wherein, described main system or remote system are suitable for receiving from least one user the painted change of the partitioning portion of described volumetric model.
39. system according to claim 34, wherein, described main system is suitable for receiving from first user selected point of the described volumetric model that is used to measure, and described second system is suitable for receiving from second user selected point of the described volumetric model that is used to measure.
40. system according to claim 34, wherein, described main system or remote system receive input from least one user and amplify to carry out zoom at shown area-of-interest.
41. system according to claim 34, wherein, described main system or remote system are suitable for receiving the input from least one user, with the transparency or the opacity of at least one cutting object of changing described volumetric model.
42. system according to claim 34, wherein, described main system or remote system are suitable for revising the change of described combination image with the physical form of the described object of representing described operation, and the tracked position of described detector is depended in described modification.
43. be used for the method used by at least one user in the 3D region executable operations of definition, described method comprises:
The image of the object of generating run shows that with described object registration ground described image gives described at least one user, and utilizes first system keeps track to have the position of detector of the longitudinal axis, and with described location transmission to first data processing equipment and second system,
Wherein, described first data processing equipment generates image according to the straight line that the longitudinal axis that is parallel to described detector extends, and described straight line has the extension of controlling according to the output of the extension opertaing device of being controlled by first user; And
Wherein, described second system with second data processing equipment generates at least one image of the object of described operation on described at least one display, described image is shown to described at least one user, wherein, receive the tracked position of the detector of described first system via communication network by described second system, and
Control described first and second data processing equipments,, revise described at least one image of the object of described operation with controlled extension according to described straight line.
44. according to the described method of claim 43, wherein, the described display of described first system generates the image of the described object that covers the described operation on the described object.
45. according to claim 43 or the described method of claim 44, wherein, described first or second data processing equipment is revised computer generated image and simulate the operation of carrying out on described object, and the operation of described simulation is controlled by the extension of controlling described straight line.
46. according to the described method of claim 45, wherein, the operation of described simulation comprises: the degree of depth of part in the patient who indicates by the extension of described straight line that removes described computer generated image.
47. according to the described method of claim 43, wherein, described first system or described second system generate the painted change of image from least one user's receiving computer.
48. according to the described method of claim 43, wherein, described first system receives the selected point that is used to measure from first user, and described second system receives the selected point that is used to measure from second user.
49. according to the described method of claim 43, wherein, described first system or described second system receive input from least one user and amplify to carry out zoom at shown area-of-interest.
50. according to the described method of claim 43, wherein, described first system or described second system receive input to change at least one of described transparency or opacity from least one user.
51. according to the described method of claim 43, wherein, described first data processing equipment or described second data processing equipment are revised the change of described image with the physical form of the described object of representing described operation, and the tracked position of described detector is depended in described modification.
52. networked interactive three-dimensional data toning system comprises:
Main system;
Display is used at least one user's display image;
Detector;
Tracking cell is used to follow the tracks of described position of detector;
Data network; And
One or more remote systems are used for the described image that alternatively develops, and are connected to described main system via described data network communications, and each of described remote system has tracked virtual tool;
Wherein, receive the tracked position and the described image of described main system detector by each remote system via described data network, and wherein, each described remote system can alternatively handle the 3D volumetric model of described combination image or common registration and with its location transmission of tracked virtual tool to described main system.
53., wherein, can repeatedly change the role of described main system and remote system according to the described system of claim 52.
54. according to the described system of claim 52, wherein, described remote system can be checked the view of the view of described main system of data set or it self.
55. according to the described system of claim 52, wherein, described remote system can be carried out any development and handle, but can not revise described data set locally.
56. according to the described system of claim 52, wherein, described main system can be at any time carried out its interface and data set and any remote system synchronously.
57. collaborative interactive ground develops and the method for manipulation volumetric objects or system, comprising:
The local user uses main workstation to obtain the realtime graphic of object or system and with they 3D data set combinations with the common registration of identical object or system, described local user shows described combination image locally, and transmits them via data network; And
One or more long-distance users, each is connected to described data network, uses remote work station to receive described combination image and handles at least one of scan image of described combination image and described common registration;
Wherein, each of described main user and described long-distance user can each be handled described combination image and described 3D data set, and wherein, described manipulation also sends to carry out the demonstration of basic synchronization on described main workstation and each remote work station via described data network.
CNA2007800018041A 2005-12-31 2007-01-03 Systems and methods for collaborative interactive visualization of 3D data sets over a network ('DextroNet') Pending CN101405769A (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US75565805P 2005-12-31 2005-12-31
US60/755,658 2005-12-31
US60/845,654 2006-09-19
US60/875,914 2006-12-19

Publications (1)

Publication Number Publication Date
CN101405769A true CN101405769A (en) 2009-04-08

Family

ID=40538816

Family Applications (1)

Application Number Title Priority Date Filing Date
CNA2007800018041A Pending CN101405769A (en) 2005-12-31 2007-01-03 Systems and methods for collaborative interactive visualization of 3D data sets over a network ('DextroNet')

Country Status (1)

Country Link
CN (1) CN101405769A (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102085101A (en) * 2009-11-16 2011-06-08 西门子公司 Method and device for identifying and assigning coronary calcification to a coronary vessel and computer program product
CN102542127A (en) * 2010-12-28 2012-07-04 通用电气公司 Systems and methods for smart medical collaboration
CN104011741A (en) * 2011-08-12 2014-08-27 Vipaar有限公司 System and method for image registration of multiple video streams
CN104068877A (en) * 2013-03-27 2014-10-01 西门子公司 Method for image support and X-ray device
US9940750B2 (en) 2013-06-27 2018-04-10 Help Lighting, Inc. System and method for role negotiation in multi-reality environments
US9959629B2 (en) 2012-05-21 2018-05-01 Help Lighting, Inc. System and method for managing spatiotemporal uncertainty
CN108140264A (en) * 2015-09-26 2018-06-08 波士顿科学医学有限公司 The dissection shell editor of adjustable-depth
CN108335336A (en) * 2017-01-20 2018-07-27 深圳市恩普电子技术有限公司 Ultrasonic imaging method and device
CN109564471A (en) * 2016-08-12 2019-04-02 波士顿科学国际有限公司 Distributed interactive medical visualization system with major/minor interaction feature

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8938106B2 (en) 2009-11-16 2015-01-20 Siemens Aktiengesellschaft Method and device for identifying and assigning coronary calcification to a coronary vessel and computer program product
CN102085101A (en) * 2009-11-16 2011-06-08 西门子公司 Method and device for identifying and assigning coronary calcification to a coronary vessel and computer program product
CN102085101B (en) * 2009-11-16 2016-02-17 西门子公司 For identifying coronary calcification thing and the method and apparatus be associated with coronary vasodilator
CN102542127A (en) * 2010-12-28 2012-07-04 通用电气公司 Systems and methods for smart medical collaboration
US10181361B2 (en) 2011-08-12 2019-01-15 Help Lightning, Inc. System and method for image registration of multiple video streams
CN104011741A (en) * 2011-08-12 2014-08-27 Vipaar有限公司 System and method for image registration of multiple video streams
CN104011741B (en) * 2011-08-12 2017-09-01 Vipaar有限公司 The system and method for the image registration of multi-path video stream
US9886552B2 (en) 2011-08-12 2018-02-06 Help Lighting, Inc. System and method for image registration of multiple video streams
US10622111B2 (en) 2011-08-12 2020-04-14 Help Lightning, Inc. System and method for image registration of multiple video streams
US9959629B2 (en) 2012-05-21 2018-05-01 Help Lighting, Inc. System and method for managing spatiotemporal uncertainty
CN104068877A (en) * 2013-03-27 2014-10-01 西门子公司 Method for image support and X-ray device
US10482673B2 (en) 2013-06-27 2019-11-19 Help Lightning, Inc. System and method for role negotiation in multi-reality environments
US9940750B2 (en) 2013-06-27 2018-04-10 Help Lighting, Inc. System and method for role negotiation in multi-reality environments
CN108140264A (en) * 2015-09-26 2018-06-08 波士顿科学医学有限公司 The dissection shell editor of adjustable-depth
CN108140264B (en) * 2015-09-26 2022-06-28 波士顿科学医学有限公司 Adjustable depth anatomical shell editing
CN109564471A (en) * 2016-08-12 2019-04-02 波士顿科学国际有限公司 Distributed interactive medical visualization system with major/minor interaction feature
CN108335336A (en) * 2017-01-20 2018-07-27 深圳市恩普电子技术有限公司 Ultrasonic imaging method and device
CN108335336B (en) * 2017-01-20 2024-04-02 深圳市恩普电子技术有限公司 Ultrasonic imaging method and device

Similar Documents

Publication Publication Date Title
CN101405769A (en) Systems and methods for collaborative interactive visualization of 3D data sets over a network ('DextroNet')
US11730545B2 (en) System and method for multi-client deployment of augmented reality instrument tracking
US20070248261A1 (en) Systems and methods for collaborative interactive visualization of 3D data sets over a network ("DextroNet")
Sauer et al. Mixed reality in visceral surgery: development of a suitable workflow and evaluation of intraoperative use-cases
AU2013370334B2 (en) System and method for role-switching in multi-reality environments
CN107822690B (en) Hybrid image/scene renderer with hands-free control
CN109564471A (en) Distributed interactive medical visualization system with major/minor interaction feature
Dubois et al. Classification Space for Augmented Surgery, an Augmented Reality Case Study.
CN106569673A (en) Multi-media case report displaying method and displaying device for multi-media case report
US20210353361A1 (en) Surgical planning, surgical navigation and imaging system
CN115315729A (en) Method and system for facilitating remote presentation or interaction
EP3790023A1 (en) Method for analyzing medical image data in a virtual multi-user collaboration, a computer program, a user interface and a system
JP2009521985A (en) System and method for collaborative and interactive visualization over a network of 3D datasets ("DextroNet")
Yu et al. VeLight: A 3D virtual reality tool for CT-based anatomy teaching and training
Le Van et al. Design and development of collaborative ar system for anatomy training
Hutchins et al. Communication in a networked haptic virtual environment for temporal bone surgery training
Rasmussen et al. The virtual temporal bone, a tele-immersive educational environment
Mutter et al. Digital technologies and quality improvement in cancer surgery
CN117826976A (en) XR-based multi-person collaboration method and system
TW202038255A (en) 360 vr volumetric media editor
Kunii et al. System to check organs, malignant tumors, blood vessel groups, and scalpel paths in DICOM with a 3D stereo immersive sensory HMD
Silverstein et al. Precisely exploring medical models and volumes in collaborative virtual reality
Tripathi et al. Augmented Reality and Its Significance in Healthcare Systems
Pinter et al. SlicerVR for image-guided therapy planning in immersive virtual reality
Lan et al. A remote virtual-surgery training and teaching system

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C02 Deemed withdrawal of patent application after publication (patent law 2001)
WD01 Invention patent application deemed withdrawn after publication

Open date: 20090408