JP2009521985A - System and method for collaborative and interactive visualization over a network of 3D datasets ("DextroNet") - Google Patents

Abstract

The present invention relates to interactive visualization of 3D datasets. In particular, it relates to the cooperative interactive visualization of one or more 3D datasets by multiple parties over a network. The suggested system receives position data of one or more remote probes of one or more remote machines over a communication link and displays a combined 3D scene including one remote probe and the 3D image on a local display. Created for display on, processed 3D images by a user local to the device, and associated with the processing by the user local to the device over the communication link Data is configured to transmit data that causes a remote machine to display a combined 3D scene including an image of a local probe that performs processing on the 3D image.
[Selection] Figure 2

Description

Cross-reference with other applications This specification includes: (i) SYSTEMS AND METHODS FOR COLLABORATIVE for collaborative and interactive visualization over a network (“DextroNet”). INTERACTIVE VISUALIZATION OVER A NETWORK ("DextroNet")), US Provisional Patent Application No. 60 / 755,658, filed December 31, 2005, (ii) "2D Interface (" Dextro In the name of the invention, “METHODS AND SYSTEMS FOR INTERACTING WITH A 3D VISUALIZATION SYSTEM USING A 2D INTERFACE (“ DextroLap ”)” US Provisional Patent Application No. 60 / 845,654, filed on September 19, 2006, and (iii) “3D Data Set Network” 2006, named “SYSTEMS AND METHODS FOR COLLABORATIVE INTERACTIVE VISUALIZATION OF 3D DATA SETS OVER A NETWORK” (“DEXTRONET”) on the network (“DextroNet”) Claims priority of Application No. issued on 19 December (Applicant reserves the right to amend this application to provide the correct application number and filing date for this latter provisional patent application) The entire contents of this provisional application are incorporated herein by reference. Further, the present specification further describes (i) “a navigation system for augmented reality surgery based on a camera mounted on a pointing device (An April 27, 2004, with the name of the invention “Augmented Reality Surgical Navigation system Based on a Camera Mounted on a Pointing Device” (“Camera Probe”) Filed co-pending US utility model application No. 10 / 832,902, (ii) “An Augmented Reality Surgical Navigation system based on a camera mounted on a pointing device Based on a Camera Mounted on a Pointing Device (“Camera Probe”), a co-pending US utility model application No. 11 / filed on July 1, 2005. No. 172,729 is incorporated by reference.

  The present invention relates to interactive visualization of three-dimensional (“3D”) data sets. In particular, it relates to cooperative interactive visualization of one or more 3D datasets using various platforms over a network.

BACKGROUND OF THE INVENTION Conventionally, 3D visualization of a 3D dataset is performed by submitting a given 3D dataset to a dedicated workstation or computer (or creating one of multiple 2D images). Yes. In general, a single user interactively visualizes a 3D data set on a dedicated workstation. This can be done, for example, with a Dextroscope (registered trademark) manufactured by Volume Interactions Pte Ltd (Singapore). The Dextroscope (registered trademark) is a high-end, true interactive visualization system that can display volumes in three dimensions and allows the user to fully control 3D. In addition, the DEX-Ray (registered trademark) system from Volume Interactions Pte Ltd (Singapore) is a dedicated 3D interactive visualization system that combines co-registered 3D scan data and real-time video. It is. DEX-Ray (registered trademark) is a combination of virtual objects segmented from pre-operative scan data and real-time video into a composite image that allows users (typically surgeons) to actually Enables you to “look behind” at the surgical site. However, traditionally 3D interactive visualization systems such as high-end Dextroscope® or DEX-Ray® have multiple display systems, many Even if one person can see their display by standing close to the actual user, only one user at a time can control (controllable) the visualization. Therefore, in such a system, no one other than the controlling person is truly participating or cooperating in the operation of the 3D data set under test.

  For example, DEX-Ray® can be used for planning complex surgical procedures such as neurosurgery procedures. In such a surgical plan, as described in the Camera Probe cited above, the neurosurgeon and his team obtain pre-operative scan data and a segmented object of interest for this data. Planning data such as approaches to be used during surgery can be added. In addition, various points in a given 3D data set can be used as “markers”, as further described in Camera Probes. The position of the tip of the user's hand with the probe from such a marker is tracked throughout the surgery and read continuously (via visualized or audible cues).

  Furthermore, it is often desirable to perform 3D input from the surgical site while performing surgery. In such a scenario, for example, by attaching a tracking ball, one or more surgical instruments can be tracked and the interaction between the surgeon and the patient can be better visualized closer to the truth. Thus, as described in the Camera Probe, once the surgery begins, a combined image of real-time data and virtual objects can be created and visualized. However, the surgeon is usually unable to dynamically adapt the displayed virtual object during surgery (including changing the point designated as a marker). This is because during surgery, there is not much time to focus on optimizing the visualization and therefore the full power of the 3D visualization system cannot be fully utilized.

  By using a DEX-Ray® type system, a small amount of virtual objects of interest, such as important nerves near the tumor or the tumor itself, are specified prior to surgery These objects can be displayed during surgery. The same is true for marker points. As mentioned above, the camera probe (Camera Probe) can specify a certain number of marker points and how the dynamic distance from the probe tip to the object is tracked during the procedure. It describes what you can do. Theoretically, the surgeon can adapt the marker points while performing the procedure, but this is also not generally done. This is because the actual procedure is limited to surgery and there is not much time to optimize the augmented reality parameters while it is in progress.

  There are other reasons why surgeons generally cannot interact with virtual data during treatment. First, most navigation system interfaces make such raw interactions very complex. Secondly, the navigation system interface is not sterile and therefore the surgeon will have to make adjustments by instructing nurses and technicians. In DEX-Ray (R), it is possible to modify the visualization just by moving the probe in the air (as described in Camera Probe) and thus the surgeon While it is possible to directly modify the display parameters while maintaining sterility, it is more convenient if the visualization environment need not be modified during surgery. In any case, it would be best to be able to assist the surgeon by visualizing the surgeon.

  Similarly, using a standard 3D interactive visualization system such as, for example, a Dextroscope® for surgical assistance, guidance, or planning, all parties in one location It is often difficult to cooperate. For example, a surgeon may perform a complex operation that is difficult to fully plan, such as a separate operation on a Siamese twin that shares brain structure. In such procedures, during the actual (long) operation, a team member who relies continuously on the pre-operative scan and relies on what is actually projected with the brain exposed, or other Most of the time you need to consult a specialist. While interactive 3D visualization of pre-operative scan data is the best way to analyze it, the surgical team will be able to around the display of such a system even when all of the participants are in the same physical location. It is difficult to get together. In addition, the more complex the case, the more often the geographical location where team members may be located is more difficult to consult preoperatively using the benefits of data visualization.

  Finally, when two distant parties discuss using an interactive 3D dataset visualization system or technology using it, for example, if one party instructs the other for its use: It is necessary that both can see the 3D dataset and the other operation at the same time.

  What is needed in the art is to have a remote participant operate as if it were actually present and operate on a 3D dataset in a 3D interactive visualization system simultaneously.

SUMMARY OF THE INVENTION An example of a system and method is provided that allows a number of physically remote people to collaborate and interactively visualize a 3D data set at approximately the same time. In an embodiment of the present invention, for example, there may be a main workstation and one or more remote workstations connected via a data network. A given main workstation can be, for example, an augmented reality surgical navigation system, or a 3D visualization system, and each workstation can have the same 3D data set deployed. In addition, a given workstation may have already acquired 3D data such as, for example, real-time video or pre-recorded video, or information such that a managed 3D ultrasound visualization system is provided. Real-time images can be combined. A user at a remote workstation can perform a given diagnostic or therapeutic procedure such as, for example, surgical navigation or fluoroscopy, or with a shared and stored 3D data set. You can receive instructions from another user at the main workstation where the lecture is taking place. A single user at the main workstation can see, for example, the virtual tools used by each remote user, and each remote user can see, for example, the main user's virtual tool and the remote workstation's dataset. You can see the effect on each. For example, if the remote workstation is the local workstation's virtual tool on the remote workstation's 3D data set via the local machine's virtual control panel, is the probe associated with that of the remote workstation? Can be displayed in the same way. In an embodiment of the present invention, each user's virtual tools can be represented by their IP address, different colors, and / or other distinguishing displays. In an embodiment of the present invention, the data network can be either low or high bandwidth. In the low-bandwidth embodiment, the 3D data set can be pre-loaded on each user's workstation and only the main user's virtual tool and data operations can be transmitted over the network. In high-band embodiments, for example, real-time images such as video, ultrasound or fluoroscopic images can be transmitted over the network as well.

Detailed Description of the Invention Overview In embodiments of the present invention, various types of 3D interactive visualization systems can be connected on a data network, so that people who are remote from each other can use the same 3D data set for various purposes. Can be operated almost simultaneously. In an embodiment of the present invention, two or more people can own a given 3D data set, each located on a workstation, while at a distance from each other. Typically, such implementations consider a “main user” and one or more “remote users”. The party can be, for example, a collaborator on a team of doctors performing a surgical planning project, for example, a separate operation on a Siamese twin. Or they can be teachers or teachers, or groups of students or listeners. Further, for example, the parties may include (i) a navigation system (eg, a DEX-Ray® system), or other diagnostic or treatment system (eg, a Cathlab machine, A surgeon or clinician performing surgery on a patient or performing a diagnostic or therapeutic procedure using a controlled ultrasound system, etc.) and (ii) physical such as a treatment or operating room like a surgeon or clinician Without being restricted, it can include visualization specialists that dynamically modify and visualize virtual objects in related 3D datasets from several remote locations. Alternatively, such parties may include teachers and one or more students in various educational, professional, review, or customer support contexts. In the following, the present invention describes the use of various possible paradigms, or examples of use, but the invention is not limited to such examples of use. It can be seen that the features of the present invention are applicable to a wide range of applications where various types of research and collaboration in sophisticated visualization of 3D datasets are desirable or useful.

  For ease of explanation, various embodiments of the present invention sometimes refer to “DextroNet”. This is a considered registered trademark used by the assignee in connection with such various embodiments.

A. Dextroscope®-Dextroscope® interaction In embodiments of the invention, for example, between remote customer training or technical support, or 3D interactive visualization system manufacturer and its customers DextroNet can be used for remote consultation. For example, DextroNet can be used by manufacturers of 3D interactive visualization systems to provide remote on-demand technical services to their customers. In such use, for example, customers who have questions regarding the use of a given 3D interactive visualization system provided by such a manufacturer can always connect to a training center. Once connected, how can a remote training technician use a contour editor on a given tool, eg, Dextroscope®, for example to the customer Can be shown. An example of such training is remote training, without requiring the customer to send all of their datasets (avoid spills on potential patient confidentiality and time wasted sending such data). This can be done using the standard data set that both parties have at the start of the session.

  Another service that may be offered to customers that is using DextroNet is, for example, preparing (segmenting) patient data for customers offline. For example, in the medical field, customers use data related to one or more relevant patients or cases in ftp, and the “Radiology Services” department can segment and return it to the customer. For such services, in some embodiments of the present invention, online demonstrations of what was done, or even remote training on actual patient data handled by the customer could be provided.

  In embodiments of the present invention, remote consultation can also be provided. In such embodiments, for example, a team of specialists, such as surgeons and radiologists, diagnostic imaging technicians, advisory physicians, etc., should be employed in the case, the various viable surgical approaches to be employed in the case, and All similar issues can be discussed remotely over DextroNet. For example, a radiologist can provide his views, views as a neurovascular surgeon, a craniofacial specialist can provide another view, taking all into account, and the case Virtually “pointing” the various volume objects in a given 3D dataset created by scanning the data.

B. Dextroscope (R)-DEX-Ray (R) type interaction As described in Camera Probe, DEX-Ray (R) The system can be used as a surgical navigation device. Such a device can be connected to a standard interactive 3D display system over DextroNet. Thus, in an embodiment of the present invention, the ability to manipulate 3D data means that it can be “outsourced” and that the limitations imposed by the surgeon's sterile field can be eliminated. For example, when a surgeon has a DEX-Ray® probe in a patient's skull, he generally does not give instructions to someone near the base computer It is impossible to control the navigation system interface. Of course, this is not desirable. This is because such computers are usually occupied with fully rendering images / sound.

  Using a probe in the patient's skull, the surgeon tries to find his direction in the patient's brain (ie, pointing to a different area and cannot point beyond tissue that has not yet been cut) So surgeons need help examining other image modalities, other segmented structures (such as tumors, blood vessels, nerves (eg, via diffusion tensor imaging, etc.) without significantly changing the position of the probe In such a scenario, the surgeon can point with the probe and can speak and hear, but often cannot get in and out of the skull. Then it would be useful to have the assistance of another surgeon (or expert assistant) to provide different visualizations. Visualization indicates, for example, which blood vessels are closest and which nerves are around the tumor, such as “visualization assistants”, which are the surgeon's navigation system and dextronet ( Can be connected via DextroNet).

C. Dextroscope®-Cathlab®-type interaction Alternatively, in an embodiment of the present invention, DEX-Ray®-Dextroscope A paradigm similar to (registered trademark) interaction can use online information, for example, in the form of X-ray fluoroscopy display. Here, remote assistance is a registration between, for example, a real-time X-ray image (2D perspective) created from a previous scan of the same individual and a 3D image (eg, from a CT or MR scan of the heart) Helps you adjust precisely. Since the blood vessels are not visible with X-rays, interventional cardiologists have to inject the contrast agent and observe the blood vessels opaque for a few seconds. Infusion of contrast media is not directed to patients with kidney problems, and in any case for the patient, contrast media is minimal (both health and medical costs) Should be stopped. Thus, the visualization assistant can utilize a few seconds of contrast agent flow to synchronize the patient's x-ray image with, for example, pre-operative CT. This information can be presented on the X-ray image as an extended virtual object. For example, it can be displayed on another monitor next to the x-ray image. For example, the heart can beat on the X-ray image, and the CT can beat accordingly. Alternatively, it may simply indicate the moment of freezing when guiding the cardiologist to follow the correct vessel with the catheter.

  In an embodiment of the present invention, only one person controls the interaction affecting 3D data (“main user”) and other parties connected on the DextroNet (“Remote User”). Can see these effects, but it is assumed that the data itself, or more precisely, their views on the data cannot be changed. Further, in such an embodiment, DextroNet transmits only the details of the 3D interaction of the main user over the network, for example. Then, after that, the dialog is performed in each remote workstation or system. This is in contrast to conventional systems where a single user sent data about his interactions to a remote server. Then, the remote server performs computer processing, and returns the final computer-processed image without instructing the remote viewer to create a final image of 3D data, without instructing the order of steps or operations.

  For example, conventional solutions such as SGI's Viz Server® provide only a one-way network. In an embodiment of the present invention, DextroNet is a multidirectional network. Approaches such as SGI's Viz Server (R) rely on the assumption that only one user requests the final projection from a given 3D scene. If exactly one projection is needed (or a pair of projections for stereo, for example), the remote computer computes (renders) the image and sends the resulting pixels to the client. The client then sees an image that is indistinguishable from that obtained by rendering locally. The user moves the cursor, for example, performs a rotation operation, transmits a command to the remote server, and can rotate the 3D model. The model can be rendered again and the image can be sent for viewing by the client. However, such conventional approaches do not allow other users to intervene their own 3D interaction devices (such as styluses, probes, etc.) in the 3D model. This is because there is no depth information about the stylus model. It also does not allow the client to create a separable viewpoint (shown below). This is because only the pixels from the perspective of one user are available.

  Moreover, such an approach cannot accurately indicate to the remote user what steps are being taken to obtain the final rendering. If he is interested in actually learning such an interactive command sequence, he cannot do it. In contrast, in an embodiment of the invention, a 3D model is used for each workstation (e.g., teacher and student, or, e.g., visualization assistant and surgeon, or other, e.g., visualization assistant and actual patient). Available at a station (such as a diagnostician or treatment specialist), and then the 3D of the model can be combined with the 3D of the user who has each station (ie a tool or other object), Can be rendered.

  In order to influence data remotely, in embodiments of the present invention, the 3D interaction of the main user's stylus can be arranged to interact with, for example, the remote user's (remote) virtual control panel. . be able to. In such an embodiment, such control panel interactions are translated into system commands on the remote machine, for example, “press the disconnect button” or “slide the zoom slider to 2.0”. There is no need to Rather, in such an embodiment, the operation of a given main user is performed on the remote user's world, eg, on the remote user machine, as a result of the operated main user's tool on the remote user machine. It may appear by displaying the main user's tools. On the remote user side, the main user's rules appear at a position above the button on the remote user's virtual control panel, and the state is, for example, “Startup Action” (one of the four exemplary actions of the tool). And then the button is pressed. No special directive is required. In this way, the remote tool can act in the same manner as the local tool on the main user or remote user side. Therefore, this is more seamless compared to using extra system commands suggested on the main user's machine side (or using the traditional Viz Server® approach) to affect operation. It is possible to reach a perfect integration. Another reason not to use system level commands is to avoid discontinuous screens on the remote user side. For example, if the stylus interaction with the virtual control panel on the remote user's side is not continuous, the remote user will not be able to function or operate “ While "B" is happening, it is confusing as to whether there is a tool on button "A". If the display of the tool position can be synchronized as described above, no extra system commands are needed and the tool itself can do it.

  It can be seen that the scale of the virtual control panel can vary between users on different platforms. This is because there may be various 3D visualization system hardware and software configurations connected via the exemplified DextroNet. For example, describing the use of various systems provided by this assignee, Volume Interactions Pte Ltd, DextroBeam® does not include a mirror, Has no "reflected" scale. On the other hand, the DextroLAP® system uses a mouse and does not use a 3D tracker. Thus, in an embodiment of the present invention, the 3D scale of the main user can be sent to all connected remote users. The remote user is typically connected through various 3D interactive visualization system platforms of different structures and models. Such a scale transmission ensures, for example, 3D interaction performed by the main user by pressing the correct button at the correct time on the remote user side. Therefore, when connected to DextroNet, the protocol, for example, makes the remote user's control panel the same as the main user's (in terms of position, orientation, size, etc.). As a result, the operation of the main user on the control panel is replicated on the machines of various remote users.

  When the main workstation synchronizes the interface and viewpoint to the remote workstation, it sends, for example, control panel parameters to the remote workstation, as described below in connection with FIG. 55 (a). Thus, when the remote workstation receives these parameters, for example, it uses this information to update its control panel and locally displayed objects. As soon as this process is complete, the main user's tools are used, for example, to operate a remote control panel. The initial viewpoints of the main user and the remote user can be made the same.

  Various exemplary paradigms are described herein. For language savings, the features have been described in detail in connection with the “Teacher-Student” paradigm, which is also applicable to the “Surgeon-Visualization Assistant” paradigm. The converse is also true and will not be repeated. The “surgeon-visualization assistant” paradigm functions similarly to the “teacher-student” paradigm. In the latter, recorded or real-time video, real-time imaging (eg, fluoroscopy, ultrasound, etc.) or real-time device location information is added, each co-registered in a 3D data set.

D. DextroNet Education (Teacher-Student) Paradigm In an embodiment of the present invention, a DextroNet connection can be used for education. For example, DextroNet should be used to instruct students about surgical procedures and planning or, for example, the use of 3D interactive visualization systems, such as Dextroscope® Can do. In addition, the exemplary DextroNet can be used for anatomy instruction. For example, such performance can be used to familiarize students with specific complex anatomical areas of the human body. In addition, the exemplary DextroNet can be used, for example, for pathology instruction, and therefore can be used as a supplementary instructional tool for autopsy tests (which is used by patients It may be particularly useful when you die during surgery or when preoperative plans and scan data for such patients are available).

  In embodiments of the invention, familiar with Dextroscope®, DextroBeam®, or DEX-Ray® system, or equivalents thereof The instructor (e.g., a doctor, surgeon, or other specialist) may, for example, provide instruction via a DextroNet to at least one remotely located student. A student can have a 3D model that is an opponent with the same perspective as the instructor. Or, for example, it can have different viewpoints. For example, a student may have one display (or display window) with an instructor's viewpoint, or another display (or display window) with a different viewpoint. Alternatively, for example, a student or instructor could switch between different viewpoints. This is, for example, like having an in-image image where one image is a remote viewpoint and the other is a local image, and is similar to the method of video conferencing. In such an embodiment, both images are created on a local workstation (one with local parameters and the other with remote parameters). Or, for example, a remote image is rendered remotely and sent in a manner similar to the Viz Server® approach (send only the last pixel, not the interaction). Thus, a given user can see exactly what the remote user is seeing. There is no doubt, “Do you see what I see?”

  To simplify the description, the parties in such an educational aspect are referred to as “teachers” or “instructors” (main users) and “students” (remote users). Next, an example of the teacher-student paradigm will be described with reference to FIGS.

  In an embodiment of the present invention, a teacher cannot, for example, see a student's real-time view image, but is viewed by the student according to the direction of the student's tool pointer displayed in the teacher's world ( Which can suggest the student's area of interest). When it is not possible to accurately “know” the remote station's viewpoint from the appearance of the remote stylus, a well-known or assumed fact, for example, the stylus is used by the right person. You can guess from the fact that you are pointing forward using it.

  Such an example scenario is designed so that a teacher can teach multiple students. Thus, the teacher will not be disturbed unless the student asks questions with voice or other signals. Then, based on the student's desire for such a question, the exemplary teacher can, for example, rotate a given object and easily align his viewpoint with the student's viewpoint. He can practically reveal how the student sees the object from the direction of the student's tool. Alternatively, teachers and students can reverse their roles. In the embodiment of the present invention, this can be easily realized by clicking a few buttons as described below ("Role Switch"), for example. As described, in an embodiment of the present invention, in the teacher-student paradigm, a student can, for example, except for some operations that do not change a voxel or multiple 3D images belonging to a particular object: You cannot manipulate objects locally. Such operations include, for example, translation of objects, rotation and / or pointing, or object zooming (magnification change). This usually prevents the instruction or instructions from colliding with the 3D data set.

  In general, the local controls that can be performed by a remote user (eg, a student, surgeon, or visualization assistant who does not act as a teacher) can often be limited. This is because, as noted above, the manipulation of the teacher's (or visualization assistant's) data set is typically performed locally on each student's (or surgeon's) machine. That is, the main user interaction is transmitted over the network and executed locally. The main user thus has control of the remote user's (local) machine. In such an implementation, confusion can arise if the student could change the local copy of his / her data as if the teacher's processing no longer makes sense. For example, on a standard Dextroscope®, the user has full control of the segmentation of objects in the 3D dataset. For example, the segmentation can be changed by changing the object automatic color matching table. Alternatively, only the segmentation can be left as is, or the color and transparency associated with an already segmented object can be simply changed. The latter operation does not change the original content of the object, but can change its visualization parameters.

  In addition, there are many tools and operations within the Dextroscope® type system and interactive 3D dataset visualization systems that generally operate based on some assumptions about the size of the object being manipulated. Exists. For example, there are pointing tools that operate by directing light rays from the tip of the tool to the closest surface of the object. If a local user changes the object's segmentation, if the closest surface of the object approaches or moves away from such a pointing tool, the teacher lifts his pointing tool and the object On the student's screen, such a pointing tool may touch an entirely different object. In fact, a student machine can go right through the one on the teacher machine and go to the surface of the object. In general, drilling tools, picking tools, contour extraction tools, isosurface creation tools, and various other tools are similarly dependent on the size of the object they are manipulating. Thus, as described, in an embodiment of the present invention, the student's local control panel is under the control of the teacher. In such an embodiment, the student can only move away from the teacher's point of view and adjust the rotation, translation, magnification (zoom), or color or transparency (not size) he wants. . Therefore, the relationship between the teacher and each object is preserved. There will always be a question of how many controls will be on the student. In embodiments of the present invention, the problem can generally be solved by allowing any local behavior that does not change the actual data set.

  Nevertheless, in an embodiment of the present invention, the local user has no “true” effect on his local data, but rather the effect is displayed later, while the main user's operation is “ By allowing certain operations to operate on local copies of data that appear to be "ghosted", complexity can be increased.

  For example, a teacher may describe various measurement methods between objects between 3D datasets using various tools for one or more students. The teacher can take measurements and they will be seen by students. However, a teacher may wish to test the ability of a student to make a similar measurement, and may ask the student to apply what he has just taught and take additional measurements that the teacher has not yet made. is there.

  The student can, for example, make these measurements and display, for example, on his machine. The teacher then takes a snapshot of the student's machine, for example, and evaluates the student's skill. Similarly, a student manipulates a “true” dataset and a dataset that can be displayed on his display, not as “ghosted”, but that “true” manipulation by a teacher can be shown. be able to. In this way, all of the teacher's operations act on the ghosted boundaries of the object so that they are displayed on the student's machine. The student's local manipulations then operate in the required range on the machine, ie, to the extent that it will affect the 3D data set, in an unbroken state. Thus, at the expense of great complexity and further local processing, the student is allowed to perform operations other than, for example, the translation, rotation, pointing, and / or magnification described above.

  Note that the above embodiment requires that when the student enters the “enhanced” separation mode, he makes a copy of the entire data set (including 3D objects as well as the control panel). Under this condition, the student operates on, for example, a local (copy) data set, while the teacher operates on the original data set that is shown ghosted to the remote user or student machine. be able to. Thus, two independent image processing processes can work in parallel on the same (student) workstation, for example. If the teacher wants to see a new approach developed by the student, the teacher can take a snapshot of the student's image and discuss it both.

  In an embodiment of the present invention, the student takes over control from the teacher when role switching is used to pass the teacher's role from party to party in this manner. He continues to work on the teacher's object and provides his opinion, for example, by demonstrating how to perform an operation or other procedure. This allows cooperation by all parties involved. In such an embodiment, there is one teacher on the network at a given time, and such cooperation occurs continuously without creating conflict problems.

  In an embodiment of the present invention, DextroNet is supported by various networks with different bandwidths. In one embodiment, when a DextroNet is used for educational instruction, for example, a slow network is performed. Each of the 3D interactive visualization systems connected via an exemplary DextroNet can have, for example, a surgical plan, a surgical procedure, or a copy of anatomical imaging and video data. Thus, typically small size tracking data, control signals or graphical or image parameter data can be transmitted over the network. In general, in DextroNet, network information can be classified into two types, for example, messages and files. The message can include, for example, tracking data, control signals, and the like. The file can include, for example, graphical and image data that can be compressed prior to transmission. This will be described in more detail below.

  As described, in an embodiment of the present invention, an action introduced by one user can be sent to the remote location of at least one other user (student or teacher). There, the second workstation calculates an image that corresponds locally. Voice communication between parties connected by a network can be out of band. For example, it is done by telephone in the low-speed network aspect or via a voice-over-network system, for example, Voice-over Internet Protocol (“VoIP”), for example in the high-speed network aspect. Furthermore, the high-speed network can accommodate the transfer of mono- or stereo video information created by the camera probe as described in the Camera Probe application. This video information can be combined with, for example, an image created by a computer. Or it may be viewed on a separate display or display window. The display of non-3D video (or images, snapshots) is usually only seen as “background”. This is because an image or video (continuous image) is a real world projection and does not contain 3D information. Thus, an object visible in such an image or video can be positioned exactly in front of the 3D object. The possibility of such a display will be described in more detail below.

  In embodiments of the present invention, DextroNet can support, for example, working cooperatively and independently. The exemplary system can be used, for example, as a stand-alone platform and independent training for surgical planning. In such an implementation, for example, the system configuration can be optimized locally. The position and orientation of the student system interface can therefore be different from that of the instructor system. However, if the networking function is activated, the student system can be modified to match the system of the instructor system, for example. This avoids mismatches that may exist between equivalent systems.

E. DextroNet Surgeon-Visualization Assistant Paradigm In another embodiment of the present invention, an instructor can, for example, teach at least one student located remotely. And the “Visualization Assistant” manipulates images and advises the steps necessary to see what such an assistant sees, or periodically so that he can see all of his viewpoints Depending on the role of the transferring teacher, the instructor can be assisted. For example, the visualization assistant highlights the object of interest (eg, color look table or object transparency); zooms in the object of interest; changes direction or viewpoint; plays a video image of the recorded procedure; Stop, rewind, fast forward, or pause; provide a different view on a split screen; combine video and computer-generated images for display; toggle between displaying mono and stereo images Perform (whether they are video images or computer-generated images); and process the images to perform other related manipulation and presentation tasks. By involving a visualization assistant in addition to the instructor and student, the instructor focuses, for example, on teaching the student, and more precisely on the processing of the image. In addition, when an assistant gives system control and thereby becomes a teacher, students who may not be familiar with 3D interactive imaging systems can help in obtaining the perspective and information they may need. I can get it.

  In embodiments of the present invention, the surgeon or physician and the “visualization assistant” can collaborate on a DextroNet during actual surgery or during other medical procedures. In such embodiments, the visualization assistant typically acts as the main user and the physician as the remote user. A surgeon or physician can perform, for example, surgery or perform several medical, diagnostic, or therapeutic procedures on a patient in an operating room or treatment room. The visualization assistant is then at a remote location, for example, a surgical department, a surgical gallery, an adjacent room, or all different locations. This is described in detail below as a surgeon-assistant paradigm using FIGS.

  The work of the visualization assistant can include, for example, touching the surgeon performing the operation, for example, by voice and dynamically manipulating the 3D display to provide the surgeon's maximum assistance. Other remote users can observe, but for example, as a remote user, they may not be able to manipulate the data freely.

  For example, suppose a surgeon uses DEX-Ray® in connection with a neurosurgical procedure. In such an exemplary implementation, his main display is limited to those portions of the data set that can be viewed from the camera perspective, as described in the Camera Probe application. However, the visualization assistant can see the virtual data from the viewpoint freely. In an embodiment of the invention, the visualization assistant can have two displays and can see both the surgeon's perspective and his perspective. For example, he can see the surgery from the opposite perspective (i.e. someone's fixed to the tumor or other intracranial structure and the surgeon "enters" his perspective from "external"). You can see the place.

  For example, suppose a surgeon is operating near the optic nerve. As described in the Camera Probe application, the user sets one or more points as markers in the data set, and the dynamic distance from the probe tip to those markers is continuously read out. I see you. In addition, the user can do the same with one or more tracked surgical instruments (which will not acquire video images) so that the visualization specialist can, for example, You can input when you are doing. A visualization assistant can do when the surgeon is unable to concentrate on the entire display for positions near the optic nerve.

  Thus, for example, the visualization assistant can digitally zoom the area where the surgeon is performing the operation, set five marker points along the optic nerve near the surgeon, and monitor the distance to each Advise him if there is a surgeon at the safety limit. Further, in some cases, new markers and dynamic distances set by the visualization assistant can be visualized and audible on the surgeon's system display. This is because the visualization assistant who is the main user controls the 3D data set.

  As described in Camera Probe, the user can specify various virtual structures including combined (augmented reality) images. Ideally, this is done dynamically as the surgical procedure progresses, and at each stage the segmentation associated with that stage can be visualized. In practice, however, it is usually not possible for a surgeon working alone. This is because the surgeon is not a visualization specialist, nor can he freely and mentally exploit the potential of the system. However, the visualization assistant does that. Once the surgeon is in a given area, the visualization assistant can add or remove structures to best guide the surgeon. For example, he is related to a part of the brain structure, such as a structure that was not segmented prior to surgery, such as a finger-like piece or a fibrous piece of a tumor that did not appear in a pre-operative scan , Objects can be segmented on the fly. Thus, in an embodiment of the present invention, the remote viewer can have more freedom and therefore can be done before planning, eg, between colorization, transparency, segmentation thresholds, etc., to name a few. Any image processing that was not possible can be performed.

  Using the similarity, surgeons can use the stadium's information press box to watch the game with a bird's eye from a high gaze and talk to the coach wirelessly while watching patterns in other teams' operational behavior. It can be thought of as similar to a football coach with (Visualization Assistant). A person giving such professional advice may, for example, look at both his screen and the surgeon's screen in an embodiment of the present invention and dynamically control the virtual objects displayed on both, The best assistance can be provided to the surgeon performing the operation.

II. Teacher-Student Interaction A. Exemplary Process Flow Next, a teacher-student paradigm process flow in an embodiment of the present invention will be described.

  FIG. 1 is a diagram showing an example of a teacher-type console in an embodiment of the present invention. This will be described with reference to this figure. At 101, the exemplary system can verify any connected students. Thus, at 102, it can be determined whether a new student has joined. If yes, at 103, the interface and data can be synchronized between the teacher console and the student console for the newly attended student. If no, the process flow returns to 101 and the system checks for any student that may participate.

  Return to 103. For example, when a new student joins a DextroNet by explicitly clicking a button on the console's 3D interface, the teacher will send a message to the student, for example, and each interface and object, eg, , Synchronize sliders, buttons, etc. Such messages include, for example, the position, orientation, and size of the teacher's control panel, the virtual window device ("widget") on the control panel, such as raising and lowering buttons, setting up an automatic color matching table, and in the data set Virtual object position, orientation, size, and any available video can be included. Alternatively, if the teacher finds that the student's data is too different from his own, the teacher chooses to do a full synchronization of the entire data set between himself and each of the student's consoles be able to. This is done, for example, by the teacher's console, compressing the data set (removing snapshots and records) and transferring it to the newly joined student in the DextroNet session. Once the interfaces and data shared by the teacher and the newly attending student are synchronized, preparation for their collaborative interactive visualization is complete, and the process flow is illustrated in FIG. Continue to the main loop, including the rest.

  At 110, the example teacher system can query whether student video has been received. This is done, for example, if a student operates a DEX-Ray® type system. If yes, process flow can proceed to 111 where the student's video is rendered and process flow can proceed to 125. If no, student video is received and the process flow can proceed directly to 125. Further, if the teacher's workstation makes the video available, it is read at 105, rendered at 106, and the process flow can proceed to 120, where the system can make any such video available. You can inquire whether.

  As described, at 120, the availability of the teacher's video is determined. If yes, the video frame can be transmitted at 126 and the process flow can proceed to 125. If no, the process flow can proceed to 125 where the teacher 3D device is read. Here, the teacher is responsible for the position, direction, and state of a virtual tool (eg, stylus, avatar that indicates the current position of the teacher in the dataset, or drill), thereby allowing the operator to interact with the dataset. Updates can be made from a local tracking system that continuously tracks movement and status. For example, in a standard Dextroscope (registered trademark) performing colonoscopy applications, such a 3D control is for example held by the stylus or user (here a teacher) in the right and left hands, respectively. Can be a joystick. Alternatively, for example, when another application is performed with a Dextroscope (registered trademark), the user has, for example, a 6D controller in his left hand. Once the teacher's 3D device is read at 125, the network (Student) 3D device can be read at 130, and the teacher workstation can, for example, indicate the student's representative tool position, orientation, and status. It can be updated via a message received from the station via the network.

  The process flow proceeds from 130 to, for example, 135, where the teacher-side interaction is sent to the student over the network. Here, the teacher can send the student the position, orientation and status of any tool he / she is using, eg, trimming tools, drills, slice viewer tools, etc., and keyboard events. Such positions and orientations can be converted to world coordinates as described in the data format section below.

  The process flow can proceed from 135 to, for example, 140, where it is determined whether the student currently wishes to follow the teacher's view in the local student display. If yes, at 145, the viewpoint is synchronized. That is, when a teacher receives a request for synchronization of a student's viewpoint, the teacher sends the position, orientation, and size of the virtual object in his workstation to the student. If no, at 140, the student therefore chooses to follow his point of view. The process flow then proceeds to 150 where a 3D view of the dataset is rendered. As these interactions are progressing, the process flow loop returns from 150 to 110 and 120 where it can be repeated as long as, for example, a DextroNet session is active. The interaction and manipulation of the 3D dataset is then continuously sent to the network. Furthermore, as shown below, the student is not necessarily attributed to the teacher's, but can toggle between looking at the teacher's viewpoint and looking at his (local) viewpoint. The current state of the student's viewpoint selection can therefore be tested continuously and recognized in each loop up to 140.

  Before describing the process flow at the corresponding student console, various exemplary configurations of DextroNet in an embodiment of the present invention will now be described with reference to FIG.

  FIG. 2 shows an example of a DextroNet network 250 and various consoles connected to it. All of the exemplary connected exemplary devices are products or prototypes of products from Volume Interactions Pte Ltd (Singapore). For example, a DEX-Ray® type workstation 210 can be connected to a DextroNet. As noted above, the DEX-Ray (registered trademark) type workstation is a workstation that performs the techniques described in the Camera Probe application. Such techniques are useful in a variety of contexts including, for example, neurosurgical planning and navigation. In a DEX-Ray (R) type workstation, for example, creating pre-operative 3D scan data of the brain, or virtual data, and real-time video segmented and processed views be able to. The DEX-Ray® workstation can function as a teacher console, where a neurosurgeon uses a DEX-Ray® workstation. Thus, such combined images can be shown to others connected via the network even when performing surgical planning or performing a surgical procedure. Or, more conveniently, the DEX-Ray (R) type workstation allows the visualization assistant to function as a teacher and operate as a student. Continuing with FIG. 2, 220 and 230 show a DEX-Ray® type workstation connected to a DextroNet. A standard DEX-Ray (R) workstation can interactively visualize 3D datasets, but what is a DEX-Ray (R) console? Unlike in general, live video cannot be captured and integrated. However, it can be integrated into a pre-recorded video 3D dataset by a standard Dextroscope® and manipulated as occurs, for example, in “post-mortem” analysis and review of neurosurgery. There, DEX-Ray (registered trademark) is used.

  The DextroBeam workstation 220 and the Dextroscope® workstation 230 are essentially the same equipment and have different features. It is mainly different in display interface. Instead of having a connected display, such as a standard computer workstation, DextroBeam uses a projector to project the display onto a wall or screen, as shown at 220. Conversely, a Dextroscope (registered trademark) usually has two displays. One is the integration of projecting the image into the mirror so that the operator can have a reach-in-interactive feeling as if he was actually operating on the 3D dataset while grabbing the 3D controller under the mirror. Monitor. The other is a standard display monitor that displays the same content that was projected onto the mirror.

  Accordingly, as shown in FIG. 2, various cooperative interactive paradigms using the exemplary DextroNet network in an embodiment of the present invention are available. The DEX-Ray® workstation 210 can cooperate with a Dextroscope® workstation 230 or a DextroBeam workstation 220. Alternatively, for example, it can also cooperate with a DEX-Ray® workstation 210 (not shown). Further, a DextroBeam workstation 220 can interact with another DextroBeam workstation 220 or Dextroscope® workstation 230 via the network 250.

  Further, although not shown in FIG. 2, other 3D interactive visualization systems can be connected to the DextroNet. For example, RadioDexter (R), a software that runs on the Dextroscope (R) and DextroBeam systems, also provided by Volume Interactions There is one version. This is activated on a laptop (or other PC) where 3D operations are mapped to 2D controls, for example with a standard mouse or keyboard. The function of such software is described in detail in the DextroLap application. Such software is referred to herein as “DextroLap”. A DextroLap console is not shown in FIG. 2, but it can also be well connected to a DextroNet.

  In addition, there are two or more cooperating workstations on DextroNet, especially in the Teacher-Student Context. There, one teacher manipulates the data set and all of the many students connected to the network 250 can participate or bystand. In such an embodiment, any other student can see, for example, each student's virtual tools as well as that of the teacher. As will be described with reference to FIGS. 10-30, the teacher can see, for example, each student's IP address and the location and snapshot of the virtual tool or a real-time video of their display.

  In another embodiment of the present invention, the control function, i.e., the operational functionality described herein as a teacher, is handed from one party to the other, and any connected system communicates with the data. Making it possible to spread the interaction to other parties. In such an embodiment, the icon or message identifies, for example, which system is the next operating teacher.

  Given the various collaboration possibilities that DextroNet can provide, the process flow on a student workstation will now be described. It can be seen that the teacher-student interaction can utilize any of the possible connections described in the context of FIG. 2 (whether or not described in FIG. 2).

  Referring to FIG. 3, at 301, the student console looks for a connection with an available teacher. This can be done, for example, by the student console first broadcasting a request to join a DextroNet session over the Internet. If the teacher is available, the student can receive an acknowledgment, for example, and then be connected. If no teacher is active on the Internet (or other data network, eg, VPN), the student will connect to the server, for example, as defined by a given system configuration, I will wait for the teacher.

  Once the teacher is connected (at 301), the process flow can move to 302, where the student's console is the interface associated with the 3D dataset so that the student and the teacher interact cooperatively. And wants to update the data. Thus, at 302, the student can update his control panel, for example, with the teacher's interface and the data set's position, orientation and size parameters from the teacher's data sent over the network. it can. He can also align the state of the widget on the control panel with that on the teacher's system via a message sent over the network. These messages are present, for example, on buttons on the virtual control panel, on module pages (eg, Dextroscope®); such pages enter various possible modules or users It shows the operation mode (for example, registration, segmentation, visualization, etc.), the current value of the slider bar, selection of a combo box, automatic color matching table setting, etc.

  The student receives, for example, at 302, a data synchronization message, such as the location, orientation, size, transparency, and level of details of the virtual object in the 3D data set when his interface is updated. You can also. As noted, in the same dataset, if it is determined that the difference between the student version dataset and the teacher version dataset is too great (or by the student's system during the automation process) He can also request that the teacher send the entire compressed data set under analysis. In such a case, the student's console can then decompress the data and reload the remote scenario. Alternatively, a teacher can send an interaction list instead of the entire data set by performing all of the data set interactions at a point in time on the student machine locally. So that the student workstation continuously receives the teacher version of the dataset. Once the interface and data are updated and the student's console is synchronized with the teacher's console, the process flow proceeds to 310 and 320 described below.

  At 310, it is determined whether the teacher's console has sent any video. If yes, the teacher's video is rendered at 314 and the process flow can proceed to 325. If no, at 310, the process flow can proceed directly to 325. If the opposite of video transmission is addressed, that is, if the student's console has video to send to the teacher's console, it is determined at 320 whether a video frame is available. Before describing the process flow at 320, if there is a student video, the student console reads it at 305 and renders it at 306. It is available at 320 and then transmits it at 326. From 326, the process flow can also move to 325. For example, if student video is not available, the process flow proceeds directly to 320-325.

  At 325, the student's console reads its own 3D device (eg, stylus and controller or their equivalent, ie, the actual physical interface with which the user interacts). This allows the position, orientation, and state of the virtual tool to be calculated, for example, by the tracked actual position of the stylus and 3D controller, as shown. If the student is using a DextroLap system, a mouse or keyboard instead of a 3D device is read at 325. At 330, the student's console can, for example, read the teacher 3D device, and the teacher's representative tools and keyboard event location, orientation, and status from the message received on the DextroNet. Update. From 330, the process flow can proceed to 340 where it is determined whether the student chooses to control his own view or follows the teacher's. If no, the student chooses to control his own viewpoint, and the process flow can then proceed to 345 where the student, for example, the tool on the left hand side of the teacher (in this example, Networking related to where the given user is in the 3D dataset, the left hand tool controls which object is currently selected (similar to a mouse moving a cursor in 2D) Ignore the message. He then reads his own left-hand position, direction, and state directly from his local joystick or 6D controller. If yes at 340, therefore, if the student chooses to follow the teacher's perspective, his machine sends a message to the teacher at 346 asking for the current location and direction of the teacher's virtual object. As soon as he receives the reply, he updates his object.

  Regardless of which selection is made at 340, the process flow can proceed to either 345 or 346-350. And then, a 3D view of the student console is rendered, after which the process flow returns to decisions 310 and 320, so that, as noted above, the DextroNet interactive process flow is continuously passed through the session. repeat.

B. Example Features of Teacher-Student Interaction From the above teacher-student paradigm, the following features are implemented in embodiments of the present invention. The following assumes a user on a system with a virtual tool for two people, one tool with one button switch, buttons and sliders, and a virtual control panel that controls the visualization application. Such an exemplary system is, for example, a Dextroscope® or Dextroscope from Volume Interactions Pte Ltd (Singapore) that runs RadioDexter® software. It can be a DextroBeam® system. .

1. Tracking load reduction In embodiments of the present invention, to handle network data transfer rate variability, to reduce the traffic load imposed on the network, and not to miss key user interactions on 3D objects. To ensure that, for example, the four states of the virtual tool button switch, confirmation, start action, do action, and stop action can be utilized. In the “confirmed” state, the user does not click on the stylus that controls the virtual tool, but simply moves it (this produces position and orientation data, but the “button pressed” data is not Does not occur). Thus, such virtual tools do not perform any active manipulation and appear only as roaming in the virtual world, but can be viewed as pointing at objects, not in a persistent manner, but in practice some objects Trigger to change its state. For example, a drill tool can show a see-through view of a 3D object when a virtual tool is interacting with it, but does not drill the object until a button is pressed. In the “Start Action” state, a button on the stylus can be pressed, which activates the “Do Action” state if it remains depressed. For example, as soon as the user releases the stylus button, the tool can enter a “stop action” state and return to a “confirm” state. Thus, most of the time, the virtual tool is in a “confirmed” state. This state is less important when the tool is actually functioning.

  Thus, in the “confirmed” state, the virtual tool appears only as roaming in the virtual world, and no active operations are performed, such as moving from one location to another. A data packet that transmits the position and direction of such a tool in this state can be considered to be a “nonsense” data packet. If there is a dense network, the teacher's “send” buffer may be full of unsent packets. In such a situation, in an embodiment of the present invention, DextroNet software can check the “send” buffer and discard packets in such “check” state. Most of the time, the virtual tool is in a “confirmed” state, which can greatly reduce the traffic load. This will be described with reference to FIG. For example, assume that a teacher has a set of messages in his “send” buffer, as shown in FIG. If the network connection is slow, he only sends a message to the student, for example as shown in FIG. 53 (b). In this case, the student sees the teacher's tool “jump” from position 1 to position N, then performs a drilling operation, for example, but is lost in the information of the actual drilling operation There is no.

  In an embodiment of the present invention, DextroNet traffic load control can take advantage of this fact. Critical messages can be assigned to preferential forwarding. Normally, when the network speed is fast, all movements of the teacher's tool can be transferred to the student, for example. In this way, the student can see the continuous movement of the teacher's tools. However, when the teacher's system detects that the network speed has slowed down (eg, by noticing that there are too many messages waiting in the send buffer), in an embodiment of the present invention, a “confirm” Status messages can be discarded. In this way, the student can keep up with the teacher's pace, even under dense network conditions. The exchange with such a message “compression” (lossy) scheme is done so that the teacher's tool in the student's local view, for example, appears not to move very smoothly. Nevertheless, important operations on virtual objects will not be lost. This mitigation is not necessary when the network speed is fast. This switch can be, for example, the length of the row in the teacher's “send” buffer. In the absence of “mitigation”, the teacher's tools can move smoothly in the student world. When “mitigation” occurs, the teacher's tools are not continuous. However, the manipulation of the teacher on the virtual object will not be lost, and the student tool will be able to keep up with the pace of the teacher tool. Thus, in embodiments of the present invention, available network speeds can be applied dynamically, and teacher tools can be displayed in multiple resolutions (ie, coarse and smooth resolutions).

2. Viewpoint Control As described, in an embodiment of the present invention, a DextroNet student can either control a given 3D object or follow the teacher's view. If the student chooses a local control of the 3D object, his local processing, such as object rotation, translation and zoom (digital magnification), will affect the position of the 3D object in the world of teachers or other students. Does not reach. However, the teacher can see, for example, the position of the student's virtual tool and the orientation relative to the 3D object, thus allowing the teacher to reference the student's viewpoint, as described above. In such an example scenario, the student can rotate his world objects to find a different region of interest or direction than the one currently selected by the teacher. It can then be communicated with the teacher, for example by voice and / or by pointing to it. Thereafter, when the student's message is received and the location pointed to by the student in the teacher's world is noticed, the teacher can, for example, reach a particular location. On the other hand, if the student chooses to follow the teacher's view, all movements of the teacher's world will be shared with the local world of the student.

  In embodiments of the present invention, there may be specific instructions that can be sent over the network to switch between these two states and reposition the remote virtual tool associated with the user's viewpoint. Such a command can, for example, drive the decisions at 140 and 340 of FIGS.

3. Data Synchronization In an embodiment of the present invention, there are two synchronization modes, single and complete. An exemplary single sync mode can communicate, for example, the object's unique position, orientation, size, transparency, and level of detail. These parameters can be considered “lightweight” in the sense that they are transferred over a data network without requiring large bandwidth. Thus, in an embodiment of the present invention, the teacher module can perform such a single synchronization as soon as it detects, for example, the participation of a new student in the network.

  In an embodiment of the present invention, full synchronization can be performed only when, for example, the user specifically wants it (eg, 145 in FIG. 1). This involves the compression and transfer of all teacher data (except for data captured for reporting purposes, eg snapshots and 3D recordings). In an embodiment of the invention, this synchronization is time consuming and can be arbitrary. In such an embodiment, the teacher can only initiate full synchronization when the student requests it. If there are a large number of students, for example, data is sent only to the student who made such a request. Furthermore, full synchronization can be used as a recovery strategy when a substantial deviation occurs between the two sides. In the absence of throughput limitations due to bandwidth availability, such full synchronization can occur automatically, periodically, or when a specifically defined event occurs.

C. 3D Interface (Control Panel) Synchronization In an embodiment of the present invention, the exemplary synchronization can include two processes: tick mark synchronization and widget synchronization, as described below.

1. Tick synchronization When using a Dextroscope (R) or DextroBeam (R) type system, for example, the virtual control panel is interactive, the 3D tracker is stationary and the virtual tool is virtual If the corresponding part of the panel can be contacted, it must be accurately measured using a physical substrate (eg made of acrylic). Thus, the scale and configuration is local and varies from machine to machine. This can cause mismatch problems during networking. For example, a graticule and / or configuration mismatch may prevent the teacher's tool from contacting a given student's control panel. In an embodiment of the present invention, the teacher and student control panels can be synchronized as follows. While the networking session is active, the position, orientation, and size of the teacher's control panel can be sent to the student, thereby allowing the student's own control panel parameters to be replaced. . When the networking session ends, the original configuration of the student machine can be restored, which allows him to work alone.

2. Widget synchronization When a networking session is started, the initial state of the control panels on both sides, for example, the position of the slider bar, the state of buttons and tabs, the list of lookup tables, etc. may be different. All these parameters need to be aligned for networking.

D. Connections between different interactive platforms In embodiments of the present invention, support for connections between different types of interactive platforms, eg, between Dextroscope® and DextroBeam® be able to. Thus, a teacher on a Dextroscope (R) can teach a number of remote students gathering in front of a DextroBeam (R). Or, for example, a teacher can teach a local student in front of a DextroBeam® and a remote student on the Dextroscope® can see.

  In an embodiment of the invention, another supported connection is between a 3D interactive platform and a 2D desktop workstation, such as those of the Dextroscope® and DextroLap systems described above. is there. With such a system, the stakeholder can use only a desktop workstation if the interaction between the user and the system is via a mouse or keyboard (as described in the Dextro Wrap application), and a stylus and joystick. The 3D input device is not used. In such a scenario, the teacher and student input devices may not be identical. This is most useful because a student who does not have a 3D stylus or joystick can see the teacher operating in all 3D and communicate with the teacher using a mouse or keyboard. On the desktop workstation, keyboard events can be transferred and interpolated in the same way as actions from the tracking system. In such an implementation, the key name and its modifier need to be included in the network message. For example, a DextroLAP cursor has a 3D position. This position can be sent to the teacher so that the teacher can see the movement of the student's cursor in 3D.

  In an embodiment of the present invention, various interactive visualization systems running on Unix and Windows systems (registered trademark) all connect with DextroNet. be able to.

E. Automatic Teacher Detection In an embodiment of the present invention, DextroNet can automatically detect a teacher if there are teachers and students on a given Internet. As described more fully below, when a client gets a reply from a server, it initiates a network connection with the server and tells the server whether a teacher or student exists. For example, the server can maintain a registration list (role, IP address, port) of all clients. When the server knows that the newly joined client is a student, it checks whether the teacher is available and sends the teacher's IP address and port to the student. Therefore, the student can automatically acquire the teacher information. If the teacher does not exist, the server can, for example, alert the student to leave the networking connection. Alternatively, the student can wait until the teacher appears online. Thus, in such an embodiment, the student does not have to worry about low-level networking tasks, such as configuring server IP addresses and ports. As soon as the teacher is present, the student's system can automatically detect it.

III. Surgeon-Visualization Assistant Interaction A. Overview An exemplary cooperative interactive visualization of an exemplary 3D data set between a surgeon and a visualization assistant in one embodiment of the present invention will now be described with reference to FIGS. FIG. 4 is a process flow diagram illustrating an example of a surgeon's console. FIG. 5 is a diagram illustrating an example of a flowchart of the console of the visualization assistant. 6-8 are diagrams illustrating examples of views of the surgeon and visualization assistant in such a scenario.

  In general, such a paradigm is not limited to the “surgeon” itself, but when one party (the “surgeon”) applies diagnostic or therapeutic treatment to a given subject, and for that subject If pre-treatment image data is available and co-registered with a physical object, and if another party can see the object from a 3D dataset created from the pre-treatment image data, Any scenario can be included. Such a “surgeon” as described, for example, is an sonographer (as described, eg, in “SonoDex”), an intervention using a Cathlab® machine. Cardiologists, surgeons using Medtronic surgical navigation systems, and the like can be included.

  In the interaction described, the surgeon is expected to use a DEX-Ray® type workstation and thus capture position data and real-time video. The visualization assistant can also use any type of workstation.

  Referring to FIG. 4, at 401, the surgeon can connect to the network. At 402, the visualization assistant side data may be synchronized to the surgeon side data. As soon as the data synchronization is complete, the process flow can proceed, for example, from 402 along two parallel paths. First, at 410, the surgeon may request a visualization assistant scenario. If requested, at 411 it can be rendered. If a visualization assistant scenario is not required, the process flow can proceed directly from 410 to 420 where the surgeon's console can read the position and orientation of the video probe. Here, the surgeon's console reads the position and orientation of the video probe (which is local) and translates it into virtual world coordinates. The process flow can then proceed from 420 to, for example, 425.

  Note that there is local video processing in parallel with the above processing. It is recalled that the surgeon's console, a DEX-Ray (R) type workstation, can also capture real-time video. Thus, at 405, the surgeon's console can retrieve its own bidet and render it at 406. The process flow can then proceed to 425 as well.

  At 425, the surgeon's console can send the local video it has acquired over the network to the visualization assistant. From there, the process flow can proceed to 430, where the surgeon's console sends the video probe information. Here, the surgeon's console sends the position and orientation of the video probe in the virtual world coordinates to the visualization assistant. This is just a transmission of the information acquired at 420. At 435, the surgeon's console can communicate the assistant's representative tools. Here, the surgeon's console can receive and communicate the position and orientation of the visualization assistant's representative tools in the surgeon's virtual world. This data can be obtained from the visualization assistant console over the network. Finally, at 450, the surgeon's console can render a 3D view of the 3D dataset that may include video and augmented reality, including the position and orientation of the visualization assistant's representative tools. Current 3D rendering will be the result of interaction with 3D data sets by both surgeons and visualization assistants. Another side of the surgeon-visualization assistant paradigm that focuses on the visualization assistant side will now be described with reference to FIG.

  At 501, the visualization assistant can connect to the network, and at 502, he or she can update his data. Here, the visualization assistant needs to update its virtual object position, orientation and size with surgeon data entered on the network. Process flow can proceed from 510 to the determinations 510 and 520 in parallel. First, at 510, it can be determined whether video has been received from the surgeon's console. If yes, at 511, the surgeon's video can be rendered. If no, the process flow can proceed to 530. Second, at decision 520, it can be determined whether an assistant scenario should be sent. This can be done, for example, in response to an inquiry or request from the surgeon's workstation.

  If the surgeon has requested a visualization assistant scenario, at 525, the assistant scenario can be sent and the assistant can send a snapshot or a video of his view. Such a snapshot may be, for example, three-dimensional or monoscopic. If the surgeon has not requested a visualization assistant scenario at 520, the process flow can then proceed to 530. Accordingly, the process flow is similarly reached from 510. The visualization assistant console can then read out its own 3D device. Here, the visualization assistant has full control of his tools, so the position, orientation and size of his tools can be done by his own stylist and manipulator it can. From here, the process flow can proceed to 540 where the surgeon's representative tools can be updated. Here, the visualization assistant console can receive and update the position and orientation of the surgeon's representative tools in the visualization assistant virtual world.

  Finally, the process flow proceeds to 550, where the visualization assistant console renders a 3D view of the 3D dataset. This 3D view will include updates to that of the visualization assistant's own 3D device as well as the surgeon's representative tools and may include any bidet received from the surgeon. As mentioned above, the surgeon-visualization assistant paradigm is assumed to be involved in the DEX-Ray (R) vs. Dextroscope (R) type interaction on the DextroNet Is done. This scenario is a variation of the above teacher-student paradigm, as already mentioned. In the surgeon-visualization assistant scenario, the surgeon usually plays the role of student, while the assistant takes the teacher. This is because surgeons (students) are busy with surgery and are more limited in their ability to interact with data. Therefore, he is mainly involved in seeing how the visualization assistant (teacher) controls the 3D visual virtual world.

B. Independent Viewpoint When using a DEX-Ray® type console in the operating room, for example, the surgeon's perspective is limited by the orientation of the video probe as shown in FIG. As described in more detail in the Camera Probe application, the DEX-Ray® type device is based on the position of the video probe and therefore in the video probe. Render from the viewpoint of the camera. Thus, for example, a surgeon using a DEX-Ray® console can see how the tip of the video probe is fixed from the inside to the target, i.e. Can't see how to approach a viewpoint fixed to a tumor or other intracranial structure. However, in a planning system such as that performed on a Dextroscope® type machine, the assistant has an unrestricted view to examine the target as shown in FIG. Thus, in an embodiment of the present invention, DextroNet assists the surgeon and allows him to move indefinitely via a 3D data set associated with the actual patient on which the surgeon is performing surgery. Can be used to connect a DEX-Ray® with a Dextroscope®. This is very useful for surgeons engaged in real-time complex surgery. Here, many visualizations that are logically impossible for the surgeon involved in the surgery will certainly help the process. It is in such a scenario that the visualization assistant can contribute significantly to surgery or other field treatment (diagnostic) efforts.

C. Interchangeable viewpoint Alternatively, in an embodiment of the present invention, the surgeon can view the assistant's viewpoint. An example is shown in FIG. Thus, in an embodiment of the present invention, there are two types of views available to the surgeon. One is a normal camera probe augmented reality view, one is a video with three 2D trihedral images as shown in FIG. 6 and the other is FIG. 8 is a viewpoint on the visualization assistant side as shown in FIG. 7. These two types of views are displayed in two separate windows that the surgeon can toggle between. Thus, the surgeon can toggle between FIGS. 6 and 7, for example. From the visualization assistant's view (FIG. 7), the surgeon can see the relationship between the virtual object and his tool from different perspectives, eg, a viewpoint positioned on the target object, such as a tumor.

  Furthermore, in an embodiment of the present invention, the visualization assistant can also see the surgeon's scenario in his display. This is illustrated, for example, in FIG. The main window is the visualization assistant's view. This is not limited by the camera position of the video probe tool held by the surgeon. Further, FIG. 8 includes a view “in the drawing”. This is shown in the small window 810 in the upper left corner of FIG. This shows the surgeon the view he can see. This can be transferred as a video frame, so the visualization assistant cannot manipulate it in any way. Accordingly, FIG. 8 shows a main window displaying a virtual 3D world in which the surgeon's tools are also visualized (as a line from top to bottom). The other small window 810 is the surgeon's actual view and is available on the live video signal and the surgeon's scenario, the augmented reality of the 3D object whose display parameters are only selected and manipulated by the surgeon. Is included. Thus, the visualization assistant can see the surgeon's tools in both the 3D world of the main window of FIG. 6 and the 2D video of the image in the image window 810 in the upper left part of FIG.

IV. Interface Interaction Example In an embodiment of the present invention, DextroNet networking functionality can be controlled, for example, from the “Networking” tab in the main Dextroscope® 3D interface.

A. Establishing a connection In an embodiment of the present invention, a user networking module interface may be provided. Such an interface comprises, for example, three buttons: “teacher”, “student” and “stop networking”. Through such an exemplary interface, the user can thus choose to be either a teacher or student, or choose to stop a DextroNet session. In an embodiment of the present invention, a student can establish a connection only when a teacher is present in the network. Otherwise, the student will receive a “teacher does not exist” message and close the connection. If the student does not have a 3D input device, for example, in a DextroLAP application, a mouse can be used for communication as described above.

B. Teacher Action In addition, additional interface features can be displayed to the main user, ie, the user acting as a “teacher”. In such an embodiment, the teacher may be provided with a student IP list and a “sync” button. This panel may only be available to teachers. When a new student joins, for example, a snapshot of the student's first scenario is sent to the teacher, displayed in the student list, and so on. In addition, the student's IP address can be shown, for example, in a text box. If there are multiple students, the teacher can scroll through the list to see other students' IP addresses and snapshots. In an embodiment of the present invention, such a snapshot allows a teacher to accurately measure the relative synchronization between his dataset and the student dataset. For example, there are two ways to achieve synchronization when starting networking: single synchronization and full synchronization. In general, if the teacher and student have the same data set, only a single synchronization is required. However, if the data sets they have are different, the teacher must initiate a full synchronization. Otherwise, a failure will occur later. Snapshots sent from various students display their initial state (environment and data) on the teacher side. Thus, the teacher can check whether the student has the same data that he has through the snapshot. If not, he can alert the student and perform a full synchronization.

C. Student Action In an embodiment of the present invention, as soon as the user chooses to become student, he loses control of his virtual control panel. He can see the teacher's operation and point to the part he is interested in. However, there is a limit to what he can do. This is because he cannot touch his virtual control panel controlled by the teacher. As stated, the limited set of interactions he can perform is either via a dedicated button (similar to that used to perform the PC described in DextroLap) or a teacher Can be easily done via a dedicated “Student” virtual control panel that is different from the “Standard” control panel that operates via the exemplary DextroNet.

  In an embodiment of the present invention, there are two ways for the student to see the operation of the teacher. For example, he can (i) follow a teacher's point of view, or (ii) view a data set from a different perspective than that of the teacher. In such an embodiment, he can toggle between these two modes by simply clicking on a signal defined by his stylus or some other system.

  In an embodiment of the present invention, when the student is in “engaged” (ie, according to the teacher's perspective) mode, for example, a red text display of the word “LockOn” may be provided. it can. In such a mode, he cannot rotate or move the object. If he clicks on the stylus or sends some other signal and disconnects, the “LockOn” text disappears, for example. This shows that he can see the data set from his own perspective. In such a “cut” mode, the student can, for example, rotate or move the displayed object using a left hand tool.

D. Stopping the Network Connection In the illustrated implementation, only the teacher's tool can touch the virtual control panel, so it is the responsibility of the teacher to end the networking function by clicking the “Stop Networking” button. When networking ends, the teacher can save all of the changes he / she made to the data during the networking session. However, the student can, for example, recover his scenario to the one before the networking and the one saved prior to entering the networking session, if desired.

  In an embodiment of the present invention, during networking, student data may be required to be synchronized with teacher data. Thus, to avoid damaging the student's local data, if the student presses the networking button before the networking session actually begins, for example, his own data can be automatically copied to some backup directory . Thus, when the networking session ends, he can recover by copying his data from the backup directory. In an embodiment of the invention, this can be done automatically when the “Stop Networking” button is pressed.

E. Server Example In an embodiment of the present invention, a DextroNet can be established on a server client architecture, as shown in FIG. In such a configuration example, both the teacher 930 and the student 920 are clients. They are physically connected to a server 910 that can be used, for example, to manage multiplying connections, connection registrations, and connection queries. All information from the transmission source (client) is first passed to the server 910, for example. The server, for example, analyzes the receiver IP address and then passes the message to a specific destination. Thus, in such an embodiment, such a server application must first be launched before networking functions are activated on either the teacher or student side. Exemplary server features are described below.

  1. From a low level perspective, in embodiments of the present invention, the server can support multiple connections using multiplexing techniques. It can be, for example, a single process concurrent server where arrival of data triggers execution. Time sharing is taken over, for example, when the load is too great for the CPU to handle it. Furthermore, from a high level perspective, based on the IP address from the source (client), the server can unicast, multicast, or broadcast the message to the destination using the TCP / IP protocol.

2. Connection Registration When a client connects to a server, it registers, for example, its IP address and networking role (eg, teacher or student) on the server. For example, it is the server's responsibility to ensure that there are two criteria: (1) the teacher exists before the student joins, and (2) only one teacher connects to the server. It can be. If criterion (1) is not met, the student may be alerted, for example, to leave the networking, or may be advised to wait for the teacher to connect, for example. If criterion (2) is not met, the second putative teacher may be warned, for example, to leave the networking or, for example, query whether he is desired to be connected as a student It can be done.

3. Connection Query In an embodiment of the present invention, the client can query the server as to how many fellow clients are in the current connection environment and who it is. In this way, the client can know who is involved in the communication. This is important for teachers who maintain a dynamic list of current students. When the server receives a request such as “buddy inquiry”, the server can return, for example, the IP addresses and ports of all the buddy clients to the requester.

4). Reply to Server Query In an embodiment of the present invention, the server can be automatically detected on the LAN. For example, when a server's UDP socket receives a client broadcast query about a server application that it was expecting from the LAN, it can check the name of the running application to see if it is available. it can. If available, the server can reply to the client querying its address (IP: port).

  Thus, in an embodiment of the present invention, if a user selects his / her networking role (eg, pressing a “teacher” or “student” button on the networking interface in the visualization environment when joining a networking connection) He can broadcast a query containing the server program name over the intranet. This message can be sent, for example, to a dedicated port on all intranet machines. If the server program is running, it can keep listening to the dedicated port. When it receives a broadcast message from the client, it can check the names of all running programs on the server machine to know if they are consistent. If there is consistency, the server can reply to a client that is querying its own address (IP: port), for example. On the other hand, the client can wait for an answer from the server. If a negative answer is returned after a certain period of time, the client can report a “server not working” error, eg, resume normal stand-alone work state.

5. Server Startup In an embodiment of the present invention, a DextroNet server can run as a stand-alone application without being installed on a visualization machine. If communication is performed within the LAN, for example, neither the teacher nor the student need to know the IP address of the server accurately. DextroNet software, for example, automatically locates the server. If the communication is over a WAN, the server IP and port must be provided to DextroNet. For example, if a local server is not detected, and a remote server does not exist, the teacher automatically launches a server application on its own machine, for example, when attempting to initiate a networking function.

  Alternatively, for example, if the server function can be combined with the teacher role, that is, if the teacher machine can take the server role, the student machine can maintain the client. However, in such an embodiment, the burden on the teacher machine may be relatively heavy. This is because a request for visualization from 3D visualization software and a request for communication from DextroNet occur simultaneously. Thus, for example, the speed of a DextroNet communication loop can be slowed, for example, by a Dextroscope® visualization loop. This can cause the input and output data remaining in the standby buffer to increase. Thus, in an embodiment of the present invention, a “nonsense” data packet can be dropped, for example, if the data string becomes long. That is, such data packets can be removed from the queue without being processed, for example. A “nonsense” data packet in this context is a data packet that does not affect the result of the program / visualization, eg, does not perform any major operations (eg, drilling, trimming, etc.) and is therefore a data set Packets that transfer movements of tools (both teacher and student) that do not affect.

  In this context, the tool has four basic states: “confirm”, “start action”, “do action”, and “stop action”, as will be shown in greater detail below. In the “confirmed” state, the virtual tool appears only as roaming in the virtual world, and no active operations are performed, such as moving from one location to another. A data packet that transmits the position and direction of such a tool in this state can be considered to be a “nonsense” data packet. If there is a dense network, the teacher's “send” buffer will be filled with unsent packets. In such a situation, in an embodiment of the invention, the software can check the “send” buffer and discard those “confirmed” packets. Most of the time, the virtual tool is in a “confirmed” state, which can greatly reduce the traffic load. For example, a teacher has (a) a set of messages in his “send” buffer. If the network connection is slow, he sends a message to the student (b). In this case, the student sees the teacher's tool “jump” from position 1 to position n and then performs, for example, a drilling operation.

V. Examples of Performance FIGS. 10-49 are diagrams illustrating various exemplary performances of a DextroNet. Such an implementation is designed to integrate with, for example, a Dextroscope (R) that runs RadioDexter (R) software, etc., and the additional functionalities of such a system Can appear as a simple module. In the described example, the teacher and student communicate with each other via a server running a server program. As noted, there can be many students at one time during each session, but only one teacher.

  FIGS. 10-25 illustrate the relationship over time between an exemplary teacher and the views of two students. Each of FIGS. 10, 14, 18 and 22 shows three views side by side. Each of them is shown in an enlarged manner in the immediately following three figures.

  FIGS. 10-25 illustrate exemplary interactions between respective views seen by an exemplary teacher and each of two students, in an embodiment of the present invention. In FIG. 10, the teacher confirms whether the student is connected. A view of the teacher is shown in FIG. FIG. 10B and FIG. 10C show views of Student 1 and Student 2, respectively, which are two exemplary students. In the example shown, Teacher and Student 1 are using a Dextroscope® type system. Student 2 uses a DextroLap type system. Figures 11 to 13 show these views in more detail and are described below.

  In FIG. 11, the teacher can see his own tool 1100, Student 1 remote tool 1101, and Student 2 remote tool 1102. Note that Student 2's remote tool is actually a cursor because Student 2 uses DextroLap. Therefore, it does not have an overall 3D control. Further, as shown in FIG. 11, each student's remote tool has the IP address of the student's computer. In an embodiment of the present invention, such an IP address can be changed to display the student's name. Alternatively, for example, both can be displayed. In an embodiment of the present invention, it is particularly useful that the name of each student displayed on the teacher is shown and used to teach the number of students at the same time. In this way, the teacher can know who he is giving a lecture to.

  Furthermore, it can be seen from FIG. 11 that the teacher's view is a display window 1120 for various student snapshots. This snapshot window 1120 allows the teacher to see the student's local view. Finally, there are three networking tools: “Synchronize” 1150, “Remote Snapshot” 1160 and “Role Switch” 1170. Synchronization 1150 can be used to synchronize between the student and the teacher. This tool can be used when it is clear to the teacher (or to the student) that the state of each data set is floating significantly apart, causing confusion. Remote snapshot 1160 requests a snapshot of a specific student displayed in snapshot window 1120. This can be easily done, for example, by a teacher's control panel with a scrollable list of connected students. The teacher can then select one person, for example, by scrolling through the student list. The teacher then presses a “capture” button, for example, and a snapshot is requested from the student workstation. Finally, the role switching 1170 can switch the roles of the teacher and the student. This process can be limited to teacher-led ones. Thus, as will be shown in more detail below, this button is active only on a teacher machine, for example.

  FIG. 12 is a diagram showing a view of student 1. It shows Student 1 tool 1201 as well as Teacher tool 1200. Note that Student 1 himself 1's IP address is shown along with his tools. Turn this feature off. Or, for example, it can be replaced with the student's name or some other identifier, just as with the teacher's view. The 3D object 1225 that the student sees is the same as the teacher's view shows, and exists in the same exact perspective or viewpoint. This is because Student 1's view is locked on to the teacher's. Therefore, a lock-on sign 1290 is displayed at the upper right of Student 1's screen. The teacher tool 1200 is visible as three networking tools: synchronization 1250, remote snapshot 1260, and role switch 1270. Since student 1 is a student and does not control networking functionality, no snapshot is displayed in the snapshot window 1220. In fact, if the student wants to see the teacher's view, he just needs to lock on the teacher's view, as already shown in FIG. Therefore, the remote snapshot 1260 is ghosted. The synchronization 1250 and the role switching 1270 are similarly ghosted. Only the teacher can perform three functions. FIG. 13 shows the view of Student 2 as in FIG. According to it, the same 3D object 1325 is shown. It can be seen that the view of Student 1's 3D object looks more transparent than that of Teacher or Student 2. This is because even though all workstations are running the same program, such workstations have different configurations. For example, some features may be disabled, eg, “ghost 3D objects when control panel is touched by stylus”. Such a feature can facilitate, for example, a stylus and a control panel.

  This is a graphic available to the user (which may be using a DextroLap system running on a laptop, or a high-speed Dextroscope (R)) This is because it depends on the scand. The more “decorative” features of 3D interactive visualization software may not give exactly the same visual results, but will behave identically in terms of 3D object effects (of course, synchronous To maintain a brilliant view, it must be exactly the same). One such decorative function is illustrated, for example, in FIG. Just play the control panel and the stylus to facilitate the display when the user interacts with the control panel (and therefore responsiveness requires that you click the buttons and sliders exactly), and the 3D object Can be captured and pasted transparently onto the 3D view. If not replicated to the student workstation, the application results are the same, but the interaction speed is slower. And the display content of the object will be different (i.e. not transparent).

  In FIG. 13, Student 2 is also in the lock-on mode, so the lock-on sign 1390 is also displayed in FIG. Student 2 can see his own tool, in this case the cursor 1302. Student 2's local IP address is also displayed. In another embodiment, it is replaced or expanded, for example as described above. As in the Student 1 case, no snapshot is displayed in the snapshot window 1320, and both the networking tool synchronization 1350 and the remote snapshot 1360 are ghosted, similar to the role switch 1370.

  Next, FIGS. 14 to 17 will be described. This shows the teacher of FIG. 10 and the two students at an exemplary time following that shown in FIGS. In FIG. 14, the teacher drew a line from the cubic object at the right rear corner of the 3D object to the tip of the conical object visible on the left rear side of the 3D object. The teacher did this using the exemplary measurement tool. Therefore, a measurement box appears near the end of the measurement, which is labeled “47.68 mm”. Both students are in lock-on mode. Next, the three views will be described in detail with reference to FIGS.

  FIG. 15 is a diagram showing a teacher's view. The teacher tool 1500 used to measure from the corner cube object to the tip of the conical object visible on the left back side of the 3D object 1525 is visible. Also visible are the Student 1 remote tool 1501 with the Student 1 IP address and the Student 2 remote tool (cursor) 1502 with the Student 2 IP address. As can be seen from the teacher's view of FIG. 15, Student 1 points to his remote tool near a point on the cube where the teacher's measurement began. Student 2 points somewhere near the base of the cone.

  In the Student 1 view, FIG. 16, the teacher's tool 1600 can be seen. The teacher's tools are, of course, far from Student 1. In addition, the Student 1 tool 1601, 3D object 1625, and the measurement line drawn by the teacher and the measurement box 1692 are visible. In addition, a lock on sign 1690 is also visible. This indicates that Student 1 is locked on to the teacher's view. FIG. 17 shows a view of Student 2. Student 2 is locked on to the teacher's view. Therefore, the lock on sign 1790 is displayed. Student 2 remote tool 1702 along with teacher tool 1700 is shown along with his IP address. Further, a measurement line drawn by the teacher can be seen between the cubic object and the conical object in the 3D object 1725. Accordingly, a measurement box 1792 indicating the length of the measurement performed by the teacher is also shown.

  18 to 21 are diagrams showing teachers and students that are significantly different from those shown in FIGS. In FIGS. 18-21, Student 1 separates his view from that of the teacher. As shown in detail in FIG. 19, the teacher can see his own tool 1900, Student 1 remote tool 1901 and Student 2 remote tool 1902, respectively. As can be seen from FIG. 19, the teacher has just measured with his tool 1900 from the top left near the corner of the object of the legislature to the tip of the conical object. Similarly, in FIG. 20, student 1 is no longer locked on. Therefore, the lock-on sign does not appear in his view. Student 1 has separated his view from that of the teacher so he can see the 3D object 2025 from above. Or you can see every other viewpoint he chooses. Furthermore, in disconnected mode, the control panel need not affect local interaction, for example, although not shown here. The student may be presented with a set of control buttons that appear on the side of the display as described, for example, in the DextroLap application. Alternatively, for example, a control panel in which the local part is omitted may be provided. Such a local control panel can be ghosted, for example, with a shortened look as described above. If you can easily recognize that it is not a “true” control panel under the control of a teacher, you do not need to ghost. FIG. 21 shows a view of Student 2. Since student 2 is still locked on by the teacher, a lock-on sign 2102 is displayed. Student 2 can have both a teacher tool 2100 as well as his own tool 2102. Student 2 can also see the 3D object 2125 and the measurement line applied by the teacher.

  Next, FIGS. 22 to 25 will be described. These are the same as those in FIG. 18 except that the position of the teacher's pen has changed. It can be seen that the view of the teacher in FIG. 23 is the same as in FIG. 19 except that the teacher's tool has moved slightly. Thus, the teacher's tool 2300 is essentially rotating around the tip of the conical object in the 3D object 2325. Student 1's remote tool 2301 as well as Student 2's remote tool 2302 are still visible in the same position they occupied in FIG. In the separated view of Student 1 in FIG. 24, the only point changed is the position and orientation of the teacher tool 2400. The student 1 tool 2401 is now the same here. As can be seen from a comparison of FIG. 20 and FIG. 24, the Student 1 tool is rotated downward about the measurement endpoint (essentially the cone end of the 3D object 2425) and the measurement line in FIG. The angle formed with the measurement line is made smaller than the angle formed with. It can be seen that Student 1's view does not show Student 2's cursor. This is because, as described above, only teachers can see all students, and each student can only see teachers. Thus, each student has virtually forgotten the presence of another student (of course, unless one of the students switches roles to become a teacher; the process is described more fully below).

  Similarly, FIG. 25 is a diagram showing a view of Student 2. FIG. 25 is substantially the same as FIG. 21 except for the position and direction of the teacher's remote tool 2500. Neither Student 2's own tool 2502 nor 3D object 2525 is moving. Student 2 is still in lock-on mode with respect to the teacher's view, so a lock-on sign 2590 is displayed in this Student 2 view.

  26-29 illustrate another embodiment in which a teacher and two students are participating in a networking session. In FIG. 26, the teacher is connected to the network. The system displays the message "You are a teacher" and you can see the teacher's virtual tools. The student is not yet connected. In FIG. 27, the first student participates. His tools and IP address are seen in the data session, and his first (timed he entered) snapshot can be seen as well. In FIG. 28, a second student joins and his tool and IP address are available to the teacher at this point. Each student's IP address appears as a text box next to their virtual tool. 29 and 30, the teacher is synchronized with the first and second students, respectively. As described above, the teacher selects which student to synchronize by selecting the student from the student list displayed on the left panel of the networking control panel.

  FIGS. 31-43 are diagrams illustrating the order of the various “surgeons” and “visualization assistant” views in an embodiment of the present invention. These figures are shown, for example, by a surgeon performing surgery using a DEX-Ray® system, for example, Dextroscope® or DextroBeam® Figure 2 illustrates how an example collaboration can occur when a visualization assistant using a trademark type system is connected to a surgeon's machine over a network. This paradigm is, for example, that one person has a diagnostic or therapeutic procedure and thus obtains real-time information about the subject, and the other person receives such real-time data and uses it and 3D data about the subject. It can be applied to any situation that uses to create a visualization of an object to assist a first person. Examples of such visualization assistants, for example, take advantage of the available viewpoint freedom (ie he is free to rotate, translate and enlarge objects in the 3D data as he desires. See things that are not visible to the surgeon), not limited to seeing such objects in the position and direction they are looking at. Thus, for example, the surgeon is cooperatively guided to the patient's tissue at the surgical site. These figures are described next.

  FIG. 31 illustrates two examples of visualization assistant (“VA”) views. In general, in the surgeon / visualization assistant paradigm, a surgeon creates a VA that locks on and generally will not correlate with the viewpoint and approach angle physically used in patient or subject processing. Unless you have decided to see the best visualization, you are disconnected from the VA view. In the context of FIG. 31, and in general, VA is used to assist surgeons using surgical navigation systems, such as the DEX-Ray® type system. As described above, the surgeon acts like a student, and as mentioned above, the visualization assistant acts like a teacher. This is due to the fact that the operating surgeon has less freedom with respect to the display of 3D virtual data, as his viewpoint is limited to the probes and other instruments he holds. For example, using a DEX-Ray® system to view a viewpoint from 3D data is the same as viewing with a camera in a hand held probe. On the other hand, since VA sees only 3D virtual data, it is possible to change the viewpoint without limitation. Because the surgeon generally wants to display augmented reality (or if using other surgical navigation systems, the 3D data seen by the viewpoint is almost the same as what he tracks) The surgeon's view is usually disconnected from the VA.

  Thus, in FIG. 31, the visualization assistant rotates the skull representing the object of interest in the surgeon's surgery and procedure so that the coronal view is visible on the left and the sagittal plane is on the right. In each visualization assistant's view, the surgeon's (acting as a student) remote tool 3100 is visible. In the sagittal view, the skull opening can be easily seen. The perspective of the surgeon dividing is different from each of these. This will be described more fully with reference to FIG. As shown in FIG. 34, the surgeon's actual view is more along the axis of the surgeon's remote tool 3100 than is in his surgical path as shown below. 32 and 33 are enlarged views of FIG. 31, respectively.

  FIG. 34 illustrates a surgeon's view corresponding to the visualization assistant view of FIG. This view is the actual view that the surgeon has and is shown locally on his, eg, DEX-Ray® type system. The surgeon's point of view can be the camera point of view of the camera probe described in the Camera Probe application, or his physical orientation and patient's position, for example, when using another surgical navigation system. It corresponds to (and is limited by) the path of approach to the interior. In FIG. 34, the surgeon's tool 3400 corresponds to the actual camera probe he has or a navigation probe or instrument. Also shown in FIG. 34 is a skull opening 2410 and a surgeon's IP address 3416, and a partially visible sphere 3405, which can be replaced or expanded with some other identifier, as described above. It is precisely this object that the visualization assistant obtains a better view by optimizing his viewpoint so that the surgeon locates the point. 31-43, it is assumed that the sphere 3405 (with respect to FIG. 34) represents an object of interest, such as a tumor, that the surgeon is processing. FIG. 35 shows a view of the visualization assistant and a corresponding surgeon's isolated view showing the skull including the skull opening as described above. In FIG. 35, the visualization assistant helps the surgeon locate the point on the sphere. The visualization assistant clearly shows the sphere in this view by cutting away the sides of the skull. On the other hand, the surgeon's view, FIG. 35 (b), is limited to the fact that his probe is moving in the real world and moves only into the actual hole in the skull or the opening of the skull. Can do. In contrast, the visualization assistant's view is unrestricted and is free to process the data to best visualize this sphere. In each view of FIG. 35 (a), the surgeon's tool 3500 is visible. As shown in FIG. 35 (b), the surgeon's tool axis corresponds more or less to his point of view. On the other hand, in FIG. 35A, the visualization assistant is looking at the object from the rear with respect to the surgeon's actual path. 36 and 37 are enlarged views of FIGS. 35 (a) and 35 (b), respectively. As can be seen from FIG. 37, the surgeon's IP address 3716 is clearly displayed.

VI. Integration with Other Surgical Navigation Systems In embodiments of the present invention, devices with 3D tracking (or 2D tracking) capabilities from manufacturers of various 3D interactive visualization systems, such as dextros on a DextroNet It is possible to connect a Dextroscope (registered trademark) or the like. Such “outpatient” devices will provide predetermined information to the DextroNet in response to questions or instructions sent over such networks. For example, Medtronic (Louisville, Colorado, USA) manufactures various navigation (or image guided) systems for neurosurgical procedures. One such navigation system is, for example, the TREON system. In addition, Medtronic also manufactures application program interface (API) network interface software. It can stream data from such navigation systems to external applications in real time.

  Similarly, another manufacturer, BrainLAB AG (Munich, Germany) has a similar software product. This product uses a custom-designed client / server architecture named VectorVision Link (VV Link) and extends functionality from the Visualization Toolkit (VTK). VV Link allows bi-directional data to transfer image data sets, visualizations, tool positions, etc. in real time. These devices provide registration information and probe coordinate information in a manner similar to the DEX-Ray (R) system, but they are not augmented reality based, so video information is provided Not. In an embodiment of the present invention, a modified DextroNet server could incorporate Stealthlink® or VV Link software or the like. After connecting the details of patient information and registration information are interchangeable, for example, DextroNet queries these systems, for example, to obtain their probe coordinates. Thus, in an embodiment of the present invention, such a system can function as a surgeon's workstation and provide spatial coordinates to a teacher (VA) workstation. Unlike the DEX-Ray® implementation described above, these currently configured systems offer an option to display to the surgeon's view during surgery from the VA workstation. Does not have. To do so, it is necessary to incorporate software that embodies the method of an embodiment of the present invention in a navigation system.

  Similarly, as described above, in an embodiment of the present invention, the machines connected via the exemplary DextroNet can be products from different manufacturers, in different configurations, and in different types. Can be. As mentioned above, surgical navigation systems, such as the DEX-Ray (registered trademark) system, or Medtronic or BrainLAB systems via DextroNet It can be connected to a standard 3D interactive visualization workstation such as Dextroscope®, DextroBeam®, etc. Furthermore, as described above, a system that transmits 3D position data, for example, a surgical navigation system that does not use augmented reality can be connected.

  44-49, described next, is yet another use of DextroNet in one embodiment of the present invention. This example relates to an example of a 2D fluoroscopic image created by a cardiologist sent to a 3D interactive visualization system such as Dextroscope® via DextroNet. is there. The paradigm is similar to that of the surgeon and visualization assistant described above. However, in this case there is no surgeon, instead there is an interventional cardiologist. In such an embodiment, the visualization assistant will allow the interventional cardiologist to analyze his anatomical interest by means of unrestricted 3D processing by the visualization assistant of the CTA scan that was acquired preoperatively. Help visualize the structure.

  Accordingly, in FIG. 44, an exemplary fluoroscopic image obtained from a Cathlab® procedure is shown. The image was acquired by applying X-rays to the patient's rib cage. In the image, an artery, more precisely, a portion inside the artery taken by administration of a contrast medium can be seen.

  FIG. 45 illustrates an example view of a standard interventional cardiologist. In FIG. 45, the interventional cardiologist sees only the 2D projection of the blood vessel taken by administering the contrast agent from the viewpoint provided by the exemplary fluoroscope. The image shown in FIG. 45 is a simulated projection of such a conventional interventional cardiologist's view (matching a series of actual fluoroscopic images. The relevant CTA is also available. could not). Simulations were obtained by manipulating on the CTA data (segmenting the CTA coronary artery (so only the arteries are visible and no other tissue is visible. The contrast agent flows through the arteries, the fluorescence Interact with the x-rays emitted by the fluoroscopy device), color them darkly as if the result of fluoroscopy, and direct the segmented artery, take a snapshot, and then snap the CTA Take a shot without segmentation).

  FIG. 46 is a diagram illustrating an example of a visualization assistant view corresponding to the clinician view of FIG. 45. Such an example visualization assistant can work with the interventional cardiologist of FIG. As described, such visualization assistants perform unrestricted 3D processing on pre-operative CTA. FIG. 46 is an enlarged view of the coronary artery examined by the VA. FIGS. 46-48 illustrate an example of an interaction between an interventional cardiologist and a visualization assistant in an embodiment of the present invention. These figures show a method for manually registering a CTA diagram together with a fluoroscopic view. Here, for example, VA obtains fluoroscopy as shown in FIG. 45 via DextroNet, and then uses CTA (eg, segmented coronary artery) by using the image, for example. Can be adjusted to match this received image. The VA makes a direction determination and what the cardiologist sees (there are limitations on fluoroscopy devices, usually not tracked, only a few standard locations that the cardiologist knows well) As soon as it knows), unrestricted 3D processing can tell the cardiologist what it is in real time. He labels those vessels, for example, annotating “left coronary artery” or “LCA”, as well as “right coronary artery” or “RCA”, for example. Alternatively, he creates a narrowing in the blood vessel that can be provided to an interventional cardiologist with a Cathlab® image (projection view or stereoscopic if such a display is available). Can be pointed (in 3D). Further, for example, VA can be measured. And if the VA can see a new image from the fluoroscope, he can identify where the catheter is, estimate the 3D position, and then determine the distance to the important anatomical landmark Can be communicated to a cardiologist.

  FIG. 47 is a diagram showing a scenario similar to FIG. 8 described above. The visualization assistant can see both the entire 3D and the main image, and can also see the snapshot or “in-picture” image in the upper left corner. The image in the figure is created by, for example, an exemplary fluoroscopy device in an interventional cardiologist's Cathlab® machine. It is essentially the image shown in 45 above. The visualization assistant uses a Dextroscope® or equivalent device to process and segment (or re-segment) pre-operative CTA data, optimally visible in a fluoroscope It is possible to visualize blood vessels with the same viewpoint. This can be done, for example, by aligning the viewpoint with what he sees in the figure. In addition, on the far right side of the view of the VA, an example interaction function button is seen, similar to that normally seen in a DextroLap performance. It can be seen that VA can be used as required, for example, DextroLap. Or, for example, he could make maximum use of a Dextroscope®.

  FIG. 48 illustrates yet another example 3D view that a VA can create. That is, as described, the VA has access to all CTA data, so the acquisition plane can be grouped, for example, as shown in the left image of FIG. Or, for example, the data can be segmented to clearly show only the coronary arteries, as shown in the right image of FIG.

  Finally, FIG. 49 shows side by side images that may be displayed, for example, on an interventional cardiologist's Cathlab® device. Intervention cardiologist has a fluoroscopy (left) acquired with his local machine and a visualization assistant created with, for example, a Dextroscope® or DextroLap machine You can compare on the same display. This comparison is easy, allowing the cardiologist to better interpret the fluorescence projections. By communicating between the interventional cardiologist and the visualization assistant, the visualization assistant can improve, for example, improve the view he created locally, so that it may be desirable and useful to the interventional cardiologist. Re-segment and optimize and send it to the interventional cardiologist via DextroNet for display (as shown in Figure 49) for comparison by the interventional cardiologist . This can be done, for example, using features such as the Cathlab® system with several monitors showing fluoroscopic procedures. Adding a 3D image to another monitor would be a simple task. These latter images may or may not coincide with fluoroscopy, for example. In addition, such a system also shows the position of the fluoroscope, as well as other patient information. Alternatively, the VA visualization can be returned to the clinician using other displays such as, for example, a monitor with a simple touch screen used under aseptic conditions.

VII. Role Switching In an embodiment of the present invention, a role switching function can be supported. Role switching is a feature in the exemplary DextroNet that allows teachers and students (or surgeons and visualization assistants) to exchange their respective roles online. In the Teacher-Student paradigm, a student cannot process objects in a 3D dataset except translation, rotation, and inching on objects. If the student takes over control from the teacher by switching roles, he can continue to work with the teacher on the object, for example. This suggests a mode for some degree of cooperation. Furthermore, at one point in time, there is only one teacher on the network. This cooperation is continuous and does not cause collision problems.

  In the exemplary DextroNet, teachers and students can both be clients at a low level, so that role switching occurs naturally. Role switching can use the current communication session (ie, networking does not have to be stopped and reconnected), and teacher and student roles can be performed at a high level. In this way, role switching can be done fairly quickly, avoiding using time for reconnection.

  Role switching supports a large number of students. The teacher decides which student will transfer his control rights. The other student can maintain the state as it is, but can know, for example, that the teacher has changed. As described above, in the embodiment of the present invention, both the teacher and the student are, for example, clients in the low-level sense. As shown in FIG. 9, in the embodiment of the present invention, the DextroNet uses a server client architecture. In such an architecture, both teachers and students are clients. Servers can also be used, for example, to communicate between a teacher and multiple students. Thus, role switching uses the current communication session without stopping networking or reconnecting. During the entire process, no change in physical connection occurs. Only one logical new role needs to be assigned and reassigned to the server. A typical process for communicating role changes between all clients is, for example: The student requests control rights, the teacher tells other students, and the server prepares to switch roles. Then, for example, both the teacher and the student reset and update their tools according to the example role changes shown below.

  • The teacher's machine that will be the student changes the student tool to a local tool and changes his teacher tool to a dedicated remote tool (ie, a new teacher). Other remote student tools are removed from the display.

  • The student's machine that will be the teacher changes his teacher tool to a local tool and adds other students and a representation of the tool for the teacher who will be the student (student tool).

  -Other student machines will turn their teacher tools into new teachers. Their tools remain as student tools.

  As soon as these role changes are complete, all clients re-register their new roles on the server. Thus, the role switching is completed.

  A number of students can be supported using an ongoing communication session. Data flow management may require caution. Data received before and after the exemplary role switch needs to be separated, for example. Also, re-registration of roles on the server needs to be synchronized, for example. To achieve this, the entire process is divided into three phases, for example, preparation for role switching (Phase I), role change (Phase II), and registration of new roles to the server (Phase III). Can be divided. At the end of each phase, the state of all clients can be synchronized.

  This process can be summarized as follows using FIGS.

A. Phase I (Figure 50)
(1) The teacher will signal all students who have transferred their controls. After getting a positive response from all students, his tool is reset and the server notifies him that he is now ready to switch roles and is in a zombie state (ie, the machine can only receive data) In the state where transmission is not possible.

  (2) When a student receives a signal from a teacher, he checks with his machine whether he is a teacher or remains a student. On the other hand, the server also notifies that it is ready to switch roles after resetting the tool. This reset does not affect changes already made to the volume object. He then enters a passive or “zombie” state. That means, as described, a state where the machine only receives a message but does not send it.

  In an embodiment of the invention, the user does not have to worry about all notifications in the role switching process. All role switching processing can be done automatically when the user presses the “Switch Role” button (and “do something” refers to the teacher or student in the description above and below. Actually refers to an example of software that does that).

  In an embodiment of the present invention, the server can be used to manage communications between teachers and students. For example, it can be a stored list that is a memorandum of their role. Therefore, the server must be notified when the role is switched. Furthermore, during role switching, the state between the teacher and the student must be synchronized at the end of each phase. The server may coordinate this process, for example. For example, at the end of Phase I, the student / teacher must inform the server whether it is ready to switch roles. After this, he enters a zombie state and waits for instructions from a server capable of changing roles. The server can, for example, count how many clients are ready for role switching. After knowing that all clients are ready, he gives everyone instructions and they can enter Phase II.

  (3) When the server knows that all clients are ready to switch roles, it sends a message to them and resumes from the zombie state.

  (4) When the client (teacher-student) resumes, the client changes his role, notifies the server and enters the zombie state again.

C. Phase III (Figure 52)
(5) When the server receives a message from all his clients that he has switched roles, he first restarts the teacher.

  (6) The teacher re-registers with the server and notifies the server. The server then resumes the student.

  (7) The student re-registers with the server, and after the teacher receives the first snapshot from the student, the teacher requests the server to resume the next student.

  (8) Step (7) can be repeated, for example, until all students have re-registered.

  Furthermore, in the embodiment of the present invention, the role of the teacher can be turned from the related person to the related person with respect to a plurality of related persons by performing each role switching through the above process.

VIII. Data Format In embodiments of the present invention, the following measures can be used to ensure proper display on the remote terminal.

A. Data Each side (teacher-student) has a copy of the same data. If the data is different, the entire data set can be synchronized (data file compression and transmission).

B. Initialization In an embodiment of the present invention, the virtual control panel can be synchronized on both sides when initializing the networking function. For example, in a Dextroscope (R) / DextroBeam (R) workstation, the virtual control panel uses a physical acrylic base when the 3D tracker is paused during interpretation Must be measured accurately. Thereby, the virtual tool can touch the corresponding part on the virtual control panel. Thus, the scale and configuration are local and vary from machine to machine. This can lead to mismatches during networking, for example, the teacher's tool cannot touch the student's control panel. To circumvent this problem, in some embodiments of the present invention, the control panel can be synchronized. While the networking function is running, the position, orientation, and size of the teacher's control panel can be sent to the student, and the student's own control panel parameters can be replaced. When the networking function ends, the student's previous configuration can be restored to operate standalone. In an embodiment of the invention, the viewpoints on both sides can be synchronized when initializing the networking function. For proper display, teachers and students should share the same eye position, viewing position, projection width and height, rotation angle, etc. Therefore, all information about the viewpoint should be synchronized at the start of the networking function.

  In an embodiment of the present invention, the zoom boxes on both sides can be synchronized when initializing the networking function. Therefore, when the networking function starts, the zoom boxes on both sides must be synchronized by position, direction, boundary, computer screen area, and the like.

C. Synchronizing widgets In the embodiment of the present invention, at the start of the networking function, the initial state such as the position of the slider bar of the control panel on both sides, the state of buttons and tabs, the list of color automatic matching table, etc. Absent. All these parameters need to be aligned.

D. Communication In embodiments of the present invention, two types of coordinate systems can be used. World coordinates and object coordinates. World coordinates are coordinates attached to the virtual world. Object coordinates are attached to each virtual object in the virtual world. In an embodiment of the present invention, all virtual tools can be displayed in world coordinates. Teachers and students can each communicate with the type of coordinate system they use.

  When in the student engaged mode ("lock on"), both the teacher and the student send their tool name, state, position, direction, and size in world coordinates to his peers.

  If the teacher's tool touches the control panel on the teacher's side when the student is in the released mode (not "lock on"), the teacher will learn the tool name, state, position, direction, and size in world coordinates. Send to. Otherwise, he sends the position, direction and size in object coordinates to the student. When the student receives information related to the teacher's tool, he converts the received information from object coordinates to world coordinates, and then displays the teacher's tool in his world. Meanwhile, the student sends his tool name, state, position, orientation, and size in object coordinates to the teacher. The teacher can then convert them to world coordinates. This can be done before the student tool is displayed. In an embodiment of the present invention, a student can determine what action is taking place based on the position, orientation, and state of his world teacher's virtual tools.

  In an embodiment of the present invention, DextroNet can synchronize the modes of the teacher-side and student-side viewpoints. The student can choose to open the viewpoint. He can, for example, press a button on his stylus. This action causes a message to be sent to the teacher's machine, resulting in the effect “I will leave”. The student cannot, for example, actually switch his viewpoint. He continues to use world coordinates from the teacher. When the teacher's machine receives a student message, for example, it can then send an acknowledgment to the student and then change to object coordinates. Upon receiving such an acknowledgment from the teacher machine, the student machine can then make changes to actually leave and then use the object coordinates.

  For example, if the student is engaged again, the situation is the same. Thus, in an embodiment of the present invention, when the teacher machine knows that the student has decided to leave, the student machine can only change the viewpoint of the student. In this way, it is possible to avoid the occurrence of a collision with respect to the coordinate type between the teacher machine and the student machine before and after leaving or re-engaging.

E. Telegraph format In embodiments of the present invention, for example, there may be two telegraph formats. One can be used for updating messages, for example, and the other can be used for files, for example.

1. Format I for message update
FIG. 54 shows an example of a first format for updating a message. For example, the following fields with the following attributes can be used:

Begin Tag: indicates the start of telegraph (unsigned character);
Data Type: Indicates whether the content is a message update or a file. For message updates, this value is 2 (an unsigned integer);
IP: IP address of the source (unsigned characters);
Object name: the object assigned to use this message;
Co-ord System: A coordinate system for interpreting position, direction, and size in telegraphs (unsigned characters). For example, there are two possible values: “wld” for world coordinates, “app” for object coordinates;
Position: The position of the object in the “object name”. A position contains three values x, y, z while floating;
Orientation: The orientation of the object in the “object name”. The direction is a 4x4 matrix. Each element in the matrix is floating;
State: The state of the object in the “object name” (unsigned character) as necessary. If the object is a tool, its value is one of four states: MK_CHECK, MK_START_ACTION, MK_DO_ACTION, MK_END_ACTION.
Size: The size of the object in the “object name”. The magnitude has three values: x, y, z in the float;
End Tag: Indicates the end of telegraph (unsigned character)

  55A to 55C are diagrams for explaining three examples using the format of FIG. FIG. 55A is a diagram for explaining update of a message for synchronization with the control panel. FIG. 55B is a diagram for explaining update of a message for synchronization with the widget. FIG. 55C is a diagram for explaining update of a message for synchronizing with the virtual tool.

2. File Transfer In an embodiment of the present invention, a long file can be transferred in several blocks. Each telegraph can include such a block. In the embodiment of the present invention, before the file is actually transferred, an update message can be transmitted to inform a friend that the file is to be transferred. The "Size" fields are the Total Block Number (unsigned integer), Block Size (unsigned integer), and Last Block Size (sign 54 can be sent in such an update message, if it can be modified to include (no integer). FIG. 56 shows an example of such an update message. FIG. 56 shows an exemplary update message for the transfer of an exemplary file of 991 KB (1,014, 921 bytes). Therefore, considering the data in the size field, the peers must have a file that is 248 blocks, each block excluding the last block is 4096 bytes, and the last block is 3209 bytes. Know. The file itself can be sent using the second format for update messages employed in file transfer.

3. Format for message update II
FIG. 57 shows an example of a second format for updating a message. In an embodiment of the present invention, the following fields can be used.

Begin Tag: indicates the start of telegraph (unsigned character);
Data Type: Indicates whether the content is a message update or a file. For files, this value is 1 (unsigned integer);
File Block: File block in binary (unsigned characters);
End Tag: Indicates the end of telegraph (unsigned character)

  In the exemplary file transfer transmission, the first 247 blocks, each of 4096 bytes, can be transmitted, for example, as shown in FIG. 58 (a), and the last block, each of 3209 bytes, is formatted II. Can be transmitted as shown in FIG.

  While the invention has been described in connection with one or more embodiments, it is not intended to be limited thereto, and the appended claims not only include the specific format and variations of the invention shown, It is to be construed as further including those that can be modified by those skilled in the art without departing from the true scope of the invention.

FIG. 1 is a process flowchart illustrating an example of a teacher type workstation in an embodiment of the present invention. FIG. 2 is a system level diagram illustrating examples of various workstations connected via a network in an embodiment of the present invention. FIG. 3 is a process flow diagram illustrating an example of a student workstation in an embodiment of the present invention. FIG. 4 is a process flow diagram illustrating an example of a surgeon workstation in an embodiment of the present invention. FIG. 5 is a process flow diagram illustrating an example of a visualization assistance workstation in an embodiment of the present invention. FIG. 6 is a standard (empty) view of the surgeon in an embodiment of the present invention. FIG. 7 is a standard view of a surgeon in use in one embodiment of the present invention. FIG. 8 is a diagram showing an example of visualization assistance in an embodiment of the present invention. FIG. 9 is a diagram illustrating an example of a teacher-student paradigm architecture in an embodiment of the present invention. Figures 10a to 10e are diagrams illustrating an example of a teacher-student paradigm architecture connected on top of an example of a dextronet (DextroNet) in an embodiment of the present invention. 11 to 13 are enlarged views of FIGS. 10a to 10c. FIGS. 14a to 14c are diagrams showing examples of 3D data sets observed in the lock-on mode by the teacher of FIG. 10 and two students as an example. FIGS. 15 to 17 are enlarged views of FIGS. 14a to 14c. FIGS. 18a to 18c are diagrams showing examples of the 3D data set shown in FIGS. 14a to 14c, in which the teacher performs the measurement and the student 1 switches to the empty diagram mode. 19 to 21 are enlarged views of FIGS. 18a to 18c. FIGS. 22a to 22c show the examples of FIGS. 18a to 18c, respectively, after the teacher has moved the pen. 23 to 25 are enlarged views of FIGS. 22a to 22c. FIGS. 26-39 are diagrams illustrating an example order from a teacher's point of view in which two students are participating in a networking session in one embodiment of the present invention. FIG. 31 is a diagram showing two examples of visualization assistance in an embodiment of the present invention. 32 to 33 are enlarged views of FIG. 31 continuously. 34 is a diagram illustrating an example of a surgeon corresponding to the visualization aid shown in FIG. 31 in one embodiment of the present invention. FIG. 35 is a diagram showing the visualization assistance and the surgeon's viewpoint, respectively, when both parties position a given point on the phantom object in an embodiment of the present invention. 36 to 37 are enlarged views of FIG. 35 continuously. FIG. 38 is a diagram illustrating visualization assistance in assisting a surgeon to position a point on an object in an embodiment of the present invention. 39 to 40 are enlarged views of FIG. 38 continuously. FIG. 41 is a diagram further illustrating an example of positioning a point on an object in cooperation with a surgeon in an embodiment of the present invention. 42 to 43 are enlarged views of FIG. 41 continuously. It is a figure which shows the example of the image by a fluoroscopy method. FIG. 45 is a diagram illustrating an example of a viewpoint of a cardiologist performing an intervention. FIG. 46 is a diagram further illustrating an example of positioning a point on an object in cooperation with a surgeon in the embodiment of the present invention in cooperation with a surgeon. FIG. 47 is a diagram showing an example of the figure created by the example of the visualization assistance shown in FIG. 46 in an embodiment of the present invention. FIG. 48 is a diagram illustrating another example of the visualization assistance 3D diagram illustrated in FIGS. 46 to 47 according to an embodiment of the present invention. FIG. 49 is a diagram showing another example of the viewpoint of a cardiologist performing an intervention according to an embodiment of the present invention. FIG. 50 is a diagram showing an example of a first stage role switching process in an embodiment of the present invention. FIG. 51 is a diagram illustrating an example of the role switching process of the second stage in an embodiment of the present invention. FIG. 52 is a diagram illustrating an example of the role switching process of the third stage and the final stage in an embodiment of the present invention. 53a and 53b are diagrams illustrating a data transmission queue in an example of a teacher (or main user) system in an embodiment of the present invention. FIG. 54 is a diagram showing an example of a message for updating the format according to an embodiment of the present invention. FIGS. 55a to 55c are diagrams illustrating examples of messages for updating examples of control panels, widgets, and tools using the message format of FIG. 56 is a diagram showing an example of a message related to file transfer using the message format of FIG. 54 in an embodiment of the present invention. FIG. 57 is a diagram showing an example of another message for updating the format according to an embodiment of the present invention. 58a to 58b are diagrams showing a file transfer message using an example of the message format of FIG. 54 in an embodiment of the present invention.

Claims (57)

An apparatus for interactively processing a three-dimensional image, wherein the image includes volumetric data created from imaging data relating to an object, the apparatus comprising:
Receiving position data of one or more remote probes of one or more remote machines over a communication link;
Creating a combined 3D scene including the at least one remote probe and the 3D image for display on a local display;
In response to processing of a local probe by a user local to the device, processing the 3D image;
Data relating to the processing by the user that is local to the device on the communication link, the image including a local probe image processed on the three-dimensional image on the at least one remote machine Send enough data to display the rendered 3D scene,
The device is configured as follows.   The apparatus of claim 1, further configured to filter the data transmitted for the processing by the user that is local to the apparatus in response to network conditions.   The apparatus of claim 3, wherein the filtering drops data packets that do not substantially modify the three-dimensional image.   The apparatus of claim 3, wherein the filtering drops packets associated with one of tool movement and tool status.   The apparatus of claim 1, further configured to display at least one of an IP address, a user name, and a user identifier of the one or more remote machines.   The data sent to the one or more remote machines and related to the processing by the user that is local to the device is as if the local probe is a probe associated with the one or more remote machines. 2. The method of claim 1, wherein interacting with a local machine virtual control panel is sufficient to cause the one or more remote machines to display an image of the local probe for processing on the 3D image. apparatus.   The apparatus of claim 1, further configured to receive a snapshot of the display of the one or more remote machines in response to an instruction of the user that is local to the apparatus.   The system is further configured to cause a three-dimensional image synchronization between the device and any of the one or more remote machines in response to a command from the user that is local to the device. The apparatus according to 1.   The apparatus of claim 8, wherein causing the synchronization includes sending a compressed copy of the 3D image stored on the apparatus to the one or more remote machines.   The apparatus of claim 8, wherein causing the synchronization includes sending a list of interactive commands executed on the three-dimensional image of the apparatus.   The apparatus according to any of claims 1 to 10, wherein the apparatus is further configured to switch roles of the local machine and the remote machine in response to a request by a remote user of the remote machine. An apparatus for interactively processing a three-dimensional image, wherein the image includes volumetric data created from imaging data relating to an object, the apparatus comprising:
Receive the position data of one remote probe of one remote machine on the communication link,
Receive the position data of one local probe,
Creating a combined 3D scene including the remote probe, the local probe, and the 3D image for display on a display;
In response to processing of the remote probe, processing the 3D image in a manner substantially equivalent to processing of the 3D image by a user local to the remote machine via the local probe;
The device is configured as follows.   As if the remote probe is a probe associated with the device, the processing of the 3D image in response to the processing of the remote probe is displayed as an image of the remote probe interacting with the virtual control panel of the local machine. The apparatus according to claim 12.   The apparatus according to claim 12, further configured to display at least one of an IP address, a user name (usemame), and a user identifier of the remote machine.   The apparatus of claim 12, further configured to send a snapshot of the display in response to a command of the remote machine.   The apparatus according to claim 12, further configured to synchronize a three-dimensional image using image data transmitted from a remote machine over a communication link in response to a command of the remote machine.   Furthermore, it is configured to display a three-dimensional image having the same viewpoint as that of a user local to the remote machine and an arbitrary viewpoint different from that of the user local to the remote machine. The apparatus of claim 12.   The apparatus of claim 17, further configured to display a virtual control panel.   Further, when the 3D image is displayed at an arbitrary viewpoint different from that of a user local to the remote device, a dedicated local The apparatus of claim 17, configured to display a control panel.   Furthermore, the apparatus in any one of Claim 12 thru | or 19 comprised so that a user may perform local operation on the said three-dimensional image.   21. The apparatus of claim 20, wherein the local operation includes a process that does not affect which voxels are considered part of the object.   21. The apparatus of claim 20, wherein the local operations include one or more of object translation and rotation, and magnification and transparency settings.   The method is further configured to create an additional local copy of the three-dimensional image and process the additional copy in response to processing received from the user that is local to the device. The apparatus according to 12 or 17.   The device is further configured to receive additional data relating to the subject from a remote device that is local to the subject and local to one of the remote machines, the additional data comprising: The apparatus of claim 1, co-registered with the three-dimensional image of the object.   25. The apparatus of claim 24, wherein the additional data regarding the subject is acquired in near real time.   The additional data about the object is one of real-time video, pre-recorded video, probe or instrument location data local to the object, fluoroscopic images, ultrasound images, and multimodal images. The apparatus according to claim 24, which is the above.   13. The system of claim 1 further configured to display additional data about the object from a remote device that is local to the object and local to one of the remote machines. The device according to any one of the above.   28. The apparatus of claim 27, wherein the additional data is co-registered in the three-dimensional image of the object.   28. The apparatus of claim 27, wherein the additional data is near real time.   The additional data is: the additional data for the subject (as is), real-time video, pre-recorded video, probe or instrument location data local to the subject, fluoroscopic images, ultrasound 28. The apparatus of claim 27, wherein the apparatus is one or more of an image and a multimodal image. A system for interactively processing a three-dimensional image, wherein the image includes volumetric data created from imaging data about an object, the system comprising:
A main workstation comprising the apparatus according to any of claims 1 to 11 or 27 to 30;
One or more remote workstations comprising the apparatus of any of claims 12 to 26;
Data network,
A system wherein each of the main workstation and the one or more remote workstations is connected via the data network. A method of interactively processing a three-dimensional image, wherein the image includes volumetric data created from imaging data about an object, the method comprising:
Providing a first device according to any of claims 1 to 11 or 27 to 30;
Providing one or more second devices according to any of claims 12 to 26;
Providing a data network, and connecting the first device and the one or more second devices via a data network;
In operation, the user of the first device and the user of the one or more second devices cooperate to visualize a common 3D data set.   34. The method of claim 33, wherein the user of the first device and the user of a second device switch roles in the initialization of the user of the first device. A networked interactive 3D data visualization system,
A main system for acquiring real-time images of the object and combining them with parts of the co-registered 3D volumetric model of the object;
A display for displaying the combined image to at least one user;
With a probe;
A tracking unit for tracking the position of the probe;
With a data network;
One or more remote systems for interactive visualization of the combined images communicatively connected to the main system by the data network, each with a tracked virtual tool;
The tracked position of the main system probe and the combined image are received by each remote system on the data network;
The remote system interactively processes the combined image or the co-registered 3D volumetric model and sends such processed image to the display of the main system and all other remote systems .   35. The system of claim 34, wherein the combined image is a 2D real-time video placed on the object.   36. The system of claim 34 or 35, wherein the main workstation and the remote workstation are employed to modify computer-generated images and simulate operations performed on the object.   40. The system of claim 36, wherein the simulated operation includes removal of a portion of the subject volumetric model.   35. The system of claim 34, employing the main system or a remote system to receive changes from at least one user to the coloring of a segmented portion of the volumetric model.   Employing the main system to receive selected points of a volumetric model for measurement from a first user and employing the second system to adopt a volumetric for measurement from a second user. 35. The system of claim 34, receiving a selected point of a model.   35. The system of claim 34, wherein the main system or one remote system receives input from at least one user and expands over a displayed region of interest.   The main system or one remote system is employed to receive input from at least one user to change the transparency or opacity of at least one segmented object of the volumetric model. 34. The system according to 34.   Employing the main system or one remote system to modify the combined image to represent changes in the physical shape of the object to be manipulated, the modification being the tracked position of the probe 35. The system of claim 34, depending on A method used by at least one user operating in a limited three-dimensional region, the method comprising:
Creating an image to be manipulated;
Displaying the image to at least one user in a co-registration relationship with the subject;
Tracking the position of the probe having a longitudinal axis by the first system;
Transmitting the position to a first data processing device and a second system, comprising:
The first data processing device creates an image according to a line extending parallel to the longitudinal axis of the probe, and the line is controlled for extension according to the output of an extension control device controlled by the first user. Has been
The second system having a second data processing device creates at least one image to be manipulated on at least one display for displaying the image to the at least one user; And
The tracked position of the probe of the first system is received on a communication network by the second system;
Controlling the first and second data processing devices to modify the at least one image to be manipulated according to a controlled extension of the line.   44. The method of claim 43, wherein the display of the first system creates an image of the operation target placed on the target.   The first or second data processing device modifies computer generated images to simulate operations performed on the object, and the simulated operations are controlled by controlling the extension of the line. 45. A method according to claim 43 or 44.   46. The method of claim 45, wherein the simulated operation includes removal of a portion of the computer-generated image that extends deep into the patient as indicated by an extension of the line.   44. The method of claim 43, wherein the first system or the second system receives a change to the coloring of the computer-generated image from at least one user.   The first system receives a selected point for measurement from a first user, and the second system receives a selected point for measurement from a second user. 44. The method according to 43.   44. The method of claim 43, wherein the first system or the second system receives input from at least one user and zooms in on a displayed region of interest.   44. The method of claim 43, wherein the first system or the second system receives input from at least one user and changes at least one transparency or opacity.   The first data processing device or the second data processing modifies an image and represents a change in the physical shape of the object to be manipulated, the modification depending on the tracked position of the probe 44. The method of claim 43. A networked interactive 3D data visualization system,
With the main system;
A display for displaying a plurality of images to at least one user;
With a probe;
A tracking unit for tracking the position of the probe;
With a data network;
One or more remote systems for interactive visualization of the images communicatively connected to the main system by the data network, each with a tracked virtual tool,
The tracked location of the main system probe and the image are received by each remote system on the data network;
Each of the remote systems interactively processes the combined image or the co-registered 3D volumetric model and sends the location of the tracked virtual tool to the main system.   53. The system of claim 52, wherein the roles of the main system and the remote system can be changed repeatedly.   53. The system of claim 52, wherein the remote system can see either the main system view of the dataset or its own view.   53. The system of claim 52, wherein the remote system can perform any visualization process locally, but cannot modify the data set.   53. The system of claim 52, wherein the main system can synchronize its interface and data set with any remote system at any time. A method for collaboratively and interactively visualizing and processing a volumetric object or system comprising:
A local user obtains real-time images of an object or system and combines them with a co-registered 3D dataset of the same object or system using a main workstation, and the local user localizes the combined image locally. Displaying automatically and transmitting them over the data network;
One or more remote users, each connected to the data network, receive the combined image and receive at least one of the combined image and the co-registered scan image using a remote workstation. Processing steps,
Each of the main user and the remote user can process the combined image and the 3D dataset,
The method wherein the process is transmitted to a data network for a substantially periodic display on the main workstation and each remote workstation.

Images