Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating 3D models or images for computer graphics
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
  • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
  • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Definitions

• the invention relates to a method for processing image objects in three-dimensional representations, wherein an image object is represented by an image generation means in a virtual image space, image points of the three-dimensional image space can be reached by a selection device and corresponding coordinate values are assigned to the image object by a computer.
• the invention also relates to a device for editing and changing three-dimensional representations, which has a computer for generating image data, a screen for three-dimensional representation of the image data in a virtual image space and a pointing device for manually influencing the three-dimensional representation. According to the invention, this device can be used as a selection device in the above-mentioned method.
• Shutter technology in which the respective perspectives for the two eyes are shown one after the other.
• the channeling in relation to one eye takes place via glasses that connect the perspectives with the correct eye.
• the autostereoscopic display in which no additional glasses are required. Both perspectives are presented simultaneously. The channeling is done by optical means, such as lenses or prisms. The viewer's movement is detected and fed to the respective eye via the optical means. Both perspectives are alternately contained in the LCD columns of a screen. The perspectives are fed to the right or left eye via prism masks on the screen. The three-dimensional image appears to float in front of the screen. Selection devices for image processing are also known, for example a computer mouse, via which individual pixels of the representation can be reached. However, the representation is in two-dimensional or quasi three-dimensional form, so that the selection of a pixel is difficult.
• a so-called mouse which can be moved back and forth on the desk surface by means of rollers, is used to process representations on a computer screen.
• the movement of the mouse in relation to the surface is recorded and converted into cursor movements.
• image points in a virtual, three-dimensional image space for example in an image layer lying behind them, can only be selected with difficulty.
• the three-dimensional representation is usually broken down into a two-dimensional view, so that a specific image layer can be selected.
• the invention is based on the object of specifying a method and a device for processing and changing three-dimensional representations, according to which simple selection and processing of three-dimensional image objects is possible.
• this object is achieved by the features of claim 1. Thereafter, at least a spatial sub-area of the image object is marked for processing by the selection device.
• an image object in the case of a three-dimensional representation generating an image volume, an image object can be marked in the image space itself.
• the marking is done with conventional image shield with the usual two-dimensional representation by rotating the body or changing its position and subsequent two-dimensional or quasi three-dimensional marking.
• an image object projected into the space is marked directly in the virtual image space.
• the operator can easily observe the marking of the spatial subarea either by changing the position of the viewer or by changing the position of the body.
• a computer with a vector graphic and various input and output devices supports the three-dimensional, in particular autostereoscopic, calculation, calculates the position of the marking in the image object and displays the marked partial areas of the image object in the three-dimensional representation.
• the method according to the invention can be used particularly advantageously in object detection and measurement in molecular physics, in inspection and quality assurance in the semiconductor industry, in metallurgy and in metal processing.
• the use in CAD processes in connection with object recognition, image acquisition, reconstruction, representation and measurement of image objects is also advantageous.
• the method can be used for virtual prototyping (mock up) and in the virtual workshop. Further areas of application are in microbiology the segmentation and measurement of spatial arrangements such as cell nucleus structures, chromosomes etc. and in medicine cytology, ophthalmology, neurodiagnostics and angiography.
• the marked image object is segmented in relation to other image objects represented in the image space.
• the image object is subdivided into individual sections, the computer storing the boundary areas of the sections in three-dimensional space and data for assigning a section during segmentation.
• the segmentation is the basis for the image-analytical processing of the three-dimensional representation.
• Image volumes are delimited in the three-dimensional representation and divided into sections that are accessible to different image processing processes.
• morphometry ie the segmentation and measurement of 3D image objects in three-dimensional images, is supported.
• the selection device in the virtual image space is directed at the image object or is guided along the delimitation areas of the image object.
• the image object marked in this way can be highlighted, in particular colored or enlarged. As a result, the selected spatial sub-area can be better recognized in the three-dimensional representation.
• the marked spatial sub-area of the image object is measured.
• three-dimensional representations can be generated from this.
• at least two image points or preferably partial areas of an image object must be marked.
• the individual information of the known two-dimensional or quasi-three-dimensional representations can be used in the three-dimensional representation and can be measured in the image space with little effort using the autostereoscopic representation.
• the accuracy during the measurement is advantageously increased and the analysis or diagnosis process is objectified.
• the marked partial area of the image object can preferably be cut out of the three-dimensional representation and displayed separately.
• the image object can be automatically segmented and / or measured by processing procedures on the computer; For this purpose, data are preferably stored in a storage device of the computer in order to generate reference objects. Based on the reference objects, an automatic segmentation and measurement of relevant image object areas is then carried out in the three-dimensional image space.
• Image analysis, cognitive and knowledge-based systems, in particular also neural networks, can be used in the segmentation, measurement or selection of a specific image object in the display.
• the automatic segmentation for example the segmentation of a liver within a medical, three-dimensional representation of a patient performance, the analysis / diagnosis process can be objectified.
• the manual segmentation or measurement of relevant sub-areas of an image object in the three-dimensional image space can be used to prepare the automated process or for verification.
• a preselection can be made in the three-dimensional representation by processing procedures of the computer and manual post-processing can be carried out using the selection device or using the keyboard of the computer.
• the computer can provide processing procedures such as preprocessing by filtering, image data generation or image data compression.
• the processing procedures can also provide for cognitive modeling, calibration or morphometric evaluation of the three-dimensional representation.
• the three-dimensional representation can be changed by the processing procedures until there is a satisfactory, formalized representation for the special knowledge of an expert, for example on the basis of output signals from neural networks or knowledge-based systems.
• basic processing procedures for processing the image objects and for the application of the method and specific analysis, diagnosis, calibration and interface procedures, in particular on the computer are provided.
• the core of the process is the coupling of a process supported by autostereoscopic visualization, which is preprocessed by basic processing procedures such as alignment, filtering, deconvolution, bit mapping, etc.
• Specific processing procedures enable 3D manipulation, manual segmentation, measurement and morphometric evaluation.
• a dialog-oriented, interactive coupling of manual and automatic sub-processes is possible via a menu guidance shown in the three-dimensional representation in order to create an analysis and diagnostic model with sufficient performance.
• the three-dimensional representation is carried out autostereoscopically, the imaging means being a screen in front of which the three-dimensional representation appears to float in the image space.
• the selection device is a 3D pointer or a stylus, which can be used to display a specific pixel in the virtual image space and to be selected using a button on the stylus.
• the method is used in the diagnosis of brain tumors, epilepsy or similar diseases in medicine.
• the head of a patient is analyzed using computer tomography, magnetic spin tomography or X-ray methods and the results are used for the three-dimensional display.
• tumors can be automatically located and measured by means of knowledge-based systems via their shading in the X-ray images.
• interesting image objects i.e. Areas in which a tumor is suspected are segmented, cut out and enlarged with foreign colors. In this way, the medical diagnosis can be significantly improved.
• a tomogram is created by means of positron emission tomography (PET) and a three-dimensional representation is generated therefrom.
• PET positron emission tomography
• the reconstructed image is displayed autostereoscopically to assess the topology of the tumor.
• the object areas relevant for the test are segmented manually, the morphological features determined and measured. It also creates reference objects for a learning set as part of a cognitive model of the analysis or diagnosis problem.
• the assessment is based on the previously determined diagnostic model. This includes the automatic segmentation and measurement of relevant sub-areas in the image object.
• Interactive can be done via a manual sub-process, the diagnostic model, for example with the aid of a neural network, is determined and updated to keep pace.
• a third embodiment of the method according to the invention relates to quality assurance in the production of electronic devices.
• the usual test criteria are the solder distribution, solder quantity, component positioning and errors in the solder volume.
• An autostereoscopic, three-dimensional representation floating in the image space in front of the screen is generated via a screen connected to a computer.
• the 3D graphics are vector graphics, which means that the solder joints can be displayed with high resolution.
• the three-dimensional representation is segmented and measured.
• the three-dimensional image determined from x-rays or tomography is displayed autostereoscopically.
• Measured picture objects can be stored as reference objects for learning quantities in the memory of the computer.
• input data for neural networks can be determined with the aim of automatic segmentation or measurement on the basis of a diagnostic model, for example a combination of knowledge-based systems and neural networks.
• the automatic segmentation and / or measurement can be interactively connected to the manual segmentation and / or measurement. It is also possible to create artificial picture objects, e.g. a desired solder distribution to draw with the pen as reference objects in the three-dimensional image space.
• the object is achieved according to the invention by the features of claim 17.
• the pointing device can then be moved freely in the image space and has a transmission means for transmitting its position, the computer assigning a pixel of the three-dimensional representation to the pointing device according to its position.
• An object can be selected in the virtual image space by means of the pointing device.
• the pointing device is not guided on a two-dimensional surface, but can be moved freely in space by the operator.
• the position of the pointing device is transmitted to the computer via a transmission means.
• the movement in space takes place, for example, by the operator's hand in front of the body.
• a change in the position of the pointing device in space leads to the displacement of a virtual cursor in the virtual image space.
• the method can be advantageously applied to stereoscopic methods using a pair of viewing glasses, such as, for example, the anaglyph technique, in which two perspectives are produced for one eye each in red and green, or the shutter technique, in which the perspectives are in succession are shown and can be switched on through the viewing glasses.
• a pair of viewing glasses such as, for example, the anaglyph technique, in which two perspectives are produced for one eye each in red and green, or the shutter technique, in which the perspectives are in succession are shown and can be switched on through the viewing glasses.
• the invention can be applied particularly advantageously to autostereoscopic three-dimensional representations in which no additional viewing glasses are required.
• the perspectives for both eyes are shown at the same time and optical means create a perspective for the corresponding eye.
• the observer movement is recorded and both perspectives are conveyed to the individual eye, for example by prisms.
• a pixel in the three-dimensional object shown is selected, processed and changed via the movement of the pointing device.
• the computer determines the associated pixels of the three-dimensional representation.
• the computer assigns to the pointing device exactly the pixel to which the pointing device in the virtual image space is oriented.
• the pointing device can be movable in the virtual image space, ie within the three-dimensional display. In this way, a three-dimensional object can be selected with the pointing device.
• the position of the pointing device is transmitted by the transmission means to the computer, which calculates the corresponding image point of the three-dimensional representation and displays it in particular via a cursor in the three-dimensional object.
• the pointing device has a transmission means.
• the data is preferably transmitted wirelessly via a transmitting and / or receiving unit.
• position and control signals are transmitted via a radio or infrared transmission link from the pointing device to the receiving unit on the computer.
• the position in space can, however, also be ascertainable via a metal detector which detects a metal head of the pointing device.
• the transmission means could also be a simple connecting cable between the pointing device and the computer.
• the pointing device is preferably designed as a pointing pen, in particular with a tip that can be located in space.
• the pointer can have the size and shape of a ballpoint pen, the pointer itself being designed as an antenna for the transmitting and / or receiving unit.
• the pointing device can easily be guided into a position in the virtual image space, so that a three-dimensional object can be processed and changed.
• the pointing device can have at least one selection means, in particular a button, for selecting a pixel and / or for starting processing.
• the start or end point of a movement in the virtual image space or the selection of a specific three-dimensional object can be determined via the selection means.
• the selection means can also be used to select certain processing procedures from a menu guidance that is virtually displayed in the three-dimensional image space via display windows.
• the pointing device can be assigned one of different processing procedures as a virtual tool head in order to process the three-dimensional representation.
• the processing procedures include, for example, modeling, such as increasing or deepening contours, applying engravings or textures, segmenting, filtering, hatching, coloring or drawing.
• Either one of the processing procedures of the pointing device is assigned in advance via the computer, or the processing procedure is selected during processing from menu windows which are virtually represented in space.
• the processing procedure is started, for example using a button.
• the button can be used by continuous operation to create lines or for modeling.
• the change in the three-dimensional representation by the processing procedure linked to the pointing device can be calculated in the computer and, in particular in real time, can be displayed on the screen.
• the changes made by the pointing device are displayed immediately, so that the operator has feedback about the processing of the three-dimensional representations.
• the changes in the three-dimensional representations can be saved on the computer.
• the processing procedure is preferably assigned to the pointing device via the keyboard or via a menu guidance displayed in the three-dimensional representation, e.g. via pull-down menus of a user interface.
• a symbol can be displayed in the virtual image space that corresponds to the position of the pointing device. That the symbol can be displayed on the pixel assigned to the pointing device.
• the symbol is preferably a blinking cursor, a hand or a tool.
• a scalpel for example, can be displayed for medical use.
• the positioning of the pointing device can preferably be supported by a virtual scale in the three-dimensional image space or other measuring functions. This can be a metric or logarithmic scale that can be displayed in three-dimensional image space.
• icons or marking elements can also be attached to the display by the pointing device. Both the measurement functions and the processing procedures can be stored in a library of the computer.
• the three-dimensional display on the screen is preferably autostereoscopic. This requires a 3D screen that creates a perspective for one eye and reworks it using optical means for one of the two eyes. This creates a three-dimensional representation that appears to float in front of the screen in the image space. Further advantageous embodiments of the invention result from the subclaims and the description. Two advantageous embodiments of the invention are described below.
• the inventive device has a computer, an autostereoscopic screen and a 3D pointing device.
• the tip of the pointing device contains an infrared transmitter, whereby the coordinates of the tip can be determined in the image space and can be calculated by the computer.
• the pointing device is assigned processing procedures which act as a virtual tool head on the tip of the pointing device. Certain parameters such as size, type and strength of the changes can also be adjustable.
• the processing, influenced by the computer also affects the autostereoscopically presented representations.
• the computer has a fast vector graphic which shows the changes by the pointing device in the three-dimensional representation.
• the device has a pointing device designed as a 3D pen, a stereoscopic observation system as an observation and transmission system for the position of the pen or the surgeon, an autostereoscopic 3D screen, an interface system, a processing system in the form of a computer and a calibration system.
• the 3D pen is equipped, for example, with one or more infrared diodes and is moved accordingly by the surgeon in the sensitive image area.
• the light points of the infrared diodes are located by the observation system, which has two cameras. The positions of the light spots recorded by the cameras are measured and stored in the computer by the observation system.
• a second point of light which is to be distinguished from the point of light of the 3D pen tip by temporal, spatial structure or color, is provided in order to be able to determine the position of the pen in space as well as the position. Because of the second point of light, the angle of rotation of the pin during rotary movements can also be determinable. Due to the light points, the surgeon sees the position and location of the pen in the optical reproduction through the autostereo- scopic screen and can thus orient and move the 3D pen accordingly.
• a stereo image specific to the operator can be determined by calibration and used for a display.
• the coordinates of the virtual representation of the observation system and the surgeon can be coordinated with each other using the stylus.
• the metric determined from the measurement process for example by the cameras, can be adopted for the overall display on the computer screen.
• the position of the pen and thus of the surgeon can possibly be recorded via a further stereo camera system.
• the position of the operator for the stereo image is calculated, for example, in such a way that the virtual object is stable in the room.
• the 3D pen can have an ergonomically designed, ballpoint pen-like housing that contains one to five IR transmission diodes, which are coupled to one or more mechanically stable optical fibers.
• the IR transmitter diodes are supplied externally with electricity via cables or batteries, and thus independently of electrical conductors.
• the IR transmit diodes emit continuous or modulated signals, which creates two different infrared light marks with a certain distance from each other in virtual space. These IR light marks are recorded and decoded to determine the position and location of the pen.
• the 3D pen can have two buttons, one activating the send function and the second enabling the determination of several processing functions, including calibration, status via cable, status via transmitter, unidirectional or bidirectional reception.
• the observation system is designed as a stereoscopic two or three camera system. Another camera system can also record the position, line of sight or eye relief of the surgeon. The arrangement of the camera systems defines the sensitive image space, whereby this sensitive image space can be limited by brightness and depth of field.
• CCD cameras fast frame grabbers, image memories, PCI bus-oriented acquisition and processing techniques can preferably be used for the observation system.
• the display is designed as an autostereoscopic 3D display, the objects shown appearing to the viewer in a virtual image space in front of the actual display.
• the digital interface must ensure communication between the pen and the observation system and the computer.
• This interface can be designed as an infrared interface or as a conventional interface based on electrical conductors.
• the processing system for example the computer, records the current pen position, stores it and uses it in connection with graphic representations as required.
• the calibration system has the task of ensuring the coordination and transformation of the reference systems.
• the 3D pen is captured by the camera systems in the observer area and displayed in the image area in front of the autostereoscopic screen.
• the calibration system adjusts the autostereoscopic display in relation to the real position of the 3D pen.
• the calibration system allows adjustments of the images to individual imaging properties of the surgeon.
• the surgeon is provided with a virtual 3D calibration room in which the surgeon can click on the selected points Calibration.
• This calibration room can have the shape of a Cartesian cube room.
• the sensitive image space must be scaled by detecting the position of the stylus. This can be done by showing defined scales in the coordinates and the corresponding calculation in the images to be measured by the calibration system.
• the camera image of the real pointing device is compared with the image in the virtual object space.
• a calibration object drawn in the virtual object space e.g. an octahedron can be clicked on pixels which are seen stereoscopically by the surgeon and these points can then be recorded via the observation system and can be offset using the computer for the reproduction of the stereo images.
• the position of the pointing device, as seen by the surgeon is reflected by the observation system and can be adjusted to one another by moving the pen and clicking on the calibration key if the positions match. In this way, the operator can calibrate the system himself before starting an application using a calibration key on the pointing device.
• the device according to the invention can be used for design tasks in medical diagnostics and therapy as well as in technical applications, for example CAD, as well as in artistic and manual 3D design.
• the pointing device can be used in particular for interactive tasks, for example when highlighting or correcting representations of programs running on the computer.
• the processing takes place in the autostereoscopic display, which results in a change in the image data stored on the computer.

Landscapes

• Engineering & Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Human Computer Interaction (AREA)
• Computer Graphics (AREA)
• Computer Hardware Design (AREA)
• Software Systems (AREA)
• Processing Or Creating Images (AREA)
• Position Input By Displaying (AREA)

Priority Applications (4)

Applications Claiming Priority (4)

Publications (2)

Family

ID=26041196

Family Applications (1)

Country Status (5)

Cited By (8)

Families Citing this family (28)

Family Cites Families (16)

• 1998
  • 1998-10-26 JP JP2000519376A patent/JP2001522098A/ja active Pending
  • 1998-10-26 DE DE59805342T patent/DE59805342D1/de not_active Expired - Lifetime
  • 1998-10-26 EP EP98962182A patent/EP1025520B1/de not_active Expired - Lifetime
  • 1998-10-26 WO PCT/DE1998/003138 patent/WO1999023586A2/de not_active Ceased
  • 1998-10-28 US US09/530,503 patent/US6734847B1/en not_active Expired - Fee Related

Also Published As

Similar Documents

Legal Events