JP2003099808A - Image processing apparatus, virtual operation apparatus, program for causing computer to achieve image processing function, and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing technology to achieve collision judgment on a real time basis. SOLUTION: A selection unit 26 selects a cross-sectional image based on a virtual strength inputted by a user. A detection unit 40 detects an object based on the selected cross-sectional image, an object table 56, and position information provided through an input. A response processing unit 42 generates a response to the user based on the detected result. Using the cross-sectional image rather than a three-dimensional image, the processing amount can be substantially reduced, thereby achieving collision judgment of a high-speed object.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a technique for performing image processing. In particular, the present invention relates to an image processing apparatus capable of realizing a novel image processing technique, a virtual surgery apparatus, a system capable of using these apparatuses, and a program for causing a computer to realize an image processing function.

[0002]

2. Description of the Related Art In recent years, the development of image processing technology using a computer has been remarkably remarkable in the medical field. Among image processing techniques, image diagnosis by computer tomography (CT) capable of observing internal tissues has become an indispensable part of the current clinical practice.
The cross-sectional image data obtained from the CT scanner is very useful especially in internal medicine and neurosurgery, because the internal information of the human body can be expressed as a two-dimensional image.

However, since the cross-sectional image data is merely two-dimensional information, a doctor or the like needs to imagine a three-dimensional state from the two-dimensional image in order to observe the internal tissue three-dimensionally. There is. This task requires a high degree of knowledge and experience and is very difficult for unfamiliar people. In order to make it easy for non-experts to handle this two-dimensional image data, CT images taken in succession
Currently, researches for reconstructing a three-dimensional image of a target tissue using a cross-sectional image are being actively conducted.

There are various methods for reconstructing a three-dimensional object from a cross-sectional image. For example, first, a specific body tissue is extracted as a two-dimensional object in each cross-sectional image, and the pixels forming the two-dimensional object are expanded to a unit cell having a thickness corresponding to a slice interval of a CT scan,
Construct a three-dimensional object. The three-dimensional object configured in this manner is used in a wide range of applications such as education, remote surgery, and surgery simulation.

[0005]

Surgical simulation by virtual reality is a technique useful for improving the skills of medical students, interns, etc. by allowing the user to experience a feeling similar to that of actual surgery. Currently, the above-described three-dimensional object is used for this surgery simulation, but it is difficult to perform arithmetic processing on a general purpose computer because the three-dimensional object has a huge amount of data. Therefore, large-scale workstations and other equipment are required, and it is hard to say that this system can be handled easily anywhere. Therefore, realization and popularization of a surgery simulation device that allows users to easily experience virtual surgery are highly desired from the standpoints of medical progress and improvement of medical personnel.

Therefore, the present invention provides an image processing apparatus, a virtual surgery apparatus, a system in which these apparatuses can be used, and a program for causing a computer to realize an image processing function, which can solve the above problems. The purpose is to These objects are achieved by a combination of features described in independent claims of the invention. The dependent claims define further advantageous specific examples of the present invention.

[0007]

In order to achieve the above object, an image processing apparatus according to a first embodiment of the present invention includes a display processing section for displaying an image of an appearance on a screen, and a virtual image from the appearance image. A first storage unit that stores a two-dimensional cross-sectional image existing in the depth direction, an input device that allows a user to input an operation, an input receiving unit that receives an input from the input device, and a cross-sectional image is selected based on the input. And a response processing unit that generates a response to an input based on the selected cross-sectional image. Since this image processing apparatus generates a response to an input by using a two-dimensional cross-sectional image, it can reduce the amount of data to be handled and realize a high-speed arithmetic processing as compared with the conventional image processing technique using a three-dimensional image. Is possible.

The first storage section may store a plurality of cross-sectional images obtained by slicing a virtual three-dimensional object in a predetermined direction. The predetermined direction is a direction perpendicular to the depth direction set for the three-dimensional object, and therefore, there are a plurality of cross-sectional images in the depth direction. Here, at least one cross-sectional image preferably has a colored region of interest. The image processing apparatus further includes a detection unit that detects a color at a position designated by input in the selected cross-sectional image, and the response processing unit generates a response based on the color detected by the detection unit. May be. The image processing apparatus may further include a table in which the color of the region of interest and the response are associated with each other, and the response processing unit may generate the response based on the detected color and the table. .

The detecting section detects an object virtually existing in the depth direction by the detected color. This object may be visible in the appearance image or may not be included in the appearance image. The depth direction is, for example, a direction that is perceived by the user as depth or depth when the appearance image is three-dimensionally considered as a three-dimensional model. It may be the direction of the virtual force to be given or a direction close to it. The input reception unit detects the magnitude of the virtual force applied by the input, and the selection unit can select the cross-sectional image based on the magnitude of the detected force. The input receiving unit may calculate virtual depth information for the appearance image from the detected virtual force, and the selection unit may select the cross-sectional image based on the depth information. The depth information may be distance data in the depth direction from the reference position. The cross-sectional image preferably has distance data from the reference plane. The image processing apparatus may further include a second storage unit that stores the two-dimensional appearance image data, and the display processing unit may display the two-dimensional appearance image data on the screen as an appearance image. By generating the appearance image from the two-dimensional data,
It is possible to significantly reduce the amount of data processing in the image processing apparatus as compared with the related art.

The first storage unit stores a plurality of two-dimensional sectional images in M
A conversion function or table for converting the time information included in the MPEG image data into virtual depth information may be stored together in the PEG format. MPEG
By storing the data in the format, it is possible to efficiently compress the data amount of the cross-sectional image, and by associating the time information with the virtual depth information, the selection unit can utilize the MPEG format data format while Allows selection of cross-sectional images.

A virtual surgery apparatus according to a second aspect of the present invention is a display processing unit for displaying an external image of a human body on a screen, and a storage unit for storing a plurality of sectional images of the human body including at least one internal tissue. An input device that allows a user to perform an operation input on the appearance image, an input reception unit that receives an input from the input device, and that detects the magnitude or direction of the virtual force expressed by the input,
Based on the magnitude or direction of the detected force, a selection unit that selects a cross-sectional image, and based on the selected cross-sectional image,
And a response processing unit that generates a response to an input. This virtual surgery device allows a user to perform virtual surgery on an external image of a human body displayed on a screen. Since the two-dimensional cross-sectional image is used, the amount of data to be processed can be significantly reduced as compared with the related art, and it becomes possible to generate a response such as a warning, an instruction, or an impression to the virtual surgery of the user at high speed.

The cross-sectional image may have a region of interest in which a predetermined internal tissue is colored. The virtual surgery apparatus further includes a detection unit that detects the color of the position designated by the input in the selected cross-sectional image, and the response processing unit generates a response based on the detection result of the detection unit. be able to. The detection unit, depending on the detected color,
Internal tissue may be detected.

A program according to the third aspect of the present invention is
A function of displaying an appearance image on a screen of a computer, a function of receiving an input from an input device, and a storage unit that stores a two-dimensional cross-sectional image existing in a virtual depth direction from the appearance image based on the input. A function of selecting a cross-sectional image and a function of generating a response to an input based on the selected cross-sectional image are realized.

An image processing system according to a fourth aspect of the present invention comprises an image distribution device and a terminal, and the image distribution device comprises:
The terminal has a first storage unit that stores an appearance image, a second storage unit that stores a two-dimensional cross-sectional image existing in a virtual depth direction from the appearance image, and a transmission unit that transmits the appearance image. ,
A receiving unit that receives the appearance image, a display device that displays the appearance image, an input device that performs an operation input on the appearance image, an input receiving unit that receives an input from the input device, and a transmission that transmits information related to the input. Part and
Further, the image distribution device, based on the information about the input,
It has a selection part which selects a section image, and a response processing part which generates a response to an input using the selected section image.

In the image processing apparatus according to the fifth aspect of the present invention, a display processing unit for displaying an external image on a screen and brightness information of each pixel are set in association with a distance from a virtual reference plane. A storage unit that stores a two-dimensional image including the object, an input device that allows a user to input an operation to the appearance image, and an input device that receives an input and generates a virtual force represented by the input. An input receiving unit that detects a position to be applied and calculates virtual depth information corresponding to the magnitude or direction of the force;
In the two-dimensional image, a detection unit that determines the contact between the object in the two-dimensional image and the virtual force represented by the input by using the brightness information and the depth information of the pixel corresponding to the position. Prepare By using a two-dimensional image in which pixel brightness information is associated with virtual depth information, it is possible to realize three-dimensional hit determination with a two-dimensional image with a small amount of data, and also to visually It is possible to use a meaningful image for the judgment. This two-dimensional image may have a colored region of interest, and it may be possible to make a hit determination using color information as well as brightness information.

The above summary of the invention does not enumerate all the necessary features of the present invention, and a combination of these feature groups can also be an invention. Further, the expression of the present invention converted between the device, the method, the system, and the computer program is also effective as an aspect of the present invention.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the scope of claims and are described in the embodiments. Not all of the combinations of features described above are essential for the solution of the invention. Further, the devices and systems according to the following embodiments are realized by a CPU, a memory, a program loaded in a memory, and the like of an arbitrary computer in terms of hardware components. It depicts the functional blocks realized by. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms only by hardware, only software, or a combination thereof, and a program that causes a computer to realize these functions, and A computer-readable recording medium recording a program is also included in the following embodiments.

FIG. 1 is a block diagram of an image processing apparatus 100 according to the first embodiment of the present invention. The image processing device 100 may be a general-purpose computer that performs image processing, and may be a medical diagnosis / training device, a game device, a DVD reproducing device, a decoder for receiving broadcasts, a set top box, a PDA terminal, or a mobile phone. In short, any device that can realize the image processing function, such as a mobile terminal including the device, may be used. Hereinafter, as a representative of these devices, the case where the image processing apparatus 100 is used as a virtual surgical operation apparatus that allows a user to experience a virtual reality operation by computer simulation will be described as an example.

The image processing apparatus 100 includes an input receiving section 24,
The selection unit 26, the display processing unit 30, the detection unit 40, the response processing unit 42, the speaker 44, and the storage device 50 are provided. The display device 20 or the input device 22 is the image processing device 1.
00, or may be present as an external device. The display device 20 has a screen that displays an image, and the user can perform various input operations while looking at the appearance image displayed on the screen. The appearance image includes, for example, an image of the head of a human body on which virtual surgery is performed. The input device 22 enables a user to input an operation to the displayed appearance image. The input device 22 may be a pointing device, a glove-type device, an input device with a reaction force, or the like, in short, any input interface capable of giving desired input on the computer to the appearance image. In particular, in order to realize an environment close to reality, the input device 22 transmits a force virtually given by the user to the appearance image to the input reception unit 24 and returns a reaction force to the force to the user. It is preferable that the input interface is simple.

The storage device 50 includes a display image data storage unit 5
2, including a cross-sectional image data storage unit 54 and an object table 56. The display image data storage unit 52 stores appearance image data to be displayed on the display device 20.
This image data may be three-dimensional appearance image data, but may be two-dimensional image data in order to reduce the amount of data to be processed. It is preferable that the display processing unit 30 appropriately change the appearance image based on the input from the user.

On the other hand, the section image data storage section 54 stores a plurality of non-display section image data for detecting an object. The “non-display” image means the display device 2
At 0, it means that the image is not to be displayed to the user, and does not mean that the cross-sectional image itself cannot be displayed. That is, in the present embodiment, the non-display image may have a data format that can be displayed, but is used in the sense that the non-display image is not displayed on the display device 20 in a normal use state by the user. The cross-sectional image may be a slice of an object itself including internal tissue in a predetermined direction, such as a CT scan image of a human body, and the CT scan image is reconstructed into a virtual three-dimensional object. It may be configured to slice the three-dimensional object in a predetermined direction.

At least one of the plurality of sectional images is
It has a region of interest in which object identification information is added to the included object. Here, the object identification information is information for identifying an included object, and may be, for example, information in which a two-dimensionally expressed object is colored in a predetermined manner. A plurality of cross-sectional images are prepared in the virtual depth direction of the appearance image. When the user gives an input instruction regarding the depth, a cross-sectional image conforming to the instruction is selected from the cross-sectional image data storage unit 54 and used for detecting the object. In this way, the cross-sectional image is used for determining whether or not the user input indicates the region of interest, that is, so-called hit determination.

The object table 56 stores the object identification information, the object name, and the response to the user in association with each other. This response includes a warning to the user and a message telling that the operation is good.

First, the user inputs an operation to the appearance image displayed on the display device 20 via the input device 22. When the image processing apparatus 100 is used as a virtual surgical apparatus, an external image showing a part of the human body is displayed on the display device 20, and a user such as a medical student or an intern operates the input device 22 to display it on the screen. With respect to the displayed appearance image, an incision is made on the affected area with a virtual scalpel interlocking with the input device 22.

The input receiving section 24 receives an input from the input device 22. In this embodiment, the input receiving unit 24 determines the magnitude of the virtual force applied to the virtual object based on the input from the input device 22, the position in the appearance image to which the force is applied, Further, the direction of the force is detected, and the depth at which the knife is incised is calculated and obtained. This depth can be determined by the magnitude and / or the direction of the force. The magnitude of the virtual force is the magnitude of the force applied to the input device 22 by the user, the number of times the input button provided on the input device 22 is pressed,
It is calculated by the combination of push buttons. The virtual force direction is determined by the user using the input device 22.
May be determined by the direction of the force applied to the screen, or may be set in advance, for example, in a direction perpendicular to the screen. The calculated depth information is displayed by the display processing unit 30.
The display processing unit 30 can select the appearance image stored in the display image data storage unit 52 based on the depth information and display it on the display device 20.
In addition, the display processing unit 30 preferably moves the virtual knife on the screen of the display device 20 in synchronization with the input.

The selection unit 26 receives the depth information of the virtual knife and the position information in the appearance image in which the virtual knife is arranged from the input reception unit 24. The selection unit 26 selects the cross-sectional image stored in the cross-sectional image data storage unit 54 based on the depth information. The selected cross-sectional image is supplied to the detection unit 40. At this time, the selection unit 26 supplies the position information of the virtual knife to the detection unit 40. This location information is
It may be directly sent from the input reception unit 24 to the detection unit 40.

The detecting section 40 detects the object identification information at the position designated by the position information of the virtual knife in the selected cross-sectional image. If the image size of the appearance image and the cross-sectional image are the same, the position information of the virtual knife can be used as position information on the cross-sectional image. If the image size is different, the position information of the virtual knife is used as the position on the cross-sectional image. Convert the information and detect the object identification information. In this example, the color information at the specified position is detected as the object identification information. When the designated position is the region of interest, the object is identified by the object table 56 based on the color information. The object table 56 includes a response to the user, and when the detection unit 40 detects an object, the response processing unit 42.
Receives it and generates a response based on the object table 56. This response may be, for example, a warning such as "The scalpel is overpowered", "I have injured the brain tissue", or a message such as "The surgery is going well." The response processing unit 42 sends this warning or message to the display device 2 via the display processing unit 30.
0 may be displayed, or the user may be notified as audio information from the speaker 44.

FIG. 2 is an example of an appearance image showing the head displayed on the display device 20. In FIG. 2A, a region for inserting a virtual knife is shown by a dotted line. A user such as a medical student operates the virtual scalpel on the appearance image via the input device 22 to make an incision in the head. Figure 2
(B) shows a state in which the scalp is incised in a semicircle using the virtual knife 46. In the figure, the hatched region indicates the surface tissue of the skull inside the scalp. Figure 2
In the process from (a) to FIG. 2 (b), it is preferable to develop the appearance image in real time according to the movement of the virtual knife 46 by the user.

FIG. 3 shows a cross-sectional image of the head. Cross section image
Is the base with a reference depth of 0 mm on the tangent plane of the top of the three-dimensional head.
A tangential plane set from the top of the head to the inside of the head
Included in the head in the direction along the normal of (depth direction)
It is an image obtained by slicing a skull that is parallel to the reference plane.
It In the depth direction, 0 mm <d₁mm <d _Two
mm <d_ThreeThere is a relationship of mm. Figure 3 (a) shows that
The cross-sectional image in depth is shown. In FIG. 3 (a), the person
Shows the body objects, the internal tissues of the skull
There is no region of interest.

FIG. 3B shows a sectional image at a depth of d ₁ mm. In the figure, the area where multiple points are shown is
It is colored blue to show that the virtual scalpel is only a short distance away from the skull. Referring to FIG.
When the detection unit 40 detects that the position specified by the virtual knife is blue in this cross-sectional image, the response processing unit 42 issues a warning instructing that the incision should not be made any deeper. In this example, the blue region does not represent the skull itself, but shows the internal tissue near the skull, that is, not only the skull itself but also its surrounding tissues are treated as one object (skull). There is. In another example, a skull,
It goes without saying that the surrounding tissues may be treated as separate objects.

FIG. 3C shows a sectional image at a depth of d ₂ mm. In the figure, the shaded area is colored green, indicating that the virtual knife has reached the skull. When the detection unit 40 detects that the position specified by the virtual knife in this cross-sectional image is green, the response processing unit 42 immediately issues a warning to stop the incision.

FIG. 3D shows a sectional image at a depth of d ₃ mm. In the figure, the filled area is colored red, which indicates that the skull is severely damaged when the virtual knife is inserted up to this area. When it is detected that the position specified by the virtual knife is red in this cross-sectional image, the response processing unit 42
Issues a warning that the patient has been fatally injured. In this way, by changing the color to be added to the same object according to its importance, it becomes possible to give a user a useful instruction according to the situation.
Further, in order to realize a surgery simulation close to reality, the magnitude of the reaction force applied to the input device 22 may be changed according to the color.

FIG. 4 shows an example of the object table 56 associated with the cross-sectional image shown in FIG. The object table 56 holds the object identification information, the object name, and the response to the input from the user in association with each other. In this example, the object name is a skull, and the response to the user is recorded in association with the color information that is the object identification information. Specifically, when the color of the region of interest is blue, warning 1 is associated, when it is green, warning 2 is associated, and when it is red, warning 3 is associated.

The image processing apparatus 100 has been described above by taking an incision operation on the head as an example, but other important internal tissues, for example,
It is also possible to color the nervous system and blood vessels as regions of interest, prepare cross-sectional images of various parts of the human body, and configure it so as to be compatible with various surgical simulations. In the example described above, the same object is color-coded, but when there are a plurality of objects, it is preferable to apply different coloring to each object. Image processing apparatus 1 according to the present embodiment
00 detects an object using a two-dimensional cross-sectional image in which the amount of data is significantly reduced as compared with the conventional technique, and therefore does not pose a problem of holding data for surgery simulation for many parts of the human body, and can perform various operations. It is possible to support virtual surgery. In addition, since the amount of data to be handled is small, it is possible to generate a response at high speed without spending time for calculation.

FIG. 5 is an explanatory diagram for explaining an example in which the image processing technique described in the first embodiment is applied to a game. Here, it is assumed that the game character searches for a treasure hunt for an object embedded in the ground with a hoe. FIG. 5A shows a game progress screen. This character digs an appropriate place on the screen according to an instruction from the user. Here, a state in which the character is digging at the point X is shown. The depth dug by the character is calculated by the input reception unit 24 as described with reference to FIG.

FIG. 5B is a three-dimensional view of an object (jar) virtually embedded in the ground. This object is sliced on three planes, that is, an A plane, a B plane, and a C plane, and the slice image is held as a slice image in the slice image data storage unit 54 (see FIG. 1). Each of these cross-sectional images has depth information from the ground surface.

FIG. 5C is an explanatory diagram for explaining the positional relationship between the ground surface and an object in the ground. The point X is located right above the object in the depth direction, and the point Y is located at the left end of the object.

FIG. 6 shows a cross-sectional image of an object.
Specifically, FIG. 6A shows a sectional image on the A surface, FIG. 6B shows a sectional image on the B surface, and FIG. 6C shows a sectional image on the C surface. An area surrounded by a circle is an area of interest indicating that an object exists in the plane and is filled with the same color.

As shown in the figure, when the character is digging at the point X, when the character is dug down to the depth of the plane A, the hoe
Hit the jar that is the object. It is determined from the fact that the input position X of the user is included in the region of interest on the A surface. On the other hand, when the character digs at the point Y, the character hits the jar only when it is dug down to the depth of the C plane. Therefore, if you dig only to the B side, you cannot find the urn. In this way, by using the cross-sectional image existing in the depth direction for the object detection processing,
It is possible to add to the game the element of luck where you should dig in to find an object quickly, and it is possible to give the game new fun.

Although the treasure hunting game has been described with reference to FIGS. 5 and 6, the image processing technique in this embodiment can be applied to other games including a fighting game and the like. In this case, the position where the attack from the opponent hits and the virtual force at that time are detected, and the damage is derived based on the distance in the depth direction from the position where the hit. A cross-sectional image having a region of interest in the depth direction is prepared in advance, and based on the cross-sectional image selected according to the distance, it is determined whether the damage received by the attack is fatal or mild. In the current 3D fighting game, there is a reality that the real-time arithmetic processing is difficult because of the huge amount of calculation. By using the image processing technique according to the present embodiment, the damage of an attack can be detected using a two-dimensional image, so that the calculation time is short and real-time high-speed processing can be realized.

It goes without saying that the image processing technique according to this embodiment can be used alone as an image processing device such as a virtual surgery device or a game device, but it is understood by those skilled in the art that it can be used in combination with a three-dimensional image processing technique. It is about to be done. For example, a 3D object is used for a display image, and the image processing technique according to the present embodiment is used for detecting an object or generating a response to an input. The system may be configured. Especially 3
When a two-dimensional object has a very complicated shape, it is necessary to process a huge amount of data for hitting the object. Therefore, it is very effective to use the image processing technique according to the present embodiment together. It can be said that it is a method.

FIG. 7 is a flow chart for explaining the image processing technique in this embodiment. In S10,
This flow starts. In S12, the appearance image is displayed on the display device 20, and the input by the user is awaited. When the input from the user is obtained by the input device 22, the virtual depth information and the position information given to the object by the input are detected in S14.
Then, in S16, a cross-sectional image is selected based on the depth information. Then, in S18, based on the position information, the color information of the position designated by the user in the cross-sectional image is detected, and whether the position is the region of interest, in other words, the structure corresponding to the object. Determine whether or not. If it is not the region of interest, the flow returns to S14, and the flow up to the determination (S18) is executed again. If it is the region of interest, a response to the user's input is generated in S20. After the response is generated, if the operation by the user is ended in S22, the whole flow is ended in S24. When continuing, the above flow is repeated from S14. It is preferable that the image displayed on the display device 20 is appropriately changed according to the input.

As described above, in the first embodiment, an example has been described in which the collision determination between the object and the virtual force represented by the input is realized by using the color information. In the following, as a second embodiment, an example will be described in which a two-dimensional image in which brightness information is provided for each pixel is used to realize a collision determination between an object and a virtual force.

FIG. 8 shows an example of a two-dimensional image used in the second embodiment of the present invention. This two-dimensional image shows a front view of a human face and has brightness information for each pixel. This two-dimensional image is generated by assigning brightness information to each pixel in association with the distance from the virtual reference plane. Here, the virtual reference plane is a plane parallel to this image plane, and the depth direction is a direction perpendicular to this reference plane. In this example, the brightness information of the pixel is set to darken in proportion to the distance in the depth direction,
Specifically, the brightness information is assigned such that the tip of the nose is brightest and the vicinity of the contour of the face is darkest. In order to grasp the relationship between the brightness information and the distance in the depth direction (depth information), a conversion function that uniquely associates the pixel brightness information with the depth information may be prepared, or the brightness You may prepare the table which matched information and depth information. The function for converting the brightness information and the depth information may be a linear function or a logarithmic function, or may be another function.

In the present embodiment, the face image to which the brightness information is assigned means that it is set as an object. That is, here the entire face is 1
It is set as one object. Further, in this two-dimensional image, a region of interest colored in a fan shape is set on the forehead. As described in the first embodiment, this region of interest is used for hitting determination, and in this facial image, the forehead portion is also set as an object. That is, in this two-dimensional image, one object is set on the entire face, and one object is also set on the forehead part of the face.
Since the hit determination using the region of interest has already been described in the first embodiment, hereinafter, referring to FIG.
The hit determination using the brightness information will be described.

When the hit determination is performed using the brightness information, the storage device 50 uses a two-dimensional image storage unit for storing the surface image of the face shown in FIG. 8 instead of the sectional image data storage unit 54. Prepare Since this face image has brightness information, it can visually represent the unevenness of the face, and can also be used as an appearance image to be displayed on the display device 20.

The two-dimensional image storage unit may have this two-dimensional image for each work process by the user. For example, in order to perform treatment in the brain, first, the scalp is peeled off, the skull is cut off, and the pretreatment for exposing the brain is required.
The three-dimensional image storage unit stores the image used in the process of stripping the scalp and the image used in the process of cutting off the skull as separate images. The two-dimensional image shown in FIG. 8 is used in the process of stripping the scalp. In this way, by preparing a plurality of two-dimensional images, it becomes possible to execute various surgical simulations. In another example, the display image data storage unit 52 stores two-dimensional surface image data having brightness information as an image for display, and the detection unit 40 determines the two-dimensional surface for the contact with the virtual knife. You may make it utilize an image.

Referring to FIG. 1, the input receiving section 24 receives a user input from the input device 22 and detects a virtual force represented by the input. Specifically, the position where the virtual knife operated on the screen is arranged and the magnitude or direction of the force by the virtual knife are detected. then,
Virtual depth information of the virtual knife with respect to the object is calculated and obtained based on the magnitude or direction of the force of the virtual knife.

The selection unit 26 receives the depth information of the virtual knife and the position information in the appearance image in which the virtual knife is arranged from the input reception unit 24. The selection unit 26 selects a two-dimensional image stored in a two-dimensional image storage unit (not shown) of the storage device 50 according to the work process. In the process of stripping the scalp, the image shown in FIG.
Is supplied to. At this time, the selection unit 26 supplies the position information and the depth information of the virtual knife to the detection unit 40. The information of the virtual knife may be directly sent from the input reception unit 24 to the detection unit 40.

The detecting unit 40 uses the brightness information and the depth information of the pixel at the position specified by the position information of the virtual knife in the selected two-dimensional image to determine whether the virtual knife reaches the face. Detect whether or not. As previously mentioned,
The pixel brightness information and the virtual depth information are associated with each other by a conversion function or a table. The depth information of the virtual knife is converted into the brightness information, and the pixel of the pixel assigned to the position is converted. By comparing with the brightness information, the contact between the face, which is the object, and the force applied by the virtual scalpel is determined.

Color information and brightness information are optically independent, and the hit determination using the color information in the first embodiment and the hit determination using the brightness information in the second embodiment are performed. By combining them, it becomes possible to expand the possibility of surgical simulation using a two-dimensional image. It is also easily understood by those skilled in the art that the surgery simulation can be executed using only the hit determination using the brightness information.

FIG. 9 is a block diagram showing the hardware components of the image processing apparatus 100 according to the first or second embodiment of the present invention. The image processing apparatus 100 includes a display device 20, an input device 22, a VRAM (video random access memory) 58, a CPU 60, a RAM (random access memory) 62, a hard disk 64, and a drive device 66. The display device 20 and the input device 22 denoted by the same reference numerals as those in FIG. 1 realize the functions and operations described with reference to FIG. These components are electrically connected by a signal transmission path such as a bus 68.

The hard disk 64 is a large-capacity magnetic storage device, and in the image processing device 100, it stores display (appearance) image data, cross-sectional image data, two-dimensional image data, an object table and the like. VRAM5
Reference numeral 8 denotes a memory that holds the screen content displayed on the display device 20, and the frame data of the display image is written therein. This data is converted into a display signal and the display device 20
Is output to.

The recording medium 70 has the functions of the image processing apparatus 100 described in connection with the first or second embodiment,
A program to be realized by the PU 60 is recorded. When the recording medium 70 is inserted into the drive device 66, the program is read into the RAM 62 or the hard disk 64, and the CPU 60 performs image processing by the read program. This recording medium 70 is a CD-RO.
It is a computer-readable medium such as M, DVD, FD. The recording medium 70 records display image data, cross-sectional image data, two-dimensional image data, an object table, etc., and when the recording medium 70 is inserted into the drive device 66, even if those data are written in the hard disk 64. Good.

Here, an example in which the image processing program is recorded in the recording medium 70 has been described, but in another example, the program may be transmitted from an external server regardless of whether it is wireless or wired. Good. In the hardware configuration shown in FIG. 9, the program only needs to cause the computer to realize the image processing function, and may be stored in the hard disk 64 in advance as well as being supplied from the outside. It is understood by the trader.

FIG. 10 is a block diagram of an image processing system 200 according to the third embodiment of the present invention. The image processing system 200 includes an image distribution device 210 and a terminal 240, and is configured so that the image distribution device 210 and the terminal 240 can exchange signals via the network 230. In this specification, the term "network" is used as a concept including both wired and wireless communication means. That is, the image distribution device 210 and the terminal 240 may be connected by a wire such as a telephone line, a dedicated line, or a public network, or may exchange signals wirelessly via a carrier wave. The image distribution device 210 includes a server that provides contents on the Internet or an intranet, a broadcasting station that broadcasts programs, and the like. In addition, the terminal 24
0 may be a general-purpose computer, and is capable of realizing image processing functions, such as game devices, medical diagnosis / training devices, broadcast receivers, and mobile terminals such as PDA terminals and mobile phones. It is sufficient that the device has a communication function.

The image distribution device 210 comprises a control unit 220, a storage unit 212 and a transmission / reception unit 222. Storage unit 21
Reference numeral 2 has a display image data storage unit 214 that stores data of an appearance image that is a display image, a cross-sectional image data storage unit 216 that stores data of a cross-sectional image of an object, and an object table 218. Transmitter / receiver 22
2 has a function of transmitting and receiving a data signal.

The terminal 240 includes a transmission / reception unit 232 and the image processing apparatus 100. The image processing apparatus 100 is as described in the above embodiment.
The transmission / reception unit 222 has a function of transmitting / receiving a data signal.

Upon receiving the image distribution request from the terminal 240, the image distribution device 210 reads the display image data, the cross-sectional image data, and the object table from the storage unit 212 by the control unit 220, and stores these data. It is transmitted from the transmission / reception unit 222. The terminal 240 receives the data at the transmission / reception unit 232, and the image processing apparatus 10 receives the data.
0 in the storage device. Image processing device 100
Performs the image processing described in connection with the first embodiment using the received data. Image processing device 100
May receive data for all display images, cross-sectional images and object tables at once, or may receive new image data each time the image displayed on the screen is changed. Further, for the hit determination processing in the terminal, it is preferable that the transmission / reception unit 222 transmits the object table before transmitting the display image data and the cross-sectional image data.

In the third embodiment, an example in which the image processing system 200 performs the image processing in the first embodiment has been described, but the image processing system 200 is also described.
However, those skilled in the art can easily understand that it is possible to provide a two-dimensional image data storage unit instead of the cross-sectional image data storage unit 216 to perform the image processing described in the second embodiment. This is where you can do it.

FIG. 11 is a block diagram of an image processing system 200 according to the fourth embodiment of the present invention. The image processing system 200 includes an image distribution device 210 and a terminal 240, and is configured so that the image distribution device 210 and the terminal 240 can exchange signals via the network 230.

The image distribution device 210 comprises a control unit 220, a storage unit 212, a transmission / reception unit 222, an input acquisition unit 250, a selection unit 252, a response processing unit 254 and a detection unit 256. The terminal 240 includes a transmission / reception unit 232, a display device 20, a display processing unit 30, an input device 22, an input reception unit 24, and a storage device 234. 1 or 10 are designated by the same reference numerals as those shown in FIG. 1 or 10.
The same or similar function and operation as those of the corresponding configuration are realized. Further, the functional blocks having the same names as those in FIG. 1 realize the functions and operations described in the first embodiment.

In the image distribution device 210, when the image distribution request is received from the terminal 240, the control unit 220 causes the storage unit 212 to operate.
The display image data is read from and is transmitted from the transmitting / receiving unit 222. The terminal 240 receives the display image data by the transmission / reception unit 232 and stores it in the storage device 234. The display processing unit 30 causes the display device 20 to display this display image.

The user of the terminal performs desired operation input on the image displayed on the display device 20 via the input device 22. The operation input information is received by the input receiving unit 24, and depth information for the object and position information in the cross-sectional image or the display image based on the virtual force applied by the input, the direction, and the action position of the force. To detect. The input information including the depth information and the position information is transmitted from the transmission / reception unit 232 to the image distribution device 210.

In the image distribution device 210, the transmission / reception unit 2
22 receives the input information and supplies it to the input acquisition unit 250. The selection unit 252 selects a cross-sectional image based on the depth information included in the input information. The detection unit 256 detects an object based on the selected cross-sectional image and the positional information of the input supplied from the selection unit 252. The input position information may be received from the input acquisition unit 250. The response processing unit 254 generates a response to the user based on the detection result of the detection unit 256 and the object table. This response is sent to the terminal 2 via the transmission / reception unit 222.
40. The terminal 240 displays the response on the display device 20.
, Or a speaker (not shown)
You may make it output a voice from. In this embodiment, since the image distribution apparatus 210 side performs the hit determination associated with the detection of the object, it is not necessary to transmit the cross-sectional image data to the terminal 240, and the effect of reducing the transmission data amount can be realized.

In the fourth embodiment, an example in which the image processing system 200 performs the image processing in the first embodiment has been described. However, the image processing system 200
However, those skilled in the art can easily understand that it is possible to provide a two-dimensional image data storage unit instead of the cross-sectional image data storage unit 216 to perform the image processing described in the second embodiment. This is where you can do it.

Although the present invention has been described above based on the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is understood by those skilled in the art that the above-described embodiments are mere examples, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and that such modifications are also within the scope of the present invention. This is where

For example, referring to FIG. 1, the cross-sectional image data storage unit 54 may compress a plurality of cross-sectional image data in MPEG format and store them together. Originally, the MPEG coding is characterized in that the data size is suppressed by extracting the difference between adjacent frames in the time direction, but in this modification, the cross-sectional image prepared in the depth direction does not change significantly. In particular, the MPEG compression technology that can expect high compression effect is used. At this time, it is preferable that the cross-sectional image data storage unit 54 stores a conversion function for converting the time information for synchronous reproduction included in the MPEG image data into virtual depth information. With this conversion function, it becomes possible to extract the cross-sectional image used for the hit determination from the virtual depth information obtained from the user input. Instead of the conversion function, virtual depth information and M
It goes without saying that a table in which the time information of the PEG image is associated with each other may be prepared.

[0069]

According to the present invention, it is possible to provide an apparatus and system having a highly useful image processing function, and further a program for causing a computer to realize the image processing function.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 (a) shows a region for inserting a virtual knife by a dotted line, and FIG. 2 (b) shows a state in which the scalp is incised in a semicircle using the virtual knife.

3A is a sectional image at a depth of 0 mm, FIG. 3B is a sectional image at a depth of d ₁ mm, and FIG. 3C is a sectional image at a depth of d ₂ mm. And (d) shows a cross-sectional image at a depth of d ₃ mm.

FIG. 4 shows an example of an object table associated with a cross-sectional image.

5A is an explanatory diagram for explaining an example in which an image processing technique is applied to a game, showing a progress screen of the game, and FIG. 5B is virtually embedded in the ground. It is a three-dimensional representation of an existing object (jar).

6A is a cross-sectional image taken along plane A in FIG. 5B, FIG. 6B is a cross-sectional image taken along plane B, and FIG.
The cross-sectional image in a surface is shown.

FIG. 7 is a flowchart illustrating an image processing technique according to the first embodiment of the present invention.

FIG. 8 shows an example of a two-dimensional image used in the second embodiment of the present invention.

FIG. 9 is a configuration diagram showing hardware components of the image processing apparatus according to the first or second embodiment of the present invention.

FIG. 10 is a configuration diagram of an image processing system according to a third embodiment of the present invention.

FIG. 11 is a configuration diagram of an image processing system according to a fourth embodiment of the present invention.

[Explanation of symbols]

20 ... Display device, 22 ... Input device, 24 ...
..Input receiving unit, 26 ... Selecting unit, 30 ... Display processing unit, 40 ... Detecting unit, 42 ... Response processing unit, 50
... Storage device, 100 ... Image processing device

Continued front page    F term (reference) 4C093 AA22 AA26 CA29 FF28 FF42                       FG01                 5B050 AA02 BA03 BA09 CA07 DA02                       EA19 EA28 FA02 FA17                 5B087 AA07 AB12 CC26 DD03 DE03                       DE07

Claims (18)

[Claims] 1. A display processing unit for displaying an appearance image on a screen, a first storage unit for storing a two-dimensional cross-sectional image existing in a virtual depth direction from the appearance image, and a user's operation input. An input device, an input reception unit that receives an input from the input device, a selection unit that selects a cross-sectional image based on the input, and a response processing unit that generates a response to the input based on the selected cross-sectional image. An image processing apparatus comprising: 2. The image processing apparatus according to claim 1, wherein the first storage unit stores a plurality of cross-sectional images obtained by slicing a virtual three-dimensional object in a predetermined direction. 3. The image processing apparatus according to claim 1, wherein at least one cross-sectional image has a colored region of interest. 4. A detection unit that detects a color at a position designated by an input in the selected cross-sectional image is further provided, and the response processing unit generates a response based on the color detected by the detection unit. The image processing apparatus according to claim 3, wherein 5. The image according to claim 4, further comprising a table in which a color of a region of interest and a response are associated with each other, and the response processing unit generates a response based on the table. Processing equipment. 6. The image processing apparatus according to claim 4, wherein the detection unit detects an object that virtually exists in the depth direction based on the detected color. 7. The input receiving unit detects the magnitude of a virtual force given by the input, and the selecting unit selects the cross-sectional image based on the detected magnitude of the force. The image processing apparatus according to any one of claims 1 to 6, which is characterized in that. 8. The input receiving unit detects a magnitude of a virtual force given by an input to calculate virtual depth information for an appearance image, and the selecting unit is based on the depth information. The image processing apparatus according to claim 1, wherein the cross-sectional image is selected. 9. The second storage unit for storing two-dimensional appearance image data is further provided, and the display processing unit displays the two-dimensional appearance image data on a screen as an appearance image.
The image processing device according to any one of 1 to 8. 10. The conversion function for collectively storing a plurality of two-dimensional sectional images in MPEG format in the first storage unit, and further for converting time information included in MPEG image data into virtual depth information. Alternatively, the image processing apparatus according to any one of claims 1 to 9, which stores a table. 11. A display processing unit for displaying an appearance image of a human body on a screen, a storage unit for storing a plurality of sectional images of the human body including at least one internal tissue, and a user's operation input for the appearance image. Based on the detected force magnitude or direction, and an input accepting unit that accepts an input from the input device and detects the magnitude or direction of the virtual force represented by the input. A virtual surgical apparatus, comprising: a selection unit that selects a cross-sectional image; and a response processing unit that generates a response to an input based on the selected cross-sectional image. 12. The virtual surgical apparatus according to claim 11, wherein the cross-sectional image has a region of interest in which a predetermined internal tissue is colored. 13. A detection unit for detecting a color of a position designated by an input in the selected cross-sectional image is further provided, wherein the response processing unit generates a response based on a detection result of the detection unit. The virtual surgical device according to claim 12, characterized in that 14. The virtual surgical apparatus according to claim 13, wherein the detection unit detects the internal tissue based on the detected color. 15. A function of displaying an appearance image on a screen of a computer, a function of receiving an input from an input device, and a two-dimensional cross-sectional image existing in a virtual depth direction from the appearance image based on the input. A program for realizing a function of selecting a cross-sectional image from a storage unit that stores and a function of generating a response to an input based on the selected cross-sectional image. 16. An image processing system comprising an image distribution device and a terminal, wherein the image distribution device includes: a first storage unit for storing an appearance image; and 2 existing in a virtual depth direction from the appearance image. A second storage unit that stores a three-dimensional cross-sectional image; a transmission unit that transmits the appearance image; the terminal that receives the appearance image; a display device that displays the appearance image; An input device for performing an operation input with respect to the input device, an input reception unit that receives an input by the input device, and a transmission unit that transmits information related to the input, and the image distribution device is based on the information related to the input. An image processing system comprising: a selection unit that selects a cross-sectional image; and a response processing unit that generates a response to an input using the selected cross-sectional image. 17. A display processing unit for displaying an appearance image on a screen, and a storage for storing a two-dimensional image including an object in which brightness information of each pixel is set in association with a distance from a virtual reference plane. Section, an input device for inputting an operation input by a user to the appearance image, and an input received by the input device, detecting a position to which a virtual force expressed by the input is applied, and detecting the force. An input receiving unit that calculates virtual depth information corresponding to a size or a direction; and, in the two-dimensional image, a two-dimensional image using the brightness information of the pixel corresponding to the position and the depth information. An image processing apparatus, comprising: an object inside and a detection unit that determines a contact with a virtual force represented by an input. 18. The image processing apparatus according to claim 17, wherein the two-dimensional image has a colored region of interest.