NL2024819B1

NL2024819B1 - Method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation

Info

Publication number: NL2024819B1
Application number: NL2024819A
Authority: NL
Inventors: Wu Haibin; Zhang Jitao; Chen Yinsheng; Wang Aili; Wang Beiyi; Yu Xiaoyang; Zhao Yanqiao; Sun Xiaoming; Yu Shuang
Original assignee: Univ Harbin Science & Tech
Priority date: 2019-03-16
Filing date: 2020-02-03
Publication date: 2021-02-09
Also published as: CN109846550B; CN109846550A; NL2024819A

Abstract

This invention relates to a method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, and belongs to the technical field of medical science, three-dimensional imaging, digital image processing and the like. The method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation comprises tracking doctor’s visual angle, 3D modeling inner cavity and generating image sequence of patient’s body surface projection. The method according to this invention is applied in a body surface projection-based system of virtual and transparent inner cavity observation for minimally invasive surgeries, to generate surface projection-based inner cavity image, by combining the doctor’s visual angle and inner cavity 3D model, on the patient body surface of non-quadratic surface, and provide assistance to the doctor in observation of the inner cavity without impact on the doctor and the operation environment during the minimally invasive surgery.

Description

INNOTRACK20001NLCN -1-

METHOD FOR VIRTUAL AND TRANSPARENT OBSERVATION OF INNER CAVITY IN BODY

SURFACE PROJECTION DURING MINIMALLY INVASIVE OPERATION Field of invention

[0001] This invention relates to a method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, and belongs to the technical field of medical science, three-dimensional imaging, digital image processing and the like. Background of invention

[0002] Compared with traditional open surgery, the minimally invasive surgery has such advantages as small wound, light pain and quick healing, catering to modern aesthetic concepts, so it is increasingly popular and applied in partial abdominal and cranial surgery. At present, the endoscope is used as main observation method during the minimally invasive surgery. By means of a plurality of pinhole cameras and xenon lamp light source, the endoscope captures the inner cavity image after inflation and presents the image on the display for doctors to observe. At present, a high quality endoscopy image can be obtained by the most advanced endoscope, and then surgical equipment positioning technology was derived from the endoscopy image with accuracy 1mm. However, the endoscopy did not satisfy the doctor's observation requirements, as shown in Fig. 1: (1) when the image of the inner cavity is displayed on the monitor, the doctor is required to constantly move his eyes between the operative site and the display, thus affecting the continuity of the surgeon's operation and easily causing fatigue to the doctor; (2) only local observation can be achieved by using single endoscope, and if the doctor needs a large scope of observation, a plurality of the endoscopes should be used in the human body simultaneously, which not only increases the burden of the patient, but also increases the complexity and bulkiness of endoscopic cables, emitter/receiver, and interventional cannula; (3) sometimes the assistant is required to move the plurality of the endoscopes to match the doctor's observation, which brings inconvenience to the doctor's surgical operation.

[0003] For the above shortcomings, some researchers proposed a patient body surface projection method based on the augmented reality technology, that is, the image captured by the endoscope is projected onto the patient's body surface by projector, so as to assist doctors in judgment when needed, and form a virtual skin transparency effect, as shown in Fig. 2 and Fig.

3. These solutions are not satisfied with simple projection, but further explore the use of azimuth tracking device to locate the head position of the doctor so as to adjust the position of the image projected by the projector on the patient's body surface.

[0004] The above solutions are still in the trial stage, and require improvements in the following aspects: (1) Current azimuth tracking technology requires a transmitter or marker to be worn on a doctor's head, increasing the burden on the doctors; (2) The inner cavity image is a two- dimensional image, which does not reflect the three-dimensional morphology of the inner cavity;

INNOTRACK20001NLCN -2- (3) The body surface of the patient is not a plane, and the projection image on the patient's body surface is distorted, therefore, the projection image should be corrected according to the three- dimensional morphology of the patient's body surface, so as to observe the undistorted inner cavity image on the patient's body surface.

Summary of invention

[0005] According to the aspects requiring the improvement, this invention discloses a method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, comprising a doctor's visual angle tracking method, an 3D modeling inner cavity method and a patient body surface projection image sequence generation method; the method for virtual and transparent observation of inner cavity in body surface projection according to this invention is applied in a body surface projection-based virtual and transparent inner cavity observation system for minimally invasive surgeries to generate surface projection-based inner cavity image by the doctor's visual angle and inner cavity 3D model on the patient body surface of non-quadratic surface, in order to get more real virtual abdominal transparent effect and finally meet the requirements of doctors’ assisted observation of internal cavity during ordinary minimally invasive surgery without impact on the doctor and the operation environment during the minimally invasive surgery.

[0006] The purpose of this invention is achieved as below:

[0007] A method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, characterized in that, the said method comprises the following steps:

[0008] Step a: Kinect tracking doctor’s visual angle, for determining the current head position and visual angle of a doctor;

[0009] Step b: PTAM (Parallel Tracking and Mapping) 3D modeling inner cavity, for generating the 3D model capable of rotating with the said doctor’s visual angle in Step a;

[0010] Step c: generating image sequence of patient's body surface projection, projecting the said 3D model of inner cavity in Step b on the patient's body surface, according to the said doctor's visual angle in Step a.

[0011] In the method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, the said Kinect tracking doctor's visual angle in Step a comprises the following steps:

[0012] Step a1: selecting the depth data of pupil and nasal tip as a key points, establishing a mathematical model for the doctor's visual angle;

[0013] Step a2: forecasting the doctor's visual angle and correcting the trace data by Kalman filter;

[0014] Step a3: analyzing the location relationship of the doctor's head, the patient's body

INNOTRACK20001NLCN -3- surface, the patient's inner cavity, and a projector, unifying a coordinate system, and designing the parameter of the patient body surface projection system.

[0015] In the method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, the said PTAM 3D modeling inner cavity in Step b comprises the following steps:

[0016] Step b1: establishing characteristic quantity of inner cavity, comprising:

[0017] Step b11: preprocessing the image of inner cavity by distinguishing specula reflection area from blood vessel;

[0018] Step b12: detecting blood vessel with single-pixel;

[0019] Step b13: determining branch point and branch segment;

[0020] Step b2: 3D modeling inner cavity: storing 3D inner cavity data and texture images in file format Stanford PLV, comprising providing 3D node data and texture information in PTAM multiframe image, by using OpenGL and according to the doctor's visual angle, generating a rotary 3D inner cavity model which is observable at multiple angle.

[0021] In the method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, the said generating image sequence of patient's body surface projection in Step c comprises the following steps:

[0022] Step c1: calculating the inner cavity image which the doctor expects to see by a typical camera imaging model according to the doctor’s visual angle and the 3D inner cavity model.

[0023] Step c2: calculating the image sequence of body surface protection according to the inner cavity image, 3D patient body surface morphology, the doctor's visual angle and the projector location.

[0024] Advantageous effect:

[0025] This invention discloses a method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation, comprising a doctor's visual angle tracking method, an 3D modeling inner cavity method and a patient body surface projection image sequence generation method; the method for virtual and transparent observation of inner cavity in body surface projection according to this invention is applied in a body surface projection-based virtual and transparent inner cavity observation system for minimally invasive surgeries to generate surface projection-based inner cavity image by the doctor's visual angle and inner cavity 3D model on the patient body surface of non-quadratic surface, in order to get more real virtual abdominal transparent effect and finally meet the requirements of doctors' assisted observation of internal cavity during ordinary minimally invasive surgery without impact on the doctor and the operation environment during the minimally invasive surgery.

[0026] Figures attached in specification

INNOTRACK20001NLCN -4-

[0027] Fig. 1 shows schematic diagram for failure of endoscope to the observation requirement of the doctor.

[0028] Fig. 2 shows the schematic diagram for virtual transparent skin projection effect.

[0029] Fig. 3 shows the real figure of virtual transparent skin projection effect.

[0030] Fig. 4 shows the schematic diagram for body surface projection system.

[0031] Fig. 5 shows the figure of the technical route for the method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation.

[0032] Fig. 6 shows the schematic diagram for Kinect doctor's visual angle tracking method.

[0033] Fig. 7 shows the schematic diagram for PTAM 3D modeling inner cavity method.

[0034] Fig. 8 shows the inner cavity image including specular reflection and blood vessel.

[0035] Fig. 9 shows the schematic diagram for patient body surface projection image sequence generating method.

[9036] Fig. 10 shows the schematic diagram for calculation and projection of inner cavity image.

[0037] Fig. 11 shows the schematic diagram for the whole structure of an adjusting device for body surface projection in minimally invasive operation.

[0038] Fig. 12 shows the structural diagram for the detail shown in Fig. 11.

[0039] Fig. 13 shows the structural schematic diagram for the foot supporting plate shown in Fig.

12.

[0040] Fig. 14 shows the structural diagram for the fixed plate structure shown in Fig. 12.

[0041] Fig. 15 shows the structural diagram for a shaft adjusting device for body surface projection adjusting gear in minimally invasive operation.

[0042] Fig. 16 shows the structural diagram for shaft drive shown in Fig. 15.

[0043] Fig. 17 shows the structural diagram for second fixed part shown in Fig. 15.

[0044] Fig. 18 shows the schematic diagram for sample taken in Embodiment 24.

[0045] Fig. 19 shows the schematic diagram for the table setting angle in Embodiment 24.

[0046] Fig. 20 shows the simplified diagram of the structure according to Embodiment 24.

[0047] Fig. 21 shows the simplified diagram of the structure shown in Fig. 20 after adjustment.

[0048] In the figures: 1 thigh supporting plate; 2 shank supporting plate; 3 foot supporting plate; 4 base; 5 first angle sensor; 6 second angle sensor; 7 shaft adjusting part; 8 fixing pad; 1-1 haunch supporting segment; 1-2 thigh supporting segment; 1-3 first push rod; 2-1 upper connection segment; 2- 2 lower connection segment; 2-3 second push rod; 3-1 rotary table; 3-2 tiptoe splint; 3-3 heel splint; 3-4 third push rod; 3-5 first cambered surface; 3-6 second cambered surface; 3-7 first key; 3-8 second key; 4-1 moving plate; 4-2 fixed plate; 4-3 base spring; 4-4 ball wheel; 7-1 shaft drive; 7-2 first waist strip; 7-3 first fixing part; 7-4 second waist strip; 7-5 table; 7-6 second fixing part; 8-1deformable pipe; 8-2 balance pipe; 8-3 stop element; 8-4 balance bag; 7-1-1 shaft

INNOTRACK20001NLCN -5- fixing fork; 7-1-2 first shaft drive push rod; 7-1-3 second shaft drive push rod; 7-5-1 table sensor; 7-6-1 fixed barrel; 7-6-2 fixed barrel chute; 7-6-3 fixed barrel push rod. Specific embodiments

[0049] Embodiment 1

[0050] This embodiment embodies the method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation.

[0051] The method for virtual and transparent observation of inner cavity in body surface projection during minimally invasive operation according to this embodiment corresponds to the body surface projection system shown in Fig. 4, the technical route of the method is shown in Fig. 5, and the method comprises the following steps:

[0052] Step a: Kinect doctor's visual angle tracking, used to determinate the current head position and visual angle of doctor;

[0053] Step b: PTAM (Parallel Tracking and Mapping) 3D modeling inner cavity , used for generating the 3D model capable of rotating with the doctor's visual angle shown in Step a;

[0054] Step c: generating patient body surface projection image sequence, meaning the projection of the inner cavity 3D model generated in Step b on the patient's body surface according to the doctor's visual angle shown in Step a.

[0055] Embodiment 2

[0056] This embodiment embodies the Kinect doctor'’s visual angle tracking method.

[0057] The Kinect doctor's visual angle tracking method according to this embodiment may separately exist as well as further limits Embodiment 1. As shown in Fig. 8, the Kinect doctor's visual angle tracking method comprises the following steps:

[0058] Step at: selecting the depth data of pupil and nasal tip as a key points to establish a mathematical model for the doctor's visual angle;

[0059] Step a2: forecasting the doctor's visual angle and correct the trace data by Kalman filter;

[0060] Step a3: analyzing the location relationship of the doctor's head, the patient's body surface, the patient's inner cavity, and a projector, unifying a coordinate system, and designing the parameter of the patient body surface projection system.

[0061] Embodiment 3

[0062] This embodiment embodies the kalman filter.

[0063] The kalman filter according to this embodiment may separately exist as well as further limits Embodiment 2. The kalman filter is described as below:

[9084] In the one-frame Kinect data at time t, the state vector X;and observation vector Z; of the kalman filter is defined to: [DOES X= (x (1), y (1), Z (t), vx (1), vy (1), vz (1)

INNOTRACK20001NLCN -6-

[0066] Zi = (x (t), y (1), z (0)

[0067] Where, x (1), y (1), and z (1) are the three-dimensional coordinate of the center of the nasal tip, and vx (1), vy (1), vz (t) are the speed of the center of the nasal tip;

[0068] Hereby the kalman system equation is listed as below:

[0069] X1= AX + BUi+ GW,

[0070] Zu = HX + Vi

[0071] Where, A is a state matrix, B is a control matrix, which approximates to 0 without controlled quantity during face movement, G is a drive matrix, H is an observation matrix, U: is a control vector, WW is Kinect measuring error and noise of X;, and V, is observation error and noise.

[0072] Embodiment 4

[0073] This embodiment embodies of the PTAM 3D modeling inner cavity method.

[0074] The PTAM 3D modeling inner cavity method according to this embodiment may separately exist as well as further limits Embodiment 1. As shown in Fig. 7, the PTAM 3D modeling inner cavity method comprises the following steps:

[0075] Step b1: establishing characteristic quantity of inner cavity, comprising:

[0076] Step b11: preprocessing the image of inner cavity by distinguishing specula reflection area from blood vessel;

[0077] Step b12: detecting single-pixel blood vessel;

[0078] Step b13: determining branch point and branch segment.

[0079] Step b2: 3D modeling inner cavity: storing 3D inner cavity data and texture images in file format Stanford PLV, namely providing 3D node data and texture information in PTAM multiframe image and generating a rotary 3D inner cavity model which is observable at multiple angle by using OpenGL and according to the doctor's visual angle.

[0080] Embodiment 5

[0081] This embodiment embodies the method of preprocessing the image of inner cavity by distinguishing specula reflection area from blood vessel;.

[0082] The method of preprocessing the image of inner cavity by distinguishing specula reflection area from blood vessel may separately exist as well as further limits Embodiment 4. The method of preprocessing the image of inner cavity by distinguishing specula reflection area from blood vessel comprises the following steps:

[0083] Step b111: the image of inner cavity which comprising specula reflection area and blood vessel as shown in Fig. 8, selecting the green component of RGB image from the image;

[0084] Step b112: calculating the scale space L of the green component image as below:

[0085] L(x, y, 0) =G (Xx, y, @)* l(x,y)

[0086] Where G (x, y, 0) is the Gaussian function of standard deviation co and I(x,y) is the green

INNOTRACK20001NLCN -7- component image;

[0087] Step b113: calculating L per pixel by Hessian matrix H and use the calculated results as the follow-up feature detection base; the Hessian matrix is given below: ZL on

[0088] H = | 2] dydx ay?

[0089] Step b114: calculating two orthogonal feature values As and A2 of Hessian matrix H, feature vector V1 and V2; the negative feature value is bright and the positive feature value is dark; for PTAM inner cavity image, the blood vessel is dark, and background is bright, then 0 < Ay < Az;

[0090] Step b115: according to the criteria of A, Ash and 1—exp (2 (A/A2) 2) commonly used at present and the following relationship, and after consideration of the sensibility of image feature to noise, distinguish background noise, microvasculature, branch point or minute feature, blood droplet or specular reflection area.

[0091] Relationship 1: if A: is close to 0, and Az is close to 0, ground noise;

[0092] Relationship 2: if A is close to 0, and Az is close to 1, microvasculature;

[0093] Relationship 3: if A; is close to 0.5, and A; is close to 1, branch point or minute feature;

[0094] Relationship 4: if As is close to 1, and A; is close to 1, blood droplet or specular reflection area.

[0095] Embodiment 6

[0096] This embodiment embodies the single pixel blood vessel detection method.

[0097] The single pixel blood vessel detection method according to this embodiment may separately exists and further limit Embodiment 4. The single pixel blood vessel detection method comprises the following steps:

[0098] Step b121: calculating the characteristic parameter of Frangi blood vessel feature F as below:

[0099] F(x,y,0) = exp (22%) . (1 — exp (32)

[0100] Where, B and c is set threshold value;

[0101] Step 122: replacing the Frangi blood vessel characteristic quantity by the center line of blood vessel characteristic, where the center line of blood vessel characteristic is the location of the pixel with gray level of pixel with change of first difference symbol in direction V2;

[0102] Step b123: to overcome the misjudgment caused by microvascular or background noise, weighted calculation of the center line R as below:

[0103] R(x,y,0) = F(x,y,0) - sign(VI(x + uy, y + Evy, a) — sign(VI(x — euy, vy — v,,0))/2

[0104] Where (uz, v2) 7 = Va, € is pixel width.

INNOTRACK20001NLCN -8-

[0105] Embodiment 7

[0106] This embodiment embodies the branch point and branch segment determining method.

[9107] The branch point and branch segment determining method according to this embodiment may separately exists and further limit Embodiment 4. The branch point and branch segment determining method comprises the following steps:

[0108] Step b131: according to the feature that the branch point is three or four blood vessel joint points, drawing a circle by using each single pixel blood vessel feature point Ri as center of circle and radius d, where d is set according to the resolution ratio of the camera, and the distance between the camera and the target.

[0109] Step b132: Determining circumferential point set C; meets the following two conditions simultaneously:

[0110] Condition 1: the point ([XC)-KR)|<t) with gray level closing to Ri in Ci is a point of intersegment, its quantity is 3 or 4, where t is determined by the gray level of R;;.

[0111] Condition 2: gray level between points of intersegment |{C)-{R)|>t

[0112] If

[0113] Yes, Riis quasi-branch point, starting Step b133;

[0114] Not, R; is not quasi-branch point;

[0115] Step 133: constituting some adjacent R; into an adjacent quasi-branch point field, and using the sub pixel center as the branch point;

[0116] Step b134: after determining the adjacent branch point field, detecting the branch segment along the adjacent pixel from the starting point adjacent branch point field. The branch segment is full branch segment if another adjacent branch point field is found, otherwise, half branch segment.

[0117] Embodiment 8

[0118] This embodiment embodies the patient body surface protection image sequence generating method.

[0119] The patient body surface protection image sequence generating method according to this embodiment may separately exist and further limit Embodiment 1. As shown in Fig. 9, the patient body surface protection image sequence generating method comprises the following steps:

[0120] Step c1: calculating the inner cavity image which the doctor expects to see by a typical camera imaging model according to the doctor's visual angle and the 3D inner cavity model.

[0121] Step c2: calculating the body surface protection image sequence according to the inner cavity image, 3D patient body surface morphology, the doctor’s visual angle and the projector location.

[0122] Embodiment 9

INNOTRACK20001NLCN -9-

[0123] This embodiment embodies the patient body surface protection image sequence generating method.

[0124] The patient body surface protection image sequence generating method according to this embodiment may separately exist and further limit Embodiment 8. In the patient body surface protection image sequence generating method, as shown in Fig. 10, the external surface of the patient's abdominal cavity is nearly quadric, according to classical visual angle change principle, the conversion relation Tp: between the coordinate system of the projector and the coordinate system of the doctor head relative the external surface of the abdominal cavity Q may be expressed by function y, and y is 4 x 4 symmetric matrix; Q is obtained by fitting according to the could data X of Kinect point, namely XTQX = 0; for a point X on Q, the relation between the coordinate system of the projector x, and the coordinate system of the doctor's eye x. is given below:

[0125] x, = (B —eq")x. £ Vat (qq" — Qas)xc- e

[0126] Where, B and e are the rotary-translation matrix of Tp, which is obtained by standardization; Q33 and q are obtained

[9127] according to

[0128] q = [4 dl

[0129] The image coordinate system is a special case of the world coordinate system, so the function y is deemed as the relation between the projected image and the observed image, and thereout the projected image is calculated.

[0130] Embodiment 10

[0131] This embodiment embodies the patient body surface protection image sequence generating method.

[0132] The patient body surface protection image sequence generating method according to this embodiment may separately exist and further limit Embodiment 8. In the patient body surface protection image sequence generating method, as shown in Fig. 10, to increase the observation scope of the doctor, a plurality of the projectors are used for protection. Exampling body surface paint p, the image is mapped from the point p to the coordinate Xp1 and Xp2 of the projector, and the coordinate Xc of the doctor's eyes by the typical camera/projector model, spatial coordinate transformation matrix to calculate the projector projected image 1 and the projector projected image 2. Because point correspondence is used rather non-surface fitting, the overlaid coverage area of the doctor’s field of view on the patient’ s body surface and the field of view of the projector can be used to accurately calculate the projected image, which is irrelative to the complexity of the patient's body surface.

[0133] The above embodiments are the core of this invention. The method according to this

INNOTRACK20001NLCN -10 - invention is applied in a body surface projection-based virtual and transparent inner cavity observation system for minimally invasive surgeries to generate surface projection-based inner cavity image by the doctor's visual angle and inner cavity 3D model on the patient body surface of non-quadratic surface, in order to get more real virtual abdominal transparent effect and finally meet the requirements of doctors' assisted observation of internal cavity during ordinary minimally invasive surgery without impact on the doctor and the operation environment during the minimally invasive surgery. It is hoped by this technology that the doctors concentrate their visual angle upon the body of the patient furthest. However, when the doctor requires large scale observation or local observation, a good technical methods is not proposed to solve the problem in the above embodiment. Therefore, Embodiments 11-27 are used to disclose an adjusting device for body surface projection during minimally invasive operation, which is used in the body surface projection system so that the doctors can achieve the image adjustment by lower limbs without change of their visual angle and use of their hands.

[0134] Embodiment 11

[0135] This embodiment discloses an adjusting device for body surface projection during minimally invasive operation, as shown in Fig. 11, comprising a thigh supporting plate 1, a shank supporting plate 2, a foot supporting plate 3, a base 4, a first angle sensor 5 and a second angle sensor 6. A supporting structure is arranged at the upper end of the base 4, and the supporting structure is a member bar for supporting, and is connected with the thigh supporting plate 1. The thigh supporting plate is used to support the thigh of the doctor. The thigh supporting plate 1 is arranged horizontally and the shank supporting plate 2 vertically. One end of the thigh supporting plate 1 is hinged with the upper end of the shank supporting plate 2. The hinged rotatable surface comprises a first rotatable surface in the length direction of the thigh supporting plate 1 and the shank supporting plate 2. When the doctor places his or her thigh on the thigh supporting plate, the doctor may drive shank supporting plate 3 by his or her shank to swing in the front and the back of the doctor. The lower end of the shank supporting plate 2 is rotatablely connected with the foot supporting plate 3. The rotatable surface of the rotatable connection is a second rotatable surface in the plane vertical to the plate of the length direction of the shank supporting plate 2, and the foot supporting plate 3 is vertically arranged with the shank supporting plate 2. The first angle sensor 5 is arranged at the hinging location of the thigh supporting plate 1 and the shank supporting plate 2 and used to detect the rotating angle of the shank supporting plate 2 relative to the thigh supporting plate 1 in the first rotatable surface. The second angle sensor 6 is arranged at the hinging location of the shank supporting plate 2 and the foot supporting plate 3 and used to detect the rotating angle of the foot supporting plate 3 relative to the shank supporting plate 2 inthe second rotatable surface.

[0136] During the minimally invasive operation, the internal image of the patient is project on the

INNOTRACK20001NLCN -1- patient's body surface by body surface projection, the doctor performs surgery by double hands.

When it is required to adjust the projected image, the doctor drives the shank supporting plate 2 by his or her shank to rotate in the first rotatable surface, the first angle sensor 5 obtains the information on the first rotatable angle, and drives the foot supporting plate 3 by his or her foot to rotate in the second rotatable surface , the second angle sensor 6 obtains the information on the second rotatable angle, uses and enters the angle value contained in the first angle information and the second angle information as coordinate value in the coordinate system. The angle information may be used as location information, and the location information may be used as instruction representing the intent of the doctor to adjust the projected image, and the adjustment includes dragging, zooming in, and zooming out, and luminance adjustment. During the surgery, the doctor may not need an assistant in adjustment, and can complete the adjustment only by moving his or her shank and foot, and fixing his or her thigh on the thigh supporting plate 1 without impact on the operation process and the consistency operation of the doctor, and avoiding the impact of the image adjustment on the operation process.

[0137] Specifically, the adjusting device further comprises a shaft adjusting part 7. The shaft adjusting part 7 is used to adjust the location of the thigh supporting plate 1 and the shank supporting plate 2 on the first rotatable surface by the change of the surface angle between the foot sole surface and the foot supporting plate 3 when the shank supporting plate 2 swings along the articulated shaft at the upper end so that the axis of the articulated shaft coincides the rotatable virtual axis of the thigh and the shank.

[0138] Due to different size of the doctors legs, after the leg is fixed with the device, the virtual axis of the rotation of the thigh and the shank is different from the axis of the articulated shaft of the thigh supporting plate 1and the shank supporting plate 2. When the virtual axis does not coincide with the articulated shaft, during rotation of the shank with the shank supporting plate 2, the shank has upward and downward movement trend relative to the foot supporting plate 3 in the length direction of the shank to fixe the tiptoe at the foot supporting plate 3. There is gap between the foot sole surface and the foot supporting plate 3, the foot sole surface changes relative to the upper surface angle of the foot supporting plate 3 when the shank swings front and back. The deviation of the virtual axis and the articulated shaft is determined according to the angle change rule, and adjusted by the shaft adjusting part 7 during swing.

[0139] The thigh supporting plate 1 is provide with a fixing pad 8 on the upper surface, and the fixing pad 8 can adapt the shape of thigh and fixes one connection end of the thigh and the shank.

The fixing direction is rectilinear direction vertical to the first rotatable surface.

[0140] The existing elastic cushion adapts to the body shape by elastic deformation, but the area with large elastic deformation also have large supporting forces. Due to the characteristic of elastic deformation, the existing elastic cushion has no fixing effect, the shape is fixed by the fixing

INNOTRACK20001NLCN -12 - pad 8 after adapting to the body shape while a certain elasticity is provided to meet comfortable requirement after fixing, and the thigh is fixed to avoid impact on the operation resulting from transmitting the foot movement for adjustment to the hands during the minimally invasive operation.

[0141] Embodiment 12

[0142] In this embodiment, on the basis of Embodiment 11, specifically, as shown in Fig. 12, the thigh supporting plate 1 comprises a haunch supporting segment 1-1, a thigh supporting segment 1-2 and a first push rod 1-3. The supporting surface of the haunch supporting segment 1-1 is arranged horizontally, the supporting surface of the thigh supporting segment 1-2 is arranged in inclined form, and the lower end of the haunch supporting segment 1-1 is hinged with the supporting structure, the rotatable surface of the hinged connection is parallel with the first rotatable surface, one end of the first push rod 1-3 is hinged with the supporting structure. Another end of the first push rod 1-3 is hinged with the lower surface of the thigh supporting segment 1-2.

[0143] The haunch supporting segment 1-1 is used to support the haunch of the doctor. One end of the thigh supporting segment 1-2 which is away from the haunch supporting segment 1-1 is arranged in downward inclined form. During use, the thigh supporting segment 1-2 is driven by the first push rod 1-3 so that the haunch supporting segment 1-1 and the thigh supporting segment 1-2 rotate along the articulated shaft at the lower end of the haunch supporting segment 1-1 to adjust sitting position.

[0144] Embodiment 13

[0145] In this embodiment, on the basis of Embodiment 11, specifically, as shown in Fig. 12, the shank supporting plate 2 comprises an upper connection segment 2-1, a lower connection segment 2-2 and a second push rod 2-3. The lower end of the upper connection segment 2-1 is connected with the lower connection segment 2-2 by a sliding structure arranged in the length direction of the upper connection segment 2-1, a horizontal segment vertical to the length direction of the upper connection segment 2-1 is arranged on the lower end of the lower connection segment 2-2, and the upper surface of the horizontal segment is rotatablely connected with the foot supporting plate 3. One end of the second push rod 2-3 is connected with the upper connection segment 2-1. Another end of the second push rod 2-3 is connected with the lower connection segment 2-2.

[0146] The upper connection segment 2-1 and the lower connection segment 2-2 enable adjustment of the length of the shank supporting plate 2 by the sliding structure to adapts the shank length of the doctor, and ensure contact of the foot sole surface with the foot supporting plate 3.

[0147] Embodiment 14

[0148] In this embodiment, on the basis of Embodiment 11, specifically, as shown in Fig. 13, the

INNOTRACK20001NLCN -13 - foot supporting plate 3 comprises a rotary table 3-1, a tiptoe splint 3-2, a heel splint 3-3 and a third push rod 3-4. The rotary table 3-1 is rotatably connected with the shank supporting plate 2. The tiptoe splint 3-2 and the heel splint 3-3 are respectively arranged on two ends of the rotary table 3-1. The tiptoe splint 3-2 is connected with the rotary table 3-1 by the sliding structure, and the heel splint 3-3 is in fixed connection with the rotary table 3-1. One end of the third push rod 3- 4 is connected with the tiptoe splint 3-2. Another end of the third push rod 3-4 is connected with the rotary table 3-1. After the doctor steps on the foot supporting plate 3, the third push rod 3-4 shrinks, the foot is fixed by the tiptoe splint 3-2 and the heel splint 3-3 in the length direction so that the foot supporting plate 3 can rotate with the rotation of the foot.

[0149] Specifically, a first cambered surface 3-5 is arranged on the clamping surface of the tiptoe splint 3-2, a second cambered surface3-6 is arranged on the clamping surface of the heel splint 3-3, and the circle center of the first cambered surface 3-5 coincides with the circle center of the second cambered surface 3-6, and the circle center is located at the axis of the foot rotating relative to the shank in the first rotatable surface; a lengthways raised lines may be arranged on the first cambered surface 3-5 and the second cambered surface 3-6 to improve the friction force between the foot and the surface during rotation in the second rotatable surface and reduce the friction force between the foot and the surface during rotation in the first rotatable surface.

[0150] Specifically, the foot supporting plate 3 further comprises a first key 3-7 and a second key 3-8. The first key 3-7 is arranged close to one end of the tiptoe splint 3-2 on the upper surface of the rotary table 3-1. The second key 3-8 is arranged on the lower surface of a fixing block extending toward the direction of the rotary table 3-1 on the upper end of the tiptoe splint 3-2. The foot supporting plate 3 can be driven to output momentum only by the foot. Also the tiptoe may be moved upward or downward to touch the first key 3-7 or the second key 3-8. The different operation intents of the doctor may be output by the first key 3-7 and the second key 3-8 to adjust the protected image.

[0151] Embodiment 15

[0152] In this embodiment, on the basis of Embodiment 11, specifically, as shown in Fig. 12, the base 4 comprises a moving plate4-1, a fixed plate 4-2, a base spring 4-3 and a ball wheel 4-4. The upper end of the fixed plate 4-2 is connected with the supporting structure, a chute in which the moving plate 4-1 is arranged at the lower end of the fixed plate 4-2 in vertical direction, the annular surface at the outer side of the chute in the lower surface of the fixed plate 4-2 is a fixed plane for supporting. The base spring 4-3 is arranged between the moving plate 4-1and the fixed plate 4-2. Several ball wheels 4-4 are arranged on the lower surface of the moving plate 4-1. A sliding contact surface is formed by several ball wheels 4-4 to form a sliding surface. When the doctor moves, the pressure of the body to the device is reduced by the foot supporting surface not lifted, the upper part of the device is lifted by the base spring 4-3, the lower edge of the fixed

INNOTRACK20001NLCN -14 - plate 4-2 is lifted from the ground and supported on the ground by the foot of the doctor, the ball wheel 4-4 at the bottom of the moving plate 4-1 rotates, and the device is moved in desirable direction. After movement, the support of the foot to the body is reduced, the pressure of the body to the device is increased so that the fixed plate 4-2 moves towards the ground, the lower surface of the fixed plate 4-2 touches with the ground to fix the device.

[0153] Embodiment 16

[0154] This embodiment discloses a fixing pad for the adjusting device for body surface projection during minimally invasive operation. The fixing pad is used for the adjusting device for body surface projection during minimally invasive operation, and used to adapt the shape of the thigh, and fix the connecting end of the thigh and the shank. The fixing direction is vertical with the rectilinear direction of the first rotatable surface.

[0155] Specifically, as shown in Fig. 14, the fixing pad comprises a deformable pipe 8-1, a balance pipe 8-2, a stop element 8-3 and a balance bag 8-4. Several deformable pipes 8-1 are arranged in circular arc to form a circular arc contact surface. Water is charged in the deformable pipes 8- 1, and each deformable pipe 8-1 is connected with the balance bag 8-4 by the balance pipe 8-2. Several balance pipes 8-2 pass thought the stop element 8-3. The stop element 8-3 comprises a movable pressing surface used for pressing the balance pipes 8-2.

[0156] After the thigh is placed on the fixing pad, the deformable pipe 8-1 is pressed by the thigh the water in the deformable pipe 8-1 is pressed into the balance bag 8-4 by the balance pipe 8-

2. After decoration of several deformable pipes 8-1, the arc contact surface adapts the thigh shape, several balance pipes are pressed simultaneously by the stop element 8-3 to disconnect the deformable pipe 8-1 from the balance bag 8-4, and the shape of the arc contact surface is fixed. Because the deformable pipe 8-1 has elasticity and adapts the shape of the foot, compared with the common elastic material, the deformable pipe provides comfort and can be used to fix the foot with better adaptive capacity.

[0157] Embodiment 17

[0158] This embodiment discloses a shaft adjusting part for the body surface projection adjusting device for minimally invasive operation. The adjusting part is used in the body surface projection adjusting device for minimally invasive operation, and used to adjust the location of the articulated shaft of the thigh supporting plate 1 and the shank supporting plate 2 in the first rotatable surface with surface angle change of the foot sole surface and the foot supporting plate 3 when the shank supporting plate 2 swings with the articulated shaft at the upper end so that the axis of the articulated shaft coincides with the rotatable virtual axis of the thigh and the shank.

[0159] Specifically, as shown in Fig. 15, the shaft adjusting part comprises a shaft drive 7-1, a first waist strip 7-2, a first fixing part 7-3, a second waist strip 7-4, a table 7-5 and a second fixing part 7-6. The shaft drive 7-1 is arranged on the thigh supporting plate 1 and used to adjust the

INNOTRACK20001NLCN -15- location of the articulated shaft of the thigh supporting plate 1 and the shank supporting plate 2 in the first rotatable surface. The first waist strip 7-2 is fixed on the tiptoe splint 3-2. The second waist strip 7-4 is fixed on the heel splint 3-3. A sliding sleeve is sleeved on the outer sides of the first waist strip 7-2 and the second waist strip7-4. A spring structure for restoration of the sliding sleeve is arranged on the sleeve. The first fixing part 7-3 is arranged on the side of the first waist strip 7-2. The first fixing part 7-3 comprises a movable fixing head. The fixing head can press the sliding sleeve on the first waist strip to fix the location of the sliding sleeve. The table 7-5 is hinged on the upper surface of the foot supporting plate 3, the articulated shaft is located below the axis at the foot rotates relative to the shank in the first rotatable surface. The upper surface of the table 7-5 contacts with the foot sole surface. The table 7-5 can swing toward the tiptoe splint 3-2 or the heel splint 3-3. A table sensor 7-5-1 is arranged on the table 7-5 to detect the amplitude of swing of the table 7-5 toward the tiptoe splint 3-2 or the heel splint 3-3. The second fixing part 7-6 is arranged on the foot supporting plate 3. The second fixing part 7-6 comprises a fixed plane located below the table 7-5. The fixed plane comprises a fixed surface with same height from the upper surface of the foot supporting plate 3 below two swing ends of the table 7-5. The fixed plane can move in the direction vertical to the surface of the foot supporting plate 3; the table is adjusted by the second fixing part 7-6 so that the foot sole surface on the table 7-5 is parallel with the upper surface of the foot supporting plate 3, and the location is determined as the initial location. The sliding sleeve on the surface of the first waist strip 7-2 is fixed by the first fixing part 7-3 to fix the location of the tiptoe and the tiptoe splint 3-2. When the shank supporting plate 2 is driven to swing front and back by the shank, the tiptoe location is fixed. Because of the virtual axis of the thigh and shank connection is different from the axis of the articulated shaft of the thigh supporting plate 1 and the shank supporting plate 2, and during swing front and back, the foot has trend to upward move relative to the foot supporting plate 3 in the length direction of the shank. After the table 7-5 is released by the second fixing part 7-6, during swing front and back, the shank moves relative to the foot supporting plate 3 to slid the heel on the surface of the second waist strip7-4 and then swing the table 7-5. If the virtual axis is coaxial with the articulated shaft, the table 7-5 cannot swing, from this it is determined whether the virtual axis is coaxial with the articulated shaft to adjust the location of the articulated shaft and the coaxial of the virtual axis.

[0160] Specifically, as shown in Fig. 16, the shaft drive 7-1comprises a shaft fixing fork 7-1-1, a first shaft drive push rod 7-1-2 and a second shaft drive push rod 7-1-3. The upper end of an articulated ear is respectively hinged with two fork ends of the upper end of the shaft fixing fork 7-1-1. The lower end of two articulated ears fixed on the shank supporting plate 2. The lower end of the shaft fixing fork 7-1-1 is in fixed connection with one end of the first shaft drive push rod 7- 1-2, and another end of the first shaft drive push rod 7-1-2 is fixed connection with one end of the second shaft drive push rod 7-1-3, the another end of the second shaft drive push rod 7-1-3 fixed

INNOTRACK20001NLCN -16 - on the thigh supporting plate 1, and the first shaft drive push rod 7-1-2 is vertically arranged with the second shaft drive push rod 7-1-3. When the location of the articulated shaft is adjusted, the location of the shaft fixing fork 7-1-1 is adjusted by adjusting the extension length of the first shaft drive push rod 7-1-2 and he second shaft drive push rod 7-1-3 to move the articulated shaft.

[0161] Specifically, as shown in Fig. 17, the second fixing part 7-6 comprises a fixed barrel 7-6- 1, a fixed barrel chute 7-6-2 and a fixed barrel push rod 7-6-3. The axis of the fixed barrel 7-6-1 arranged below the table 7-5 vertically to the upper surface of the foot supporting plate 3. The fixed barrel chute 7-6-2 is arranged on the foot supporting plate 3, and used to guide the sliding of the fixed barrel 7-6-1 in the direction vertically to the direction of the foot supporting plate 3.

The side of the fixed barrel 7-6-1 is connected with one end of the fixed barrel push rod 7-6-3. Another end of the fixed barrel push rod 7-6-3 is fixed on the foot supporting plate 3. When the table 7-5 is fixed, the fixed barrel 7-6-1 is pulled by the fixed barrel push rod 7-8-3 to slid to the lower surface of the table 7-5 along the fixed barrel chute 7-6-2, with gap with the lower surface of the table 7-5. The table 7-5 is downward pressed by the foot, and the table 7-5 slides along the chute in the vertical direction vertical to the surface of the foot supporting plate 3, the elastic piece at the lower end of the table 7-5 is pressed while the lower surface of the table 7-5 moves and contact with the upper surface of the fixed barrel 7-6-1 to complete parallel locating of the foot sole surface and the foot supporting plate 3, the lower end of the table 7-5 is supported by the elastic piece to ensure that the foot sole surface always contacts with the upper surface of the table 7-5 during swing.

[0162] Specifically, the table sensor 7-5-1 is a third angle sensor. One end of the third angle sensor is connected with the table 7-5, and another end of the third angle sensor is fixe on the foot supporting plate 3. The rotation amplitude of the articulated shaft of the table relative to the surface of the foot supporting plate 3 is detected by the third angle sensor and used as the swing amplitude of the table 7-5.

[0163] Specifically, the table sensor 7-5-1 comprises two distance sensors. Two distance sensors are respectively arranged below two swing ends of the table 7-5. The distance between two swing ends of the table 7-5 relative to the plane of the fixing end at the upper surface of the foot supporting plate 3 is directly detected by two distance sensors, and used as the swing amplitude ofthe table 7-5.

[0164] Embodiment 18

[0165] This embodiment discloses a shaft drive of the shaft adjusting part for the body surface projection during minimally invasive operation. The shaft drive according to this embodiment is used for the shaft adjusting part for the body surface projection during minimally invasive operation to adjust the articulated shafts of the thigh supporting plate 1 and the shank supporting plate 2 on the first rotatable surface so that the axis of the articulated shaft coincides with the

INNOTRACK20001NLCN -17 - rotatable virtual axis of the thigh and the shank.

[0166] The shaft drive 7-1 comprises a shaft fixing fork 7-1-1, a first shaft drive push rod 7-1-2 and a second shaft drive push rod 7-1-3. The upper end of an articulated ear is respectively hinged with two fork ends of the upper end of the shaft fixing fork 7-1-1. The lower end of two articulated ears are fixed on the shank supporting plate 2. The lower end of the shaft fixing fork 7-1-1 is in fixed connection with one end of the first shaft drive push rod 7-1-2. Another end of the first shaft drive push rod 7-1-2 is in fixed connection with one end of the second shaft drive push rod 7-1-3. Another end of the second shaft drive push rod 7-1-3 is fixed on the thigh supporting plate 1. The first shaft drive push rod 7-1-2 is arranged vertically with the second shaft drive push rod 7-1-3. During adjustment of the location of the articulated shaft, the location of the shaft fixing fork 7-1-1 is adjusted by adjusting the extension length of the first shaft drive push rod 7-1-2 and the second shaft drive push rod 7-1-3 to move the articulated shaft.

[0167] Embodiment 19

[0168] This embodiment discloses a second fixing part of the shaft adjusting part for the body surface projection during minimally invasive operation. The second fixing part according to this embodiment is used for the shaft adjusting part for the body surface projection during minimally invasive operation to locate the initial location of the table 7-5.

[0169] The second fixing part7-6 is characterized in comprising a fixed barrel 7-6-1, a fixed barrel chute 7-6-2 and a fixed barrel push rod 7-6-3. The axis of the fixed barrel 7-6-1 is arranged below the table 7-5 vertically to the upper surface of the foot supporting plate 3. The fixed barrel chute 7-6-2 is arranged on the foot supporting plate 3, and used to guide the sliding of the fixed barrel 7-6-1 in the direction vertically to the direction of the foot supporting plate 3. The side of the fixed barrel 7-6-1 is connected with one end of the fixed barrel push rod 7-6-3. Another end of the fixed barrel push rod 7-6-3 is fixed on the foot supporting plate 3. When the table 7-5 is fixed, the fixed barrel 7-6-1 is pulled by the fixed barrel push rod 7-8-3 to slid to the upper surface of the table 7- 5 along the fixed barrel chute 7-6-2, with gap with the lower surface of the table 7-5. The table 7- 5 is downward pressed by the foot, and the table 7-5 slides along the chute in the vertical direction vertical to the surface of the foot supporting plate 3, the elastic piece at the lower end of the table 7-5 is pressed while the lower surface of the table 7-5 moves and contact with the upper surface of the fixed barrel 7-6-1 to complete parallel locating of the foot sole surface and the foot supporting plate 3, the lower end of the table 7-5 is supported by the elastic piece to ensure that the foot sole surface always contacts with the upper surface of the table 7-5 during swing.

[0170] Embodiment 20

[0171] This embodiment discloses an adjustment method for the body surface projection during the minimally invasive operation. The method according to this embodiment is used in the adjusting device for body surface projection during the minimally invasive operation according to

INNOTRACK20001NLCN -18 - Embodiment 11, Embodiment 12, Embodiment 13, Embodiment 14 or Embodiment 15 to adjust the body surface projection to facilitate the observation without interruption of the operation.

[0172] Specifically, the method comprises the following steps:

[0173] Step a: fixation of thigh: the doctor lifts her or his one leg, and sit on haunch supporting segment 1-1, place the lifted thigh on the thigh supporting segment 1-2 in the length direction of the thigh supporting segment 1-2 to fix the thigh of the doctor with the thigh supporting plate 1 so that the movement of the shank and the foot is separated from the movement of the upper body to avoid the impact on the operation during adjustment.

[0174] Step b: locating based on shank length: the lifted shank is drooped naturally, the second push rod 2-3 is shrunk to move the lower connection segment 2-2 of the shank supporting plate toward the upper connection segment 2-1 so that the foot supporting plate 3 move upwards by shortening the length of the shank supporting plate 2 till the foot sole surface contacts with the foot supporting plate 3. The shank supporting plate 3 is located acceding to the shank length of the doctor.

[0175] Step c: foot locating: the tiptoe splint 3-2 is pulled by the third push rod 3-4 to move toward the tiptoe, the foot is clamped by the tiptoe splint 3-2 and the heel splint 3-3 in the length direction so that he foot is fixed with the foot supporting plate 3.

[0176] Step d: obtaining y- coordinate value: when the doctor is required to enter the y-coordinate quantity, the shank supporting plate 2 is driven by the shank to swing the articulated shaft of the shank supporting plate 2 and the thigh supporting plate 1 front and back in the first rotatable surface, by using the articulated shaft of the shank supporting plate 2 and the thigh supporting plate 1 as axis, the swing amplitude of the shank supporting plate 2 is obtained by the first angle sensor 5, and the swing amplitude of the shank supporting plate 2 is used as the y- coordinate quantity.

[0177] Step e: obtaining x-coordinate value: when the doctor is required to enter the x-coordinate quantity, the foot supporting plate 3 is driven by moving the foot leftward or rightward to swing in the second rotatable surface by using the rotating shaft connected with the foot supporting plate 3 and the shank supporting plate 2 as the axis, the swing amplitude of the foot supporting plate 3 is obtained by the second angle sensor 8, and the swing amplitude of the shank supporting plate 3 is used as the x- coordinate quantity.

[0178] Step f: obtaining operation intent: when the doctor operates at the designated area projected on the body surface, the doctor slides tiptoe upward or downward along the first cambered surface 3-5 to touch the first key 3-7 or the second key 3-8 with the tiptoe, and the doctor's operation intent is obtained by the first key 3-7 or the second key 3-8.

[0179] The value of the y-coordinate and the x-coordinate with the doctor's intent corresponds to the location of the body surface projection. After the doctor enters the value of the y-coordinate

INNOTRACK20001NLCN -19 - and the x-coordinate, the projected area requiring amplification can be selected to facilitate the observation without the assistance of the assistant and impact on the operation continuity.

[0180] Specifically, the method further comprises a articulated shaft location adjusting step. After locating the foot, the location of the articulated shaft of the thigh supporting plate 1 and the shank supporting plate 2 in the first rotatable surface is adjusted to achieve coincidence of the articulated shaft of the rotatable virtual axis of the doctor's thigh and shank.

[0181] Specifically, when the thigh is fixed as shown in Step a, the fixing pad 8 adapts the shape of the thigh, and is used to fix the end connecting with doctor's thigh and shank. The fixing direction is the rectilinear direction vertical to the first rotatable surface.

[0182] Embodiment 21

[0183] This embodiment discloses y- coordinate value output method used in the adjusting method for body surface projection during minimally invasive operation. The method according to this embodiment is used in the adjusting method for body surface projection during minimally invasive operation to output the y-coordinate with the doctor's intent.

[0184] Specifically, the included angle of the shank supporting plate 2 in the length direction and the vertical direction is set to the zero point of the y- coordinate, while setting the output value of the first angle sensor 5 when the shank supporting plate 2 is located at the zero point of the y- coordinate to zero. When the doctor enters the y- coordinate value, the shank supporting plate 2 is driven by moving the shank frontward or backward to swing the articulated shaft of the shank supporting plate 2 and the thigh supporting plate 1 in the first rotatable surface, the swing amplitude of the shank supporting plate 2 is obtained by the first angle sensor 5, and the value corresponding to the swing amplitude of the shank supporting plate 2 is output as the y- coordinate value.

[0185] Embodiment 22

[0186] This embodiment discloses x- coordinate value output method used in the adjusting method for body surface projection during minimally invasive operation. The method according to this embodiment is used in the adjusting method for body surface projection during minimally invasive operation to output the x--coordinate with the doctor’s intent.

[0187] The included angle of the connecting line between the tiptoe splint 3-2 and the heel splint 3-3 of the foot supporting plate 3 and the first rotatable surface is set to the zero point of the x- coordinate while setting the output value of the second angle sensor 8 when the foot supporting plate 3 is located at the zero point of the x- coordinate to zero. When the doctor enters the x- coordinate value, the foot supporting plate 3 is driven by moving the shank frontward or backward to swing the articulated shaft of the foot supporting plate 3 and the shank supporting plate 2 in the second rotatable surface, the swing amplitude of the foot supporting plate 3 is obtained by the second angle sensor 6, and the value corresponding to the swing amplitude of the foot supporting

INNOTRACK20001NLCN - 20 - plate 3 is output as the x- coordinate value.

[0188] Embodiment 23

[0189] This embodiment discloses an operation intent output method for the adjusting method for body surface projection during the minimally invasive operation. The method according to this embodiment is used for the adjusting method for body surface projection during the minimally invasive operation to output the intent of the doctor to operate the body surface projection.

[0190] When the doctor moves his or her tiptoe upwards or downwards, the first key 3-7 and the second key 3-8 tiptoe are set according to the distribution of the locations contactable with the sliding route of tiptoe along the first cambered surface 3-5. The first key 3-7 and the second key 3-8 tiptoe are set respectively to correspond to the intent of the doctor to operate the designated body surface projection area. When doctor moves his or her tiptoe upwards or downwards, the tiptoe slides along the first cambered surface 3-5 to contact with the first key 3-7 or the second key 3-8. The operation intend of the doctor is output by the first key 3-7 or the second key 3-8.

[0191] Embodiment 24

[0192] This embodiment discloses a shaft adjusting method for the adjusting method for body surface projection during the minimally invasive operation. The method according to this embodiment is used for the adjusting method for body surface projection during the minimally invasive operation according to Embodiment 20 to achieve coincidence of the axis of the articulated shaft with the rotatable virtual axis of the thigh and the shank by adjusting the location of the articulated shaft of the thigh supporting plate 1 and the shank supporting plate 2 in the first rotatable surface with the surface angle of the foot sole surface and the foot supporting plate 3 during swing of the shank supporting plate 2 along the articulated shaft at the upper end.

[0193] Specifically, the method comprises the following steps:

[0194] Step a foot sole surface locating: during locating the foot, the fixed plane arranged below the table 7-5 is lifted by second fixing part 7-6, the fixed plane contacts with the lower surface of the table 7-5, the upper surface of the table 7-5 is paralleled with the upper surface of the foot supporting plate 3 to the locate the foot sole surface. The foot sole surface is paralleled with the upper surface of the foot supporting plate 3.

[0195] Step b tiptoe fixation: during locating the foot, after the foot is clamped by the tiptoe splint 3-2 and the heel splint 3-3 in the length direction, the fixing head of the first fixing part 7-3 is moved to the first waist strip 7-2, the fixing head clamps and fixes the sliding sleeve on the first waist strip 7-2. The sliding sleeve fixes the tiptoe of the doctor at the sliding surface of the first waist strip 7-2 by the friction force.

[0196] Step c obtaining foot sole angle change: as shown in Fig. 18, the shank supporting plate 2 is driven by the shank to swing the articulated shaft of the shank supporting plate 2 and the thigh supporting plate 1 front and back in the first rotatable surface, the reading of the first angle

INNOTRACK20001NLCN -21- sensor 5 is read, the range between adjacent maximum and minimum reading of the first angle sensor 5 is set to period. One period is selected to read maximum a and minimum b in this period. a is compared with —b. if a > -b, the segment from —b to b of this period is taken as sampling segment. If a< -b, the segment from a to -a of this period is taken as sampling segment. The reading of the table sensor 7-5-1 is read, the reading of the table sensor 7-5-1 corresponds to the reading of the first angle sensor 5 at the time axis. The reading of the table sensor 7-5-1 in the sampling segment is obtained to determine the change value of the foot sole surface relative to the foot supporting plate 3 in the sampling segment.

[0197] Step d determining offset direction: read the readings c and d of the table sensor 7-5-1 corresponding to the two ends of the obtained segment, read the negative and positive sign of ¢ and d, define the horizontal direction in the first rotatable surface as the horizontal axis and the vertical direction as the vertical axis. The reading of the table sensor 7-5-1 is zero when the upper surface of the table 7-5 is parallel to the upper surface of the foot supporting plate 3. The negative value is output by the table sensor 7-5-1 when the table 7-5 rotates clockwise in the first rotatab surface, and the positive value is output by the table sensor 7-5-1 when the table 7-5 rotates anticlockwise in the first rotatable surface. The rotatable virtual axis of thigh and shank is located in the intersegment point of the horizontal axis and the vertical axis. If c is positive, and d is negative, the virtual axis is located at the right of the articulated shaft. If c is negative, and d is positive, the virtual axis is located at the left of the articulated shaft. If c is positive and d is positive, the virtual axis is located below the articulated shaft. If c is negative, and d is negative, the virtual axis is located above the articulated shaft. As shown in Fig. 18 and Fig. 19, when the virtual axis of the foot is located at the left of the articulated shaft, and the foot is swung frontward, the positive value is output by the first angle sensor 5. Because the virtual axis is not coaxial with the articulated shaft, the shank has trend to move downward in the length direction of the shank, because the tiptoe is fixed, the table is driven by the foot sole surface to deflect clockwise relative to the upper surface of the foot supporting plate 3, and the negative value is output by the table sensor. When the foot is swung backwards, the negative value is output by the first angle sensor, the shank has trend to move upwards in the length direction of the shank, because the tiptoe is fixed, the table is driven by the foot sole surface to deflect anticlockwise relative to the upper surface of the foot supporting plate 3, and the positive value is output by the table sensor.

[0198] Step e adjusting articulated shaft: length m is set. When c has different sign with d, the articulated shaft is moved by the length m along the horizontal axis. If c is positive and d is negative, the articulated shaft is adjusted rightward. If c is negative and d is positive, the articulated shaft is moved leftward, Step d is repeated. If the sign of c and d is unchanged, the articulated shaft is continuously moved by the length m till the sign of c and d is changed. If the sign of c and d is changed, the articulated shaft is continuously moved by 0.5m length till c has

INNOTRACK20001NLCN -22 - same sign with d. The articulated shaft is moved by the length m along the vertical axis, if c is positive and d is positive, the articulated shaft is moved downward. If c is negative and d is negative, the articulated shaft is moved upwards. Step d is repeated till c has different sign with d.

[0199] If c has same sign with d, the articulated shaft is moved by the length m along the vertical axis. If c is positive and d is positive, the articulated shaft is moved downward. If c is negative and d is negative, the articulated shaft is moved upwards. Step d is repeated till c has different sign with d. The articulated shaft is moved by the length m along the horizontal axis. If c is positive and d is negative, the articulated shaft is moved rightward. If c is negative and d is positive, the articulated shaft is moved leftward. Step d is repeated. If the sign of c and d is unchanged, the articulated shaft is continuously moved by the length m till the sign of ¢ and d is changed. If the sign of c and d is changed, the articulated shaft is continuously moved by 0.5m length till c has same sign with d.

[0200] As shown in Fig. 20 and Fig. 21, the foot length is ignored, the shank is simplified to a line segment e, the shank supporting plate is simplified to the line segment f. When the shank line segment e is at the left of the vertical line, it is located at the outer side of the supporting plate line f, c is negative. When the shank line segment e is at the right of the vertical line, it is located at the outer side of the supporting plate line f, d is positive, and it is determined that the virtual axis is located at the left of the articulated shaft. The articulated shaft is move leftward by m, in one period, c is negative and d is positive to determine that the virtual axis is still located at the left of the articulated shaft. The articulated shaft is continuously move leftward by m till c is positive, and d becomes auxiliary, it is determined that the virtual axis is located at the right of the articulated shaft and the distance of the virtual axis and the articulated shaft in horizontal direction is less than m. The articulated shaft is continuously move by reducing m to 0.5m, and continuously moved if the sign of c and d are unchanged. If the sign of c and d is changed, the movement distance is reduced continuously till c has same sign with d while the height of the virtual axis and the articulated shaft is adjusted. At this moment, the trace of the shank line segment is located at the outer side of the supporting plate f, c and d are positive, and it is determined that virtual axis is located above the articulated shaft. The articulated shaft is moved by m value to increase the length of the shank supporting plate by m value. If c has same sign with d, the articulated shaft is moved continuously in the vertical direction. If c has different sign with d, adjustment is performed in the horizontal direction. Threshold value n is set, when m is reduced to n, it is considered that the error of the virtual axis and the articulated shaft is within allowable range, and the adjustment is stopped.

[0201] Specifically, in Step d, the third angle sensor is arranged at the table 7-5 for detection. The output value of the third angle sensor is set to zero when the upper surface of the table 7-5 is

INNOTRACK20001NLCN -23- parallel to the upper surface of the foot supporting plate 3, the output value of table sensor 7-5-1 is negative when the table 7-5 rotates clockwise in the first rotatable surface, and the output value is positive when the table 7-5 rotates anticlockwise in the first rotatable surface.

[0202] Embodiment 25

[0203] This embodiment discloses a foot sole surface angle obtaining method for body surface projection shaft adjustment during minimally invasive operation. This method according to this embodiment is used in the shaft adjustment method for the adjusting method for body surface projection in minimally invasive operation according to Embodiment 24.

[9204] Specifically, the shank supporting plate 2 is driven by the shank to swing the articulated shaft of the shank supporting plate 2 and the thigh supporting plate 1 front and back in the first rotatable surface, the reading of the first angle sensor 5 is read, the range between adjacent maximum and minimum reading of the first angle sensor 5 is set to period. One period is selected to read maximum a and minimum b in this period. a is compared with —b. if a > -b, the segment from —b to b of this period is taken as sampling segment. If a< -b, the segment from a to -a of this period is taken as sampling segment. The reading of the table sensor 7-5-1 is read, the reading of the table sensor 7-5-1 corresponds to the reading of the first angle sensor 5 at the time axis. The reading of the table sensor 7-5-1 in the sampling segment is obtained to determine the angle change value of the foot sole surface relative to the foot supporting plate 3 in the sampling segment.

[0205] Embodiment 26

[9206] This embodiment discloses a offset direction determination method for the body surface projection shaft adjustment in the minimally invasive operation. This method according to this embodiment is used for the body surface projection shaft adjustment in the minimally invasive operation. According to Embodiment 24.

[0207] Specifically read the readings c and d of the table sensor 7-5-1 corresponding to the two ends of the obtained segment, read the negative and positive sign of c and d, define the horizontal direction in the first rotatable surface as the horizontal axis and the vertical direction as the vertical axis. The reading of the table sensor 7-5-1 is zero when the upper surface of the table 7-5 is parallel to the upper surface of the foot supporting plate 3. The negative value is output by the table sensor 7-5-1 when the table 7-5 rotates clockwise in the first rotatable surface, and the positive value is output by the table sensor 7-5-1 when the table 7-5 rotates anticlockwise in the first rotatable surface. The rotatable virtual axis of thigh and shank is located in the intersegment point of the horizontal axis and the vertical axis. If c is positive, and d is negative, the virtual axis is located at the right of the articulated shaft. If c is negative, and d is positive, the virtual axis is located at the left of the articulated shaft. If c is positive and d is positive, the virtual axis is located below the articulated shaft. If c is negative, and d is negative, the virtual axis is located above the

INNOTRACK20001NLCN -24 - articulated shaft.

[0208] Embodiment 27

[0209] This embodiment discloses an articulated shaft adjusting method for the shaft adjustment for body surface projection in minimally invasive operation. The method according to this embodiment is used for the shaft adjustment for body surface projection in minimally invasive operation according to Embodiment 24.

[0210] Specifically, Step e adjusting articulated shaft: length m is set. When c has different sign with d, the articulated shaft is moved by the length m along the horizontal axis. If c is positive and d is negative, the articulated shaft is adjusted rightward. If ¢ negative is and d is positive, the articulated shaft is moved leftward, Step d is repeated. If the sign of c and d is unchanged, the articulated shaft is continuously moved by the length m till the sign of ¢ and d is changed. If the sign of c and d is changed, the articulated shaft is continuously moved by 0.5m length till c has same sign with d. The articulated shaft is moved by the length m along the vertical axis, if c is positive and d is positive, the articulated shaft is moved downward. If c is negative and d is negative, the articulated shaft is moved upwards. Step d is repeated till c has different sign with d.

[0211] If c has same sign with d, the articulated shaft is moved by the length m along the vertical axis. If c is positive and d is positive, the articulated shaft is moved downward. If c is negative and d is negative, the articulated shaft is moved upwards. Step d is repeated till c has different sign with d. The articulated shaft is moved by the length m along the horizontal axis. If c is positive and d is negative, the articulated shaft is moved rightward. If c is negative and d is positive, the articulated shaft is moved leftward. Step d is repeated. If the sign of c and d is unchanged, the articulated shaft is continuously moved by the length m till the sign of c and d is changed. If the sign of c and d is changed, the articulated shaft is continuously moved by 0.5m length till c has same sign with d.

Claims

INNOTRACK20001NLCN -25- CONCLUSIONS

A method for virtual and transparent observation of an internal cavity in body surface projection during minimally invasive surgery, characterized in that said method comprises the following steps: Step a: Kinect tracing of the physician's field of view, for determining the current position of a doctor's head and field of vision; Step b: PTAM (Parallel Tracking and Mapping) 3D modeling of the internal cavity, to generate the 3D model that can rotate with said field of view of the physician in Step a; Step c: Generating an image sequence of the body surface projection of a patient, wherein said 3D model of the internal cavity in Step b is projected onto the body surface of the patient, in accordance with said field of view of the physician in Step a.

The method for virtual and transparent observation of an internal cavity in body surface projection during minimally invasive surgery according to claim 1, characterized in that said Kinect tracking of the physician's field of view in Step a comprises the following steps: Step a1: selecting of depth data from the pupil and tip of the nose as key points to create a mathematical model for the physician's field of vision; Step a2: predicting the physician's field of view and correcting the tracking data by means of a Kalman filter; Step a3: Analyzing the location relationship of the doctor's head, the patient's body surface, the patient's internal cavity and a projector, unifying a coordinate system and designing the projection system parameter of a body surface patient.

The method for virtual and transparent observation of an internal cavity in body surface projection during minimally invasive surgery according to claim 1, characterized in that said PTAM 3D modeling of the internal cavity in Step b comprises the following steps: Step b1: determining of a typical quantity of the internal cavity, comprising: Step b11: preprocessing the image of the internal cavity by distinguishing a specular reflection area of a blood vessel; Step b12: detecting the single pixel blood vessel;

INNOTRACK20001NLCN - 26 - Step b13: determining a node and node segment; Step b2: 3D modeling the internal cavity: storing 3D data and texture images of the internal cavity in file format Stanford PLV, including providing 3D node data and texture information in multi-frame PTAM image, by using OpenGL and in accordance with the physician's field of view to generate a rotatable 3D model of the internal cavity that can be observed from multiple angles.

The method for virtual and transparent observation of an interior space in body surface projection during minimally invasive surgery according to claim 1, characterized in that said generating an image sequence of the body surface projection of a patient in Step c comprises the following steps: Step c1: the calculating the image of the internal cavity that the physician expects to see by means of a conventional camera imaging model, in accordance with the physician's field of view and the 3D model of the internal cavity; Step c2: Calculating the image sequence of body surface projection in accordance with the image of the internal cavity, 3D morphology of the body surface of the patient, the field of view of the doctor and the location of the projector.