CN117942164A - Information feedback method, image trolley and electronic equipment - Google Patents
Information feedback method, image trolley and electronic equipment Download PDFInfo
- Publication number
- CN117942164A CN117942164A CN202211292310.6A CN202211292310A CN117942164A CN 117942164 A CN117942164 A CN 117942164A CN 202211292310 A CN202211292310 A CN 202211292310A CN 117942164 A CN117942164 A CN 117942164A
- Authority
- CN
- China
- Prior art keywords
- image
- region
- interest
- strain
- grid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000006073 displacement reaction Methods 0.000 claims description 33
- 238000012545 processing Methods 0.000 claims description 27
- 238000013507 mapping Methods 0.000 claims description 26
- 238000004590 computer program Methods 0.000 claims description 17
- 238000010586 diagram Methods 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 16
- 230000008520 organization Effects 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 10
- 238000009499 grossing Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 abstract description 11
- 210000001519 tissue Anatomy 0.000 description 50
- 238000001356 surgical procedure Methods 0.000 description 11
- 238000006243 chemical reaction Methods 0.000 description 9
- 238000007405 data analysis Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 239000003086 colorant Substances 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 210000004204 blood vessel Anatomy 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 241001408665 Timandra griseata Species 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013016 damping Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000338 in vitro Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000000399 orthopedic effect Effects 0.000 description 1
- 244000144985 peep Species 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 210000000115 thoracic cavity Anatomy 0.000 description 1
- 230000002485 urinary effect Effects 0.000 description 1
- 210000001835 viscera Anatomy 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Image Processing (AREA)
Abstract
The invention provides an information feedback method, an image trolley and electronic equipment, and relates to the technical field of medical equipment, wherein the method comprises the following steps: receiving an image of a region of interest; obtaining the surface deformation of the region of interest according to the image of the region of interest; obtaining a strain field of the region of interest according to the surface deformation of the region of interest; and displaying a strain cloud image of the region of interest based on the strain field of the region of interest. The image trolley is used for executing the method. According to the information feedback method, the image trolley and the electronic equipment provided by the embodiment of the invention, the strain feedback of the interested area in the operation process is provided in a non-contact mode, and the operation safety is improved.
Description
Technical Field
The invention relates to the technical field of medical equipment, in particular to an information feedback method, an image trolley and electronic equipment.
Background
Modern surgery can directly observe internal organs of a body through a peep tube or an endoscope connected with a miniature television camera, and has been widely used in hepatobiliary surgery, thoracic surgery, obstetrics and gynecology, urinary surgery, orthopedics and the like.
Currently, in modern surgical procedures, surgical instruments are either powerless or force feedback is required by mounting sensors on the surgical instruments in a contact manner, which, although capable of providing load changes at the contact points, still present the risk of mishandling causing damage to the patient's internal tissues. Therefore, how to provide relevant feedback information more safely during surgery to assist doctors in performing surgery is an important issue to be solved in the art.
Disclosure of Invention
Aiming at the problems in the prior art, the embodiment of the invention provides an information feedback method, an image trolley and electronic equipment, which can at least partially solve the problems in the prior art.
In a first aspect, the present invention proposes an information feedback method, including:
Receiving an image of a region of interest;
obtaining the surface deformation of the region of interest according to the image of the region of interest;
Obtaining a strain field of the region of interest according to the surface deformation of the region of interest;
And displaying a strain cloud image of the region of interest based on the strain field of the region of interest.
In a second aspect, the present invention provides an image dolly, comprising an image receiving module, an image processing module and an image display module, wherein:
the image receiving module is used for receiving the image of the region of interest;
The image processing module is used for obtaining the surface deformation of the region of interest according to the image of the region of interest and obtaining the strain field of the region of interest according to the surface deformation of the region of interest;
the image display module is used for displaying a strain cloud image of the region of interest based on the strain field of the region of interest.
In a third aspect, the present invention provides an information feedback system, including the image trolley according to any one of the above embodiments.
In a fourth aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the information feedback method according to any of the above embodiments when the program is executed by the processor.
In a fifth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the information feedback method according to any of the above embodiments.
According to the information feedback method, the image trolley and the electronic equipment provided by the embodiment of the invention, the image of the region of interest can be received, the surface deformation of the region of interest is obtained according to the image of the region of interest, the strain field of the region of interest is obtained according to the surface deformation of the region of interest, the strain cloud image of the region of interest is displayed based on the strain field of the region of interest, the strain feedback of the region of interest in the operation process is provided in a non-contact mode, and the safety of the operation is improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
fig. 1 is a schematic structural view of a surgical robot system according to a first embodiment of the present invention.
Fig. 2 is a flowchart of an information feedback method according to a second embodiment of the present invention.
Fig. 3 is a flowchart of an information feedback method according to a third embodiment of the present invention.
Fig. 4 is a flowchart of an information feedback method according to a fourth embodiment of the present invention.
Fig. 5 is a flowchart of an information feedback method according to a fifth embodiment of the present invention.
Fig. 6 is a schematic diagram of a strain cloud of a region of interest provided by a sixth embodiment of the invention.
Fig. 7 is a flowchart of an information feedback method according to a seventh embodiment of the present invention.
Fig. 8 is a schematic structural diagram of an information feedback system according to an eighth embodiment of the present invention.
Fig. 9 is a schematic structural view of an image dolly provided in a ninth embodiment of the present invention.
Fig. 10 is a schematic physical structure of an electronic device according to a tenth embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present application and their descriptions herein are for the purpose of explaining the present application, but are not to be construed as limiting the application. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be arbitrarily combined with each other.
Fig. 1 is a schematic structural diagram of a surgical robot system according to a first embodiment of the present invention, and as shown in fig. 1, the surgical robot system according to an embodiment of the present invention includes a doctor-side apparatus 1, a patient-side apparatus 2, an image carriage 3, and an auxiliary apparatus 4, wherein:
The doctor-side apparatus 1 is communicatively connected to the patient-side apparatus 2 and the image dolly 3, respectively. The patient-side device 2 is provided with at least one mechanical arm on which the surgical instrument is mounted. The patient-side device 2 is further provided with an endoscope for acquiring images of the region of interest. The doctor terminal device 1 is provided with a main manipulator and a display screen, the display screen is convenient for a doctor to observe the view scene of the operation, and the doctor operates the main manipulator to perform the operation. Wherein the region of interest includes, but is not limited to, a tissue surface.
The doctor controls the surgical instrument of the patient end device 2 to perform surgery through the main manipulator of the doctor end device 1, and the image trolley 3 receives the image of the region of interest acquired by the endoscope, and processes and displays the image of the region of interest. It is understood that the image dolly 3 is provided with a processor for performing image data processing and a display screen for displaying an image.
In the embodiment of the present invention, the image dolly 3 is used for executing the information feedback method provided in the embodiment of the present invention.
The endoscope of the patient-side apparatus 2 acquires an image of the region of interest and transmits the image to the image dolly 3. The image trolley 3 performs gray level distribution analysis on the acquired image information, performs image data processing and conversion into a strain field of the region of interest, and displays the strain field on a display screen in the form of strain cloud images. The image trolley 3 can send the strain cloud image to the doctor-side device 1 to be displayed in the operating view of the doctor so as to be convenient for the doctor to check.
Meanwhile, strain data can be transmitted to a doctor trolley end, and the strain data is input to a joint controller for doctor end operation control;
the final execution module is two main mechanical hands operated by doctors, and the force received by the instrument can be sensed while the mechanical hand at the patient end is controlled.
The following describes a specific implementation procedure of the information feedback method provided by the embodiment of the present invention, taking an image trolley as an execution subject. It can be understood that the execution subject of the information feedback method provided by the embodiment of the invention is not limited to the image dolly.
Fig. 2 is a flow chart of an information feedback method according to a second embodiment of the present invention, and as shown in fig. 2, the information feedback method according to the embodiment of the present invention includes:
s201, receiving an image of a region of interest;
Specifically, during a surgical operation, an endoscope acquires an image of a region of interest, and then transmits the acquired image to an image dolly. The image trolley receives an image of the region of interest.
For example, images under the surgical field are acquired by two charge coupled device (charge coupled device, abbreviated as CCD) cameras of an endoscope and sent to an image trolley as images of the region of interest. The three-dimensional coordinates of each point of the tissue surface in the image can be calculated on the basis of the known camera parameters and the position parameters.
S202, obtaining surface deformation of the region of interest according to the image of the region of interest;
specifically, the image trolley processes the image of the region of interest, and the surface deformation of the region of interest can be obtained through the difference of the images of the adjacent moments of the region of interest. Wherein the surface deformations of the region of interest may be surface deformations of one or more of the tissues muscle, fat, blood vessels, skin, etc.
S203, obtaining a strain field of the region of interest according to the surface deformation;
Specifically, the image trolley performs smoothing treatment on the surface deformation of the region of interest, and then obtains a strain field of the region of interest through green-lagrangian strain calculation.
S204, displaying a strain cloud image of the region of interest based on the strain field.
Specifically, after the strain field of the region of interest is obtained, the image trolley distinguishes different strains in the strain field of the region of interest in different colors, so that the difference of strain values is expressed as the difference of colors, and a strain cloud image of the region of interest is obtained and displayed. The strain cloud image of the region of interest can be sent to a display screen of doctor terminal equipment for display, so that a doctor can intuitively know the strain condition of the current operation position, and the operation safety is improved. The corresponding relation between the strain value and the color is set according to actual needs, and the embodiment of the invention is not limited.
Further, a segmentation threshold can be set for strain, safety prompts are carried out in the operation process, and the situation that excessive loading is caused on the tissue surface due to the operation of the instrument is prevented by the segmentation threshold.
In the non-instrument contact area under the operation observation view, when the strain value of the tissue surface exceeds the prompt threshold, safety prompt is carried out under the doctor view, and the doctor is informed of the risk of damaging the tissue in the body through operation. The segmentation threshold can be set as well as a time threshold, and the time threshold is used for preventing false alarm caused by data jump and avoiding the influence of false alarm and shutdown on operation fluency.
For example, a prompt threshold and an alarm threshold are set, and the prompt threshold is smaller than the alarm threshold. In addition, a time threshold may also be set. The prompting threshold, the alarming threshold and the time threshold are set according to actual needs, and the embodiment of the invention is not limited.
When the strain of a certain position of the region of interest is greater than the prompt threshold and less than the alarm threshold, prompt information can be displayed on an operation interface of doctor-side equipment: there is a risk of damaging the tissue. At this time, the doctor can normally operate the manipulator.
When the strain of a certain position of the region of interest is larger than the alarm threshold value and the duration is larger than the time threshold value, a fault code can be displayed on an operation interface of doctor-side equipment to prompt that a large risk exists to damage tissues, and a doctor is blocked in the direction of increasing the strain when operating the manipulator.
When the strain of a certain position of the region of interest is larger than the alarm threshold value and the duration is smaller than or equal to the time threshold value, the false alarm caused by data jump can be considered, and the alarm prompt is not carried out so as to avoid affecting the operation of doctors.
Further, in order to prevent the strain cloud image from obstructing the surgical field of the doctor and interfering with the doctor's surgery, the region of interest may be covered with a color having transparency to display the strain cloud image, thereby reducing the visual impact of the strain cloud image on the surgery.
The information feedback method provided by the embodiment of the invention can receive the image of the region of interest, obtain the surface deformation of the region of interest according to the image of the region of interest, obtain the strain field of the region of interest according to the surface deformation of the region of interest, display the strain cloud picture of the region of interest based on the strain field of the region of interest, provide the strain feedback of the region of interest in the operation process in a contactless manner, and improve the safety of the operation. In addition, the strain field is displayed in the form of strain cloud pictures, so that a doctor can intuitively feel the strain of the tissue surface through different colors in the operation process, the doctor is assisted to perform reasonable operation, and the improvement of operation safety is facilitated.
Fig. 3 is a schematic flow chart of an information feedback method according to a third embodiment of the present invention, as shown in fig. 3, further, based on the above embodiments, the obtaining, according to the image of the region of interest, surface deformation of the region of interest includes:
S301, carrying out grid division on the current image of the region of interest to obtain coordinates and gray values of each grid image of the current image;
Specifically, when the endoscope acquires images of the region of interest, the image acquisition is performed at a certain frequency, the image trolley continuously receives the images of the region of interest, and then each image is sequentially processed according to the acquisition sequence of the images. In the embodiment of the invention, the current image of the region of interest, namely the image of the region of interest being processed by the image trolley. The image trolley performs grid division on the current image of the region of interest, divides the current image of the region of interest into grid images with the same size, and acquires coordinates of each grid image and gray values of each grid image. The coordinates of the grid image include plane coordinates of each pixel point in the grid image, i.e., pixel coordinates, and height coordinates, i.e., coordinates in a height direction perpendicular to a plane in which the plane coordinates are located. The size of the grid is set according to actual needs, for example, a pixel block with the size of 15x15 is set, which is not limited in the embodiment of the present invention.
It can be understood that the gray scale processing may be performed on the grid image, so as to obtain a gray scale value of each pixel point in the grid image, which is used as the gray scale value of the grid image. The purpose of the meshing is to calculate the deformation of each mesh image so that the strain field of the image of the entire region of interest can be obtained.
For example, plane coordinates and height coordinates of each pixel point within the image of the region of interest may be obtained by stereoscopic techniques.
S302, according to the gray value of each grid image of the previous image of the current image and the gray value of each grid image of the current image, obtaining the mapping relation between each grid image of the current image and each grid image of the previous image; wherein the gray values of each grid image of the previous image of the current image are obtained in advance;
Specifically, one grid image of the current image corresponds to a certain grid image of a previous image of the current image, the similarity of each grid image can be calculated through a normalized product correlation coefficient or a normalized error square sum based on gray values of the two grid images, and whether the two grid images correspond or not can be determined through the similarity. For each grid image of the current image, the image trolley calculates the similarity of each grid image of the grid image and each grid image of the previous image of the current image, then obtains the maximum similarity from the similarity of each grid image of the grid image and each grid image of the previous image of the current image, compares the maximum similarity with a similarity threshold, and if the maximum similarity is larger than the similarity threshold, corresponds the grid image of the previous image of the current image corresponding to the maximum similarity to the grid image. And repeating the process to obtain the grid image of the previous image of the current image corresponding to each grid image of the current image, thereby obtaining the mapping relation between each grid image of the current image and each grid image of the previous image. The similarity threshold is set according to practical experience, and the embodiment of the invention is not limited. The current image may be regarded as the last image pass of the current image
For example, the similarity of two grid images is calculated by a calculation formula of normalized product correlation coefficient. The calculation formula of the normalized product correlation coefficient is:
Wherein C cc represents the similarity of two mesh images, (x refi,yrefj) represents the coordinates of one pixel point in the mesh image of the current image, f (x refi,yrefj) represents the gray value corresponding to each pixel point (x refi,yrefh) of the mesh image of the current image, f m represents the average value of the gray values corresponding to each pixel point of the mesh image of the current image, (x curi,ycurj) represents the coordinates of one pixel point in the mesh image of the previous image of the current image, f (x curi,ycurj) represents the gray value corresponding to each pixel point (x curi,ycurj) in the mesh image of the previous image of the current image, g m represents the average value of the gray values corresponding to each pixel point of the previous image of the mesh image of the current image, i and j are positive integers, S 1 represents the set of all pixel points of the mesh image of the current image, S 2 represents the set of all pixel points of the mesh image of the previous image of the current image, and in the case of uniform mesh division, the number of pixels in each of each mesh image of the previous image of the current image is equal.
S303, obtaining the in-plane displacement increment of each grid image of the current image according to the plane coordinates of each grid image of the current image with the mapping relation, the plane coordinates of each grid image of the previous image and the coordinate mapping relation equation; obtaining the out-of-plane displacement increment of each grid image of the current image according to the height coordinates of the center point of each grid image of the current image with the mapping relation and the height coordinates of the center point of each grid image of the previous image; wherein the mapping relation equation is preset;
Specifically, the mesh image of the current image may be regarded as being obtained after the mesh image of the previous image having the mapping relationship is deformed. The image trolley solves a coordinate mapping relation equation through plane coordinates of each grid image of the current image and plane coordinates of each grid image of the previous image with a mapping relation, and can obtain in-plane displacement increment of each grid image of the current image. The coordinate mapping relation equation is preset and reflects the relation between the coordinates of the grid image of the current image with the mapping relation and the coordinates of the grid image of the corresponding previous image, and the coordinate mapping relation equation relates to in-plane displacement increment, namely displacement increment of the plane coordinates of the grid image of the current image relative to the coordinates of the grid image of the corresponding previous image in a first direction and displacement increment of the grid image of the corresponding previous image in a second direction in a plane, and the first direction and the second direction are perpendicular.
The image trolley can obtain the height coordinates of the center point of each grid image of the current image and the height of the center point of each grid image of the previous image, then calculate the displacement of the height coordinates of the center point of the grid image of the current image with the mapping relation and the corresponding height coordinates of the center point of the grid image of the previous image in the height direction, and obtain the out-of-plane displacement increment of the grid image of the current image. The height direction is perpendicular to the plane where the first direction and the second direction are located.
S304, obtaining deformation of each grid image of the current image according to the in-plane displacement increment and the out-of-plane displacement increment of each grid image of the current image;
specifically, for each grid image of the current image, the image trolley can determine deformation of the grid image through in-plane displacement increment and out-of-plane displacement increment of the grid image, wherein the deformation of the grid image of the current image is the deformation of the grid image of the current image relative to the grid image of the previous image of the current image.
S305, obtaining the current surface deformation of the region of interest according to the deformation of each grid image of the current image.
Specifically, after obtaining the deformation of each grid image of the current image, the image trolley combines the deformation of each grid image of the current image, namely, the deformation of each grid image is spliced into the deformation of the current image of the region of interest, and the deformation of the current surface of the region of interest is obtained.
Further, on the basis of the above embodiments, the coordinate mapping relation equation includes:
Wherein, (x curi,ycurj) represents the pixel coordinates of the mesh image of the current image, (x refi,yrefj) represents the pixel coordinates of the mesh image of the previous image of the current image, (x curi,ycurj) corresponds to (x refi,yrefj), (x refc,yrefc) represents the pixel coordinates of the center point of the mesh image of the previous image of the current image, u represents the displacement increment of the mesh image of the current image in the first direction with respect to the mesh image of the corresponding previous image of the current image, v represents the displacement increment of the mesh image of the current image in the second direction with respect to the mesh image of the corresponding previous image of the current image, u rc represents the coordinate conversion increment corresponding to u, v rc represents the coordinate conversion increment corresponding to v, x rc represents the coordinate conversion in the first direction from the mesh image of the previous image of the current image to the mesh image of the current image, and y rc represents the coordinate conversion in the second direction from the mesh image of the previous image of the current image to the mesh image of the current image.
Specifically, the coordinate mapping relation equation comprises u, v,And/>The transformation parameters are formed, and the transformation parameters can be solved by forming an equation set through bringing the pixel coordinates of the grid images of a plurality of current images, the pixel coordinates of the grid images of the previous image of the current image and the pixel coordinates of the center point of the grid image of the previous image of the current image, so that u and v can be obtained, namely the in-plane displacement increment of each grid image of the current image can be obtained.
Fig. 4 is a schematic flow chart of an information feedback method according to a fourth embodiment of the present invention, as shown in fig. 4, further, based on the above embodiments, the obtaining the strain field of the region of interest according to the surface deformation of the region of interest includes:
S401, carrying out smoothing treatment on the surface deformation of the region of interest to obtain a smooth displacement field of the region of interest;
Specifically, the image trolley performs smoothing processing on the surface deformation of the region of interest, and a smooth displacement field of the region of interest can be obtained. The smoothing processing for the surface deformation can adopt a plane elastic finite element analysis method with a penalty function, a delta smoothing algorithm, wavelet transformation and the like, and is selected according to actual needs, and the embodiment of the invention is not limited.
S402, carrying out Green-Lagrange strain calculation on the smooth displacement field of the region of interest to obtain a strain field of the region of interest.
Specifically, the image trolley performs green-lagrangian strain calculation on the smooth displacement field of the region of interest, so that the strain field of the region of interest can be obtained.
Fig. 5 is a flowchart of an information feedback method according to a fifth embodiment of the present invention, as shown in fig. 5, further, based on the strain field of the region of interest, displaying a strain cloud image of the region of interest includes:
S501, obtaining the corresponding color and the strain direction of each grid image according to the strain-color difference relation diagram and the strain of each grid image of the strain field of the region of interest; wherein the strain-color difference relationship map is pre-established;
Specifically, the strain field of the region of interest may be subdivided into strains corresponding to each grid image, and the color corresponding to each grid image may be obtained by querying the color corresponding to the strain corresponding to each grid image through the strain-color difference relationship diagram. The strain direction corresponding to each grid image can be obtained through the strain of each grid image of the strain field of the region of interest. The size of the mesh-divided image of the strain field of the region of interest may be the same as the size of the mesh division in step S301.
The strain-color difference relation diagram is pre-established, a maximum value and a minimum value of the strain are obtained, a color screenshot of the color changing from light to dark is obtained, the minimum value of the strain is corresponding to the lightest color, and the maximum value of the strain is corresponding to the darkest color. And carrying out N equal division on the difference value between the maximum value of the strain and the minimum value of the strain, carrying out N equal division on the color between the lightest color and the darkest color, carrying out one-to-one correspondence on the difference value of the N equal division and the color of the N equal division according to the relation of the strain from small to large and the color from light to dark, and finally establishing a strain-color difference relation graph.
S502, obtaining a strain cloud image of the region of interest according to the colors corresponding to the grid images.
Specifically, after the color and the strain direction corresponding to each grid image of the strain field of the region of interest are obtained, the strain field of the region of interest is displayed in the strain direction corresponding to each grid image to display the corresponding color, and a strain cloud image of the region of interest is obtained.
For example, fig. 6 is a schematic diagram of a strain cloud image of a region of interest according to a sixth embodiment of the present invention, where, as shown in fig. 6, arrows in the graph indicate strain directions of respective grid images of the region of interest, and the darker the color of the arrows indicates the greater the corresponding strain. After the strain field of the region of interest is obtained, the strain of each grid image is corresponding to the color shade, and the darker the color is, the greater the strain is; for ease of viewing, the color is displayed on an arrow identifying the direction of strain. The strain cloud chart shown in fig. 6 can be displayed on the doctor-side equipment, when the doctor operates the main manipulator on the doctor-side equipment, the size and the direction of the strain of the region of interest can be visually seen, the size and the direction of the strain of the region of interest can be changed in real time along with the operation of the doctor, the doctor can intuitively feel the size and the direction of the strain received by the tissue surface, and the larger area and the strain direction in the region of interest can be distinguished through the color, so that the doctor can reasonably judge the operation.
Further, in the strain cloud chart, the strain direction corresponding to each grid image is indicated by the arrow direction, and the magnitude of the strain is indicated by the length of the arrow.
Specifically, in the obtained strain cloud image of the region of interest, the strain direction corresponding to each grid image is represented in the arrow direction, and meanwhile, the size of the strain is represented in the arrow length, and the arrow length corresponding to the larger the strain is. When the size and the direction of the strain corresponding to the grid image are changed, the length and the direction of the arrow corresponding to the grid image are also changed correspondingly, so that a doctor can more intuitively check the change of the size and the direction of the strain of the region of interest.
Fig. 7 is a schematic flow chart of an information feedback method according to a sixth embodiment of the present invention, as shown in fig. 7, further, based on the foregoing embodiments, the information feedback method according to the embodiment of the present invention further includes:
s701, obtaining tissue information of the image of the region of interest according to the image of the region of interest and the tissue identification model; wherein the tissue recognition model is pre-established;
Specifically, the image trolley inputs the image of the region of interest into a tissue recognition model, and tissue in the image of the region of interest is recognized through the tissue recognition model, so that tissue information of the image of the hand part is obtained, wherein the tissue information of the image comprises tissues included in the image and positions of the tissues in the image. Wherein the tissue recognition model is pre-established. The tissue information includes, but is not limited to, blood vessels, fat, muscles, organs, and the like.
S702, obtaining the organization information of each grid image of the region of interest according to the organization information of the image of the region of interest and each grid image of the region of interest;
Specifically, the image trolley performs grid division on the current image of the region of interest, and can obtain each grid image of the region of interest. The respective grid images of the image of the region of interest are corresponding to the organization information of the image of the region of interest, so that the organization information of each grid image of the region of interest can be obtained, i.e. an image of which organization the grid image is can be determined.
For example, based on the position in the image according to the pixel coordinates of the center point of the mesh image, and to which tissue the position belongs in the image, the tissue information of the mesh image can be determined.
S703, obtaining a stress field of the region of interest according to the tissue information of each grid image of the region of interest and the strain field of the region of interest;
Specifically, through the tissue information of each grid image of the region of interest, a stiffness tensor corresponding to each grid image can be determined, each tissue has a unique corresponding stiffness tensor, and the stiffness tensor of the tissue information of the grid image is taken as the stiffness tensor corresponding to the grid image. Based on the strain cloud image of the region of interest and each grid image of the region of interest, the strain corresponding to each grid image of the region of interest can be obtained, and based on the stiffness tensor corresponding to each grid image and the strain corresponding to each grid image, the stress corresponding to each grid image can be calculated and obtained, and the stress field of the region of interest can be obtained through the stress corresponding to each grid image of the region of interest.
Wherein the stiffness tensors of different tissues during the solving of the stress field from the strain field. The rigidity tensors of different tissues can be obtained by in-vitro test experiments on the tissues, and the corresponding relation between the different tissues and the rigidity tensors can be established.
For example, according to the formula σ=cε, the stress σ corresponding to the grid image is calculated, C represents the stiffness tensor corresponding to the grid image, ε represents the strain corresponding to the grid image.
S704, transmitting a stress field of the region of interest to doctor-side equipment.
Specifically, the image trolley transmits the stress field of the region of interest to doctor-side equipment, which can implement force feedback to a main manipulator operated by a doctor according to the stress field of the region of interest.
On the basis of the above embodiments, further, the step of establishing the tissue identification model includes:
Acquiring a surgical sample image and corresponding tag information;
And training to obtain the tissue identification model according to the original model, the surgical sample image and the corresponding tissue label information.
Specifically, a number of images of the actual surgical procedure may be collected and the images may be preprocessed to obtain the surgical sample image. The number of the sample images included in the surgical sample image is set according to actual needs, and the embodiment of the invention is not limited. Labeling each sample image, and labeling different tissue labels in the sample image to serve as tissue label information corresponding to the sample image. It will be appreciated that the tissue can be distinguished by the image, as different tissues exhibit different image characteristics, e.g. blood vessels are tree-rooted and blood is packed inside, while fat has no significant blood veins inside, etc. The preprocessing of the image includes, but is not limited to, image denoising, image normalization processing, etc., and is set according to actual needs, and the embodiment of the invention is not limited.
Based on the surgical sample image and the corresponding tissue label information, training the original model, and training to obtain a tissue identification model, wherein the tissue identification model is used for identifying different tissues in the image of the region of interest. The original model can be selected according to actual needs by adopting a deep learning model and the like, and the embodiment of the invention is not limited. The specific training process of the model is the prior art, and will not be described here in detail.
Fig. 8 is a schematic structural diagram of an information feedback system according to an eighth embodiment of the present invention, and as shown in fig. 8, the information feedback system according to the embodiment of the present invention includes an image acquisition module 801, an image data processing module 802, a data analysis calculation module 803, and an image display module 804, where:
The image acquisition module 801 is connected to the image data processing module 802, the image data processing module 802 is connected to the data analysis and calculation module 803, and the data analysis and calculation module 803 is connected to the image display module 804.
The image acquisition module 801 is configured to acquire an image of a region of interest in an operation field of view, and obtain an image of the region of interest. The image acquisition module 801 may be a three-dimensional electronic endoscope, which includes components such as a miniature industrial camera, an optical sensor, a cold light source, and an optical system. In the operation process, three-dimensional image information of the region of interest can be acquired in real time through the three-dimensional electronic endoscope, so that an anatomical structure can be recognized more easily, and the operation accuracy can be improved. The three-dimensional electronic endoscope is adopted to collect images, each point on the surface of the object is focused on pixels of two CCD camera imaging surfaces of the three-dimensional electronic endoscope, the points on the surface of the object in the two camera images are identified, the same viewpoint is found, imaging parameters are adopted, the shape of the surface of the object is calculated, and the three-dimensional coordinates of the object under the current image are obtained. The imaging parameters comprise the distance between the axes of the two lenses, the focal length of the lenses and other information.
The image data processing module 802 is configured to store the acquired image and perform gray level distribution processing on the image. The image data processing module 802 may include a memory and a processor. The data analysis and calculation module 803 calculates a strain field of the region of interest by using the image data at each time transmitted from the image data processing module 802, and the obtained strain field is displayed in a strain cloud image manner. The data analysis computation module 803 may employ a processor. The image display module 804 is configured to display a strain cloud image.
Further, as shown in fig. 8, the information feedback system provided by the embodiment of the present invention further includes a surgical instrument joint controller 805, where the surgical instrument joint controller 805 is communicatively connected to the data analysis and calculation module 803. The surgical instrument joint controller 805 is used to control the movement of the various joints of the master manipulator.
The data analysis and calculation module 803 is further configured to calculate a stress field of the region of interest based on the strain field of the region of interest, and obtain stress data. The data analysis and calculation module 803 sends stress data to the surgical instrument joint controller 805, and the surgical instrument joint controller 805 performs feedback control on the main manipulator operated by the doctor through the stress data, so that the doctor can feel a corresponding damping force when operating the main manipulator.
Because of the different surface tension of different tissues, it is necessary to identify the different tissues in the image of the region of interest, and the stress calculation of the tissue surface has been performed. The data analysis and calculation module 803 configures an organization recognition model to perform organization recognition.
Fig. 9 is a schematic structural diagram of an image trolley according to a ninth embodiment of the present invention, and as shown in fig. 9, the image trolley according to the embodiment of the present invention includes an image receiving module 901, an image processing module 902 and an image display module 903, where:
The image receiving module 901 is used for receiving an image of a region of interest; the image processing module 902 is configured to obtain a surface deformation of the region of interest according to the image of the region of interest and obtain a strain field of the region of interest according to the surface deformation of the region of interest; the image display module 903 is configured to display a strain cloud image of the region of interest based on the strain field of the region of interest.
Further, based on the above embodiments, the image processing module 902 is specifically configured to:
Performing grid division on the current image of the region of interest to obtain coordinates and gray values of each grid image of the current image;
obtaining the mapping relation between each grid image of the current image and each grid image of the previous image according to the gray value of each grid image of the previous image of the current image and the gray value of each grid image of the current image; wherein the gray values of each grid image of the previous image of the current image are obtained in advance;
Obtaining the in-plane displacement increment of each grid image of the current image according to the plane coordinates of each grid image of the current image with the mapping relation, the plane coordinates of each grid image of the previous image and the coordinate mapping relation equation; obtaining the out-of-plane displacement increment of each grid image of the current image according to the height coordinates of the center point of each grid image of the current image with the mapping relation and the height coordinates of the center point of each grid image of the previous image; wherein the mapping relation equation is preset;
Obtaining deformation of each grid image of the current image according to the in-plane displacement increment and the out-of-plane displacement increment of each grid image of the current image;
and obtaining the current surface deformation of the region of interest according to the deformation of each grid image of the current image.
Further, on the basis of the above embodiments, the coordinate mapping relation equation includes:
Wherein, (x curi,ycurj) represents the pixel coordinates of the mesh image of the current image, (x refi,yrefj) represents the pixel coordinates of the mesh image of the previous image of the current image, (x curi,ycurj) corresponds to (x refi,yrefj), (x refc,yrefc) represents the pixel coordinates of the center point of the mesh image of the previous image of the current image, u represents the displacement increment of the mesh image of the current image in the first direction with respect to the mesh image of the corresponding previous image of the current image, v represents the displacement increment of the mesh image of the current image in the second direction with respect to the mesh image of the corresponding previous image of the current image, u rc represents the coordinate conversion increment corresponding to u, v rc represents the coordinate conversion increment corresponding to v, x rc represents the coordinate conversion in the first direction from the mesh image of the previous image of the current image to the mesh image of the current image, and y rc represents the coordinate conversion in the second direction from the mesh image of the previous image of the current image to the mesh image of the current image.
Further, based on the above embodiments, the image processing module 902 is specifically configured to:
carrying out smoothing treatment on the surface deformation of the region of interest to obtain a smooth displacement field of the region of interest;
and carrying out Green-Lagrange strain calculation on the smooth displacement field of the region of interest to obtain a strain field of the region of interest.
Further, based on the above embodiments, the image display module 803 is specifically configured to:
Obtaining the corresponding color and the strain direction of each grid image according to the strain-color difference relation diagram and the strain of each grid image of the strain field of the region of interest; wherein the strain-color difference relationship map is pre-established;
and obtaining the strain cloud image of the region of interest according to the corresponding color and strain direction of each grid image.
Further, in the strain cloud chart, the strain direction corresponding to each grid image is indicated by the arrow direction, and the magnitude of the strain is indicated by the length of the arrow.
Further, the image processing module 902 is further configured to:
Obtaining tissue information of the image of the region of interest according to the image of the region of interest and the tissue identification model; wherein the tissue recognition model is pre-established;
Obtaining the organization information of each grid image of the region of interest according to the organization information of the image of the region of interest and each grid image of the region of interest;
obtaining a stress field of the region of interest according to the tissue information of each grid image of the region of interest and the strain field of the region of interest;
transmitting the stress field of the region of interest to a doctor-side device.
The embodiment of the image trolley provided by the embodiment of the invention can be specifically used for executing the processing flow of each method embodiment, and the functions of the embodiment of the image trolley are not repeated herein, and can be referred to in the detailed description of the method embodiment.
The embodiment of the invention provides an information feedback system, which comprises the image trolley in any embodiment.
Fig. 10 is a schematic physical structure of an electronic device according to a tenth embodiment of the present invention, as shown in fig. 10, the electronic device may include: a processor 1001, a communication interface (Communications Interface) 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 perform communication with each other through the communication bus 1004. The processor 1001 may call logic instructions in the memory 1003 to perform the following method: receiving an image of a region of interest; obtaining the surface deformation of the region of interest according to the image of the region of interest; obtaining a strain field of the region of interest according to the surface deformation; and displaying a strain cloud image of the region of interest based on the strain field.
Further, the logic instructions in the memory 1003 described above may be implemented in the form of software functional units and sold or used as a separate product, and may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a grid device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, are capable of performing the methods provided by the above-described method embodiments, for example comprising: receiving an image of a region of interest; obtaining the surface deformation of the region of interest according to the image of the region of interest; obtaining a strain field of the region of interest according to the surface deformation; and displaying a strain cloud image of the region of interest based on the strain field.
The present embodiment provides a computer-readable storage medium storing a computer program that causes the computer to execute the methods provided by the above-described method embodiments, for example, including: receiving an image of a region of interest; obtaining the surface deformation of the region of interest according to the image of the region of interest; obtaining a strain field of the region of interest according to the surface deformation; and displaying a strain cloud image of the region of interest based on the strain field.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description of the present specification, reference to the terms "one embodiment," "one particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (10)
1. An information feedback method, comprising:
Receiving an image of a region of interest;
obtaining the surface deformation of the region of interest according to the image of the region of interest;
Obtaining a strain field of the region of interest according to the surface deformation;
and displaying a strain cloud image of the region of interest based on the strain field.
2. The method of claim 1, wherein the obtaining a surface deformation of the region of interest from the image of the region of interest comprises:
Performing grid division on the current image of the region of interest to obtain coordinates and gray values of each grid image of the current image;
obtaining the mapping relation between each grid image of the current image and each grid image of the previous image according to the gray value of each grid image of the previous image of the current image and the gray value of each grid image of the current image; wherein the gray values of each grid image of the previous image of the current image are obtained in advance;
Obtaining the in-plane displacement increment of each grid image of the current image according to the plane coordinates of each grid image of the current image with the mapping relation, the plane coordinates of each grid image of the previous image and the coordinate mapping relation equation; obtaining the out-of-plane displacement increment of each grid image of the current image according to the height coordinates of the center point of each grid image of the current image with the mapping relation and the height coordinates of the center point of each grid image of the previous image; wherein the mapping relation equation is preset;
Obtaining deformation of each grid image of the current image according to the in-plane displacement increment and the out-of-plane displacement increment of each grid image of the current image;
and obtaining the current surface deformation of the region of interest according to the deformation of each grid image of the current image.
3. The method of claim 1, wherein the obtaining a strain field of the region of interest from the surface deformation of the region of interest comprises:
carrying out smoothing treatment on the surface deformation of the region of interest to obtain a smooth displacement field of the region of interest;
and carrying out Green-Lagrange strain calculation on the smooth displacement field of the region of interest to obtain a strain field of the region of interest.
4. The method of claim 1, wherein the displaying a strain cloud image of the region of interest based on the strain field of the region of interest comprises:
Obtaining the corresponding color and the strain direction of each grid image according to the strain-color difference relation diagram and the strain of each grid image of the strain field of the region of interest; wherein the strain-color difference relationship map is pre-established;
and obtaining the strain cloud image of the region of interest according to the corresponding color and strain direction of each grid image.
5. The method according to any one of claims 1 to 4, wherein in the strain cloud, a strain direction corresponding to each grid image is indicated by an arrow direction, and a magnitude of strain is indicated by a length of the arrow.
6. The method as recited in claim 1, further comprising:
Obtaining tissue information of the image of the region of interest according to the image of the region of interest and the tissue identification model; wherein the tissue recognition model is pre-established;
Obtaining the organization information of each grid image of the region of interest according to the organization information of the image of the region of interest and each grid image of the region of interest;
obtaining a stress field of the region of interest according to the tissue information of each grid image of the region of interest and the strain field of the region of interest;
transmitting the stress field of the region of interest to a doctor-side device.
7. An image trolley is characterized by comprising an image receiving module, an image processing module and an image display module, wherein:
the image receiving module is used for receiving the image of the region of interest;
The image processing module is used for obtaining the surface deformation of the region of interest according to the image of the region of interest and obtaining the strain field of the region of interest according to the surface deformation of the region of interest;
the image display module is used for displaying a strain cloud image of the region of interest based on the strain field of the region of interest.
8. An information feedback system comprising the image trolley of claim 7.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 6 when executing the computer program.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211292310.6A CN117942164A (en) | 2022-10-21 | 2022-10-21 | Information feedback method, image trolley and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211292310.6A CN117942164A (en) | 2022-10-21 | 2022-10-21 | Information feedback method, image trolley and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117942164A true CN117942164A (en) | 2024-04-30 |
Family
ID=90791138
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211292310.6A Pending CN117942164A (en) | 2022-10-21 | 2022-10-21 | Information feedback method, image trolley and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117942164A (en) |
-
2022
- 2022-10-21 CN CN202211292310.6A patent/CN117942164A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4171833B2 (en) | Endoscope guidance device and method | |
| US20170296032A1 (en) | Branching structure determination apparatus, method, and program | |
| JP6083103B2 (en) | Image complementation system for image occlusion area, image processing apparatus and program thereof | |
| US20150255005A1 (en) | Movement evaluation device and program therefor | |
| CN111067468B (en) | Method, apparatus, and storage medium for controlling endoscope system | |
| CN112618026B (en) | Remote operation data fusion interactive display system and method | |
| US20190066390A1 (en) | Methods of Using an Imaging Apparatus in Augmented Reality, in Medical Imaging and Nonmedical Imaging | |
| WO2022073342A1 (en) | Surgical robot and motion error detection method and detection device therefor | |
| EP3910592A1 (en) | Image matching method, apparatus and device, and storage medium | |
| CN108814717B (en) | Surgical robot system | |
| US20220361733A1 (en) | Endoscopic examination supporting apparatus, endoscopic examination supporting method, and non-transitory recording medium recording program | |
| CN112802185B (en) | Endoscope image three-dimensional reconstruction method and system facing minimally invasive surgery space perception | |
| CN114022547A (en) | Endoscope image detection method, device, equipment and storage medium | |
| EP2901935A1 (en) | Method and device for generating virtual endoscope image, and program | |
| US20170270678A1 (en) | Device and method for image registration, and non-transitory recording medium | |
| CN112704566B (en) | Surgical consumable checking method and surgical robot system | |
| JP2016209399A (en) | Image processing device and method, and computer program | |
| CN112712016B (en) | Surgical instrument identification method, identification platform and medical robot system | |
| CN117942164A (en) | Information feedback method, image trolley and electronic equipment | |
| CN115209783A (en) | Processing device, endoscope system, and method for processing captured image | |
| CN113317874B (en) | Medical image processing device and medium | |
| CN115281584B (en) | Flexible endoscope robot control system and flexible endoscope robot simulation method | |
| WO2024029502A1 (en) | Endoscopic examination assistance device, endoscopic examination assistance method, and recording medium | |
| CN116864098B (en) | Medical instrument stress information acquisition method and system based on position information | |
| CN113362324B (en) | Bone health detection method and system based on video image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |