CN115699782A

CN115699782A - Medical observation system, image processing method, and program

Info

Publication number: CN115699782A
Application number: CN202180043119.5A
Authority: CN
Inventors: 户松景
Original assignee: Sony Group Corp
Current assignee: Sony Group Corp
Priority date: 2020-06-22
Filing date: 2021-06-08
Publication date: 2023-02-03
Also published as: US20230218143A1; US12070183B2; WO2021261236A1

Abstract

The present disclosure relates to a medical observation system, an image processing method, and a program that enable image display more suitable for an operator. A correction processing unit performs correction processing on a wide-angle image obtained by capturing a medical target, a trimming processing unit moves a trimming area of a display image trimmed from the wide-angle image, and a correction control unit controls the correction processing according to a state of the trimming area with respect to the wide-angle image. The present disclosure may be applied to, for example, an endoscopic surgical system.

Description

Medical observation system, image processing method, and program

Technical Field

The present disclosure relates to a medical observation system, an image processing method, and a program, and more particularly to a medical observation system, an image processing method, and a program capable of providing an image display more helpful to a surgeon.

Background

Conventionally, as disclosed in PTL 1, for example, in order to ensure a viewing angle, an endoscope is provided with a wide-angle lens at its distal end.

In a wide-angle image, the closer to its edge, the more distorted. Therefore, in general, an image cropped from the distortion-corrected wide-angle image is displayed as a display image on a display or the like.

[ list of references ]

[ patent document ]

[PTL 1]

JPH10-290777A

Disclosure of Invention

[ problem ] to

However, for a surgeon performing an operation while observing a display image as a trimming image instead of the entire wide-angle image, it is not easy to recognize the position of a trimming area for trimming the display image in the entire wide-angle image.

The present disclosure has been made in view of such a situation, and provides an image display more helpful to a surgeon.

[ solution of problem ]

The medical observation system according to the present invention is a medical observation system including: a correction processing unit that performs correction processing on a wide-angle image of a captured image as a medical target; a trimming processing unit that moves a trimming area of a display image trimmed from the wide-angle image; and a correction control unit that controls the correction processing according to a relative state of the trimming area with respect to the wide-angle image.

An image processing method of the present invention is an image processing method executed by a medical observation system, including: performing correction processing on a wide-angle image of a captured image as a medical target; moving a trimming area of a display image trimmed from the wide-angle image; and controlling the correction processing according to a relative state of the trimming area with respect to the wide-angle image.

A program according to the present disclosure is a program that causes a computer to perform the operations of: performing correction processing on a wide-angle image of a captured image as a medical target; moving a trimming area of a display image trimmed from the wide-angle image; and controlling the correction processing according to a relative state of the trimming area with respect to the wide-angle image.

In the present disclosure, correction processing is performed on a wide-angle image of a captured image as a medical target, a trimming area of a display image trimmed from the wide-angle image is moved, and the correction processing is controlled according to a relative state of the trimming area with respect to the wide-angle image.

Drawings

Fig. 1 is a schematic view of an endoscopic surgical system applying techniques according to the present disclosure.

Fig. 2 is a schematic view of a microsurgical system applying techniques in accordance with the present disclosure.

Fig. 3 is a block diagram showing a configuration example of a medical observation system according to an embodiment of the present disclosure.

Fig. 4 is a flowchart showing a flow of a basic operation of the medical observation system.

Fig. 5 is a flowchart showing the flow of the operation of the correction control unit according to the first embodiment.

Fig. 6 is a schematic diagram showing an example of trimming a display image.

Fig. 7 is a schematic diagram showing an example of trimming a display image.

Fig. 8 is a flowchart showing the flow of the operation of the correction control unit according to the second embodiment.

Fig. 9 is a schematic diagram for explaining a correction amount according to the moving speed of the trimming area.

Fig. 10 is a schematic diagram showing the relationship between the moving speed of the trimming area and the correction amount of the wide-angle image.

Fig. 11 is a flowchart showing a flow of an operation of the display control unit according to the third embodiment.

Fig. 12 is a schematic diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 13 is a diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 14 is a flowchart showing a flow of an operation of the display control unit according to the fourth embodiment.

Fig. 15 is a schematic diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 16 is a schematic diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 17 is a block diagram showing another configuration example of the medical observation system according to the embodiment of the present disclosure.

Fig. 18 is a flowchart showing a flow of an operation of the display control unit according to the fifth embodiment.

Fig. 19 is a diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 20 is a schematic diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 21 is a flowchart showing a flow of an operation of the display control unit according to the sixth embodiment.

Fig. 22 is a schematic diagram showing an example of displaying an image in the display mode according to the position of the trimming area.

Fig. 23 is a block diagram showing an example of the hardware configuration of the control apparatus.

Detailed Description

Hereinafter, modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described. Here, the description will be made in the following order.

1. Embodiments applicable in accordance with the techniques of this disclosure

2. Configuration example and basic operation of medical viewing system

3. First embodiment (correction control 1 according to the moving state of the trimming area)

4. Second embodiment (correction control according to the moving state of the trimming area 2)

5. Third embodiment (display control according to position of trimming area 1)

6. Fourth embodiment (display control according to position of trimming area 2)

7. Other configuration examples of medical viewing system

8. Fifth embodiment (display control according to position of trimming area 3)

9. Sixth embodiment (display control according to position of trimming area 4)

10. Hardware configuration

<1. Examples to which the technique according to the present disclosure is applicable >

The technique according to the present disclosure is applicable to a medical observation system capable of moving a cutout area of a display image that is cut out from a wide-angle image.

(endoscopic surgery system)

Fig. 1 is a schematic view of an endoscopic surgical system 1 to which techniques according to the present disclosure are applied.

Fig. 1 shows a state in which a surgeon (doctor) SG performs an operation on a patient PT on a patient bed BD.

As shown in fig. 1, the endoscopic surgery system 1 includes an endoscope 10, other surgical instruments 20, and a support arm device 30 for supporting the endoscope 10. The endoscopic surgery system 1 includes various types of devices used for endoscopic surgery, although not shown.

In endoscopic surgery, a number of tubular laparotomy instruments, referred to as trocars 41a-41d, are inserted into the abdominal wall, rather than cutting the abdominal wall to open the abdomen. Then, the lens barrel 11 of the endoscope 10 and other surgical instruments 20 are inserted into the body cavity of the patient PT from the trocars 41a to 41 d. In the example of fig. 1, as other surgical instruments 20, a pneumoperitoneum tube 21, an energy treatment instrument 22 for incising and dissecting a tissue and sealing a blood vessel by high-frequency current or ultrasonic vibration, and forceps 23 are inserted into a body cavity of a patient PT. However, the surgical instrument 20 shown in the drawings is merely an example, and various types of surgical instruments that are generally used in endoscopic surgery, such as forceps and shrinkers, may be used as the surgical instrument 20.

An image of the surgical site in the body cavity of the patient PT captured by the endoscope 10 is displayed on the display device 50. The surgeon SG performs a treatment such as excision of an affected part using the energy treatment instrument 22 or the forceps 23 while observing an image of the surgical site displayed on the display device 50 in real time. Although not shown, the operator SG or an assistant during surgery supports the pneumoperitoneum tube 21, the energy treatment device 22, and the forceps 23.

The support arm arrangement 30 comprises an arm portion 32 extending from a base portion 31. In the example of fig. 1, the arm portion 32 includes

joints

33a, 33b, and 33c and

links

34a and 34b, and is driven under the control of a control device (not shown). The endoscope 10 is supported by the arm 32, and its position and orientation are controlled. Thereby, the endoscope 10 can be stably fixed in position.

The endoscope 10 includes a lens barrel 11 and a camera 12 connected to a bottom end of the lens barrel 11, and a portion of the lens barrel 11 having a predetermined length from a distal end thereof is inserted into a body cavity of a patient PT. Although the endoscope 10 configured as a so-called rigid endoscope having the rigid lens barrel 11 is shown in the example of fig. 1, the endoscope 10 may be configured as a so-called flexible endoscope having the flexible lens barrel 11.

The distal end of the lens barrel 11 is provided with an opening into which a wide-angle lens serving as an objective lens is fitted. A light source device (not shown) is connected to the endoscope 10, and light generated by the light source device is guided to the distal end of the lens barrel 11 by a light guide extending inside the lens barrel 11, and irradiates an observation target in the body cavity of the patient PT through a wide-angle lens. The endoscope 10 may be a direct-view endoscope, or may be a see-through endoscope or a side-view endoscope.

An optical system and an imaging element are provided inside the camera 12, and light reflected from an observation target (observation light) is focused on the imaging element through the optical system. The observation light is photoelectrically converted by the capture element, and an electric signal corresponding to the observation light, that is, an image signal corresponding to an observation image is generated. The image signal is transmitted as RAW data to a Camera Control Unit (CCU) (not shown). The camera 12 has a function of adjusting the magnification and the focal length by driving the optical system as appropriate.

For example, the camera 12 may be provided with a plurality of imaging elements in order to cope with stereoscopic vision (3D display) or the like. In this case, a plurality of relay optical systems are provided inside the lens barrel 11 to guide the observation light to each of the plurality of imaging elements.

Although not shown, the endoscopic surgery system 1 is provided with an input device serving as an input interface of the endoscopic surgery system 1. The input device includes, for example, a mouse, a keyboard, a touch panel, a switch, a foot switch 61, a lever, and the like. In the case where the input device is configured as a touch panel, the touch panel may be provided on the display surface of the display device 50.

The input device may include a device worn by the surgeon, such as a glasses-type wearable device or Head Mounted Display (HMD), a camera that can detect the movement of the surgeon, or the like, to be able to receive various types of input according to the gesture or line of sight of the surgeon.

With this configuration, the surgeon SG can perform various processes while viewing the display image of the surgical site displayed on the display device 50.

(microsurgical system)

Fig. 2 is a schematic diagram of a microsurgical system 100 applying techniques in accordance with the present disclosure.

Fig. 2 shows a state in which the surgeon SG performs an operation on the patient PT on the bed BD using a surgical instrument 121 such as a scalpel, forceps, tweezers, or the like.

The surgical video microscope device 110 is installed beside the hospital bed BD.

The surgical video microscope device 110 includes a base 111 serving as a base, an arm 112 extending from the base 111, and an imaging unit 115 connected to a distal end of the arm 112. The surgical video microscope device 110 can be an endoscope.

The arm 112 has a plurality of

joints

113a, 113b, and 113c, a plurality of

links

114a and 114b connected by the

joints

113a and 113b, and a capturing portion 115 attached to the tip of the arm 112.

In the example of fig. 2, the arm 112 has three engaging portions 113a to 113c and two

links

114a and 114b for the sake of simplicity. In fact, in consideration of the degree of freedom of the position and posture of the imaging unit 115, the number and shape of the engaging portions 113a to 113c and the

links

114a and 114b, the direction of the drive shafts of the engaging portions 113a to 113c, and the like may be appropriately set so as to achieve the desired degree of freedom.

The joints 113a to 113c have a function of rotatably connecting the

links

114a and 114b, and drive of the arm 112 is controlled by driving the joints 113a to 113c to rotate.

The imaging unit 115 is a unit including an optical system for acquiring an optical image of a subject to acquire an image of an imaging target, and is configured, for example, as a microscope equipped with a camera capable of capturing a moving image and a still image. In the embodiments of the present disclosure, a wide-angle lens is applied as an optical system. As shown in fig. 2, the orientation and position of the imaging unit 115 are controlled by the surgical video microscope device 110 so that the imaging unit 115 attached to the distal end of the arm 112 can capture the state of the surgical site of the patient PT. The imaging unit 115 may be configured to be detachable from the arm portion 112.

The display device 150 is mounted in a position facing the surgeon SG. For example, after being subjected to various types of image processing by an image processing device built in or externally provided to the surgical video microscope device 110, an image of the surgical site acquired by the imaging unit 115 is displayed on the display device 150.

With this configuration, the surgeon SG can perform various treatments while viewing the display image of the surgical site displayed on the display device 150.

<2. Example of configuration and basic operation of medical Observation System >

Next, a configuration example and a basic operation of a medical observation system that can be applied to the endoscopic surgery system 1 or the microsurgical system 100 described above are described.

(configuration example of medical Observation System)

Fig. 3 is a block diagram showing a configuration example of the medical observation system of the embodiment of the present invention.

The medical viewing system 200 shown in fig. 3 includes a camera 210, an input device 220, a display device 230, and a control device 240.

The camera 210 corresponds to the endoscope 10 (the camera 12) of the endoscopic surgery system 1 or the microscope (the imaging unit 115) of the microsurgical system 100, and includes an optical system and an imaging element. The wide-angle lens is used as an optical system included in the camera 210. Specifically, the camera 210 acquires a wide-angle image of the treatment target and provides the wide-angle image to the control device 240. The wide-angle image is a moving image of the surgical site captured in real time, but may be a still image. As the wide-angle lens, for example, an ultra wide-angle lens capable of obtaining an angle of view of 135 degrees or 140 degrees is used, but a fisheye lens capable of obtaining a larger angle of view (for example, an angle of view close to 180 degrees) is applicable. As the wide-angle lens, a wide-angle lens capable of obtaining an angle of view of about 60 to 100 degrees may be applied.

The input device 220 includes an operation device such as a button, a switch, a mouse, a joystick, or a touch panel, and an audio input device such as a microphone, and provides an operation signal indicating an operation by a user, specifically, an operator of the medical observation system 200 to the control device 240.

The display device 230 is configured as a monitor for displaying an image output from the control device 240. The user (surgeon) observes the image displayed on the display device 230.

The control device 240 is configured as an image processing device to which the technique according to the present disclosure is applied, and is configured as, for example, a Camera Control Unit (CCU) including a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and the like.

The control device 240 generally controls image capturing with the camera 210 and display with the display device 230 based on an operation signal from the input device 220. For example, the control device 240 cuts out a part of the wide-angle image obtained from the camera 210, and outputs the cut-out image to the display device 230 as a display image.

The control device 240 includes a correction processing unit 241, a trimming processing unit 242, a display control unit 243, and a correction control unit 244.

The correction processing unit 241 performs correction processing on the wide-angle image obtained from the camera 210. Specifically, the correction processing unit 241 performs distortion correction on the distorted wide-angle image to generate a corrected wide-angle image without distortion, and supplies the corrected wide-angle image to the clipping processing unit 242.

In a known wide-angle image, the closer to the edge thereof, the darker, i.e., the smaller the amount of light. In this viewpoint, the correction processing unit 241 may perform luminance correction on the wide-angle image to generate a corrected wide-angle image having constant luminance.

The correction processing unit 241 may also supply the wide-angle image, which is not subjected to the correction processing, to the clipping processing unit 242 as it is, under the control of a correction control unit 244, which will be described later.

The clipping processing unit 242 clips a part of the wide-angle image obtained from the correction processing unit 241, and supplies the clipped image as a display image to the display control unit 243. Based on detecting a trigger for moving a trimming area, a trimming area for trimming a display image from a wide-angle image is moved on the wide-angle image. For example, the trimming area is moved in real time on the wide-angle image based on an operation signal instructing a user operation through the input device 220. For example, in a case where an image of a surgical instrument in the wide-angle image is recognized and the cutting area is moved to follow the movement of the surgical instrument whose image has been recognized, the cutting area is moved in real time on the wide-angle image based on a detection signal indicating that the movement of the surgical instrument to be followed has been detected.

The clipping processing unit 242 also supplies relative state information indicating the relative state of the clipping area with respect to the wide-angle image to the display control unit 243 and the correction control unit 244. The relative state includes at least one of a moving state of the trimming area in the wide-angle image (at least one of presence or absence of movement, moving speed, moving direction, and the like) and a position of the trimming area in the wide-angle image.

The display control unit 243 causes the display device 230 to display the display image obtained from the trimming processing unit 242 by controlling the display device 230. The display control unit 243 can also control display of the display image in accordance with the relative state of the trimming area with respect to the wide-angle image, which is indicated by the relative state information obtained from the trimming processing unit 242.

The correction control unit 244 controls the correction processing of the correction processing unit 241 according to the relative state of the cropped area and the wide-angle image indicated by the relative state information obtained from the cropping processing unit 242.

Details of the control of the correction process are described in the following embodiments. In short, when the relative state of the trimming area changes, the correction control unit 244 disables the correction process or adjusts the correction amount for the correction process according to the change of the relative state. When the relative state of the trimming area is not changed without change, the correction control unit 244 enables the correction process.

(basic operation of medical Observation System)

Next, the flow of the basic operation of the medical observation system 200 will be described with reference to the flowchart of fig. 4.

For example, the process of fig. 4 is started in a case where the relative state of the trimming area and the wide-angle image is constant, that is, in a case where the correction process is performed on the wide-angle image from the camera 210 and then the display image trimmed from the corrected wide-angle image is displayed on the display device 230.

In step S1, the trimming processing unit 242 of the control device 240 determines whether a trigger for moving the trimming area has been detected. For example, step S1 is repeated until a user operation for moving the trimming area is received, and when an operation for moving the trimming area is received, the process proceeds to step S2.

The user operation for moving the cutting area may be received based on a voice indicating the movement of the cutting area, or may be received by an operating device such as a joystick. A user operation for moving the trimming area may be received based on a result of detecting the line of sight of the user. As described above, in step S1, the movement of the surgical instrument to be followed in the wide-angle image may be detected, and then the process may proceed to step S2.

In step S2, the trimming processing unit 242 of the control device 240 starts moving the trimming area based on the user operation to move the trimming area. This will change the relative state of the cropped area to the wide-angle image.

Thus, in step S3, the correction control unit 244 of the control device 240 controls the correction processing of the correction processing unit 241 according to the relative state of the trimming area and the wide-angle image.

Specifically, when the relative state of the trimming area changes, the correction control unit 244 disables the correction process or adjusts the correction amount for the correction process according to the change of the relative state. When the relative state of the trimming area does not change, the correction control unit 244 enables the correction process.

In step S4, the clipping processing unit 242 of the control device 240 clips a display image from the wide-angle image or the original wide-angle image subjected to the correction processing by the correction processing unit 241.

In step S5, the display control unit 243 of the control device 240 causes the display device 230 to display a display image cropped from the wide-angle image.

As described above, when the relative state of the trimming area with respect to the wide-angle image changes, at least the display image is cropped from the wide-angle image on which the correction processing is not completely performed.

In the case where it is impossible to recognize the position of the cutting area in the entire wide-angle image, even if the cutting area is at the edge of the wide-angle image, the surgeon cannot recognize that the cutting area is prevented from moving further, resulting in a risk of hindering the progress of the operation.

In contrast, according to the above-described processing, when the surgeon moves the cutting area so that the cutting area is at the edge of the wide-angle image, the distortion or darkness of the display image enables the surgeon to recognize that the cutting area is prevented from further movement. Thus, an image display more helpful to the surgeon can be provided.

A specific embodiment of correction control according to the relative state of the trimming area and the wide-angle image will be described below. In the embodiment described below, it is assumed that distortion correction is performed on a wide-angle image by the correction processing unit 241. However, the luminance correction may be performed thereon as described above.

<3. First embodiment (correction control 1 according to the moving state of the trimming area) >

Fig. 5 is a flowchart showing the flow of operation of the correction control unit 244 according to the first embodiment. The correction control unit 244 of the present embodiment disables or enables the correction processing by the correction processing unit 241 depending on whether the trimming area is moving or stopping in the wide-angle image.

In step S11, the correction control unit 244 determines whether the trimming area is moving.

If it is determined in step S11 that the trimming area is moving, the processing proceeds to step S12, and the correction control unit 244 invalidates the correction processing by the correction processing unit 241.

On the other hand, if it is determined in step S11 that the trimming area has not moved, that is, if the trimming area is stopped, the process proceeds to step S13, and then the correction control unit 244 enables the correction process by the correction processing unit 241.

Clipping of a display image according to the present embodiment will now be described with reference to fig. 6 and 7. For convenience of explanation, the object of the wide-angle image is assumed to be a checkerboard pattern in which a distorted image is easily recognized, and the same applies to the following embodiments.

As shown in fig. 6, in the case where the trimming area CA is located at the center of the wide-angle image 310, the trimming area CA is movable upward, downward, leftward and rightward. From this state, when the trimming area CA is moved to the lower right based on the user operation of moving the trimming area CA, the display image 311 corresponding to the trimming area CA is trimmed from the wide-angle image 310 as it is without being subjected to the correction processing (distortion correction).

On the other hand, as shown in fig. 7, in a case where the clipping area CA is stopped at the lower right of the wide-angle image 310, the display image 312 corresponding to the clipping area CA is clipped from the corrected wide-angle image 310c on which the correction processing (distortion correction) has been performed on the wide-angle image 310.

According to the above-described processing, when the surgeon moves the trimming area, the closer the trimming area is to the edge of the wide-angle image, the more distorted the display image is, so that the trimming area can be easily recognized as being close to the edge of the wide-angle image. When the surgeon stops the cutting area, even if the cutting area is located near the edge of the wide-angle image, a displayed image without distortion is displayed so that the operation can be smoothly performed.

As described above, according to the present embodiment, it is possible to provide an image display more helpful to the surgeon.

<4. Second embodiment (correction control 2 according to the moving state of the trimming area) >

Fig. 8 is a flowchart showing the flow of operation of the correction control unit 244 according to the second embodiment. The correction control unit 244 of the present embodiment sets the correction amount of the correction processing by the correction processing unit 241 according to the moving speed of the trimming area in the wide-angle image.

In step S21, the correction control unit 244 determines whether the moving speed of the trimming area exceeds the first speed.

If it is determined in step S21 that the moving speed of the trimming area exceeds the first speed, the process proceeds to step S22, and the correction control unit 244 then sets the correction amount for the correction process by the correction processing unit 241 to 0%.

On the other hand, if it is determined in step S21 that the moving speed of the trimming area does not exceed the first speed, the process proceeds to step S23, and then the correction control unit 244 determines whether the moving speed of the trimming area exceeds the second speed. The second speed is a moving speed lower than the first speed.

If it is determined in step S23 that the moving speed of the trimming area exceeds the second speed, the process proceeds to step S24, and then the correction control unit 244 sets the correction amount for the correction process of the correction processing unit 241 to 50%. The correction amount set in step S24 is not limited to 50%, and may be another value between 0% and 100%.

On the other hand, if it is determined in step S23 that the moving speed of the trimming area does not exceed the second speed, the process proceeds to step S24, and then the correction control unit 244 sets the correction amount for the correction process by the correction processing unit 241 to 100%.

Clipping of a display image according to the present embodiment will now be described with reference to fig. 9. In fig. 9, the moving speed of the cutting area CA is represented by the length and thickness of the block arrow.

As shown in the left part of fig. 9, when the trimming area CA is moved relatively fast, the display image corresponding to the trimming area CA is trimmed from the wide-angle image 320 obtained by the correction processing with the correction amount of 0% (without being subjected to the correction processing as it is).

As shown in the center of fig. 9, when the trimming area CA is moved relatively slowly, the display image corresponding to the trimming area CA is trimmed from the corrected wide-angle image 320c1 to which the correction processing with the correction amount of 50% has been applied to the wide-angle image 320.

As shown on the right side of fig. 9, when the position of the trimming area CA is finely adjusted or stopped, the display image corresponding to the trimming area CA is trimmed from the corrected wide-angle image 320c2 to which the correction processing with the correction amount of 100% has been applied to the wide-angle image 320.

According to the above-described processing, when the surgeon moves the trimming area quickly, the rough position where the trimming area is located in the wide-angle image can be determined from the display image trimmed from the wide-angle image on which the correction processing is not performed. Further, when the surgeon slowly moves the trimming area, it is possible to determine the position of the trimming area in the wide-angle image from the display image trimmed from the wide-angle image with less distortion, and then determine the exact position of the trimming area.

In the process of fig. 8, the correction amount for the correction process is set stepwise according to whether or not the moving speed of the trimming area exceeds a predetermined threshold (first speed or second speed). However, the correction amount for the correction process may be changed according to the moving speed of the trimming area. For example, the correction amount for the correction process may be linearly changed.

For example, as shown in fig. 10, when the moving speed of the trimming area is lower than the first speed v1 and higher than the second speed v2, the correction amount for the correction process may be increased (decreased) in accordance with the decrease (increase) of the moving speed.

The embodiment in which the correction control unit 244 controls the correction process of the correction processing unit 241 according to the moving state of the trimming area in the wide-angle image has been described above. The correction control unit 244 is not limited to this embodiment, and may control the correction processing by the correction processing unit 241 according to the position of the cropped area in the wide-angle image.

As described above, the display control unit 243 can also control the display of the display image according to the relative state of the trimming area and the wide-angle image. Now, an embodiment in which the display control unit 243 controls the display of the display image according to the relative state of the trimming area and the wide-angle image will be described below.

<5. Third embodiment (display control 1 according to the position of the trimming area) >

Fig. 11 is a flowchart showing a flow of the operation of the display control unit 243 according to the third embodiment. The display control unit 243 according to the present embodiment changes the display mode of the display image according to the distance between the trimming area and the edge of the wide-angle image.

In step S31, the display control unit 243 determines whether the trimming area is within an area defined by 20% of the outer periphery of the wide-angle image. Here, if the trimming area does not include an area defined by 20% of the outer periphery of the wide-angle image, it is determined that the trimming area is within an area defined by 20% of the outer periphery of the wide-angle image.

If it is determined in step S31 that the trimming area is within the area defined by 20% of the outer periphery of the wide-angle image, the process proceeds to step S32, and then the display control unit 243 causes the display device 230 to display the corresponding display image in the first display mode. The first display mode is a display informing that the trimming area is near the center of the wide-angle image.

On the other hand, if it is determined in step S31 that the trimming area is not within the area defined by 20% of the outer periphery of the wide-angle image, the process proceeds to step S33, and then the display control unit 243 determines whether the distance between the trimming area and the edge of the wide-angle image is greater than or equal to a predetermined threshold value.

If it is determined in step S33 that the distance between the trimming area and the edge of the wide-angle image is greater than or equal to the predetermined threshold, the process proceeds to step S34, and then the display control unit 243 causes the display device 230 to display the corresponding display image in the second display mode. The second display mode is a display that notifies that the trimming area is close to the edge of the wide-angle image.

On the other hand, if it is determined in step S33 that the distance between the cropped area and the edge of the wide-angle image is not greater than or equal to the predetermined threshold, the process proceeds to step S35, and then the display control unit 243 causes the display device 230 to display the corresponding display image in the third display mode. The third display mode is a display notifying that the trimming area is at a position very close to the edge of the wide-angle image.

A display image in a display mode according to the position of a trimming area according to the present embodiment will now be described with reference to fig. 12 and 13. In fig. 12, it is assumed that the trimming area moves in the wide-angle image. Thus, the display image is cropped from the wide-angle image in which the correction processing is not performed.

As shown in fig. 12, for example, when the trimming area CA _1 is substantially at the center of the wide-angle image 330 on which the correction processing is not performed, a display image 331 with a predetermined frame added is displayed as indicated by an arrow # 1.

When the trimming area CA _2 is at a position deviated from the center of the wide angle image 330 and the trimming area includes an area defined by 20% of the outer periphery of the wide angle image 330, as indicated by an arrow #2, a display image 332 with an added two-line frame is displayed.

When the trimming area CA _3 is at a position very close to the edge of the wide-angle image 330, as indicated by an arrow #3, the display image 333 with the three-line frame added is displayed.

In fig. 13, it is assumed that the trimming area stops in the wide-angle image. Accordingly, the display image is cropped from the corrected wide-angle image on which the correction processing has been performed.

As shown in fig. 13, for example, when the trimming area CA _11 is approximately at the center of the correction wide-angle image 330c on which the correction processing has been performed, a display image 336 with a predetermined frame added is displayed as indicated by an arrow # 11.

When the trimming area CA _12 is at a position deviated from the center of the wide-angle image 330c and the trimming area includes an area defined by 20% of the outer periphery of the corrected wide-angle image 330c, as indicated by an arrow #12, the display image 337 with the added double-line frame is displayed.

When the trimming area CA _13 is located at a position very close to the edge of the corrected wide-angle image 330c, as indicated by an arrow #13, a display image 338 with the three-line frame added is displayed.

In fig. 12 and 13, the line type of the frame of the display image is changed according to the distance between the trimming area and the edge of the wide-angle image. However, the color and thickness of the frame displaying the image may be changed.

For example, the frame to be added to the display image may be blue for a cropped area near the center of the wide angle image, yellow for a cropped area at a position including an area defined by 20% of the outer periphery of the wide angle image, and red for a cropped area at a position very close to the edge of the wide angle image.

Instead of or in addition to changing the display mode added to the entire frame of the display image, the portion (side) of the frame added to the display image near or in contact with the edge of the wide-angle image may be changed according to the distance between the trimming area and the edge of the wide-angle image. In this case, the surgeon can move the cutting area while taking care of the direction in which the edge of the wide-angle image is located.

Instead of or in addition to changing the display mode of the frame of the display image, the display mode of the entire display image may be changed, for example, by changing the brightness or resolution of the display image.

According to the above-described processing, the position where the trimming area is located in the wide-angle image can be easily determined from the display mode for the respective display images.

For example, in the case of clipping a display image from a corrected wide-angle image in which distortion correction has been performed, or in the case of an object whose distortion is difficult to recognize even in a wide-angle image in which distortion correction has not been performed, it may not be possible to determine whether or not a clipping area is located near the edge of the wide-angle image.

For example, in the case where the image of the surgical site within the abdominal cavity is a wide-angle image, it is not easy to determine whether the cutting area is located near the edge of the wide-angle image in accordance with the shape of the organ or the like appearing in the display image.

Even in this case in particular, the position of the trimming area in the wide-angle image can be easily determined from the display mode for the respective display images.

<6. Fourth embodiment (display control 2 according to the position of the trimming area) >

Fig. 14 is a flowchart showing the flow of operation of the display control unit 243 according to the fourth embodiment. In a state where the trimming area reaches the edge of the wide-angle image, if it cannot be determined from the trimmed display image that the trimming area is located at the edge of the wide-angle image, the surgeon may attempt to move the trimming area further outside the edge of the wide-angle image. Therefore, when the surgeon may move the trimming area beyond the edge of the wide-angle image, the display control unit 243 according to the present embodiment changes the display mode of the display image. As a user operation to move the trimming area used here, an operation that may move the trimming area beyond the edge of the wide-angle image (note that the trimming area is not actually placed outside the edge of the wide-angle image) is referred to as trimming area beyond the edge of the wide-angle image.

In step S41, the display control unit 243 determines whether the trimming area is located outside the edge of the non-mask area in the wide-angle image.

The endoscopic image including the wide-angle image shown in the above-described embodiments includes a black shadow called "vignetting" in a portion (left and right sides on the screen) where the imaging element (image size) is larger than the image circle of the lens. On the other hand, portions (upper and lower portions on the screen) smaller in image size than the image circle are not imaged, and the subject is cut off. Then, a black image is synthesized as a mask image in the peripheral portion of the endoscope image according to the diameter of the scope of the endoscope. The black shadow or black image area in the wide-angle image is referred to herein as the occlusion region.

In other words, in step S41, it is determined whether the trimming area is located outside the upper edge or the lower edge of the wide-angle image on the screen.

If it is determined in step S41 that the trimming area is not located outside the edge of the non-mask area in the wide-angle image, the process proceeds to step S42, and then the display control unit 243 causes the display device 230 to display the trimming display image including the mask area.

On the other hand, if it is determined in step S41 that the trimming area is located outside the edge of the unshaded area in the wide-angle image, the process proceeds to step S43, and then the display control unit 243 adds a pseudo-shading area to the trimmed display image and causes the display device 230 to display the resulting image.

Fig. 15 is a diagram showing a display image cropped from a wide-angle image by a conventional method.

As shown in fig. 15, for example, when the trimming area CA _21 is at the left end of the wide-angle image 340 where the correction processing is not performed, a display image 341 including a mask area is displayed as indicated by an arrow # 21. In this case, the presence of the mask area makes it possible to easily determine that the trimming area is located at the edge of the wide-angle image.

On the other hand, when the trimming area CA _22 is in the lower end of the wide image 340, the display image 342 not including the mask area is displayed as indicated by an arrow # 22. In this case, since there is no mask area, it is not possible to easily determine that the trimming area is located at the edge of the wide-angle image.

Fig. 16 is a diagram showing a display image cropped from a wide-angle image by the method according to the present embodiment.

In the method according to the present embodiment, the pseudo mask area BK340 (pseudo mask area) is added to the wide-angle image 340 in the upper and lower portions on the screen on which the object is cut. The trimming area CA is movable on the wide-angle image 340 including the pseudo mask area BK 340.

In fig. 16, the display image 341 corresponding to the trimming area CA _21 trimmed from the wide-angle image 340 is the same as in the example of fig. 15, and therefore, the description thereof will be omitted.

When the trimming area CA _23 is in the lower end of the wide-angle image 340 including the pseudo mask area BK340, the display image 343 including the pseudo mask area BK340 is displayed as indicated by an arrow # 23. As a result, even when the trimming area is close to the edge of the non-masked area in the wide-angle image 340, the pseudo-masked area BK340 can easily determine that the trimming area is located at the edge of the wide-angle image.

Instead of adding the pseudo mask area BK340 in the upper and lower portions of the wide-angle image 340, when the trimming area CA is detected to be located outside the upper edge or the lower edge of the wide-angle image 340, a pseudo mask image may be synthesized and displayed at the upper end or the lower end of the display image.

<7. Other configuration example of medical Observation System >

Fig. 17 is a block diagram showing another configuration example of the medical observation system according to the embodiment of the present invention.

The medical viewing system 400 shown in fig. 17 includes a camera 410, an arm 420, an input device 430, a display device 440, and a control device 450.

The camera 410, the input device 430, and the display device 440 are the same as the camera 210, the input device 220, and the display device 230 in fig. 3, respectively, and thus, descriptions thereof will be omitted.

The arm 420 corresponds to the arm 32 of the endoscopic surgical system 1 or the arm 112 of the microsurgical system 100, and supports the camera 410.

The control device 450 includes a correction processing unit 451, a trimming processing unit 452, a display control unit 453, a correction control unit 454, and a drive control unit 455.

The correction processing unit 451, the display control unit 453, and the correction control unit 454 are the same as the correction processing unit 241, the display control unit 243, and the correction control unit 244 in fig. 3, respectively, and therefore, descriptions thereof will be omitted.

The clipping processing unit 452 is the same as the clipping processing unit 242 in fig. 3 in that the clipping processing unit moves the clipping area on the wide-angle image in real time based on the operation signal from the input device 430. When the trimming area is placed outside the range of the wide-angle image, the trimming processing unit 452 instructs the drive control unit 455 to drive the arm portion 420.

The drive control unit 455 controls the position and orientation of the camera 410 supported by the arm 420 by controlling the drive of the arm 420 based on an instruction from the cutting processing unit 452. The drive control unit 455 also outputs information indicating the current position of the arm portion 420 within the drive range to the cutting processing unit 452 in real time.

Using the information from the drive control unit 455, the clipping processing unit 452 provides the display control unit 453 and the correction control unit 454 with relative state information indicating the relative state of the clipping area and the allowable range for capturing the wide-angle image based on the drive range of the arm portion 420. The relative state includes a moving state (presence or absence of movement, moving speed, or the like) of the trimming area within the allowable range for capturing the wide-angle image and a position of the trimming area within the allowable range for capturing the wide-angle image.

An embodiment will be described below in which the display control unit 453 controls display of a display image according to the relative state of the trimming area and the allowable range for capturing the wide-angle image. In the following embodiment, the correction control unit 454 may also control the correction processing by the correction processing unit 241 according to the relative state of the trimming area and the allowable range for capturing the wide-angle image.

<8. Fifth embodiment (display controller 3 according to position of trimming area) >

Fig. 18 is a flowchart showing the flow of operation of the display control unit 453 according to the fifth embodiment. The display control unit 453 according to the present embodiment changes the display mode of the display image according to the distance between the trimming area and the edge of the capture allowable range.

In step S51, the display control unit 453 determines whether the trimming area is within an area defined by 20% of the outer periphery of the capturing permission range. Here, if the trimming area does not include an area defined by 20% of the outer periphery of the capture allowable range, it is determined that the trimming area is within an area defined by 20% of the outer periphery of the capture allowable range.

If it is determined in step S51 that the trimming area is within the area defined by 20% of the outer periphery of the capturing allowable range, the process proceeds to step S52, and then the display control unit 453 causes the display device 440 to display the corresponding display image in the first display mode.

On the other hand, if it is determined in step S51 that the trimming area is not within the area defined by 20% of the outer periphery of the capture allowable range, the process proceeds to step S53, and then the display control unit 243 determines whether the distance between the trimming area and the edge of the capture allowable range is greater than or equal to a predetermined threshold value.

If it is determined in step S53 that the distance between the trimming area and the edge of the capture allowable range is greater than or equal to the predetermined threshold, the process proceeds to step S54, and then the display control unit 453 causes the display device 440 to display the corresponding display image in the second display mode.

On the other hand, if it is determined in step S53 that the distance between the trimming area and the edge of the capture allowable range is not greater than or equal to the predetermined threshold, the process proceeds to step S54, and then the display control unit 453 causes the display device 440 to display the corresponding display image in the third display mode.

A display image in a display mode according to the position of the trimming area according to the present embodiment will now be described with reference to fig. 19. In fig. 19, it is assumed that the trimming area stops in the wide-angle image. Accordingly, the display image is cropped from the corrected wide-angle image on which the correction processing has been performed. The trimming area may be moved in the wide-angle image, and the display image may be trimmed from the wide-angle image in which the correction processing is not performed.

In the example of fig. 19, the wide-angle image 510 is an image of an object captured at the lower end of the allowable range of the capture SA.

As shown in fig. 19, for example, when the trimming area CA _31 is at the upper end of the wide-angle image 510 but substantially at the center of the allowable range of capturing SA, the display image 511 with the predetermined frame added is displayed as indicated by an arrow # 31.

When the trimming area CA _32 is at a position slightly deviated from the center of the allowable range for capturing SA, as indicated by an arrow #32, the display image 512 with the predetermined frame added is displayed.

When the trimming area CA _33 is located at a position deviated from the center of the range in which the capture of SA is permitted and it includes an area defined by 20% of the outer periphery of the range in which the capture of SA is permitted, as indicated by an arrow #33, the display image 513 with the added double-line frame is displayed.

When the trimming area CA _34 is at a position very close to the edge of the allowable range where the SA is captured, as indicated by an arrow #34, the display image 514 with the three-line frame added is displayed.

In fig. 19, the line type of a frame of a display image is changed according to the distance between the trimming area and the edge of the capture allowable range. However, the color and thickness of the frame displaying the image may be changed.

For example, the frame to be added to the display image is blue for a trimming area near the center of the capture allowable range, yellow for a trimming area at a position including an area defined by 20% of the outer periphery of the capture allowable range, and red for a trimming area at a position very near the edge of the capture allowable range.

Instead of or in addition to changing the display mode added to the entire frame of the display image, a part (side) of the frame added to the display image, which is close to or in contact with the edge of the wide-angle image, may be changed according to the distance between the trimming area and the edge of the capture allowable range. In this case, the surgeon can move the cutting area while considering the direction in which the edge of the allowable capture range is located.

According to the above-described processing, it is possible to easily determine where the trimming area is located within the capture permission range from the display mode for the respective display images.

In the case where the above-described third embodiment is applied to the configuration in which the position and posture of the camera 410 can be controlled by driving the arm part 420, even if the trimming area can be moved by driving the arm part 420, the display mode for displaying the image is changed according to the distance between the trimming area and the edge of the wide-angle image, so that it is not possible to accurately present the restriction of the movement of the trimming area.

On the other hand, according to the present embodiment, the display mode of the trimming area is changed according to the distance between the trimming area and the edge of the capture allowable range, so that the restriction of the movement of the trimming area can be presented more accurately.

<9. Sixth embodiment (display controller 4 according to position of trimming area) >

In the above-described fifth embodiment, the display mode of the trimming area is changed only according to the distance between the trimming area and the edge of the capture allowable range. Therefore, for example, it is impossible to determine that the trimming area CA _31 is located at the edge of the wide-angle image 510, such as the display image 511 in fig. 19.

Therefore, as shown in fig. 20, for example, when the trimming area CA _41 is substantially at the center of the wide-angle image 520, as indicated by an arrow #41, a display image 521 without a frame is displayed.

When the trimming area CA _42 is located at a position deviated from the center of the wide-angle image 520, the display image 332 without frames is displayed as indicated by an arrow # 42.

When the trimming area CA _43 is at a position very close to the edge of the wide-angle image 520, as indicated by an arrow #43, the display image 523 with the predetermined frame added is displayed. At this time, as will be described later, the display mode for displaying the frame of the image 523 is changed according to the distance between the trimming area and the edge of the allowable range for capturing.

As a result, it can be determined whether the trimming area is located at the edge of the wide-angle image, and the trimming area is located at a position within the allowable capture range.

In the example of fig. 20, even when the trimming area CA _42 is at a position distant from the edge of the wide-angle image 520, if the trimming area CA _42 includes an area defined by 20% of the outer periphery of the allowable range of the captured SA, the display image 532 to which frames are added in the corresponding display mode is displayed.

Fig. 21 is a flowchart showing the flow of operation of the display control unit 453 according to the sixth embodiment. The display control unit 453 according to the present embodiment changes the display mode of the display image according to the distance between the trimming area and the edge of the captured allowable range and the distance between the trimming area and the edge of the wide-angle image. The processing of steps S61 and S65 to S67 in the flowchart of fig. 21 is the same as the processing of steps S51 and S53 to S55 in the flowchart of fig. 18, and thus, the description thereof will be omitted.

Thus, if it is determined in step S61 that the trimming area is within the area defined by 20% of the outer periphery of the capture allowable range, the process proceeds to step S62, and then the display control unit 453 determines whether the distance between the trimming area and the edge of the wide-angle image is greater than or equal to a predetermined threshold value.

If it is determined in step S62 that the distance between the trimming area and the edge of the wide-angle image is greater than or equal to the predetermined threshold, the process proceeds to step S63, and then the display control unit 453 causes the display device 440 to display the trimmed display image as it is.

On the other hand, if it is determined in step S62 that the distance between the trimming area and the edge of the wide-angle image is not greater than or equal to the predetermined threshold, the process proceeds to step S64, and then the display control unit 453 causes the display device 440 to display the corresponding display image in the first display mode.

A display image in a display mode according to the position of the trimming area according to the present embodiment will now be described with reference to fig. 22. In fig. 22, it is assumed that the trimming area stops in the wide-angle image. Thus, the display image is cropped from the corrected wide-angle image on which the correction processing has been performed. The trimming area may be moved in the wide-angle image, and the display image may be trimmed from the wide-angle image in which the correction processing is not performed.

As shown in fig. 22, even when the trimming area CA is substantially at the center of the allowable range for capturing SA _51, if the trimming area CA is at a position very close to the edge of the wide-angle image 530, a display image 531 with the predetermined frame added is displayed as shown by an arrow # 51.

When the trimming area CA is at a position where it includes an area defined by 20% of the outer periphery of the allowable range where SA _52 is captured, regardless of the position of the trimming area CA in the wide-angle image 530, as indicated by arrow #52, the display image 532 with the added double-line frame is displayed.

When the trimming area CA is at a position very close to the end of the allowable range for capturing SA _53, the display image 533 with the three-line frame added is displayed as indicated by arrow #53 regardless of the position of the trimming area CA in the wide-angle image 530.

In fig. 22, as shown in fig. 19, the line type of the frame of the display image is changed according to the distance between the trimming area and the edge of the captured allowable range. However, the color and thickness of the frame of the display image, the display mode added to a part (side) of the frame of the display image (which is near or in contact with the edge of the wide-angle image), or the entire display image may be changed.

According to the above-described processing, it is possible to easily determine whether or not the trimming area is located very close to the edge of the wide-angle image, in accordance with the display mode of the corresponding display image and the position of the trimming area within the allowable capture range.

The embodiment has been described above in which the display control unit 243 or the display control unit 453 controls the display of the display image according to the position of the trimming area in the wide-angle image or the allowable range of capture. The display control unit 243 or the display control unit 453 is not limited to this embodiment, and may control to display the display image (presence or absence of movement, movement speed, movement direction, or the like) of the trimming area in a wide-angle image or a captured allowable range.

For example, the display mode for displaying an image and/or the display mode for displaying a frame of an image may be changed according to the moving speed or the capture allowable range of the cropped area in the wide-angle image (in other words, the time to reach the edge of the wide-angle image or the capture allowable range).

<10. Hardware configuration >

Finally, an example of the hardware configuration of the control device included in the medical imaging system according to the embodiment of the present invention will be described with reference to fig. 23.

As shown in fig. 23, the control device 600 includes a cpu 601, a rom 603, and a ram 605. The control device 600 further includes a host bus 607, a bridge 609, an external bus 611, an interface 613, an input device 615, an output device 617, and a storage device 619. The control device 600 may include a drive 621, a connection port 623, and a communication device 625.

The central processor 601 functions as an arithmetic processing device and a control device, and controls all or some operations in the control device 600 according to various programs recorded on the read only memory 603, the random access memory 605, the storage device 619, or the removable recording medium 627.

The read only memory 603 stores programs, arithmetic parameters, and the like used by the central processing unit 601. The random access memory 605 mainly stores programs used by the central processing unit 601, parameters appropriately changed during the execution of the programs, and the like. These units are connected to each other by a host bus 607 including an internal bus such as a central processor bus. Each configuration of the control device 240 in fig. 3 and the control device 450 in fig. 17 is realized by, for example, a central processor 601.

The host bus 607 is connected to an external bus 611, such as a peripheral component interconnect/interface (PCI) bus, through a bridge 609. The input device 615, the output device 617, the storage device 619, the drive 621, the connection port 623, and the communication device 625 are connected to the external bus 611 via the interface 613.

The input device 615 is an operation member such as a mouse, a keyboard, a touch panel, a button, a switch, a lever, and a pedal operated by the operator. The input device 615 may be a remote control device (so-called remote control), for example using infrared light or other radio waves, or may be an externally connected device 629, such as a mobile phone or PDA, which supports operations on the control device 600.

The input device 615 includes an input control circuit or the like that generates an input signal based on information input by the surgeon using the operation device and outputs the input signal to the central processor 601.

The operator can operate the input device 615 to input various types of data to the control device 600 and instruct the control device 600 to perform processing operations.

The output device 617 is configured as a device capable of visually or audibly notifying the operator of the acquired information. Specifically, for example, the output device 617 is configured as a display device such as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, and a lamp, a sound output device such as a speaker and an earphone, and/or a printer device.

For example, the output device 617 outputs results obtained by various types of processing performed by the control device 600. Specifically, the display device displays results obtained by various types of processing performed by the control device 600 as text or images. On the other hand, the sound output device converts an audio signal formed of reproduced sound data, or the like into an analog signal, and outputs the converted analog signal.

The storage device 619 is a data storage device configured as an example of a storage unit of the control device 600. The storage device 619 includes, for example, a magnetic storage device such as a Hard Disk Drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 619 stores programs executed by the central processing unit 601 or various types of data.

The drive 621 is a recording medium reader/writer, and is built in or externally connected to the control device 600. The drive 621 reads information recorded on a removable recording medium 627 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory mounted thereon, and outputs the information to the random access memory 605. The drive 621 can also record information on a removable recording medium 627 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory mounted thereon.

The removable recording medium 627 is, for example, a DVD medium, an HD-DVD medium, or a blu-ray (registered trademark) medium. The removable recording medium 627 may be a compact flash (registered trademark), a flash memory, a Secure Digital (SD) memory card, or the like. The removable recording medium 627 may be, for example, an Integrated Circuit (IC) card or an electronic device on which a non-contact type IC chip is mounted.

The connection port 623 is a port for directly connecting the external connection device 629 to the control device 600. Examples of connection ports 623 include a Universal Serial Bus (USB) port, an IEEE1394 port, and a Small Computer System Interface (SCSI) port. Other examples of the connection port 623 include an RS-232C port, an optical audio terminal, and a high-definition multimedia interface (HDMI) (registered trademark) port. When the external connection device 629 is connected to the connection port 623, the control device 600 directly acquires various types of data from the external connection device 629 or supplies various types of data to the external connection device 629.

The communication device 625 is, for example, a communication interface including a communication device or the like for connecting to a communication network (network) 631. The communication device 625 is a communication card for wired or wireless Local Area Network (LAN), bluetooth (registered trademark), or Wireless USB (WUSB), for example. The communication device 625 may be a router for optical communications, a router for Asymmetric Digital Subscriber Line (ADSL), or any of various types of communication modems.

The communication device 625 may, for example, transmit and receive signals and the like to and from the internet or another communication device in accordance with a predetermined protocol, such as TCP/IP. The communication network 631 connected to the communication device 625 may be constituted by a network or the like connected in a wireless or wired manner. The communication network 631 may be, for example, the internet, a home LAN, or a communication network through which infrared communication, radio wave communication, or satellite communication is performed.

Each component of the control apparatus 600 may be configured using a general-purpose member, or may be configured using hardware dedicated to the function of each component. Therefore, when any of the above-described embodiments is implemented, the hardware configuration to be used may be appropriately changed according to the technical level.

A computer program for realizing each function of the control device 600 included in the medical observation system according to the above-described embodiment can be generated, and the computer program is installed in a personal computer or the like. A computer-readable recording medium storing the computer program may also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may be distributed via a network, for example, without using a recording medium.

In this specification, a system means a set of plural constituent elements (devices, modules (parts), etc.), and all of the constituent elements may be located in the same housing or not. Therefore, a plurality of apparatuses accommodated in separate housings and connected via a network, and one apparatus in which a plurality of modules are accommodated in one housing are all systems.

It is to be noted that the embodiments of the present disclosure are not limited to the above-described embodiments, and may be modified in various ways without departing from the scope and spirit of the present disclosure.

For example, the present disclosure may be configured as cloud computing in which a plurality of apparatuses share and cooperatively process one function via a network.

In addition, each step described in the above-described flowcharts may be performed by one apparatus or performed in a shared manner by a plurality of apparatuses.

Further, in the case where a plurality of kinds of processing are included in a single step, the plurality of kinds of processing included in the single step may be executed by one apparatus or by a plurality of apparatuses in a shared manner.

The present disclosure may be configured as follows.

(1) A medical viewing system, comprising:

a correction processing unit that performs correction processing on a wide-angle image of a captured image as a medical target;

a trimming processing unit that moves a trimming area of a display image trimmed from the wide-angle image; and a correction control unit that controls the correction processing according to a relative state of the trimming area with respect to the wide-angle image.

(2) The medical observation system according to (1), wherein the correction processing unit performs distortion correction on the wide-angle image.

(3) The medical observation system according to (1), wherein the correction processing unit performs brightness correction on the wide-angle image.

(4) The medical observation system according to any one of (1) to (3), wherein the relative state includes a moving state of the cutout area in the wide-angle image.

(5) The medical observation system according to (4), wherein,

the correction control unit

Disabling the correction process when the trimming area moves, an

Enabling the correction process when the trimming area is stopped.

(6) The medical observation system according to (4), wherein the correction control unit sets a correction amount for the correction process in accordance with a moving speed of the trimming area.

(7) The medical observation system according to (6), wherein the correction control unit sets the correction amount step by step according to whether or not the moving speed of the trimming area exceeds a predetermined threshold.

(8) The medical observation system according to (6), wherein the correction control unit linearly changes the correction amount corresponding to the moving speed of the cutting area.

(9) The medical observation system according to any one of (1) to (8), further comprising a display control unit that controls display of a display image according to a relative state of the cutting area with respect to the wide-angle image.

(10) The medical observation system according to (9), wherein the relative state includes at least one of a position and a moving state of the cutout area in the wide-angle image.

(11) The medical observation system according to (10), wherein the display control unit changes a display mode of the display image according to a distance between the cutout area and an edge of the wide-angle image.

(12) The medical observation system according to (11), wherein the display control unit changes a display mode of a frame of the display image according to the distance.

(13) The medical observation system according to (10), wherein the display control unit changes a display mode of the display image when the trimming area is located outside an edge of the wide-angle image.

(14) The medical observation system according to (13), wherein the display control unit displays the display image with a pseudo-occlusion region added when the trimming region is located outside an edge of an non-occlusion region in the wide-angle image.

(15) The medical observation system according to any one of (1) to (14), further comprising a display control unit that controls display of the display image according to a relative state of the cutting area with respect to an allowable range of the wide-angle image, based on a driving range of an arm portion for supporting a camera that captures the wide-angle image.

(16) The medical observation system according to (15), wherein the relative state includes at least one of a position and a moving state of the cutting area within the capture permission range.

(17) The medical observation system according to (16), wherein the display control unit changes a display mode of the display image according to a distance between the trimming area and an edge of the capture allowable range.

(18) The medical observation system according to (17), wherein the display control unit changes a display mode of the display image according to a distance between the cutout area and an edge of the capture allowable range and a distance between the cutout area and an edge of the wide-angle image.

(19) An image processing method performed by a medical viewing system, the image processing method comprising:

performing correction processing on a wide-angle image of a captured image as a medical target;

moving a trimming area of a display image trimmed from the wide-angle image; and

the correction processing is controlled in accordance with a relative state of the trimming area with respect to the wide-angle image.

(20) A program that causes a computer to execute:

[ list of reference numerals ]

1. Endoscopic surgery system

100. Microsurgical system

200. Medical observation system

210. Camera with a camera module

220. Input device

230. Display device

240. Control device

241. Correction processing unit

242. Cutting processing unit

243. Display control unit

244. Correction control unit

400. Medical observation system

410. Camera with a camera module

420. Arm part

430. Input device

440. Display device

450. Control device

451. Correction processing unit

452. Cutting processing unit

453. Display control unit

454. Correction control unit

455. A drive control unit.

Claims

1. A medical viewing system comprising:

a correction processing unit that performs correction processing on a wide-angle image that is a captured image of a medical target;

a trimming processing unit that moves a trimming area of a display image trimmed from the wide-angle image; and

a correction control unit that controls the correction processing according to a relative state of the trimming area with respect to the wide-angle image.

2. The medical observation system according to claim 1, wherein the correction processing unit performs distortion correction on the wide-angle image.

3. The medical observation system according to claim 1, wherein the correction processing unit performs brightness correction on the wide-angle image.

4. The medical viewing system of claim 1, wherein the relative state includes a state of movement of the cropped area in the wide-angle image.

5. The medical viewing system of claim 4,

the correction control unit

Disabling the correction process when the trimming area moves, an

When the trimming area is stopped, the correction processing is started.

6. The medical observation system according to claim 4, wherein the correction control unit sets a correction amount for the correction process in accordance with a moving speed of the trimming area.

7. The medical observation system according to claim 6, wherein the correction control unit sets the correction amount step by step according to whether or not a moving speed of the trimming area exceeds a predetermined threshold.

8. The medical observation system according to claim 6, wherein the correction control unit linearly changes the correction amount corresponding to the moving speed of the cutting area.

9. The medical viewing system of claim 1, further comprising: a display control unit that controls display of the display image according to the relative state of the trimming area with respect to the wide-angle image.

10. The medical viewing system of claim 9, wherein the relative state includes at least one of a position and a movement state of the cropped area in the wide angle image.

11. The medical observation system according to claim 10, wherein the display control unit changes a display mode of the display image according to a distance between the cutout area and an edge of the wide-angle image.

12. The medical observation system according to claim 11, wherein the display control unit changes a display mode of a frame of the display image in accordance with the distance.

13. The medical observation system according to claim 10, wherein the display control unit changes a display mode of the display image when the trimming area is located outside an edge of the wide-angle image.

14. The medical observation system according to claim 13, wherein the display control unit displays the display image with a pseudo-occlusion region added when the trimming region is located outside an edge of an non-occlusion region in the wide-angle image.

15. The medical viewing system of claim 1, further comprising: a display control unit that controls display of the display image according to a relative state of the trimming area with respect to an allowable range of the wide-angle image, based on a driving range of an arm for supporting a camera that captures the wide-angle image.

16. The medical viewing system according to claim 15, wherein the relative state includes at least one of a position and a moving state of the cutting area within a capture allowance range.

17. The medical observation system according to claim 16, wherein the display control unit changes a display mode of the display image in accordance with a distance between the trimming area and an edge of the capture allowable range.

18. The medical observation system according to claim 17, wherein the display control unit changes a display mode of the display image in accordance with a distance between the cutout area and an edge of the capture allowable range and a distance between the cutout area and an edge of the wide-angle image.

19. An image processing method performed by a medical viewing system, the image processing method comprising:

20. A program for causing a computer to execute: