JP7244926B2 - SURGERY IMAGE DISPLAY DEVICE, CONTROL METHOD FOR SURGICAL IMAGE DISPLAY DEVICE, AND PROGRAM - Google Patents

Description

The present invention relates to a system that selects images to be displayed on a display by capturing images of procedures such as surgical operations with a multi-view camera.

Conventionally, a visual object-centered fixed viewing interface has already been proposed as a viewing interface that allows easy selection of viewpoints when viewing multi-viewpoint video content (for example, Non-Patent Document 1). However, this assumes that the whole body of a person who performs sports or performing arts is viewed from all angles, and does not assume that the operator's hand operations are viewed from all angles. If there is no obstacle between the camera and the object to be photographed, the object to be photographed is included regardless of which image is selected. However, when photographing operations at hand such as during surgical operations, there is a problem that the operator's head and body are caught between the camera and the hand, and the hand cannot be seen. Such videos have no informational value, and videos from such viewpoints should not be included in options for viewing interfaces for multi-view videos. When selecting a viewpoint, if a video with no informational value is selected, the viewpoint must be selected again.
In this specification, practitioners include surgeons, as well as “manufacturers, furniture craftsmen, cutlery craftsmen, metal engravers, charcoal burners, weaving technicians, glass product manufacturers, ceramic technicians, cloisonne workers, brick workers, tile manufacturing workers, casting workers, candle manufacturing workers, mold making workers, etc.

Traditional crafts, glass blowers, doll artists, metalworkers, bamboo craftsmen, swordsmiths, wax craftsmen, Yuzen dyeing, Buddhist altar and altar fittings craftsmen, Japanese umbrella craftsmen, fireworks craftsmen, calligraphy craftsmen, Japanese paper craftsmen, joinery craftsmen, oshie battledore craftsmen Decorator, potter, tile craftsman, lacquer craftsman, lantern, fan, folding fan craftsman, mounting technician, archer, arrow craftsman, mask craftsman (Noh mask, Kyogen mask, Kagura mask), lacquerware craftsman, etc.

Architects, civil engineers, plasterers, carpenters, masonry, temple carpenters, painters, sheet metal workers, tatami mat craftsmen, welders, electricians, landscapers (gardeners), etc.

Food-related chefs, sushi chefs, Japanese confectionery artisans, salt artisans, tofu artisans, bakers, sake makers, soba artisans, coffee roasters, miso artisans, soy sauce artisans, wine makers, tofu artisans

Service (fashion, medical)
・Hairdresser ・Barber ・Cleaner ・Shoemaker ・Japanese dressmaker ・Esthetician ・Total beautician ・Kimono dresser ・Leather worker Acupuncturist (acupuncture and moxibustion), magician, etc.

It means an artist, a painter, a calligrapher, etc.

As described above, when viewing multi-view video content of operations at hand such as surgical operations, it is desirable that the viewing interface excludes images in which the hand is obscured by obstacles from options.

In addition, illumination is used in medical procedures such as surgery. A surgical light is usually used for this illumination so that the affected area to be treated is not shaded by the operator's body, especially the head, arms, hands, etc., and surgical instruments.
The shadowless light includes a movable and tiltable circular support base attached to the tip of an arm suspended from the ceiling, a center light arranged in the center of the support base, and a center light arranged around the center light. A structure having a plurality of ambient lights attached to the support base is often used.
By the way, during the operation, the person who assists the operation, such as preparing the next surgical instrument, and the person who grasps the progress of the operation in a separate room cannot directly observe the treatment affected area. It is normal to know the condition of the affected area.
At this time, a camera is used, and since it is convenient to move and tilt along with the movement and tilting of the surgical lamp, the camera is often attached to the surgical lamp. Japanese Patent Application Laid-Open No. 2000-102547 (Patent Document 1) and International Publication No. 2015/173851 (Patent Document 2) have been proposed as examples in which a camera is provided in a shadowless light. A camera is provided at a close position or inside the handle portion that operates the surgical light.
In the case of treatment using a shadowless light with a camera with this structure, even if the affected area to be treated is illuminated, the captured images show the operator's body, especially the head, arms, hands, etc., as well as the surgical instruments. A situation arises in which the affected area is not reflected in the image. This problem will be described in detail below.

2. Description of the Related Art When photographing an affected area to be treated in surgery, visual disturbance occurs in which the affected area to be imaged is hidden due to the operator's peek, hand or finger movements, surgical instruments, or the like. In order to avoid visual disturbances and complete imaging, the position and direction of the camera should be adjusted according to the constantly changing positional relationship between the affected area to be treated, the operator, surgical instruments, etc., which may interfere with imaging. , it is necessary to adjust the angle of view. However, in practice, it is difficult to immediately operate the camera in accordance with changes in the position of the affected area to be treated and the position of the obstacle.

First, it is difficult to allocate personnel for operating the camera as described above in the operating room. The surgeon who participates in the operation as a surgeon or an assistant, and the nurse who hands the surgical instruments to the operator concentrates on the progress of the operation, so it is difficult to grasp the situation of the captured images during the operation. is. In the operating room, there are circulating nurses who provide additional surgical instruments to the instrument-delivering nurses, and sometimes operate the camera between duties. Even so, it is difficult to operate the camera immediately every time the position of the diseased part to be treated and the position of the obstacle change.

Second, there are situations in which it is difficult to change the position of the camera. At the surgical site, the affected area to be treated is sterilized, the patient lying on the operating bed is covered with a sterile cloth, the operator and the nurse who removes the instruments are wearing sterile gowns on top of the surgical gowns. Surroundings are arranged instrument tables on which sterilized surgical instruments are placed. When photographing the affected area to be treated, the camera is not allowed to touch these sterile people or objects, and the camera can be placed either directly above the affected area to be treated or further away from the sterile person or object. position. In addition, the operating light is usually placed near the affected area to be treated. If the camera is located directly above the affected area to be treated, the circulating nurse who can manipulate the position of the camera is not wearing a sterile gown, making it difficult to approach the camera without touching a sterile person or object. be.

In addition, as in Patent Documents 1 and 2, when only one camera is provided in the shadowless lamp, although the affected area to be treated is illuminated, the operator's head and the affected area to be treated are placed between the camera and the affected area. Situations can occur where visual field impairment occurs due to intervening obstacles such as hands. At this time, it is necessary to operate the position of the camera in order to avoid obstacles and photograph the affected area to be treated. Because it is installed, when the position and direction of the camera are operated according to the movement of the operator, the position and direction of the operating light will also change in conjunction with this, making it difficult to illuminate the affected area to be treated. produces

What is common to all cases is that in the operating room, the completion of a safe operation takes precedence over the capture of good images. takes precedence. For these reasons, it is difficult for any of the cameras disclosed in Patent Documents 1 and 2 to photograph the affected area to be treated while avoiding obstruction of the field of vision due to obstacles.

Also, when the camera's visual field is obstructed by an obstacle during imaging, the operator is concentrating on the progress of the surgery and it is impossible to see the display at the same time. It is difficult to notice this, and the operation proceeds without imaging being performed.

On the other hand, the most necessary information in surgical images required in the medical field is the content of medical operations on the affected area to be treated. It is necessary to perform imaging of the affected area to be treated while avoiding obstruction of the imaging field of the patient's head, body, arms, and hands.

JP-A-2000-102547 WO2015/173851 JP 2012-120812 A

Kenji Mase, Shogo Tokai, Tetsuya Kawamoto, Toshiaki Fujii, "Considerations on the fixed viewing method of multi-view images and design of operation interface", Human Interface Society Research Report, Vol.11, No.1, pp.7-12, 2009.3.10-11 .

SUMMARY OF THE INVENTION The present invention has been made in view of the above needs and circumstances, and an object of the present invention is to provide a technique for avoiding visual disturbance as much as possible when displaying surgical images. It is in.

In order to achieve the above object, the present invention provides an acquisition unit that acquires a plurality of image data obtained by photographing the same operative field from different viewpoints using a plurality of cameras, and an image for each frame of each image data. an evaluation unit that evaluates the operating field exposure, which is the degree to which the operating field is reflected in the image data, based on the operating field exposure of each image of the plurality of video data for each frame; Provided is a surgical image display device comprising: a selection unit for selecting image data from among them; and an image display unit for displaying an image of the image data selected by the selection unit on a display device.

Advantageous Effects of Invention According to the present invention, visual field disturbance can be avoided as much as possible when displaying surgical images.

FIG. 1 is a diagram showing an example of the configuration of the first invention of the present application. FIG. 2 is a flow chart showing an example of an algorithm for selecting a video in the first invention of the present application. FIG. 3 is a flow chart showing an example of an algorithm for calculating an average score from hand recognition regions in the first invention of the present application. FIG. 4 is a diagram showing an example of a recognition area obtained by detecting a hand area included in an image in the first invention device of the present application. FIG. 5 shows that in the first inventive device of the present application, the area of the hand included in the image is detected, the center point of each area is obtained, the average point is calculated from the average of the coordinates of the center point of each area, and the image is obtained. FIG. 10 is a diagram showing an example of the process of calculating the distance to the center point; FIG. 6 shows that in the first invention apparatus of the present application, the hand area included in the image is detected, the center point of each area is obtained, the average point is calculated from the average of the coordinates of the center point of each area, and the image is obtained. FIG. 10 is a diagram showing an example of the process of calculating the distance to the center point; FIG. 7 shows that in the first invention device of the present application, the hand area included in the image is detected, the center point of each area is obtained, the average point is calculated from the average of the coordinates of the center point of each area, and the image is obtained. FIG. 10 is a diagram showing an example of the process of calculating the distance to the center point; FIG. 8 is a schematic diagram of a camera system according to a second invention embodiment of the present application. FIG. 9 is a diagram showing a configuration for selecting an image to be displayed on a display device by means of an operation device from images of a plurality of cameras. FIG. 10 is a diagram showing a configuration for simultaneously recording images from a plurality of cameras using a recording unit. FIG. 11 is a diagram showing an embodiment in which a plurality of videos shot at the same time are recorded with a common shooting date and time and played back at the same time. FIG. 12 is a diagram showing an embodiment in which the tilting unit of the surrounding camera unit is tilted by rotating the operating handle. 13A and 13B are diagrams showing an embodiment example of the guide light irradiated to the object to be photographed when the apparatus of the present invention is provided with a guide light irradiation device for informing the direction of the image of the camera. FIG. 14 shows an embodiment in which the device of the present invention is equipped with a microphone. FIG. 15 is a diagram showing an embodiment in which a plurality of images and sounds acquired at the same time are recorded with a common shooting date and time and played back at the same time. FIG. 16 shows an embodiment in which the apparatus of the present invention is equipped with a semi-spherical camera. In FIG. 17, the operation handle is equipped with an operation device for automatic adjustment of the light source, the angle of view and the focus, the recording unit, and the recording device, and the focal length operation device automatically adjusts the focus by input from the operation device example of the monitor mechanism or by tilting the unit base. Fig. 3 shows an example embodiment of the device of the present invention for actuating regulation; FIG. 18 is a diagram showing a configuration for extracting a plurality of recorded video and audio within a common shooting date and time range and outputting them as a plurality of divided data. FIG. 19 is a flow chart showing an example of an algorithm for switching images when the display device recognizes an image failure. FIG. 20 is a diagram showing a configuration for selecting an image to be displayed on the display device by the CPU provided in the distance measuring device and the monitor mechanism. FIG. 21 is a diagram showing a configuration for automatically recognizing the content of moving image data and audio data to create character string data, and adding tag information of character strings to photographing time information to create list data. FIG. 22 is a diagram showing a configuration example of a surgical image display apparatus according to the third invention. FIG. 23 is a flow chart showing an example of processing of the information processing apparatus according to the third invention. 24A and 24B are diagrams showing examples of images captured from a camera installed above the surgical field. FIG. 25 is a diagram showing an example of changes in the operating field exposure for each frame and switching of images to be displayed. FIG. 26 is a diagram showing an example of an image switching control algorithm to which the Dijkstra method is applied. FIG. 27 is a diagram showing another configuration example of the surgical image display device according to the third invention.

<First Invention>
A first invention of the present application performs image recognition of a hand in an information processing device for image data captured by a plurality of video cameras and moving image (video) data that is a frame sequence arranged in a time domain. By doing so, the images displayed on the image display unit are selected in advance and presented to the viewer by selecting the one in which the hand is displayed in the image or by not selecting the one in which the hand is not displayed in the image. .

The first invention of the present application provides a function for supporting effective viewing of multi-viewpoint images related to procedures such as surgery, with the configuration described above. Preliminarily exclude images with no informational value, such as those that are out of the angle of view and do not appear, or images with high informational value, such as many hands appearing near the center of the screen. By selecting and presenting images in advance, it is expected that the viewer's stress when selecting an appropriate image from among many options will be reduced.

An embodiment embodying the first invention of the present application will be described in detail below with reference to the drawings. The figures are drawn schematically for convenience of explanation. Note that the application range of the multi-view video viewing system according to the embodiment of the present invention is not limited to the configuration shown below, and can be applied to any configuration.

In FIG. 1, three video images of an object 10 captured by three video cameras each composed of one main camera 12 and two sub cameras 14 are sent to an image acquisition unit 22 of an information processing device 20, and are sent to a control unit 24. 4 shows an embodiment of the invention in which the image selected by is sent to the display device 40 via the image display unit 34. FIG. The number of video cameras is two or more and there is no upper limit. A camera ID is assigned to each camera. At the same time, each camera transmits image data to which the same time information is added to the image acquisition unit 22 together with its own camera ID information. The image data sent from the control unit 24 to the image display unit 34 at a certain point in time is image data including hand image data whose hands have been recognized by the image recognition unit 26, among the plurality of image data input to the image acquisition unit 22. is. The number of image data selected by the image (video) selection unit 32 and sent to the video display unit 34 is not limited to one, and a plurality of image data may be selected. The information processing device 20 and the display device 40 may be a PC (personal computer), a smart phone, or a tablet terminal. The processing and functions of the information processing device 20, which will be described later, develop a program non-temporarily stored in a storage device or storage medium (for example, a hard disk drive, a solid state drive, a flash memory, a magnetic disk, etc.) into a memory, It may be implemented in software by being executed by a processor (eg, CPU, GPU, etc.). Alternatively, part of the processing and functions of the information processing device 20 may be realized by hardware such as ASIC, or may be executed by another computer such as a cloud server.

FIG. 2 is a flow chart showing an example of an algorithm in the control section 24. As shown in FIG. First, images captured by the respective video cameras in time synchronization are sent to the image acquisition section 22 of the information processing device 20 and passed to the control section 24 . At this time, in the control unit 24, the image recognition unit 26 recognizes the recognition area, which is the area where the hand is displayed in the image in the frame at a certain point in time, for each of the transferred image data (step S1, FIG. 4, 52, 54, 56, 58). As for image recognition, conventional imaging devices such as digital cameras and digital video cameras detect a person's face and hands from the image, track the detected face and adjust the focus, or detect the movement of the hand. It has been proposed to execute a predetermined process accordingly. Methods for recognizing a hand from an image include, for example, a method based on color features and a method based on shape features. When the surgical image is the target image, the operator wears prescribed surgical gloves, so it is relatively easy to grasp the color features. Alternatively, to facilitate image recognition, a special colored surgical glove (a color clearly different from other objects in the field of view of the camera) is worn, or surgical gloves are used for image recognition. may be marked with Recognition methods based on shape features include, for example, a method using HOG features, a method of recognizing a structure consisting of a palm and five fingers, and the like. A hand recognizer may also be generated by machine learning such as deep learning. Examples of technologies that can be used as image recognition technology for moving images include APIs (application programming interfaces) such as Cloud Video Intelligence provided by Google and Video Indexer provided by Microsoft. interface). Alternatively, a posture detection algorithm such as OpenPose may be used.
In addition, image data obtained using a thermography, which can visually display the temperature distribution as an image, and a depth camera, which can acquire distance information to the subject as an image, is used as the target image for hand recognition. You may

Next, the counting unit 28 counts the number of recognition areas of each image data (step S2). Next, the average point calculator 30 calculates the coordinates of the average point from the recognition area of each image data (step S3, 70 in FIG. 6). A method for calculating the average score will be described later. As a result, the number of recognition regions and the coordinate information of the average point are added to each image data of each camera at the same time. Based on these, the image (video) selection unit 32 first confirms the maximum number of recognition regions (step S4). If the hand is not recognized in any of the image data, that is, if the number of recognition regions is 0 in any of the image data, the image (video) selection unit 32 selects the main camera 12 video. An image is selected (step S9) and transmitted to the image display section . Note that it is possible to redefine one of the secondary cameras 14 as the main camera 12 and at the same time replace the primary camera 12 with the secondary camera 14 by the operation of the viewer or the algorithm of the computer. be.

If the maximum number of recognition regions is 1 or more in step S4, the image (video) selection unit 32 selects image data with the maximum number of recognition regions (step S5). At this time, it is confirmed whether or not there are a plurality of image data to be selected (step S6). If there is only one image data with the maximum number of recognition regions, the image (video) selection unit 32 selects the video including this image data and transmits it to the video display unit 34 . If there are two or more pieces of image data with the maximum number of recognition regions, the image (video) selection unit 32 calculates the distance from the center point of the image from the coordinates of the average point for each of these images. ( 90 in FIG. 7), the video including the image data with the smallest distance is selected and transmitted to the video display unit 34 .

FIG. 3 is a flow chart showing an example of an algorithm for calculating the coordinates of the average point from the recognition area of each image data in the average point calculator 30. In FIG. First, the average score calculation unit 30 confirms the number of recognition regions obtained by the counting unit 28 for each image data (step S31). The average score is not calculated for image data in which the number of recognition regions is 0, that is, the hand is not recognized (step S36). If the number of recognition regions is one or more, the average point calculation unit 30 calculates the coordinates of the central point of each region from the coordinate information of the recognition regions detected by the image recognition unit 26 (step S32; see FIG. 5). Here is an example of recognizing a region as a rectangle, where the center point is equal to the average of the coordinates of the four corners of the rectangle). Next, it is confirmed whether or not the number of recognition areas is two or more (step S33). When the number of recognition areas is 1, the center point of this area becomes the average point of this camera image (step S35). If the number of recognition areas is two or more, the average value is calculated from all the coordinates of the center point of each recognition area and defined as the coordinates of the average point (step S34, 70 in FIG. 6).

Note that the algorithm shown in FIGS. 2 and 3 is only an example, and even if the order of steps is changed or steps are added/deleted, the expected result, that is, an image in which the hand is not recognized, is selected. Any algorithm that selects a video with no hands, a video with many hands, or a video with hands positioned around the center of the video can be included within the scope of the present invention.

In addition, regarding the image sent to the image acquisition unit 22, the object for which the control unit 24 performs processing from image recognition to image selection does not have to be image data of all frames. can be adjusted by viewer selection or computer algorithms every 5 seconds, 10 seconds, etc.

FIG. 4 is a diagram showing an example of recognition areas 52, 54, 56, and 58 where hands are recognized by the image recognition unit 26. As shown in FIG. Although this figure shows a case where the recognition area is rectangular, the shape of the recognition area may be any shape such as an ellipse or a polygon. Also, although the rectangles are shown on the screen for the sake of clarity, the rectangles do not have to be displayed on the display device as an actual user interface, as long as they are internally processed by a computer. Also, for example, when imaging surgery, the hand may also be the target of treatment. In this case, it is not desirable to include the hand to be treated in the recognition area. Therefore, in the image recognition means, a device that can distinguish between a gloved hand and a non-gloved hand may be adopted.

FIG. 5 is a diagram showing an example in which center points 62, 64, 66 and 68 of the recognition regions 52, 54, 56 and 58 are calculated by the average point calculator 30. In FIG. When the recognition areas 52, 54, 56, 58 are rectangular, the center points 62, 64, 66, 68 may be obtained by averaging the coordinates of the four corners of the rectangle, or by obtaining the intersection of diagonal lines. . There are many other ways to find the center point, but any method may be used as long as a point identical or similar to the center point is calculated as a result. Moreover, when the recognition area is not rectangular, a method of calculating the area of the recognition area and setting the center point as the center of gravity may be used.

FIG. 6 is a diagram showing an example in which an average point 70 of the center points 62, 64, 66 and 68 is calculated by the average point calculator 30. As shown in FIG. The coordinates of the average point 70 may be calculated as the coordinates obtained by calculating the average value from the respective coordinate values of the center points 62, 64, 66 and 68. FIG.

FIG. 7 is a diagram showing an example in which the image (video) selection unit 32 calculates the distance between the average point 70 and the center point 80 of the video from the coordinates of the average point 70 calculated by the average point calculation unit 30. is. Although the distance in this figure is calculated considering both the x-coordinate and the y-coordinate, it may be calculated using either the x-coordinate or the y-coordinate, for example. Further, the distance 90 between the average point 70 and the center point 80 of the image may be obtained by the image (video) selection unit 32 as in the above configuration, or by the average point calculation unit 30 in advance. A distance 90 between the average point 70 and the center point 80 of the image may be calculated. In this case, the value of the distance 90 is transferred to the image (image) selection unit 32 and used for image selection. may

The first invention can be understood as follows.
(1) In a multi-viewpoint video viewing system in which a plurality of multi-viewpoint video data indicating the state of an object being processed is obtained by a practitioner, and the video data is processed and displayed on a display device,
a plurality of video cameras that concurrently capture the hands of a practitioner processing an object from different viewpoints and output video data consisting of a large number of continuous image data;
An information processing device that receives and processes a plurality of video data simultaneously captured by the plurality of video cameras,
The information processing device is
an image recognition unit for detecting individual image data of a plurality of video data multiplexed and recognizing hand image data relating to hands in the individual image data;
an image selection unit for selecting, from among the plurality of image data, image data in which hand image data is recognized by the image recognition unit in time series; 1. A multi-view video viewing system comprising a video display unit for transmitting video to a display device.
(2) The image display unit performs editing by combining the image data selected in time series by the image selection unit, creates one piece of edited image data to be output to the display device, and displays the edited image data. The multi-view video viewing system according to (1), which transmits to the display device.
(3) the information processing device includes a counting unit, counts the number of recognition regions, which are regions of the hand image data in which hands are recognized by the image recognition unit, for each image data, and sends the number to the image selection unit;
The multi-view video viewing system according to (1) or (2), wherein the image selection unit selects image data with a large number of recognition regions and sends the image data to the video display unit.
(4) the plurality of video cameras are composed of one primary camera and one or more secondary cameras;
The information processing device includes a counting unit, counts the number of recognition regions, which are regions of the hand image data in which the hand is recognized by the image recognition unit, for each image data, and sends the number to the image selection unit,
The image selection unit according to any one of (1) to (3), wherein when the number of recognition regions is 0 in any image, the image data of the main camera is selected and sent to the image display unit. A multi-view video viewing system.
(5) The multi-view video viewing system according to (4), wherein the multi-view video viewing system can define any video camera as the main camera.
(6) the image recognition unit outputs the displayed position of the recognition area in which the hand is recognized and the area in the image;
The information processing device includes an average point calculation unit, calculates an average point of the center points from the center points of the one or more recognition regions output by the image recognition unit, and sends the average point to the image selection unit,
The multi-view video viewing system according to any one of (1) to (5), wherein the image selection unit selects image data in which the average point is close to the center point of the image and sends the image data to the video display unit.
(7) The multi-view video viewing according to any one of (1) to (6), wherein the image recognition unit recognizes only the image data of the gloved hand and does not recognize the image data of the ungloved hand. system.
(8) In the method of processing a plurality of multi-viewpoint image data showing the state of the object being processed by the operator's hands and selecting the image to be displayed on the display device,
an image recognition step of detecting individual image data of a plurality of video data multiplexed and recognizing hand image data relating to hands in the individual image data;
and an image selection step of selecting, from among the plurality of image data, image data in which hand image data has been recognized by the image recognition step in time series.
(9) The method according to (8) is
a counting step of counting the number of recognition regions, which are regions of the hand image data in which hands are recognized by the image recognition step, for each of the image data;
A method in which the image selection step selects image data having a large number of the recognition regions in the individual image data.
(10) the plurality of video data are composed of one main video data and one or more sub-video data;
The method according to (9), wherein the image selection step selects the image data of the main video data when the number of the recognition regions is 0 in the individual image data at the same time of any video data.
(11) The method according to any one of (8) to (10),
outputting the displayed position and the area in the image of the recognition area where the hand was recognized in the image recognition step;
calculating an average point of the center points from the center points of the one or more recognition regions output by the image recognition step;
having an average score calculation step for sending to the image selection step;
The image selection step selects image data in which the average point is close to the center point of the image.
(12) The method according to any one of (8) to (11), wherein the image recognition step recognizes only the image data of the gloved hand and does not recognize the image data of the ungloved hand.
(13) A computer program that causes an information processing device or an image recognition unit to execute each step of the method according to any one of (8) to (12).
(14) A computer-readable recording medium recording the computer program according to (13).

<Second invention>
Next, the second invention of the present application will be described. The purpose of the second invention is that the operator can complete the imaging of the affected area to be treated while concentrating on the operation without being conscious of the imaging, and one of the captured images has a high probability. To provide a camera system that captures an affected area to be treated.

A second aspect of the invention of the present application relates to a camera unit composed of a camera and a plurality of light sources, and a shadowless light is provided by arranging a plurality of the camera units so that the range of illumination by the light source is within the range photographed by the camera. With this configuration, it is assumed that a plurality of cameras are distributed and arranged inside a set of a plurality of light sources of the shadowless lamp, and when any light source hits the affected area to be treated, the vicinity of this light source can be detected at the same time. To realize a situation in which photographing is completed by a camera located at . In this situation, the practitioner can capture images of the affected area without changing the position of the camera while maintaining stable lighting on the affected area to be treated without being conscious of the imaging situation. It is possible to complete. Also, even when the surgical lamp is moved to improve the stable illumination of the affected area to be treated, the affected area to be treated can be photographed by any one of the cameras.

In order to achieve the above object, the second invention of the present application is a camera equipped with a shadowless light-equipped camera mechanism for photographing an affected area to be treated during medical treatment such as surgery, and a monitor mechanism for displaying the image photographed by the camera mechanism. system, the camera mechanism with the surgical light has the same overall form as a surgical light commonly used in surgery, is equipped with a plurality of light sources, and can be moved and tilted in the same manner as a general surgical light. A possible overall base, an operating handle to which a sterile cover can be attached approximately in the center of the overall base, a central camera unit in the center of the overall base, and at least one peripheral camera unit around the central camera unit. It is characterized by The overall form and configuration are similar to those of a general operating light, and the practitioner can illuminate the affected area to be treated as if operating a general operating light.

At this time, the surrounding camera unit includes a surrounding camera in the vicinity of a plurality of light sources provided therein. If any of the lights reaches the affected area to be treated, there is a high possibility that the affected area is being imaged by the surrounding cameras located near the light source that has reached the affected area. Conversely, if the illumination does not reach the affected area to be treated, there is a high possibility that the affected area will not appear in the image being captured, but in a situation where the illumination does not reach the affected area, surgery will not be possible. Because it is difficult to implement, it is generally unnecessary for a practitioner to grasp the operating handle to move and tilt the entire base of the surgical light camera mechanism so that illumination reaches the affected area. Similar to the operation in a shadow light, this operation allows a return to the situation where the affected area is imaged by a camera near any light source. During this operation, the practitioner does not need to be concerned about the imaging situation. A light source is essential for the peripheral camera unit, but it is optional whether or not the central camera unit is provided with a light source.

Images from a plurality of cameras are input to the monitor mechanism, and the image displayed on the display device is preferably that of the central camera in the initial setting, but depending on the surgical situation, the images of the surrounding cameras may have more information. By operating an operation device provided in the monitor mechanism, it is possible to switch images on the display device or display a plurality of images at the same time.

The second invention of the present application provides a function to support effective utilization of moving images related to medical care such as surgery with the above configuration,
・The operator can complete imaging of the affected area to be treated simply by performing the operation with the same feeling as operating a conventional operating light without being conscious of imaging.
・There is no need to allocate personnel for filming.
・When obstructions obstruct the field of view of the camera, the illumination from the light source will not reach the affected area to be treated. Visual impairment can also be improved by moving and tilting.
・It is possible to display the image of the affected area to be treated more reliably on the display, and staff such as nurses involved in surgery, anesthesiologists, doctors and students who are observers of surgery, and administrators of the operating room , more accurate surgery progress can be grasped without having to peer into the actual site of surgery from behind the operator, efficient operation of the operating room and occurrence of wound infection due to reduction of contamination. It is expected that the number will decrease.
・Other uses of video footage of medical procedures, including surgery, in medical settings include investigating the causes of medical accidents, improving patient satisfaction through more specific explanations, and improving surgeons and young medical staff. It can be expected to have effects such as improving the education of

In the field of dental care, Japanese Patent Application Laid-Open No. 2012-120812 (Patent Document 3) proposes means for combining a shadowless light with a video camera system that outputs multi-viewpoint images from a plurality of video cameras. Even if this means is simply applied to a medical operation that treats the whole body, good imaging cannot be achieved.

First of all, in order to photograph the affected area to be treated in dentistry from multiple viewpoints, multiple cameras can photograph the cleft, which is the entrance to the oral cavity, and the structures at least down to the molars through the space between the upper and lower incisors. so you need to place a video camera. Even in adult males, the cleft is about 6 cm wide, the distance between the upper and lower incisors is about 5 cm, and the depth to the third molar is about 6 cm. In order for the dentist to operate the operating light in the position of examination in dentistry, the operating light should be at a distance of at most 60 to 70 cm from the patient. In order to photograph molar teeth from a plurality of viewpoints, it is necessary to arrange the cameras within a distance of about 15 cm. Of course, if the mouth is small, such as a child or a woman, the camera should be placed at a closer position. In this case, the distance between the cameras is narrower than the width of the operator's head, which may cause visual disturbance during surgery, and it is difficult to photograph the affected area to be treated in medical surgery. Medical surgery treats a wider range of the body than dental care, and the number of staff involved in the treatment is large. It is necessary to provide a light source in the area of . If we do not pay attention to the positional relationship between the camera and the light source, even if this operating light is equipped with multiple cameras, as described above, the affected area to be treated will be illuminated, but not captured by the camera. . In order to notice this situation, it is necessary for a third party to monitor the display, and in order to recover from the situation in which shooting is not achieved, it is necessary to operate the position and direction of the camera together with the operating light. , which interferes with the progress of therapy and may cause harm to the patient.

Next, an embodiment embodying the second invention of the present application will be described in detail below with reference to FIG. 8 and subsequent drawings. The figures are drawn schematically for convenience of explanation.

FIG. 8 shows an embodiment of the invention in which the camera system 110 consists of one central camera unit 116 and six surrounding camera units 126 . The light source of the shadowless light is not essential for the light source 124 of the central camera unit, and the light source 132 of the surrounding camera unit may be arranged so as to surround the surrounding camera 130 as in the embodiment of FIG. 1, or side by side. can be arranged as follows. The central camera 122 may be arranged side by side with the operating handle 120 as in the embodiment of FIG. 8, or may be installed inside the operating handle. Images from each camera are displayed on the display device 142 of the monitor mechanism 140 . The number of light camera mechanisms 112 and monitor mechanisms 140 is at least one to one, but may be many to one, one to many, or many to many. The monitor mechanism 140 is not limited to the case where it is fixed to the wall surface in the operating room, or the case where it is movable on a cart, and the case where it is installed in the operating room or the staff room of the medical office via the hospital network, without the consent of the patient. For example, it may be installed in another facility and the status of surgery may be viewed as live surgery via the Internet. Images to be displayed on the display device 142 can be selected using the operation device 144 . The operation device 144 may be a button installed on the monitor mechanism 140, a mouse or a keyboard, a touch panel as the display device 142, or a remote controller wirelessly connected to the monitor mechanism 140. , may be a combination of these. Putting the remote control in a sterile plastic bag allows staff wearing sterile gowns to select images. The display device 142 can magnify and display a part of the displayed image, and when the affected area to be treated appears small, the affected area can be selected and enlarged for display. . By displaying a pointer on the display device 142 and operating it with a mouse or the like, or by using the display device 142 as a touch panel, it is possible to select an area to be enlarged.

FIG. 9 shows a configuration in which the operation device 144 selects an image to be displayed on the display device 142 from images of the central camera 122 and the six peripheral cameras 130 . At the site of surgery, it is not the operator who views the images on the display, but the operating room nurses and visitors who do not participate in the surgery. The present invention provides means for selecting, displaying, and enlarging a plurality of captured images by means of a monitor mechanism. It is possible to specify ON or OFF of display for each image, and if there is no specification, only the image of the central camera 122 is displayed on the display device 142 . The screen area of the display device is automatically divided according to the number of cameras designated to be ON, and images are displayed side by side. can be adjusted with Also, the image displayed in each divided area can be enlarged.

FIG. 10 shows a configuration in which the recording unit 162 simultaneously inputs images from the central camera 122 and six surrounding cameras 130 . A switch (not shown) for starting and stopping recording is provided, and operation of one switch starts and stops recording for all cameras. The switch may be provided in the recording unit, provided in the operation handle 120 illustrated in FIG. 8, provided in the remote controller, or a combination thereof.

In FIG. 11, a plurality of images taken simultaneously by the central camera 122 and the two surrounding cameras 130 are recorded in the recording device 164 with the common shooting date and time added, and output in any combination via the output means 168. A plurality of moving images are played back at the same time with the recorded shooting date/time signal Sg1 from the shooting date/time signal adding device 166, and the images are displayed on the display device 142 of the monitor mechanism 140. When playing back multiple videos at the same time, in order to play back the same scene by matching the actual time it was shot, if there is a discrepancy in the recording time recorded for each video, make fine adjustments to the playback time. there were times when it was necessary. In order to eliminate this trouble, the present invention includes means for recording a plurality of images in parallel at the same recording time, and means for simultaneously reproducing or outputting the plurality of recorded images as data at the same time. offer. In the present invention, by playing back a plurality of moving images with the shooting date and time aligned by the common shooting date and time signal Sg1, it is possible to switch the screen to the video of another camera without any time lag, or to display them at the same time. is. Although two peripheral cameras 130 are schematically illustrated, in practice, three or more peripheral cameras 130 surround the central camera 122 in order to avoid the possibility of visual disturbance. is desirable. The recording unit 162 may be configured separately from the monitor mechanism 140 as illustrated in FIG. 11, or may be incorporated inside the monitor mechanism 140 . In some cases, the output means 168 is provided in the recording unit 162 as illustrated in FIG. may play a role. 11, when the output destination of the moving image is the display device 142, while simultaneously reproducing a plurality of moving images output simultaneously, through the operation device 144, similar to the case illustrated in FIG. In addition, it is possible to select a moving image to be displayed on the display device 142 and adjust the configuration of the screen. Also, the output destination may be a storage medium such as a DVD instead of a display. may be output as one moving image file in which the moving images of the combination of are displayed side by side in an arbitrary arrangement.

FIG. 12 shows an embodiment in which the operating handle 120 and the tilting unit 184 of each surrounding camera unit 126 are mechanically linked by a linking device 182 and the tilting unit 184 is operated by rotating the operating handle 120 . The link device 182 may use bevel gears or helical gears (not shown) to change the direction of rotational force and mechanically transmit the force to the tilting unit 184 . In FIG. 12, the surrounding camera unit 126 has a circular shape, and by rotating on an axis that matches the diameter of this circle, the shadowless light-equipped camera mechanism 112 is tilted with respect to the center. The shape of the unit 126 is not limited to a circle, and the tilting center or fulcrum may be shifted from the center of the peripheral camera unit 126 as long as the method of tilting is relative to the center of the shadowless light-equipped camera mechanism 112 . As another example, the linkage 182 may be electrically operated to transmit an electronic signal to activate a motor provided in the tilting unit 184 to achieve tilting.

FIG. 13 shows an embodiment of the guide light irradiated to the photographing object 190 when the camera mechanism with the shadowless light is equipped with a guide light irradiation device for informing the direction of the image of the camera. By irradiating the subject 190 with guide light 192 along the vertical direction of the image and guide light 194 along the horizontal direction of the image, the practitioner can perform the treatment without checking the image on the display while watching the guide light. By manipulating the shadow-lit camera mechanism, it is possible to adjust the tilt of the photographing object 190 appearing in the image. The guide light is a combination of linear lights and should display a pointer in one direction to indicate a specific direction of the image. In this example, the linear light 192 with an arrowhead indicates a vertically upward direction, but this form is arbitrary. If the tilts of the images of a plurality of cameras are parallel to the horizontal direction, one guide light irradiation device is sufficient.

It is preferable that the irradiation of the guide light is automatically output when the camera mechanism with the shadowless lamp is moved or tilted, and that the guide light is automatically extinguished after a certain period of time has elapsed. An acceleration sensor is used to detect the occurrence of movement and tilting of the surgical light-equipped camera mechanism. Of course, the occurrence of these motions is detected not only by the acceleration sensor, but also by means such as an internal sensor that measures the movement and tilting of the camera mechanism with the shadowless light and the angles of the joints required for the tilting of the peripheral camera unit. It is possible.
Also, the guide light must be safe for the human body, especially the eyeball.

FIG. 14 shows an embodiment of the present invention in which a camera mechanism 112 with shadowless light is provided with a microphone 202, and a recording device 204 and a voice output device 206 are provided with a microphone 202 for outputting sound. In surgery, staff conversation is also important information. In particular, when instructing intern staff who participate in surgery wearing sterile gowns, it is difficult to accurately remember what was instructed because it is not possible to take notes during the operation. In addition, although the content of the guidance deserves to be shared with the operating room nurses and the visitors, there are cases where the sound volume of the operator's utterances is not sufficient and the surrounding people cannot hear them. For convenience of explanation, the central camera 122, the monitor mechanism 140, and the like in FIG. 8 are omitted, but their configurations are the same as in FIG. FIG. 14 illustrates a case where the microphone 202 is arranged side by side with the operation handle 120, but for effective sound collection, the microphone 202 should be installed near the tip of the operation handle 120 near the operator. is desirable. In addition, since the sound emitted by medical equipment such as electrocardiograms and blood oxygen saturation monitors and medical equipment such as electric scalpels may echo in the operating room, the microphone 202 has a noise canceling function. A unidirectional microphone along the direction of lighting and shooting is desirable.

FIG. 15 shows an embodiment of the present invention in which a microphone 202, an audio output device 206 and a recording device 204 are added to FIG. The audio output device 206 may be incorporated in the monitor mechanism 140 as shown, may be provided independently, or may be a device such as an earphone worn by the staff. It is preferable that the recording device 204 is incorporated in the recording unit 162 as shown in the drawing, and the recording is performed by adding the same shooting date and time as the simultaneously recorded moving image. When outputting moving images, it is possible to simultaneously output moving images and sounds through the output means 168 with the shooting date and time aligned. By aligning the common shooting date and time and reproducing multiple moving images and sounds, it is possible to switch the screen to the image of another camera without time lag, or to display it at the same time.

FIG. 16 shows an embodiment of the present invention in which the central camera 122 of the camera mechanism 112 with shadowless light is a semi-spherical camera, a stereo camera, or an infrared camera 222, and one of the cameras 222 is provided at the tip of the operating handle 120. be.
In the operating room, it is desirable to grasp not only images of the affected area to be treated, but also a wide range of conditions such as the arrangement and operation of staff and surgical instruments. The present invention provides means for acquiring a wide range of images by a hemisphere camera. When a semi-spherical camera is used, a central camera may be provided in the central camera unit 116 separately from the semi-spherical camera. It is desirable that the semi-spherical camera be installed at the tip of the operation handle 120 . Although the monitor mechanism 140 in FIG. 8 is omitted in FIG. 16, the image of the semi-spherical camera 222 is also input to the monitor mechanism 140 in the same manner as the other cameras, and the operation device 144 is used to select and control images. The expansion range can be adjusted.

In addition, in terms of grasping the condition of the affected area to be treated, the information obtained from the two-dimensional display image is inferior to the information obtained from the three-dimensional display image. The present invention provides means for displaying images in three dimensions by the stereo camera 222 . When a stereo camera is used, it can be selected by the operation device 144 of the monitor mechanism 140 whether the image on the display device 142 is to be displayed two-dimensionally or three-dimensionally. The stereo camera 222 may be a combination of any two of the central camera 122 and the surrounding cameras 130 , or one or more of the cameras may be the stereo camera 222 .

In addition, an infrared camera is often used in an operating room to evaluate an affected area to be treated. At this time, it is troublesome to bring the infrared camera into the vicinity of the affected area in a sterilized bag and take pictures. The present invention provides a means for displaying the image of the infrared camera by omitting this trouble. When infrared cameras are used, it is sufficient if one of the central camera 122 or the surrounding cameras 130 is an infrared camera or can be switched to an infrared camera.

FIG. 17 shows switches 234, 236, 242, 246, and 248 on the side of the operating handle 120, which are operated by the operator during surgery to change the light source, angle of view, and focus of each camera via electronic signals. The operation of the automatic adjustment function can be switched, recording can be started and stopped, and the automatic focus adjustment function of each camera can also be operated by detecting the tilt of the unit base body by the tilt detector 244. 1 shows a possible configuration in an embodiment of the present invention; In the past, camera settings, recording, and recording operations could not be performed by staff wearing sterile gowns. It was difficult. The present invention provides a camera system having means for operating the light source, camera angle of view, autofocus setting, sound recording, and sound recording on the handle portion to which the sterile cover is attached in a general shadowless lamp. The operating handle 120 is fitted with a sterile cover so that it can be operated by an operator wearing a sterile gown during surgery. At this time, it is desirable that the cover be made of a transparent and soft material such as vinyl or thin plastic so that the switches 234, 236, 242, 246, and 248 can be operated without hindrance. Also, the positions of switches 234, 236, 242, 246, 248 may be incorrectly set when the operating handle 120 is touched to move and tilt the entire surgical light camera mechanism or tilt the ambient camera unit. The switches 234 , 236 , 242 , 246 , 248 should be positioned such that they cannot be actuated.
It is desirable that the automatic adjustment function of the camera automatically operates when the positional relationship between the camera and the affected area to be treated changes. , or the occurrence of tilting of the surrounding camera unit 126 can be detected by using an acceleration sensor provided in any of the surrounding camera units 126. Therefore, it is desirable to use an acceleration sensor. Of course, not only the acceleration sensor, but also means such as an internal sensor for measuring the movement and tilting of the camera mechanism 112 with the shadowless light and the angles of the joints required for the tilting of the peripheral camera unit 126 can detect the occurrence of these motions. It may be detected.

FIG. 18 shows that when outputting a plurality of recorded moving images 252, 254 and audio 256 to an external recorder, instead of outputting data over the entire recording time, the range T1 to T2 of the shooting time is displayed. This embodiment of the present invention shows an embodiment of the present invention in which each data is cut out as data in the time range of the same position, and voice is synthesized with each moving image and output when data is output by limiting . In the present invention, as shown in FIGS. 11 and 15, any moving image data or audio data shares the same shooting date and time data, so if a time range T1 to T2 for extracting data is specified, all data can be extracted at once. It is possible to divide multiple moving image data by .

FIG. 19 is a flow chart showing an example of an algorithm for switching images when the display device recognizes an image failure. If there are no staff in the operating room who do not wear sterile gowns to operate the monitor mechanism, it is difficult to switch images displayed on the display. cause inconvenience. The present invention provides a camera system having means for automatically switching to another image when the monitor mechanism detects that the displayed image does not display the affected area to be treated due to image disturbance. As described with reference to FIG. 9, when no image is designated, the image of the central camera is displayed on the display device (step S101). The display device can detect from the pixel configuration of the image that there is a high possibility that the displayed image does not display the affected area, and can automatically switch to another camera's image. In surgery, the pixels of the image of the affected area have a large range of colors such as skin color and red. It can be blue or green, the common colors of surgical gowns and caps, or green, the color of the practitioner's gloves. The display device recognizes that an image disturbance has occurred when a certain proportion of the image is occupied by blue or green pixels (step S102). If YES in step S102, a monitoring time timer is set (step S103). Until the timer set time elapses, it continues to recognize whether or not the image disturbance continues (step S104). If there is no image obstruction, YES is determined in step S105, and the image of the central camera is continued to be displayed. If the image failure continues, NO is determined in step S105, and the image failure monitoring continues until the timer set time elapses. If the timer set time elapses while the image disturbance continues, YES is determined in step S104, and the image is automatically switched to another camera and displayed (step S106). After that, it is possible to detect a video failure in the displayed video in the same way and automatically switch to the video of another camera (steps S107 to S111). This recognition of image disturbance is performed simultaneously for each image when a plurality of images are displayed simultaneously. In addition, it is also possible to designate the image to be switched to as a video with no image disturbance by similarly monitoring the image disturbance in the background for the image that is not displayed.

FIG. 20 shows a configuration example of the present invention in which a distance measuring device 272 is provided near the central camera 122 and each peripheral camera 130, and an image to be displayed on the display device 142 is automatically selected by the CPU 274 provided in the monitor mechanism 140. FIG. When the distance measuring device 272 uses a laser range finder, it is preferable to adopt laser class 1 (near infrared rays) as a light source to ensure safety to the human body, especially the eyeball. The measured value of each distance measuring device 172 is transmitted to the CPU 274, and the CPU 274 detects the distance when the value changes greatly over a certain period of time from the measured value in the situation in which the affected area to be treated is photographed. It is possible to control so that the images of the cameras 122 and 130 positioned near the measuring device 272 are not displayed on the display device 142 .

FIG. 21 automatically recognizes the contents of moving image data 282 and voice data 288 by moving image recognition means 284 and voice recognition means 290, respectively, creates character string data, adds tag information of character strings to photographing time information, and lists them. Figure 2 shows the configuration of the present invention for creating data 286,292; Recorded movies are long, often monotonous in overall color and composition, and it is often difficult to identify the recording time of a surgical scene that one wishes to replay. As a means for this, there is a method in which the viewer of the video manually adds tag information to the video, but it requires a lot of time and effort. In the present invention, the recorded video and audio are mechanically and automatically recognized, and by adding tag information of character strings to the time information of the data, the time information of the recording can be searched, so that the scene to be played back can be found. provide a means by which they can be identified. Specific examples of the moving image recognition means 284 and the voice recognition means 290 include Cloud Video Intelligence and Cloud Speech APIs (application programming interfaces) provided by Google, respectively. However, since the video data 182 and the audio data 288 actually contain patient information, it is necessary to obtain the patient's permission in advance when using the API that requires communication with the Google server. be. In order to carry out the present invention without going through a network outside the hospital, it is necessary to construct and operate the moving image recognition means 284 and the voice recognition means 290 within the hospital.

The second invention can be understood as follows.
(15) A camera mechanism with a shadowless light that can be used for surgical procedures such as surgery and can capture static and moving images of the affected area to be treated and/or the hands of the operator, and a monitor that displays the images captured by this camera mechanism. In a camera system provided with a mechanism, the camera mechanism includes a movable and tiltable overall base, a central camera unit disposed substantially in the center of the overall base, and a camera provided on the overall base, the central camera unit At least one peripheral camera unit is provided on the periphery, and the central camera unit is erected on a central unit base, a central portion of the central unit base, and is operated by a practitioner to move and move the entire camera mechanism. An operation handle that can be tilted, and a central camera built into or adjacent to the operation handle, wherein the surrounding camera units are arranged on the surrounding unit base and the surrounding unit base, respectively. and a plurality of light sources arranged adjacent to the surrounding camera on the surrounding unit base body, wherein the central camera and the surrounding camera provide views of the affected area to be treated and/or the treatment target from different viewpoints. The monitor mechanism normally includes at least one display device that displays the image of the central camera, and the display device is controlled by the viewer's operation. 1. A camera system characterized by comprising an operating device capable of selecting an image to be displayed in the camera from images taken by surrounding cameras.
(16) The camera system according to (15), wherein the central camera unit comprises a plurality of light sources arranged near the central camera, the plurality of light sources forming a shadowless light.
(17) The operation device has a function of displaying a combination of two or more images among the images captured by the central camera unit and the surrounding camera units on the display device (15). Or the camera system according to (16).
(18) The camera system according to any one of (15) to (17), further comprising a recording unit that simultaneously records images taken by the central camera unit and the surrounding camera units in parallel.
(19) The camera according to (18), wherein the recording unit is capable of synchronizing and simultaneously playing back two or more images shot simultaneously by the central camera unit and the surrounding camera units. system.
(20) The surrounding camera unit further includes a tilting unit that tilts the surrounding unit base body, the operating handle and the tilting unit are connected via a link device, and the operating handle rotates along a rotation axis. actuating the tilting unit via the link device to tilt the base of the surrounding camera unit with respect to the central camera unit, thereby adjusting the direction of the surrounding camera. The camera system according to (15) or (16).
(21) The shadowless light-equipped camera mechanism includes a guide light irradiation device for irradiating a subject with guide light, which is a light beam indicating the vertical and horizontal directions of an image of either the central camera or the surrounding cameras, and a display. The camera system according to any one of (15) to (20), in which the inclination of the photographed image with respect to the affected area to be treated can be known without looking at the camera.
(22) Any of (15) to (21) provided with a microphone for acquiring voice information including conversation during surgery, a recording device for recording the acquired voice information, and a voice output device for outputting the acquired voice. The camera system according to
(23) The recording unit can synchronize time and simultaneously reproduce the video and audio shot and recorded by the central camera unit, the surrounding camera units and the microphones of (18) to (22). A camera system according to any one of the preceding claims.
(24) The camera system according to any one of (15) to (23), wherein the operation handle is provided with a semi-spherical camera at the tip of the handle and capable of recording the surrounding environment over 360°.
(25) The camera system according to any one of (15) to (23), wherein the central camera is a semi-spherical camera and capable of recording the surrounding environment over 360°.
(26) The operation handle has a device for operating the light sources of the central unit and the peripheral unit base on the side of the handle, and can send an instruction signal regarding turning on/off the light source and adjustment of illuminance (15) to (24). ).
(27) The center camera and the surrounding cameras each have a view angle operation device, and the operation handle has a switch on the side of the handle for sending an instruction signal to the view angle operation device (15) to (26). ).
(28) The central camera and the surrounding cameras each have a focal length operating device, and the focal length operating device can automatically adjust the focus according to the object to be photographed by an automatic focus adjustment function (15)-( 27) The camera system according to any one of the above.
(29) The operation handle is capable of operating activation and deactivation of the automatic focus adjustment function of the focal length operation devices of the central camera and the peripheral cameras via a switch on the side of the handle (15). The camera system according to any one of (28).
(30) The focal length operation device adjusts the angle of view of the central camera and the peripheral cameras and moves the entire camera mechanism via the operation handle when the automatic focus adjustment function is stopped. Alternatively, any of (15) to (29) capable of automatically stopping the automatic focus adjustment function after activating the automatic focus adjustment function and adjusting the focus when it is detected that a tilting operation has occurred. The camera system according to
(31) The camera system according to any one of (15) to (30), wherein the operation handle has an operation device for the recording unit attached to the side of the handle, and can be operated to start and stop recording.
(32) The camera system according to any one of (21) to (31), wherein the operation handle has an operation device for the recording device attached to the side of the handle, and can be operated to start and stop recording.
(33) Any one of (15) to (22), wherein the operation device of the monitor mechanism is capable of operating activation and deactivation of an automatic focus adjustment function of the focal length operation devices of the central camera and the peripheral cameras. The camera system according to
(34) Any one of (15) to (33), wherein at least one of the central camera and the surrounding cameras is a stereo camera, and the display device is capable of three-dimensionally displaying images captured by the stereo camera. The described camera system.
(35) The apparatus according to any one of (15) to (34), wherein at least one of the central camera and the surrounding cameras is an infrared camera, and the display device can display images captured by the infrared camera. camera system.
(36) The monitor mechanism allows the operation device for selecting an image to be displayed on the display device to display an image captured by any one of the central camera, the peripheral camera, and the semi-celestial camera within an arbitrary shooting range. The camera system according to any one of (24) to (35), in which the image can be cut out, enlarged, and displayed on the display.
(37) Any of (18) to (36), wherein the recording unit can cut out a plurality of recorded images and sounds in an arbitrary time range while maintaining time synchronization and output them as data. The camera system according to
(38) The display device detects by recognizing the pixels of the image that the displayed image does not display the imaging target such as the treatment target area and/or the hand of the practitioner, The camera system according to any one of (15) to (37), which is capable of automatically switching to display of another image when there is no improvement.
(39) The central unit base body and the surrounding unit base body are provided with a distance measuring device for optically measuring the distance to the photographed object near the central camera and the surrounding camera, and distance information is sent to the CPU provided in the monitor mechanism. , and when the CPU detects that the distance suddenly changes greatly and does not return for a certain period of time, it determines that the object to be photographed has not been photographed, and displays the image on the display device. The camera system according to any one of (15) to (38), which is capable of automatically switching to display of video from another camera.
(40) The recording unit and the recording device mechanically and automatically recognize the image information and audio information of the recorded video, and add tag information to the time information of the data, so that the recording is performed based on the tag information. A camera system according to any one of (18) to (39), which enables retrieval of time information and reproduction from a corresponding point in time.
(41) A camera mechanism with a shadowless light that can be used for surgical procedures such as surgery and can capture static and moving images of the affected area to be treated and/or the hands of the operator, and a monitor that displays the images captured by this camera mechanism. In a camera system equipped with a mechanism,
The camera mechanism is
whole base that can be moved and tilted,
at least two camera units mounted on the overall substrate; and
An operation handle that can move and tilt the entire camera mechanism,
each of the at least two camera units has a unit base and at least one camera mounted on the unit base;
The cameras of the at least two camera units are capable of simultaneously capturing fixed images and moving images of the affected area to be treated and/or the operator's hand from different viewpoints,
A camera system, wherein the camera mechanism further comprises image selection means for selecting an image to be displayed on the display device from images captured by the cameras of the at least two camera units. .

<Third invention>
A third aspect of the present invention is an acquisition unit that acquires a plurality of video data obtained by photographing the same surgical field from different viewpoints using a plurality of cameras; an evaluation unit that evaluates an operating field exposure that is the degree to which the field is captured; and one of the plurality of video data based on the operating field exposure of each image of the plurality of video data for each frame. The present invention relates to a surgical image display device including a selection unit for selecting image data and an image display unit for displaying an image of the image data selected by the selection unit on a display device. A surgical image display device is a system for recording, reproducing, and displaying surgical images such as surgical operations. The multi-view video viewing system according to the first invention of the present application described above is also a specific example of the surgical video display device.

According to the third invention of the present application, appropriate video data is selectively selected based on the operative field exposure (the degree to which the operative field is captured in the image) from a plurality of video data captured from different viewpoints. to be displayed. As a result, the user (person viewing the surgical video) can avoid videos in which the surgical field is blocked by obstacles and can selectively view images in which the surgical field is exposed. Therefore, the user can view only images showing the operative field in good condition throughout the operation, and the user can easily observe the state of the operative field (affected area or incision) and the progress of the operation.

An embodiment embodying the third invention of the present application will be described in detail with reference to the drawings. FIG. 22 shows a configuration example of a surgical image display device. As shown in FIG. 22 , the surgical image display device 300 of this embodiment has three cameras 310 , an information processing device 320 and a display device 330 .

The camera 310 is a color digital video camera, and can capture video data at 30 frames per second, for example. Each camera 310 is installed at a position such that the surgical field of the patient on the surgical bed can be imaged from above the surgical bed 350 . Each camera 310 may be attached to the operating light as shown in FIG. 8, or may be attached to a location (ceiling, arm, or the like) other than the operating light. The three cameras 310 are installed at different positions and orientations so that the operative field can be photographed from different viewpoints (directions). When the position/orientation of each camera 310 is fixed, the viewpoint information (information defining the three-dimensional position and orientation) of each camera 310 is registered in advance in the information processing device 320 . If the position/posture of each camera 310 is variable, information on the viewpoint may be sent to the information processing device 320 when the position/posture of each camera 310 is changed. Note that the number of cameras is not limited to three, and any number of cameras may be provided as long as they are two or more. In addition, it is not necessary for all cameras to have the same configuration and specifications, and cameras with different configurations and specifications may be mixed (for example, a visible light camera and an infrared light camera, a high resolution camera and a low resolution camera, etc.). cameras, color and monochrome cameras, etc.).

The information processing device 320 has an image acquisition unit 321 , an image recognition unit 322 , a score calculation unit 323 , a video selection unit 324 and a video display unit 325 . The image acquisition unit 321 has a function of acquiring video data from each of the cameras 310 . Here, the image acquisition unit 321 is synchronously input with three pieces of video data obtained by photographing the same operative field from different viewpoints. The image recognition unit 322 has a function of applying image recognition processing to the image of each frame of each video data, and detecting a predetermined object or a region (pixel group) that satisfies a predetermined condition from the image. The score calculation unit 323 has a function of evaluating the degree to which the operative field is captured in the image based on the detection result of the image recognition unit 322 and calculating an evaluation score indicating the degree. Hereinafter, this evaluation score is referred to as "operative field exposure" in the sense of the degree of exposure of the operative field to the field of view of the camera. The video selection unit 324 has a function of selecting one video data from three video data for each frame based on the operative field exposure of each image of the three video data. The image display unit 325 has a function of displaying the image of the image data selected by the image selection unit 324 on the display device 330 . In the configuration of this embodiment, the image acquisition unit 321 is an example of the “acquisition unit” of the present invention, the image recognition unit 322 and the score calculation unit 323 are examples of the “evaluation unit” of the present invention, and the video selection unit 324 is an example of the "selection section" of the present invention, and the image display section 325 is an example of the "image display section" of the present invention.

The information processing device 320 can be configured by, for example, a general-purpose computer equipped with a processor (CPU, GPU, etc.), memory, nonvolatile storage device (hard disk drive, solid state drive, flash memory, etc.), input device, and the like. can. In that case, the above-described functions 321 to 325 are implemented in software by developing a program non-temporarily stored in a storage device into a memory and executing the program by a processor. Alternatively, part of the processing and functions of the information processing device 320 may be realized by hardware such as ASIC, or may be executed by another computer such as a cloud server. A personal computer, a smart phone, a tablet terminal, or the like can be used as the information processing device 320 .

The display device 330 is a device that displays images output from the image display unit 325 . A liquid crystal display, an organic EL display, or the like can be used as the display device 330 . Note that the information processing device 320 and the display device 330 may be configured integrally.

FIG. 23 is a flow chart showing an example of processing of the information processing device 320 . FIG. 23 shows the processing performed for each frame. First, the image acquisition unit 321 acquires data of three images captured in time synchronization by the three cameras 310 (step S300). These three images are a set of images of the same surgical field taken at the same time from different viewpoints. Next, the image recognition unit 322 applies image recognition processing to each of the three images (step S301). Next, the score calculator 323 calculates the operating field exposure of each image based on the result of image recognition (step S302). Next, the image selection unit 324 selects the image with the highest operative field exposure from the three images (step S303). Then, the image display unit 325 displays the image selected in step S303 on the display device 330 (step S304). By executing the above-described processing for each frame, an image that best captures the operative field is selected from the images of the plurality of cameras 310 and displayed on the display device 330 .

Next, a specific example of processing by the image recognition unit 322 and the score calculation unit 323 will be described. In this embodiment, focusing on the fact that the surgical field is often blocked by the heads of doctors and nurses, a "hat" is detected by image recognition. Then, the smaller the area of the hat region in the image (that is, the ratio of the hat region in the image), the higher the operative field exposure, and conversely, the larger the area of the hat region in the image, the smaller the operative field exposure. evaluate. The color and shape of the surgical cap is known in advance, and the cap cannot become contaminated during surgery. Therefore, image recognition of the hat is relatively easy, and has the advantage of obtaining stable recognition results.

24A and 24B show examples of images captured from a camera 310 placed above the surgical field. In FIG. 24A, the surgical field (incision 400) is shown approximately in the center of the image, and the surgeon's hands 401, surgical instruments 402, and a cap 403 worn by the surgeon are shown around it. FIG. 24B shows an example in which the surgeon is in a position to look into the incision 400 so that the hat 403 blocks most of the operative field from the view of the camera 310 . For these examples, it should be appreciated that the operative field exposure is greater in the image of FIG. 24A than in the image of FIG. 24B.

Any algorithm may be adopted as a method for recognizing the hat 403 from the image. For example, the surgical cap 403 has a predetermined color, such as blue, green, and white, so a recognition algorithm based on color features is preferred. The color feature of the hat may be set (instructed) by the user, or machine-learned using a large number of surgical images (training data). Alternatively, since the shape of the head is generally round, a recognition algorithm that combines color features and shape features may be employed. Furthermore, in order to facilitate image recognition or improve accuracy, doctors and nurses may be made to wear hats with special colors, textures, or shapes, or hats with markers for image recognition.

When the image recognition unit 322 in FIG. 22 detects the hat 403, it extracts the area of the hat 403 from the image. A known segmentation algorithm can be used for region extraction. After that, the score calculator 323 calculates the area of the hat region (for example, the number of pixels), and obtains the operative field exposure using the following equation.

Operating field exposure = (Area of entire image - Area of hat region) / Area of entire image

The operating field exposure has a maximum value of 1 when the hat is not captured in the image, and a minimum value of 0 when the entire image is completely blocked by the hat (when the operating field is not captured at all). .

FIG. 25 shows an example of changes in the operating field exposure for each frame and switching of images to be displayed. In the frame at time t1 shown in the upper part of FIG. 25, the image I2 has the highest surgical field exposure among the images I1 to I3 of the three cameras, and the image I2 is displayed on the display device 330 . Thereafter, in the frame at time t2, as shown in the lower part of FIG. 25, the operating field exposure of image I3 is the highest, and image I3 is displayed on display device 330 (FIG. 22). By selectively displaying an image of a camera with a high operative field exposure degree in this way, the user can always observe the state of the operative field.

In this embodiment, an example of detecting a "hat" as an object blocking the operative field has been described, but the method of evaluating the operative field exposure is not limited to this. For example, the image recognizer 322 may detect objects that may be present around the surgical field (eg, hands, surgical instruments, etc.). This is because there is a high probability that the operative field will also appear in the image when objects that may exist around the operative field (hands, surgical instruments, etc.) are captured in the image. In this case, it can be evaluated that the image in which the hand and the surgical instrument are detected has a higher surgical field exposure. In addition, it may be evaluated that the greater the number of hands and surgical instruments detected from the image, the greater the exposure of the surgical field. The same algorithm as described in the first invention can be used for hand detection. Recognition algorithms based on color features or shape features, for example, may be used to detect surgical instruments. Image recognition may also be facilitated by attaching markers for image recognition to gloves or surgical instruments.

The image recognition unit 322 may detect the surgical field itself (for example, an incision). Since areas other than the surgical field (incision) are generally covered with a sterile drape, the skin color and reddish color often appear only in the surgical field (incision). Therefore, a region that satisfies the color condition that it has color characteristics such as skin color or reddish color may be recognized as the operative field. In addition, since the operating field is always exposed to the light of the shadowless light, a brighter image can be obtained than the area other than the operating field. may be recognized as the operative field. Note that the color features and luminance thresholds to be recognized as the surgical field may be set (instructed) by the user, or may be machine-learned using a large number of surgical images (training data). In this case, the larger the area of the region that satisfies the color condition or the luminance condition, the higher the operative field exposure.

The image selection unit 324 of this embodiment selects the image with the highest surgical field exposure from among the three images as the image for display. It may make viewers feel uncomfortable. Therefore, it is preferable that the image selection unit 324 determines the selection image not only based on the magnitude of the operative field exposure but also considering the switching frequency of the cameras. For example, the image selection unit 324 determines the selected image by solving an optimization problem set to maximize the exposure of the surgical field of the image and to reduce the switching frequency as much as possible. You may As an example, the reciprocal of the surgical field exposure is defined as the surgical field shielding degree, and an objective function (cost function) including a term of the surgical field shielding degree and a term of the switching frequency is set, and the value of this objective function is the minimum. A method for controlling switching of images can be adopted as follows. At this time, the user may be allowed to adjust parameters for determining the respective weights of the surgical field shielding degree and switching frequency in the objective function. By providing such a parameter adjustment function, it is possible for the user to adjust whether to give priority to displaying an image in which the exposure of the operative field is as large as possible or to giving priority to reducing the switching of images as much as possible. Become.

FIG. 26 shows an example of an image switching control algorithm to which Dijkstra's method, which is a method for solving the shortest path problem, is applied. For convenience of explanation, FIG. 26 shows switching control of three cameras (three images), but it is also applicable to switching of four or more cameras. First, preset the minimum number of frames n (n is an integer of 2 or more) for prohibiting image switching (for example, if n = 10, the same camera image will continue to be displayed for at least 10 frames). guarantee that.). The camera C1 selected at a certain frame t is considered as a vertex, and the next camera that can be selected is connected by an edge. In the case of the same cameras, the frame t and the next frame t+1 are connected by a side 500, and as the weight of the side 500, the operative field shielding degree of the image of the camera C1 in the frame t+1 is set. In the case of different cameras, the side 501 connects the camera C1 in the frame t and the camera C2 in the frame t+n, and the weight of the side 501 is the sum of the degree of operative field shielding of the images of the camera C2 from the frame t+1 to the frame t+n. set. Similarly, the camera C1 in the frame t and the camera C3 in the frame t+n are connected by a side 502, and as the weight of the side 502, the total value of the degree of operative field shielding of the images of the camera C3 from the frame t+1 to the frame t+n is set. . After creating the graph structure in this way, the optimal combination problem is solved by Dijkstra's algorithm. Note that when the side 501 (switching to a different camera) is selected, the camera is switched to the camera C2 at the time of the frame t+1, and the image of the camera C2 is continuously displayed for n frames from the frame t+1 to the frame t+n. do. The same is true when side 502 is selected. With such an algorithm, images are selected so that the degree of operative field occluded is minimized while prohibiting image switching from occurring in a period shorter than n frames.

The image display unit 325 in FIG. 22 extracts an area including the operative field (referred to as “operative field area”) from the image selected by the image selection unit 324, and enlarges and displays the extracted operative field area on the display device 330. may This makes it possible to observe the operative field more appropriately. Extraction of the surgical field region may be performed, for example, by recognizing the surgical field region using a classifier that has learned image features of the surgical field region using a large number of surgical images (training data), or by recognizing the surgical field region. It may be performed by extracting the luminance region, or may be extracted based on the instruction of the surgical field region by the user. Methods of teaching the operating field region by the user include a method of specifying the region with a pointing device, a method of teaching the region of interest by inputting a line of sight, and the like. Note that these teaching methods can also be applied when setting an operating field region in training data.

FIG. 27 shows another configuration example of the surgical image display device. The configuration differs from that of FIG. 22 in that an image conversion section 326 is provided between the video selection section 324 and the video display section 325 . This image conversion unit 326 has a function of converting an image of the camera into an image viewed from a reference viewpoint, based on the information of the viewpoint of the camera.

ここで、（ｘ１，ｙ１）は画像Ｉ１の座標であり、（ｘｉ，ｙｉ）は画像Ｉｉ（ｉ＝２，３）の座標であり、Ａｉは画像Ｉｉの座標を画像Ｉの座標に変換するための変換行列である。変換行列Ａｉにより、画像の回転、拡大／縮小などの変換が可能である。なお、前述のように、各カメラの視点の情報は既知であるため、変換行列Ａｉはこれらの視点の情報から幾何学的計算により求めることができる。For example, if three cameras are C1 to C3, and the viewpoint of camera C1 is set as a reference viewpoint, the following conversion formula converts images I2 to I3 of cameras C2 to C3 into images viewed from the viewpoint of camera C1. can be converted.

where (x1,y1) are the coordinates of image I1, (xi,yi) are the coordinates of image Ii (i=2,3), and Ai transforms the coordinates of image Ii to the coordinates of image I. is a transformation matrix for The transformation matrix Ai enables transformation such as image rotation and enlargement/reduction. As described above, since the viewpoint information of each camera is known, the transformation matrix Ai can be obtained by geometric calculation from the viewpoint information.

By performing such image conversion, it is possible to align the viewpoints of the images to be displayed (that is, the appearance of the images). Therefore, the discontinuity of the image at the time of camera switching can be reduced as much as possible, so that even when the camera is switched according to the exposure of the surgical field, viewing can be continued without discomfort.

Further, the image conversion unit 326 can acquire depth information of each pixel included in each camera image based on the images of a plurality of cameras, and convert the acquired depth information into a three-dimensional image. The image display unit 325 can display the operative field as a three-dimensional image in which the head disappears by omitting pixels whose depth information is smaller than a certain value. Also, the video display unit 325 can rotate the three-dimensional image created by the image conversion unit 326 and display it from an arbitrary viewpoint according to the viewer's operation. By adopting a head-mounted display as the display device 330, the viewer can more intuitively observe the three-dimensional image, and can have a viewing experience as if he were participating in an actual surgery.

10 subject 12 main camera 14 sub camera 20 information processing device 22 image acquisition unit 24 control unit 26 image recognition unit 28 counting unit 30 average score calculation unit 32 video selection unit (image selection unit)
34 video display unit 40 display devices 52, 54, 56, 58 areas where hands are recognized (recognition areas)
62, 64, 66, 68 Center point 70 of recognition area Average point 80 of center points of recognition area Center point 90 of image Distance between average point and center point of image

Claims (20)

A surgical image display device,
an acquisition unit that acquires a plurality of image data obtained by photographing the same surgical field from different viewpoints using a plurality of cameras;
an evaluation unit that evaluates the operative field exposure, which is the degree to which the operative field is captured in the image, for each frame image of each video data;
a selection unit that selects video data from among the plurality of video data based on the operative field exposure of each image of the plurality of video data for each frame;
a video display unit for displaying an image of the video data selected by the selection unit on a display device;
with
The evaluation unit detects an exposed area that is not covered with a sterile drape from the image as the operative field, and evaluates that the larger the area of the detected area, the greater the degree of exposure of the operative field. Surgical image display device. A surgical image display device,
an acquisition unit that acquires a plurality of image data obtained by photographing the same surgical field from different viewpoints using a plurality of cameras;
an evaluation unit that evaluates the operative field exposure, which is the degree to which the operative field is captured in the image, for each frame image of each video data;
a selection unit that selects video data from among the plurality of video data based on the operative field exposure of each image of the plurality of video data for each frame;
a video display unit for displaying an image of the video data selected by the selection unit on a display device;
with
The selection unit solves an optimization problem set so as to increase the exposure of the surgical field of the image as much as possible and to reduce the switching frequency of the image data as much as possible, thereby selecting the image data to be selected. A surgical image display device characterized by determining the 3. The surgical image display apparatus according to claim 2, wherein the selector has a function of allowing a user to adjust parameters for determining the weight of the surgical field exposure and the switching frequency. A surgical image display device,
an acquisition unit that acquires a plurality of image data obtained by photographing the same surgical field from different viewpoints using a plurality of cameras;
an evaluation unit that evaluates the operative field exposure, which is the degree to which the operative field is captured in the image, for each frame image of each video data;
a selection unit that selects video data from among the plurality of video data based on the operative field exposure of each image of the plurality of video data for each frame;
a video display unit for displaying an image of the video data selected by the selection unit on a display device;
with
further comprising an image conversion unit that converts an image of video data captured by each camera into an image viewed from the same viewpoint based on viewpoint information of each camera;
The surgical image display device, wherein the image display unit displays the image converted by the image conversion unit on a display device. The evaluation unit detects at least one of the operative field, objects that may exist around the operative field, and objects that block the operative field from the image, and determines whether the operative image is processed based on the detection result. 5. The surgical image display device according to any one of claims 1 to 4, wherein field exposure is evaluated. The evaluation unit detects a hand as an object that may exist around the surgical field from the image, and the image in which the hand is detected has a higher surgical field exposure than the image in which the hand is not detected. 6. The surgical image display device according to claim 5 , wherein the evaluation is such that the is large. 7. The surgical image display apparatus according to claim 6 , wherein the evaluation unit evaluates that the greater the number of hands detected from the image, the greater the exposure of the surgical field. The evaluation unit detects a cap from the image as an object that obstructs the operative field, and evaluates that the operative field exposure is lower in the image in which the hat is detected than in the image in which the hat is not detected. 6. The surgical image display device according to claim 5 , characterized in that: 9. The surgical image display apparatus according to claim 8 , wherein the evaluation unit evaluates that the larger the area of the hat region detected from the image, the smaller the surgical field exposure. The evaluation unit detects an incision as the operative field from the images, and evaluates that the image in which the incision is detected has a higher operative field exposure than the image in which the incision is not detected. 6. The surgical image display device according to claim 5 , characterized in that: The evaluation unit detects a region from the image as the operative field as the operative field, and evaluates that the larger the area of the region that satisfies the predetermined color condition, the higher the operative field exposure. 6. The surgical image display device according to claim 5 . The evaluation unit detects a region as the surgical field from the image, and evaluates that the larger the area of the region that satisfies the predetermined brightness condition, the higher the surgical field exposure. 6. The surgical image display device according to claim 5 . The surgical image display according to any one of claims 1 to 12 , wherein the selection unit selects, from among the plurality of image data, image data having the highest surgical field exposure of the image. Device. 13. The image display unit extracts an area including the surgical field from the image of the image data selected by the selection unit, and enlarges and displays the extracted area on a display device. The surgical image display device according to any one of the above. further comprising a shadowless light for illuminating the surgical field;
The surgical image display apparatus according to any one of claims 1 to 14 , wherein the plurality of cameras are provided in the shadowless lamp. A control method for a surgical image display device,
Using a plurality of cameras to acquire a plurality of video data of the same operative field photographed from different viewpoints;
a step of evaluating the degree of exposure of the surgical field, which is the degree to which the surgical field is shown in the image, for each frame image of each piece of video data;
selecting video data from among the plurality of video data based on the operative field exposure of each image of the plurality of video data for each frame;
displaying an image of the selected video data on a display device;
has
In the evaluating step, the area exposed without being covered with a sterile drape is detected as the operative field from the image, and the larger the area of the detected area, the greater the operative field exposure is evaluated. A control method for a surgical image display device. A control method for a surgical image display device,
Using a plurality of cameras to acquire a plurality of video data of the same operative field photographed from different viewpoints;
a step of evaluating the degree of exposure of the surgical field, which is the degree to which the surgical field is shown in the image, for each frame image of each piece of video data;
selecting video data from among the plurality of video data based on the operative field exposure of each image of the plurality of video data for each frame;
displaying an image of the selected video data on a display device;
has
In the selecting step, the image to be selected is solved by solving an optimization problem set to maximize the exposure of the surgical field of the image and to reduce the switching frequency of the image data as much as possible. A control method for a surgical image display device, comprising determining data. A control method for a surgical image display device,
Using a plurality of cameras to acquire a plurality of video data of the same operative field photographed from different viewpoints;
a step of converting an image of video data captured by each camera into an image viewed from the same viewpoint based on the viewpoint information of each camera;
a step of evaluating the degree of exposure of the surgical field, which is the degree to which the surgical field is shown in the image, for each frame image of each piece of video data;
selecting video data from among the plurality of video data based on the operative field exposure of each image of the plurality of video data for each frame;
displaying an image of the selected video data on a display device;
has
A control method for a surgical image display device, wherein an image that has been converted into an image viewed from the same viewpoint is displayed on the display device. A program for causing a processor provided in a surgical image display device to execute each step of the control method according to any one of claims 16 to 18 . A computer-readable recording medium storing the program according to claim 19 non-temporarily.

Images