JP6192602B2 - Image recording device - Google Patents

Description

  The present invention relates to an image recording apparatus that records an image so as to facilitate editing.

In recent years, endoscopes have been widely used in the medical field. For example, Japanese Patent Laid-Open No. 2006-61469, which is a first conventional example, images a patient's body cavity in time series using a capsule endoscope, and a time scale indicating an imaging period of a series of images taken is attached. A time bar is displayed, and an index indicating a position on the time scale corresponding to the imaging time of the selected desired image is displayed. This publication also discloses measuring the color distribution of an image, determining whether the image is an extracorporeal image from the measurement result, and displaying a mark indicating the imaging period of the extracorporeal image.
Japanese Patent Laid-Open No. 2005-124824 as a second conventional example describes a surgical support system for generating a surgical simulation image in which surgical moving image data is necessary for each procedure step necessary for constructing the surgical simulation image. It is disclosed to divide into optimal procedure step moving image data and register the divided procedure step moving image data in a database. This publication also discloses that, during surgery, index data is added to the surgical video data based on the depression of an operation button or the like of an endoscope for a mark that divides the surgical video data.

JP 2006-61469 A JP 2005-124824 A

Usually, when performing a single operation, it is desirable to efficiently extract an image portion of a desired scene in order to record an endoscopic image in an image recording apparatus from the start to the end of the operation. In the first conventional example, an imaging position of a selected image in a series of images recorded in time series can be easily estimated from a display screen. It does not disclose efficient extraction.
In addition, the second conventional example discloses that a mark to be divided is attached by an operation of the user during the operation. In the conventional example, the user concentrates on the operation during the operation. However, there are drawbacks that require cumbersome operations, and forgetting to add marks to be divided.
SUMMARY An advantage of some aspects of the invention is that it provides an image recording apparatus that can easily extract an image of a desired scene.

  An image recording apparatus according to an aspect of the present invention receives an image signal input unit to which an image signal is input, and an image analysis unit that extracts an image feature amount in the image signal input to the image signal input unit. An image type database unit that stores a database of a plurality of image types associated with feature amounts for discrimination for each image type corresponding to the progress of surgery in advance for the image of the image signal, and extracted by the image analysis unit An image discriminating unit that discriminates a corresponding image type in the image type database unit with respect to the image of the feature amount, and a change corresponding to the change in the image type discriminated by the discrimination result of the image discriminating unit An edit point for specifying the image signal at the specified timing and a first meta information related to the image signal are added to the image signal for recording. A recording data generating unit for generating data, a recording unit for recording the recording data including the image signal, the editing point and the first meta information generated by the recording data generating unit; An information input unit for inputting information on the operating state of a plurality of peripheral devices to be used, and a state for determining each operating state of the plurality of peripheral devices corresponding to the information from the information input to the information input unit A second meta information related to the plurality of peripheral devices at the timing when the determination point corresponding to the change in the determination result of the determination unit and the determination unit of the state determination unit and the determination result of the state determination unit is changed in the image signal And a control unit that controls to record in the recording unit.

  According to the present invention, an image of a desired scene can be easily extracted.

FIG. 1 is a diagram illustrating an overall configuration of an endoscope system including an image recording apparatus according to a first embodiment of the present invention. 2 is a diagram showing an overall configuration of an endoscope system showing an internal configuration of the image recording apparatus in FIG. FIG. 3 is a table showing a plurality of peripheral devices used in the operation and main operation states (status) of the peripheral devices in a table format. FIG. 4 is a diagram showing, in a tabular form, images of categories and subcategories that are classified and stored for each image type in the image data type database. FIG. 5 is a diagram showing representative color average values in an intra-body cavity image, an extra-body cavity image, and a color bar image. FIG. 6 is a flowchart showing the operation of the image recording apparatus according to the first embodiment. FIG. 7A is a timing diagram showing how the status of peripheral devices and image categories change according to the progress of surgery when recording is performed during surgery, and edit points and meta information are created at the time of change. chart. FIG. 7B is a diagram showing details of the edit points and peripheral device statuses in FIG. 7A in a table format. FIG. 7C is a diagram showing an example of a recording format of recording data in which meta information is recorded together with an image. FIG. 7D is a diagram showing a specific example of the meta information in FIG. 7C. FIG. 8 is a flowchart showing details of processing for determining the image type in FIG. FIG. 9 is a diagram illustrating a 3D image determined by edge processing. FIG. 10 is a diagram showing a PIP (Picture in Picture) image determined by edge processing. FIG. 11 is a diagram showing a POP (picture out picture) image determined by edge processing. FIG. 12A is a diagram showing a menu screen before designating a scene to be extracted from recorded data. 12B is a diagram showing a screen on which a list of images of the corresponding scene is displayed when the category in the body cavity is designated in FIG. 12A. FIG. 12C is a diagram showing a screen on which a list of images of the corresponding scene is displayed when the treatment subcategory is further specified in FIG. 12B. 12D is a view showing a screen on which a list of images of the corresponding scene is displayed when the status of the peripheral device is further specified in FIG. 12C. FIG. 13 is a flowchart showing the processing contents when only the processing related to the image type is performed without performing the processing related to the status of the peripheral device in the flowchart of FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
As shown in FIG. 1, an endoscope system 1 including an image recording apparatus 10 according to a first embodiment of the present invention is an endoscope for a patient 3 placed on a surgical bed (hereinafter referred to as a bed) 2. 4, a light source device 5 that generates illumination light as a plurality of peripheral devices used together with the endoscope 4, a video processor (hereinafter referred to as a processor) 6 as a signal processing device that performs signal processing, An electric scalpel device 7 for performing treatment, an insufflation device 8 for performing pneumoperitoneum, and a monitor 9 as a display device for displaying an endoscopic image are provided.
The image recording device 10, the light source device 5, the processor 6, the electric knife device 7, the pneumoperitoneum device 8, and the monitor 9 are mounted on a movable trolley 11.
The endoscope 4 has an elongated insertion portion 12, and a signal cable 13 extending from the proximal end (rear end) of the insertion portion 12 is connected to the processor 6. In addition, the light guide cable 14 extended from the vicinity of the proximal end of the insertion portion 12 is connected to the light source device 5.

When performing surgery as shown in FIG. 1, the insertion portion 12 of the endoscope 4 is inserted into, for example, the abdominal cavity of the patient 3 as a body cavity. In order to make it easy to observe the inside of the abdominal cavity, the insufflation apparatus 8 supplies air to the trocar 16 inserted into the abdominal cavity through the air supply tube 15 and inhales the abdominal cavity.
The electric scalpel device 7 is inserted into the abdominal cavity via the cable 18 a and supplies a high-frequency current to the electric scalpel 17 that performs a treatment such as an incision. Incisions are made, and the bleeding part is coagulated. Note that the high-frequency current that has flowed toward the lesioned tissue when the tip of the electric knife 17 comes into contact with the patient 3 via a counter electrode 19 that contacts the patient 3 over a wide area and a return cable 18b connected to the counter electrode 19. Return to the female device 7.
In addition, the operator may perform a procedure of anastomosing the incised site with an anastomosis thread.
FIG. 2 shows an internal configuration of the image recording apparatus 10 in FIG. 1 and a schematic configuration of a plurality of peripheral devices.

A light guide 21 is inserted into the endoscope 4, and a proximal end of the light guide 21 is connected to the light source device 5. The light source device 5 generates illumination light and supplies it to the light guide 21.
In addition, the imaging element 22 disposed at, for example, the distal end portion of the endoscope 4 is connected to the processor 6 via the signal line 23.
In addition, the endoscope 4 is provided with a changeover switch 24 that performs an operation of switching the observation mode, and an operation signal of the changeover switch 24 is sent to the light source device 5 and the processor 6.
In a normal state in which the changeover switch is turned off, the light source device outputs white light corresponding to normal light observation (mode) as illumination light. When the changeover switch 24 is turned on, the special light observation is performed. It switches to the state which outputs special light corresponding to (mode) as illumination light. That is, the changeover switch 24 has a function of turning on / off the special light observation.
The light source device 5 includes a lamp 5a that generates white light, a power supply circuit 5b that supplies lighting power to the lamp 5a, a filter 5c that is inserted and removed from the optical path, a condenser lens 5d, a power switch 5e, A control circuit 5f for controlling the light source device 5. The control circuit 5f is connected to the image recording apparatus 10 via the communication line 25a.

In the state where the changeover switch 24 is not operated, the filter 5c is disposed at a position retracted from the optical path. When the changeover switch 24 is operated, the control circuit 5f controls the filter 5c to be disposed at a position on the optical path, and the light source device 5 emits special light, more specifically narrowband light, that has passed through the filter 5c. The light guide 21 is supplied.
The processor 6 includes a signal processing circuit 6 a that performs signal processing on the image sensor 22 and generates an image signal, a power switch 6 b, and a control circuit 6 c that controls the processor 6. The image signal generated by the signal processing circuit 6a is input to the image recording apparatus 10 (the image signal input unit 31 thereof).
The control circuit 6c is connected to the image recording apparatus 10 via the communication line 25b, and the control circuit 6c transmits information on the operating state (status) of the processor 6 to the image recording apparatus 10 via the communication line 25b. When a transmission signal is transmitted from the image recording apparatus 10 side to the control circuit 6c, the control circuit 6c performs a control operation corresponding to the transmission signal.

In a state where the changeover switch 24 is not operated, the control circuit 6c performs control so that the signal processing circuit 6a generates an image signal in a state where the image pickup device 22 takes an image under white light illumination. In addition, when the changeover switch 24 is operated, the control circuit 6c is configured such that the signal processing circuit 6a captures an image signal (more specifically, a narrow-band signal) captured by the image sensor 22 under illumination of special light. (Image signal) is generated.
In addition, the control circuit 6 c transmits to the image recording apparatus 10 information indicating whether the status of the processor 6 is ON / OFF of the changeover switch 24, in other words, whether the special light observation is ON / OFF.
Note that the control circuit 5f of the light source device 5 also transmits to the image recording device 10 information indicating which status is ON / OFF of the changeover switch 24, in other words, whether the special light observation is ON / OFF. The image recording apparatus 10 can acquire information indicating whether the special light observation status is ON / OFF from both the light source device 5 and the processor 6.

For this reason, information regarding the status of special light observation ON / OFF may be acquired from only one peripheral device. In FIG. 3, for example, the acquisition of information indicating whether the special light observation status is ON / OFF by the processor 6 is indicated as a processor (light source device). Since the special light observation ON / OFF is also a switching status between special light observation and normal light observation by the endoscope 4, it may be regarded as a special light observation ON / OFF status of the endoscope 4 (FIG. 7D). reference).
The signal processing circuit 6a has a second signal input end as shown by a dotted line in addition to the first signal input end inputted from the endoscope 4 as shown by a solid line, and the signal processing circuit 6a has , PIP (Picture in Picture) image signal, POP (Picture Out Picture), and 3D image signal.
In FIG. 2, the case of a 2D endoscope including one image sensor 22 as the endoscope 4 is shown, but in FIG. 2, two image sensors 22a, The 3D endoscope 4b provided with 22b may be used for the endoscope system 1.

As shown in FIG. 2, the electric knife device 7 includes a high-frequency generator circuit 7a that generates a high-frequency signal, an output switch 7b that outputs the generated high-frequency signal, a mode switch 7c that changes the output mode of the high-frequency signal, and an electric knife. And a control circuit 7d for controlling the device 7. The control circuit 7d is connected to the image recording apparatus 10 via the communication line 25c. The control circuit 7d turns on and off the output of the high-frequency signal by the output switch 7b in the electric scalpel device 7, and changes the waveform of the high-frequency power according to the selection of the incision mode and the coagulation mode as the output mode by the mode switch 7c. Control switching for Further, the control circuit 7 d transmits output ON / OFF status information and output mode status information to the image recording apparatus 10.
As shown in FIG. 2, the insufflation apparatus 8 includes an insufflation apparatus 8 a that performs insufflation for insufflation, a switch 8 b that performs ON / OFF of the insufflation, and a control circuit 8 c that controls the insufflation apparatus 8. Have The air supply device 8a includes a sensor 8d that detects a flow rate when air is supplied and a pressure in the abdominal cavity when air is supplied. The control circuit 8c is connected to the image recording apparatus 10 via the communication line 25d.

The control circuit 8c controls air supply ON / OFF by a switch 8b that performs ON / OFF of air supply in the insufflation apparatus 8. In addition, the control circuit 7 d transmits information on the status of air supply ON / OFF in the insufflation apparatus 8 to the image recording apparatus 10.
The monitor 9 includes a display unit 9a such as a liquid crystal display, a structure emphasis switch for changing the intensity (mode) when structure emphasis is turned on and off, and structure emphasis switch when the image is displayed (in FIG. (Structure emphasis) 9b, flip switch 9c for flip ON / OFF (simply flip in FIG. 2), PIP / POP switch 9d for simply ON / OFF PIP / POP (simply PIP in FIG. 2) 9d, monitor 9 And a control circuit 9e for controlling. The control circuit 9e is connected to the image recording apparatus 10 via the communication line 25e. When the flip is turned on, the image displayed on the monitor 9 is reversed upside down. In FIG. 2, the status by ON / OFF of the structure emphasis switch 9 b and the flip switch 9 c is described as the status of the monitor 9. However, the status may be assigned to the status of the processor 6.
The control circuit 9e is configured to turn ON / OFF the structure emphasis by the structure emphasis switch 9b in the monitor 9, the status of the structure emphasis intensity (mode) when the structure emphasis is ON, the status of the flip ON / OFF by the flip switch 9c, PIP The PIP / POP status information by the / POP switch 9d is transmitted to the image recording apparatus 10.

Although not shown in FIG. 2, when performing an operation, an ultrasonic dissection (coagulation) apparatus may be used as shown in FIG. The ultrasonic incision device replaces the high frequency generation circuit 7a in the electric scalpel device 7 with an ultrasonic drive signal generation circuit that generates an ultrasonic drive signal, replaces the high frequency signal with an ultrasonic drive signal, and converts the electric scalpel 17 into an ultrasonic wave. It has almost the same function as the structure replaced with the probe.
In this case, the control circuit of the ultrasonic incision apparatus transmits information about the ON / OFF status of the ultrasonic output and the status of the output mode to the image recording apparatus 10.
As shown in FIG. 2, the image recording apparatus 10 includes an image signal input unit (or image signal input circuit) 31 to which an image signal from the processor 6 is input, and an image signal in the image signal input to the image signal input unit 31. An image analysis unit (or image analysis circuit) 32 that performs image analysis for extracting feature quantities, and a record data generation unit (or record data) that generates recording data for recording from an image signal that has passed through the image analysis unit 32 Generation circuit) 33.

In addition, the image recording apparatus 10 reads the recording data recorded on the hard disk 34 and the hard disk 34 that records the recording data generated by the recording data generation unit 33, and outputs the image data from the image recording apparatus 10 as an image signal. Part (or image signal output circuit) 35. The image signal output unit 35 can output the image signal input from the image signal input unit 31 to the monitor 9 in addition to outputting the image signal generated from the recording data to the external monitor 9.
Further, the image recording apparatus 10 determines, for each image type corresponding to the progress of the operation, a feature value for each image type for the image of the image signal input to the image analysis unit 32 in advance. An image type database unit (or image type database device) 36 that stores a database of a plurality of image types associated with (or associated with) and an image of a feature amount extracted by the image analysis unit 32 And an image discriminating unit (or image discriminating circuit) 37 that discriminates corresponding image types in a plurality of image types stored in the database unit 36.

The image discriminating unit 37 notifies the recording data generating unit 33 when the image type discriminated in the previous time changes (timing) in the subsequent time, and the recording data generating unit 33 changes the image type. Information of editing points (or extraction points) that serve as editing (or extraction) points indicating the above, and meta information (more specifically, first meta information as information related to at least the image type at the timing of the editing points) Information) is added to the recording data of the image (signal).
That is, when the image type discriminated by the image discriminating unit 37 changes with time, the recording data generating unit 33 adds the editing point and (first) meta information to the recording data of the image. Also called integrated recording data). The recording data (integrated recording data) to which the editing point and the (first) meta information are added may be a file integrated with the image recording data, or an image file (generally a moving image file). ) And a separate file associated (or attached).
FIG. 4 shows image type information stored in advance in the image type database unit 36. The image type is classified by a category (or a large classification) that is a rough classification of the image type and a sub-category (or a medium classification) that is a more detailed classification in the category classification.

The category is classified into body cavity (image), body cavity (image), and other (image). In addition, in the body cavity (image), outside the body cavity (image), and the other (image), discrimination feature amounts corresponding to the respective image types are linked (associated). That is, the image type database unit 36 determines the feature quantity for discrimination (or the feature quantity for specification) for discriminating (specifying) the inside of the body cavity (image), the outside of the body cavity (image), and the other (image). ), Outside the body cavity (image), and other (image), respectively (linked) and stored.
In addition, similarly for the following images of a plurality of subcategories, determination feature amounts corresponding to the images of each subcategory are associated (linked). Then, the image type database unit 36 stores the discrimination feature amounts corresponding to the images of the respective subcategories in association (linked). For this reason, the image type database unit 36 includes a determination feature amount storage unit 36a that stores in advance (information about) a determination feature amount associated (linked) with images of categories and subcategories. Then, the feature amount of the image of the image signal input to the image recording apparatus 10 is extracted, and the extracted feature amount is compared with the feature amount for discrimination, so that the extracted image has an image type of any category or subcategory. It is possible to determine whether the image is an image.

When the category is in the body cavity (image), the treatment (image), composition (image), anastomosis (image), and other (image) subcategories in the case of 2D (image), and 3D (image) Are classified into subcategories of treatment (image), composition (image), anastomosis (image), and other (image). An anastomosis (image) is also called an anastomosis thread (image).
The treatment (image) is an image using a treatment tool and a probe, the composite (image) is a PIP / POP image, the anastomosis (image) is an image of an anastomosis thread performing anastomosis, and the other (image ) Is an image not corresponding to the above-mentioned subcategory.
On the other hand, when the category is in a body cavity (image), it is classified into a black image, a color bar, and other subcategories. The black image is an image when there is no signal, the color bar is a pre-operative test image, and the other (image) is an image that does not fall within the above subcategory. In addition, you may employ | adopt the classification different from the classification mentioned above. Further, as a subcategory of composition (image), a subcategory of PIP (image) or POP (image) may be used without using composition (image).
The image analysis unit 32 analyzes (calculates) a pixel histogram (simply abbreviated as a histogram) of R, G, B color component images (simply abbreviated as R, G, B) in an input image signal (image thereof). Thus, a histogram analysis circuit 32a that extracts the feature amount of the image and an edge extraction circuit 32b that extracts the feature amount of the image by extracting the edge of the image are included. Note that, for example, when the anastomosis (image) is determined, the image analysis unit 32, in addition to the two circuits of the histogram analysis circuit 32a and the edge extraction circuit 32b, as described later, Depending on whether or not the image portion obtained by extracting the distal end portion of the treatment instrument corresponds to a characteristic motion pattern of a pre-registered image (whether or not it is an anastomosis (image)), it may be determined. .

The R, G, and B histogram analysis results analyzed by the histogram analysis circuit 32 a and the edge extraction results extracted by the edge extraction circuit 32 b are sent to the image discrimination unit 37.
The image discriminating unit 37 compares the feature amount extracted by the analysis of the image analysis unit 32 with the discrimination feature amount accumulated in the image type database unit 36, etc. It is determined whether an image corresponds to which image type among a plurality of image types or belongs to which image type.
Details of the image discrimination processing by the image discrimination unit 37 will be described later with reference to FIG. 8, but the outline of the image discrimination will be described with reference to FIG. FIG. 5 shows typical image examples inside and outside the body cavity, and also shows the average values of R, G, and B in one frame image. 5 (A) and 5 (B) show examples of images different from each other in the body cavity. Regarding the average values of R, G, and B in the respective images, the R average value is There is a feature that the deviation amount between the average values of R, G, and B is large so as to be considerably large.

On the other hand, FIG. 5C, FIG. 5D, and FIG. 5E show typical image examples outside the body cavity, and R, G, B regarding the average values of R, G, B in the respective images. The deviation between the average values has a small feature. For this reason, the image type database unit 36 stores, for example, a deviation amount between the average values of R, G, and B as a threshold value, for example, 20 as a threshold value as a discrimination feature value for discriminating between inside and outside the body cavity. 37 discriminate | determines the image which becomes more than a threshold value from the image in a body cavity, and discriminate | determines the image which becomes less than a threshold value from the image outside a body cavity. Note that whether or not the average value of R is greater than or equal to a threshold value with respect to the average values of G and B may be used as a distinguishing feature amount for discriminating between the body cavity and the outside of the body cavity.
FIG. 5E shows an image of the color bar (outside the body cavity) and displays color components other than RGB in a predetermined pattern. In this case, compared to the case of an image outside the body cavity, R The difference amount between the average values of G, B, and B is, for example, 5 or less as the second threshold value, and the average value of R, G, B does not vary in different frames, and the like is used as a distinguishing feature amount. The image discriminating unit 37 can discriminate whether or not it is a color bar image by discriminating whether or not the extracted feature amount corresponds to such a discriminating feature amount. In addition to the contents described in FIG. 5, the determination as described in FIG. 8 may be performed.

In the present embodiment, the image determination unit 37 inputs status information of a plurality of peripheral devices as incidental information when performing image determination by a status determination unit 38a described later. When the accuracy of determining the image type using the feature amount is low, the image determination unit 37 further refers to the status information of a plurality of peripheral devices to determine the image type using only the feature amount. Improve the accuracy of discrimination.
In the present embodiment, when the discriminated image type changes with time, the recording data generation unit 33 adds the editing point and the first meta information including at least image type information to the recording data of the image. The added (integrated) recording data is generated, and the hard disk 34 of the recording unit 34a that records the (integrated) recording data in which the first meta information including the edit point and at least the image type information is added to the image recording data. It has a function. Note that if the determined image type has not changed over time, the recording data generation unit 33 uses an image and various meta information (the various meta information includes the first meta information) in a predetermined format. However, it may not be included) to generate recording data, and the hard disk 34 records the recording data in the recording unit 34a.

When starting image recording, the recording data generation unit 33 also includes the first editing point P1 and the meta information (the first information related to the image type and status) even when the image type or the status described later does not change. (Integrated) recording data to which the first meta information and the second meta information) are added is generated, and the hard disk 34 records the (integrated) recording data (see FIG. 7A described later). Note that the first editing point P1 may be omitted, and only meta information may be added.
The image recording apparatus 10 also includes a communication unit (or communication circuit) 38 that communicates with a plurality of peripheral devices. The communication unit 38 includes a status determination unit (status determination circuit) 38a that acquires information on the status (operation state) of each peripheral device in a plurality of peripheral devices by communication to determine the status. 2 shows a configuration example in which the status determination unit 38a is provided inside the communication unit 38, the status determination unit 38a may be provided outside the communication unit 38.
In addition, the image recording apparatus 10 controls the image recording apparatus 10 and also controls a plurality of peripheral devices and acquires information from the plurality of peripheral devices via the communication unit 38 (or a control circuit). ) 39, a touch panel 40 that is operated by a touch operation for performing operations on the image recording apparatus 10, and a storage destination switching unit (or storage destination switching) that switches a storage destination when recording data recorded in the hard disk 34 is stored externally. Circuit) 41.

The touch panel 40 has a function of an operation / display unit 40a with an operation by a touch operation and a display unit to display. Further, the touch panel 40 performs designation of an image type and designation of arbitrary statuses in a plurality of peripheral devices in order to extract or search an image portion of a desired scene with respect to the hard disk 34 having the recording unit 34a. It has the function of the designation | designated part (designated input device) 40b to perform.
In addition, the control unit 39 searches for (or extracts) meta information of conditions that match the image type specified by the specifying unit 40b and the specified operation state from the recording data recorded in the recording unit 34a. Section (or meta information retrieval device) 39a. The meta information search unit 39a sends the searched meta information or information of the image portion corresponding to the meta information to the touch panel 40, for example, and the operation / display unit 40a of the touch panel 40 corresponds to the meta information or the meta information. Displays the information of the image part to be used. For this reason, the operation / display unit 40a forms a display unit that displays information of the image portion corresponding to the searched meta information.
As a recording device external to the image recording device 10, a USB memory 42a, an optical medium 42b, a PC / server 42c, and the like can be used.

The control unit 39 monitors the status status of each peripheral device in the plurality of peripheral devices determined by the status determination unit 38a, and transmits the changed status to the recording data generation unit 33 at the timing when the status changes. The recording data generation unit 33 controls to generate (integrated) recording data to which the editing point and the meta information (second meta information) at the status change timing are added. Further, the hard disk 34 has a function of a recording unit 34 a for recording (integrated) recording data generated by the recording data generating unit 33.
The recording data generation unit 33 generates recording data for recording an image of the image signal from the image signal input from the image signal input unit 31 (through the image analysis unit 32). Meta information (first meta information) including the image type determined by the determination unit 37 is added, and an edit point is also added at the timing when the image type changes. Further, the recording data generation unit 33 adds meta information (second meta information) including status information of a plurality of peripheral devices transmitted (input) from the control unit 39 to the recording data, and the status changes. Edit points are also added at the same timing.

  The image recording apparatus 10 according to the present embodiment includes an image signal input unit 31 to which an image signal is input, an image analysis unit 32 that extracts a feature amount of an image in the image signal input to the image signal input unit 31, An image type database unit 36 for storing a database of a plurality of image types associated with feature amounts for discrimination for each image type corresponding to the progress of surgery for the image of the input image signal; and the image An image discriminating unit 37 for discriminating a corresponding image type in the image type database unit 36 for the image of the feature amount extracted by the analyzing unit 32, and the image type discriminated by the discrimination result of the image discriminating unit 37. In response to the change in the image, the edit point for specifying the image signal at the changed timing and the first meta information related to the image signal A recording data generation unit 33 that generates recording data to be recorded in addition to an image signal, and the recording that includes the image signal, the editing point, and the first meta information generated by the recording data generation unit 33 A hard disk 34 having a recording unit for recording data, and a communication unit 38 constituting an information input unit to which information on operation states of a plurality of peripheral devices as peripheral devices for an endoscope used during the operation is input. A status determination unit 38a that forms a state determination unit for determining each operation state of the plurality of peripheral devices corresponding to the information from the information input to the information input unit, and a determination result of the state determination unit Second meta information related to the plurality of peripheral devices at the timing when the edit point corresponding to the change and the determination result of the state determination unit have changed is added to the image signal Te, characterized in that it comprises a control unit 39 for controlling so as to record in the recording unit.

Next, the operation of this embodiment will be described. FIG. 6 shows processing according to a typical operation in the image recording apparatus 10 of the present embodiment when performing surgery.
For example, as shown in FIG. 2, after setting the endoscope 4 and a plurality of peripheral devices, the image recording apparatus 10 enters an operating state by turning on the power switch of the image recording apparatus 10.
The image signal from the processor 6 is input to the image signal input unit 31 of the image recording device 10 as shown in step S1 in FIG. 6, and the communication unit 38 of the image recording device 10 is input to the communication unit 38 as shown in step S2. The status information of a plurality of peripheral devices is input.
FIG. 7A shows an example in which the image type changes, an example in which the statuses of a plurality of peripheral devices change according to the progress of the operation after the operation is started, and a state in which edit points are generated. FIG. 7B shows details of statuses of a plurality of peripheral devices in the case of FIG. 7A in a table format. As shown in the specific example of FIG. 7A and the like, even when only one of the image type and the status changes, the first meta information and the second meta information may be added and recorded. In the present embodiment, when the image type changes, or when any status in a plurality of peripheral devices changes, meta information including information regarding the image type and information on each status of the plurality of peripheral devices is recorded. (See FIG. 7D).

As shown in FIG. 7A, at the time t0 of the start of surgery, as the status of a plurality of peripheral devices, the insufflation apparatus 8 is turned off, the electrosurgical unit 7 is turned off, and the output mode is an incision mode (incision mode in FIG. 7A). ), Processor 6 is set to special light observation OFF (special light OFF in FIG. 7A), and monitor 9 is set to structure enhancement OFF.
In the state where the operation is started, the endoscope 4 is not yet inserted into the abdomen of the patient 3, and the image signal input to the image recording apparatus 10 is an image signal outside the body cavity.
In this case, the image discriminating unit 37 in the image recording device 10 corresponds to the category outside the body cavity in which the feature amount extracted by the image analyzing unit 32 is compared with the discriminating feature amount in the image type database unit 36, Further, it is determined that the image is not a black image or a color bar, and is an image of another subcategory. Details of the image discrimination by the image discrimination unit 37 will be described with reference to FIG.
A staff member such as a surgeon performs a recording start (recording start) instruction operation on the image recording apparatus 10 (such as the hard disk 34) from the touch panel 40 or the like.

Upon receiving a recording start instruction operation, the control unit 39 controls to record (record) the image of the image signal input from the image signal input unit 31 on the hard disk 34 via the recording data generation unit 33 and the like. As shown in step S3, the image recording apparatus 10 starts recording.
In the present embodiment, as shown in step S4, the recording start time is set as the first editing point P1, and the control unit 39 records the image recording data in the hard disk 34 in association with the meta information (P1). Control to do. Note that the i-th meta information (Pi) including the case of the first edit point P1 represents meta information at the time of the edit point Pi. For example, in the example shown in FIG. 7C, the edit point Pi and the meta information (Pi) are recorded in mdat (indicated by B3) in association with, for example, time information from the start of recording in the image recording data.
As shown in FIG. 7A, the image of the image signal input from the processor 6 into the image recording apparatus 10 at the recording start time t1 is converted into recording data (compressed or the like) by the recording data generation unit 33, and the recording is performed. The editing point P1 at the start of recording, image type information as meta information (P1) at this editing point P1, status information of a plurality of peripheral devices, and patient information of the main surgery are added (or associated) to the data. ) And recorded.

In the present embodiment, meta information including at least image type information is referred to as first meta information, and meta information including at least a plurality of peripheral device status information is referred to as second meta information.
In the example shown in FIG. 7A, the recording data (recording file) is added or associated with information on the category outside the body cavity and information on other subcategories as image type information at the editing point P1, and the statuses of a plurality of peripheral devices are added. As information, the insufflation apparatus 8 is air supply OFF, the electrosurgical apparatus 7 is output OFF, the output mode is incision mode, the processor 6 is special light observation OFF, and the monitor 9 is structure enhancement OFF information, meta information (P1). Is added to or associated with the hard disk 34. This meta information (P1) is abbreviated as meta (P1) in FIG. 7A.
As shown in step S5 of FIG. 6, the image analysis unit 32 of the image recording device 10 extracts the image feature amount of the image signal input to the image recording device 10, and as shown in step S6, the image determination unit 37 The feature amount of the extracted image is compared with the feature amount for discrimination in the image type database unit 36. Based on the comparison result, the image discriminating unit 37 discriminates the image type of the extracted image as shown in step S7.

In the next step S8, the image determination unit 37 monitors whether the image type determined by the image determination unit 37 has changed. When the image type changes, as shown in step S9, an edit point is added to the recording data at the time when the image type changes, and the first meta information is associated. Of course, status information may be included in the first meta information. The hard disk 34 records (integrated) recording data in which editing points and first meta information are added to the recording data. Then, the process proceeds to the next step S10.
On the other hand, if the image type has not changed in step S8, the process proceeds to step S10 without performing step S9.
In step S10, the control unit 39 acquires the status information of the plurality of peripheral devices determined by the status determination unit 38a, and determines whether or not the determined status has changed. If the status changes, in the next step S11, an edit point is added to the recording data and the second meta information is associated. Of course, image type information may be included in the second meta information. The hard disk 34 records the recording data in which the editing point and the second meta information are added to the recording data. Then, the process proceeds to the next step S12.

In the example shown in FIG. 7A, after the start of recording, for example, at time t2, the pneumoperitoneum 8 changes from the air supply OFF state to the ON state. Then, the abdominal cavity is inhaled with carbon dioxide gas. At this time t2, an edit point P2 is generated, and second meta information (P2) is generated at the edit point P2. The second meta information (P2) includes status information of at least a plurality of peripheral devices.
After the process of step S11 or in the case of a determination result that the status does not change in step S10, the control unit 39 performs a process of waiting for the elapse of a fixed time (for example, several seconds) in step S12, and then performs control in the next step S13. The unit 39 determines whether or not a recording stop operation has been performed. If the recording stop operation has not been performed, the process returns to step S5. Then, a change in image type is determined in step S8, a change in status is determined in step S10, and if changed, the process of adding edit points and meta information is repeated. Hereinafter, an example of processing in steps S8 and S10 will be described with reference to FIG. 7A.

On the other hand, when the recording stop operation is performed, in step S14, the control unit 39 stops the operation of recording the recording data generated by the recording data generation unit 33 on the hard disk 34, and ends the process of FIG.
In the example shown in FIG. 7A, the endoscope 4 is inserted into the body cavity (abdominal cavity) that is inhaled from outside the body cavity, for example, at time t3 after the air is fed at time t2 and the stomach is inflated.
Then, in step S8, the image determination unit 37 determines from the feature amount extracted by the image analysis unit 32 that the image type of the extracted feature amount has changed from outside the body cavity to a category inside the body cavity. In addition, the image determination unit 37 also determines which subcategory belongs to the category in the body cavity from the extracted feature amount. When the endoscope 4 is inserted and there is no treatment tool in the field of view of the endoscope 4, the image determination unit 37 determines that the extracted feature amount image belongs to the other subcategory in FIG. 4. To do.
In this case, as shown in FIG. 7A, an edit point P3 is generated at time t3, and the first meta information (P3) including image type information (category in body cavity and other subcategories) is included in this edit point P3. ) Is generated. The hard disk 34 records (integrated) recording data in which the editing point P3 and the first meta information (P3) are added to the recording data.

Further, in FIG. 7A, the operator performs a display operation for enhancing the structure with the intensity of the mode M1 on the monitor 9 at time t4. Then, when performing the process of step S10 in FIG. 6, the status determination unit 38a determines the change in the status and sends information on the determined status to the control unit 39. The control unit 39 receives the status at time t4. The editing point P4 is generated based on the changed determination, and the second meta information (P4) is added.
Further, in FIG. 7A, when the operator places the electric scalpel 17 within the field of view of the endoscope 4 with the electric scalpel 17 in order to perform, for example, a treatment for incising the lesioned part, for example, at time t5, the image discriminating unit 37 It is determined that the feature amount of the image including 17 as a treatment tool has changed from the other subcategory to the treatment (image) subcategory. Then, the image discriminating unit 37 controls to generate the edit point P5 at time t5 and add the first meta information (P5).
When the surgeon performs an output ON operation at time t6 in order to incise the lesion with the electric scalpel 17, the status determination unit 38a determines the output ON status from the electric scalpel device 7 to the electric scalpel 17. The control unit 39 determines the status change from the output OFF of the electrosurgical knife device 7 to ON and controls to add the edit point P6 and the second meta information (P6). The surgeon performs an operation to turn off the output from the electric knife device 7 to the electric knife 17 at time t7, for example, after performing a treatment for incising the lesioned part.

Then, the control unit 39 determines that the status of the electric knife device 7 from the determination result of the status determination unit 38a has changed to the output OFF, generates the edit point P7 at time t7, and generates the second meta information (P7 ) Is added.
The surgeon also determines whether coagulation is necessary. If it is determined that coagulation treatment is necessary, for example, the output mode of the electric knife device 7 is switched to the coagulation mode at time t8.
Then, the control unit 39 determines from the determination result by the status determination unit 38a that the output mode of the electric knife device 7 has changed to the coagulation mode status, generates an edit point P8 at time t8, and generates the second meta information. Control to add (P8).
Further, the operator performs an operation of switching to a special light mode in which, for example, the traveling state of the blood vessels near the surface layer of the mucous membrane is more easily recognized and the coagulation state is easily confirmed, for example, at time t9.
Then, the control unit 39 determines that the image processing mode of the processor 6 has changed to the special light observation mode status from the determination result by the status determination unit 38a, generates the edit point P9 at time t9, Control is performed to add the second meta information (P9).
When the surgeon performs an operation to turn ON the output from the electric knife device 7 to the electric knife 17 in the special light observation mode, for example, at time t10, the control unit 39 determines the electric knife device 7 from the determination result by the status determination unit 38a. It is determined that the output ON status has changed, and at time t10, an edit point P10 is generated and control is performed to add the second meta information (P10). After performing the treatment for coagulating the bleeding site, the surgeon performs an operation to turn off the output from the electric knife device 7 to the electric knife 17 at time t11, for example.

Then, the control unit 39 determines that the status of the electric knife device 7 to the output OFF has changed from the determination result by the status determination unit 38a, generates the edit point P11 at time t11, and generates the second meta information (P11). Control to add.
Further, after confirming the site where the coagulation treatment has been performed in the special light observation mode, the surgeon performs an operation of switching the special light observation mode to OFF, that is, switching to the normal light observation mode, for example, at time t12. In FIG. 7A, structure enhancement is also turned off.
Then, the control unit 39 determines from the determination result by the status determination unit 38a that the status of the special light observation mode is OFF and the structure enhancement is OFF, generates the edit point P12 at time t12, and the second The meta information (P12) is controlled to be added.
Further, the surgeon performs an operation for turning off the air supply by the pneumoperitoneum device 8 at, for example, time t13. Then, the control unit 39 determines that the status of air supply OFF has changed from the determination result by the status determination unit 38a, generates the edit point P13 at time t13, and adds the second meta information (P13). To control.

Thereafter, the operator performs an operation of pulling out the endoscope 4 from the body cavity to the outside of the body cavity, for example, at time t14. Then, the image discriminating unit 37 discriminates that it is a category outside the body cavity based on the extracted feature quantity and falls into other subcategories outside the body cavity.
In this case, an edit point P14 is generated at time t14, and first meta information (P14) including image type information (category in the body cavity and other categories) is generated at the edit point P14. Thereafter, for example, when a recording stop operation is performed at time t15, the recording is stopped. Even at the recording stop time t15, the editing point P15 and the meta information (P15) (first meta information and second meta information) (P15) may be added and recorded. Anyway. FIG. 7A shows an example in which edit point P15 and meta information (P15) are not added. For example, at the time t16 after the recording stop time t15, the operation ends.
FIG. 7B shows details of status changes in FIG. 7A in tabular form. FIG. 7B shows the statuses whose status has changed in a plurality of peripheral devices by hatching.

FIG. 7C shows a format example of (integrated) recording data recorded on the hard disk 34.
In the example shown in FIG. 7C, the (integrated) recording data adopts, for example, an MP4 file format suitable for handling moving images, audio data, etc., and this MP4 file format is data called three boxes as shown. Consists of blocks. “Ftyp” indicated by B1 is arranged at the head, a box indicating the file type, “moov” indicated by B2 is a box for storing various metadata, and “mdat” indicated by B3 is a box for storing video and audio data. In the present embodiment, image recording data is stored in mdat indicated by B3, and text including edit points P1, P2,..., Meta information (P1), meta information (P2),. The file is stored.
FIG. 7D shows specific examples of meta information (P1), meta information (P2), and meta information (P3). The meta information (P1) is a symbol or data P1 indicating the editing point P1 (simply shown by the same code as the editing point P1 but may be a different code) and data of surgical patient information recorded as this MP4 file. Date, Patient ID, Patient name, BOD, Age, Sex, including the title 1 item of the first operation and data of the recording start time 00:00:00 of the first editing point P1, and an image Includes category of category, category, and status information of multiple peripheral devices.

The meta information (P2) corresponds to the case where the status of the pneumoperitoneum 8 changes, and the symbol or data P2 indicating the editing point P2, the recording time of the moving image corresponding to the editing point P2, and the editing point P2. Includes image type category, category, and status information of multiple peripheral devices.
The meta information (P3) corresponds to the case where the category changes, and the symbol or data P3 indicating the edit point P3 in the meta information (P3), the recording time of the moving image corresponding to the edit point P3, and the edit point P3. Includes image type category, category, and status information of multiple peripheral devices.
The meta information (P4) or the like is data in the same format. In the example shown in FIG. 7D, the status (operating state) of the plurality of peripheral devices and the meta information regarding the image type are displayed together with the editing point regardless of whether the statuses of the plurality of peripheral devices change or the image type changes. Is generated.

In this way, files such as MP4 including moving image files constituting recording data are recorded on the hard disk 34, and the files include information on image types classified by image feature amounts, and progress of surgery. The information on the statuses of a plurality of peripheral devices that change in accordance with is recorded in association with at least an editing point at a time when the image type changes and an editing point at a time when the status changes. Actually, on the hard disk 34, information on the image type classified by the feature amount of the image and information on the status of a plurality of peripheral devices are added to each editing point and recorded.
Accordingly, by inputting the information of the scene to be extracted or edited from the recording data recorded on the hard disk 34, for example, from the patch panel 40, the moving image portion corresponding to the scene to be extracted or edited can be extracted easily and in a short time. be able to.
The operation when the scene to be extracted or edited is simply extracted will be described later with reference to FIGS. 12A to 12D.
In the processing shown in FIG. 6, as a processing procedure, it is determined whether or not the image type has changed in step S8, and then it is determined whether or not the status has changed in step S10. After determining whether or not the status has changed, it may be determined whether or not the image type has changed.

FIG. 8 shows details of the processing in step S7 in FIG.
When the image type determination process starts as shown in FIG. 8, in the first step S <b> 21, the image determination unit 37 determines the feature amount extracted by the image analysis unit 32 from the body cavity and the outside of the body cavity. Compare with quantity. Specifically, as described with reference to FIG. 5, the image discriminating unit 37 has a threshold (20) as a deviation amount between the average values of the R, G, and B images extracted by the histogram analysis circuit 32a of the image analyzing unit 32. It is determined whether or not the above is true. Alternatively, the image determination unit 37 determines whether or not the amount of deviation between the average values of the R, G, and B images is equal to or greater than a value that is 1/10 of the average value of R as a threshold value. Alternatively, the image discriminating unit 37 determines whether or not the average value of the R image is larger than the threshold value of 1/10 of the average value of R as compared with the average value of the G and B images.
When the determination result is satisfied, as shown in step S22, the image determination unit 37 determines that the feature amount image extracted by the image analysis unit 32 is an image of a category in the body cavity.

On the other hand, when the determination result is not equal to or greater than the threshold value but less than the threshold value, the image determination unit 37 determines that the feature amount image extracted by the image analysis unit 32 is in a category outside the body cavity, as shown in step S23. It is determined that it is an image.
If it is determined that the image is a category outside the body cavity, in the next step S24, the image determination unit 37 determines whether the image signal of the feature amount image extracted by the image analysis unit 32 is at a no-signal level. Determine whether. The image determination unit 37 determines whether or not the image signal is equal to or lower than a threshold level that is slightly larger than the 0 level for determining the no-signal level. If the image signal is equal to or lower than the threshold level, in step S25, the image determination unit 37 determines that the image is a subcategory of a black image (image), and ends the process of FIG.
In the case of a determination result that is not below the threshold level, in step S26, the image determination unit 37 determines whether or not the color distribution value of the image remains constant for at least several frames or more, or the R, G, G average It is determined whether or not the value shift amount is 3/100 or less and is constant or not changed in a period of at least several frames.

If the determination result is satisfied, the image determination unit 37 determines in step S27 that the image is a subcategory of the color bar, and ends the process of FIG. If the determination result does not correspond to this, the image determination unit 37 determines in step S28 that the image is in another subcategory, and ends the process of FIG.
If it is determined in step S22 that the category is a body cavity category, in step S29, the image determination unit 37 determines that the edge extracted by the edge extraction circuit 32b of the image analysis unit 32 causes the image (the entire image) to be left or right. It is determined whether or not an edge portion that is equally divided vertically is included.
FIGS. 9A and 9B show patterns in the case of a 3D image. The 3D image includes a first image I1 that is a left-eye image and a second image I2 that is a right-eye image. In this way, the entire screen (image) is displayed in the same size so as to be divided into two equal parts on the left and right or top and bottom.

For this reason, when an edge is extracted from the image of FIG. 9, the border line portion between the first image I1 and the second image I2 that are equally divided in the horizontal direction or the vertical direction is a regular pattern that is extracted as an edge.
Therefore, in the case where the edge extracted in step S29 is a determination result including a regular pattern including an edge that equally divides the entire image left and right or up and down, in the next step S30, the image determination unit 37 Is determined to fall under the 3D (image) subcategory. On the other hand, in the case of a determination result that does not include an equally divided edge, in step S31, the image determination unit 37 determines that the image falls under the 2D (image) subcategory.
In step S32 after the process of step S30 or S31, the image determination unit 37 determines whether or not the image includes a straight edge. In the case of a one-low result including a straight edge, in the next step S33, the image determination unit 37 determines whether or not an edge of a regular constant pattern is included. In this case, it is determined whether the regular constant pattern includes an edge having a frame-like pattern extending in the horizontal direction and / or the vertical direction as shown in FIG.

(A) to (D) in FIG. 10 show images in the case of PIP that constitutes a composite image, and a first image I1 that is a parent screen and a second image that is a child screen smaller than the first image I1. The second image I2 includes, for example, about 1/2 of the length in the horizontal direction and the vertical direction in the first image I1 at the lower right, upper right, upper left, and lower left corners. Is not limited to a value).
When an edge is extracted from the image of FIG. 10, a boundary line portion extending in the horizontal direction and the vertical direction of the second image I2 is extracted as an edge at one of the four corners of the image.
11A to 11C show images in the case of a POP constituting a composite image, and an image I for one screen (shown by hatching) is a first image I1 and a second image. The first image I1 and the second image I2 are displayed in the same size (A) or different sizes (B) or (C) adjacent to each other in the horizontal direction (left-right direction). .

When an edge is extracted from the image of FIG. 11, the boundary line portion between the first image I1 and the second image I2 adjacent in the left-right direction is extracted as an edge.
FIG. 10 and FIG. 11 clearly differ in the boundary pattern. 10 and 11 is stored in the discrimination feature quantity storage unit 36a as a discrimination feature quantity (edge pattern), and the image discrimination unit 37 discriminates the extracted edge. PIP or POP or the like is discriminated in comparison with the discrimination feature quantity (the edge pattern) stored in the feature quantity storage unit 36a.
In the case of the determination result having a regular pattern in step S33, in the next step S34, the image determination unit 37 determines whether or not the regular pattern forms one of the four corners. If the regular pattern is a determination result that forms one of the four corners, in step S35, the image determination unit 37 determines that the image having the regular pattern falls under the PIP subcategory.

On the other hand, if the determination result indicates that the regular pattern does not form one of the four corners, the image determination unit 37 determines in step S36 that the image having the regular pattern falls under the POP subcategory.
If the determination result is not a regular pattern in step S33, the image determination unit 37 determines in step S37 that the image having the linear edge pattern falls under the sub-category of the treatment image including the treatment tool.
In addition, in the case where the determination result does not have a linear edge in step S32 and in the case where the determination result has it, the image determination unit 37 determines whether the image is an image including a thread-like edge as shown in step S38. Do. If it is determined that the image includes a thread-like edge, in step S39, the image determination unit 37 determines that the image falls under an anastomosis (image) subcategory.

When it is determined whether or not the image is an anastomosis (image) by the process of step S38, a characteristic pattern of movement for anastomosis of the anastomosis thread is registered in advance as a determination feature amount. Alternatively, whether or not the image is an anastomosis (image) may be determined based on whether or not the thread-like edge in the images of a plurality of frames corresponds to the registered determination feature amount. Also, a characteristic pattern at the tip of the treatment tool when performing anastomosis is registered in advance as a feature value for discrimination, and the motion of the tip of the treatment tool in a plurality of frames of images corresponds to the registered feature amount Whether or not the image is an anastomosis (image) may be determined depending on whether or not to do so.
On the other hand, in the case of the determination result of the image that does not include the thread-like edge in the process of step S38, the image determination unit 37 determines in step S40 that the image corresponds to the other (image) subcategory.

Not only the above-mentioned treatment tools and anastomosis thread discrimination methods, but also past cases using commonly known local feature extraction / recognition techniques such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Feature) A discrimination database may be constructed in advance in the image type database unit 36 using an image with a device such as a treatment instrument (including surgical forceps) and anastomosis thread as learning data.
Then, a local feature amount is extracted from the input image to be discriminated, and a device such as a treatment instrument or anastomosis thread may be discriminated by comparing it with learning data using a discrimination database. . Further, as described above, in order to improve the accuracy of discrimination, discrimination processing may be performed on images of a plurality of frames, or status information of a plurality of peripheral devices is referred to when discriminating image types. You may do it.
After the processes of steps S35, S36, S37, S39, and S40, the process of FIG.

Next, an operation when a user such as an operator extracts (or searches) an image portion of a desired scene from recorded data recorded on the hard disk 34 will be described with reference to FIGS. 12A to 12D. A menu screen before extraction is shown in FIG. 12A. This menu screen is displayed on the operation / display unit 40a of the touch panel 40, for example. In this menu screen, in the image list display area A1 for displaying a list of images before or after extraction, for example, thumbnail images of images when recording is started are displayed, and scene extraction items arranged next to the thumbnail images are displayed. In the designation area A2, a plurality of scene extraction items that specify items for scene extraction, specifically, a category representing an image type, a subcategory item, and a peripheral device status item are displayed. The display example in the image list display area A1 in FIG. 12A schematically shows that there are 14 images composed of edit points from P1 to P14. In FIG. 12A to FIG. 12D, the edit point Pi is surrounded by a square frame to schematically show that the image is the edit point Pi.
12A to 12D, only one peripheral device status item is displayed, but a plurality of peripheral device status items are displayed, and any one or more statuses in the plurality of peripheral devices are displayed. You may be allowed to specify the item.

In the initial state of the menu display of FIG. 12A, all of the plurality of scene extraction items are in an unspecified state. For example, in order to specify scene extraction for a category item, For example, the selection button arranged on the right side is operated. In response to the operation of the selection button, the control unit 39 displays all designated items that can be designated in the category as indicated by arrows in a selection display area (abbreviated as selection display in FIG. 12A) A3.
For example, when the user designates an item in the body cavity, the control unit 39 (the meta information search unit 39a) retrieves an image corresponding to the item in the body cavity from the meta information in the recording data recorded in the hard disk 34, As shown in FIG. 12B, control is performed so that thumbnail images of images at the edit point time corresponding to the search are displayed in the image list display area A1. When applied to the example of FIG. 7A, 13 thumbnail images at edit points P3 to P13 (timing or time) are displayed. In addition, when there are many numbers, you may display by overlapping a part.

Further, in the state of FIG. 12B, when the user designates a treatment as a subcategory item, the control unit 39 searches the meta information in the recording data for an image in which the category is the body cavity and the subcategory corresponds to the treatment, and FIG. As shown in FIG. 6, thumbnail images of images at the edit point corresponding to the search are displayed in the image list display area A1.
When applied to the example of FIG. 7A, thumbnail images of edit points P5 to P13 are displayed. In the state of FIG. 12C, when the electric knife device ON is designated as the device status, the control unit 39 displays an image corresponding to the device status in which the category is the body cavity, the subcategory is treatment, and the electric knife device is ON. A search is performed from the information, and as shown in FIG. 12D, thumbnail images of images at the edit point (time) corresponding to the search are displayed in the image list display area A1. When applied to the example of FIG. 7A, thumbnail images of edit points P6 and P10 are displayed.
12A to 12D, for example, when extracting a scene in which treatment is performed by the electric knife 17 in the body cavity, information on the status of output ON of the electric knife device 7 is input. By doing so, the moving image files of the edit points P6 and P10 can be easily extracted. Furthermore, by adding information on the incision mode and coagulation mode, one of the moving image files can be extracted.

FIG. 7A shows an example in which treatment is performed using the electric knife 17. However, since the subcategory of the anastomosis using the anastomosis thread and the subcategory of the composite image of PIP / POP can also be classified. Even when various operations are performed, a moving image portion corresponding to a scene to be extracted or edited can be extracted easily and in a short time.
According to this embodiment that operates as described above, an image of a desired scene can be easily extracted. More specifically, it is specified by specifying the image type corresponding to the scene that the user wants to extract (or search or edit) and the status (operation state) information of a plurality of peripheral devices in the scene. Since the image portion (starting image) that matches the conditions can be displayed, an image of a desired scene can be easily extracted.

6 shows a flowchart including processing for determining a change in image type and a status change of a plurality of peripheral devices. In this case, for example, in response to a selection instruction from the touch panel 40, FIG. Alternatively, the process shown in FIG. 13 may be performed in which step S2 in step 6 and steps S10 to S11 are omitted.
In addition, in the configuration shown in FIG. 2, for example, the image determination unit 37 is changed to a configuration including (or also serving as) the function of the control unit 39, or the control unit 39 is changed to a configuration including (or also serves as the function of the image determination unit 37). May be.
Further, a part of the constituent elements or a part of the disclosed contents may be extracted from the constituent elements within the scope disclosed in the present specification to generate a claim.

DESCRIPTION OF SYMBOLS 1 ... Endoscope system, 3 ... Patient, 4 ... Endoscope, 5 ... Light source device, 6 ... Processor, 7 ... Electric scalpel device, 8 ... Pneumo-abdominal device, 9 ... Monitor, 10 ... Image recording device, 11 ... Trolley, 12 ... insertion part, 16 ... trocar, 17 ... electric knife, 22 ... imaging element, 24 ... changeover switch, 25a-25e ... communication line, 31 ... image signal input part, 32 ... image analysis part, 33 ... recording data Generation unit 34... Hard disk 35. Image signal output unit 36... Image type database unit 37... Image discrimination unit 38 38 communication unit 39 control unit 40 touch panel 41 storage destination switching unit

Claims (9)

An image signal input unit to which an image signal is input;
An image analysis unit that extracts a feature amount of an image in the image signal input to the image signal input unit;
An image type database unit that stores a database of a plurality of image types associated with feature amounts for discrimination for each image type corresponding to the progress of surgery for the image of the input image signal;
An image discriminating unit that discriminates a corresponding image type in the image type database unit with respect to the image of the feature amount extracted by the image analysis unit;
Corresponding to the change of the image type determined by the determination result of the image determination unit, the editing point for specifying the image signal at the changed timing and the first meta information related to the image signal, A recording data generation unit for generating recording data to be recorded in addition to the image signal;
A recording unit that records the recording data including the image signal, the editing point, and the first meta information generated by the recording data generation unit;
An information input unit for inputting information on operation states of a plurality of peripheral devices used in the operation;
A state determination unit for determining each operation state of the plurality of peripheral devices corresponding to the information from the information input to the information input unit;
Adding the second meta information related to the plurality of peripheral devices at the timing when the edit result corresponding to the change in the determination result of the state determination unit and the determination result of the state determination unit has changed, to the image signal; A control unit that controls to record in the recording unit;
An image recording apparatus comprising:   The image type database unit includes a large classification unit that classifies the image type according to the progress of the operation into a large classification of whether the image type is an image inside or outside the body cavity. The image recording apparatus described.   The image type database unit includes a treatment image, a PIP image / POP image, or an anastomosis image in which an anastomosis is performed during surgery. The image recording apparatus according to claim 2, further comprising a middle classification unit that classifies the images into middle classifications. The image analysis unit includes a histogram analysis circuit that extracts a feature amount of the image from a pixel histogram in an R, G, B color component image of the input image signal, an edge extraction circuit that extracts an edge of the image, Have
The image determination unit includes the plurality of images in which the first feature amount extracted by the histogram analysis circuit and the second feature amount extracted by the edge extraction circuit are stored in the image type database unit in advance. The image recording apparatus according to claim 1, wherein a corresponding image type is determined by comparison with a determination feature amount associated with the type.   The state discriminating unit discriminates each operation state in the plurality of peripheral devices, and the control unit determines the plurality at the timing when at least one operation state in the plurality of peripheral devices changes from the discrimination result by the state discriminating unit. 2. The image recording apparatus according to claim 1, wherein control is performed such that information on all operating states of the peripheral device is recorded in the recording unit as the second meta information. Further, for the recording unit, a designation unit that performs designation of the image type and designation of an arbitrary operation state in the plurality of peripheral devices;
A meta information retrieval unit that retrieves meta information of a condition that matches the image type designated by the designation unit and the designated operation state from the recording data recorded in the recording unit;
A display unit that displays information of an image portion corresponding to the searched meta information;
The image recording apparatus according to claim 1, further comprising:   The image recording apparatus according to claim 1, wherein the state determination unit determines each operation state of at least an insufflation apparatus and an electric scalpel apparatus or an ultrasonic incision apparatus as the plurality of peripheral devices.   The image type database unit is any image in which the image type belonging to the outside of the body cavity in the large classification unit is a non-signal black image, a color bar image representing a test image, or other images not corresponding to both images. The image recording apparatus according to claim 2, further comprising a middle classification unit that is classified into the middle classification. The recording data generation unit generates, as the first meta information, third meta information including information on each operation state of the plurality of peripheral devices at a timing when the first meta information is generated, The recording unit records the third meta information,
Furthermore, the control unit relates to an image type determined by the image determination unit at a timing when the second meta information is generated as the second meta information when the determination result by the state determination unit is changed. The image recording apparatus according to claim 1, wherein control is performed so that fourth meta information including information is recorded in the recording unit.

Images