WO2019216084A1 - Examination moving image processing device, examination moving image processing method, and examination moving image processing program - Google Patents
Examination moving image processing device, examination moving image processing method, and examination moving image processing program Download PDFInfo
- Publication number
- WO2019216084A1 WO2019216084A1 PCT/JP2019/015361 JP2019015361W WO2019216084A1 WO 2019216084 A1 WO2019216084 A1 WO 2019216084A1 JP 2019015361 W JP2019015361 W JP 2019015361W WO 2019216084 A1 WO2019216084 A1 WO 2019216084A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- moving image
- unit
- inspection
- video
- frames
- Prior art date
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/045—Control thereof
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B23/00—Telescopes, e.g. binoculars; Periscopes; Instruments for viewing the inside of hollow bodies; Viewfinders; Optical aiming or sighting devices
- G02B23/24—Instruments or systems for viewing the inside of hollow bodies, e.g. fibrescopes
Definitions
- the present invention relates to an inspection moving image processing apparatus, an inspection moving image processing method, and an inspection moving image processing program, and more particularly to an inspection moving image processing apparatus, an inspection moving image processing method, and an inspection moving image processing program for processing an inspection moving image of an endoscope.
- Patent Literature 1 as a technique for acquiring a learning sample image from a moving image, a still image that is a learning sample image is extracted from the moving image at a constant time interval, and the portion where the change in the image is severe is more detailed. Techniques for extracting still images at intervals have been proposed.
- Patent Document 1 a large amount of images of scenes where the operator is not gazing for lesion detection, such as a scene in which the endoscope is moved greatly, are acquired, and high-quality learning sample images are obtained. There is a drawback that it cannot be collected efficiently.
- the present invention has been made in view of such circumstances, and an object of the present invention is to provide an inspection moving image processing apparatus, an inspection moving image processing method, and an inspection moving image processing program capable of efficiently collecting high quality learning sample images.
- An inspection video acquisition unit that acquires an inspection video of an endoscope, a unit video extraction unit that extracts a unit video having a plurality of frames from the inspection video acquired by the inspection video acquisition unit, and a unit video extraction unit
- a scene change determination unit that analyzes the extracted unit video and determines whether there is a scene change in the unit video, and at least a part of the unit video that is determined to have no scene change by the scene change determination unit
- An inspection moving image processing apparatus comprising: a recording control unit that records the frame of the first recording medium on a first storage medium.
- unit moving images are extracted in time-series order from the acquired inspection moving images of the endoscope.
- the scene change determination unit determines whether or not there is a scene change for each unit moving image. Then, at least a part of the frames included in the unit moving image determined to have no scene change is recorded on the first storage medium.
- the part where there is no change in the scene is considered to be the part that the operator is gazing to detect lesions. Will not change.) Therefore, by selecting and recording only unit moving images that do not change the scene, only the portion that the operator has focused on can be extracted from the inspection moving images. Thereby, it is possible to efficiently collect scene images (moving images or still images) that the operator gazes from the inspection moving images, and it is possible to efficiently collect high-quality learning sample images for machine learning.
- the range recognized as “no scene change” includes a range recognized as substantially unchanged. That is, the range of “almost no change” is included.
- the scene change determination unit extracts frames from a unit video in a time-series order at regular time intervals, detects whether there is a change between the extracted frames before and after, and whether there is a scene change in the unit video
- the inspection moving image processing apparatus according to (1), wherein:
- frames are extracted from a unit moving image in a time-series order at regular time intervals, and the presence / absence of a change between the extracted frames is detected. Based on the detection result, the presence or absence of a scene change is determined. Specifically, it is determined that there is no scene change when there is no change between all the extracted frames. Thereby, the presence or absence of a scene change can be determined efficiently.
- the scene change determination unit is configured to change between the previous and next frames based on a change in the shape of the graph of the total luminance value in the X direction and / or a change in the shape of the graph of the total luminance value in the Y direction.
- the graph of the total luminance value in the X direction of the image is a graph of the total luminance value in the X direction of the image.
- the horizontal axis represents each position in the X direction and the vertical axis represents the luminance at each position. Expressed as the sum of values.
- the graph of the total luminance value in the Y direction of the image is a graph of the total luminance value in the Y direction of the image.
- the horizontal axis represents each position in the Y direction and the vertical axis represents each position. Expressed as the sum of luminance values.
- the graph of the total luminance value in the X direction is configured by the sum of the luminance values of a single or a plurality of pixel columns in the X direction. That is, the graph of the total luminance value in the X direction is not necessarily for all the pixel columns in the X direction, and may be a graph in which only a specific pixel column is extracted.
- the graph of the total luminance value in the Y direction is configured by the sum of the luminance values of a single or a plurality of pixel columns in the Y direction.
- the graph of the total luminance value in the Y direction is not necessarily for all the pixel columns in the Y direction, and may be a graph in which only a specific pixel column is extracted.
- the inspection moving image under inspection is acquired in real time and processed in real time. That is, it is processed in the background in parallel with the inspection.
- unit moving images are extracted in time-series order from the acquired inspection moving image of the endoscope, and the presence or absence of a scene change is determined for each extracted unit moving image. Then, at least a part of the frames included in the unit moving image determined to have no scene change is recorded on the first storage medium.
- An inspection moving image processing program for causing a computer to realize a function of determining presence / absence and a function of recording at least a part of frames included in a unit moving image determined to have no scene change in a first storage medium.
- unit moving images are extracted in time-series order from the acquired inspection moving image of the endoscope, and the presence or absence of a scene change is determined for each extracted unit moving image. Then, at least a part of the frames included in the unit moving image determined to have no scene change is recorded on the first storage medium.
- inspection moving image processing apparatus The figure which shows schematic structure of an endoscope apparatus.
- Block diagram of functions realized by inspection video processing device Conceptual diagram of processing by unit video extractor
- Conceptual diagram of processing by scene change determination unit Conceptual diagram of detection of change between frames based on image profile
- Flowchart showing the procedure of inspection movie processing in the inspection movie processing apparatus
- Block diagram of functions realized by the inspection moving image processing apparatus when it has a function of extracting a frame with the highest sharpness from a unit moving image Block diagram of functions realized by the inspection moving image processing apparatus when it has a function of transferring unit moving images extracted as gaze scene data
- FIG. 1 is a diagram illustrating a system configuration of an endoscope system including an examination moving image processing apparatus.
- the endoscope system 1 includes an endoscope apparatus 100, an image recording apparatus 200 that records inspection images (moving images and still images) of the endoscope, and an inspection moving image of the endoscope. And an inspection moving image processing apparatus 300 for processing.
- Each device is connected to be communicable with each other via the network 10.
- FIG. 2 is a diagram illustrating a schematic configuration of the endoscope apparatus.
- the endoscope apparatus 100 includes an electronic endoscope 110, a light source device 120, a processor device 130, a display device 140, a console 150, and the like.
- the electronic endoscope 110 includes an insertion unit 112, an operation unit 114, and a connection unit 116, and is connected to the light source device 120 and the processor device 130 via the connection unit 116.
- the insertion part 112 is a part inserted into the body cavity.
- the distal end of the insertion portion 112 is bent vertically and horizontally by the operation of the operation portion 114.
- the distal end portion of the insertion portion 112 is provided with an observation window, an illumination window, a forceps opening, a nozzle, and the like.
- An image is photographed from the observation window via a photographing optical system and an image sensor incorporated in the distal end portion of the insertion portion 112. Further, the illumination light is emitted from the illumination window via an illumination optical system incorporated in the distal end portion of the insertion portion 112.
- the operation unit 114 is a part that performs various operations.
- the operation unit 114 includes an angle knob for controlling the bending of the distal end of the insertion unit 112, a photographing button for instructing photographing, an air / water feeding button for instructing air feeding or water feeding from the nozzle, and a suction for instructing suction from the forceps opening.
- a button, a forceps port for inserting a treatment tool, and the like are provided.
- the light source device 120 supplies illumination light to the electronic endoscope 110.
- the light source device 120 generates light (so-called white light) having characteristics (wavelength distribution) close to natural light (sunlight) and supplies the light to the electronic endoscope 110.
- the light source device 120 is connected to the processor device 130.
- the light source device 120 automatically adjusts the brightness of the illumination light in conjunction with the processor device 130 (so-called automatic light control).
- the processor device 130 processes a signal output from the image sensor of the electronic endoscope 110 to generate a video signal.
- the processor device 130 is communicably connected to the image recording device 200 and the inspection moving image processing device 300 via the network 10.
- the processor device 130 transmits an inspection image (moving image and still image) to the image recording device 200 and the inspection moving image processing device 300 in accordance with an instruction from the surgeon.
- the display device 140 is a device that displays an image captured by the image sensor of the electronic endoscope 110.
- the display device 140 is constituted by a liquid crystal display, for example.
- the video signal generated by the processor device 130 is input to the display device 140.
- an image captured by the image sensor of the electronic endoscope 110 is displayed on the display device 140.
- the console 150 functions as a user interface that receives input operations such as function settings of the endoscope apparatus 100 and various instructions.
- the image recording apparatus 200 records inspection images (moving images and still images) of the endoscope.
- the image recording apparatus 200 includes a so-called file server, and includes a computer 210 in which a predetermined file management program is installed, and an external storage device 220 connected to the computer 210.
- the computer 210 is connected to the network 10 and is communicably connected to the endoscope apparatus 100 and the inspection moving image processing apparatus 300 via the network 10.
- An inspection image of the endoscope is acquired from the endoscope apparatus 100 via the network 10.
- the acquired inspection image is recorded in the external storage device 220.
- the external storage device 220 is composed of, for example, a hard disk drive (HDD).
- the inspection image of the endoscope recorded by the image recording device 200 includes a still image in addition to a moving image.
- a moving image is composed of a number of frames arranged in time series.
- a still image is composed of an image for one frame (one frame) of the moving image.
- Moving images and still images are taken according to instructions from the operator. That is, shooting of a moving image starts in response to a start instruction from the surgeon and ends in response to an end instruction. Shooting is performed at a predetermined frame rate. Further, still image shooting is performed according to a shooting instruction from the operator.
- the inspection moving image processing apparatus 300 performs a process of extracting and recording a portion watched by the operator from the inspection moving image of the endoscope. More specifically, a process of extracting and recording a part where the scene is not changed from the examination moving image of the endoscope as a part watched by the surgeon is performed.
- the part where there is no change in the scene is considered to be the part that the operator is gazing to detect lesions. Will not change.) Therefore, by extracting only the part where the scene does not change from the examination moving image of the endoscope, it is possible to extract only the part that the operator is gazing at. Thereby, it is possible to efficiently collect images of scenes that the operator gazes at from the examination moving image, and it is possible to efficiently collect high quality learning sample images.
- the inspection moving image processing apparatus 300 is configured by a computer in which a predetermined inspection moving image processing program is installed.
- FIG. 3 is a diagram showing a schematic configuration of a computer constituting the inspection moving image processing apparatus.
- the computer constituting the inspection moving image processing apparatus 300 includes a CPU (Central Processing Unit) 310, a ROM (Read Only Memory) 311, a RAM (Random Access Memory) 312, a hard disk drive (Hard Disk Drive, HDD). 313, an optical disk drive 314, a communication interface (interface, I / F) 315, an input / output interface (interface, I / F) 316, and the like.
- a keyboard 320, a mouse 321, and a display 322 are connected to this computer via an input / output interface 316.
- the computer is connected to the network 10 via the communication interface 315.
- the inspection moving image of the endoscope is acquired from the endoscope apparatus 100 via the network 10 in real time. That is, a moving image being examined by the electronic endoscope 110 is transmitted from the processor device 130 to the examined moving image processing device 300 in real time and acquired.
- the communication interface 315 corresponds to an inspection moving image acquisition unit.
- the inspection moving image acquired via the communication interface 315 is stored in a predetermined storage area (temporary storage area) of the RAM 312 or the hard disk drive 313, and is used for subsequent processing.
- FIG. 4 is a block diagram of functions realized by the inspection moving image processing apparatus.
- the inspection video processing device 300 extracts a unit video extracted by the unit video extraction unit 300A and the unit video extraction unit 300A for extracting unit videos from the acquired endoscope inspection video in chronological order.
- a scene change determination unit 300B that analyzes and determines whether there is a scene change in the unit moving image, and a recording control that records in the hard disk drive 313 a unit moving image that is determined to have no scene change by the scene change determination unit 300B It functions as the unit 300C.
- the unit moving image extracting unit 300A extracts unit moving images from the acquired endoscope inspection moving images in chronological order.
- the unit moving image is composed of a moving image including a specified number of frames.
- FIG. 5 is a conceptual diagram of processing by the unit moving image extraction unit.
- the inspection moving image M includes a large number of frames f arranged in time series.
- the unit moving image extraction unit 300A extracts a moving image including a specified number of frames f in order from the top, and extracts a unit moving image UM.
- a moving image including a certain number of frames f can be extracted in time series by sequentially extracting moving images at certain time intervals. For example, when the frame rate is 30 fps (frames per second), a moving image of 30 frames can be extracted by extracting a moving image every second. Accordingly, if the shooting time of a moving image including a specified number of frames f is a unit time T, moving images are sequentially extracted at intervals of the unit time T, so that a specified number of frames f are included in time series from the inspection moving image M. Unit video UM can be extracted.
- the scene change determination unit 300B analyzes the unit moving image extracted by the unit moving image extraction unit 300A, and determines whether or not there is a scene change in the unit moving image.
- frames are extracted from a unit video in time series order at a fixed time interval, and the presence / absence of a change in the unit video is determined by detecting the presence / absence of changes between the extracted frames. . More specifically, the presence / absence of image shift between the previous and next frames is detected to determine the presence / absence of a scene change. That is, it is determined that there is no scene change when there is little image shift between the previous and next frames.
- FIG. 6 is a conceptual diagram of processing by the scene change determination unit.
- the scene change determination unit 300B extracts frames f from the unit moving image UM in a time-series order at regular time intervals. Then, the presence or absence of a change between the extracted frames before and after is detected. If a change is detected between any of the frames, it is determined that the scene has changed. On the other hand, if no change is detected between any frames, it is determined that there is no change in the scene.
- the case where the frame f is extracted for every 5 frames from the unit moving image UM composed of 30 frames is shown.
- the 5th, 10th, 15th, 20th, 25th and 30th frames are extracted.
- the tenth and fifteenth frames, the fifteenth and twenty-fifth frames, the twenty-fifth and twenty-fifth frames, and the twenty-fifth and thirty-fifth frames Detected.
- the presence or absence of a change between frames is detected based on the image profile. Specifically, based on the change in the shape of the graph of the total luminance value in the X direction of the image constituting the frame and the change in the shape of the graph of the total luminance value in the Y direction, it is determined whether there is a change between the previous and next frames. To do.
- FIG. 7 is a conceptual diagram of detection of a change between frames based on an image profile.
- (A) shows a graph of the total luminance value in the X direction of the images constituting the frame
- (B) shows a graph of the total luminance value in the Y direction of the images constituting the frame.
- the graph of the total luminance value in the X direction of the image is a graph of the total luminance value in the X direction of the image.
- the horizontal axis represents each position in the X direction and the vertical axis represents the total luminance value in each position.
- Is represented as The sum of the luminance values at each position is calculated as the sum of the luminance values of the pixels arranged in the Y direction at each position.
- the luminance value of each pixel can be acquired as a pixel value of a G (Green) image.
- a graph GX1 indicated by a solid line is a graph of the total luminance value in the X direction in the previous frame
- a graph GX2 indicated by a broken line is a graph of the total luminance value in the X direction in the subsequent frame.
- the graph of the total luminance value in the Y direction of the image is a graph of the total luminance value in the Y direction of the image.
- the horizontal axis represents each position in the Y direction of the image, and the vertical axis represents the total luminance value at each position. Is represented as The sum of the luminance values at each position is calculated as the sum of the luminance values of the pixels arranged in the X direction at each position.
- a graph GY1 indicated by a solid line is a graph of the total luminance value in the Y direction in the previous frame
- a graph GY2 indicated by a broken line is a graph of the total luminance value in the Y direction in the subsequent frame.
- the presence or absence of is detected. Specifically, the presence or absence of a change is detected by the following procedure.
- P X1 (X) is the total luminance value at each position in the X direction of the image constituting the previous frame
- P X2 (X) is the total luminance value at each position in the X direction of the image constituting the previous frame
- the previous frame Let P Y1 (Y) be the total luminance value at each position in the Y direction of the image forming the image, and P Y2 (Y) be the total luminance value at each position in the Y direction of the image forming the subsequent frame.
- the calculated D is compared with a threshold value. If the calculated D exceeds the threshold, it is determined that there is a change between the previous and next frames. On the other hand, when the calculated D is equal to or smaller than the threshold, it is determined that there is no change between the preceding and succeeding frames.
- the recording control unit 300C processes the unit moving image determined by the scene change determination unit 300B based on the determination result. Specifically, the unit moving image determined as having no scene change by the scene change determination unit 300B is recorded in a predetermined storage area (gaze scene storage area) of the hard disk drive 313.
- the hard disk drive 313 is an example of a first storage medium.
- the unit moving image recorded in the hard disk drive 313 is a moving image of a portion where there is no scene change.
- the fact that there is no change in the scene means that there is no movement of the endoscope, and this is considered to be a moving image of the part that the operator is gazing at.
- the recording control unit 300C records the unit moving image determined to have no scene change in a predetermined storage area (gaze scene storage area) of the hard disk drive 313, and then stores the unit movie stored in the RAM 312 or the temporary storage area of the hard disk drive 313.
- the moving image is deleted from the RAM 312 or the temporary storage area of the hard disk drive 313.
- FIG. 8 is a flowchart showing the processing procedure of the inspection moving image in the inspection moving image processing apparatus.
- a moving image (inspection moving image) captured by the electronic endoscope 110 is transmitted from the endoscope device 100 to the inspection moving image processing device 300 via the network 10 in real time.
- the inspection moving image processing apparatus 300 acquires the inspection moving image transmitted from the endoscope apparatus 100 in real time (step S10).
- the acquired inspection moving image is recorded in the RAM 312 or the temporary storage area of the hard disk drive 313.
- the unit moving image extraction unit 300A extracts unit moving images from the acquired inspection moving image in chronological order (step S11).
- the scene change determination unit 300B determines whether or not there is a scene change (step S12).
- FIG. 9 is a flowchart illustrating a procedure of processing for determining whether or not there is a scene change in the scene change determination unit.
- the scene change determination unit 300B first extracts frames from the unit moving image in a time series order at a constant time interval (step S12A). Next, the extracted frames are compared before and after to detect the presence or absence of the change (step S12B).
- the presence / absence of a scene change is determined based on the detection result (step S12C). That is, it is determined whether or not there is a change between all the frames. If there is no change between all the frames, it is determined that there is no scene change in the unit moving image (step S12D). On the other hand, if there is a change between any of the frames, it is determined that there is a scene change in the unit moving image (step S12E).
- the scene change determination unit 300B compares the frames extracted at regular time intervals before and after to detect the presence or absence of a scene change.
- step S13 After the determination process by the scene change determination unit 300B, based on the determination result, it is determined whether or not to record the unit moving image as the gaze scene data (step S13).
- the scene change determination unit 300B determines that there is no scene change, it is determined that the unit moving image is recorded as data of the gaze scene.
- the scene change determination unit 300B determines that there is a scene change, it is determined that the unit moving image is not recorded as data of the gaze scene.
- the unit moving image is stored in a predetermined storage area (gaze scene storage area) of the hard disk drive 313 (step S14). Thereafter, the unit moving image recorded in the temporary storage area of the RAM 312 or the hard disk drive 313 is deleted (step S15).
- step S16 the presence / absence of the next unit moving image is determined. That is, it is determined whether or not there is a unit moving image extracted next. Next, if there is an extracted unit moving image, the process returns to step S12 and the above process is performed again. On the other hand, if there is no unit moving image extracted next, the process is terminated.
- unit moving images are extracted from the inspection moving image of the endoscope in chronological order, and the presence or absence of a scene change is determined for each extracted unit moving image. .
- Only the unit moving image determined to have no scene change is recorded in the hard disk drive 313.
- the part where there is no change in the scene is considered to be the part that the operator is gazing to detect lesions. Will not change.
- the acquisition form of a test moving image is not limited to this.
- the inspection moving image recorded in the image recording apparatus 200 may be acquired and processed.
- an inspection moving image stored on an optical disc such as a CD (Compact Disc) or a DVD (Digital Versatile Disc) may be read from the optical disc drive 314 and processed.
- FIG. it is good also as a structure which acquires and processes a test
- the number of frames or the length may be arbitrarily set.
- the minimum is 2 frames.
- ⁇ Modification of scene change determination unit> ⁇ Frame extraction>
- the frame is extracted from the unit video at regular time intervals and the presence or absence of a change between frames is detected.
- the presence or absence of a change is detected between all the frames constituting the unit video. It is good also as a structure.
- the unit moving image is determined to have a scene change.
- the configuration is such that the presence / absence of a change between frames is detected based on the difference between frames of the image profile (total luminance value). Is not limited to this. Any configuration that can detect the presence or absence of an image shift due to the movement of the endoscope may be used.
- the change occurs when the X-direction difference value sequence P X1 (X) -P X2 (X) or the Y-direction difference value sequence P Y1 (Y) -P Y2 (Y) includes a value outside a predetermined range. It may be determined that there is.
- the profile may not be for all pixel columns in the X or Y direction, but may be a profile obtained by extracting only specific pixel columns.
- a configuration may be adopted in which a motion vector is detected between previous and subsequent frames to detect the presence or absence of a change.
- the detected motion vector is compared with a threshold, and if it is equal to or less than the threshold, it is determined that there is no change between frames.
- a known technique can be adopted as a technique for detecting the presence or absence of a scene change in a moving image regardless of the comparison of the previous and subsequent frames.
- the scene change determination unit 300B determines that there is a scene change
- the entire unit moving image is recorded as a gaze scene data in a predetermined storage area (gaze scene storage area) of the hard disk drive 313.
- a part of the frames of the unit moving image may be recorded in a predetermined storage area of the hard disk drive 313.
- the first frame of the unit moving image may be recorded in a predetermined storage area of the hard disk drive 313.
- the final frame, the intermediate frame, and the like may be recorded in a predetermined storage area of the hard disk drive 313.
- a configuration may be adopted in which a plurality of frames are recorded instead of a single frame.
- it may be configured to record frames extracted from a unit moving image at a constant time interval.
- the frame with the highest sharpness may be extracted from the unit moving image, and the frame may be recorded as a gaze scene data in a predetermined storage area (gaze scene storage area) of the hard disk drive 313.
- FIG. 10 is a block diagram of functions realized by the inspection moving image processing apparatus when it has a function of extracting a frame with the highest sharpness from a unit moving image.
- the inspection moving image processing apparatus 300 further includes a high sharpness frame extraction unit 300D.
- the high-sharpness frame extraction unit 300D performs processing for analyzing a unit moving image and extracting a frame with the highest sharpness when the unit moving image to be processed is determined to have no scene change by the scene change determination unit 300B. I do. Sharpness is generally expressed as a measure of articulation of details in an image or transition of edges.
- the high sharpness frame extraction unit 300D measures the sharpness of each frame constituting the unit moving image, and extracts the frame having the highest sharpness.
- a well-known method is employ
- the recording control unit 300C records the frame extracted by the high sharpness frame extraction unit 300D as a gaze scene data in a predetermined storage area (gaze scene storage area) of the hard disk drive 313.
- the frame with the highest sharpness is extracted for all the frames constituting the unit moving image, but the frame may be thinned out and processed.
- a configuration may be adopted in which frames are extracted from a unit moving image at regular time intervals, and a frame with the highest sharpness is extracted from the extracted frames.
- a plurality of frames with high sharpness may be extracted and recorded.
- the top N frames with high sharpness may be recorded.
- the unit moving image is recorded as the gaze scene data in a predetermined storage area (gaze scene storage area) of the hard disk drive 313 provided in the inspection moving image processing apparatus 300.
- the recording destination of the extracted unit moving image is not limited to this. For example, it may be configured to transfer to the image recording apparatus 200 via the network 10 and record in the image recording apparatus 200.
- FIG. 11 is a block diagram of functions realized by the inspection moving image processing apparatus when it has a function of transferring a unit moving image extracted as gaze scene data.
- the inspection moving image processing apparatus 300 further includes a transfer control unit 300E.
- the transfer control unit 300E transfers the unit moving image recorded in a predetermined storage area (gaze scene storage area) of the hard disk drive 313 to the image recording apparatus 200 via the network 10.
- the image recording device 200 receives the unit moving image transferred from the inspection moving image processing device 300 and records it in the external storage device 220.
- the image recording device 200 is an example of a second storage medium.
- the transfer control unit 300E may sequentially transfer the unit moving images extracted as the data of the gaze scene, or may transfer them collectively after the inspection of the endoscope is completed. Alternatively, the unit moving images may be collectively transferred when a certain amount of unit moving images are accumulated in a predetermined storage area (gazing scene storage area) of the hard disk drive 313.
- the unit moving image extracted as the gaze scene data may be recorded in the image recording apparatus 200 in association with the inspection source moving image.
- a report is created after the examination.
- the report may be recorded in the image recording apparatus 200 in association with the report.
- Information such as diagnosis results may be added and recorded.
- an observation target is imaged using light of a specific wavelength band (narrow-band light) as illumination light.
- the visualization image which emphasized the blood vessel in the specific depth area
- an image suitable for blood vessel observation can be displayed on the display device 140.
- the light source included in the light source device 120 may be a light source that generates light in a plurality of wavelength bands as white band light or white band light, or generates light in a specific wavelength band narrower than the white wavelength band.
- a light source may be used.
- the specific wavelength band may be a visible blue band or a green band, or a visible red band.
- the specific wavelength band is the visible blue band or the green band, it includes a wavelength band of 390 nm to 450 nm, or 530 nm to 550 nm, and peaks within a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. It may have a wavelength.
- the specific wavelength band is the visible red band
- the wavelength band of 585 nm or more and 615 nm or less, or 610 nm or more and 730 nm or less is included, and the light of the specific wavelength band is 585 nm or more and 615 nm or less or 610 nm or more It may have a peak wavelength in a wavelength band of 730 nm or less.
- the light of the specific wavelength band described above includes a wavelength band having a different absorption coefficient between oxyhemoglobin and reduced hemoglobin, and has a peak wavelength in a wavelength band having a different absorption coefficient between oxyhemoglobin and reduced hemoglobin.
- the specific wavelength band includes a wavelength band of 400 ⁇ 10 nm, 440 ⁇ 10 nm, 470 ⁇ 10 nm, or 600 nm to 750 nm, and 400 ⁇ 10 nm, 440 ⁇ 10 nm, 470 ⁇ 10 nm, or 600 nm to 750 nm
- the light generated from the light source includes a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm, and may have a peak wavelength in a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.
- the light source may include a light source that emits excitation light having a peak of 390 nm or more and 470 nm or less. In this case, an image having information on the fluorescence emitted by the fluorescent substance in the subject can be acquired.
- the type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. are preferably configured according to the type of observation target, the purpose of observation, etc. In observation, it is preferable to combine and / or switch the wavelength of the illumination light according to the type of observation object, the purpose of observation, and the like.
- switching the wavelength for example, by rotating a disk-shaped filter (rotary color filter) provided in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of light to be irradiated is changed. You may switch.
- the functions realized by the computer can be realized by various processors.
- Various processors include a general-purpose processor that functions as a processing unit that executes programs and performs various processes, and a PLD that is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array). (Programmable Logic Device), a dedicated electric circuit which is a processor having a circuit configuration designed exclusively for executing specific processing such as ASIC (Application Specific Integrated Circuit).
- One function may be realized by two or more processors of the same type or different types.
- a configuration realized by a plurality of FPGAs or a configuration realized by a combination of a CPU and an FPGA may be used.
- a plurality of functions may be configured by a single processor.
- a single processor As an example of a configuration that realizes a plurality of functions by a single processor, first, as represented by a computer such as a client and a server, a single processor is configured by a combination of one or more CPUs and software. There is a form in which the processor realizes a plurality of functions. Secondly, as represented by a system on chip (SoC), there is a form in which a processor that realizes a plurality of functions with one IC chip (IC: Integrated Circuit) is used. In this way, various functions are realized by using one or more of the various processors as a hardware structure.
- SoC system on chip
- the hardware structure of these various processors is more specifically an electric circuit in which circuit elements such as semiconductor elements are combined.
- Endoscope system 10 network 100 Endoscope device 110 Electronic endoscope 112 Insertion part 114 Operation unit 116 connections 120 light source device 130 processor unit 140 Display device 150 console 200 Image recording device 210 computer 220 External storage device 300 Inspection video processing device 300A unit video extractor 300B Scene change determination unit 300C recording controller 300D high sharpness frame extraction unit 300E Transfer control unit 312 RAM313 Hard disk drive 314 Optical disk drive 315 Communication interface 316 I / O interface 320 keyboard 321 mouse 322 display GX1 Graph of the total luminance value in the X direction in the previous frame GX2 Graph of the total luminance value in the X direction in the subsequent frame GY1 Graph of the total luminance value in the Y direction in the previous frame GY2 Graph of the total luminance value in the Y direction in the subsequent frame En brightness value Pn1 Number of pixels in the previous frame at the brightness value En Pn2 Number of pixels in the subsequent frame at the brightness value En M inspection video T Unit time UM video f frame S10-S16 Inspection
Landscapes
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Optics & Photonics (AREA)
- Surgery (AREA)
- Biomedical Technology (AREA)
- Animal Behavior & Ethology (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Astronomy & Astrophysics (AREA)
- General Physics & Mathematics (AREA)
- Endoscopes (AREA)
- Image Analysis (AREA)
Abstract
Provided are an examination moving image processing device, an examination moving image processing method, and an examination moving image processing program, with which good-quality learning sample images can be collected efficiently. An endoscopic examination moving image is acquired from an endoscopic device. A unit moving image extraction unit (300A) extracts, from the acquired examination moving image and in chronological order, unit moving images each having a plurality of frames. A scene change determination unit (300B) analyzes each of the extracted unit moving images, and determines whether or not there is a scene change in the unit moving image. A recording control unit (300C) records, as watched scene data, a unit moving image determined to have no scene change in a predetermined storage area of an HDD (313).
Description
本発明は検査動画処理装置、検査動画処理方法及び検査動画処理プログラムに係り、特に内視鏡の検査動画を処理する検査動画処理装置、検査動画処理方法及び検査動画処理プログラムに関する。
The present invention relates to an inspection moving image processing apparatus, an inspection moving image processing method, and an inspection moving image processing program, and more particularly to an inspection moving image processing apparatus, an inspection moving image processing method, and an inspection moving image processing program for processing an inspection moving image of an endoscope.
近年、機械学習を活用して、内視鏡の検査動画から病変部を自動的に検出する技術が注目されている。本技術を開発するためには、適切な観察シーンの学習用サンプル画像を大量に用意する必要がある。
In recent years, a technique for automatically detecting a lesion from an inspection moving image of an endoscope by using machine learning has attracted attention. In order to develop this technology, it is necessary to prepare a large number of sample images for learning appropriate observation scenes.
特許文献1には、動画から学習用サンプル画像を取得する技術として、動画から一定の時間間隔で学習用サンプル画像とする静止画を抽出し、かつ、映像の変化が激しい部分については、より細かい間隔で静止画を抽出する技術が提案されている。
In Patent Literature 1, as a technique for acquiring a learning sample image from a moving image, a still image that is a learning sample image is extracted from the moving image at a constant time interval, and the portion where the change in the image is severe is more detailed. Techniques for extracting still images at intervals have been proposed.
しかしながら、特許文献1の手法では、内視鏡を大きく動かしているシーンなど、術者が病変検出のために注視していないシーンの画像も大量に取得されてしまい、良質な学習用サンプル画像を効率よく収集できないという欠点がある。
However, in the method of Patent Document 1, a large amount of images of scenes where the operator is not gazing for lesion detection, such as a scene in which the endoscope is moved greatly, are acquired, and high-quality learning sample images are obtained. There is a drawback that it cannot be collected efficiently.
本発明は、このような事情に鑑みてなされたもので、良質な学習用サンプル画像を効率よく収集できる検査動画処理装置、検査動画処理方法及び検査動画処理プログラムを提供することを目的とする。
The present invention has been made in view of such circumstances, and an object of the present invention is to provide an inspection moving image processing apparatus, an inspection moving image processing method, and an inspection moving image processing program capable of efficiently collecting high quality learning sample images.
上記課題を解決するための手段は、次のとおりである。
Means for solving the above problems are as follows.
(1)内視鏡の検査動画を取得する検査動画取得部と、検査動画取得部で取得された検査動画から複数のフレームを有する単位動画を抽出する単位動画抽出部と、単位動画抽出部で抽出された単位動画を解析し、単位動画内でのシーンの変化の有無を判定するシーン変化判定部と、シーン変化判定部でシーンの変化がないと判定された単位動画に含まれる少なくとも一部のフレームを第1記憶媒体に記録する記録制御部と、を備えた検査動画処理装置。
(1) An inspection video acquisition unit that acquires an inspection video of an endoscope, a unit video extraction unit that extracts a unit video having a plurality of frames from the inspection video acquired by the inspection video acquisition unit, and a unit video extraction unit A scene change determination unit that analyzes the extracted unit video and determines whether there is a scene change in the unit video, and at least a part of the unit video that is determined to have no scene change by the scene change determination unit An inspection moving image processing apparatus comprising: a recording control unit that records the frame of the first recording medium on a first storage medium.
本態様によれば、取得された内視鏡の検査動画から時系列順に単位動画が抽出される。抽出された単位動画は、シーン変化判定部において、単位動画ごとにシーンの変化の有無が判定される。そして、シーンの変化がないと判定された単位動画に含まれる少なくとも一部のフレームが第1記憶媒体に記録される。
According to this aspect, unit moving images are extracted in time-series order from the acquired inspection moving images of the endoscope. In the extracted unit moving image, the scene change determination unit determines whether or not there is a scene change for each unit moving image. Then, at least a part of the frames included in the unit moving image determined to have no scene change is recorded on the first storage medium.
一般に、内視鏡の検査動画において、シーンの変化がない部分は、術者が病変検出のために注視している部分と考えられる(注視する場合は、内視鏡の動きを止めるので、シーンは変化しないと考えられる。)。したがって、シーンの変化がない単位動画のみを選択して記録することにより、検査動画から術者が注視した部分のみを抽出できる。これにより、検査動画から術者が注視したシーンの画像(動画又は静止画)を効率よく収集でき、機械学習のための良質な学習用サンプル画像を効率よく収集できる。
In general, in the examination video of an endoscope, the part where there is no change in the scene is considered to be the part that the operator is gazing to detect lesions. Will not change.) Therefore, by selecting and recording only unit moving images that do not change the scene, only the portion that the operator has focused on can be extracted from the inspection moving images. Thereby, it is possible to efficiently collect scene images (moving images or still images) that the operator gazes from the inspection moving images, and it is possible to efficiently collect high-quality learning sample images for machine learning.
なお、「シーンの変化がない」と認められる範囲には、実質的に変化がないと認められる範囲が含まれる。すなわち、「ほぼ変化なし」の範囲が含まれる。
It should be noted that the range recognized as “no scene change” includes a range recognized as substantially unchanged. That is, the range of “almost no change” is included.
(2)シーン変化判定部は、単位動画から一定の時間間隔で時系列順にフレームを抽出し、抽出した前後のフレーム間で変化の有無を検出して、単位動画内でのシーンの変化の有無を判定する、上記(1)の検査動画処理装置。
(2) The scene change determination unit extracts frames from a unit video in a time-series order at regular time intervals, detects whether there is a change between the extracted frames before and after, and whether there is a scene change in the unit video The inspection moving image processing apparatus according to (1), wherein:
本態様によれば、単位動画から一定の時間間隔で時系列順にフレームが抽出され、抽出された前後のフレーム間で変化の有無が検出される。そして、その検出結果に基づいて、シーンの変化の有無が判定される。具体的には、抽出されたすべてのフレーム間で変化がない場合にシーンの変化がないと判定される。これにより、効率よくシーンの変化の有無を判定できる。
According to this aspect, frames are extracted from a unit moving image in a time-series order at regular time intervals, and the presence / absence of a change between the extracted frames is detected. Based on the detection result, the presence or absence of a scene change is determined. Specifically, it is determined that there is no scene change when there is no change between all the extracted frames. Thereby, the presence or absence of a scene change can be determined efficiently.
(3)シーン変化判定部は、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及び/又はY方向における合計輝度値のグラフの形状の変化に基づいて、前後のフレーム間の変化の有無を判定する、上記(2)の検査動画処理装置。
(3) The scene change determination unit is configured to change between the previous and next frames based on a change in the shape of the graph of the total luminance value in the X direction and / or a change in the shape of the graph of the total luminance value in the Y direction. The inspection moving image processing apparatus according to (2), wherein the presence / absence of a change is determined.
本態様によれば、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及び/又はY方向における合計輝度値のグラフの形状の変化に基づいて、前後のフレーム間の変化の有無が判定される。これにより、より正確にシーンの変化の有無を判定できる。なお、画像のX方向における合計輝度値のグラフとは、画像のX方向における輝度値の合計をグラフにしたものであり、横軸を画像のX方向の各位置、縦軸を各位置における輝度値の合計として表わされる。同様に、画像のY方向における合計輝度値のグラフとは、画像のY方向における輝度値の合計をグラフにしたものであり、横軸を画像のY方向の各位置、縦軸を各位置における輝度値の合計として表わされる。
According to this aspect, based on the change in the shape of the graph of the total luminance value in the X direction and / or the change in the shape of the graph of the total luminance value in the Y direction of the images constituting the frame, Presence / absence is determined. Thereby, the presence or absence of the change of a scene can be determined more correctly. The graph of the total luminance value in the X direction of the image is a graph of the total luminance value in the X direction of the image. The horizontal axis represents each position in the X direction and the vertical axis represents the luminance at each position. Expressed as the sum of values. Similarly, the graph of the total luminance value in the Y direction of the image is a graph of the total luminance value in the Y direction of the image. The horizontal axis represents each position in the Y direction and the vertical axis represents each position. Expressed as the sum of luminance values.
(4)X方向における合計輝度値のグラフは、X方向の単一又は複数の画素列の輝度値の合計である、上記(3)の検査動画処理装置。
(4) The inspection moving image processing apparatus according to (3), wherein the graph of the total luminance value in the X direction is a total of luminance values of a single or a plurality of pixel columns in the X direction.
本態様によれば、X方向における合計輝度値のグラフが、X方向の単一又は複数の画素列の輝度値の合計で構成される。すなわち、X方向における合計輝度値のグラフは、必ずしもX方向の全画素列のものではなく、特定の画素列のみを取り出したものでもよい。
According to this aspect, the graph of the total luminance value in the X direction is configured by the sum of the luminance values of a single or a plurality of pixel columns in the X direction. That is, the graph of the total luminance value in the X direction is not necessarily for all the pixel columns in the X direction, and may be a graph in which only a specific pixel column is extracted.
(5)Y方向における合計輝度値のグラフは、Y方向の単一又は複数の画素列の輝度値の合計である、上記(3)又は(4)の検査動画処理装置。
(5) The inspection moving image processing apparatus according to (3) or (4), wherein the graph of the total luminance value in the Y direction is a total of luminance values of one or a plurality of pixel columns in the Y direction.
本態様によれば、Y方向における合計輝度値のグラフが、Y方向の単一又は複数の画素列の輝度値の合計で構成される。すなわち、Y方向における合計輝度値のグラフは、必ずしもY方向の全画素列のものではなく、特定の画素列のみを取り出したものでもよい。
According to this aspect, the graph of the total luminance value in the Y direction is configured by the sum of the luminance values of a single or a plurality of pixel columns in the Y direction. In other words, the graph of the total luminance value in the Y direction is not necessarily for all the pixel columns in the Y direction, and may be a graph in which only a specific pixel column is extracted.
(6)シーン変化判定部は、X方向における合計輝度値のグラフのフレーム間の差分値に基づく、上記(3)の検査動画処理装置。
(6) The inspection moving image processing apparatus according to (3), wherein the scene change determination unit is based on a difference value between frames of the graph of the total luminance value in the X direction.
本態様によれば、X方向における合計輝度値のグラフのフレーム間の差分値に基づいて、前後のフレーム間の変化の有無が判定される。
According to this aspect, based on the difference value between frames of the graph of the total luminance value in the X direction, whether or not there is a change between the previous and subsequent frames is determined.
(7)シーン変化判定部は、Y方向における合計輝度値のグラフのフレーム間の差分値に基づく、上記(3)又は(4)の検査動画処理装置。
(7) The inspection moving image processing device according to (3) or (4), wherein the scene change determination unit is based on a difference value between frames of the graph of the total luminance value in the Y direction.
本態様によれば、Y方向における合計輝度値のグラフのフレーム間の差分値に基づいて、前後のフレーム間の変化の有無が判定される。
According to this aspect, based on the difference value between frames of the graph of the total luminance value in the Y direction, it is determined whether there is a change between the preceding and succeeding frames.
(8)シーン変化判定部でシーンの変化がないと判定された単位動画を解析し、最も鮮鋭度の高いフレームを抽出する高鮮鋭度フレーム抽出部を更に備え、記録制御部は、高鮮鋭度フレーム抽出部で抽出されたフレームを第1記憶媒体に記録する、上記(1)から(7)のいずれか一の検査動画処理装置。
(8) The scene change determination unit further includes a high sharpness frame extraction unit that analyzes a unit moving image determined to have no scene change and extracts a frame with the highest sharpness, and the recording control unit has a high sharpness level. The inspection moving image processing apparatus according to any one of (1) to (7), wherein the frame extracted by the frame extraction unit is recorded in a first storage medium.
本態様によれば、シーン変化判定部でシーンの変化がないと判定されると、当該単位動画から最も鮮鋭度の高いフレームが抽出される。そして、抽出されたフレームが第1記憶媒体に記録される。これにより、一部のフレームのみを記録する場合に良質なフレームを選択して記録できる。
According to this aspect, when the scene change determination unit determines that there is no scene change, a frame with the highest sharpness is extracted from the unit moving image. The extracted frame is recorded on the first storage medium. This makes it possible to select and record a high-quality frame when only a part of the frames is recorded.
(9)検査動画取得部は、検査中の検査動画をリアルタイムに取得する、上記(1)から(8)のいずれか一の検査動画処理装置。
(9) The inspection moving image processing apparatus according to any one of (1) to (8), wherein the inspection moving image acquisition unit acquires an inspection moving image under inspection in real time.
本態様によれば、検査中の検査動画がリアルタイムに取得され、リアルタイムに処理される。すなわち、検査と平行して、バックグラウンドで処理される。
According to this aspect, the inspection moving image under inspection is acquired in real time and processed in real time. That is, it is processed in the background in parallel with the inspection.
(10)第1記憶媒体に記録したフレームを第2記憶媒体に転送する転送制御部を更に備えた、上記(1)から(9)のいずれか一の検査動画処理装置。
(10) The inspection moving image processing apparatus according to any one of (1) to (9), further including a transfer control unit configured to transfer a frame recorded in the first storage medium to the second storage medium.
本態様によれば、必要に応じて、第1記憶媒体に記録されたフレームが第2記憶媒体に転送される。これにより、大量のデータを効率よく保存できる。
According to this aspect, the frame recorded on the first storage medium is transferred to the second storage medium as necessary. Thereby, a large amount of data can be stored efficiently.
(11)内視鏡の検査動画を取得するステップと、取得した検査動画から複数のフレームを有する単位動画を抽出するステップと、抽出した単位動画を解析し、単位動画内でのシーンの変化の有無を判定するステップと、シーンの変化がないと判定された単位動画に含まれる少なくとも一部のフレームを第1記憶媒体に記録するステップと、を含む検査動画処理方法。
(11) A step of acquiring an inspection moving image of an endoscope, a step of extracting a unit moving image having a plurality of frames from the acquired inspection moving image, analyzing the extracted unit moving image, and determining a scene change in the unit moving image An inspection moving image processing method comprising: a step of determining presence / absence; and a step of recording at least a part of frames included in a unit moving image determined to have no scene change in a first storage medium.
本態様によれば、取得された内視鏡の検査動画から時系列順に単位動画が抽出され、抽出された単位動画ごとにシーンの変化の有無が判定される。そして、シーンの変化がないと判定された単位動画に含まれる少なくとも一部のフレームが第1記憶媒体に記録される。これにより、術者が注視したシーンの画像(動画又は静止画)を効率よく収集でき、良質な学習用サンプル画像を効率よく収集できる。
According to this aspect, unit moving images are extracted in time-series order from the acquired inspection moving image of the endoscope, and the presence or absence of a scene change is determined for each extracted unit moving image. Then, at least a part of the frames included in the unit moving image determined to have no scene change is recorded on the first storage medium. As a result, it is possible to efficiently collect scene images (moving images or still images) that the operator gazes at, and to efficiently collect high-quality learning sample images.
(12)内視鏡の検査動画を取得する機能と、取得した検査動画から複数のフレームを有する単位動画を抽出する機能と、抽出した単位動画を解析し、単位動画内でのシーンの変化の有無を判定する機能と、シーンの変化がないと判定された単位動画に含まれる少なくとも一部のフレームを第1記憶媒体に記録する機能と、をコンピュータに実現させる検査動画処理プログラム。
(12) A function of acquiring an inspection video of an endoscope, a function of extracting a unit video having a plurality of frames from the acquired inspection video, and analyzing the extracted unit video to detect a scene change in the unit video An inspection moving image processing program for causing a computer to realize a function of determining presence / absence and a function of recording at least a part of frames included in a unit moving image determined to have no scene change in a first storage medium.
本態様によれば、取得された内視鏡の検査動画から時系列順に単位動画が抽出され、抽出された単位動画ごとにシーンの変化の有無が判定される。そして、シーンの変化がないと判定された単位動画に含まれる少なくとも一部のフレームが第1記憶媒体に記録される。これにより、術者が注視したシーンの画像(動画又は静止画)を効率よく収集でき、良質な学習用サンプル画像を効率よく収集できる。
According to this aspect, unit moving images are extracted in time-series order from the acquired inspection moving image of the endoscope, and the presence or absence of a scene change is determined for each extracted unit moving image. Then, at least a part of the frames included in the unit moving image determined to have no scene change is recorded on the first storage medium. As a result, it is possible to efficiently collect scene images (moving images or still images) that the operator gazes at, and to efficiently collect high-quality learning sample images.
本発明によれば、良質な学習用サンプル画像を効率よく収集できる。
According to the present invention, it is possible to efficiently collect high quality learning sample images.
以下、添付図面に従って本発明の好ましい実施形態について詳説する。
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[内視鏡システム]
図1は、検査動画処理装置を含む内視鏡システムのシステム構成を示す図である。
[Endoscope system]
FIG. 1 is a diagram illustrating a system configuration of an endoscope system including an examination moving image processing apparatus.
同図に示すように、この内視鏡システム1は、内視鏡装置100と、内視鏡の検査画像(動画及び静止画)を記録する画像記録装置200と、内視鏡の検査動画を処理する検査動画処理装置300と、を備える。各装置は、ネットワーク10を介して互いに通信可能に接続される。
As shown in the figure, the endoscope system 1 includes an
《内視鏡検査装置》
図2は、内視鏡装置の概略構成を示す図である。
《Endoscopic inspection device》
FIG. 2 is a diagram illustrating a schematic configuration of the endoscope apparatus.
図2に示すように、内視鏡装置100は、電子内視鏡110、光源装置120、プロセッサ装置130、表示装置140及びコンソール150等を備える。
As shown in FIG. 2, the
〈電子内視鏡〉
電子内視鏡110は、挿入部112、操作部114及び接続部116を備え、接続部116を介して光源装置120及びプロセッサ装置130と接続される。
<Electronic endoscope>
The
挿入部112は、体腔内に挿入される部分である。挿入部112は、操作部114の操作によって先端部が上下左右に湾曲する。挿入部112の先端部には、観察窓、照明窓、鉗子口、ノズル等が備えられる。画像は、挿入部112の先端部に組み込まれた撮影光学系及びイメージセンサを介して観察窓から撮影される。また、照明光は、挿入部112の先端部に組み込まれた照明光学系を介して照明窓から照射される。
The
操作部114は、各種操作を行う部分である。操作部114には、挿入部112の先端の湾曲を制御するアングルノブ、撮影を指示する撮影ボタン、ノズルからの送気又は送水を指示する送気送水ボタン、鉗子口からの吸引を指示する吸引ボタン、処置具を挿入する鉗子口等が備えられる。
The
〈光源装置〉
光源装置120は、照明光を電子内視鏡110に供給する。光源装置120は、自然光(太陽光)に近い特性(波長分布)の光(いわゆる白色光)を発生させて、電子内視鏡110に供給する。
<Light source device>
The
光源装置120は、プロセッサ装置130に接続される。光源装置120は、プロセッサ装置130と連動して、照明光の明るさを自動的に調整する(いわゆる自動調光)。
The
〈プロセッサ装置〉
プロセッサ装置130は、電子内視鏡110のイメージセンサから出力される信号を処理して、映像信号を生成する。
<Processor unit>
The
プロセッサ装置130は、ネットワーク10を介して画像記録装置200及び検査動画処理装置300と通信可能に接続される。プロセッサ装置130は、術者からの指示に応じて、検査画像(動画及び静止画)を画像記録装置200及び検査動画処理装置300に送信する。
The
〈表示装置〉
表示装置140は、電子内視鏡110のイメージセンサで捉えた映像を映し出す装置である。表示装置140は、たとえば、液晶ディスプレイで構成される。表示装置140には、プロセッサ装置130で生成された映像信号が入力される。これにより、電子内視鏡110のイメージセンサで捉えた映像が表示装置140に映し出される。
<Display device>
The
〈コンソール〉
コンソール150は、内視鏡装置100の機能設定、各種指示等の入力操作を受け付けるユーザインターフェースとして機能する。
<console>
The
《画像記録装置》
画像記録装置200は、内視鏡の検査画像(動画及び静止画)を記録する。画像記録装置200は、いわゆるファイルサーバで構成され、所定のファイル管理プログラムがインストールされたコンピュータ210と、そのコンピュータ210に接続された外部記憶装置220と、を備える。コンピュータ210は、ネットワーク10に接続され、ネットワーク10を介して内視鏡装置100及び検査動画処理装置300と通信可能に接続される。
<Image recording device>
The
内視鏡の検査画像は、ネットワーク10を介して内視鏡装置100から取得される。取得された検査画像は、外部記憶装置220に記録される。外部記憶装置220は、たとえば、ハードディスクドライブ(Hard Disk Drive , HDD)で構成される。
An inspection image of the endoscope is acquired from the
画像記録装置200が記録する内視鏡の検査画像は、動画の他に静止画を含む。動画は、時系列で並ぶ多数のフレームで構成される。静止画は、その動画の1コマ(1フレーム)分の画像で構成される。動画及び静止画の撮影は、術者からの指示に応じて実施される。すなわち、動画の撮影は、術者からの開始の指示に応じて開始され、終了の指示に応じて終了される。撮影は、あらかじめ定められたフレームレートで実施される。また、静止画の撮影は、術者からの撮影指示に応じて実施される。
The inspection image of the endoscope recorded by the
《検査動画処理装置》
検査動画処理装置300は、内視鏡の検査動画から術者が注視した部分を抽出して記録する処理を行う。より具体的には、内視鏡の検査動画からシーンの変化がない部分を術者が注視した部分として抽出して記録する処理を行う。一般に、内視鏡の検査動画において、シーンの変化がない部分は、術者が病変検出のために注視している部分と考えられる(注視する場合は、内視鏡の動きを止めるので、シーンは変化しないと考えられる。)。したがって、内視鏡の検査動画からシーンの変化がない部分のみを抽出することにより、術者が注視している部分のみを抽出できる。これにより、検査動画から術者が注視したシーンの画像を効率よく収集でき、良質な学習用サンプル画像を効率よく収集できる。
《Inspection video processing device》
The inspection moving
検査動画処理装置300は、所定の検査動画処理プログラムがインストールされたコンピュータで構成される。
The inspection moving
図3は、検査動画処理装置を構成するコンピュータの概略構成を示す図である。
FIG. 3 is a diagram showing a schematic configuration of a computer constituting the inspection moving image processing apparatus.
同図に示すように、検査動画処理装置300を構成するコンピュータは、CPU(Central Processing Unit)310、ROM(Read Only Memory)311、RAM(Random Access Memory)312、ハードディスクドライブ(Hard Disk Drive , HDD)313、光学ディスクドライブ314、通信インターフェース(interface , I/F)315、入出力インターフェース(interface , I/F)316等を備える。このコンピュータには、入出力インターフェース316を介して、キーボード320、マウス321、ディスプレイ322が接続される。また、このコンピュータは、通信インターフェース315を介してネットワーク10に接続される。
As shown in the figure, the computer constituting the inspection moving
内視鏡の検査動画は、ネットワーク10を介して内視鏡装置100からリアルタイムに取得される。すなわち、電子内視鏡110による検査中の動画が、プロセッサ装置130から検査動画処理装置300にリアルタイムに送信され、取得される。本実施の形態の検査動画処理装置300では、通信インターフェース315が、検査動画取得部に相当する。通信インターフェース315を介して取得された検査動画は、RAM312又はハードディスクドライブ313の所定の記憶領域(一時記憶領域)に格納され、以後の処理に供される。
The inspection moving image of the endoscope is acquired from the
図4は、検査動画処理装置が実現する機能のブロック図である。
FIG. 4 is a block diagram of functions realized by the inspection moving image processing apparatus.
同図に示すように、検査動画処理装置300は、取得された内視鏡の検査動画から単位動画を時系列順に抽出する単位動画抽出部300A、単位動画抽出部300Aで抽出された単位動画を解析し、単位動画内でのシーンの変化の有無を判定するシーン変化判定部300B、及び、シーン変化判定部300Bでシーンの変化がないと判定された単位動画をハードディスクドライブ313に記録する記録制御部300Cとして機能する。
As shown in the figure, the inspection
〈単位動画抽出部〉
単位動画抽出部300Aは、取得された内視鏡の検査動画から単位動画を時系列順に抽出する。単位動画は、規定数のフレームを含んだ動画で構成される。
<Unit video extractor>
The unit moving
図5は、単位動画抽出部による処理の概念図である。
FIG. 5 is a conceptual diagram of processing by the unit moving image extraction unit.
同図に示すように、検査動画Mは、時系列で並ぶ多数のフレームfで構成される。単位動画抽出部300Aは、先頭から順に規定数のフレームfを含む動画を取り出して、単位動画UMを抽出する。
As shown in the figure, the inspection moving image M includes a large number of frames f arranged in time series. The unit moving
なお、検査動画は、あらかじめ定められたフレームレートで撮影されるので、一定の時間間隔で順次動画を抽出してゆくことにより、一定数のフレームfを含む動画を時系列順に抽出できる。たとえば、フレームレートが30fps(frames per second;フレーム毎秒)の場合、1秒ごとに動画を抽出することにより、30フレームの動画を抽出できる。したがって、規定数のフレームfを含む動画の撮影時間を単位時間Tとすると、単位時間Tの間隔で順次動画を抽出してゆくことにより、検査動画Mから時系列順に規定数のフレームfを含んだ単位動画UMを抽出できる。
Since the inspection moving image is shot at a predetermined frame rate, a moving image including a certain number of frames f can be extracted in time series by sequentially extracting moving images at certain time intervals. For example, when the frame rate is 30 fps (frames per second), a moving image of 30 frames can be extracted by extracting a moving image every second. Accordingly, if the shooting time of a moving image including a specified number of frames f is a unit time T, moving images are sequentially extracted at intervals of the unit time T, so that a specified number of frames f are included in time series from the inspection moving image M. Unit video UM can be extracted.
〈シーン変化判定部〉
シーン変化判定部300Bは、単位動画抽出部300Aで抽出された単位動画を解析し、単位動画内でのシーンの変化の有無を判定する。本実施の形態では、単位動画から一定の時間間隔で時系列順にフレームを抽出し、抽出した前後のフレーム間で変化の有無を検出して、単位動画内でのシーンの変化の有無を判定する。より具体的には、前後のフレーム間で画像のズレの有無を検出して、シーンの変化の有無を判定する。すなわち、前後のフレーム間で画像のズレが少ない場合にシーンの変化がないと判定する。
<Scene change determination unit>
The scene
図6は、シーン変化判定部による処理の概念図である。
FIG. 6 is a conceptual diagram of processing by the scene change determination unit.
同図に示すように、シーン変化判定部300Bは、単位動画UMから一定の時間間隔で時系列順にフレームfを抽出する。そして、抽出した前後のフレーム間で変化の有無を検出する。いずれかのフレーム間で変化が検出された場合、シーンが変化したと判定する。一方、いずれのフレーム間においても変化が検出されない場合、シーンの変化がないと判定する。
As shown in the figure, the scene
図6に示す例では、30フレームで構成される単位動画UMから5フレームごとにフレームfを抽出する場合を示している。この場合、5フレーム目、10フレーム目、15フレーム目、20フレーム目、25フレーム目及び30フレーム目のフレームが抽出される。そして、5フレーム目と10フレーム目、10フレーム目と15フレーム目、15フレーム目と20フレーム目、20フレーム目と25フレーム目、25フレーム目と30フレーム目において、フレーム間の変化の有無が検出される。
In the example shown in FIG. 6, the case where the frame f is extracted for every 5 frames from the unit moving image UM composed of 30 frames is shown. In this case, the 5th, 10th, 15th, 20th, 25th and 30th frames are extracted. In the fifth and tenth frames, the tenth and fifteenth frames, the fifteenth and twenty-fifth frames, the twenty-fifth and twenty-fifth frames, and the twenty-fifth and thirty-fifth frames, Detected.
本実施の形態では、画像プロファイルに基づいて、フレーム間の変化の有無を検出する。具体的には、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及びY方向における合計輝度値のグラフの形状の変化に基づいて、前後のフレーム間の変化の有無を判定する。
In the present embodiment, the presence or absence of a change between frames is detected based on the image profile. Specifically, based on the change in the shape of the graph of the total luminance value in the X direction of the image constituting the frame and the change in the shape of the graph of the total luminance value in the Y direction, it is determined whether there is a change between the previous and next frames. To do.
図7は、画像プロファイルに基づくフレーム間の変化の検出の概念図である。
FIG. 7 is a conceptual diagram of detection of a change between frames based on an image profile.
同図において、(A)は、フレームを構成する画像のX方向における合計輝度値のグラフを示し、(B)は、フレームを構成する画像のY方向における合計輝度値のグラフを示している。
In the same figure, (A) shows a graph of the total luminance value in the X direction of the images constituting the frame, and (B) shows a graph of the total luminance value in the Y direction of the images constituting the frame.
画像のX方向における合計輝度値のグラフは、画像のX方向における輝度値の合計をグラフにしたものであり、横軸を画像のX方向の各位置、縦軸を各位置における輝度値の合計として表わされる。各位置における輝度値の合計は、各位置においてY方向に並ぶ画素の輝度値の合計として算出される。なお、各画素の輝度値は、G(Green;緑)画像の画素値として取得できる。
The graph of the total luminance value in the X direction of the image is a graph of the total luminance value in the X direction of the image. The horizontal axis represents each position in the X direction and the vertical axis represents the total luminance value in each position. Is represented as The sum of the luminance values at each position is calculated as the sum of the luminance values of the pixels arranged in the Y direction at each position. The luminance value of each pixel can be acquired as a pixel value of a G (Green) image.
図7の(A)において、実線で示すグラフGX1は、前フレームにおけるX方向の合計輝度値のグラフであり、破線で示すグラフGX2は、後フレームにおけるX方向の合計輝度値のグラフである。
In FIG. 7A, a graph GX1 indicated by a solid line is a graph of the total luminance value in the X direction in the previous frame, and a graph GX2 indicated by a broken line is a graph of the total luminance value in the X direction in the subsequent frame.
画像のY方向における合計輝度値のグラフは、画像のY方向における輝度値の合計をグラフにしたものであり、横軸を画像のY方向の各位置、縦軸を各位置における輝度値の合計として表わされる。各位置における輝度値の合計は、各位置においてX方向に並ぶ画素の輝度値の合計として算出される。
The graph of the total luminance value in the Y direction of the image is a graph of the total luminance value in the Y direction of the image. The horizontal axis represents each position in the Y direction of the image, and the vertical axis represents the total luminance value at each position. Is represented as The sum of the luminance values at each position is calculated as the sum of the luminance values of the pixels arranged in the X direction at each position.
図7の(B)において、実線で示すグラフGY1は、前フレームにおけるY方向の合計輝度値のグラフであり、破線で示すグラフGY2は、後フレームにおけるY方向の合計輝度値のグラフである。
In FIG. 7B, a graph GY1 indicated by a solid line is a graph of the total luminance value in the Y direction in the previous frame, and a graph GY2 indicated by a broken line is a graph of the total luminance value in the Y direction in the subsequent frame.
上記のように、本実施の形態では、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及びY方向における合計輝度値のグラフの形状の変化に基づいて、フレーム間の変化の有無を検出する。具体的には、次の手順で変化の有無を検出する。
As described above, in the present embodiment, the change between frames based on the change in the shape of the graph of the total luminance value in the X direction and the change in the shape of the graph of the total luminance value in the Y direction of the images constituting the frame. The presence or absence of is detected. Specifically, the presence or absence of a change is detected by the following procedure.
いま、前フレームを構成する画像のX方向の各位置における合計輝度値をPX1(X)、後フレームを構成する画像のX方向の各位置における合計輝度値をPX2(X)、前フレームを構成する画像のY方向の各位置における合計輝度値をPY1(Y)、後フレームを構成する画像のY方向の各位置における合計輝度値をPY2(Y)とする。
Now, P X1 (X) is the total luminance value at each position in the X direction of the image constituting the previous frame, P X2 (X) is the total luminance value at each position in the X direction of the image constituting the previous frame, and the previous frame. Let P Y1 (Y) be the total luminance value at each position in the Y direction of the image forming the image, and P Y2 (Y) be the total luminance value at each position in the Y direction of the image forming the subsequent frame.
まず、X方向の各位置について、前フレームの合計輝度値PX1(X)と後フレームの合計輝度値PX2(X)の差の絶対値(|PX1(X)-PX2(X)|)を求め、その総和(ΣX|PX1(X)-PX2(X)|)を算出する。同様に、Y方向の各位置について、前フレームの合計輝度値PY1(Y)と後フレームの合計輝度値PY2(Y)の差の絶対値(|PY1(Y)-PY2(Y)|)を求め、その総和(ΣY|PY1(Y)-PY2
(Y)|)を算出する。
First, for each position in the X direction, the absolute value of the difference between the total luminance value P X1 (X) of the previous frame and the total luminance value P X2 (X) of the subsequent frame (| P X1 (X) −P X2 (X) |) And the sum (Σ X | P X1 (X) −P X2 (X) |) is calculated. Similarly, for each position in the Y direction, the absolute value of the difference between the total luminance value P Y1 (Y) of the previous frame and the total luminance value P Y2 (Y) of the subsequent frame (| P Y1 (Y) −P Y2 (Y ) |) And find the sum (Σ Y | P Y1 (Y) −P Y2
(Y) |) is calculated.
次に、算出されたX方向の各位置の合計輝度値の差の絶対値の総和(|PX1(X)-PX2(X)|)とY方向の各位置の合計輝度値の差の絶対値の総和(ΣY|PY1(Y)-PY2(Y)|)の和Dを算出する。すなわち、D=ΣX|PX1(X)-PX2(X)| +ΣY|PY1(Y)-PY2(Y)|を算出する。
Next, the sum of absolute values of differences between the calculated total luminance values at each position in the X direction (| P X1 (X) −P X2 (X) |) and the difference between the total luminance values at each position in the Y direction The sum D of the sum of absolute values (Σ Y | P Y1 (Y) −P Y2 (Y) |) is calculated. That, D = Σ X | P X1 (X) -P X2 (X) | + Σ Y | P Y1 (Y) -P Y2 (Y) | is calculated.
次に、算出されたDを閾値と比較する。算出されたDが、閾値を超える場合、前後のフレーム間で変化あり、と判定する。一方、算出されたDが、閾値以下の場合、前後のフレーム間で変化なし、と判定する。
Next, the calculated D is compared with a threshold value. If the calculated D exceeds the threshold, it is determined that there is a change between the previous and next frames. On the other hand, when the calculated D is equal to or smaller than the threshold, it is determined that there is no change between the preceding and succeeding frames.
このように、本実施の形態では、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及びY方向における合計輝度値のグラフの形状の変化に基づいて、前後のフレーム間の変化の有無を判定する。
As described above, in the present embodiment, based on the change in the shape of the graph of the total luminance value in the X direction and the change in the shape of the graph of the total luminance value in the Y direction of the images constituting the frame, Determine if there is a change.
〈記録制御部〉
記録制御部300Cは、シーン変化判定部300Bで判定処理された単位動画を判定結果に基づいて処理する。具体的には、シーン変化判定部300Bでシーンの変化なしと判定された単位動画をハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録する。ハードディスクドライブ313は第1記憶媒体の一例である。
<Recording control unit>
The
ハードディスクドライブ313に記録される単位動画は、シーンの変化のない部分の動画である。シーンの変化がないということは、内視鏡の動きのないということであり、術者が注視している部分の動画であると考えられる。このような部分を選択的に抽出して記録することにより、学習用サンプル画像となり得る良質な画像を効率よく収集できる。
The unit moving image recorded in the
記録制御部300Cは、シーンの変化なしと判定された単位動画をハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録後、RAM312又はハードディスクドライブ313の一時記憶領域に記憶された当該単位動画をRAM312又はハードディスクドライブ313の一時記憶領域から消去する。シーンの変化ありと判定された単位動画についても同様であり、シーン変化判定部300Bによる判定処理後、RAM312又はハードディスクドライブ313の一時記憶領域から消去する。これにより、RAM312又はハードディスクドライブ313の記憶領域を効率よく使用できる。
The
[検査動画処理方法]
図8は、検査動画処理装置における検査動画の処理手順を示すフローチャートである。
[Inspection video processing method]
FIG. 8 is a flowchart showing the processing procedure of the inspection moving image in the inspection moving image processing apparatus.
内視鏡検査が開始されると、電子内視鏡110で撮影された動画(検査動画)が内視鏡装置100からネットワーク10を介して検査動画処理装置300にリアルタイムに送信される。検査動画処理装置300は、内視鏡装置100から送信される検査動画をリアルタイムに取得する(ステップS10)。取得された検査動画は、RAM312又はハードディスクドライブ313の一時記憶領域に記録される。
When the endoscopic inspection is started, a moving image (inspection moving image) captured by the
検査動画が取得されると、単位動画抽出部300Aにおいて、取得された検査動画から時系列順に単位動画が抽出される(ステップS11)。
When the inspection moving image is acquired, the unit moving
抽出された単位動画は、シーン変化判定部300Bにおいて、シーンの変化の有無が判定される(ステップS12)。
In the extracted unit moving image, the scene
図9は、シーン変化判定部におけるシーンの変化の有無を判定する処理の手順を示すフローチャートである。
FIG. 9 is a flowchart illustrating a procedure of processing for determining whether or not there is a scene change in the scene change determination unit.
シーン変化判定部300Bは、まず、単位動画から時系列順に一定の時間間隔でフレームを抽出する(ステップS12A)。次に、抽出した各フレームを前後で比較し、その変化の有無を検出する(ステップS12B)。
The scene
本実施の形態では、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及びY方向における合計輝度値のグラフの形状の変化に基づいて、前後のフレーム間の変化の有無が判定される。具体的には、D=ΣX|PX1(X)-PX2(X)| +ΣY|PY1(Y)-PY2(Y)| を算出し、算出されたDが閾値以下か否かに基づいて、前後のフレーム間の変化の有無を判定する。算出されたDが閾値を超える場合、前後のフレーム間で変化あり、と判定され、閾値以下の場合、前後のフレーム間で変化なし、と判定される。
In the present embodiment, based on the change in the shape of the graph of the total luminance value in the X direction of the image constituting the frame and the change in the shape of the graph of the total luminance value in the Y direction, whether or not there is a change between the previous and next frames. Determined. Specifically, D = Σ X | P X1 (X) -P X2 (X) | + Σ Y | P Y1 (Y) -P Y2 (Y) | is calculated, whether the calculated D is less than or equal to the threshold Based on whether or not, the presence or absence of a change between the preceding and succeeding frames is determined. When the calculated D exceeds the threshold, it is determined that there is a change between the previous and next frames, and when it is equal to or less than the threshold, it is determined that there is no change between the previous and subsequent frames.
すべてのフレーム間で変化の有無を検出後、その検出結果にシーンの変化の有無を判定する(ステップS12C)。すなわち、すべてのフレーム間において、変化がなかったか否かを判定する。すべてのフレーム間において、変化がない場合、当該単位動画内でシーンの変化なし、と判定する(ステップS12D)。一方、いずれかのフレーム間において、変化があった場合、当該単位動画内でシーンの変化あり、と判定する(ステップS12E)。
After detecting the presence / absence of a change between all the frames, the presence / absence of a scene change is determined based on the detection result (step S12C). That is, it is determined whether or not there is a change between all the frames. If there is no change between all the frames, it is determined that there is no scene change in the unit moving image (step S12D). On the other hand, if there is a change between any of the frames, it is determined that there is a scene change in the unit moving image (step S12E).
このように、シーン変化判定部300Bでは、一定の時間間隔で抽出したフレームを前後で比較して、シーンの変化の有無を検出する。
In this manner, the scene
シーン変化判定部300Bによる判定処理後、その判定結果に基づいて、当該単位動画を注視シーンのデータとして記録するか否か判定される(ステップS13)。シーン変化判定部300Bでシーンの変化なし、と判定されると、当該単位動画を注視シーンのデータとして記録する、と判定される。一方、シーン変化判定部300Bでシーンの変化あり、と判定されると、当該単位動画を注視シーンのデータとして記録しない、と判定される。
After the determination process by the scene
注視シーンのデータとして記録する、と判定されると、当該単位動画が、ハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記憶される(ステップS14)。その後、RAM312又はハードディスクドライブ313の一時記憶領域に記録された当該単位動画が消去される(ステップS15)。
If it is determined to record as data of the gaze scene, the unit moving image is stored in a predetermined storage area (gaze scene storage area) of the hard disk drive 313 (step S14). Thereafter, the unit moving image recorded in the temporary storage area of the
一方、注視シーンのデータとして記録しない、と判定されると、RAM312又はハードディスクドライブ313の一時記憶領域に記録された当該単位動画が消去される(ステップS15)。
On the other hand, if it is determined not to record as the data of the gaze scene, the unit moving image recorded in the
この後、次の単位動画の有無が判定される(ステップS16)。すなわち、次に抽出された単位動画の有無が判定される。次に抽出された単位動画ある場合、ステップS12に戻り、上記処理を再度実施する。一方、次に抽出された単位動画がない場合、処理を終了する。
Thereafter, the presence / absence of the next unit moving image is determined (step S16). That is, it is determined whether or not there is a unit moving image extracted next. Next, if there is an extracted unit moving image, the process returns to step S12 and the above process is performed again. On the other hand, if there is no unit moving image extracted next, the process is terminated.
このように、本実施の形態の検査動画処理装置300によれば、内視鏡の検査動画から時系列順に単位動画が抽出され、抽出された単位動画ごとにシーンの変化の有無が判定される。そして、シーンの変化がないと判定された単位動画のみがハードディスクドライブ313に記録される。一般に、内視鏡の検査動画において、シーンの変化がない部分は、術者が病変検出のために注視している部分と考えられる(注視する場合は、内視鏡の動きを止めるので、シーンは変化しないと考えられる。)。したがって、シーンの変化がない単位動画のみを選択して記録することにより、検査動画から術者が注視した部分のみを抽出できる。これにより、検査動画から術者が注視したシーンの画像を効率よく収集でき、機械学習のための良質な学習用サンプル画像を効率よく収集できる。
Thus, according to the inspection moving
[変形例]
《検査動画の取得形態の変形例》
上記実施の形態では、検査中の動画をリアルタイムに取得する構成としているが、検査動画の取得形態は、これに限定されるものではない。たとえば、画像記録装置200に記録されている検査動画を取得して、処理する構成としてもよい。あるいは、CD(Compact Disc)、DVD(Digital Versatile Disc)などの光ディスクに格納された検査動画を光学ディスクドライブ314から読み出して処理してもよい。また、検査動画処理装置300に接続された外部ストレージに格納された検査動画を取得して、処理する構成としてもよい。その他、ネットワークを介して他の機器から検査動画を取得して、処理する構成としてもよい。
[Modification]
《Variation of acquisition form of inspection video》
In the said embodiment, although it is set as the structure which acquires the moving image under test | inspection in real time, the acquisition form of a test moving image is not limited to this. For example, the inspection moving image recorded in the
《単位動画抽出部の変形例》
検査動画から抽出する単位動画については、フレーム数又は長さを任意に設定できるようにしてもよい。最小は、2フレームとなる。
<Modification of unit video extractor>
For the unit moving image extracted from the inspection moving image, the number of frames or the length may be arbitrarily set. The minimum is 2 frames.
《シーン変化判定部の変形例》
〈フレームの抽出〉
上記実施の形態では、単位動画から一定の時間間隔でフレームを抽出して、フレーム間の変化の有無を検出する構成としているが、単位動画を構成するすべてのフレーム間で変化の有無を検出する構成としてもよい。
<Modification of scene change determination unit>
<Frame extraction>
In the above embodiment, the frame is extracted from the unit video at regular time intervals and the presence or absence of a change between frames is detected. However, the presence or absence of a change is detected between all the frames constituting the unit video. It is good also as a structure.
また、上記実施の形態では、抽出したすべてのフレーム間で変化の有無を検出する構成としているが、変化が検出された段階で処理を終了してもよい。この場合、変化が検出された段階で当該単位動画は、シーンの変化ありと判定される。
Moreover, in the said embodiment, although it has set as the structure which detects the presence or absence of a change between all the extracted frames, you may complete | finish a process in the stage where a change was detected. In this case, when the change is detected, the unit moving image is determined to have a scene change.
〈フレーム間の変化の有無の検出〉
上記実施の形態では、画像のプロファイル(輝度値の合計)のフレーム間の差分に基づいて、フレーム間の変化の有無を検出する構成としているが、前後のフレーム間の変化の有無を検出する方法は、これに限定されるものではない。内視鏡の動きによる画像のズレの有無を検出できる構成であればよい。
<Detection of change between frames>
In the above embodiment, the configuration is such that the presence / absence of a change between frames is detected based on the difference between frames of the image profile (total luminance value). Is not limited to this. Any configuration that can detect the presence or absence of an image shift due to the movement of the endoscope may be used.
たとえば、X方向の差分値列PX1(X)-PX2(X)又はY方向の差分値列PY1(Y)-PY2(Y)が、所定の範囲外の値を含む場合に変化ありと判定してもよい。
For example, the change occurs when the X-direction difference value sequence P X1 (X) -P X2 (X) or the Y-direction difference value sequence P Y1 (Y) -P Y2 (Y) includes a value outside a predetermined range. It may be determined that there is.
また、プロファイルは、XあるいはY方向の全画素列のものではなく、特定の画素列のみを取り出したものでもよい。
In addition, the profile may not be for all pixel columns in the X or Y direction, but may be a profile obtained by extracting only specific pixel columns.
また、X方向及びY方向のいずれか一方の画像のプロファイルに基づいて、フレーム間の変化の有無を検出する構成としてもよい。
Moreover, it is good also as a structure which detects the presence or absence of the change between frames based on the profile of the image of any one of a X direction and a Y direction.
また、その他の方法として、前後のフレーム間で動きベクトルを検出して、変化の有無を検出する構成としてもよい。この場合、たとえば、検出された動きベクトルを閾値と比較し、閾値以下の場合にフレーム間の変化なしと判定する。
As another method, a configuration may be adopted in which a motion vector is detected between previous and subsequent frames to detect the presence or absence of a change. In this case, for example, the detected motion vector is compared with a threshold, and if it is equal to or less than the threshold, it is determined that there is no change between frames.
この他、前後のフレームの比較に関わらず、動画内のシーンの変化の有無を検出する技術に関して、公知の技術を採用できる。
In addition, a known technique can be adopted as a technique for detecting the presence or absence of a scene change in a moving image regardless of the comparison of the previous and subsequent frames.
《記録制御部の変形例》
〈記録対象〉
上記実施の形態では、シーン変化判定部300Bでシーンの変化ありと判定された場合、当該単位動画の全体を注視シーンのデータとして、ハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録する構成としているが、当該単位動画の一部のフレームをハードディスクドライブ313の所定の記憶領域に記録する構成としてもよい。たとえば、当該単位動画の先頭フレームをハードディスクドライブ313の所定の記憶領域に記録する構成としてもよい。この他、最終フレーム、中間フレーム等をハードディスクドライブ313の所定の記憶領域に記録する構成としてもよい。また、この場合、単一のフレームではなく複数枚のフレームを記録する構成としてもよい。たとえば、単位動画から一定の時間間隔で抽出したフレームを記録する構成としてもよい。
<Modification of recording control unit>
<Recording target>
In the above-described embodiment, when the scene
また、単位動画から最も鮮鋭度の高いフレームを抽出し、当該フレームを注視シーンのデータとしてハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録する構成としてもよい。
Alternatively, the frame with the highest sharpness may be extracted from the unit moving image, and the frame may be recorded as a gaze scene data in a predetermined storage area (gaze scene storage area) of the
図10は、単位動画から最も鮮鋭度の高いフレームを抽出する機能を備える場合に検査動画処理装置が実現する機能のブロック図である。
FIG. 10 is a block diagram of functions realized by the inspection moving image processing apparatus when it has a function of extracting a frame with the highest sharpness from a unit moving image.
同図に示すように、この場合、検査動画処理装置300には、高鮮鋭度フレーム抽出部300Dが更に備えられる。
As shown in the figure, in this case, the inspection moving
高鮮鋭度フレーム抽出部300Dは、処理対象の単位動画が、シーン変化判定部300Bでシーンの変化なしと判定された場合に、当該単位動画を解析し、最も鮮鋭度の高いフレームを抽出する処理を行う。鮮鋭度は、一般的に、画像内の細部の明瞭度又はエッジ部の遷移度の測定値で表される。
The high-sharpness
高鮮鋭度フレーム抽出部300Dは、単位動画を構成する各フレームの鮮鋭度を測定し、最も鮮鋭度の高いフレームを抽出する。鮮鋭度の測定には、公知の手法が採用される。
たとえば、画像のエッジ領域における高周波成分の強度の平均によって鮮鋭度を測定する方法などを採用できる。
The high sharpness
For example, a method of measuring the sharpness by the average of the intensities of the high frequency components in the edge region of the image can be adopted.
記録制御部300Cは、高鮮鋭度フレーム抽出部300Dによって抽出されたフレームを注視シーンのデータとしてハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録する。
The
このように、最も鮮鋭度の高いフレームを注視シーンのデータとしてハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録することにより、より良質な学習用サンプル画像を収集できる。
In this way, by recording the frame with the highest sharpness as the data of the gaze scene in the predetermined storage area (gaze scene storage area) of the
なお、上記の例では、単位動画を構成するすべてのフレームを対象にして最も鮮鋭度の高いフレームを抽出する構成としているが、間引いて処理してもよい。たとえば、単位動画から一定の時間間隔でフレームを抽出し、抽出したフレームの中から最も鮮鋭度の高いフレームを抽出する構成としてもよい。
In the above example, the frame with the highest sharpness is extracted for all the frames constituting the unit moving image, but the frame may be thinned out and processed. For example, a configuration may be adopted in which frames are extracted from a unit moving image at regular time intervals, and a frame with the highest sharpness is extracted from the extracted frames.
また、鮮鋭度の高い複数のフレームを抽出して記録してもよい。たとえば、鮮鋭度の高い上位N枚のフレームを記録してもよい。
A plurality of frames with high sharpness may be extracted and recorded. For example, the top N frames with high sharpness may be recorded.
〈記録先〉
上記実施の形態では、注視シーンのデータとして単位動画を検査動画処理装置300に備えられたハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録する構成としているが、注視シーンのデータとして抽出した単位動画の記録先は、これに限定されるものではない。たとえば、ネットワーク10を介して画像記録装置200に転送し、画像記録装置200に記録する構成としてもよい。
<Record destination>
In the above embodiment, the unit moving image is recorded as the gaze scene data in a predetermined storage area (gaze scene storage area) of the
図11は、注視シーンのデータとして抽出した単位動画を転送する機能を備える場合に検査動画処理装置が実現する機能のブロック図である。
FIG. 11 is a block diagram of functions realized by the inspection moving image processing apparatus when it has a function of transferring a unit moving image extracted as gaze scene data.
同図に示すように、この場合、検査動画処理装置300には、転送制御部300Eが更に備えられる。転送制御部300Eは、ハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に記録された単位動画を、ネットワーク10を介して画像記録装置200に転送する。画像記録装置200は、検査動画処理装置300から転送された単位動画を受け取り、外部記憶装置220に記録する。画像記録装置200は、第2記憶媒体の一例である。
As shown in the figure, in this case, the inspection moving
なお、転送制御部300Eは、注視シーンのデータとして抽出した単位動画を逐次転送してもよいし、また、内視鏡の検査終了後にまとめて転送してもよい。また、ハードディスクドライブ313の所定の記憶領域(注視シーン記憶領域)に一定容量の単位動画が蓄積された段階でまとめて転送してもよい。
Note that the
〈検査画像等との関連付け〉
注視シーンのデータとして抽出した単位動画は、抽出元の検査動画に関連付けて、画像記録装置200に記録してもよい。
<Association with inspection images>
The unit moving image extracted as the gaze scene data may be recorded in the
また、一般に内視鏡検査では、検査後にレポートが作成されるが、そのレポートに関連付けて画像記録装置200に記録してもよい。診断結果の情報等を付加して記録してもよい。
In general, in an endoscopic examination, a report is created after the examination. However, the report may be recorded in the
《内視鏡装置の変形例》
〈内視鏡の種類〉
上記実施の形態では、いわゆる軟性内視鏡の検査動画を処理する場合を例に説明したが、硬性内視鏡の検査動画を処理する場合にも同様に適用できる。
<< Modified example of endoscope apparatus >>
<Endoscope types>
In the above-described embodiment, the case where a so-called flexible endoscope inspection moving image is processed has been described as an example, but the present invention can be similarly applied to the case of processing a rigid endoscope inspection moving image.
〈観察の種類〉
上記実施の形態は、いわゆる通常観察の例で説明したが、NBI(Narrow Band Imaging;狭帯域光観察)、BLI(Blue LASER Imaging;狭帯域光観察)、FICE(Flexible spectral Imaging Color Enhancement ;分光推定処理)等の特殊光観察を行う場合にも同様に適用できる。特殊光観察を行う場合は、その特殊光観察に対応した光を光源装置120から供給し、特殊光観察に対応した画像処理をプロセッサ装置130で実施する。
<Type of observation>
Although the above-described embodiment has been described with an example of so-called normal observation, NBI (Narrow Band Imaging), BLI (Blue LASER Imaging), FICE (Flexible spectral Imaging Color Enhancement), spectral estimation The same applies to special light observation such as processing. When performing special light observation, light corresponding to the special light observation is supplied from the
たとえば、狭帯域光観察では、照明光に特定の波長帯域の光(狭帯域光)を使用して、観察対象を撮影する。そして、撮影により得られた信号から観察対象の特定深さ領域にある血管を強調した可視化画像を生成する。これにより、血管の観察に適した画像を表示装置140に表示できる。
For example, in narrow-band light observation, an observation target is imaged using light of a specific wavelength band (narrow-band light) as illumination light. And the visualization image which emphasized the blood vessel in the specific depth area | region of observation object is produced | generated from the signal obtained by imaging | photography. Thereby, an image suitable for blood vessel observation can be displayed on the
〈光源の波長帯域〉
光源装置120に備える光源は、白色帯域の光、又は、白色帯域の光として複数の波長帯域の光を発生する光源でもよいし、白色の波長帯域よりも狭い特定の波長帯域の光を発生する光源でもよい。
<Wavelength band of light source>
The light source included in the
特定の波長帯域は、可視域の青色帯域若しくは緑色帯域、あるいは、可視域の赤色帯域であってもよい。特定の波長帯域が可視域の青色帯域若しくは緑色帯域である場合、390nm以上450nm以下、又は530nm以上550nm以下の波長帯域を含み、かつ、390nm以上450nm以下又は530nm以上550nm以下の波長帯域内にピーク波長を有していてもよい。また、特定の波長帯域が可視域の赤色帯域である場合、585nm以上615nm以下、又は610nm以上730nm以下、の波長帯域を含み、かつ、特定の波長帯域の光は、585nm以上615nm以下又は610nm以上730nm以下の波長帯域内にピーク波長を有していてもよい。
The specific wavelength band may be a visible blue band or a green band, or a visible red band. When the specific wavelength band is the visible blue band or the green band, it includes a wavelength band of 390 nm to 450 nm, or 530 nm to 550 nm, and peaks within a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. It may have a wavelength. In addition, when the specific wavelength band is the visible red band, the wavelength band of 585 nm or more and 615 nm or less, or 610 nm or more and 730 nm or less is included, and the light of the specific wavelength band is 585 nm or more and 615 nm or less or 610 nm or more It may have a peak wavelength in a wavelength band of 730 nm or less.
上述した特定の波長帯域の光は、酸化ヘモグロビンと還元ヘモグロビンとで吸光係数が異なる波長帯域を含み、かつ、酸化ヘモグロビンと還元ヘモグロビンとで吸光係数が異なる波長帯域にピーク波長を有していてもよい。この場合、特定の波長帯域は、400±10nm、440±10nm、470±10nm、又は、600nm以上750nmの波長帯域を含み、かつ、400±10nm、440±10nm、470±10nm、又は600nm以上750nm以下の波長帯域にピーク波長を有していてもよい。
The light of the specific wavelength band described above includes a wavelength band having a different absorption coefficient between oxyhemoglobin and reduced hemoglobin, and has a peak wavelength in a wavelength band having a different absorption coefficient between oxyhemoglobin and reduced hemoglobin. Good. In this case, the specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm to 750 nm, and 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm to 750 nm You may have a peak wavelength in the following wavelength bands.
また、光源が発生する光は、790nm以上820nm以下、又は905nm以上970nm以下の波長帯域を含み、かつ、790nm以上820nm以下又は905nm以上970nm以下の波長帯域にピーク波長を有していてもよい。
The light generated from the light source includes a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm, and may have a peak wavelength in a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.
また、光源は、ピークが390nm以上470nm以下である励起光を照射する光源を備えていてもよい。この場合、被検体内の蛍光物質が発する蛍光の情報を有する画像を取得できる。
The light source may include a light source that emits excitation light having a peak of 390 nm or more and 470 nm or less. In this case, an image having information on the fluorescence emitted by the fluorescent substance in the subject can be acquired.
光源の種類(レーザ光源、キセノン光源、LED光源(LED:Light-Emitting Diode)等)、波長、フィルタの有無等は、観察対象の種類、観察の目的等に応じて構成することが好ましく、また、観察の際は観察対象の種類、観察の目的等に応じて照明光の波長を組合せ、及び/又は、切り替えることが好ましい。波長を切り替える場合、たとえば、光源の前方に配置され、特定波長の光を透過又は遮光するフィルタが設けられた円板状のフィルタ(ロータリカラーフィルタ)を回転させることにより、照射する光の波長を切り替えてもよい。
The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. are preferably configured according to the type of observation target, the purpose of observation, etc. In observation, it is preferable to combine and / or switch the wavelength of the illumination light according to the type of observation object, the purpose of observation, and the like. When switching the wavelength, for example, by rotating a disk-shaped filter (rotary color filter) provided in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of light to be irradiated is changed. You may switch.
[その他]
上記の実施の形態において、コンピュータに実現させている機能は、各種のプロセッサに実現させることができる。各種のプロセッサには、プログラムを実行して各種の処理を行う処理部として機能する汎用的なプロセッサであるCPU、FPGA(Field Programmable Gate Array)などの製造後に回路構成を変更可能なプロセッサであるPLD(Programmable Logic Device)、ASIC(Application Specific Integrated Circuit)などの特定の処理を実行させるために専用に設計された回路構成を有するプロセッサである専用電気回路などが含まれる。
[Others]
In the above embodiment, the functions realized by the computer can be realized by various processors. Various processors include a general-purpose processor that functions as a processing unit that executes programs and performs various processes, and a PLD that is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array). (Programmable Logic Device), a dedicated electric circuit which is a processor having a circuit configuration designed exclusively for executing specific processing such as ASIC (Application Specific Integrated Circuit).
一つの機能は、同種又は異種の二つ以上のプロセッサに実現させてもよい。たとえば、複数のFPGAで実現させる構成としてもよいし、CPU及びFPGAの組み合わせで実現させる構成としてもよい。
One function may be realized by two or more processors of the same type or different types. For example, a configuration realized by a plurality of FPGAs or a configuration realized by a combination of a CPU and an FPGA may be used.
また、複数の機能を一つのプロセッサで構成してもよい。複数の機能を一つのプロセッサで実現する構成の例としては、第1に、クライアント、サーバなどのコンピュータに代表されるように、一つ以上のCPUとソフトウェアとの組合せで一つのプロセッサを構成し、このプロセッサに複数の機能を実現させる形態がある。第2に、システムオンチップ(SoC:System On Chip)などに代表されるように、複数の機能を一つのICチップ(IC:Integrated Circuit)で実現するプロセッサを使用する形態がある。このように、各種の機能は、ハードウェア的な構造として、上記各種のプロセッサを一つ以上用いて実現される。
A plurality of functions may be configured by a single processor. As an example of a configuration that realizes a plurality of functions by a single processor, first, as represented by a computer such as a client and a server, a single processor is configured by a combination of one or more CPUs and software. There is a form in which the processor realizes a plurality of functions. Secondly, as represented by a system on chip (SoC), there is a form in which a processor that realizes a plurality of functions with one IC chip (IC: Integrated Circuit) is used. In this way, various functions are realized by using one or more of the various processors as a hardware structure.
更に、これらの各種のプロセッサのハードウェア的な構造は、より具体的には、半導体素子などの回路素子を組み合わせた電気回路である。
Furthermore, the hardware structure of these various processors is more specifically an electric circuit in which circuit elements such as semiconductor elements are combined.
1 内視鏡システム
10 ネットワーク
100 内視鏡装置
110 電子内視鏡
112 挿入部
114 操作部
116 接続部
120 光源装置
130 プロセッサ装置
140 表示装置
150 コンソール
200 画像記録装置
210 コンピュータ
220 外部記憶装置
300 検査動画処理装置
300A 単位動画抽出部
300B シーン変化判定部
300C 記録制御部
300D 高鮮鋭度フレーム抽出部
300E 転送制御部
312 RAM313 ハードディスクドライブ
314 光学ディスクドライブ
315 通信インターフェース
316 入出力インターフェース
320 キーボード
321 マウス
322 ディスプレイ
GX1 前フレームにおけるX方向の合計輝度値のグラフ
GX2 後フレームにおけるX方向の合計輝度値のグラフ
GY1 前フレームにおけるY方向の合計輝度値のグラフ
GY2 後フレームにおけるY方向の合計輝度値のグラフ
En 輝度値
Pn1 輝度値Enにおける前フレームの画素数
Pn2 輝度値Enにおける後フレームの画素数
M 検査動画
T 単位時間
UM 単位動画
f フレーム
S10~S16 検査動画の処理手順
S12A~S12E シーンの変化の有無を判定する処理の手順
1 Endoscope system
10 network
100 Endoscope device
110 Electronic endoscope
112 Insertion part
114 Operation unit
116 connections
120 light source device
130 processor unit
140 Display device
150 console
200 Image recording device
210 computer
220 External storage device
300 Inspection video processing device
300A unit video extractor
300B Scene change determination unit
300C recording controller
300D high sharpness frame extraction unit
300E Transfer control unit
312 RAM313 Hard disk drive
314 Optical disk drive
315 Communication interface
316 I / O interface
320 keyboard
321 mouse
322 display
GX1 Graph of the total luminance value in the X direction in the previous frame
GX2 Graph of the total luminance value in the X direction in the subsequent frame
GY1 Graph of the total luminance value in the Y direction in the previous frame
GY2 Graph of the total luminance value in the Y direction in the subsequent frame
En brightness value
Pn1 Number of pixels in the previous frame at the brightness value En
Pn2 Number of pixels in the subsequent frame at the brightness value En
M inspection video
T Unit time
UM video
f frame
S10-S16 Inspection video processing procedure
S12A to S12E Processing procedure for determining presence / absence of scene change
Claims (12)
-
内視鏡の検査動画を取得する検査動画取得部と、
前記検査動画取得部で取得された前記検査動画から複数のフレームを有する単位動画を抽出する単位動画抽出部と、
前記単位動画抽出部で抽出された前記単位動画を解析し、前記単位動画内でのシーンの変化の有無を判定するシーン変化判定部と、
前記シーン変化判定部でシーンの変化がないと判定された前記単位動画に含まれる少なくとも一部のフレームを第1記憶媒体に記録する記録制御部と、
を備えた検査動画処理装置。
An inspection video acquisition unit for acquiring an inspection video of the endoscope;
A unit video extractor for extracting a unit video having a plurality of frames from the test video acquired by the test video acquisition unit;
Analyzing the unit moving image extracted by the unit moving image extracting unit, and determining whether or not there is a change in the scene in the unit moving image; and
A recording control unit for recording in the first storage medium at least a part of the frames included in the unit moving image determined by the scene change determination unit to have no scene change;
Inspection video processing apparatus comprising:
-
前記シーン変化判定部は、前記単位動画から一定の時間間隔で時系列順にフレームを抽出し、抽出した前後のフレーム間で変化の有無を検出して、前記単位動画内でのシーンの変化の有無を判定する、
請求項1に記載の検査動画処理装置。
The scene change determination unit extracts frames from the unit moving image in a time-series order at a constant time interval, detects whether there is a change between the extracted frames before and after the extraction, and whether there is a scene change in the unit moving image. Determine
The inspection moving image processing apparatus according to claim 1.
-
前記シーン変化判定部は、フレームを構成する画像のX方向における合計輝度値のグラフの形状の変化及び/又はY方向における合計輝度値のグラフの形状の変化に基づいて、前後のフレーム間の変化の有無を判定する、
請求項2に記載の検査動画処理装置。
The scene change determination unit changes between the previous and next frames based on a change in the shape of the graph of the total luminance value in the X direction and / or a change in the shape of the graph of the total luminance value in the Y direction of the images constituting the frame. The presence or absence of
The inspection moving image processing apparatus according to claim 2.
-
前記X方向における合計輝度値のグラフは、X方向の単一又は複数の画素列の輝度値の合計である、
請求項3に記載の検査動画処理装置。
The graph of the total luminance value in the X direction is the total luminance value of a single or a plurality of pixel columns in the X direction.
The inspection moving image processing apparatus according to claim 3.
-
前記Y方向における合計輝度値のグラフは、Y方向の単一又は複数の画素列の輝度値の合計である、
請求項3又は4に記載の検査動画処理装置。
The graph of the total luminance value in the Y direction is the total luminance value of a single or a plurality of pixel columns in the Y direction.
The inspection moving image processing apparatus according to claim 3 or 4.
-
前記シーン変化判定部は、前記X方向における合計輝度値のグラフのフレーム間の差分値に基づく、
請求項3に記載の検査動画処理装置。
The scene change determination unit is based on a difference value between frames of the graph of the total luminance value in the X direction.
The inspection moving image processing apparatus according to claim 3.
-
前記シーン変化判定部は、前記Y方向における合計輝度値のグラフのフレーム間の差分値に基づく、
請求項3又は4に記載の検査動画処理装置。
The scene change determination unit is based on a difference value between frames of the graph of the total luminance value in the Y direction.
The inspection moving image processing apparatus according to claim 3 or 4.
-
前記シーン変化判定部でシーンの変化がないと判定された前記単位動画を解析し、最も鮮鋭度の高いフレームを抽出する高鮮鋭度フレーム抽出部を更に備え、
前記記録制御部は、前記高鮮鋭度フレーム抽出部で抽出されたフレームを前記第1記憶媒体に記録する、
請求項1から4のいずれか1項に記載の検査動画処理装置。
The unit change analysis unit analyzes the unit moving image determined to have no scene change, and further includes a high sharpness frame extraction unit that extracts a frame with the highest sharpness,
The recording control unit records the frame extracted by the high-sharpness frame extracting unit in the first storage medium;
The inspection moving image processing apparatus according to any one of claims 1 to 4.
-
前記検査動画取得部は、検査中の前記検査動画をリアルタイムに取得する、
請求項1から7のいずれか1項に記載の検査動画処理装置。
The inspection video acquisition unit acquires the inspection video under inspection in real time,
The inspection moving image processing apparatus according to any one of claims 1 to 7.
-
前記第1記憶媒体に記録したフレームを第2記憶媒体に転送する転送制御部を更に備えた、
請求項1から9のいずれか1項に記載の検査動画処理装置。
A transfer control unit for transferring the frame recorded in the first storage medium to the second storage medium;
The inspection moving image processing apparatus according to any one of claims 1 to 9.
-
内視鏡の検査動画を取得するステップと、
取得した前記検査動画から複数のフレームを有する単位動画を抽出するステップと、
抽出した前記単位動画を解析し、前記単位動画内でのシーンの変化の有無を判定するステップと、
シーンの変化がないと判定された前記単位動画に含まれる少なくとも一部のフレームを第1記憶媒体に記録するステップと、
を含む検査動画処理方法。
Obtaining an inspection video of the endoscope;
Extracting a unit video having a plurality of frames from the acquired inspection video;
Analyzing the extracted unit video and determining whether there is a scene change in the unit video; and
Recording at least a part of the frames included in the unit moving image determined to have no scene change in the first storage medium;
Inspection video processing method including
-
内視鏡の検査動画を取得する機能と、
取得した前記検査動画から複数のフレームを有する単位動画を抽出する機能と、
抽出した前記単位動画を解析し、前記単位動画内でのシーンの変化の有無を判定する機能と、
シーンの変化がないと判定された前記単位動画に含まれる少なくとも一部のフレームを第1記憶媒体に記録する機能と、
をコンピュータに実現させる検査動画処理プログラム。
A function to obtain an inspection video of the endoscope;
A function of extracting a unit video having a plurality of frames from the acquired inspection video;
A function of analyzing the extracted unit video and determining the presence or absence of a scene change in the unit video;
A function of recording at least a part of the frames included in the unit moving image determined to have no scene change in the first storage medium;
Inspection video processing program that makes computer realize.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2020518198A JP7158471B2 (en) | 2018-05-09 | 2019-04-08 | Inspection video processing device, operation method of inspection video processing device, and inspection video processing program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2018-090616 | 2018-05-09 | ||
JP2018090616 | 2018-05-09 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019216084A1 true WO2019216084A1 (en) | 2019-11-14 |
Family
ID=68467917
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2019/015361 WO2019216084A1 (en) | 2018-05-09 | 2019-04-08 | Examination moving image processing device, examination moving image processing method, and examination moving image processing program |
Country Status (2)
Country | Link |
---|---|
JP (1) | JP7158471B2 (en) |
WO (1) | WO2019216084A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7495063B2 (en) | 2021-05-21 | 2024-06-04 | 富士フイルム株式会社 | Endoscope system and method of operation thereof |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2010187756A (en) * | 2009-02-16 | 2010-09-02 | Olympus Corp | Image processing apparatus, image processing method, and image processing program |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4652694B2 (en) * | 2004-01-08 | 2011-03-16 | オリンパス株式会社 | Image processing method |
JP5147308B2 (en) * | 2007-06-20 | 2013-02-20 | オリンパス株式会社 | Image extraction apparatus and image extraction program |
JP5959168B2 (en) * | 2011-08-31 | 2016-08-02 | オリンパス株式会社 | Image processing apparatus, operation method of image processing apparatus, and image processing program |
JP5863439B2 (en) * | 2011-12-21 | 2016-02-16 | Hoya株式会社 | Endoscope device |
JP6067264B2 (en) * | 2012-07-17 | 2017-01-25 | Hoya株式会社 | Image processing apparatus and endoscope apparatus |
CN109475284B (en) * | 2016-10-20 | 2021-08-27 | 奥林巴斯株式会社 | Endoscope system, terminal device, server, and transmission method |
-
2019
- 2019-04-08 JP JP2020518198A patent/JP7158471B2/en active Active
- 2019-04-08 WO PCT/JP2019/015361 patent/WO2019216084A1/en active Application Filing
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2010187756A (en) * | 2009-02-16 | 2010-09-02 | Olympus Corp | Image processing apparatus, image processing method, and image processing program |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7495063B2 (en) | 2021-05-21 | 2024-06-04 | 富士フイルム株式会社 | Endoscope system and method of operation thereof |
Also Published As
Publication number | Publication date |
---|---|
JP7158471B2 (en) | 2022-10-21 |
JPWO2019216084A1 (en) | 2021-06-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7346285B2 (en) | Medical image processing device, endoscope system, operating method and program for medical image processing device | |
JP6049518B2 (en) | Image processing apparatus, endoscope apparatus, program, and operation method of image processing apparatus | |
JP6150583B2 (en) | Image processing apparatus, endoscope apparatus, program, and operation method of image processing apparatus | |
CN111295127B (en) | Examination support device, endoscope device, and recording medium | |
CN113543694B (en) | Medical image processing device, processor device, endoscope system, medical image processing method, and recording medium | |
JP7135082B2 (en) | Endoscope device, method of operating endoscope device, and program | |
CN113573654A (en) | AI system for detecting and determining lesion size | |
JP2017148392A (en) | Calculation system | |
JP7337073B2 (en) | MEDICAL IMAGE PROCESSING APPARATUS, ENDOSCOPE SYSTEM, AND METHOD OF OPERATION OF MEDICAL IMAGE PROCESSING APPARATUS | |
JP6949999B2 (en) | Image processing equipment, endoscopic systems, image processing methods, programs and recording media | |
JP7326308B2 (en) | MEDICAL IMAGE PROCESSING APPARATUS, OPERATION METHOD OF MEDICAL IMAGE PROCESSING APPARATUS, ENDOSCOPE SYSTEM, PROCESSOR DEVICE, DIAGNOSTIC SUPPORT DEVICE, AND PROGRAM | |
WO2019016912A1 (en) | Diagnosis supporting device, diagnosis supporting method and program | |
JPWO2020170791A1 (en) | Medical image processing equipment and methods | |
US20200170484A1 (en) | Diagnosis support system, endoscope system, processor and diagnosis support method | |
KR20220054340A (en) | Systems, devices, and methods for turbidity analysis | |
CN116133572A (en) | Image analysis processing device, endoscope system, method for operating image analysis processing device, and program for image analysis processing device | |
JP7158471B2 (en) | Inspection video processing device, operation method of inspection video processing device, and inspection video processing program | |
JP6128989B2 (en) | Image processing apparatus, endoscope apparatus, and operation method of image processing apparatus | |
CN116134363A (en) | Endoscope system and working method thereof | |
WO2022014258A1 (en) | Processor device and processor device operation method | |
JP7214886B2 (en) | Image processing device and its operating method | |
WO2022230607A1 (en) | Medical image processing device, endoscope system, and operation method for medical image processing device | |
JP6168878B2 (en) | Image processing apparatus, endoscope apparatus, and image processing method | |
WO2022210508A1 (en) | Processor device, medical image processing device, medical image processing system, and endoscopic system | |
JP7470776B2 (en) | ENDOSCOPE SYSTEM, CONTROL METHOD, AND CONTROL PROGRAM |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19799099 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2020518198 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19799099 Country of ref document: EP Kind code of ref document: A1 |