WO2019116630A1

WO2019116630A1 - Endoscopic image observation support system

Info

Publication number: WO2019116630A1
Application number: PCT/JP2018/029398
Authority: WO
Inventors: 亮猪股; 卓志永田
Original assignee: オリンパス株式会社
Priority date: 2017-12-12
Filing date: 2018-08-06
Publication date: 2019-06-20
Also published as: JP2021052810A

Abstract

According to the present invention, an image generation unit 50 generates a diagnostic reading screen including a reproduction region of an endoscopic image. A display processing unit 58 displays a stripe-shaped color bar which indicates a temporal change in a color value of the endoscopic image in the diagnostic reading screen, and arranges and displays, in association with the color bar, extraction images extracted from among a plurality of endoscopic images. The display processing unit 58 arranges and displays the extraction images so that time information on the color bar is equal to time information on the extraction images.

Description

Endoscope image observation support system

The present invention relates to a system for supporting observation of an endoscopic image taken by a capsule endoscope.

Patent Document 1 converts color information extracted along a time series from a group of in-vivo images into a band-like image (average color bar) along a time axis, and incorporates an average color bar into an image reading screen. Disclose the technology to be displayed. When interpreting the capsule endoscope image, the user first performs an operation of specifying an image obtained by capturing the start position of the region. Since the average color bar indicates the temporal change of the color value of the in-vivo image, the user refers to the average color bar to register the start position of the site.

International Publication No. 2011/0013475

However, the color information that composes the average color bar is affected by the imaging environment such as illumination, so the user's experience is also needed to make a mark on the start position of the site. In addition, since the specification of the start position of a site | part is a preparation work of observation to the last, it is preferable to be performed as soon as possible. Therefore, in addition to the average color bar, it is preferable that more useful information be presented in order to register the start position of the site.

The present invention has been made in view of these circumstances, and an object thereof is to realize a function of presenting information useful for registration of the start position of a site.

In order to solve the above-mentioned subject, an endoscope image observation support system of an embodiment of the present invention is an observation support system which supports observation of a plurality of endoscope images which a capsule endoscope photographed the inside of a subject. Display processing for displaying a strip-shaped color bar indicating temporal changes in color values of the captured endoscopic image on the diagnostic image generation screen for generating a diagnostic reading screen including a reproduction area of the endoscopic image and the diagnostic reading screen And a unit. The display processing unit arranges and displays extracted images extracted from the plurality of endoscopic images in association with the color bars.

It is to be noted that any combination of the above-described constituent elements, one obtained by converting the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program and the like is also effective as an aspect of the present invention.

It is a figure for demonstrating the outline | summary of the image observation assistance system of the capsule endoscope concerning an Example. It is a figure which shows the structure of a management server and a recording device. It is a figure which shows the example of the interpretation screen of an endoscopic image. It is a figure which shows the component of a color bar. It is a figure which shows a mark display area when the expansion button is operated. It is a figure which shows a mark display area when the expansion button is operated. It is a figure which shows a mark display area when the expansion button is operated.

FIG. 1 is a view for explaining an outline of an image observation support system of a capsule endoscope according to an embodiment. The endoscopic image observation support system 1 supports observation of a capsule endoscopic image by a reader. In an examination using a normal endoscope, a doctor observes an image taken by the endoscope inserted into a patient's body in real time on a display to make a diagnosis. However, in a capsule endoscopy, a reader reads a capsule This is different from ordinary endoscopy in that a large number of images captured in the past by the endoscope are collectively observed.

In capsule endoscopy, a patient (subject) has a plurality of receiving antennas (not shown) attached to the abdomen, and the receiving device 4 is attached to the waist by a belt. Swallow the endoscope 3 from the mouth. The capsule endoscope 3 is an image obtained by A / D-converting an imaging signal output from an imaging unit that is a microminiature camera, an illumination unit that illuminates the inside of a subject, and the imaging unit, and adding an image ID and imaging time information. A signal processing unit that generates data, a memory that temporarily stores image data, a communication module that transmits image data stored in the memory, and a battery that supplies power to each unit are provided. The capsule endoscope 3 periodically captures a still image while moving the digestive tract, and transmits image data to the receiving device 4 via an antenna.

The receiving device 4 incorporates a recording medium 5, and the receiving device 4 adds related information including radio wave intensity information at the time of reception at each reception antenna to the image data received by each reception antenna, Record 5 When the capsule endoscope 3 captures an image of the body every 0.5 seconds, when imaging in the body is completed in about 8 hours, about 60,000 pieces of endoscopic image data are recorded on the recording medium 5.

The image ID is information for identifying an image, and may be information to which a serial number indicating the imaging order is added. For example, “1” may be added to the image ID of the endoscope image captured first, and “2” may be added to the image ID of the endoscope image captured second. By generating the image ID in this manner, the serial number included in the image ID expresses the photographing order, and the duplication of the image ID can be avoided. The image ID and the photographing time information may be added to the photographed image as related information by the receiving device 4 when the receiving device 4 receives the photographed image. In any case, the image captured by the capsule endoscope 3 is recorded in the recording medium 5 in association with related information such as an image ID, shooting time information and received radio wave intensity.

When the antenna and the receiving device 4 are recovered from the patient, the data terminal of the receiving device 4 is connected to the data reading device connected to the management server 10, and the data reading device is about 60,000 sheets recorded on the recording medium 5. The endoscope image data and the related information are read out and transmitted to the management server 10. The data reading device may be an external device connected to the management server 10 by a USB cable or the like. The recording medium 5 may be a memory card that can be attached to and detached from the receiving device 4, and the recording medium 5 is removed from the receiving device 4 and mounted on a data reading device to read endoscope image data and related information. It may be done. The recording medium 5 may be attached to a data reading slot provided in the management server 10 to read endoscope image data and related information.

The management server 10 performs predetermined image processing on the endoscope image read from the recording medium 5 and records the image in the recording device 12. The recording device 12 may be configured by an HDD (hard disk drive) or may be configured by a flash memory. The endoscopic image recorded in the recording medium 5 is an uncompressed RAW (raw) image or a RAW image subjected to only lossless compression, so the data size is very large. Therefore, the management server 10 performs predetermined lossy compression processing on the endoscopic image which is the RAW image, reduces the data size, and records the result in the recording device 12. In the embodiment, the management server 10 takes charge of the image processing of the endoscope raw image, but another apparatus, for example, the terminal device 20 performs the image processing on the endoscope raw image and records it in the recording device 12 The recording device 12 may be provided in the terminal device 20.

The plurality of terminal devices 20 are connected to the management server 10 by a network 2 such as a LAN (local area network). The terminal device 20 is a personal computer or the like assigned to an image reading person such as a doctor or an engineer (hereinafter may be simply referred to as “user”) and is connected to the display device 22 so as to allow screen output. 20 may be a laptop computer integrated with a display device, or may be a portable tablet. The terminal device 20 accesses the management server 10 and displays the endoscopic image recorded in the recording device 12 on the display device 22.

When compressing the endoscopic RAW image, the management server 10 has a function of executing an analysis application to analyze the endoscopic image. Image analysis by the analysis application is performed on all endoscopic RAW images captured in one capsule endoscopy, and the result of the image analysis is added as additional information to the compressed endoscopic image .

One of the purposes of capsule endoscopy is to look for a source of bleeding in the digestive tract. When the management server 10 acquires the endoscope RAW image from the recording medium 5, the management server 10 executes the analysis application to perform image processing, thereby specifying the endoscope RAW image which may have taken a bleeding state. For example, when the redness of the endoscopic image exceeds a predetermined threshold value, the management server 10 determines that the image is likely to have taken a bleeding state, and compresses the endoscopic RAW image, Flag information indicating that the image is red is added.

The management server 10 derives color information of each endoscopic RAW image in the process of determining a reddish image. The color information indicates the average color value of the image, and the average color value expresses the color tone of the image. The management server 10 uses the color information of each image to determine whether the image is highly likely to have taken a bleeding state. The derived color information is also used when constructing a color bar to be displayed on an interpretation screen or the like. The management server 10 adds color information when compressing the endoscopic RAW image.

There is a difference in the moving speed of the capsule endoscope 3 in the digestive tract, and the change in the endoscopic image to be captured is small at the place where the moving speed is slow. Therefore, it is inefficient and burdensome for the image reader to observe all the images including a plurality of substantially unchanged images equally. Therefore, the analysis application performs a process of specifying an image (similar image) with a small change by comparing endoscopic RAW images captured continuously in time. Hereinafter, this process is referred to as "image summarizing process".

In the image summarizing process, a reference image is set, and the ratio of the covered area of the reference image to the determination target image to be determined to be similar to the reference image is calculated as the coverage. The determination target image is an image captured later than the reference image. The analysis application determines the determination target image as a similar image of the reference image when the coverage is equal to or greater than the threshold. When compressing the endoscopic RAW image which is the reference image, the management server 10 adds flag information indicating that it is the reference image, and when compressing the endoscopic RAW image which is the similar image, the similar image And flag information indicating that the

The reproduction application executed in the management server 10 or the terminal device 20 has a reproduction mode for shortening the reproduction time of the endoscope image with reference to the flag information added by the image summarizing process, and the reader By selecting this reproduction mode, shortening of the observation time can be realized.

The playback application of the embodiment is configured to have four playback modes.
(First playback mode)
The first reproduction mode is a manual reproduction mode using an operation of a user interface connected to the terminal device 20. In the first reproduction mode, the endoscope image can be frame-by-frame displayed one by one by rotating the wheel of the mouse. Therefore, the first reproduction mode is used when specifying the image in which the lesion is most clearly imaged among a plurality of images in which the lesion is imaged. When the user rotates the wheel backward, the endoscopic image is continuously reproduced and displayed in the forward direction (direction from the old image to the new image at the photographing time), and when the user rotates the wheel forward, the endoscopic view The mirror image is continuously reproduced and displayed in the reverse direction (direction from the new image at the shooting time to the old image).

(2nd playback mode)
The second reproduction mode is an automatic reproduction mode in which the endoscope image is continuously reproduced and displayed in the forward or reverse direction at the set reproduction speed. The second reproduction mode is used for normal endoscopic image observation.

(3rd playback mode)
In the third playback mode, similar images are forward-directed at a speed higher than the preset playback speed while continuously playing back and displaying the reference image specified by the image summarizing process in the forward or reverse direction at the preset playback speed. This is an automatic playback mode in which continuous playback display is performed in the reverse direction. The third reproduction mode realizes shortening of the observation time as compared to the second reproduction mode by reproducing at high speed a similar image having a small change with respect to the reference image.

(4th playback mode)
The fourth reproduction mode is an automatic reproduction mode in which the display of the similar image specified by the image summarizing process is omitted, and only the reference image is reproduced and displayed in the forward or reverse direction at the set reproduction speed. The fourth reproduction mode realizes shortening of the observation time as compared with the third reproduction mode by omitting the display of the similar image. A fourth back reproduction mode which is a back mode may be set to the fourth reproduction mode. The fourth reverse reproduction mode is an automatic reproduction mode in which the display of the reference image is omitted and only the similar image is reproduced and displayed in the forward direction or the reverse direction at the set reproduction speed. The fourth back reproduction mode is used to confirm that there is no observation omission after the observation in the fourth reproduction mode.

The first to third reproduction modes are continuous reproduction modes in which the temporally consecutive endoscopic images are reproduced and displayed in order, and the fourth reproduction mode (fourth back reproduction mode) is a temporally continuous endoscopic view. This is a thinning reproduction mode in which a mirror image is thinned and reproduced and displayed. The reproduction application carries out an endoscope image reproduction process according to the reproduction mode selected by the user. The reproduction application may be executed by the management server 10 or may be executed by the terminal device 20.

The terminal device 20 is connected to a user interface such as a keyboard and a mouse. The terminal device 20 has a function of cooperating with the management server 10 to support the image interpretation work by the image reader. The terminal device 20 causes the display device 22 to display the interpretation screen of the endoscope image, and the user observes the endoscope image reproduced and displayed on the interpretation screen and captures the endoscope image obtained by imaging a lesion or the like. .

FIG. 2 shows the configuration of the management server 10 and the recording device 12. The management server 10 includes an acquisition unit 30, an image processing unit 40, a screen generation unit 50, an operation reception unit 52, a trajectory processing unit 54, a section setting unit 56, and a display processing unit 58. The acquisition unit 30 includes a RAW image acquisition unit 32 and a related information acquisition unit 34. The image processing unit 40 includes a color information generation unit 42, a redness determination unit 44, an image summary processing unit 46, and a compression processing unit 48. Each function of the management server 10 may be realized by executing various applications such as an analysis application and a reproduction application. In the embodiment, the management server 10 executes various applications, but the terminal device 20 may execute various applications.

The recording device 12 includes an endoscope image recording unit 80, a related information recording unit 82, a position information recording unit 84, a trajectory image recording unit 86, an examination information recording unit 88, and an observation content recording unit 90. The endoscopic image recording unit 80 records the endoscopic image subjected to the image processing by the image processing unit 40. The examination information recording unit 88 records information on endoscopy. The observation content recording unit 90 records the observation content registered for the past examination of the capsule endoscope, for example, finding information, a finding image associated with the finding information, and the like.

The configuration of the management server 10 can be realized as hardware in any processor, memory, or other LSI, and software can be realized as a program loaded in the memory, etc. Here, it is realized by cooperation of them. Are drawing functional blocks. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The RAW image acquisition unit 32 acquires approximately 60,000 sheets of endoscope RAW images transmitted from the data reading device and causes the recording device 12 to temporarily store them. The related information acquisition unit 34 acquires related information transmitted from the data reading device and causes the related information recording unit 82 to record the related information. The image processing unit 40 performs the following image processing on all endoscopic raw images. In addition, image ID is matched with endoscopic raw image data.

<Generation of color information>
The color information generation unit 42 derives color information of each endoscopic RAW image. The color information generation unit 42 derives color information of each image as a color value obtained by averaging pixel values. The color information generation unit 42 may divide the endoscopic RAW image into a plurality of pieces at regular intervals in the height direction, and derive color information for each divided area. The color information generation unit 42 provides color information of each image to the compression processing unit 48.

<Identification of reddish image>
The redness determination unit 44 searches for a reddish endoscopic RAW image from the color information of each image derived by the color information generation unit 42, and specifies an image whose redness is stronger than a predetermined threshold. The redness determination unit 44 provides the compression processing unit 48 with the image ID of the specified redness image.

<Image summary processing>
The image summarization processing unit 46 performs an image summarization process that classifies all endoscopic images into a reference image and a similar image similar to the reference image. First, the image summarizing processing unit 46 sets an image captured first as a reference image. The image summary processing unit 46 performs similarity determination as to whether or not the determination target image captured next to the reference image is similar to the reference image. The image summary processing unit 46 obtains a covered area including a deformed image obtained by deforming the reference image in the determination target image, and calculates a ratio of the covered area in the determination target image as a coverage.

The image summary processing unit 46 determines the determination target image as a similar image of the reference image when the coverage is equal to or greater than the threshold. The image summary processing unit 46 performs the similarity determination as to whether or not the image is similar to the reference image, using the image captured next to the image determined as the similar image as the determination target image. If the moving speed of the capsule endoscope 3 is low, several dozen images taken after the reference image may be determined as similar images.

On the other hand, when the coverage is less than the threshold value, the image summarizing processing unit 46 determines the determination target image as a dissimilar image. The image summary processing unit 46 sets an image determined as a non-similar image as a new reference image, and performs similarity determination using an image captured next as a determination target image. The image summarization processing unit 46 performs this image summarization processing on all about 60,000 sheets of images and classifies them into a reference image and a similar image.

The number ratio of the reference image to the similar image is adjusted by setting of the threshold. When the threshold is increased, the number of reference images is increased, and when the threshold is decreased, the number of reference images is decreased. In the fourth reproduction mode, since only the reference image is reproduced and displayed, the setting of the threshold is important for suppressing the oversight of the lesion and the like, but according to the results so far, about 60,000 endoscopic images Among them, it has been found that the interpretation of only the reference image prevents the oversight of the lesion image by setting a threshold value to extract about 20,000 sheets as the reference image. The image summary processing unit 46 provides the compression processing unit 48 with the image ID of the classified reference image and the image ID of the similar image.

<Compression process of endoscopic RAW image>
The image analysis processing by the color information generation unit 42, the redness determination unit 44, and the image summarization processing unit 46 is performed when the compression processing unit 48 compresses the endoscopic RAW image. The compression processing unit 48 performs irreversible compression processing on the endoscopic RAW image to generate an image file to which the image ID and photographing time information are added, and causes the endoscopic image recording unit 80 to record the image file. For example, the compression processing unit 48 may compress the endoscopic RAW image in an image format such as JPEG.

The compression processing unit 48 adds information indicating the analysis result provided from the color information generation unit 42, the redness determination unit 44, and the image summary processing unit 46 to the compressed image file. Specifically, the compression processing unit 48 adds the color information provided from the color information generation unit 42 to each compressed image. Further, the compression processing unit 48 adds information indicating that the image is a reddish image to the compressed image having the image ID provided from the redness judging unit 44. This information may be added as flag information. Further, the compression processing unit 48 adds flag information indicating that it is a reference image to the reference image based on the result of the image summarization processing by the image summarization processing unit 46, and that the similar image is a similar image Add flag information to indicate. Since it is in the relation of front and back whether it is a reference image or a similar image, flag value 1 may express a reference image and flag value 0 may express a similar image.

In the embodiment, the color information generation unit 42, the redness determination unit 44, and the image summarization processing unit 46 perform image processing on the endoscopic RAW image before the compression processing of the endoscopic RAW image by the compression processing unit 48. Have been implemented. In a modification, the color information generation unit 42, the redness determination unit 44, and the image summarization processing unit 46 may perform image analysis on the compressed image, and information indicating the analysis result may be added to the compressed image. In the endoscopic image recording unit 80, an image file subjected to image processing by the image processing unit 40 is recorded, and the user uses the image file recorded in the endoscopic image recording unit 80 to Conduct observation.

<Tracking process of capsule endoscope>
The related information acquisition unit 34 acquires related information transmitted from the data reading device and causes the related information recording unit 82 to record the related information. The locus processing unit 54 executes processing for specifying the movement locus of the capsule endoscope 3 from the reception intensity information recorded in the related information recording unit 82.

The locus processing unit 54 detects the position of the capsule endoscope 3 when the endoscopic image is taken based on the reception intensity information when the plurality of reception antennas receive the image data among the related information of the image data. Get the position information to represent. This position is a three-dimensional position in the subject and represents a position when the capsule endoscope 3 captures an image. The method of detecting the imaging position of the capsule endoscope 3 is not limited to the method based on the reception intensity of the wireless signal, and a method using a magnetic field or other known method may be used. For example, the trajectory processing unit 54 derives the amount of movement and the direction of movement within the subject from the difference between the endoscopic image and the endoscopic image taken before and after the endoscopic image, and combines it with the reception intensity information at a plurality of receiving antennas. Position information representing the position of the capsule endoscope 3 at the time of imaging may be derived. This position information may be position coordinates in a three-dimensional space. The locus processing unit 54 causes the position information recording unit 84 to record the acquired position information in association with the image ID.

The locus processing unit 54 generates locus image data representing the movement locus of the capsule endoscope 3 in the subject based on the position information recorded in the position information recording unit 84. The trajectory processing unit 54 may generate three-dimensional trajectory image data by connecting the position information recorded in the position information recording unit 84 in the order of the image ID. In addition, when connecting, trajectory image data may be generated so that connected line segments are connected smoothly. The trajectory processing unit 54 causes the trajectory image recording unit 86 to record the generated trajectory image data.

Hereinafter, the screen displayed on the display device 22 at the time of image interpretation will be described.
The doctor B who is a user inputs a user ID and a password to the terminal device 20 and logs in. When the user logs in, the management server 10 provides the examination information recorded in the examination information recording unit 88 to the terminal device 20, and the display device 22 displays a list of capsule endoscope examinations. On the examination list screen, examination information such as patient ID, patient name, examination ID, examination date and time, and the like are displayed, and the user selects an examination to be subjected to the interpretation report creation. When an examination having a patient ID “1111”, a patient name “A”, and an examination ID “0001” is selected from the examination list, the screen generation unit 50 reads an interpretation screen for the user to interpret an endoscopic image. It is generated and displayed on the display device 22.

FIG. 3 shows an example of an interpretation image screen of an endoscopic image. At the upper center of the image interpretation screen, a reproduction area 100 for switching and reproducing an endoscopic image is provided. The diagnostic reading screen is displayed on the display device 22 with the playback mode selection button 102a at the upper left corner of the screen selected. When the overview mode selection button 102 b is selected, the screen generation unit 50 generates an overview screen in which the images extracted from the plurality of reference images are arranged in a grid, and causes the display device 22 to display the overview screen.

The number-of-reproductions switching button 108 is an operation button for switching the number of images displayed in the reproduction area 100. Although FIG. 3 shows an example in which single-image display is selected, the user can select two-image display or four-image display by operating the reproduction number switching button 108.

In the reproduction mode selection area 130, operation buttons for selecting a reproduction mode are arranged. The second reproduction mode selection button 110 is an operation button for selecting a second reproduction mode. The third reproduction mode selection button 112 is an operation button for selecting the third reproduction mode. The fourth reproduction mode selection button 114 is an operation button for selecting a fourth reproduction mode in which only the reference image is reproduced and displayed. The fourth back reproduction mode selection button 116 is an operation button for selecting a fourth back reproduction mode in which only the similar image is reproduced and displayed. In the fourth reproduction mode, since the reproduction display of similar images is omitted, when the user selects the fourth reproduction mode and interprets, all the endoscopic images are read in the fourth reverse reproduction mode as well. It is recommended to observe.

The user selects one of the second reproduction mode selection button 110, the third reproduction mode selection button 112, the fourth reproduction mode selection button 114, and the fourth back reproduction mode selection button 116 to set the reproduction mode. In the default state, the second reproduction mode selection button 110 is selected. The reproduction button 104 a and the reverse reproduction button 104 b are displayed in the reproduction button display area 104 provided below the reproduction area 100, and when the reproduction button 104 a is selected, the endoscopic image is displayed in the forward direction in the reproduction area 100 ( When the reverse playback button 104b is selected, the endoscopic image is displayed in the reverse direction (from the new image at the shooting time toward the old image) when the reverse playback button 104b is selected. Is displayed on the screen). The reproduction speed adjustment unit 106 includes a slider for adjusting the reproduction speed (display time of one endoscopic image). The playback speed adjustment unit 106 sets the playback speed, that is, the display frame rate of the endoscopic image, according to the position of the slider.

The display processing unit 58 reproduces and displays an endoscopic image in the reproduction area 100 in accordance with the reproduction mode selected in the reproduction mode selection area 130 and the reproduction speed (display frame rate) set by the reproduction speed adjustment part 106. . When the play button 104a or the reverse play button 104b is selected, the display processing unit 58 starts the playback display, but a pause button is displayed instead at the selected play button 104a or the reverse play button 104b. Ru. If the user operates the pause button during playback display of the endoscope image, the display processing unit 58 pauses playback display of the endoscope image. In this state, when the user operates the mouse wheel, the display processing unit 58 displays the endoscope image frame by frame in the first reproduction mode according to the rotation of the mouse wheel.

When the user places the mouse pointer on the image displayed in the playback area 100 and double-clicks the left button of the mouse, the image is captured and displayed in the captured image display area 128. The captured image displayed in the captured image display area 128 is an option of an image to be attached to the interpretation report later. In this example, it is shown that five captured images 128a to 128e are selected.

The screen generation unit 50 displays a horizontally long bar display area 120 with the one end as the imaging start time and the other end as the imaging end time below the reproduction area 100. In the initial state, the bar display area 120 expresses a time axis in which the left end is the shooting start time and the right end is the shooting end time. The slider 122 indicates the temporal position of the endoscopic image displayed in the reproduction area 100, and the time position represented by the slider 122 is relative time information from the imaging start time in the time display area 124. Is also displayed. When the user places the mouse pointer on any part of the bar display area 120 and clicks the left button of the mouse, an endoscopic image at that time position is displayed in the reproduction area 100. Also, even if the user drags the slider 122 and drops it at an arbitrary position in the bar display area 120, an endoscopic image at that time position is displayed in the reproduction area 100.

The display processing unit 58 displays, in the bar display area 120, a strip-shaped color bar 146 that indicates temporal change in color value of the captured endoscopic image. Here, the color bar 146 is configured by arranging color information extracted from a plurality of endoscopic images along a time series, that is, color values, along a time axis.

FIG. 4 shows the components of the color bar 146 displayed in the bar display area 120. As shown in FIG. The color bar 146 is constituted by a plurality of color elements 148 arranged in time series in the longitudinal direction of the bar display area 120. Here, five color elements 148 _n-2 to 148 _{n + 2} are shown. Each color element 148 is a vertically long display element having a color obtained by averaging color values of one or more images. For example, if one color element 148 corresponds to the color information of one image, the color element 148 is represented by the color information of each image itself, and thus becomes a vertically long element having the average color of each image. In this case, if there are 60,000 endoscope images, 60,000 color elements 148 will be present. When one color element 148 corresponds to the color information of 100 images, the color element 148 is a vertically long element having a color obtained by averaging the color information of 100 consecutive images in time series. If there are 60,000 sheets of endoscopic images, 600 color elements 148 will be present. In the embodiment, the display processing unit 58 configures one color element 148 with the average color of 100 endoscopic images.

The display processing unit 58 calculates the color information of the color element 148 from the color information of the continuous 100 images, derives the width of the color element 148 so as to fit in the bar display area 120, and a plurality (here, 600) Are arranged in the bar display area 120 along the time series. The color bar 146 is thereby formed. The color bar 146 indicates the temporal change of the color value of the photographed endoscopic image, and the user looks at the color bar 146 and makes a notion of the start position of each site.

The red image display button 126 is a button for displaying a red mark at the shooting time of a reddish image in the bar display area 120. When the red image display button 126 is operated, the display processing unit 58 displays a red mark at the photographing time of the reddish image. By displaying a red mark in the bar display area 120, the user can recognize the presence of an image that is highly likely to have taken a hemorrhage.

The enlarged display button 118 is a button for enlarging the playback area 100. When the enlarged display button 118 is operated, the capture image display area 128 is hidden, and the reproduction area 100 is enlarged by that amount.

The display processing unit 58 displays a trajectory image in the trajectory display area 150. The locus display area 150 is displayed as a window, and the display processing unit 58 can enlarge the locus display area 150 by a user operation and move it to any position. The display processing unit 58 uses the trajectory image data recorded in the trajectory image recording unit 86 to display a three-dimensional trajectory image in the trajectory display area 150. The user can rotate the trajectory image by operating the mouse in the trajectory display area 150.

The user can add a mark to the bar display area 120 to indicate the start position of the site. The user refers to the temporal color change represented by the color bar 146, and while observing the endoscopic image reproduced and displayed in the reproduction area 100, the marking button (FIG. (Not shown) to mark the start position of each site on the bar display area 120. The section setting unit 56 receives the marking operation and sets a section of each part. By performing this marking process, when the endoscopic image is reviewed, the start position of the site can be easily known. In particular, when a user different from the doctor B observes the endoscopic image, the marking process is performed, so that the start position of the region can be easily recognized, and the image observation can be smoothly performed. Marking is generally performed on an image taken at the entrance of the stomach, small intestine, or large intestine. Marking information is added to the endoscope image displayed in the reproduction area 100 when the marking button is operated.

In the interpretation screen of the embodiment, the display processing unit 58 extracts a part of the images extracted from the plurality of endoscopic images in association with the color bar 146 in order to perform the site marking of the user efficiently. Hereinafter, they are also displayed side by side as “extracted image”. Specifically, the display processing unit 58 arranges and displays the extracted endoscopic images along the longitudinal direction of the strip-shaped color bar 146. The endoscopic images (extracted images) to be arranged may be extracted from among a plurality of reference images.

The time axis change button 142 includes an enlargement button 142 a for changing the scale of the time axis in the bar display area 120 and a reduction button 142 b. The enlargement button 142 a is a button for enlarging the time axis scale, and the reduction button 142 b is a button for reducing the time axis scale. The time axis adjustment unit 144 has a slider for adjusting the scale of the time axis, and the user can also set the scale of the time axis according to the position of the slider. As shown in FIG. 3, when the slider of the time axis adjustment unit 144 is at the left end, the enlargement ratio of the scale is minimum, and the bar display area 120 is displayed such that the entire length thereof fits in the interpretation screen. In the area 120, a color bar 146 is displayed.

When the enlargement button 142a is operated once, the screen generation unit 50 doubles the bar display area 120 in the horizontal direction. When the reduction button 142b is operated once, the screen generation unit 50 displays the bar. The area 120 is reduced by half in the lateral direction. The enlargement ratio and the reduction ratio are arbitrary and may be determined as appropriate. In the embodiment, the enlargement and reduction in the vertical direction are not performed but may be performed. When the enlargement button 142a is operated once from the state shown in FIG. 3, the screen generation unit 50 doubles the time axis to display the bar display area 120, and the display processing unit 58 matches the enlargement ratio. The color component 148 is reconstructed and displayed as a color bar 146.

FIG. 5 shows the bar display area 120 when the enlargement button 142a is operated. When the bar display area 120 is enlarged, the display processing unit 58 enlarges and displays each color element 148 of the color bar 146 and arranges the extracted endoscopic images along the longitudinal direction of the color bar 146. indicate. In this example, the display processing unit 58 superimposes and displays the endoscopic image on the color bar 146, but the endoscopic image is parallel to the color bar 146 in a region different from the color bar 146. May be displayed side by side.

The display processing unit 58 superimposes the endoscopic image on the color bar 146 so that the user can visually recognize at least a part of the color bar 146. Therefore, the user can confirm temporal changes in color values in the color bar 146 simultaneously while confirming the endoscopic image when making a reference to the start position of the site. Therefore, the display processing unit 58 shortens the longitudinal length of the endoscopic image to be shorter than the longitudinal length of the bar display area 120 (color bar 146) so that the user can visually recognize the color bar 146. Superimpose on the color bar 146.

Here, the display processing unit 58 arranges and displays the extracted endoscope images so that the time information of the color bar 146 and the time information (imaging time information) of the extracted endoscope image are aligned. Thus, the color bar 146 can be associated with the extracted image, and the user can recognize the relationship between the color of the color bar 146 and the actual image showing the color.

The display processing unit 58 may perform processing of associating one endoscopic image with each color element 148. In the embodiment, since the color component 148 is configured by the average color of 100 images, the display processing unit 58 associates one of the 100 images with the color component 148. For example, the display processing unit 58 may select an image to be associated from among the reference images included in 100 sheets.

As shown in FIG. 4, the color bar 146 is configured by arranging a plurality of vertically long color elements 148 in the horizontal direction, and thus the endoscopic image is to be superimposed on the plurality of color elements 148. . The display processing unit 58 may specify, for example, the color element 148 located at the lateral center of the endoscopic image, and display the endoscopic image associated with the specified color element 148. In any case, it is preferable that the display processing unit 58 display an endoscopic image associated with one of the plurality of superimposed color elements 148 in the bar display area 120.

As illustrated, since the display processing unit 58 enlargedly displays the time axis of the bar display area 120, the right end of the bar display area 120 does not fit on the screen. The user can move the slider 122 to move the color bar 146 left and right, and the display processing unit 58 displays the color bar 146 and the endoscopic image according to the movement of the slider 122.

FIG. 6 shows the bar display area 120 when the enlargement button 142a is operated on the image interpretation screen shown in FIG. The display processing unit 58 further displays the time axis of the bar display area 120 twice as large. As the time axis is expanded in the lateral direction by a factor of two, the number of endoscopic images displayed per unit time is also doubled.

FIG. 7 shows the bar display area 120 when the enlargement button 142a is operated on the radiogram interpretation screen shown in FIG. The display processing unit 58 further displays the time axis of the bar display area 120 twice as large. By enlarging the time axis, the number of endoscopic images to be superimposed and displayed per unit time increases, and the user can easily find an image at or near the start position of the region.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that this embodiment is an exemplification, and that various modifications can be made to the combination of each component and each processing process, and such a modification is also within the scope of the present invention. .

In the modification, the recording device 12 extracts the boundary images of the stomach, the small intestine, and the large intestine taken in the past, that is, the feature quantities extracted from the past boundary images obtained by photographing the start positions of the respective digestive tracts. The feature amount of the site start position such as size, shape, etc. may be held. The image processing unit 40 performs a process of specifying an endoscopic RAW image that is likely to be an image (boundary image) at the start position of the region by the image matching process. The image processing unit 40 may perform the image matching process on all the endoscopic images by using the feature amount of the boundary image held in the recording device 12, but is set as the reference image by the image summarizing process. An image matching process may be performed on an endoscopic image. This image matching process may utilize known techniques. When the image processing unit 40 extracts an endoscopic RAW image with high possibility of the image at the start position of the stomach, small intestine, and large intestine, the compression processing unit 48 is extracted in each image as a candidate of the start position of the region. And information indicating the part name.

When the image processing by the image processing unit 40 is performed, the display processing unit 58 associates the endoscopic image (candidate image) extracted as a candidate for the start position of the region with the color bar 146. Display. In the embodiment, the display processing unit 58 arranges and displays the extracted images extracted from the plurality of reference images, but when there is a candidate image indicating the start position of the region, the candidate displayed in the images displayed side by side It is preferred to include an image. At this time, the display processing unit 58 may display the candidate image so as to be more prominent than other images, for example, by displaying the candidate image in a blinking manner. With such a display, the user can easily recognize the presence of the candidate image indicating the start position of the site.

1 ... endoscopic image observation support system, 10 ... management server, 12 ... recording device, 20 ... terminal device, 22 ... display device, 30 ... acquisition unit, 32 · · · RAW image acquisition unit 34: related information acquisition unit 40: image processing unit 42: color information generation unit 44: redness determination unit 46: image summary processing unit 48 ... compression processing unit 50: screen generation unit 52: operation reception unit 54: trajectory processing unit 56: section setting unit 58: display processing unit 80 Endoscope image recording unit 82 Related information recording unit 84 Position information recording unit 86 Track image recording unit 88 Examination information recording unit 90 Observation contents Recording unit, 120 ... bar display area, 142 ... time axis change button, 142a ... enlargement button, 142b · · · - reduction button, 144 ... time axis adjusting section, 146 ... color bars, 148 ... color component.

The present invention can be used in the field of supporting observation of an endoscopic image taken by a capsule endoscope.

Claims

An observation support system for supporting observation of a plurality of endoscopic images obtained by photographing the inside of a subject with a capsule endoscope,
A screen generation unit that generates an interpretation screen including a reproduction area of the endoscopic image;
And a display processing unit configured to display a strip-shaped color bar indicating temporal change in color value of the endoscopic image on the interpretation screen.
The display processing unit arranges and displays extracted images extracted from the plurality of endoscopic images in association with the color bar.
An endoscopic image observation support system characterized in that.
The display processing unit arranges and displays the extracted images along the longitudinal direction of the color bar.
The endoscopic image observation assistance system of Claim 1 characterized by the above-mentioned.
The display processing unit displays the extracted images side by side so that the time information of the color bar and the time information of the extracted image are aligned.
The endoscopic image observation assistance system of Claim 2 characterized by the above-mentioned.
The display processing unit displays the extracted image superimposed on the color bar.
The endoscopic image observation assistance system of Claim 2 characterized by the above-mentioned.
Each of the plurality of endoscopic images is classified into either a reference image or a similar image similar to the reference image,
The display processing unit arranges and displays the extracted images extracted from the plurality of reference images.
The endoscopic image observation assistance system of Claim 1 characterized by the above-mentioned.
The display processing unit displays the extracted image extracted as a candidate for the start position of the part in association with the color bar.
The endoscopic image observation assistance system of Claim 1 characterized by the above-mentioned.