JP2017191609A - Image processing apparatus and image processing method - Google Patents

Abstract

An image processing apparatus and an image processing method capable of efficiently observing an image taken for a long time are provided. A plurality of temporally continuous images obtained by time-lapse observation of a living cell using a microscope are acquired. Analyzing a plurality of feature values for each acquired image, based on the analysis result, an index of the degree of change of the plurality of feature values between temporally consecutive images, and changing each of the plurality of feature values Calculate using quantity. An image whose index exceeds a predetermined threshold is selected, and the selected image is displayed in time series. [Selection] Figure 6

Description

  The present technology relates to an image processing apparatus and an image processing method for processing a microscopic image.

  In the time-lapse observation for photographing a living cell for a long time, various contrivances for shortening the time required for observation have been made in observing a photographed image (moving image).

  In order to shorten the time, there are several methods for analyzing a photographed cell image and calculating a feature amount. For example, there is a technique for recognizing a cell or the like by combining edge detection, contour extraction, filter processing, information in the Z-axis direction, and the like (see, for example, Patent Documents 1 to 3 and Non-Patent Documents 1 to 3). In these techniques, an area where a cell or the like is imaged is classified, and a temporal change in feature amount is output for each classified area.

Japanese Patent Laid-Open No. 2004-054347 JP 2006-018394 A JP 2011-027543 A

"Cell Segmentation in Microscopy Imagery Using A Bag Of Local Bayesian Classifiers", Z. Yin, R. Bise, M. Chen and T. Kanade, 2010 "High-resolution cell outline segmentation and tracking from phase-contrast microscopy images", ME Ambuhl, C. Brepsant, J.-J. Meister, AB Verkhovsky and IF Sbalzarini, 2011, Journal of Microscopy, Vol. 245, Pt 2 2012 , pp. 161-170 "Advanced Level-Set Based Multiple-Cell Segmentation and Tracking in Time-Lapse Fluorescence Microscopy Images", O. Dzyubachyk, W. Niessen and E. Meijering, 2008

  However, the conventional technique only recognizes the cells, and it takes time for the photographing time to check all the images taken for a long time. As described above, there is an insufficient point for efficiently observing an image taken for a long time.

  In view of the circumstances as described above, an object of the present technology is to provide an image processing apparatus and an image processing method capable of efficiently observing an image taken for a long time.

  In order to achieve the above object, an image processing apparatus according to an embodiment of the present technology acquires a plurality of temporally continuous images obtained by time-lapse observation of biological cells using a microscope, and acquires each of the acquired Analyzing the feature quantity of the image, and calculating an index of the degree of change in the feature quantity between the temporally continuous images based on the analysis result, and the index exceeds a predetermined threshold value A control unit configured to select a selected image and display the selected image on the display unit in time series.

  In order to achieve the above object, in the image processing apparatus according to an aspect of the present technology, the control unit sets an ROI region in the acquired image, analyzes the feature amount of the set ROI region, The index may be calculated for each ROI region.

  In order to achieve the above object, in the image processing apparatus according to an aspect of the present technology, the control unit selects, for at least one ROI region, an image in which the calculated index exceeds the threshold value. It may be configured.

  In order to achieve the above object, in the image processing apparatus according to an aspect of the present technology, the control unit accepts designation of the ROI region from a user via an input unit, and the index of the designated ROI region is A configuration may be used in which an image exceeding the threshold is selected.

  In order to achieve the above object, in the image processing apparatus according to an aspect of the present technology, the control unit may be configured to display an enlarged portion of the ROI area in each of the images.

  In order to achieve the above object, in the image processing device according to an aspect of the present technology, the control unit may be configured to set the threshold value by an input received via the input unit.

  In order to achieve the above object, in the image processing device according to an aspect of the present technology, the control unit may be configured to set the threshold based on the amount of change in the feature amount.

  In order to achieve the above object, in the image processing apparatus according to an aspect of the present technology, the control unit reproduces the plurality of images as a moving image according to the reproduction time received via the input unit. The threshold value may be adjusted so that an image is selected.

  In order to achieve the above object, in the image processing device according to an aspect of the present technology, the feature amount may be at least one of luminance, an edge, a motion vector, and a density.

In order to achieve the above object, in the image processing apparatus according to an aspect of the present technology, the control unit includes the change amount of the feature amount, a normalized change amount obtained by normalizing the change amount, and a normal amount. It is also possible to use the following formula using the total amount of change in chemical conversion.
(Index) = Σ ((change amount of the feature amount) × (normalized change amount of the feature amount) / (total normalized change amount))

  In order to achieve the above object, in the image processing method according to an aspect of the present technology, the control unit acquires a plurality of temporally continuous images obtained by observing biological cells using a microscope over time, and Analyzing the feature quantity of each acquired image, calculating an index of the degree of change of the feature quantity between each temporally continuous image based on the analysis result, and the index is predetermined An image exceeding the threshold is selected, and the selected image is displayed on the display unit in time series.

As described above, according to the present technology, an image taken for a long time can be efficiently observed.
Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure for demonstrating the image of the digest reproduction | regeneration of a moving image performed with the image processing apparatus which concerns on this embodiment. It is a figure which shows the example of GUI used with the image processing apparatus which concerns on this embodiment. It is a figure which shows the example of GUI used with the image processing apparatus which concerns on this embodiment. 1 is a configuration diagram of a microscope system 1 including an image processing apparatus 50 according to the present embodiment. 3 is a configuration diagram of a control unit 51. FIG. 5 is a flowchart for explaining an outline of a flow of processing performed by the image processing apparatus 50. It is a flowchart for demonstrating the flow of a designation | designated / detection process of ROI. It is a figure which shows the example of the ROI information table 52a. It is a flowchart for demonstrating the flow of the calculation process of the change parameter | index for every ROI. It is a figure which shows the example of the ROI number and change index table 52c. It is a flowchart for demonstrating the flow of a digest production | generation and display process. It is an example of the histogram of a change parameter | index. It is a flowchart for demonstrating the flow of the calculation process of the change parameter | index for every ROI. It is an example of the table which stores the ROI number of ROI which is a process target, and each change amount in the ROI. It is a flowchart for demonstrating the flow of a digest production | generation and display process.

<First Embodiment>
Hereinafter, embodiments according to the present technology will be described with reference to the drawings. In the following description, first, an image of how to reproduce a long-time moving image acquired by time-lapse observation or the like will be described. Next, indices used for digest creation will be described, and then the configuration of the image processing apparatus according to the present embodiment and the flow of the processing will be described.

[Image of digest playback]
First, an image of digest reproduction of a moving image performed by the image processing apparatus according to the present embodiment will be described. FIG. 1 is a diagram for explaining an image of digest reproduction of a moving image performed by the image processing apparatus according to the present embodiment.

  The upper side of the figure shows an example of a moving image that is a basis for digest reproduction. In the image at time t = 1, two circular cells are shown. Here, the region where the upper right cell is shown is ROI (Region of Interest) 1 and the region where the lower left cell is shown is ROI2.

  It is assumed that there is no change in the image at time t = 2, and that the shape of the ROI2 cell changes to a triangle at time t = 3. It is assumed that there is no change in the image at time t = 4, and that the shape of the ROI1 cell has changed to a triangle at time t = 5. It is assumed that there is no change in the image at time t = 6, and the shape of both cells has changed to a square at time t = 7.

  The center of the figure shows the state of digest playback that captures changes in the entire frame (image). That is, if there is a change in any of the ROIs shown in the image, a digest is generated so as to include all the changes. In this example, an image at time t = 3 when only ROI1 cells changed, an image at time t = 5 when only ROI2 cells changed, and an image at time t = 7 when both cells changed, a total of four images Digest playback is performed with images.

  The lower side of the figure shows a state of digest reproduction capturing changes for each ROI. For example, in the cell of ROI1, since there was a change only at time t = 5 and time t = 7, the digest is composed of a total of three images including these. In the digest that captures the change in ROI1, the ROI1 region may be enlarged as shown here, or the entire image may be displayed so that the positional relationship with other cells can be easily understood. Good.

  Heretofore, an image of digest reproduction of a moving image performed by the image processing apparatus according to the present embodiment has been described.

[Frame selection method for digest playback (outline)]
Next, an outline of a method for extracting a frame used for a digest from a frame constituting a moving image will be described.

  First, consider the above-mentioned index (change index) related to “change”. And this change index is calculated | required for every flame | frame which comprises a moving image. As a reference for selecting a frame to be used for digest, it is adopted that this change index exceeds a predetermined threshold.

  Thus, when the user changes the threshold setting, the number of frames having a change index exceeding the threshold increases or decreases. Digest playback for various playback times can be performed by increasing or decreasing the number of frames used for the digest.

  In addition, by configuring the image forming apparatus so that a plurality of types of “changes” can be captured, a digest can be created in accordance with changes specified by the user, such as cell division, migration, and neovascularization.

  As described above, in the image processing apparatus according to the present technology, the type of change that the user (researcher) wants to see and the degree of change (threshold value) are specified, and only the image of the place where the change occurs more than the threshold value is selected. Since it can be observed as a digest, the analysis time can be shortened.

  The outline of the method for extracting the frame used for the digest from the frames constituting the moving image has been described above.

[Types of changes targeted by this technology (example)]
Here, the types of “change” handled by the image processing apparatus according to the present technology will be described with three examples. In the present technology, digest reproduction can also be performed by capturing changes other than the types of changes listed here.

(1) Changes in morphology and state such as cell division (changes in edges and brightness)
In order to capture changes in form and state, a phase contrast image or a fluorescence image can be used. The phase difference image can be used to determine the contour position when setting an ROI region or when tracking (tracking) imaging of cells. The change index can be obtained using the number of edges obtained by measuring the phase difference image and the dispersion of the edges.

  Further, by using the fluorescent image, it is possible to determine the frames that will be the starting point and the ending point of the continuous change. For example, the frame where the change in fluorescence starts can be set as the start point, and the frame where the decrease is minimized while the luminance value of the fluorescence increases or decreases can be set as the end point. A change index can be obtained using a histogram of luminance values obtained by measuring a fluorescent image.

  It is also possible to use a combination of both a phase difference image and a fluorescence image obtained by photographing one biological specimen. For example, since the outline of a cell is clearly shown in the phase contrast image, the region where the cell exists is determined as the ROI region using the phase contrast image, and then the change in the ROI region is captured using the fluorescence image. You can also.

(2) Changes in dynamics (motion) such as migration and transport Each of the migrating cells is tracked, and the motion vector of each cell is calculated by a method such as ME (Motion Vector Estimation). The change index can be calculated from the total amount of change in the motion amount calculated based on the motion vector.

(3) Changes in density characteristic of immune cell activity and angiogenesis Density of phosphors shown in fluorescence image in areas where immune cells are activated and areas where angiogenesis is performed (tumor is present) A change index can be calculated from the change. For example, a change index can be calculated from a change in the sum of luminance values in the ROI region.

  The types of “changes” handled by the image processing apparatus according to the present technology have been described with three examples.

[Change index and coefficient of change index]
Here, an example of how to obtain the above-described change index will be described. In the method of obtaining the change index described here, first, a coefficient for obtaining the change index is calculated, and the change index is calculated using the calculated coefficient.

  First, calculation is performed in consideration of four types of changes, and the amount of change is A, B, C, and D. These change amounts include, for example, a change in luminance value, a change in the total number of edges, a change in density, and a change in motion.

  Next, the normalized change amount is obtained. The normalized change amount is obtained from the formula (change amount) / (total change amount). The normalized change amounts of the change amounts A, B, C, and D are a, b, c, and d, respectively.

  Next, a total n of normalized change amounts is obtained. The total normalized change amount n is obtained by summing the respective normalized change amounts. n = Σ (each normalized change)

  Next, a coefficient for calculating the change index is obtained. The coefficients for the change amounts A, B, C, and D are p, q, r, and s, respectively, and the coefficients are expressed as follows: p = a / n, q = b / n, r = c / n, s = d / n Can be requested.

  Note that the method of obtaining the coefficients p, q, r, and s is not limited to the method of obtaining the above-described mathematical expressions. For example, all may be 1, or a default value set in advance may be used.

  Finally, the change index is calculated as the sum of (coefficient × change amount). That is, the change index can be obtained by the following formula.

  (Change index) = pA + qB + rC + sD

  Although four types of changes are considered here, more types of changes may be taken into account when calculating the change index. In this case, the additional change amount is E, the normalized change amount is e, and the coefficient is u, and u = e / n, (change index) = pA + qB + rC + sD + uE.

  The example of how to obtain the change index has been described above.

[Example of GUI of image processing apparatus]
Next, an example of a GUI (Graphical User Interface) used in the image processing apparatus according to the present embodiment will be described. 2 and 3 are diagrams illustrating examples of GUIs used in the image processing apparatus according to the present embodiment.

  In the example shown in FIG. 2, a slide bar for the user to specify the playback time for digest playback and an above-described threshold value for the user to specify below the image display area in which the fluorescent image and the bright field image are displayed. A slide bar is provided. Note that there is a relationship between the playback time and the threshold value such that the threshold value decreases when the playback time is extended, and the playback time decreases when the threshold value is increased.

  In this example, the user designates a round cell reflected in the vicinity of the center of the image display area as an ROI by directly clicking or the like, and adjusts the playback time slide bar. Then, with respect to the designated ROI region, the threshold value is adjusted so as to be within the designated time, and a frame whose change index exceeds the threshold value is selected from the changed frames, and digest reproduction is performed.

  In the example illustrated in FIG. 3, corresponding slide bars are displayed according to the number of ROIs designated by the user in the image display area. Here, since three ROIs are designated by the user, three slide bars are displayed so that the user can designate a threshold value for selecting a frame to be subjected to digest reproduction in each ROI.

  The example of the GUI used in the image processing apparatus according to the present embodiment has been described above.

[Configuration of image processing apparatus]
Next, an outline of a configuration of a microscope system including the image processing apparatus according to the present embodiment will be described. FIG. 4 is a configuration diagram of the microscope system 1 including the image processing apparatus 50 according to the present embodiment.

  The microscope system 1 includes a laser light source unit 10, a scanner unit 20, a microscope 30, a microscope controller 40, and an image processing device 50. Note that processing such as image analysis performed by the image processing apparatus 50 may be performed by the image analysis server 100 on the local network or the image analysis cloud server 200 on the Internet cloud.

  The laser light source unit 10 generates excitation light for generating fluorescence from a specimen such as a fluorescently labeled cell. The generated excitation light is introduced into the scanner unit 20.

  The laser light source unit 10 includes a laser controller 11, and the laser controller 11 controls the intensity of the excitation light, the light emission interval, and the like.

  The scanner unit 20 includes a scanner controller 21, a galvano mirror 22, a dichroic mirror 23, and a photodetector 24.

  The galvanometer mirror 22 is irradiated with the laser beam for excitation introduced from the laser light source unit 10 in the X direction and the horizontal direction (XY direction) of the specimen placed on the stage 35 of the microscope 30 and irradiated. The direction of the laser beam is changed in each Y direction. The laser light whose direction is adjusted by the galvanometer mirror 22 passes through the dichroic mirror 23 and is then introduced into the microscope 30. The laser light introduced into the microscope 30 is irradiated onto the specimen, and the fluorescence excited by the irradiation returns from the microscope 30 to the scanner unit 20.

  The dichroic mirror 23 reflects only the fluorescence of the laser light and fluorescence returned from the microscope 30 toward the photodetector 24.

  The light detector 24 is generally a PMT (Photomultiplier Tube) and detects fluorescence excited in a sample irradiated with laser light. In the case of a confocal microscope, a pinhole is installed in front of the photodetector 24 on the optical path. The position where the pinhole is installed is in a conjugate position with a focus position of an objective lens 32 described later.

  The scanner controller 21 controls the galvanometer mirror 22 and the photodetector 24 in order to scan the sample in the XY directions. The signal detected by the photodetector 24 is transmitted to the image processing apparatus 50 as a luminance value for each point on the XY plane to be scanned.

  The microscope 30 includes a filter 31, an objective lens 32, a stage 35, a stage driving unit 34, an imaging lens 36, an imaging unit 37, and a temperature sensor 38. A specimen to be observed is placed on the stage 35.

  The filter 31 guides the laser light introduced from the scanner unit 20 to the objective lens 32. Further, the filter 31 guides fluorescence generated from the specimen by laser irradiation to the scanner unit 20 or guides reflected or transmitted light from the bright field light source (not shown) to the imaging lens 36. .

  The objective lens 32 condenses the laser light introduced from the scanner unit 20 via the filter 31 at the focal position of the objective lens 32. The objective lens 32 guides fluorescence generated from the specimen to the scanner unit 20 via the filter 31, and guides light emitted from the bright field light source to the imaging lens 36 via the filter 31. To do.

  A specimen is placed on the stage 35. The stage 35 is moved by the stage drive unit 34 in the XY direction perpendicular to the optical axis of the objective lens 32 and in the Z direction along the optical axis of the objective lens 32. 35 is moved.

  The imaging lens 36 images the bright field light that has passed through the objective lens 32 and the filter 31 on an imaging element (not shown) of the imaging unit 37.

  The imaging unit 37 captures a bright field image formed on the imaging element by the imaging lens 36. The captured image is sent to the image processing apparatus 50 and analyzed.

  The microscope controller 40 moves the stage 35 with respect to the stage driving unit 34 based on an instruction from the image processing apparatus 50 regarding autofocus, cell tracking exposure, and the like.

  The image processing apparatus 50 includes a control unit 51, a storage unit 52, an input unit 53, and a display unit 54. The image processing apparatus 50 is generally configured by a PC (Personal Computer).

  The control unit 51 controls the entire microscope system 1, synthesizes a fluorescence image from the luminance value detected by the photodetector 24 and the coordinate value on the XY plane from which the luminance value is detected, or is synthesized. The fluorescence image is analyzed to calculate the optimum laser beam intensity and control the laser beam intensity. Moreover, the control part 51 controls the microscope controller 40 as mentioned above, and implement | achieves functions, such as an autofocus function and cell tracking exposure, of the microscope 30.

  The control unit 51 is realized by executing a program stored in the storage unit 52 by a CPU (Central Processing Unit). Details of the configuration of the control unit 51 will be described later.

  The storage unit 52 includes a hard disk drive or a semiconductor memory, and the above-described program group executed by the CPU, the fluorescent image acquired from the scanner unit 20, the ROI information table 52a, the image analysis result 52b, and the ROI number. A change index table 52c, a moving image (frame) 52d, and the like are stored.

  The input unit 53 includes a pointing device such as a mouse and a touch panel, a keyboard, and the like, and inputs an instruction from the user via a GUI or the like.

  The display unit 54 includes a display device such as a liquid crystal display and a display control unit for drawing a GUI or an image on the display device.

  The outline of the configuration of the microscope system 1 including the image processing apparatus 50 according to the present embodiment has been described above. Although the image processing apparatus 50 has been described as a part of the microscope system 1 here, the present invention is not limited to this configuration, and the image processing apparatus 50 may be configured by a PC or the like independent of the microscope system 1.

[Configuration of Control Unit 51]
Here, the configuration of the control unit 51, which is a functional block realized by executing a program by the CPU, will be described. FIG. 5 is a configuration diagram of the control unit 51. Here, of the functions performed by the control unit 51, only the part related to digest playback is shown.

  The control unit 51 includes an ROI designation unit 51a, an ROI detection unit 51b, an image measurement unit 51c, a measurement result analysis unit 51d, a change amount calculation unit 51e, a coefficient calculation unit 51f, a change index calculation unit 51g, a change index histogram creation unit 51h, An ROI / frame selection unit 51i, a viewpoint conversion unit 51j, and a digest creation unit 51k are provided.

  The ROI designation unit 51 a displays a micrographed image on the display screen of the display unit 54, and accepts an area on the image designated by the user using the GUI and the input unit 53 as an ROI.

  When the user does not designate a desired ROI using the ROI designation unit 51a, the ROI detection unit 51b detects a region that should be the ROI instead from the target image.

  The image measuring unit 51c measures a luminance value, an edge, an area, a luminance centroid, and the like in the ROI set on the target image. The measurement result is analyzed by the measurement result analysis unit 51d.

  The measurement result analysis unit 51d analyzes the measurement result obtained by the image measurement unit 51c. For example, based on the luminance value measured by the image measurement unit 51c, a histogram of luminance values is created, or the variance of luminance values is calculated. Further, based on the edges measured by the image measuring unit 51c, the sum of the edges and the variance of the edges are calculated. Also, the luminance density is calculated based on the area and the luminance centroid measured by the image measuring unit 51c.

  The change amount calculation unit 51e calculates the change amount between frames or the change amount per unit time from the analysis result by the measurement result analysis unit 51d and the analysis result 52b of the past image stored in the storage unit 52. To do.

  The coefficient calculation unit 51f calculates the coefficient of the change index described above based on the change amount calculated by the change amount calculation unit 51e.

  The change index calculation unit 51g calculates a change index based on the change amount calculated by the change amount calculation unit 51e and the coefficient calculated by the coefficient calculation unit 51f.

  The change index histogram creation unit 51h creates a histogram for selecting a frame to be included in the digest from each frame of the moving image that is the source of the digest and the value of the change index that the frame has.

  The ROI / frame selection unit 51i selects a frame for performing digest reproduction in the designated or detected ROI based on the histogram created by the change index histogram creation unit 51h.

  The viewpoint conversion unit 51j is not an essential component for digest reproduction, and is not limited to the viewpoint used by the user for capturing a biological specimen, that is, the viewpoint from directly above the specimen, but viewed from a viewpoint arbitrarily designated by the user. An image is reconstructed and displayed. In order to realize this function, there is an image (Z stack) obtained by photographing the XY plane by moving the focus to a plurality of positions in the Z direction at each time of photographing, and a 3D image is constructed. Is the premise.

  The digest creation unit 51k displays the frames selected by the ROI / frame selection unit 51i on the display unit 54 in chronological order, thereby performing digest reproduction of a moving image obtained by photographing a biological specimen for time-lapse observation. Note that the digest creating unit 51k may represent the process of change in a format in which thumbnail images are arranged in a time series by using the changed image as a still image thumbnail image in addition to reproducing the digest as a moving image. In addition, the digest creation unit 51k may extract an image in which a change has occurred and output it in a format used for a paper or report.

  The configuration of the control unit 51 has been described above.

[Outline of processing performed by image processing apparatus]
Next, an outline of the flow of processing performed by the image processing apparatus 50 will be described. FIG. 6 is a flowchart for explaining an outline of the flow of processing performed by the image processing apparatus 50.

  First, the control unit 51 performs designation or detection (ROI designation / detection processing) by the user of the ROI in the frame to be processed (step S10).

  Next, the control unit 51 performs a change index calculation process for each ROI in the frame to be processed (step S20).

  Next, the control unit 51 determines whether or not change indices have been calculated for all ROIs (step S30). When the ROI for which the change index has not yet been calculated remains (N in step S30), the process returns to step S20, and the change index calculation process is continued. When the calculation of the change index has been completed for all ROIs (Y in step S30), the process proceeds to the next step S40.

  Next, the control unit 51 creates a digest by selecting a frame to be used for display from the original moving image using the change index calculated in step S20, and displays the created digest (digest creation and display). (Step S40).

  Although not shown, the control unit 51 does not obtain the change index for all the ROIs for only one frame constituting the moving image, but for all the ROIs for all the frames constituting the moving image. Needless to say, it seeks change indicators.

  The outline of the flow of processing performed by the image processing apparatus 50 has been described above. In other words, the control unit 51 of the image processing apparatus 50 acquires a plurality of temporally continuous images obtained by observing biological cells with a microscope over time, and acquires each of the acquired images. Analyzing the feature quantity of the image, and calculating an index of the degree of change in the feature quantity between the temporally continuous images based on the analysis result, and the index exceeds a predetermined threshold value It can be said that the selected image is selected and displayed on the display unit 54 in time series. Below, the detailed flow of each process demonstrated here is demonstrated.

[ROI designation / detection processing]
Here, the flow of ROI designation / detection processing by the control unit 51 will be described. FIG. 7 is a flowchart for explaining the flow of ROI designation / detection processing.

  First, the control unit 51 reads an image for one frame from the moving image 52d that is a basis for creating the digest stored in the storage unit 52 (step S11). The read image is displayed in the image display area of the display unit 54.

  Next, the ROI designation unit 51a receives the designation of the ROI from the user, and determines whether or not the ROI is designated (step S12). When the ROI is designated by the user (yes in step S12), the process proceeds to step S17. When the ROI is not designated by the user (no in step S12), the process proceeds to step S13.

  If there is no ROI designation by the user in step S12, the control unit 51 next determines whether or not the image to be processed is a fluorescent image (step S13).

  When the image is a fluorescent image (yes in step S13), the process proceeds to step S14. If the image is not a fluorescent image (no in step S13), the process proceeds to step S15.

  If it is determined in step S13 that the image is a fluorescent image, then the ROI detection unit 51b detects a part having a high luminance value (fluorescent part) from the image, and the detected fluorescent part is designated as ROI. (Step S14).

  When it is determined in step S13 that the image is not a fluorescent image or after the processing in step S14, the control unit 51 determines whether or not the image to be processed is a bright field image (step S15). If the image is a bright field image (yes in step S15), the process proceeds to step S16. If the image is not a bright field image (no in step S15), the process proceeds to step S17.

  When it is determined in step S15 that the image is a bright field image, the ROI detection unit 51b detects a high-luminance part or an edge part from the image, and sets the detected part as the ROI ( Step S16).

  Finally, the control unit 51 stores the designated or detected ROI information (ROI information) in the ROI information table 52a (step S17).

  The flow of ROI designation / detection processing has been described above.

[ROI information table]
Next, an example of the ROI information table 52a will be described. FIG. 8 is a diagram illustrating an example of the ROI information table 52a.

  One row of this table is composed of an ROI number for uniquely identifying a designated or detected ROI, a center coordinate of the ROI, a size of the ROI, and a frame number of a frame including the ROI. Yes.

  Here, although the size of the ROI is represented by the width (W) and the height (H), the shape of the ROI is not limited to a quadrangle, and may be any shape. It may be point-like.

  The example of the ROI information table 52a has been described above.

[Calculation processing of change index for each ROI]
Next, the flow of the change index calculation process for each ROI by the control unit 51 will be described. FIG. 9 is a flowchart for explaining the flow of processing for calculating a change index for each ROI.

  First, the image measuring unit 51c performs measurement within the region of the ROI that is the processing target on the image that is the processing target (step S21). The measurement may be measurement of a luminance value, measurement of an edge, measurement of an area of a high luminance part, measurement of a luminance center of gravity.

  Next, the measurement result analysis unit 51d analyzes the measurement result obtained by the image measurement unit 51c (step S22). When the luminance value is measured in the previous step, the measurement result analysis unit 51d creates a histogram of luminance values or calculates the variance of luminance values. When the edge is measured in the previous step, the measurement result analysis unit 51d calculates the total number of edges or calculates the variance of the edges. In the previous step, when the area or the luminance center of gravity is measured, the measurement result analysis unit 51d calculates the density of the high luminance part.

  Next, the measurement result analysis unit 51d stores the result of the analysis performed in the previous step in the storage unit 52 (step S23).

  Next, the change amount calculation unit 51e reads the analysis result 52b of the past (the previous frame in the moving image) from the storage unit 52 (step S24).

  Next, the change amount calculation unit 51e calculates the change amount based on the past analysis result 52b read in the previous step and the result analyzed in step S22 (step S25). Examples of the change amount include change amounts such as luminance change, edge change, density change, and motion change. The change amount may be calculated by obtaining a difference between two analysis results.

  Next, the coefficient calculation unit 51f calculates a coefficient for obtaining a change index based on the change amount calculated in the previous step (step S26). The calculation method is as described above.

  Next, the change index calculation unit 51g calculates a change index of the ROI that is the processing target (step S27). The calculation method is as described above.

  Finally, the change index calculation unit 51g writes the ROI number to be processed and the calculated change index into the ROI number / change index table 52c in the storage unit 52 and stores it (step S28).

  The flow of change index calculation processing for each ROI has been described above.

[ROI number / change index table]
Next, an example of the ROI number / change index table 52c will be described. FIG. 10 is a diagram illustrating an example of the ROI number / change index table 52c.

  One row of this table is composed of the ROI number for uniquely identifying the designated or detected ROI and the value of the change index in the ROI.

  In addition to the ROI number and the change index value, a frame number may be recorded.

  The example of the ROI number / change index table 52c has been described above.

[Create and display digest]
Next, a flow of digest creation and display processing by the control unit 51 will be described. FIG. 11 is a flowchart for explaining the flow of digest creation and display processing.

  First, the control unit 51 obtains information regarding the length of the reproduction time, the threshold that affects the reproduction time, and the designation of the ROI for digest reproduction (step S41).

  Next, the control unit 51 reads the ROI number and change index information in the ROI from the ROI number / change index table 52c in the storage unit 52 (step S42).

  Next, the change index histogram creating unit 51h determines whether or not the information acquired in step S41 includes designation of ROI by the user (step S43).

  When the ROI for digest reproduction is designated by the user (step S43: yes), the change index histogram creation unit 51h creates a change index histogram in the designated ROI region (step S44). An example of the change index histogram will be described later.

  When the ROI for digest playback is not specified by the user (no in step S43), the change index histogram creating unit 51h creates a change index histogram for the entire frame (step S45). In this case, the histogram is created using the highest value among the change indices of ROIs included in one frame (OR operation is performed between a plurality of ROIs).

  Next, the ROI / frame selection unit 51i uses the digest of the change index created in the previous step and the frame constituting the original video based on the playback time length and the threshold acquired in step S41. A frame to be used is selected (step S46). If the ROI is designated by the user, in this step, processing for extracting only the designated ROI image from the selected frame may be performed.

  Finally, the digest creation unit 51k performs digest reproduction using the frame or ROI selected in the previous step (step S47).

  The digest creation and display process flow has been described above. In the above description, attention is paid to the amount of change, and a digest is generated by picking up a frame with a large amount of change. However, the present invention is not limited to this configuration. A configuration in which a digest is generated by selecting is also conceivable.

[Histogram of change index]
Next, an example of a change index histogram will be described. FIG. 12 is an example of a change index histogram.

  In the graph shown in this example, the horizontal axis indicates the time corresponding to each frame constituting the moving image, and the vertical axis indicates the change index value. And the value of the change index at each time for each ROI is plotted. In addition, the vertical axis indicates the position of the value serving as the threshold value.

  When the graph and the threshold shown in this example are used, since the change index value at time t = 1, 2, and 4 exceeds the threshold for ROI1, digest playback is performed using a total of three frames at these times. Is done.

  Regarding ROI2, since the value of the change index at times t = 1 and 2 exceeds the threshold value, digest reproduction is performed using a total of two frames at these times.

  The example of the change index histogram has been described above.

[ROI setting method]
Here, a method for setting an ROI on an image when a cell or the like observed by a user moves with time will be described. Examples of ROI setting methods include the following two methods.

  In the first method, first, when setting an ROI so as to surround a cell or the like that the user wants to observe, the range of movement of the cell is grasped over the entire frame selected for use in the digest. Then, the position and range of the ROI are determined with a size including the moving range. Once determined, the position and range of the ROI is fixed and unchanged over all selected frames.

  In the second method, a cell that a user wants to observe is tracked for each image constituting a frame, and a frame used for tracking is set as an ROI. The position and range of the ROI will be different for each frame.

  In some cases, the image processing apparatus 50 may use the first method and the second method. For example, when it is desired to view a moving image shot by time-lapse observation immediately without waiting for processing for all the frames, the second method that does not need to analyze all the selected frames is suitable.

  In addition, two types of ROI are set using both the first method and the second method, and the movement of cells and the like are observed using the ROI set by the first method, and set by the second method. It can also be said that the inside of a cell or the like is observed in detail by using the ROI.

  As described above, the method of setting the ROI on the image when the cell or the like observed by the user moves with time has been described.

  The first embodiment has been described above.

<Modification 1 (Automatic selection of frames constituting a digest)>
In the configuration of the image processing apparatus 50 according to the first embodiment, the user designates a threshold value for selecting a frame constituting the digest. On the other hand, in this modification, the image processing device 50a automatically sets a threshold value for selecting a frame constituting the digest, and the digest is created without inputting the threshold value from the user.

  In addition, as a method of automatically setting the threshold value, there is a method of using the above-described change amount and setting the average value as the threshold value. Further, the differential value of the change amount may be taken, and the average of the taken differential values may be used as the threshold value.

  As described above, the modification example in which the image processing apparatus 50a automatically sets the threshold for selecting the frames constituting the digest and creates the digest without inputting the threshold from the user has been described.

<Modification 2 (Calculation of change index at the time of digest creation)>
In the first embodiment, a change index is calculated in the change index calculation process for each ROI and stored in the ROI number / change index table 52c. In the digest creation and display process, the stored change index for each ROI is Reading out and creating a digest.

  On the other hand, in this modification, the change index is not calculated in the calculation process of the change index for each ROI, and information regarding the ROI number and each change amount is stored in the table. Then, in the digest creation and display processing, the stored ROI number and information on each change amount are read to calculate a change index, and the digest is created using the calculated change index.

  The difference in the processing in the configuration of the present modification from the processing in the first embodiment is the change index calculation processing for each ROI and the digest creation and display processing. Therefore, only these different portions will be described below. .

[Calculation processing of change index for each ROI]
The flow of change index calculation processing for each ROI by the control unit 51 ′ will be described. FIG. 13 is a flowchart for explaining the flow of change index calculation processing for each ROI. In addition, since the process from step S21 to step S26 is the same as 1st Embodiment, description is abbreviate | omitted and it demonstrates from the following step S28a.

  Next, the change amount calculation unit 51e stores the ROI number of the ROI to be processed and each change amount in the ROI in the table of the storage unit 52 (step S28a).

  Next, the coefficient calculation unit 51f stores the coefficient for obtaining the change index calculated in step S26, together with the ROI number, in a table in the storage unit 52 (step S28b).

  The above is the flow of change index calculation processing for each ROI in this modification.

[Example of ROI number and change amount table]
Next, an example of a table that stores the ROI number of the ROI that is the processing target described above and each change amount in the ROI will be described. FIG. 14 is an example of a table storing the ROI number of the ROI to be processed and each change amount in the ROI.

  In this example, the ROI number that uniquely identifies the ROI and the amount of change in the ROI classified for each type of change are stored in one row of the table.

  The example of the table storing the ROI number of the ROI to be processed and each change amount in the ROI has been described above.

[Create and display digest]
Next, the flow of digest creation and display processing by the control unit 51 ′ will be described. FIG. 15 is a flowchart for explaining the flow of digest creation and display processing.

  First, the control unit 51 ′ acquires information regarding the length of the playback time, the threshold that affects the playback time, and the designation of the ROI for digest playback (step S41).

  Next, the control unit 51 ′ reads the ROI number and information on the amount of change in the ROI from the ROI number and change amount table in the storage unit 52 (step S42a).

  Next, the control unit 51 ′ reads the ROI number and the coefficient for obtaining the change index in the ROI from the table of the ROI number and change index coefficient in the storage unit 52 (step S42b).

  Next, the change index calculation unit 51g calculates a change index of the ROI that is the processing target (step S27). The calculation method is as described above.

  Since the processing from the next step S43 to step S47 is the same as that of the first embodiment, description thereof is omitted.

  The flow of digest creation and display processing in this modification has been described above.

  In the above, the modification which calculates a change parameter | index at the time of digest preparation was demonstrated.

[Additional notes]
In addition, the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the scope of the present technology.

[Another configuration of this technology]
In addition, this technique can also take the following structures.
(1)
Acquire multiple images obtained by time-lapse observation of living cells using a microscope,
Analyzing the feature amount of each acquired image,
Based on the analysis result, calculate an index of the degree of change of the feature amount between the images that are temporally continuous,
Select an image where the index exceeds a predetermined threshold,
An image processing apparatus comprising: a control unit configured to display the selected image on a display unit in time series.
(2)
The image processing apparatus according to (1) above,
The controller is
Set the ROI area in the acquired image,
Analyzing the feature quantity of the set ROI region;
An image processing apparatus that calculates the index for each ROI region.
(3)
The image processing apparatus according to (2),
The controller is
An image processing apparatus that selects an image in which the calculated index exceeds the threshold value for at least one ROI region.
(4)
The image processing apparatus according to (2),
The controller is
Accepting designation of the ROI region from the user via the input unit,
An image processing apparatus that selects an image in which the index of the designated ROI region exceeds the threshold value.
(5)
The image processing apparatus according to any one of (2) to (4),
The controller is
An image processing apparatus for enlarging and displaying a portion of the ROI area in each of the images.
(6)
The image processing apparatus according to any one of (1) to (5),
The controller is
An image processing apparatus that sets the threshold value based on an input received via an input unit.
(7)
The image processing apparatus according to any one of (1) to (5),
The controller is
An image processing apparatus that sets the threshold based on the amount of change in the feature amount.
(8)
The image processing apparatus according to any one of (1) to (7),
The controller is
When playing back the plurality of images as a movie,
An image processing apparatus that adjusts the threshold value so that the image corresponding to the reproduction time received via the input unit is selected.
(9)
The image processing apparatus according to any one of (1) to (8),
The image processing apparatus, wherein the feature amount is at least one of luminance, edge, motion vector, and density.
(10)
The controller is
An image processing apparatus that calculates the index using the following expression, using a change amount of the feature amount, a normalized change amount obtained by normalizing the change amount, and a sum of the normalized change amounts.
(Index) = Σ ((change amount of the feature amount) × (normalized change amount of the feature amount) / (total normalized change amount))
(11)
The control unit
Acquire multiple images obtained by time-lapse observation of living cells using a microscope,
Analyzing the feature amount of each acquired image,
Based on the analysis result, calculate an index of the degree of change of the feature amount between the images that are temporally continuous,
Select an image where the index exceeds a predetermined threshold,
An image processing method for displaying the selected image on a display unit in time series.

DESCRIPTION OF SYMBOLS 1 ... Microscope system 10 ... Laser light source unit 11 ... Laser controller 20 ... Scanner unit 21 ... Scanner controller 22 ... Galvano mirror 23 ... Dichroic mirror 24 ... Photo detector 30 ... Microscope 31 ... Filter 32 ... Objective lens 33 ... Sample 34 ... Stage Driving unit 35 ... Stage 36 ... Imaging lens 37 ... Imaging unit 40 ... Microscope controller 50 ... Image processing device 51 ... Control unit 52 ... Storage unit 52a ... ROI information table 52b ... Image analysis result 52c ... ROI number / change index table 52d ... Movie (frame)
53 ... Input section 54 ... Display section

Claims (11)

Acquire multiple images obtained by time-lapse observation of living cells using a microscope,
Analyzing a plurality of features for each acquired image,
Based on the analysis result, an index of the degree of change of the plurality of feature amounts between the images that are temporally continuous is calculated using the amount of change of the plurality of feature amounts,
Select an image where the index exceeds a predetermined threshold,
An image processing apparatus comprising: a control unit configured to display the selected image on a display unit in time series. The image processing apparatus according to claim 1,
The controller is
Set the ROI area in the acquired image,
Analyzing the plurality of feature quantities of the set ROI region;
An image processing apparatus that calculates the index for each ROI region. The image processing apparatus according to claim 2,
The controller is
An image processing apparatus that selects an image in which the calculated index exceeds the threshold value for at least one ROI region. The image processing apparatus according to claim 2,
The controller is
Accepting designation of the ROI region from the user via the input unit,
An image processing apparatus that selects an image in which the index of the designated ROI region exceeds the threshold value. The image processing apparatus according to any one of claims 2 to 4,
The controller is
An image processing apparatus for enlarging and displaying a portion of the ROI area in each of the images. An image processing apparatus according to any one of claims 1 to 5,
The controller is
An image processing apparatus that sets the threshold value based on an input received via an input unit. An image processing apparatus according to any one of claims 1 to 5,
The controller is
An image processing apparatus that sets the threshold based on the change amount of each of the plurality of feature amounts. The image processing apparatus according to any one of claims 1 to 7,
The controller is
When playing back the plurality of images as a movie,
An image processing apparatus that adjusts the threshold value so that the image corresponding to the reproduction time received via the input unit is selected. The image processing apparatus according to any one of claims 1 to 8,
The image processing apparatus, wherein the plurality of feature amounts are at least two of luminance, edge, motion vector, and density. The image processing apparatus according to claim 9,
The controller is
The index is calculated by the following formula using the change amount of each of the plurality of feature amounts, the normalized change amount obtained by normalizing the change amount, and the sum of the normalized change amounts. apparatus.
(Indicator) = Σ ((Change amount) × (Normalized change amount) / (Total normalized change amount)) The control unit
Acquire multiple images obtained by time-lapse observation of living cells using a microscope,
Analyzing a plurality of features for each acquired image,
Based on the analysis result, an index of the degree of change of the plurality of feature amounts between the images that are temporally continuous is calculated using the amount of change of the plurality of feature amounts,
Select an image where the index exceeds a predetermined threshold,
An image processing method for displaying the selected image on a display unit in time series.

Images