JP6821153B2 - Living tissue image analysis system, image processing system and program - Google Patents

Description

The present invention relates to a biological tissue image analysis system or the like that executes a predetermined image processing to analyze an image of a biological tissue.

In recent years, a system called CAD (Computer Aided Diagnosis) has been developed that supports diagnosis such as pointing out or identifying a lesion site or evaluating a medical condition by acquiring an image taken by MRI or CT and analyzing the image. ing.

Such a CAD system quantifies the pathological condition in a biological tissue based on the difference from the normal time at the site to be inspected by comparing the image of the biological tissue such as a reference organ with the image of the biological tissue to be inspected. It has a structure to be changed.

Then, such a CAD system makes a diagnosis after correcting the difference in the organ shape (that is, the living tissue) due to individual differences or the change in the organ shape due to heartbeat or respiration when quantifying the pathological condition of the living tissue. Is required to do.

For example, when making a diagnosis based on a brain MRI image, a CAD system uses statistical processing such as SPM (Statistical Parametric Mapping) to perform a standard brain (hereinafter referred to as "standard brain") and a patient's brain. (Hereinafter referred to as "patient brain"), shape matching is performed.

In particular, when performing shape matching between soft and deformable objects such as organs, CAD systems include non-rigid registration, FFD (Free-Form Deformation), TPS (Thin Plate Spline) or Diffusion Model. Processing is used.

In addition, when making a diagnosis based on CT images in living tissues such as lungs and abdomen whose shape changes during biological activity, the CAD system is designed to perform shape matching using image processing. The change in the shape of the living tissue is corrected by the image processing, the change in the shape of the living tissue is corrected, and then the shape matching is executed.

Then, in order to perform shape matching, the positions of each part in the living tissue must be aligned, and conventionally, in such image processing, considering that the subject is a non-rigid body, it is necessary to align the positions. There is known a system that performs alignment including correction of deformation between images based on the acquired image information of living tissue (for example, Patent Document 1).

JP-A-2010-119654

However, in Patent Document 1, although the target biological tissue is imaged by executing the image processing and the deformation correction when the biological tissue is deformed is executed as the image processing, each of the biological tissues in the biological tissue. Changes in time series during biological activity in cells cannot be sufficiently corrected for analysis as image processing.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to accurately acquire movements during biological activity in a living tissue as change information, and as a result, external factors. To provide a biological tissue image analysis system or the like capable of analyzing time-series changes in biological tissue during biological activity in the state of biological tissue based on the above or the state of biological tissue based on comparison with normal biological tissue. It is in.

In order to solve the above-mentioned problems, the present invention executes an acquisition means for acquiring a plurality of biological tissue images in which biological tissues in activity are imaged in time series as biological tissue image data, and predetermined image processing. Thereby, the shape of the biological tissue imaged in the biological tissue image of each acquired biological tissue image data is shown based on the reference shape showing the shape of the biological tissue imaged in advance as the reference image. A shape aligning means for aligning the position on the image of a given element constituting the target shape with the position on the image of the given element constituting the corresponding reference shape and the reference shape of the reference image are configured. Specific means for specifying the respective coordinate positions on each biological tissue image with respect to the aligned components indicating the components constituting the target shape aligned with a given element, and the specified positions. Based on the respective coordinate positions of the combined components on each biological tissue image, the extraction means for extracting the change information regarding the change of the biological tissue during the biological activity and the change information of the extracted biological tissue are output. Output control means and
It has a configuration including.

With this configuration, the present invention has a certain pattern of a plurality of types of fixed cells (muscles, skin, organs, etc.) during biological activity associated with the movement of the living body including beating and breathing in the acquired biological tissue image data. By aligning the shape (including a part of the shape (part) of the living tissue) of the tissue having the structure assembled in the above with the shape imaged in the reference image, the time series in the specified shape can be obtained. Changes in living tissue such as the amount of fluctuation can be extracted.

Therefore, the present invention includes time-series shape changes and local sites in a target biological tissue, such as the state of biological tissue based on external factors or the state of biological tissue based on comparison with normal biological tissue. The movement during biological activity in the biological tissue, such as the amount of expansion and contraction of the biological tissue, can be accurately extracted as change information.

As a result, the present invention relates to a biological tissue during biological activity in the state of the biological tissue based on an external factor such as when a drug is administered to the biological tissue or the state of the target biological tissue based on a comparison with a normal biological tissue. Time-series changes can be analyzed based on objective indicators and information such as quantified information, which dramatically improves the speed and accuracy of promoting new drug development and identifying pathological conditions. Can be made to.

The present invention presents a time series of living tissue during biological activity in the state of living tissue based on external factors such as when a drug is administered to living tissue or the state of target living tissue based on comparison with normal living tissue. Since changes can be analyzed, it is possible to dramatically improve the speed and accuracy of promoting new drug development and identifying pathological conditions.

It is a block diagram which shows the structure of one Embodiment of the image processing system for bioanalysis which concerns on this invention. It is a block diagram which shows the structure of the image processing apparatus of the image processing system for biological analysis of one Embodiment. The figure for demonstrating the image-to-image alignment processing, shape alignment processing, coordinate position identification processing, extraction processing and output processing included in the change information extraction processing of an ecological organization in the data processing unit of the image processing apparatus of one embodiment ( The 1). The figure for demonstrating the image-to-image alignment processing, shape alignment processing, coordinate position identification processing, extraction processing and output processing included in the change information extraction processing of an ecological organization in the data processing unit of the image processing apparatus of one embodiment ( The second). The figure for demonstrating the image-to-image alignment processing, shape alignment processing, coordinate position identification processing, extraction processing and output processing included in the change information extraction processing of an ecological organization in the data processing unit of the image processing apparatus of one embodiment ( The third). In one embodiment, it is a figure (No. 1) showing an example of an image of a living tissue imaged by a change information extraction process. In one embodiment, it is a figure (No. 2) showing an example of an image of a living tissue imaged by a change information extraction process. It is a figure which shows other example of the image of the living tissue imaged by the change information extraction processing in one embodiment. It is a flowchart which shows the operation of the change information extraction processing executed in the image processing apparatus of one Embodiment.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In addition, the following embodiment is the present invention for an image processing system for bioanalysis that analyzes an image of a living tissue during biological activity (hereinafter referred to as "biological tissue image") imaged by a multiphoton microscope. This is an embodiment when the biological tissue image analysis system, the program and the image processing system according to the above are applied. However, the present invention is not limited to the following embodiments within the scope of including the technical idea thereof.

[1] Outline of the image processing system for bioanalysis First, the configuration and outline of the image processing system 1 for bioanalysis in the present embodiment will be described with reference to FIG. Note that FIG. 1 is a system configuration diagram showing the configuration of the image processing system 1 for bioanalysis in the present embodiment.

The image processing system 1 for biological analysis of the present embodiment images the biological tissue during biological activity as a biological tissue image in time series, and image data of the plurality of imaged biological tissues (hereinafter, "biological tissue image data"). This is a system that executes predetermined image processing on () and acquires the movement of living tissue during biological activity as change information.

Usually, a biological tissue having a structure in which multiple types of fixed cells (cell clusters) such as muscles, skin, or organs during biological activities associated with the movements of the living body including beating and respiration are assembled in a certain pattern is The object to be imaged is a very small size (about several hundred μm).

Therefore, even if the imaging device is fixed or the living tissue to be the object is fixed, the angle of view of the imaging device that becomes the field of view is obtained by moving the predetermined region to be observed (that is, the imaging target region). In many cases, the imaging target region (living tissue) is removed from the inside, and it is difficult to continuously and appropriately observe the time-series changes in the imaging target region of the observation target.

Further, even in such an imaging target region, the shape constantly changes (deforms) during ecological activity, so even when the angle of view position (that is, the field of view) of the imaging target region is fixed, the image is concerned. It is often difficult to accurately track the imaging target area in chronological order.

Therefore, the image processing system 1 for bioanalysis of the present embodiment fixes the field of view of the angle of view position of the imaging target region, which is a biological tissue that is undergoing biological activity in terms of image processing, and has a shape that serves as a reference for the imaging target region. It has a configuration in which changes in the time series in the imaging target region are accurately tracked using (hereinafter, referred to as “reference shape”), and change information indicating the changes in the time series is extracted.

In particular, the image processing system 1 for biological analysis of the present embodiment changes the movement of the biological tissue during biological activity, such as a time-series shape change in the biological tissue and the amount of expansion and contraction of the biological tissue including a local site. For example, it has a configuration capable of quantifying as change information according to the amount of fluctuation and outputting the change information.

Specifically, the image processing system 1 for biological analysis of the present embodiment generates image data of a biological tissue image obtained by imaging a biological tissue to be analyzed in a time series as biological tissue image data. An image processing device 10, a network 20, an image processing device 30 that executes predetermined image processing on a plurality of biological tissue image data obtained in time series to acquire and present change information related to the biological tissue. Consists of.

The imaging device 10 is for observing a living tissue, for example, a communication function for connecting to a network 20 and exchanging data, an imaging function capable of acquiring a predetermined image such as a time-lapse image in time series, and a living tissue. It is a device having a microscope function.

In particular, the imaging device 10 uses an imaging function to image the biological tissue to be analyzed at regular intervals (for example, every 1/30 second), and generates biological tissue image data as a time-lapse image of the biological tissue image. It has a configuration in which the generated biological tissue image data is transmitted to the image processing device 30 together with the time information indicating the imaging time by the communication function.

For example, when transmitting the biological tissue image data to the image processing device 30, the imaging device 10 uses a predetermined communication standard such as LAN (LAN: Local Area Network) to directly or by wire an access point (not shown). It has a configuration in which it is wirelessly transmitted to the image processing device 30 via.

Further, the image pickup apparatus 10 has various configurations as a multiphoton microscope as a microscope function.

For example, the image sensor 10 includes an optical system as a multiphoton microscope, a CCDI sensor (Charge Coupled Device Image Sensor) that converts an optical image input from the optical system into an electric signal, and an electric signal generated by the CCDI sensor. It has a generation unit that generates image data based on the above.

Then, as a microscope function executed as, for example, a multiphoton microscope, the image pickup apparatus 10 performs analysis on the stage in order to keep the distance from the lens set in the optical system constant when performing image processing. It has a configuration using a stabilizer that limits movement in the Z-axis direction (specifically, the first axis from the stage to the lens) during the biological activity of interest.

Further, the image pickup apparatus 10 moves the living tissue in the X-axis (second axis perpendicular to the Z-axis direction) direction and the Y-axis (third axis perpendicular to the Z-axis and X-axis) directions on the stage. It has a configuration in which a plurality of biological tissue images are generated in time series as two-dimensional images, and the generated biological tissue images are output to the image processing device 30 as biological tissue image data.

In the following description, unless otherwise specified, the image pickup apparatus 10 will be described as providing the image processing apparatus 30 with an image of a biological tissue during biological activity in a state in which movement in the Z-axis direction is restricted. ..

The network 20 is composed of, for example, a wired or wireless IP (Internet Protocol) network 20, a public telephone network including a mobile phone network, or another network 20.

The image processing device 30 is a device capable of executing a predetermined arithmetic process or application such as a personal computer, and executes a predetermined image process on each biological tissue image data in conjunction with the imaging device 10. It is a device that acquires and presents information about the living tissue.

In particular, the image processing device 30 executes predetermined image processing on each of the plurality of biological tissue image data generated in time series in the image pickup device 10, and sets the target of the image processing (that is, the image pickup target region). While fixing the visual field position (image angle position) of the living tissue (hereinafter referred to as "target living tissue"), the shape of each target living tissue is specified, and the shape of each specified target living tissue is observed. It has a structure for extracting time-series changes in living tissue as change information.

Then, the image processing apparatus 30 is performing biological activity in the biological tissue such as a time-series shape change in the biological tissue and the amount of expansion and contraction of the biological tissue including a local site as time-series change information regarding the target biological tissue. It has a configuration for executing a process of extracting movement (that is, fluctuation) as information (hereinafter referred to as "change information extraction process of ecological organization").

Specifically, the image processing device 30 is used as a change information extraction process for the ecological organization.
(1) Acquire a plurality of biological tissue image data in chronological order from the imaging device 10 and obtain them.
(2) By executing a predetermined image process, each acquired biological tissue image data is imaged into a biological tissue image based on the shape of the biological tissue imaged in advance as a reference image (that is, the reference shape). The position on the image of the element (for example, pixel) constituting the shape of the living tissue (hereinafter referred to as "target shape") is the position on the image of the corresponding element forming the reference shape. According to
(3) Specify the respective coordinate positions on each biological tissue image with respect to the element constituting the target shape aligned with the element constituting the reference shape of the reference image (hereinafter, “aligned component”). And
(4) Based on the respective coordinate positions on each biological tissue image of the identified aligned component, change information regarding changes in biological tissue during biological activity is extracted.
(5) It has a configuration for executing various processes for outputting change information of the extracted biological tissue.

Further, in the change information extraction process of the ecological tissue, the image processing device 30 executes a predetermined image process on the biological tissue image of each acquired biological tissue image data in addition to each of the above processes. To align each element (eg, pixel) in each biotissue image between each biotissue image.
(7) Align each element in the biological tissue in each biological tissue image with the position on the image of the reference shape corresponding to the target shape of each biological tissue in which the alignment is executed.
It has a configuration to execute each process.

In addition, as the process of (4), the image processing apparatus 30 determines the amount of fluctuation in the biological tissue during biological activity based on each coordinate position on each biological tissue image of the specified aligned component during biological activity. It has a structure to be calculated (extracted) as change information regarding changes in living tissue.

With such a configuration, in the image processing system 1 for biological analysis of the present embodiment, the biological tissue (muscle, skin, organ) during biological activity accompanying the movement of the biological body including beating and breathing in the acquired biological tissue image data. Align the shape (including a part of the shape (site) of a living tissue) of a tissue having a structure in which a plurality of types of fixed cells such as, etc. are gathered in a certain pattern with the shape imaged in the reference image. By doing so, it is possible to extract changes in the living tissue such as the amount of time-series fluctuation in the specified shape.

Therefore, the image processing system 1 for biological analysis of the present embodiment has a time-series shape in the target biological tissue, such as the state of the biological tissue based on an external factor or the state of the biological tissue based on the comparison with the normal biological tissue. The movement during biological activity in the biological tissue such as the amount of expansion and contraction of the biological tissue including changes and local parts is accurately used as, for example, quantified (that is, quantified) change information (that is, feature amount). It is possible to extract to.

As a result, in the image processing system 1 for biological analysis of the present embodiment, the state of the biological tissue based on external factors such as when a drug is administered to the biological tissue or the target biological tissue based on the comparison with the normal biological tissue. In this state, time-series changes in biological tissues during biological activities can be analyzed based on accurate information such as objective indicators, so that the speed and accuracy of promoting new drug development and identifying pathological conditions can be analyzed. Can be dramatically improved.

[2] Image Processing Device Next, the configuration of the image processing device 30 of the present embodiment will be described with reference to FIG. Note that FIG. 2 is a block diagram showing a block of the image processing device 30 of the present embodiment.

Specifically, as shown in FIG. 2, the image processing device 30 of the present embodiment is transmitted from the data storage unit 300 that records various data used when various programs are executed and the image pickup device 10. A communication control unit 310 that receives a plurality of time-series biological tissue image data, and a data processing unit 320 that executes image processing on the received biological tissue image data and executes change information extraction processing of the ecological tissue. A display unit 340 composed of a liquid crystal panel or the like, a display control unit 350 for controlling the display unit 340, an operation unit 370, and a management control unit 380 for controlling each unit.

Then, the above-mentioned parts are connected to each other by a bus 31, and data transfer between each component is executed.

For example, the image processing device 30 of the present embodiment constitutes the image analysis device of the present invention, and the data storage unit 300 constitutes the storage means of the present invention. Further, for example, the communication control unit 310 of the present embodiment constitutes the acquisition means of the present invention in conjunction with the data processing unit 320, and the data processing unit 320 is the shape alignment means, the specific means and the extraction means of the present invention. To configure. Further, for example, the display unit 340 of the present embodiment constitutes the output control means of the present invention in conjunction with the data processing unit 320 and the display control unit 350.

The data storage unit 300 is composed of, for example, an HDD (Hard Disk Drive) or the like, and is imaged by the application storage unit 301 and the image pickup device 10 in which an application program for executing each process such as a change information extraction process of an ecological organization is stored. Obtained by the image data storage unit 302 for storing the biological tissue image data generated in the above, the reference data storage unit 303 for storing various data used for various image processing, and the change information extraction process of the ecological tissue. It has an output data storage unit 304 that stores change information about the ecological organization as data, and a ROM / RAM 306 that is used as a work area during program execution.

In particular, the image data storage unit 302 contains ecological organization image data (specifically, frame data) in which a plurality of target ecological organizations output from the imaging device 10 are imaged at predetermined timings (for example, 1/30 second). ) And data on the imaged ecological organization, such as information indicating the time of imaging, are stored in chronological order at predetermined timings.

The reference data storage unit 303 contains image data of a biological tissue image (hereinafter referred to as "reference image") in which a biological tissue that serves as a reference for image processing in the same ecological tissue as the target ecological tissue is imaged, and an image. Data used for various processes such as various parameters used for information extraction processing of changes in ecological tissues including processing are stored.

For example, in the reference data storage unit 303, as a reference image, a biological tissue image in which the shape, size, coordinate position on the image, and the like in the same ecological organization as the target ecological organization are determined and imaged as a reference is used as a reference. It is stored as image data.

The communication control unit 310 is a predetermined network interface, constructs a communication line with the image pickup device 10, and exchanges various data acquired by the image pickup device 10.

The data processing unit 320 is based on an application that executes a change information extraction process of an ecological organization including a predetermined image process recorded in the ROM / RAM 306.
(1) Acquisition of the biological tissue image data such as reception of biological tissue image data output from the imaging device 10 or reading of biological tissue image data stored from the image data storage unit 302 in conjunction with the communication control unit 310. processing,
(2) Each pixel constituting the biological tissue imaged in each biological tissue image between the biological tissue images by executing a predetermined image processing on the biological tissue image of each acquired biological tissue image data (2) Alignment process between images to align pixels),
(3) By executing the predetermined image processing, the target shape of each biological tissue in which the alignment in the image is executed in the predetermined region in each biological tissue image is configured based on the reference shape in the reference image. Shape alignment processing that aligns each pixel with the position on the image of the corresponding reference shape,
(4) For each pixel constituting the target shape aligned with each pixel constituting the reference shape of the reference image (hereinafter, referred to as “aligned constituent pixel”), each pixel on each biological tissue image Coordinate position identification process to specify the coordinate position,
(5) Based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels, change information regarding changes in biological tissue during biological activity is extracted.
(6) In conjunction with the display unit 340 and the display control unit 350, change information of the extracted biological tissue such as display output and storage in the data storage unit 300 (for example, an image obtained by imaging the change in movement and the amount of change thereof). ) Output processing,
Execute each process of.

In particular, the data processing unit 320 has the data management unit 321 that manages the acquisition process and the data storage unit 300 by executing the application of the change information extraction process of the ecological organization, and the image-to-image position that executes the image-to-image alignment process. Alignment control unit 322, shape alignment control unit 323 that executes shape alignment processing, coordinate position identification processing unit 324 that executes coordinate position identification processing, and extraction output processing unit 325 that executes extraction processing and output processing. To realize.

For example, the data management unit 321 of the present embodiment constitutes the acquisition means of the present invention together with the communication control unit 310, and the image-to-image alignment control unit 322 constitutes the image-to-image alignment means of the present invention. Further, for example, the shape alignment control unit 323 of the present embodiment constitutes the shape alignment means of the present invention, and the coordinate position identification processing unit 324 constitutes the identification means of the present invention. Then, for example, the extraction output processing unit 325 of the present embodiment constitutes the extraction means of the present invention, and also constitutes the output control means of the present invention in conjunction with the display control unit 350.

The details of each part of the data processing unit 320 and the details of the change information extraction processing of the ecological organization in the present embodiment will be described later.

The display unit 340 is composed of a panel of a liquid crystal element or an EL (Electro Luminescence) element, and displays a predetermined image based on the display data generated by the display control unit 350.

Under the control of the management control unit 380 and the data processing unit 320, the display control unit 350 generates drawing data necessary for the display unit 340 to draw a predetermined image, and transmits the generated drawing data to the display unit 340. Output.

The operation unit 370 is a large number of keys such as various confirmation buttons, operation buttons for inputting each operation command, and a numeric keypad. The operation unit 370 is composed of a touch sensor provided on the display unit 340 and is used when performing each operation. It is supposed to be.

The management control unit 380 is mainly composed of a central processing unit (CPU), and by executing a program, it performs overall management control of the image processing device 30 and various other controls.

[3] Data processing unit and change information extraction process of ecological organization Next, the details of the data processing unit 320 and the change information extraction process of the ecological organization in the image processing apparatus 30 of the present embodiment will be described with reference to FIGS. 3 to 8. To do. 3 to 5 show the data processing unit 320 of the present embodiment regarding the image-to-image alignment process, shape alignment process, coordinate position identification process, extraction process, and output process included in the change information extraction process of the ecological organization. It is a figure for demonstrating, and FIGS. 6 to 8 are figures which show the image example of the living tissue imaged by the change information extraction processing.

(Data management department)
The data management unit 321 includes a plurality of biological tissue image data output from the image pickup device 10 in chronological order via the communication control unit 310, and a plurality of time-series living organisms stored in other devices or storage devices (not shown). The tissue image data is stored in the image data storage unit 302.

Then, when the data management unit 321 executes the change information extraction process of the ecological organization, the data management unit 321 reads out a plurality of stored biological tissue image data based on the instruction of the operator for the change information extraction process of the ecological tissue. To provide to each department.

Further, the data management unit 321 executes registration of various parameters to the reference data storage unit 303 based on the instruction of the operator.

In particular, the data management unit 321 of the present embodiment transfers a biological tissue that fluctuates in the X-axis direction and the Y-axis direction in which movement in the Z-axis direction is restricted during biological activity from the image pickup device 10 or the like on the X-axis and Y-axis planes. The biological tissue image imaged two-dimensionally by the above is acquired as biological tissue image data.

For example, the data management unit 321 stores in the image data storage unit 302 a biological tissue image in which the biological tissue of the myocardium as illustrated in FIG. 3A is imaged as the biological tissue image data of the target biological tissue image. To do.

Then, the data management unit 321 reads out the biological tissue image data stored in the image data storage unit 302 based on the instruction of the operator when the change information extraction process of the ecological tissue is executed.

The data management unit 321 may directly acquire a plurality of biological tissue image data output from the image pickup apparatus 10 in chronological order via the communication control unit 310.

Further, the plurality of ecological tissue image data stored in the image data storage unit 302 may be two or more data. However, in the present embodiment, it is preferable that about 1800 frames (for example, one frame is 1/30 second and 60 seconds of ecological tissue image) of image data is stored as the biological tissue image data.

(Image alignment control unit)
When executing the change information extraction process of the ecological tissue, the image-to-image alignment control unit 322 executes image processing such as rigid registration, non-rigid registration, affine transformation, homography transformation, or pattern matching. In each of the acquired biological tissue image data, an image-to-image alignment process for aligning the position in the image of the target biological tissue imaged with each biological tissue image between the biological tissue images is executed.

Specifically, the inter-image alignment control unit 322 unifies the coordinate positions in the image of each pixel constituting the target biological tissue imaged in each biological tissue image between the biological tissue images.

That is, the inter-image alignment control unit 322 uses the coordinate system (X coordinate and Y coordinate) on the common image in which the angle of view is fixed in the image processing in the biological tissue image, and is used between the biological tissue images. The coordinate position of each corresponding pixel (each pixel having the same feature) in each biological tissue image is set so that the target biological tissue imaged in each biological tissue image is arranged within a predetermined coordinate position or coordinate range. , Perform coordinate conversion to unify between each biological tissue image.

Specifically, the inter-image alignment control unit 322 identifies and corresponds to each corresponding pixel between the biological tissue images of the acquired biological tissue image data by executing rigid body registration as image processing. The coordinate position of each pixel is converted into a predetermined coordinate or coordinate range in a common predetermined image (hereinafter, referred to as “field image”).

For example, when the myocardium is a living tissue, the image-to-image alignment process is executed on the living tissue image of the living tissue image data shown in FIG. 3 (A), and the upper left on the living tissue image (in the drawing). It is assumed that the myocardium is imaged as a living tissue in the upper left). Further, it is assumed that the image feature points of the target shape are arranged at the center coordinates in the field image as the predetermined coordinates or coordinate range after the image conversion.

In this case, as shown in FIG. 3B, the inter-image alignment control unit 322 identifies the pixels of the image feature points of the biological tissue imaged in the acquired biological tissue image, and the identified biological tissue. The pixel (representative pixel) of any of the image feature points of is converted into the center coordinates of the field image, and the image feature points other than the representative pixel and the component pixels constituting the other biological tissue are converted on the biological tissue image. Converts to pixels on the corresponding field image based on the distance to the representative point.

At this time, the inter-image alignment control unit 322 determines the pixels of the edge portion (or the portion having the high edge strength among the edge portions) in the edge detection, or the color feature amount such as the brightness value with respect to the biological tissue image. Pixels having one value after binarizing with the threshold value of are specified as image feature points, and image feature points that are assumed to be arranged near the center from the specified image feature points are set as representative pixels. To do.

The inter-image alignment control unit 322 is a time-series of the reference image stored in the reference data storage unit 303 or a plurality of acquired biological tissue image data as predetermined coordinates or coordinate ranges of the field image. The biological tissue image of the first biological tissue image data (hereinafter, also referred to as “reference image”) may be used.

Further, the image-to-image alignment control unit 322 identifies a region of the biological tissue in the target biological tissue image visually by the operator, and an image arranged near the center of the region from among the image feature points in the region. A feature point is set as a representative pixel. However, the inter-image alignment control unit 322 may manually specify the representative pixel including each image feature point based on the instruction of the operator, or the alignment itself may be manually performed based on the instruction of the operator. You may do it.

Further, the inter-image alignment control unit 322 is an operation unit for the operator with respect to the biological tissue image in each biological tissue image data acquired such as the center coordinates for arranging on the field image and the arrangement position of each component pixel. Based on the operation to 370, each living tissue may be manually arranged on the visual field image.

(Shape alignment control unit)
When the shape alignment control unit 323 executes the change information extraction process of the ecological tissue, for example, a non-rigid registration (specifically, FFD: Free Form Deformation or TPS-RPM: Thin Plate Spline-Robust Point Matching) ), Etc., and based on the reference shape of the predetermined reference image, each biological tissue in which the image-to-image alignment processing in each biological tissue image is executed in the acquired plurality of biological tissue image data. The shape alignment process for aligning the target shape of the above with the position on the image of the corresponding reference shape is executed.

Specifically, the shape alignment control unit 323 uses the non-rigid registration to display the reference shape of the biological tissue imaged in the reference image stored in the reference data storage unit 303, or a plurality of acquired biological tissue image data. Of these, the shape of the biological tissue imaged in the biological tissue image (hereinafter, referred to as "first biological tissue image") of the biological tissue image data acquired first (hereinafter, also referred to as "reference shape" in this case). Each pixel constituting the target shape of the biological tissue imaged in each target biological tissue image corresponding to each constituent pixel is specified.

For example, when the shape alignment control unit 323 uses a reference image stored in the reference data storage unit 303 as shown in FIG. 4A when the living tissue is a myocardium, the shape alignment control unit 323 is a non-rigid resist. By executing the ration, as shown in FIG. 4 (B), each pixel constituting the reference shape in the biological tissue imaged in the reference image and the corresponding pixel, each acquired biological tissue. Each pixel constituting the biological tissue imaged in the biological tissue image of the image data is specified.

Further, for example, when the shape alignment control unit 323 uses the first biological tissue image when the biological tissue is a myocardium, the shape alignment control unit 323 similarly performs non-rigid registration to obtain an image on the reference image. Each pixel constituting the reference shape in the living body tissue to be formed and each pixel constituting the living body tissue imaged in the living body tissue image of each acquired living body tissue image data, which is the corresponding pixel, are specified.

Even if the shape alignment control unit 323 executes the shape alignment process by rigid body registration for the deformed part such as projective transformation (homography conversion) for each part of the biological tissue instead of the non-rigid body registration. Good.

(Coordinate position identification processing unit)
The coordinate position specifying processing unit 324 is for each pixel (that is, the aligned constituent pixel) that constitutes the target shape imaged in each biological tissue image that is aligned with each pixel that constitutes the reference shape of the reference image. Then, the coordinate position specifying process for specifying each coordinate position on each biological tissue image is executed.

That is, the coordinate position identification processing unit 324 returns each of the aligned constituent pixels constituting the target shape imaged in each biological tissue image aligned by the shape alignment process to the original coordinate system, and returns the original coordinate system. The coordinate position in the coordinate system of is specified and set to the coordinate position.

For example, when the living tissue is the myocardium, the coordinate position specifying processing unit 324 sets each object before the alignment process shown in FIG. 3B is executed (that is, when the image-to-image alignment process is performed). The original coordinate position of each pixel in the living tissue is stored in advance, and as shown in FIG. 5A, the coordinate position of each pixel when the state alignment process is executed is returned to the stored coordinate position. ..

The coordinate position specifying processing unit 324 calculates the amount of displacement (distance from the coordinates on each target biological tissue image as a visual field image to the coordinates on the reference image) at the time of coordinate conversion when the shape alignment process is executed. In this case, the coordinate position in the original coordinate system may be specified by performing the inverse conversion in the coordinate conversion, and each aligned constituent pixel may be set in the coordinate position.

(Extraction output processing unit)
The extraction output processing unit 325 is an image in which changes in living tissue during biological activity are imaged based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels (hereinafter, “biological tissue change image”). ”) Or, the amount of fluctuation indicating the change in the biological tissue during the biological activity is extracted as change information.

Specifically, the extraction output processing unit 325 extracts (that is, tracks) the biological tissue imaged on the biological tissue image during biological activity or the time-series change of each element constituting the biological tissue. , A biological tissue change image (for example, a time-lapse image or a moving image) for continuously displaying the coordinate positions of the aligned constituent pixels on each biological tissue image in the order in which the biological tissue image data was acquired during the biological activity. Generate.

Further, the extraction output processing unit 325 compares the coordinate positions in the corresponding (that is, as the same element) aligned constituent pixels on each biological tissue image, or the aligned configuration on each biological tissue image. The distance between the feature points (for example, end points) of each aligned constituent pixel in the living tissue, and the maximum displacement of each aligned constituent pixel, as compared with the constituent pixels of the biological tissue imaged in the reference image corresponding to the pixel. The amount and the minimum displacement amount, or the average value of the displacement amounts of the aligned constituent pixels, etc., are quantified and the quantified fluctuation amount of each biological tissue is calculated.

That is, the extraction output processing unit 325 extracts as change information expressing the change of the biological tissue by generating a biological tissue change image or by calculating the amount of fluctuation of each biological tissue.

For example, when the living tissue is the myocardium, the extraction output processing unit 325 sets the coordinate position (uppermost end point) of the feature point pixel in the target shape as the target living tissue and the said, as shown in FIG. 5 (B). The coordinate position (lowermost end point) of each pixel on the biological tissue image corresponding to the reference shape pixel and the amount of displacement are calculated.

Then, the extraction output processing unit 325 calculates the time-series variation amount in the target biological tissue by comparing the displacement amount in each of the calculated biological tissue images with the time-series.

When the extraction output processing unit 325 calculates the displacement amount of each pixel in the arithmetic processing for executing the coordinate conversion of each pixel such as non-rigid registration when the shape alignment processing is executed. , The amount of change in the living tissue during biological activity may be calculated (extracted) using the amount of displacement in each of the referenced pixels.

On the other hand, the extraction output processing unit 325 interlocks with the display control unit 350 to display and output the generated time-lapse image, moving image, or the calculated fluctuation amount of the target biological tissue such as display output and storage in the data storage unit 300. And execute the output processing to be stored.

(Example of image of living tissue extracted and processed for change information)
When the original biological tissue image as shown in FIG. 6A is read out, the image of the target biological tissue imaged in each biological tissue image between the biological tissue images by the image-to-image alignment control unit 322. It is assumed that the image-to-image alignment process for aligning the internal positions is executed.

In this case, the extraction output processing unit 325 interlocks with the display control unit 350, and the display unit 340 displays each target biological tissue image (that is, the target biological tissue image) in which the image-to-image alignment process as shown in FIG. 6B is completed. The field image) is displayed.

In addition, by executing the coordinate position identification process for each target biological tissue image for which the image-to-image alignment process and the shape alignment process have been completed, the biological tissue (that is, noise) that reflects only the movement during the biological activity is generated. When a plurality of biological tissue images of each time series of the removed biological tissue) are acquired, the extraction output processing unit 325 is linked with the display control unit 350, and is displayed on the display unit 340 in FIG. 6 (C). Each biological tissue image from which noise has been removed is displayed for the movement during biological activity as shown.

Note that FIG. 6D shows an example of a reference image used for the shape alignment process. That is, the shape alignment control unit 323 executes the shape alignment process on each target biological tissue image aligned between the images using, for example, the reference image shown in FIG. 6 (D). ..

Further, as shown in FIG. 7A, the extraction output processing unit 325 is linked with the display control unit 350, and grid lines are superimposed on each target biological tissue image that has been subjected to change information extraction processing as change information. The displayed image may be displayed on the display unit 340.

In particular, the extraction output processing unit 325 uses an image in which grid lines are superimposed on the reference image as shown in FIG. 7B as a comparison image, and each target biological tissue having a grid line and undergoing change information extraction processing. It may be displayed on the display unit 340 together with the image.

6 (A) to 6 (D) and 7 (A) and 7 (B) are examples in which a part of the heart in a rat is used as a target biological tissue, and are 512 × 512 pixels (512 × 512 pixels). Images captured by a multiphoton microscope that can be imaged in FIGS. 6 (A) to 6 (D) and images obtained by cutting out the center of the captured image in 352 x 352 pixels (FIGS. 7 (A) and 7 (B)). is there. However, 1 pixel is 1 μm.

In addition, each of FIGS. 6 (A) to 6 (D) and FIGS. 7 (A) and 7 (B) shows the case where the intracellular mitochondria are labeled by adding tetramethylrhodamine ethyl ester, which is a fluorescent dye. The tissue in the red part indicates that it is active, and the tissue in the black part is the part that is not recognized or is dead.

Further, the conditions at the time of imaging in each of FIGS. 6 (A) to 6 (D) and 7 (A) and 7 (B) are the high-speed multiphoton confocal microscope A1R MP +.
(Nikon), using a confocal laser Chameleon Vision II (Coherent), excitation wavelength 800-880 nm, objective lens 25x water immersion objective lens (CFI Apo 25 × W, Nikon) ), The band pass filter is 629/53 nm.

On the other hand, FIGS. 8A and 8B are image examples when a part of the heart in a rat expressing green fluorescent protein, which is a fluorescent protein, is used as a target biological tissue. In particular, for each condition at the time of imaging in FIGS. 8A and 8B, a confocal laser Chameleon Vision II (manufactured by Coherent) was used with a high-speed multiphoton confocal microscope A1R MP + (manufactured by Nikon). The excitation wavelength is 800-880 nm, the objective lens is a 25x water immersion objective lens (CFI Apo 25 × W, manufactured by Nikon Corporation), and the bandpass filter is 525/50 nm.

In this case, as shown in FIG. 8A, the extraction output processing unit 325 executes the coordinate position identification process for each target biological tissue image for which the image-to-image alignment process and the shape alignment process have been completed. By doing so, when a plurality of biological tissue images of each time series of the biological tissue (that is, the biological tissue from which noise has been removed) reflecting only the movement during the biological activity are acquired, the display unit is linked with the display control unit 350 to obtain the display unit. 340 is displayed with each biological tissue image from which noise has been removed for movement during biological activity as shown in FIG. 8 (A).

Further, as shown in FIG. 8B, the extraction output processing unit 325 interlocks with the display control unit 350, and as change information, each element (cell) is obtained for each target biological tissue image that has been subjected to change information extraction processing. ) Is visualized by the deformation of the grid lines on the display unit 340.

[4] Eco-organization change information extraction processing Next, the details of the eco-organization change information extraction processing executed in the image processing apparatus 30 of the present embodiment will be described with reference to FIG. Note that FIG. 9 is a flowchart showing the operation of the change information extraction process of the ecological organization executed by the data processing unit 320 of the present embodiment.

This operation uses the operation of executing the change information extraction process for the target biological tissue imaged in the 1800 frame biological tissue image imaged and imaged every 1/30 second for 60 seconds in chronological order. I will explain.

Further, in this operation, the change information extraction process for the target biological tissue is executed using the reference image data stored in the image data storage unit 302, and as the image process, the image-to-image alignment process is executed by rigid body registration. A case where the shape alignment process is executed by non-rigid registration will be described.

First, when the data management unit 321 detects the start of the change information extraction process including the selection of the target biological tissue image based on the operation of the operation unit 370 by the operator (step S101), the image data storage unit 302 The plurality of corresponding biological tissue image data stored in the above are read out and acquired (step S102).

Next, the inter-image alignment control unit 322 images each of the acquired biological tissue image data into each biological tissue image between the biological tissue images using a coordinate system on a common field image. In order to arrange the target biological tissue at a predetermined coordinate position (or within the coordinate range), a coordinate transformation for setting (converting) the coordinate position in each pixel of the biological tissue image is executed (inter-image alignment process (step S103). )).

For example, the inter-image alignment control unit 322 converts the pixels indicating the center of gravity of the biological tissue imaged in each biological tissue image into the center coordinates of the visual field image, and is based on the coordinates of the image of the center of gravity on the visual field image. Then, a coordinate transformation is performed to transform the coordinate positions of the biological tissue image in other pixels.

Next, the shape alignment control unit 323 corresponds to each target biological tissue corresponding to each pixel constituting the reference shape of the biological tissue imaged in the reference image or the reference shape imaged in the first acquired biological tissue image. Each pixel constituting the target shape of the living tissue imaged in the image is specified (shape alignment process (step S104)).

At this time, the shape alignment control unit 323 identifies the coordinate positions on the original biological tissue image (field image) of each pixel and stores them in the ROM / RAM 306, respectively. The shape alignment control unit 323 calculates the displacement amount indicating the distance from the coordinate position on the original biological tissue image (field image) to the coordinate position on the reference image in each pixel, and for each of the calculated pixels. The displacement amount may be stored in the ROM / RAM 306.

Next, the coordinate position specifying processing unit 324 sets each pixel (that is, the aligned constituent pixel) that constitutes the target shape imaged in each biological tissue image aligned by the shape alignment process to the original biological tissue. It returns to the coordinate system of the image, specifies the coordinate position in the original coordinate system, and sets each pixel at the coordinate position (coordinate position identification process (step S105)).

Next, the extraction output processing unit 325 uses a biological tissue change image (time lapse image) that images changes in the biological tissue during biological activity based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels. And moving images), or the amount of fluctuation indicating changes in the biological tissue during the biological activity is extracted as change information (extraction process (step S106)).

Next, the extraction output processing unit 325 interlocks with the display control unit 350 to execute an output process for displaying the extracted change information on the display unit 340 (step S107), and ends this operation.

The extraction output processing unit 325 may store the calculated fluctuation amount in the biological tissue during biological activity in the data storage unit 300 as output processing.

[5] Modification example [5.1] Modification example 1
In the above embodiment, the image pickup apparatus 10 and the image processing apparatus 30 may be installed or used in the same room or in the same site, and each of them may be installed or used in a remote place such as a foreign country. Each of the above processes may be executed.

Further, the image processing device 30 may execute the change information extraction process of the ecological organization using a database (not shown) connected via the network 20, or may be formed by one or more devices. When a database is used, the database is responsible for a part of the functions of the data storage unit 300 (recording of target biological tissue image data).

[5.2] Modification 2
In the above embodiment, the image processing system for bioanalysis is realized by the image pickup device 10 and the image processing device 30, but it may be realized by a stand-alone type by providing an image data generation unit having a scanner and an image pickup function. ..

[5.3] Modification 3
In the above embodiment, the image-to-image alignment control unit 322 is executed in advance as an image-to-image alignment process when manually acquiring a plurality of time-series biological tissue image data, and the image-to-image alignment process has already been performed. The change information extraction process of the ecological tissue may be executed on the executed target biological image. That is, in some cases, the image-to-image alignment control unit 322 may omit the image-to-image alignment process in the change information extraction process of the ecological organization.

1 ... Image processing system for bioanalysis 10 ... Imaging device 20 ... Network 30 ... Image processing device 300 ... Data storage unit 301 ... Application storage unit 302 ... Image data storage unit 303 ... Reference data storage unit 320 ... Data processing unit 321 ... Data Management unit 322 ... Image-to-image alignment control unit 323 ... Shape alignment control unit 324 ... Coordinate position identification processing unit 325 ... Extraction output processing unit 340 ... Display unit 350 ... Display control unit 370 ... Operation unit 380 ... Management control unit

Claims (8)

An acquisition means for acquiring a plurality of biological tissue images in which biological tissues during biological activities are imaged in time series as biological tissue image data, and
By executing a predetermined image process on the biological tissue image of each acquired biological tissue image data, the pixels other than the pixels of the image feature points of the biological tissue image and the pixels of the image feature points constituting the biological tissue image are obtained. Of the pixels constituting the biological tissue in each biological tissue image between the biological tissue images based on the pixels of the image feature points of the specified biological tissue image and the component pixels of the specified biological tissue image. An image-to-image alignment means that unifies the coordinate positions or coordinate ranges of pixels having the same characteristics,
Based on the reference shape showing the shape of the biological tissue that has been imaged in advance as the reference image by performing a predetermined image processing on each biological tissue image for which the alignment between the biological tissue images has been executed. , Each pixel constituting the target shape indicating the shape of the biological tissue imaged in each biological tissue image in which the alignment between the biological tissue images is executed, and each pixel constituting the reference shape. A shape alignment means for aligning the target shape of the living tissue in association with the
A specific means for specifying each coordinate position on each biological tissue image with respect to the aligned constituent pixels indicating the pixels constituting the target shape in which the target shape of the biological tissue is aligned.
An extraction means for extracting change information regarding changes in the biological tissue during the biological activity based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels .
An output control means that outputs change information of the extracted biological tissue, and
A biological tissue image analysis system characterized by being equipped with. In the biological tissue image analysis system according to claim 1,
The extraction means
As change information regarding changes in the biological tissue during the biological activity, the amount of change in the biological tissue during the biological activity is calculated based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels . Living tissue image analysis system. In the biological tissue image analysis system according to claim 1 or 2 .
A biological tissue image in which a biological tissue image stored in advance in a storage means or a biological tissue image imaged as a reference biological tissue image among the acquired plurality of biological tissue image data is used as the reference image. Analysis system. In the biological tissue image analysis system according to any one of claims 1 to 3 ,
The acquisition means
When acquiring an image of the living tissue placed on the stage, the depth direction with respect to the stage surface is set as the first axis, and the axis along the stage surface and perpendicular to the first axis. It is defined by the axial direction of the second axis, which is the axis, and the axial direction of the third axis, which is an axis along the stage surface and is an axis perpendicular to the first axis and the second axis. The biological tissue image imaged by two dimensions is acquired as the biological tissue image data.
The extraction means
A biological tissue image analysis system that calculates two-dimensional changes in biological tissue during biological activity as the change information. Computer,
An acquisition means for acquiring a plurality of biological tissue images in which biological tissues during biological activities are imaged in time series as biological tissue image data.
By executing a predetermined image process on the biological tissue image of each acquired biological tissue image data, the pixels other than the pixels of the image feature points of the biological tissue image and the pixels of the image feature points constituting the biological tissue image are obtained. Of the pixels constituting the biological tissue in each biological tissue image between the biological tissue images based on the pixels of the image feature points of the specified biological tissue image and the component pixels of the specified biological tissue image. Inter-image alignment means that unifies the coordinate position or coordinate range of pixels with the same characteristics,
Based on the reference shape showing the shape of the biological tissue that has been imaged in advance as the reference image by performing a predetermined image processing on each biological tissue image for which the alignment between the biological tissue images has been executed. , Each pixel constituting the target shape indicating the shape of the biological tissue imaged in each biological tissue image in which the alignment between the biological tissue images is executed, and each pixel constituting the reference shape. A shape alignment means that aligns the target shape of the living tissue in association with the image .
A specific means for specifying the coordinate position on each biological tissue image with respect to the aligned constituent pixels indicating the pixels constituting the target shape in which the target shape of the biological tissue is aligned.
An extraction means for extracting change information regarding changes in the biological tissue during the biological activity based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels , and
An output control means that outputs change information of the extracted biological tissue,
A program characterized by functioning as. An imaging device that captures an image of living tissue during biological activity and generates an image of the living tissue,
An image analysis device that acquires a plurality of biological tissue images generated by the imaging device in time series as biological tissue image data and executes predetermined image processing on the acquired plurality of biological tissue image data.
With
The image analysis device
An acquisition means for acquiring a plurality of biological tissue images in which biological tissues during biological activities are imaged in time series as biological tissue image data, and
By executing a predetermined image process on the biological tissue image of each acquired biological tissue image data, the pixels other than the pixels of the image feature points of the biological tissue image and the pixels of the image feature points constituting the biological tissue image are obtained. Of the pixels constituting the biological tissue in each biological tissue image between the biological tissue images based on the pixels of the image feature points of the specified biological tissue image and the component pixels of the specified biological tissue image. An image-to-image alignment means that unifies the coordinate positions or coordinate ranges of pixels having the same characteristics,
Based on the reference shape showing the shape of the biological tissue that has been imaged in advance as the reference image by performing a predetermined image processing on each biological tissue image for which the alignment between the biological tissue images has been executed. , Each pixel constituting the target shape indicating the shape of the biological tissue imaged in each biological tissue image in which the alignment between the biological tissue images is executed, and each pixel constituting the reference shape. A shape alignment means for aligning the target shape of the living tissue in association with the
A specific means for specifying each coordinate position on each biological tissue image with respect to the aligned constituent pixels indicating the pixels constituting the target shape in which the target shape of the biological tissue is aligned.
An extraction means for extracting change information regarding changes in the biological tissue during the biological activity based on the respective coordinate positions on each biological tissue image of the specified aligned constituent pixels .
An output control means that outputs change information of the extracted biological tissue, and
An image processing system characterized by being equipped with. An acquisition means for acquiring a plurality of biological tissue images in which biological tissues during biological activities are imaged in time series as biological tissue image data, and
By executing a predetermined image process on the biological tissue image of each acquired biological tissue image data, the pixels other than the pixels of the image feature points of the biological tissue image and the pixels of the image feature points constituting the biological tissue image are obtained. Of the pixels constituting the biological tissue in each biological tissue image between the biological tissue images based on the pixels of the image feature points of the specified biological tissue image and the component pixels of the specified biological tissue image. An image-to-image alignment means that unifies the coordinate positions or coordinate ranges of pixels having the same characteristics,
Based on the reference shape showing the shape of the biological tissue that has been imaged in advance as the reference image by performing a predetermined image processing on each biological tissue image for which the alignment between the biological tissue images has been executed. , Each pixel constituting the target shape indicating the shape of the biological tissue imaged in each biological tissue image in which the alignment between the biological tissue images is executed, and each pixel constituting the reference shape. A shape alignment means for aligning the target shape of the living tissue in association with the
An output control means for causing the display means to display an image relating to the aligned constituent pixels indicating the constituent pixels constituting the target shape in which the target shape of the living tissue is aligned.
A biological tissue image analysis system characterized by being equipped with. Computer,
An acquisition means for acquiring a plurality of biological tissue images in which biological tissues during biological activities are imaged in time series as biological tissue image data.
By executing a predetermined image process on the biological tissue image of each acquired biological tissue image data, the pixels other than the pixels of the image feature points of the biological tissue image and the pixels of the image feature points constituting the biological tissue image are obtained. Of the pixels constituting the biological tissue in each biological tissue image between the biological tissue images based on the pixels of the image feature points of the specified biological tissue image and the component pixels of the specified biological tissue image. Inter-image alignment means that unifies the coordinate position or coordinate range of pixels with the same characteristics,
Based on the reference shape showing the shape of the biological tissue that has been imaged in advance as the reference image by performing a predetermined image processing on each biological tissue image for which the alignment between the biological tissue images has been executed. , Each pixel constituting the target shape indicating the shape of the biological tissue imaged in each biological tissue image in which the alignment between the biological tissue images is executed, and each pixel constituting the reference shape. Shape alignment means for aligning the target shape of the living tissue in association with each other, and
An output control means for causing a display means to display an image relating to the aligned constituent pixels indicating the constituent pixels constituting the target shape in which the target shape of the living tissue is aligned.
A program characterized by functioning as.