JP6562253B2 - Image output device, image transmission device, image reception device, image output method and program - Google Patents

Description

  The present disclosure relates to an image output device, an image transmission device, an image reception device, and the like.

  Conventionally, an optical microscope has been used to observe a microstructure in a living tissue or the like. The optical microscope uses light transmitted through an observation object or reflected light. The observer observes the image magnified by the lens. There is also known a digital microscope that takes an image magnified by a microscope lens and displays it on a display. By using a digital microscope, simultaneous observation by multiple people, observation at a remote location, etc. are possible.

  In recent years, a technique for observing a microstructure by a CIS (Contact Image Sensing) method has attracted attention. In the case of the CIS method, the observation target is arranged close to the imaging surface of the image sensor. As the image sensor, a two-dimensional image sensor in which a large number of photoelectric conversion units are generally arranged in rows and columns in the imaging surface is used. The photoelectric conversion unit is typically a photodiode formed on a semiconductor layer or a semiconductor substrate, and generates charge upon receiving incident light.

  An image acquired by the image sensor is defined by a large number of pixels. Each pixel is partitioned by a unit region including one photoelectric conversion unit. Therefore, the resolution (resolution) in the two-dimensional image sensor usually depends on the arrangement pitch of the photoelectric conversion units on the imaging surface. In this specification, the resolution determined by the arrangement pitch of the photoelectric conversion units may be referred to as “inherent resolution” of the image sensor. Since the arrangement pitch of the individual photoelectric conversion units is as short as the wavelength of visible light, it is difficult to further improve the intrinsic resolution.

  A technique for realizing a resolution (high resolution) exceeding the intrinsic resolution of an image sensor has been proposed. Patent Document 1 discloses a technique for forming an image of a subject (high-resolution image) using a plurality of images obtained by shifting the imaging position of the subject.

JP-A-62-137037

  However, there is a problem that it is difficult to display an image of a subject smoothly because a high resolution image is generated.

  Therefore, the present disclosure provides an image output device, an image transmission device, an image reception device, and the like that can smoothly display an image of a subject.

  An image output apparatus according to an aspect of the present disclosure includes an image acquisition unit that acquires a first resolution image that is an image of a subject obtained based on imaging by a digital microscope, and an image that has a higher resolution than the first resolution image. A high-resolution image generation unit that generates a second-resolution image that is an image of the subject obtained based on photographing by the digital microscope, and an enlargement ratio for a partial region in the acquired first-resolution image An enlargement reception unit that receives enlargement information including the position of the region, and enlarges the region by performing pixel interpolation on the region of the first resolution image based on the enlargement information received by the enlargement reception unit. An interpolated image generating unit that generates an interpolated magnified image, and outputting the interpolated magnified image to a display device. When the enlargement rate included in the enlargement information received by the display output unit displayed on the display device and the enlargement acceptance unit is larger than a predetermined value, the second resolution image corresponding to the enlargement information is displayed. A switching unit that causes the display output unit to switch the interpolation enlarged image output to the display device to the second resolution image corresponding to the enlargement information, which is generated by the high resolution image generation unit. .

  Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. Alternatively, it may be realized by any combination of recording media.

  According to the present disclosure, an image of a subject can be displayed smoothly.

1 is a configuration diagram illustrating an example of an image processing system including an image output device according to Embodiment 1. FIG. 6 is a diagram illustrating an example of a display screen displayed by the display device in Embodiment 1. [FIG. FIG. 10 is a diagram illustrating another example of the display screen displayed by the display device in the first embodiment. 3 is a flowchart illustrating an example of processing operations of the digital microscope and the image output apparatus according to Embodiment 1. 6 is a flowchart illustrating another example of processing operations of the digital microscope and the image output apparatus according to Embodiment 1. 6 is a configuration diagram illustrating an example of an image processing system including an image output apparatus according to Embodiment 2. FIG. 10 is a flowchart illustrating an example of processing operations of the digital microscope, the image transmission device, and the image reception device in the second embodiment. 10 is a flowchart showing a part of processing of the image receiving apparatus according to the modification of the second embodiment. 3 is a plan view schematically showing a part of a subject 2. FIG. FIG. 7B is a plan view schematically showing extracted photodiodes related to imaging of the region shown in FIG. 7A. 3 is a cross-sectional view schematically showing the direction of light rays that pass through a subject 2 and enter a photodiode 4p. FIG. 3 is a cross-sectional view schematically showing the direction of light rays that pass through a subject 2 and enter a photodiode 4p. FIG. It is a figure which shows typically 6 pixels Pa acquired by the 6 photodiodes 4p. It is sectional drawing which shows typically the state which entered the light ray from the irradiation direction different from the irradiation direction shown to FIG. 8A and FIG. 8B. It is sectional drawing which shows typically the state which entered the light ray from the irradiation direction different from the irradiation direction shown to FIG. 8A and FIG. 8B. It is a figure which shows typically 6 pixels Pb acquired under the irradiation direction shown to FIG. 9A and 9B. FIG. 9 is a cross-sectional view schematically showing a state in which light rays are incident from an irradiation direction different from the irradiation direction shown in FIGS. 8A and 8B and the irradiation direction shown in FIGS. 9A and 9B. It is sectional drawing which shows typically the state which entered the light ray from the irradiation direction shown in FIG. 8A and FIG. 8B, and the irradiation direction different from the irradiation direction shown to FIG. 9A and FIG. 9B. It is a figure which shows typically 6 pixels Pc acquired under the irradiation direction shown to FIG. 10A and 10B. 8A and 8B are cross-sectional views schematically showing a state in which light is incident from an irradiation direction shown in FIGS. 8A and 8B, an irradiation direction shown in FIGS. 9A and 9B, and an irradiation direction different from the irradiation directions shown in FIGS. 10A and 10B. is there. FIG. 11B is a diagram schematically showing six pixels Pd acquired under the irradiation direction shown in FIG. 11A. It is a figure which shows the high resolution image HR synthesize | combined from four sub-images Sa, Sb, Sc, and Sd. FIG. 6 is a cross-sectional view schematically showing an irradiation direction adjusted so that light beams that have passed through two adjacent regions of the subject 2 enter different photodiodes. It is a figure which shows typically an example of the cross-section of a module. It is a top view which shows an example of the external appearance when the module 10 shown to FIG. 14A is seen from the image sensor 4 side. It is a figure for demonstrating an example of the manufacturing method of a module. It is sectional drawing which shows the example of the irradiation angle at the time of acquisition of a sub image. It is sectional drawing which shows the example of the method of irradiating a to-be-photographed object with the irradiation angle different from the irradiation angle shown to FIG. 16A. It is a schematic diagram showing an example of composition of an image acquisition device by an embodiment of this indication. It is a perspective view which shows the example external appearance of the image acquisition apparatus 100a. It is a perspective view which shows the state which closed the cover part 120 in the image acquisition apparatus 100a shown to FIG. 18A. It is a figure which shows typically an example of the loading method of the socket 130 with respect to the stage 32 of the image acquisition apparatus 100a. It is a figure which shows typically an example of the method of changing an irradiation direction. It is a figure which shows typically the change of the direction of the light ray which injects into a to-be-photographed object when the stage 32 is inclined only angle (theta) with respect to the reference plane. It is a figure which shows the cross-sectional structure of a CCD image sensor, and the example of distribution of the relative transmittance | permeability Td of a to-be-photographed object. It is a figure which shows the cross-sectional structure of a backside illumination type CMOS image sensor, and the example of distribution of the relative transmittance | permeability Td of a to-be-photographed object. It is a figure which shows the cross-sectional structure of a backside illumination type CMOS image sensor, and the example of distribution of the relative transmittance | permeability Td of a to-be-photographed object. It is a figure which shows the cross-sectional structure of a photoelectric converting film laminated | stacked image sensor, and the example of distribution of the relative transmittance | permeability Td of a to-be-photographed object.

(Knowledge that became the basis of the present invention)
The present inventor has found that the following problems occur with respect to the high resolution technology described in the “Background Art” column.

  In a pathological examination using a microscope, the magnification of a lens to be used varies depending on the pathology of an individual specimen (also referred to as a pathological specimen) to be observed. For example, in the examination of cancer, first, observation is performed with a lens having a 10 × objective. For specimens that do not have a region with a high suspicion of cancer, the observation is completed only by observing a lens with a 20 × objective. .

  When observing an image of a specimen obtained by photographing with a digital microscope, an image with a high resolution (high resolution image) is necessary for a high magnification. However, it takes time to acquire a high-resolution image. When a high-resolution image is generated from nine sub-images (low-resolution images), it takes about 100 seconds. That is, it takes about 60 seconds to capture nine sub-images, and it takes about 40 seconds to generate a high-resolution image from the nine captured sub-images. Therefore, there is a problem that even if an attempt is made to display a high-resolution image, the image of the subject cannot be displayed smoothly, for example, a time during which nothing is displayed occurs for a while.

  In order to solve such a problem, an image output apparatus according to an aspect of the present invention includes an image acquisition unit that acquires a first resolution image that is an image of a subject obtained based on photographing by a digital microscope, and the first A high-resolution image generation unit that generates a second resolution image that is an image of the subject that is an image having a higher resolution than that of the one-resolution image and is obtained based on photographing by the digital microscope, and the acquired first resolution An enlargement accepting unit that accepts enlargement information including an enlargement ratio for a partial region in the image and the position of the region, and the region of the first resolution image based on the enlargement information accepted by the enlargement accepting unit An interpolation image generation unit that generates an interpolation enlarged image by performing pixel interpolation and enlarging the region, and outputs the interpolation enlarged image to a display device The display output unit for displaying the interpolated enlarged image on the display device, and the enlargement information included in the enlargement information received by the enlargement accepting unit is greater than a predetermined value. The corresponding second resolution image is generated in the high-resolution image generation unit, and the interpolation enlarged image output to the display device is output to the display output unit according to the enlargement information. A switching unit that switches to a resolution image.

  Thereby, first, for example, when an image of a subject such as a specimen is enlarged and displayed, an interpolation enlarged image created based on the accepted enlargement ratio and position is displayed. When the enlargement ratio is larger than a predetermined value, a second resolution image corresponding to the enlargement ratio is generated as a high resolution image, and the displayed interpolated enlarged image corresponds to the enlargement ratio or the like. Switching to the second resolution image. Therefore, even if it takes a long time to generate the second resolution image, the interpolation enlarged image is displayed until the second resolution image is displayed, so that the image of the subject can be displayed smoothly. it can.

  In the image output device according to an aspect of the present invention, a high-resolution image is generated and displayed in preference to an area that is enlarged and displayed in the first resolution image. Therefore, for example, a high-resolution image can be displayed in a time close to 60 seconds for capturing nine sub-images (first resolution images). As a result, even in the case of a rapid examination in which the resection range of a malignant tumor is determined during surgery, the time taken to display a high-resolution image can be shortened, and the burden on the patient can be reduced.

  For example, the enlargement receiving unit sequentially receives the enlargement information, and the display output unit receives the enlarged enlarged image based on the enlargement information or the enlargement every time the enlargement information is received by the enlargement receiving unit. The image displayed on the display device may be switched by outputting the second resolution image corresponding to the information to the display device.

  Thereby, the image of the subject displayed on the display device can be appropriately adjusted by changing the enlargement information.

  The enlargement receiving unit sequentially receives the enlargement information during a predetermined adjustment period, and the switching unit is included in the enlargement information received immediately before the elapsed time at the time when the adjustment period has elapsed. When the enlargement ratio is larger than a predetermined value, the display output unit is caused to switch the interpolation enlarged image output to the display device to the second resolution image corresponding to the enlargement information. Also good. For example, when the second resolution image is generated using a plurality of first resolution images, this adjustment period is required to capture the number of first resolution images (for example, nine) necessary for generating the second resolution image. It is longer than the time (for example, 60 seconds).

  Accordingly, by changing the enlargement information during the adjustment period, it is possible to appropriately adjust a partial region (interpolation enlargement image) of the first resolution image that is enlarged and displayed on the display device. Then, when the adjustment period elapses, the interpolation enlarged image can be switched to the second resolution image.

  The high-resolution image generation unit generates the second resolution image using the plurality of first resolution images acquired by the image acquisition unit, and the image output device further includes the second resolution image. When the plurality of first resolution images necessary for generating the image are acquired by the image acquisition unit, a notification unit that notifies the user of the completion of shooting may be provided.

  Accordingly, since the user is notified of the completion of shooting, the user can recognize that the second resolution image can be quickly displayed after the notification. As a result, the user can smoothly display the second resolution image on the display device by making the enlargement ratio larger than a predetermined value as necessary.

  An image output apparatus according to another aspect of the present invention is an image output apparatus including an image transmission apparatus and an image reception apparatus connected to the image transmission apparatus via a communication line, the image transmission apparatus. Is an image acquisition unit for acquiring a first resolution image that is an image of a subject obtained based on photographing with a digital microscope, and an image having a higher resolution than the first resolution image, based on photographing with the digital microscope A high-resolution image generation unit that generates a second resolution image that is an image of the subject obtained in the above, and the image reception device includes an enlargement ratio for a partial region in the acquired first resolution image, and An enlargement receiving unit that receives enlarged information including the position of the region, and the image transmission device that is transmitted based on the enlarged information received by the enlargement receiving unit. An interpolation image generation unit for generating an interpolation enlarged image by performing pixel interpolation on the region of the first resolution image and enlarging the region, and outputting the interpolation enlarged image to a display device, thereby the interpolation enlarged image When the enlargement rate included in the enlargement information received by the enlargement reception unit is larger than a predetermined value, the second resolution image corresponding to the enlargement information is displayed on the display device. Is generated by the high-resolution image generation unit, the second resolution image corresponding to the enlarged information is acquired from the high-resolution image generation unit, and is output to the display device with respect to the display output unit. A switching unit that switches the interpolated enlarged image to the second resolution image corresponding to the enlarged information.

  Thus, for example, even if there is no pathologist at the point where the specimen as the subject is placed, the pathologist at a remote location can use the image receiving apparatus to transmit the image of the specimen transmitted from the image transmission apparatus installed at that point. The specimen can be obtained and diagnosed based on the image. Even at this time, even if it takes time to generate the second resolution image, the interpolation enlarged image is displayed until the second resolution image is displayed. can do.

  Hereinafter, embodiments will be specifically described with reference to the drawings.

  It should be noted that each of the embodiments described below shows a comprehensive or specific example. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

(Embodiment 1)
FIG. 1 is a configuration diagram illustrating an example of an image processing system including an image output apparatus according to the first embodiment.

  The image processing system 1000 includes a digital microscope 1500, an image output device 1001, and a display device 1501.

  The digital microscope 1500 is an image acquisition device described later, and images a pathological specimen as a subject. Here, in this specification, the “digital microscope” is an apparatus for digitally imaging at least a part of a pathological tissue preparation specimen. For example, the digital microscope is an image acquisition device, which will be described later, or a virtual slide scanner that captures the specimen by a contact image sensing (CIS) method. Here, the image acquisition device that captures images by the CIS method is a device that acquires a first resolution image and a second resolution image that are images of the subject by directly placing the subject on the image sensor and performing the photographing. . Specifically, the image acquisition device generates a plurality of first resolution images by irradiating a subject with light from a plurality of different irradiation directions and photographing the subject. And the said image acquisition apparatus produces | generates the 2nd resolution image of the resolution higher than each 1st resolution image by rearranging each pixel of those some 1st resolution images. Further, the digital microscope 1500 may be a digital microscope such as a virtual slide scanner that generates enlarged images having a plurality of resolutions, instead of an image acquisition apparatus described later. A virtual slide scanner is a device that photographs a specimen through a microscope. The operation of shifting the relative position of the slide with respect to the microscope and the operation of photographing are repeated, and a plurality of obtained images are connected to generate an image of a wide range of specimens. In the virtual slide scanner, the resolution of an image to be acquired is selected by changing the objective lens attached to the microscope. When an objective lens with a low magnification is used, the resolution is lowered, but the range of the specimen imaged on one image is widened, so that the number of imaging parts is reduced and the imaging time is shortened. The image acquisition device will be described in detail in the third embodiment.

  The display device 1501 is a device that displays an image obtained based on photographing with a digital microscope to an operator (for example, a pathologist), and is, for example, a liquid crystal display or a projector.

  The image output device 1001 acquires an image obtained by photographing with the digital microscope 1500 and causes the display device 1501 to display the image. The image output apparatus 1001 includes an image acquisition unit 1101, a high-resolution image generation unit 1102, an enlargement reception unit 1201, a switching unit 1202, a display output unit 1203, and an interpolation image generation unit 1210.

  The image acquisition unit 1101 acquires a first resolution image that is an image of a subject obtained based on photographing by the digital microscope 1500. Note that the image of the subject obtained based on the photographing by the digital microscope 1500 may be an image directly obtained by the photographing, or an image obtained by performing processing on the directly obtained image. It may be. Here, the image acquisition unit 1101 acquires, as a first resolution image, a sub-image obtained by, for example, photographing with a contact image sensor in the digital microscope 1500. The resolution of the first resolution image is referred to as the first resolution.

  The high-resolution image generation unit 1102 generates a second resolution image that is an image having a higher resolution than the first resolution image and is a subject image obtained based on photographing by the digital microscope 1500. In the present embodiment, the high resolution image generation unit 1102 uses the plurality of first resolution images acquired by the image acquisition unit 1101 to generate the second resolution image (that is, the high resolution image). The resolution of the second resolution image is referred to as the second resolution. Here, the first resolution has a pixel pitch of 0.9 μm, and the second resolution has a pixel pitch of 0.3 μm. Further, when the ratio of the pixel pitch of the first resolution to the pixel pitch of the second resolution is a resolution ratio, the resolution ratio is 3.

  Note that the image acquisition unit 1101 may not have acquired all the number of first resolution images necessary for generating the second resolution image. The total number is 9, for example. Therefore, the high resolution image generation unit 1102 determines whether or not acquisition of all the first resolution images has been completed, and performs generation of the high resolution image after determining that acquisition has ended.

  The enlargement accepting unit 1201 accepts enlargement information including an enlargement ratio with respect to a partial area in the acquired first resolution image and a position of the area. The position of the area may be the center position (center coordinates) of the area, or may be the upper left end or the lower right end of the rectangular area. Specifically, the enlargement reception unit 1201 acquires the enlargement ratio and position through, for example, a keyboard operation. Further, the enlargement receiving unit 1201 may update the accepted enlargement rate and position by acquiring the respective change amounts of the enlargement rate and the position. Specifically, the enlargement receiving unit 1201 uses, for example, the number of rotations of a mouse having a wheel as the change value of the enlargement rate, uses the change amount of the mouse position as the change amount of the position of the region, and sets the enlargement rate and position. It may be updated. At this time, the initial value of the enlargement ratio is, for example, 1, and the initial coordinate value of the position of the area is, for example, the coordinate value of the center in the entire image.

  Here, in order to simplify the description of the processing described later, the maximum value of the enlargement rate K accepted by the enlargement accepting unit 1201 is set to a value equal to or less than the resolution ratio 3. That is, when the accepted enlargement factor K is a value larger than the resolution ratio, the enlargement acceptance unit 1201 replaces the enlargement factor K with the resolution ratio. The enlargement ratio after replacement is used to enlarge the image.

  The enlargement ratio used for image enlargement may be a value larger than the resolution ratio. In that case, when outputting the second resolution image, the display output unit 1203 causes the interpolation image generation unit 1210 to perform interpolation and enlargement of the second resolution image based on the enlargement ratio. Then, the display output unit 1203 causes the display device 1501 to display the second resolution image that has been subjected to the interpolation and enlargement.

  The interpolated image generation unit 1210 generates an interpolated enlarged image by performing pixel interpolation on a partial region of the first resolution image based on the enlargement information received by the enlargement receiving unit 1201 and enlarging the region. That is, when the interpolation image generation unit 1210 enlarges the area, the interpolation image generation unit 1210 enlarges the area at an enlargement rate included in the enlargement information. Note that pixel interpolation is a process that creates and compensates for pixels that do not originally exist between pixels included in an image, and is, for example, interpolation by a nearest neighbor method, a bilinear method, a bicubic method, or the like. When a value larger than the resolution ratio is accepted as the enlargement ratio, and the interpolation and enlargement target is the second resolution image, the interpolation image generation unit 1210 obtains a value obtained by dividing the accepted enlargement ratio by the resolution ratio, Treated as the actual magnification used for image magnification. That is, the interpolated image generation unit 1210 generates an interpolated enlarged image based on the second resolution image by enlarging the second resolution image at the actual magnification, and the resolution between the first resolution image and the second resolution image. To absorb the difference. For example, when the received enlargement ratio is 6, the interpolation image generation unit 1210 interpolates and enlarges the second resolution image by a value 2 obtained by dividing the enlargement ratio 6 by the resolution ratio 3. That is, the interpolated image generation unit 1210 converts the enlargement factor 6 to 2, and performs interpolation enlargement of the second resolution image with the enlargement factor 2 after the conversion. The interpolated image generation unit 1210 converts the coordinates of the position of the area by multiplying the coordinates of the position of the area by the resolution ratio 3.

  The display output unit 1203 outputs the above-described interpolation enlarged image to the display device 1501, thereby causing the display device 1501 to display the interpolation enlarged image. The display output unit 1203 may display the first resolution image or the second resolution image on the display device 1501 by outputting the first resolution image or the second resolution image to the display device 1501.

  The switching unit 1202 causes the high-resolution image generation unit 1102 to generate a second resolution image corresponding to the enlargement information when the enlargement rate included in the enlargement information received by the enlargement reception unit 1201 is larger than a predetermined value. . Then, the switching unit 1202 causes the display output unit 1203 to switch the interpolation enlarged image output to the display device 1501 to the second resolution image corresponding to the enlarged information. The second resolution image corresponding to the enlarged information is not an image corresponding to the entire area of the first resolution image, but an image corresponding to a part of the first resolution image corresponding to the enlarged information. . Here, the predetermined value may be a value arbitrarily set by a pathologist or an operator, for example, or may be a predetermined value. Specifically, the predetermined value may be, for example, a value equal to the resolution ratio (that is, 3 in the above example). The predetermined value is a value larger than 1. However, if the predetermined value is set to a low value, the images are frequently switched. In order to avoid this, it is desirable that the predetermined value is as high as possible.

  Further, if the predetermined value is set to a low value, the area for generating the second resolution image increases in the entire area captured by the first resolution image. Therefore, the processing load for generating the second resolution image increases, and the reaction speed for operations such as changing the display range decreases. Therefore, the predetermined value is set to 3 when the CPU performance used for generating the second resolution image is high, and the predetermined value is set to 10 when the CPU performance is low. Also, since the area of the second resolution image to be transferred increases as the area for generating the second resolution image increases, the predetermined value is set to 10 for a thin line such as a mobile phone line, and for a thick line such as an optical line, The predetermined value is 3.

  That is, the switching unit 1202 switches the image displayed by the display device 1501. For example, after the second resolution image is generated by the high resolution image generation unit 1102, the switching unit 1202 switches the image displayed on the display device 1501 from the interpolation enlarged image to the second resolution image corresponding to the enlarged information. . In addition, when the interpolation image generation unit 1210 generates an interpolation enlarged image based on the above-described second resolution image, the switching unit 1202 may switch to the interpolation enlarged image.

  FIG. 2A is a diagram illustrating an example of a display screen displayed by the display device 1501.

  The display device 1501 displays a display screen 601 including an end button 602, an overhead image display unit 603, a finding input field 604, a read button 605, an enlarged image display unit 606, and a determination result field 607.

  On the enlarged image display unit 606, the first resolution image or the second resolution image is enlarged and displayed. When the end button 602 is selected, the display device 1501 transmits a signal notifying that the image display should be ended to the image output device 1001. The overhead image display unit 603 displays the entire first resolution image or second resolution image. In the findings input field 604, for example, findings by a pathologist are written. When the reading button 605 is selected, the display device 1501 ends the display of the first resolution image and transmits a signal instructing reading of the second resolution image to the image output device 1001, for example.

  Here, as described above, the high-resolution image generation unit 1102 determines whether or not the image acquisition unit 1101 has acquired all the numbers of the first resolution images necessary for generating the second resolution image. Judgment. The display output unit 1203 receives the determination result from the high-resolution image generation unit 1102 and causes the display device 1501 to display the determination result. For example, as illustrated in FIG. 2A, the display device 1501 displays the determination result in the determination result column 607 of the display screen 601. When the determination result indicates that the digital microscope 1500 has not been acquired, that is, when the digital microscope 1500 continues to perform imaging, the display device 1501 displays a message “under imaging” in the determination result column 607 as the determination result.

  FIG. 2B is a diagram showing another example of the display screen displayed by the display device 1501.

  When the determination result by the high-resolution image generation unit 1102 indicates that the acquisition is complete, the display device 1501 displays a message “photographing complete” in the determination result column 607 as the determination result.

  FIG. 3 is a flowchart showing an example of processing operations of the digital microscope 1500 and the image output apparatus 1001.

  First, the digital microscope 1500 captures the first first resolution image (step S11). Then, the digital microscope 1500 transmits the first resolution image to the image output device 1001 (step S12). After that, the digital microscope 1500 continuously captures each of the second and subsequent first resolution images and sequentially transmits them to the image output apparatus 1001 (step S13).

  The image acquisition unit 1101 of the image output apparatus 1001 acquires the first first resolution image transmitted from the digital microscope 1500 (step S21). The display output unit 1203 outputs the first first resolution image to the display device 1501, thereby causing the display device 1501 to display the first first resolution image (step S22). Further, every time the second and subsequent first resolution images are transmitted from the digital microscope 1500 in step S13, the image acquisition unit 1101 acquires the second and subsequent first resolution images at the transmission timing. .

  Next, the enlargement accepting unit 1201 of the image output apparatus 1001 accepts enlargement information including an enlargement ratio and a position in response to an operation by the operator on, for example, a keyboard or a mouse (step S23). The enlargement ratio is an enlargement ratio for a partial area of the image displayed on the display device 1501, and the position is, for example, the center position of the area. Then, the display output unit 1203 displays the enlarged interpolation image on the display device 1501 (step S24). That is, the display output unit 1203 generates an interpolated enlarged image from the first first resolution image according to the enlarged information received in step S23, and causes the display device 1501 to display the interpolated enlarged image.

  Here, the high resolution image generation unit 1102 determines whether or not all the first resolution images necessary for generating the second resolution image have been acquired (step S25). If it is determined that the image is acquired (Yes in step S25), the display output unit 1203 displays “imaging completed” on the display device 1501 as the determination result, thereby allowing the user (operator or pathologist) to display it. The completion of shooting is notified (step S26). From this notification, the user can determine that a high-resolution image (second resolution image) can be displayed quickly if the enlargement ratio is increased thereafter. If it is determined that all the first resolution images have not been acquired (No in step S25), the image output apparatus 1001 repeatedly executes the processing from step S23.

  When the completion of shooting is notified in step S26, the switching unit 1202 determines whether or not the enlargement rate included in the enlargement information received last by the enlargement receiving unit 1201 is larger than a threshold value (predetermined value) ( Step S27). For example, the threshold is a resolution ratio value (here, 3). Here, when it is determined that the enlargement ratio is equal to or less than the threshold (No in step S27), the display output unit 1203 displays the interpolation enlarged image on the display device 1501 (step S28). That is, the display output unit 1203 generates an interpolated enlarged image from the first first resolution image according to the received enlarged information and causes the display device 1501 to display the interpolated enlarged image. The accepted enlarged information is the enlarged information accepted in step S23 or step S33. If the enlarged information is the information accepted in step S23, the display output unit 1203 continuously displays the interpolated enlarged image displayed in step S24 on the display device 1501 in step S28.

  If it is determined in step S27 that the enlargement ratio is greater than the threshold (Yes in step S27), the high resolution image generation unit 1102 has already generated and stored a second resolution image corresponding to the received enlargement information. It is determined whether or not there is (step S29). If it is determined that the second resolution image is not stored (No in step S29), the switching unit 1202 causes the high resolution image generation unit 1102 to generate and store the second resolution image corresponding to the received enlarged information. (Step S31). After step S31 or when it is determined that the image is stored in step S29 (Yes in step S29), the display output unit 1203 reads the second resolution image corresponding to the enlarged information and displays it in the display device 1501. It is displayed (step S30).

  As a result, the interpolated enlarged image is displayed first in step S24, and then the interpolated enlarged image is switched to the second resolution image. That is, the switching unit 1202 causes the display output unit 1203 to switch the interpolation enlarged image output to the display device 1501 to the second resolution image corresponding to the enlarged information.

  Then, the display output unit 1203 determines whether or not to end the display of the image based on a signal output from the display device 1501 when the end button 602 is selected (step S32). If it is determined that the processing should not be terminated (No in step S32), the enlargement receiving unit 1201 receives again enlargement information including the enlargement ratio and position in accordance with an operation performed by the operator on, for example, a keyboard or a mouse. (Step S33). Then, the image output device 1001 repeatedly executes the processing from step S27. On the other hand, when it is determined that the process should be ended (Yes in step S32), the image output apparatus 1001 ends the image display process.

  As described above, the enlargement receiving unit 1201 sequentially receives the enlargement information in step S23 and step S33. The display output unit 1203 outputs an interpolation enlarged image based on the enlarged information or a second resolution image corresponding to the enlarged information to the display device 1501 every time enlarged information is received by the enlarged receiving unit 1201. (Steps S24, S28, S30). Thereby, the image displayed on the display device 1051 is switched. Therefore, the subject image displayed on the display device 1501 can be appropriately adjusted by changing the enlargement information.

  FIG. 4 is a flowchart showing another example of processing operations of the digital microscope 1500 and the image output apparatus 1001.

  The digital microscope 1500 executes steps S11 to S13 as in the flowchart shown in FIG.

  First, the image output apparatus 1001 executes the processes of steps S21 to S23 as in the flowchart shown in FIG. Next, the display output unit 11203 of the image output device 1001 generates an interpolated enlarged image from the first first resolution image according to the enlarged information received in step S23, and displays the generated enlarged image on the display device 1501 ( Step S41).

  Here, for example, the display output unit 1203 determines whether or not a predetermined time has elapsed since the first first resolution image was acquired in step S21 (step S41). During the predetermined time, after the first first resolution image is captured in step S11, the capturing of all the first resolution images necessary for generating the second resolution image is completed. It is set to more than the time until. That is, if the predetermined time has elapsed, all the first resolution images are acquired by the image output apparatus 1001.

  Therefore, if the display output unit 1203 determines that the predetermined time has elapsed (Yes in step S42), the display device 1501 displays “shooting complete” as the determination result to notify the user (step S43). ). On the other hand, if it is determined that the predetermined time has not elapsed (No in step S42), the image output apparatus 1001 repeatedly executes the processing from step S23.

  When the completion of shooting is notified in step S43, the switching unit 1202 determines whether or not the enlargement rate included in the enlargement information received immediately before the elapse of the predetermined time is larger than the threshold (step S44). . If it is determined that the value is equal to or smaller than the threshold value (No in step S44), the display output unit 1203 continuously displays the interpolated enlarged image displayed in step S41 on the display device 1501 (step S45).

  On the other hand, if it is determined that the enlargement ratio is greater than the threshold (Yes in step S44), the switching unit 1202 causes the high-resolution image generation unit 1102 to generate a second resolution image corresponding to the enlargement information (step S46). . Then, the switching unit 1202 causes the display output unit 1203 to switch the interpolation enlarged image output to the display device 1501 to the second resolution image corresponding to the enlarged information (step S47). That is, the display output unit 1203 causes the display device 1501 to display a second resolution image corresponding to the enlarged information.

  As described above, in the example of the processing operation illustrated in FIG. 4, the above-described predetermined time is used as an adjustment period for adjusting the interpolation enlarged image based on the first resolution image. That is, the enlargement receiving unit 1201 sequentially receives the enlargement information during a predetermined adjustment period. Then, the switching unit 1202 displays an image to the display output unit 1203 when the enlargement rate included in the enlargement information received immediately before the elapsed time is larger than a predetermined value at the time when the adjustment period has elapsed. Let them switch. That is, the switching unit 1202 causes the display output unit 1203 to switch the interpolation enlarged image output to the display device 1501 to the second resolution image corresponding to the enlarged information. Accordingly, by changing the enlargement information in the adjustment period, it is possible to appropriately adjust a partial region (interpolation enlargement image) of the first resolution image displayed on the display device 1501 in an enlarged manner. Then, when the adjustment period elapses, the interpolation enlarged image can be switched to the second resolution image.

(effect)
As described above, in the present embodiment, the first resolution image is presented to the observer (user) before all the first resolution images required for generating the high resolution image are captured. It is possible to determine whether or not to increase the resolution, or to determine the area to increase the resolution.

  In this embodiment, for example, when an image of a subject such as a specimen is enlarged and displayed, an interpolation enlarged image created based on the accepted enlargement factor and position is displayed. When the enlargement ratio is larger than a predetermined value, a second resolution image corresponding to the enlargement ratio is generated as a high resolution image, and the displayed interpolated enlarged image corresponds to the enlargement ratio or the like. Switching to the second resolution image. Therefore, even if it takes a long time to generate the second resolution image, the interpolation enlarged image is displayed until the second resolution image is displayed, so that the image of the subject can be displayed smoothly. it can.

  In the present embodiment, a high-resolution image is generated and displayed in preference to an area that is enlarged and displayed in the first resolution image. Therefore, for example, a high-resolution image can be displayed in a time close to 60 seconds for capturing nine sub-images (first resolution images). As a result, even in the case of a rapid examination in which the resection range of a malignant tumor is determined during surgery, the time taken to display a high-resolution image can be shortened, and the burden on the patient can be reduced.

  In the present embodiment, the display output unit 1203 completes photographing when a plurality of first resolution images necessary for generating the second resolution image are acquired by the image acquisition unit 1101. Inform the user. That is, the display output unit 1203 includes a notification unit that notifies the user of the completion of shooting. Accordingly, since the user is notified of the completion of shooting, the user can recognize that the second resolution image can be quickly displayed after the notification.

(Embodiment 2)
The image output apparatus according to the present embodiment includes an image transmission apparatus and an image reception apparatus that are connected to each other via a communication line. Of the apparatus and its constituent elements in the present embodiment, the same apparatus and its constituent elements as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and detailed descriptions thereof are omitted. .

  FIG. 5 is a configuration diagram illustrating an example of an image processing system including the image output apparatus according to the second embodiment.

  The image processing system 2000 includes a digital microscope 1500, an image output device 2001, and a display device 1501.

  The image output apparatus 2001 includes an image transmission apparatus 1100 and an image reception apparatus 1200 that are connected to each other via a communication line. Note that the image output apparatus 2001 as a whole has the same functions as the image output apparatus 1001 of the first embodiment. For example, a set including the digital microscope 1500 and the image transmission device 1100 is arranged in a facility such as a hospital without a pathologist. The set including the image receiving device 1200 and the display device 1501 is arranged, for example, in a facility with a pathologist far from the hospital.

  The image transmission device 1100 includes an image acquisition unit 1101, a high resolution image generation unit 1102, and a notification unit 1113.

  As in the first embodiment, the high-resolution image generation unit 1102 determines whether or not all the first resolution images necessary for generating the second resolution image have been acquired by the image acquisition unit 1101.

  When the high-resolution image generation unit 1102 determines that all the first resolution images have been acquired, the notification unit 1113 notifies the image reception device 1200 of completion of shooting via the communication line.

  The image reception device 1200 includes an image acquisition unit 1215, an enlargement reception unit 1201, an interpolation image generation unit 1210, a display output unit 1203, a switching unit 1202, a notification unit 1213, and an image holding unit 1212.

  The image acquisition unit 1215 receives the first resolution image (for example, the first first resolution image) acquired from the digital microscope 1500 by the image acquisition unit 1101 of the image transmission device 1100 from the image transmission device 1100 via the communication line. get.

  The notification unit 1213 notifies the user of the completion of shooting when a plurality of first resolution images necessary for generating the second resolution image are acquired by the image acquisition unit 1101. That is, when the notification unit 1213 receives a notification of completion of imaging from the notification unit 1113 of the image transmission apparatus 1100 via the communication line, the notification unit 1213 displays a message “imaging complete” on the display device 1501. Accordingly, the notification unit 1213 notifies the user (operator) of the image reception device 1200 of the completion of shooting. Note that the notification unit 1213 may notify the user of the completion of shooting by outputting a voice “shooting is completed”.

  The image holding unit 1212 is a recording medium for holding the second resolution image transmitted from the image transmission apparatus 1100 via the communication line.

  FIG. 6A is a flowchart illustrating an example of processing operations of the digital microscope 1500, the image transmission device 1100, and the image reception device 1200.

  The digital microscope 1500 executes steps S11 to S13 as in the flowchart shown in FIG. In steps S12 and S13, the digital microscope 1500 transmits the first resolution image to the image transmission device 1100.

  When acquiring the first first resolution image from the digital microscope 1500, the image acquisition unit 1101 of the image transmission apparatus 1100 transmits the first resolution image to the image reception apparatus 1200 via the communication line (step S51). Furthermore, the image acquisition unit 1101 acquires the second and subsequent first resolution images sequentially transmitted from the digital microscope 1500 (step S52). Here, every time the first resolution image is acquired, the image acquisition unit 1101 outputs the first resolution image to the high resolution image generation unit 1102. The high resolution image generation unit 1102 determines whether or not all the first resolution images necessary for generating the second resolution image have been acquired. If it is determined that the image has been acquired, the notification unit 1113 notifies the image reception device 1200 of “shooting complete” as the determination result via the communication line (step S53).

  The image acquisition unit 1215 of the image reception device 1200 acquires the first first resolution image transmitted from the image transmission device 1100 via the communication line (step S21). The display output unit 1203 outputs the first first resolution image to the display device 1501, thereby causing the display device 1501 to display the first first resolution image (step S22).

  Next, the enlargement accepting unit 1201 of the image receiving device 1200 accepts enlargement information including an enlargement ratio and a position in response to an operation by the operator on, for example, a keyboard or a mouse (step S23). Then, the display output unit 1203 displays the enlarged interpolation image on the display device 1501 (step S24). That is, the display output unit 1203 generates an interpolated enlarged image from the first first resolution image according to the enlarged information received in step S23, and causes the display device 1501 to display the interpolated enlarged image.

  Here, the notification unit 1213 determines whether or not there is a shooting completion notification from the image transmission device 1100 in step S53 (step S25a). When it is determined that there is a notification (Yes in step S25a), the notification unit 1213 displays “imaging completion” on the display device 1501 as the determination result, thereby allowing the user (operator or pathologist) to complete the imaging. Notification is made (step S26). From this notification, the user can determine that a high-resolution image (second resolution image) can be displayed quickly if the enlargement ratio is increased thereafter. When it is determined that there is no notification (No in step S25a), the image reception device 1200 repeatedly executes the processing from step S23.

  When the completion of shooting is notified in step S26, the switching unit 1202 determines whether or not the enlargement rate included in the enlargement information received last by the enlargement receiving unit 1201 is larger than a threshold value (predetermined value) ( Step S27). Here, when it is determined that the enlargement ratio is equal to or less than the threshold (No in step S27), the display output unit 1203 displays the interpolation enlarged image on the display device 1501 (step S28).

  On the other hand, if it is determined that the enlargement ratio is greater than the threshold (Yes in step S27), the switching unit 1202 already generates a second resolution image corresponding to the accepted enlargement information and stores it in the image holding unit 1212. It is determined whether or not (step S29). If it is determined that the second resolution image is not stored (No in step S29), the switching unit 1202 sends the second resolution image corresponding to the received enlarged information to the image transmission apparatus 1100 via the communication line. A request is made (step S31a).

  When receiving the request (step S54), the high-resolution image generation unit 1102 of the image transmission device 1100 generates a second resolution image corresponding to the received enlarged information, and sends it to the image reception device 1200 via the communication line. Transmit (step S55).

  The switching unit 1202 of the image receiving apparatus 1200 acquires the second resolution image corresponding to the enlarged information transmitted from the image transmitting apparatus 1100 and stores it in the image holding unit 1212 (step S31b). After step S31b or when it is determined that the image is stored in step S29 (Yes in step S29), the display output unit 1203 reads the second resolution image corresponding to the enlarged information and displays it in the display device 1501. It is displayed (step S30).

  As a result, the interpolated enlarged image is displayed first in step S24, and then the interpolated enlarged image is switched to the second resolution image. That is, the switching unit 1202 causes the display output unit 1203 to switch the interpolation enlarged image output to the display device 1501 to the second resolution image corresponding to the enlarged information.

  Then, the display output unit 1203 determines whether or not to end the display of the image based on a signal output from the display device 1501 when the end button 602 is selected (step S32). If it is determined that the processing should not be terminated (No in step S32), the enlargement receiving unit 1201 receives again enlargement information including the enlargement ratio and position in accordance with an operation performed by the operator on, for example, a keyboard or a mouse. (Step S33). Then, the image receiving device 1200 repeatedly executes the processing from step S27. On the other hand, if it is determined that the process should be terminated (Yes in step S32), the image reception device 1200 ends the image display process.

  As described above, the enlargement receiving unit 1201 sequentially receives the enlargement information in step S23 and step S33. The display output unit 1203 outputs an interpolation enlarged image based on the enlarged information or a second resolution image corresponding to the enlarged information to the display device 1501 every time enlarged information is received by the enlarged receiving unit 1201. (Steps S24, S28, S30). Thereby, the image displayed on the display device 1051 is switched. Therefore, the subject image displayed on the display device 1501 can be appropriately adjusted by changing the enlargement information.

  Note that the notification unit 1213 notifies the user of the completion of shooting after all the first resolution images necessary for generating the second resolution image are acquired, but notifies the user before the acquisition. Also good. For example, when the predetermined number of first resolution images (for example, 90%) out of all the number of first resolution images required for generating the second resolution image are acquired, the notification unit 1213 is obtained. May notify the user of the completion of shooting. In addition, since the time required for shooting is constant, the notification unit 1213 receives a predetermined time from the acquisition of the first first resolution image regardless of the notification from the notification unit 1113 of the image transmission device 1100. The user may be notified of the completion of shooting. The predetermined time is an expected time required to shoot all the first resolution images required for generating the second resolution image. Moreover, the alerting | reporting part 1213 may alert | report before or after the predetermined time.

(effect)
In such an image output apparatus 2001 according to the present embodiment, the same effect as in the first embodiment can be obtained even if the facility for acquiring the image of the subject and the facility for observing the image are separated from each other.

  In rapid diagnosis, it is important that diagnosis is performed promptly. However, it takes time to increase the resolution of the entire image. In the present embodiment, the sub-image is transmitted to the image receiving device 1200 when the first low-resolution first resolution image (sub-image) is acquired, so that a rough diagnosis is started early with the sub-image. can do. In addition, the resolution is increased from the portion displayed on the display device 1501 in the first resolution image, and is transmitted to the image receiving device 1200 and displayed. Therefore, higher resolution can be implemented with priority from the displayed portion of the first resolution image.

  In the present embodiment, even if the second resolution image is not stored in the image holding unit 1212, the number of the first resolution images required for generating the high resolution image has been captured. After a few seconds, a second resolution image corresponding to the enlarged information is generated, transmitted, and displayed. On the other hand, if the required number of first resolution images have not been taken, the high resolution image is not displayed unless several tens of seconds are waited. In the present embodiment, the notification unit 1213 notifies the completion of shooting. Therefore, when the second resolution image is not displayed, the operator can easily grasp whether the cause is the incomplete photographing or the generation and transmission of the second resolution image. Thereby, the operator can operate the image receiving device 1200 smoothly.

(Modification)
If the image receiving device 1200 in the second embodiment determines that the second resolution image corresponding to the enlarged information is not stored as in steps S29 and S31a of FIG. 6A, the second resolution image is transmitted to the image transmitting device. Request to 1100. Even when it is determined that the second resolution image is not stored, the image receiving device 1200 according to the present modification example changes the second resolution image to the image transmitting device if the enlarged information is immediately changed. No request is made to 1100.

  FIG. 6B is a flowchart showing a part of processing of the image receiving apparatus 1200 according to the present modification.

  Similar to the image receiving apparatus 1200 in the second embodiment, the image receiving apparatus 1200 according to the present modification executes steps S21 to S32 shown in FIG. 6A and further executes the process of step S29a.

  Specifically, the switching unit 1202 determines whether or not a second resolution image corresponding to the received enlargement information has already been generated and stored in the image holding unit 1212 (step S29). Here, if it is determined that the second resolution image is not stored (No in step S29), the switching unit 1202 has passed a certain period of time without changing the enlargement information (that is, the enlargement ratio and position). It is determined whether or not (step S29a). If the switching unit 1202 determines that the predetermined time or more has elapsed (Yes in step S29a), the switching unit 1202 requests the image transmission apparatus 1100 for a second resolution image corresponding to the received enlarged information (step S31a). . Then, the image receiving device 1200 performs the same process as in the second embodiment. On the other hand, when it is determined that the predetermined time or more has not elapsed, that is, the enlarged information has been changed before the predetermined time has elapsed (No in step S29a), the display output unit 1203 causes the display device 1501 to display the interpolation enlarged image. (Step S28). As a result, even if it is determined in step S27 that the second resolution image corresponding to the received enlarged information is necessary, if the second resolution image becomes unnecessary due to the change of the enlarged information, the second resolution image Request, generation or reading can be suppressed. That is, it is possible to suppress execution of useless processing relating to the second resolution image.

(effect)
In Embodiment 2 described above, when the display center position is moved by a mouse operation in a state where the enlargement ratio is larger than the threshold value, the second resolution image must be generated for all the areas through which the display center position has passed. Don't be. Note that the display center position is, for example, a position included in the enlarged information. If the range in which the operator moves the mouse becomes wider, the area of the second resolution image that must be generated becomes wider, and the waiting time until the second resolution image of the area actually observed is generated increases. . However, the observer (pathologist) stops the mouse or suppresses the speed at which the mouse is moved at a diagnostically important position where the observation is performed firmly. In the present modification, the second resolution image is created when the display position is stopped for a certain time, so that the waiting time until the second resolution image of the portion to be observed firmly is generated can be shortened.

  In the flowchart shown in FIG. 6B, it is determined in step S29a whether or not the value of the enlarged information has been stopped for a certain time or more, but it is determined whether or not the moving speed of the display center position is equal to or less than a predetermined value. May be. Specifically, if the moving speed of the display center position is equal to or less than a predetermined value, the image receiving device 1200 executes the process of step S31a, and if the moving speed is greater than the predetermined value, executes the process of step S28. To do.

  In Embodiments 1 and 2 and the modifications thereof, the high-resolution image generation unit 1102 generates a second resolution image using a plurality of first resolution images, and these first resolution images are used. Instead, the second resolution image may be generated. For example, when the digital microscope 1500 is a virtual slide scanner or the like, the lens used for photographing the subject is changed. For example, a first resolution image is generated by a 10 × objective lens, and a second resolution image in a partial region of the first resolution image is generated by a 40 × objective lens. The high-resolution image generation unit 1102 generates a second resolution image based on the image signal from the digital microscope 1500 obtained by the 40 × objective lens. That is, the high-resolution image generation unit 1102 requests the digital microscope 1500 for an image signal corresponding to the enlargement information (for example, the enlargement ratio and position) received by the enlargement reception unit 1201, and uses the image signal as the enlargement information. A corresponding second resolution image is generated. The high-resolution image generation unit 1102 generates a second resolution image corresponding to the enlarged information every time the enlarged information is changed. For example, every time the position included in the enlarged information is changed, the high resolution image generation unit 1102 generates a second resolution image corresponding to a partial region of the first resolution image at the position.

(Embodiment 3)
Here, the digital microscope 1500 in each of the above embodiments and modifications thereof will be described in detail below. The digital microscope 1500 is hereinafter referred to as an image acquisition device (digitizer).

<Principle of high resolution image formation>
In Honkai, a plurality of images obtained by changing the irradiation direction of the illumination light and performing a plurality of shootings are used, and images having a higher resolution (resolution) than each of the plurality of images (hereinafter referred to as “high”). A resolution image "or a" high resolution image "). The high resolution image corresponds to the above-described second resolution image, and each of the plurality of images (sub-images) used for forming the high resolution image corresponds to the above-described first resolution image. First, the principle of high-resolution image formation will be described with reference to FIGS. Here, a CCD (Charge Coupled Device) image sensor will be described as an example. In the following description, components having substantially the same function are denoted by common reference numerals, and description thereof may be omitted.

  Please refer to FIG. 7A and FIG. 7B. FIG. 7A is a plan view schematically showing a part of the subject. The subject 2 shown in FIG. 7A is, for example, a thin piece of biological tissue (typically having a thickness of several tens of μm or less). When the image of the subject 2 is acquired, the subject 2 is arranged close to the imaging surface of the image sensor. The distance from the imaging surface of the image sensor to the subject 2 is typically 1 mm or less, and can be set to about 1 μm, for example.

  FIG. 7B is a plan view schematically showing extracted photodiodes related to imaging in the region shown in FIG. 7A out of the photodiodes of the image sensor. In the example described here, among the photodiodes 4p formed in the image sensor 4, six photodiodes are shown. For reference, FIG. 7B shows arrows indicating the x, y, and z directions orthogonal to each other. The z direction indicates the normal direction of the imaging surface. FIG. 7B also shows an arrow indicating the u direction, which is a direction rotated by 45 ° from the x axis toward the y axis in the xy plane. In other drawings, an arrow indicating the x direction, the y direction, the z direction, or the u direction may be illustrated.

  Components other than the photodiode 4p in the image sensor 4 are covered with a light shielding layer. In FIG. 7B, the hatched area indicates an area covered with the light shielding layer. The area (S2) of the light receiving surface of one photodiode on the imaging surface of the CCD image sensor is smaller than the area (S1) of the unit region including the photodiode. The ratio (S2 / S1) of the light receiving area S2 to the pixel area S1 is called "aperture ratio". Here, description will be made assuming that the aperture ratio is 25%.

  8A and 8B schematically show the directions of light rays that pass through the subject 2 and enter the photodiode 4p. 8A and 8B show a state where light rays are incident from a direction perpendicular to the imaging surface. As schematically shown in FIGS. 8A and 8B, here, no lens for imaging is arranged between the subject 2 and the image sensor 4, and the image of the subject 2 passes through the subject 2. Acquired using substantially parallel rays.

  FIG. 8C schematically shows an image Sa (first sub-image Sa) acquired under the irradiation direction shown in FIGS. 8A and 8B. As shown in FIG. 8C, the first sub-image Sa is composed of six pixels Pa acquired by six photodiodes 4p. Each pixel Pa has a value (pixel value) indicating the amount of light incident on the individual photodiode 4p.

  As shown in FIGS. 8A and 8B, when the subject 2 is irradiated from a direction perpendicular to the imaging surface, light that has passed through a region located directly above the photodiode 4p in the entire subject 2 is incident on the photodiode 4p. To do. In this example, the first sub-image Sa has information on areas A1, A2, A3, A4, A5, and A6 (see FIG. 7A) of the entire subject 2. Note that light that has passed through a region not directly above the photodiode 4p does not enter the photodiode 4p. Therefore, in the first sub-image Sa, information on regions other than the regions A1, A2, A3, A4, A5, and A6 in the entire subject 2 is missing.

  9A and 9B show a state in which light rays are incident from an irradiation direction different from the irradiation direction shown in FIGS. 8A and 8B. The light rays shown in FIGS. 9A and 9B are inclined in the x direction with respect to the z direction. At this time, light that has passed through a region different from the region located immediately above the photodiode 4p in the entire subject 2 enters the photodiode 4p.

  FIG. 9C schematically shows an image Sb (second sub-image Sb) acquired under the irradiation direction shown in FIGS. 9A and 9B. As shown in FIG. 9C, the second sub-image Sb is also composed of six pixels acquired by the six photodiodes 4p. However, the pixels Pb constituting the second sub-image Sb are areas B1, B2, B3, B4, B5, and B6 (outside the areas A1, A2, A3, A4, A5, and A6 of the entire subject 2 ( 7A)). In other words, the second sub-image Sb does not have information on the areas A1, A2, A3, A4, A5, and A6 in the entire subject 2, and instead, the areas B1, B2, B3, B4 , B5 and B6. Here, for example, the region B1 is a region adjacent to the right side of the region A1 in the subject 2 (see FIG. 7A).

  As understood by comparing FIGS. 8A and 8B with FIGS. 9A and 9B, by appropriately changing the irradiation direction, light beams that have passed through different regions of the subject 2 are incident on the photodiode 4p. be able to. As a result, the first sub image Sa and the second sub image Sb can include pixel information corresponding to different positions in the subject 2.

  10A and 10B show a state in which light rays are incident from an irradiation direction different from the irradiation direction shown in FIGS. 8A and 8B and the irradiation directions shown in FIGS. 9A and 9B. The light rays shown in FIGS. 10A and 10B are inclined in the y direction with respect to the z direction.

  FIG. 10C schematically shows an image Sc (third sub-image Sc) acquired under the irradiation direction shown in FIGS. 10A and 10B. As shown in FIG. 10C, the third sub-image Sc is composed of six pixels Pc acquired by the six photodiodes 4p. As shown in the figure, the third sub-image Sc has information on areas C1, C2, C3, C4, C5, and C6 shown in FIG. Here, for example, the region C1 is a region adjacent to the upper side of the region A1 in the subject 2 (see FIG. 7A).

  11A shows a state in which light rays are incident from an irradiation direction shown in FIGS. 8A and 8B, an irradiation direction shown in FIGS. 9A and 9B, and an irradiation direction different from the irradiation directions shown in FIGS. 10A and 10B. . The light ray shown in FIG. 11A is inclined in a direction that forms an angle of 45 ° with the x axis in the xy plane with respect to the z direction.

  FIG. 11B schematically shows an image Sd (fourth sub-image Sd) acquired under the irradiation direction shown in FIG. 11A. As shown in FIG. 11B, the fourth sub-image Sd is composed of six pixels Pd acquired by six photodiodes 4p. The fourth sub-image Sd has information on areas D1, D2, D3, D4, D5, and D6 shown in FIG. Here, for example, the region D1 is a region adjacent to the right side of the region C1 (see FIG. 7A). Thus, each of the sub-images Sa, Sb, Sc, and Sd includes an image composed of different parts of the subject 2.

  FIG. 12 shows a high-resolution image HR synthesized from four sub-images Sa, Sb, Sc, and Sd. As shown in FIG. 12, the number of pixels or the pixel density of the high-resolution image HR is four times the number of pixels or the pixel density of each of the four sub-images Sa, Sb, Sc, and Sd.

  For example, attention is focused on the blocks of the areas A1, B1, C1, and D1 shown in FIG. As can be seen from the above description, the pixel Pa1 of the sub-image Sa shown in FIG. 12 has information on only the area A1, not the entire block described above. Therefore, it can be said that the sub-image Sa is an image in which information on the areas B1, C1, and D1 is missing.

  However, by using the sub-images Sb, Sc, and Sd having pixel information corresponding to different positions in the subject 2, as shown in FIG. 12, the missing information in the sub-image Sa is complemented and the entire block information is included. It is possible to form a high resolution image HR. In this example, the resolution of each sub-image is equal to the intrinsic resolution of the image sensor 4. In this example, a resolution four times the intrinsic resolution of the image sensor 4 is obtained. The degree of high resolution (super-resolution) depends on the aperture ratio of the image sensor. In this example, since the aperture ratio of the image sensor 4 is 25%, the resolution can be increased up to four times by light irradiation from four different directions. When N is an integer greater than or equal to 2, if the aperture ratio of the image sensor 4 is approximately equal to 1 / N, the resolution can be increased up to N times.

  In this way, by imaging a subject by sequentially irradiating parallel light from a plurality of different irradiation directions with respect to the subject, pixel information sampled “spatially” from the subject can be increased. . By synthesizing the obtained plurality of sub-images, it is possible to form a high-resolution image having a higher resolution than each of the plurality of sub-images. Of course, the irradiation direction is not limited to the irradiation direction described with reference to FIGS. 8A to 11B.

  In the above example, the sub-images Sa, Sb, Sc and Sd shown in FIG. 12 have pixel information of different regions in the subject 2 and do not overlap. However, there may be overlap between different sub-images. In the above example, the light beams that have passed through two adjacent regions in the subject 2 are both incident on the same photodiode. However, the setting of the irradiation direction is not limited to this example. For example, as shown in FIG. 13, the irradiation direction may be adjusted so that light beams that have passed through two adjacent regions of the subject 2 enter different photodiodes.

<Module>
In the formation of a high-resolution image based on the principle described with reference to FIGS. 7A to 12, the acquisition of the sub image is performed in a state where the subject 2 is arranged close to the imaging surface of the image sensor 4. In the embodiment of the present disclosure, a sub image is acquired using a module having a structure in which the subject 2 and the image sensor 4 are integrated. Hereinafter, an example of a module configuration and an example of a module manufacturing method will be described with reference to the drawings.

  FIG. 14A schematically shows an example of a cross-sectional structure of the module. In the module 10 shown in FIG. 14A, the subject 2 covered with the encapsulant 6 is disposed on the imaging surface 4 </ b> A of the image sensor 4. In the illustrated example, a transparent plate (typically a glass plate) 8 is arranged on the subject 2. That is, in the configuration illustrated in FIG. 14A, the subject 2 is sandwiched between the image sensor 4 and the transparent plate 8. It is advantageous that the module 10 has the transparent plate 8 because workability is improved. As the transparent plate 8, for example, a general slide glass can be used. Note that each element is schematically shown in the drawing, and the actual size and shape of each element do not necessarily match the size and shape shown in the figure. The same applies to other drawings referred to below.

  In the configuration illustrated in FIG. 14A, the image sensor 4 is fixed to the package 5. FIG. 14B shows an example of the appearance when the module 10 shown in FIG. 14A is viewed from the image sensor 4 side. As shown in FIGS. 14A and 14B, the package 5 has a back electrode 5 </ b> B on the surface opposite to the transparent plate 8. The back electrode 5B is electrically connected to the image sensor 4 through a wiring pattern (not shown) formed on the package 5. That is, the output of the image sensor 4 can be taken out via the back electrode 5B. In this specification, a structure in which a package and an image sensor are integrated is referred to as an “imaging device”.

  An example of a method for manufacturing the module 10 will be described with reference to FIG. Here, a thin slice (tissue slice) of a biological tissue is illustrated as the subject 2. The module 10 having a biological tissue slice as the subject 2 can be used for pathological diagnosis.

  First, as shown in FIG. 15, the tissue section A02 is placed on the transparent plate 8. The transparent plate 8 can be a slide glass used for observing a sample with an optical microscope. Below, a slide glass is illustrated as the transparent plate 8. Next, the tissue section A02 is stained by immersing the tissue section A02 together with the transparent plate 8 in the staining solution Ss. Next, the subject 2 obtained by staining the tissue section A02 is covered with the encapsulant 6 by applying the encapsulant 6 on the transparent plate 8. The encapsulant 6 has a function of protecting the subject 2. Next, the image sensor 7 is disposed on the subject 2 such that the imaging surface of the image sensor 4 faces the subject 2. In this way, the module 10 is obtained.

  The module 10 is produced for each object to be imaged. For example, in a pathological diagnosis scene, a plurality of (for example, 5 to 20) tissue sections are prepared from one specimen. Therefore, a plurality of modules 10 having tissue sections obtained from the same specimen as the subject 2 can be produced. If a plurality of sub-images are acquired for each of the plurality of modules 10, a high-resolution image corresponding to each of the plurality of modules 10 can be formed.

  As illustrated in FIG. 14A, the module 10 includes an image sensor 4 that acquires an image of the subject 2, unlike a preparation used for observation with an optical microscope. Such a module may be called an “electronic preparation”. As shown in FIG. 14A, the use of the module 10 having a structure in which the subject 2 and the image sensor 7 are integrated has an advantage that the arrangement between the subject 2 and the image sensor 4 can be fixed.

  When acquiring an image of the subject 2 using the module 10, the illumination light is irradiated to the subject 2 through the transparent plate 8. Illumination light transmitted through the subject 2 enters the image sensor 4. Thereby, an image of the subject 2 is obtained. By sequentially performing imaging while changing the relative arrangement of the light source and the subject, a plurality of different images can be acquired at different angles during irradiation. For example, as illustrated in FIG. 16A, the light source 310 is disposed immediately above the image sensor 4. Then, if imaging is performed in a state where the subject 2 is irradiated with the collimated light CL from the normal direction of the imaging surface 4A of the image sensor 4, a sub image similar to the sub image Sa shown in FIG. 8C is obtained. Further, as shown in FIG. 16B, when the subject 2 is irradiated with the light CL collimated with the module 10 tilted, and the imaging is performed, the sub-image Sb shown in FIG. 9C (or the sub-image Sc shown in FIG. 10C). A sub-image similar to is obtained. As described above, by sequentially performing imaging while changing the posture of the module 10 with respect to the light source, it is possible to obtain a high-resolution image by applying the principle described with reference to FIGS. 7A to 12. .

<Image acquisition device>
FIG. 17 schematically illustrates an example of the configuration of an image acquisition device according to an embodiment of the present disclosure. An image acquisition apparatus 100a illustrated in FIG. In the configuration illustrated in FIG. 17, the illumination system 30 is configured such that the light source 31 that generates illumination light, the stage 32 configured to be detachably loaded with the module 10, and the attitude of the stage 32 can be changed. The stage drive mechanism 33 is included. FIG. 17 schematically shows a state in which the module 10 is loaded on the stage 32. However, illustration of the encapsulant 6 and the transparent plate 8 in the module 10 is omitted. The module 10 is not an essential component for the image acquisition apparatus 100a.

  The module 10 has an arrangement in which illumination light transmitted through the subject 2 is incident on the image sensor 7 while being connected to the stage 32. The illumination system 30 changes the irradiation direction with the subject 2 as a reference, for example, by changing the posture of the stage 32. The change in “attitude” in the present specification broadly includes a change in inclination with respect to a reference plane, a change in rotation angle with respect to a reference orientation, a change in position with respect to a reference point, and the like. The subject 2 is irradiated with illumination light generated by the light source 31 sequentially from a plurality of different irradiation directions with the subject 2 as a reference. Details of the configuration and operation of the illumination system 30 will be described later. By irradiating the subject 2 while changing the irradiation direction, a plurality of different images (sub-images) according to a plurality of different irradiation directions are acquired by the image sensor 7. A high-resolution image can be formed using the plurality of obtained images.

  An image acquisition apparatus 100a illustrated in FIG. 17 includes an irradiation direction determination unit 40a. The irradiation direction determination unit 40a determines a plurality of different irradiation directions when a plurality of sub-images are acquired by the image sensor 7. In the embodiment of the present disclosure, the acquisition of the sub-image is executed under a plurality of different irradiation directions determined by the irradiation direction determination unit. In other words, the sub-images in the embodiment of the present disclosure are a plurality of different images corresponding to a plurality of different irradiation directions determined by the irradiation direction determination unit.

  Next, an example of a method for changing the irradiation direction of the illumination light with the subject as a reference will be described with reference to FIGS. 18A to 19B.

  18A and 18B show an exemplary appearance of the image acquisition device 100a. In the configuration illustrated in FIG. 18A, the image acquisition device 100 a includes a main body 110 including a light source 31 and a stage 32, and a lid 120 that is connected to the main body 110 so as to be opened and closed. By closing the lid 120, a dark room can be formed inside the image acquisition device 100a (see FIG. 18B).

  In the illustrated example, a socket 130 for holding the module 10 is connected on the stage 32. The socket 130 may be fixed to the stage 32, or may be configured to be detachable from the stage 32. Here, a configuration in which the socket 130 is configured to be detachable from the stage 32 is illustrated. The socket 130 includes, for example, a lower base material 132 in which the module 10 is detachable and an upper base material 134 in which the opening Ap is formed. In the configuration illustrated in FIG. 18A, the socket 130 holds the module 10 by sandwiching the module 10 between the lower base material 132 and the upper base material 134.

  The lower base member 132 may have an electrical connection portion having an electrical contact for electrical connection with the image sensor 7 of the module 10. When acquiring the image of the subject, the module 10 is placed on the lower base material 132 so that the imaging surface of the imaging device 7 faces the light source 31. At this time, the electrical contact of the electrical connection portion and the back surface electrode 5B (see FIGS. 14A and 14B) of the imaging device 7 come into contact with each other, whereby the imaging device 7 of the module 10 and the electrical connection portion of the lower base member 132 are connected. Electrically connected.

  FIG. 18C shows an example of a method for loading the socket 130 into the stage 32 of the image acquisition apparatus 100a. In the configuration illustrated in FIG. 18C, the socket 130 has an electrode 136 protruding from the bottom surface. This electrode 136 may be part of the electrical connection of the lower substrate 132. In the example shown in FIG. 18C, the stage 32 of the image acquisition device 100a has a mounting portion 34 provided with a jack 36. As shown in FIG. 18C, for example, the socket 130 holding the module 10 is loaded on the stage 32 so that the electrode 136 of the socket 130 is inserted into the jack 36. Thereby, the electrical connection between the image sensor 7 in the module 10 held in the socket 130 and the image acquisition device 100a is established. The stage 32 may have a circuit that receives the output of the image sensor 4 in a state in which the socket 130 that holds the module 10 is loaded. In the embodiment of the present disclosure, the image acquisition device 100a acquires information (image signal or image data) indicating an image of the subject 2 with the electrical connection unit included in the socket 130 as an intermediary.

  When imaging a plurality of subjects using a plurality of modules 10, the same number of sockets 130 as the modules 10 are prepared, and the imaging target is changed by replacing the sockets 130 holding the modules 10. Also good. Alternatively, the imaging target may be changed by replacing the module 10 with the single socket 130 attached to the stage 32.

  As illustrated in FIG. 18C, by loading the socket 130 on the stage 32, the bottom surface of the socket 130 and the top surface of the attachment portion 34 can be brought into close contact with each other. Thereby, the arrangement of the socket 130 with respect to the stage 32 is fixed. Therefore, the arrangement of the stage 32 and the module 10 held in the socket 130 can be kept constant before and after the change in the posture of the stage 32. Typically, in a state where the socket 130 is loaded on the stage 32, the main surface of the transparent plate 8 of the module 10 and the stage 32 are substantially parallel.

  FIG. 19A shows an example of a method for changing the irradiation direction. As illustrated, the illumination light CL emitted from the light source 31 is irradiated to the module 10 held in the socket 130. The illumination light CL is incident on the subject of the module 10 through the opening Ap provided in the socket 130. The light transmitted through the subject enters the imaging surface of the imaging device 7 of the module 10.

  The light emitted from the light source 31 is typically collimated light. However, when the light incident on the subject can be regarded as substantially parallel light, the light emitted from the light source 31 may not be collimated light.

  The light source 31 includes, for example, an LED chip. The light source 31 may include a plurality of LED chips each having a peak in a different wavelength band. For example, the light source 31 may include an LED chip that emits blue light, an LED chip that emits red light, and an LED chip that emits green light. When a plurality of light emitting elements are arranged close to each other (for example, about 100 μm), these can be regarded as point light sources.

  Using a plurality of light emitting elements that emit light of different colors, for example, by sequentially irradiating light of different colors in each irradiation direction, a plurality of sub-images for each color can be acquired. . For example, a set of blue sub-images, a set of red sub-images, and a set of green sub-images may be obtained. By using the acquired set of sub-images, a color high-resolution image can be formed. For example, in a pathological diagnosis scene, more useful information regarding the presence or absence of a lesion can be obtained by using a color high-resolution image. By using a white LED chip as the light source 31 and arranging a color filter on the optical path, illumination lights of different colors may be obtained in a time sequential manner. Further, an image sensor for color imaging may be used as the image sensor 4. However, from the viewpoint of suppressing a reduction in the amount of light incident on the photoelectric conversion unit of the image sensor 4, a configuration without a color filter is more advantageous.

  The light source 31 is not limited to an LED, and may be an incandescent bulb, a laser element, a fiber laser, a discharge tube, or the like. The light emitted from the light source 31 is not limited to visible light, and may be ultraviolet light, infrared light, or the like. The number and arrangement of the light emitting elements included in the light source 31 can be arbitrarily set.

  As shown in FIGS. 17 and 19A, the image acquisition apparatus 100a includes a stage drive mechanism 33. The stage drive mechanism 33 includes a gonio mechanism, a rotation mechanism, and the like, and changes the tilt angle of the stage 32 with respect to the main body 110 and / or a rotation angle with respect to an axis passing through the center of the stage 32. The stage drive mechanism 33 may include a slide mechanism that can translate the stage 32 in a reference plane (typically a horizontal plane).

  By operating the stage drive mechanism 33, the posture of the stage 32 can be changed. Here, since the socket 130 holding the module 10 is attached to the stage 32, the posture of the module 10 can be changed by changing the posture of the stage 32. For example, it is assumed that the incident direction of the illumination light when the stage 32 is not inclined with respect to the reference plane is the normal direction of the imaging surface of the image sensor. Here, the relationship (for example, parallel) between the inclination of the stage 32 with respect to the reference plane and the inclination of the module 10 with respect to the reference plane (which may be referred to as the inclination of the transparent plate 8) is the change in the attitude of the stage 32. It is kept constant before and after. For this reason, as shown in FIG. 19B, when the stage 32 is tilted by the angle θ with respect to the reference plane, the direction of light rays incident on the subject is also tilted by the angle θ. In FIG. 19B, a broken line N indicates a normal line of the imaging surface of the image sensor.

  Thus, by changing the posture of the module 10 together with the stage 32, it is possible to sequentially irradiate the subject with illumination light from a plurality of different irradiation directions with the subject 2 as a reference. Therefore, a plurality of images corresponding to a plurality of different irradiation directions with the subject 2 as a reference can be acquired by the imaging element 7 of the module 10. The irradiation direction with respect to the subject 2 is, for example, an angle (normal angle θ shown in FIG. 19B) formed by the normal N of the imaging surface of the image sensor and the incident light to the subject 2, and a reference set on the imaging surface. It can be expressed by a set of angles (azimuth angles) formed by the azimuth and the projection of incident light onto the imaging surface.

  It is possible to irradiate the subject 2 from a plurality of different irradiation directions by moving the light source 31 in the image acquisition device 100a or by sequentially lighting a plurality of light sources arranged at different locations. It is. For example, the irradiation direction may be changed by moving the light source 31 along the direction connecting the light source 31 and the subject 2. The irradiation direction may be changed by combining the change in the posture of the stage 32 and the movement of the light source 31.

<Image sensors used in modules>
In the embodiment of the present disclosure, the image sensor 4 is not limited to a CCD image sensor, but is a CMOS (Complementary Metal-Oxide Semiconductor) image sensor or other image sensor (for example, a photoelectric conversion film laminated type described later). Image sensor). The CCD image sensor and the CMOS image sensor may be either a front side illumination type or a back side illumination type. Hereinafter, the relationship between the element structure of the image sensor and the light incident on the photodiode of the image sensor will be described.

  FIG. 20 shows a cross-sectional structure of the CCD image sensor and an example of the distribution of the relative transmittance Td of the subject. As shown in FIG. 20, the CCD image sensor generally includes a substrate 80, an insulating layer 82 on the substrate 80, and a wiring 84 disposed in the insulating layer 82. A plurality of photodiodes 88 are formed on the substrate 80. A light shielding layer (not shown in FIG. 20) is formed on the wiring 84. Here, illustration of transistors and the like is omitted. In the following drawings, illustration of transistors and the like is omitted. In general, the cross-sectional structure in the vicinity of the photodiode in the front-illuminated CMOS image sensor is substantially the same as the cross-sectional structure in the vicinity of the photodiode in the CCD image sensor. Therefore, the illustration and description of the cross-sectional structure of the front-illuminated CMOS image sensor are omitted here.

  As shown in FIG. 20, when the illumination light is incident from the normal direction of the imaging surface, the irradiation light transmitted through the region R <b> 1 immediately above the photodiode 88 in the subject enters the photodiode 88. On the other hand, the irradiation light that has passed through the region R2 of the subject that is directly above the light shielding layer on the wiring 84 is incident on the light shielding region (the region where the light shielding film is formed) of the image sensor. Therefore, when irradiation is performed from the normal direction of the imaging surface, an image showing the region R1 of the subject that is directly above the photodiode 88 is obtained.

  In order to acquire an image showing a region immediately above the light shielding film, irradiation is performed from a direction inclined with respect to the normal direction of the imaging surface so that light transmitted through the region R2 enters the photodiode 88. Good. At this time, depending on the irradiation direction, part of the light transmitted through the region R <b> 2 may be blocked by the wiring 84. In the example shown in the figure, the light beam passing through the portion indicated by hatching does not reach the photodiode 88. For this reason, the pixel value may be somewhat reduced at oblique incidence. However, since not all of the transmitted light is blocked, it is possible to form a high-resolution image using the sub-image obtained at this time.

  FIG. 21A and FIG. 21B show a cross-sectional structure of a backside illuminated CMOS image sensor and an example of a relative transmittance Td distribution of a subject. As shown in FIG. 21A, in the backside illuminated CMOS image sensor, transmitted light is not blocked by the wiring 84 even in the case of oblique incidence. However, noise caused by the light (schematically indicated by a thick arrow BA in FIG. 21A and FIG. 21B described later) that has passed through another area of the subject that is different from the area to be imaged enters the substrate 80 May occur, and the quality of the sub-image may be deteriorated. Such deterioration can be reduced by forming a light shielding layer 90 on a region other than the region where the photodiode is formed on the substrate, as shown in FIG. 21B.

  FIG. 22 shows a cross-sectional structure of an image sensor (hereinafter referred to as a “photoelectric conversion film laminated image sensor”) including a photoelectric conversion film formed of an organic material or an inorganic material, and a distribution of relative transmittance Td of the subject. And an example.

  As shown in FIG. 22, the photoelectric conversion film stacked image sensor generally includes a substrate 80, an insulating layer 82 provided with a plurality of pixel electrodes, a photoelectric conversion film 94 on the insulating layer 82, and a photoelectric sensor. And a transparent electrode 96 on the conversion film 94. As shown in the figure, in the photoelectric conversion film stacked image sensor, a photoelectric conversion film 94 for performing photoelectric conversion is formed on a substrate 80 (for example, a semiconductor substrate) instead of the photodiode formed on the semiconductor substrate. The photoelectric conversion film 94 and the transparent electrode 96 are typically formed over the entire imaging surface. Here, illustration of a protective film for protecting the photoelectric conversion film 94 is omitted.

  In the photoelectric conversion film stacked image sensor, charges (electrons or holes) generated by photoelectric conversion of incident light in the photoelectric conversion film 94 are collected by the pixel electrode 92. Thereby, a value indicating the amount of light incident on the photoelectric conversion film 94 is obtained. Therefore, in the photoelectric conversion film stacked image sensor, it can be said that a unit region including one pixel electrode 92 corresponds to one pixel on the imaging surface. In the photoelectric conversion film laminated image sensor, the transmitted light is not blocked by the wiring even in the case of oblique incidence as in the case of the back-illuminated CMOS image sensor.

  As described with reference to FIGS. 7A to 12, in the formation of a high-resolution image, a plurality of sub-images showing images composed of different parts of the subject are used. However, in a typical photoelectric conversion film laminated image sensor, since the photoelectric conversion film 94 is formed over the entire imaging surface, for example, even in the case of normal incidence, it transmits through a region other than the desired region of the subject. Photoelectric conversion can occur in the photoelectric conversion film 94 also by the light that has been applied. If excess electrons or holes generated at this time are drawn into the pixel electrode 92, there is a possibility that an appropriate sub-image cannot be obtained. Therefore, it is beneficial to selectively draw the charges generated in the region where the pixel electrode 92 and the transparent electrode 96 overlap (the shaded region in FIG. 22) to the pixel electrode 92.

  In the configuration illustrated in FIG. 22, a dummy electrode 98 is provided in the pixel corresponding to each of the pixel electrodes 92. An appropriate potential difference is applied between the pixel electrode 92 and the dummy electrode 98 when acquiring the image of the subject. As a result, the charge generated in the region other than the region where the pixel electrode 92 and the transparent electrode 96 overlap is drawn into the dummy electrode 98, and the charge generated in the region where the pixel electrode 92 and the transparent electrode 96 overlap is selectively selected. Can be drawn into. Similar effects can be obtained by patterning the transparent electrode 96 or the photoelectric conversion film 94. In such a configuration, it can be said that the ratio (S3 / S1) of the area S3 of the pixel electrode 92 to the area S1 of the pixel corresponds to the “aperture ratio”.

  As already described, when N is an integer equal to or greater than 2, if the aperture ratio of the image sensor 4 is approximately equal to 1 / N, the resolution can be increased up to N times. In other words, a smaller aperture ratio is advantageous for higher resolution. In the photoelectric conversion film laminated image sensor, the ratio (S3 / S1) corresponding to the aperture ratio can be adjusted by adjusting the area S3 of the pixel electrode 92. This ratio (S3 / S1) is set in the range of 10% to 50%, for example. A photoelectric conversion film laminated image sensor having the ratio (S3 / S1) within the above range can be used for super-resolution.

  As can be seen from FIGS. 20 and 21B, the surface of the CCD image sensor and the front-illuminated CMOS image sensor that faces the subject is not flat. For example, a CCD image sensor has a step on its surface. Further, in the backside illumination type CMOS image sensor, in order to obtain a sub-image for forming a high-resolution image, it is necessary to provide a patterned light-shielding layer on the imaging surface, and the surface facing the subject is flat. is not.

  On the other hand, the imaging surface of the photoelectric conversion film laminated image sensor is a substantially flat surface as can be seen from FIG. Therefore, even when the subject is arranged on the imaging surface, the subject is hardly deformed due to the shape of the imaging surface. In other words, a more detailed structure of the subject can be observed by acquiring a sub-image using a photoelectric conversion film laminated image sensor.

  As described above, the image output apparatus according to one or a plurality of aspects has been described based on some embodiments, but the present invention is not limited to these embodiments. Unless it deviates from the meaning of the present invention, various forms conceived by those skilled in the art may be applied to the present embodiment, and forms constructed by combining components in different embodiments may be included in the present disclosure. .

  In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the image output apparatus of each of the above embodiments is a program that causes a computer to execute each step in the flowchart of FIG. 3, FIG. 4, FIG. 6A, or FIG.

  Further, in this disclosure, all or part of a unit, a device, or all or part of the functional blocks in the block diagrams shown in FIGS. 1 and 5 may be a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large may be implemented by one or more electronic circuits including scale integration). The LSI or IC may be integrated on a single chip, or may be configured by combining a plurality of chips. For example, the functional blocks other than the memory element may be integrated on one chip. Here, it is called LSI or IC, but the name changes depending on the degree of integration and may be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA) programmed after manufacture of an LSI, or a reconfigurable logic device capable of reconfiguring junction relations inside the LSI or setting up circuit partitions inside the LSI can be used for the same purpose.

  Further, all or part of the functions or operations of the unit, the apparatus, or a part of the apparatus can be executed by software processing. In this case, the software is recorded on a non-transitory recording medium such as one or more ROMs, optical disks, hard disk drives, etc., and the software is executed when the software is executed by a processor. Causes specific functions in the software to be executed by the processor and peripheral devices. The system or apparatus may comprise one or more non-transitory recording media in which software is recorded, a processor, and required hardware devices such as an interface.

  The present disclosure has an effect that an image of a subject can be displayed smoothly, and can be applied to, for example, an image output apparatus for displaying a high-resolution image of a pathological specimen.

1000, 2000 Image processing system 1001, 2001 Image output device 1100 Image transmission device 1101 Image acquisition unit 1102 High-resolution image generation unit 1113 Notification unit 1200 Image reception device 1201 Enlargement reception unit 1202 Switching unit 1203 Display output unit 1210 Interpolated image generation unit 1213 Notification unit 1500 Digital microscope 1501 Display device

Claims (9)

An image acquisition unit for acquiring a first resolution image that is an image of a subject obtained based on photographing by a digital microscope;
A high-resolution image generation unit that generates a second resolution image that is an image of the subject that is an image having a higher resolution than the first resolution image and is obtained based on photographing by the digital microscope;
An enlargement receiving unit for receiving enlargement information including an enlargement ratio for a partial area in the acquired first resolution image and a position of the area;
An interpolation image generation unit that generates an interpolation enlarged image by performing pixel interpolation on the region of the first resolution image based on the enlargement information received by the enlargement reception unit, and enlarging the region;
A display output unit for displaying the interpolated magnified image on the display device by outputting the interpolated magnified image to a display device;
When the enlargement ratio included in the enlargement information received by the enlargement reception unit is greater than a predetermined value, the second resolution image corresponding to the enlargement information is generated by the high-resolution image generation unit, and the display A switching unit that causes the output unit to switch the interpolated enlarged image output to the display device to the second resolution image according to the enlarged information;
An image output apparatus comprising: The expansion reception unit sequentially receives the expansion information,
The display output unit outputs the interpolated enlarged image based on the enlarged information or the second resolution image corresponding to the enlarged information to the display device every time the enlarged information is received by the enlarged receiving unit. The image output device according to claim 1, wherein the image displayed on the display device is switched. The expansion receiving unit sequentially receives the expansion information during a predetermined adjustment period,
When the enlargement rate included in the enlargement information received immediately before the elapsed time is larger than a predetermined value at the time when the adjustment period has elapsed, the switching unit The image output device according to claim 1, wherein the interpolated enlarged image output to the display device is switched to the second resolution image corresponding to the enlarged information. The high-resolution image generation unit generates the second resolution image using the plurality of first resolution images acquired by the image acquisition unit;
The image output device further includes:
4. A notification unit for notifying a user of completion of photographing when a plurality of the first resolution images necessary for generating the second resolution image are acquired by the image acquisition unit. The image output device according to any one of the above. An image output device having an image transmission device and an image reception device connected to the image transmission device via a communication line,
The image transmission device includes:
An image acquisition unit for acquiring a first resolution image that is an image of a subject obtained based on photographing by a digital microscope;
A high-resolution image generating unit that generates an image having a higher resolution than the first resolution image and a second resolution image that is an image of the subject obtained based on photographing by the digital microscope;
The image receiving device includes:
An enlargement receiving unit for receiving enlargement information including an enlargement ratio for a partial area in the acquired first resolution image and a position of the area;
Based on the enlargement information received by the enlargement accepting unit, an interpolated enlarged image is generated by performing pixel interpolation on the region of the first resolution image transmitted from the image transmission device and enlarging the region. An interpolation image generation unit;
A display output unit for displaying the interpolated magnified image on the display device by outputting the interpolated magnified image to a display device;
When the enlargement rate included in the enlargement information received by the enlargement acceptance unit is greater than a predetermined value, the second resolution image corresponding to the enlargement information is generated by the high-resolution image generation unit, The second resolution image corresponding to the enlarged information is acquired from a high-resolution image generation unit, and the interpolated enlarged image output to the display device is output to the display unit according to the enlarged information. A switching unit for switching to the second resolution image,
Image output device. Obtain a first resolution image, which is an image of the subject obtained based on photography with a digital microscope,
Receiving enlargement information including an enlargement ratio for a partial area in the acquired first resolution image and a position of the area;
Generating an interpolated enlarged image by enlarging the region by performing pixel interpolation on the region of the first resolution image based on the received enlarged information;
By outputting the interpolated magnified image to a display device, the interpolated magnified image is displayed on the display device,
When the enlargement rate included in the accepted enlargement information is greater than a predetermined value,
Generating a second resolution image corresponding to the enlargement information, which is an image of a higher resolution than the first resolution image, and is an image of the subject obtained based on photographing by the digital microscope,
An image output method for switching the interpolated magnified image output to the display device to the second resolution image corresponding to the magnified information. Obtain a first resolution image, which is an image of the subject obtained based on photography with a digital microscope,
Receiving enlargement information including an enlargement ratio for a partial area in the acquired first resolution image and a position of the area;
Generating an interpolated enlarged image by enlarging the region by performing pixel interpolation on the region of the first resolution image based on the received enlarged information;
By outputting the interpolated magnified image to a display device, the interpolated magnified image is displayed on the display device,
When the enlargement rate included in the accepted enlargement information is greater than a predetermined value,
Generating a second resolution image corresponding to the enlargement information, which is an image of a higher resolution than the first resolution image, and is an image of the subject obtained based on photographing by the digital microscope,
A program for causing a computer to switch the interpolated magnified image output to the display device to the second resolution image corresponding to the magnified information. An image acquisition unit for acquiring a first resolution image that is an image of a subject obtained based on photographing by a digital microscope;
A high-resolution image generation unit that generates a second resolution image that is an image of the subject that is an image having a higher resolution than the first resolution image and is obtained based on photographing by the digital microscope;
An image transmitting apparatus comprising: an image receiving apparatus connected via a communication line;
The image receiving device includes:
An enlargement receiving unit for receiving enlargement information including an enlargement ratio for a partial area in the acquired first resolution image and a position of the area;
Based on the enlargement information received by the enlargement accepting unit, an interpolated enlarged image is generated by performing pixel interpolation on the region of the first resolution image transmitted from the image transmission device and enlarging the region. An interpolation image generation unit;
A display output unit for displaying the interpolated magnified image on the display device by outputting the interpolated magnified image to a display device;
When the enlargement rate included in the enlargement information received by the enlargement acceptance unit is greater than a predetermined value, the second resolution image corresponding to the enlargement information is generated by the high-resolution image generation unit, The second resolution image corresponding to the enlarged information is acquired from a high-resolution image generation unit, and the interpolated enlarged image output to the display device is output to the display unit according to the enlarged information. A switching unit for switching to the second resolution image,
Image receiving device. An enlargement accepting unit that accepts enlargement information including an enlargement ratio with respect to a partial area in the first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and a position of the area;
Interpolation for generating an interpolated enlarged image by performing pixel interpolation on the region of the first resolution image transmitted from the image transmission device and enlarging the region based on the enlargement information received by the enlargement receiving unit An image generator;
A display output unit for displaying the interpolated magnified image on the display device by outputting the interpolated magnified image to a display device;
When the enlargement ratio included in the enlargement information received by the enlargement reception unit is larger than a predetermined value, the image is higher in resolution than the first resolution image, and is obtained based on the photographing by the digital microscope. A second resolution image corresponding to the enlargement information, which is an image of the subject to be generated, is generated by the high resolution image generation unit of the image transmission device, and the second resolution image corresponding to the enlargement information is generated from the high resolution image generation unit. A switching unit that acquires a two-resolution image and causes the display output unit to switch the interpolation enlarged image output to the display device to the second resolution image according to the enlarged information;
An image receiving apparatus comprising: the image transmitting apparatus connected via a communication line,
The image transmission device includes:
An image acquisition unit for acquiring the first resolution image;
The high-resolution image generation unit for generating the second resolution image,
Image transmission device.

Images