JP2018082389A - Image creating system, image creating method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image creating system and the like that can create a high-quality focused image of an object on a virtual focal surface by using a plurality of photographed images.SOLUTION: An image creating system 10 creates a focused image corresponding to a virtual focal surface located between a plurality of artificial point light sources 101aj and a plurality of image sensors 102ak having objects arranged on their surfaces. Every time the respective artificial point light sources 101aj illuminate the objects, the plurality of image sensors 102ak respectively acquire photographed images. The image creating system 10 selects one of the plurality of image sensors 102ak, acquires a plurality of photographed images photographed by the selected image sensor, acquires information on a focal surface located between the plurality of artificial point light sources 101aj and the selected image sensor, creates a focused image of the focal surface by using the information on the focal surface and the plurality of photographed images, and outputs the created focused image of the focal surface.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for generating an image of an object on a virtual focal plane based on a plurality of captured images from a plurality of light sources in a lensless microscope.

  The demand for continuous observation of cultured cells without staining is in many fields where the cultured cells are used in medical and industrial fields, such as production of therapeutic cells and testing of drug efficacy. However, since most of the cells are almost colorless and transparent, it is difficult to understand the three-dimensional structure of the cultured cells by photographing with an optical microscope using transmitted light.

  In Patent Document 1, in order to evaluate the cross-sectional shape of a cell, a large number of images (that is, the focus is shifted in the height direction of the object) that are parallel to the objective lens and have different focus height positions with respect to the object. 3 shows a method of generating a focused image (pseudo-sectional image) of a surface that is not parallel to the objective lens from a large number of captured images).

  By the way, continuous observation of cultured cells is performed in a limited space called an incubator for maintaining a humid environment for culturing cells. For observation in such a humid and limited space, Patent Document 2 discloses a lensless microscope capable of observing minute cells without using a lens. A method is disclosed in which a plurality of images taken by illumination irradiated from a plurality of different positions are overlapped (Ptyography) to increase the resolution.

JP 2013-101512 A US Patent Application Publication No. 2014/0133702

  However, in the method of Patent Document 1, partial images are cut out from images at respective height positions after shooting, and the cut out partial images are joined together. Therefore, discontinuities occur at the joints of the partial images. As a result, the image quality of the pseudo cross-sectional image deteriorates due to discontinuity. Further, if the discontinuous portion is subjected to the blurring process in order to reduce the image quality deterioration due to the discontinuity, the sharpness of the pseudo sectional image is lowered.

  Therefore, the present disclosure provides an image generation system that can generate a high-quality focused image of an object on a virtual focal plane using a plurality of captured images.

  An image generation system according to an aspect of the present disclosure includes a first image sensor having first and second light sources, a surface on which the first object is placed, and a plurality of sensor pixels, and a second object. A second image sensor having a surface on which is mounted and a plurality of sensor pixels, and a virtual focal plane located between the first and second light sources and the first and second image sensors And at least one control circuit for generating a focused image corresponding to each of the first and second image sensors, each time the first and second light sources illuminate, the plurality of sensors. A plurality of captured images of the first and second objects are acquired using a luminance value based on light received by a pixel, and the at least one control circuit is (a1) the first and second Select one of the image sensors and (a (A) acquiring a plurality of captured images captured by the selected image sensor; (a3) acquiring information on the focal plane located between the first and second light sources and the selected image sensor; a4) Using the focal plane information and the plurality of photographed images, the luminance values of the sensor pixels corresponding to the plurality of focused pixels constituting the focused image on the focal plane are acquired, and thereby the focal point is acquired. A focused image of the surface is generated, and (a5) the generated focused image of the focal plane is output.

  An image generation method according to an aspect of the present disclosure is an image generation method for generating an image of an object located on an image sensor, and (b1) each time each of the first and second light sources is illuminated. First and second objects respectively positioned on the first and second image sensors using brightness values based on light received by a plurality of sensor pixels of the first and second image sensors, respectively. Acquiring a plurality of captured images of the object, (b2) selecting one of the first and second image sensors, and (b3) acquiring a plurality of captured images captured by the selected image sensor, (B4) setting a virtual focal plane located between the first and second light sources and the selected image sensor, and (b5) generating a focused image corresponding to the focal plane. The focal plane information and the A plurality of photographed images, and obtaining brightness values of the sensor pixels corresponding to a plurality of focused pixels constituting the focused image of the focal plane, thereby generating a focused image of the focal plane, (B6) An in-focus image of the focal plane is output, and at least one of the (b1) to (b6) is executed by the control circuit.

  A program according to an aspect of the present disclosure is a program to be executed by a computer, and (c1) selects one of the first and second image sensors, and (c2) the first and second images. A plurality of captured images captured by the selected image sensor are acquired from among a plurality of captured images of the first and second objects respectively positioned on the first image sensor, wherein the first and second images are acquired. Each of the plurality of captured images of the object has a luminance value based on light received by the plurality of sensor pixels of each of the first and second image sensors each time the first and second light sources are illuminated. And (c3) setting a virtual focal plane located between the first and second light sources and the selected image sensor, obtained by the first and second image sensors, and (c4) Said scorching Generating a focused image corresponding to a plane, using the focal plane information and the plurality of captured images, and corresponding to a plurality of focused pixels constituting the focused image of the focal plane By acquiring the luminance value of the sensor pixel, a focused image of the focal plane is generated, and (c5) the focused image of the focal plane is output.

  Note that these comprehensive or specific modes may be realized by a recording medium such as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM, and the apparatus, method, integrated circuit, and computer program. Also, any combination of recording media may be realized.

  According to the present disclosure, it is possible to generate a high-quality focused image of an object on a virtual focal plane using a plurality of captured images.

It is a block diagram which shows an example of a functional structure of the image generation system which concerns on embodiment. It is a block diagram which shows an example of a functional structure of the imaging device which concerns on embodiment. It is a section side view showing typically an example of the structure of the tray for cultivation containers concerning an embodiment. It is a top view which shows typically an example of the structure of the tray for culture containers which concerns on embodiment. It is a perspective view which shows typically an example of the culture container which concerns on embodiment. It is a perspective view which shows typically another example of the culture container which concerns on embodiment. It is a top view which shows typically an example of arrangement | positioning of the illuminator which concerns on embodiment. It is a side view which shows typically an example of arrangement | positioning of the illuminator which concerns on embodiment. It is a typical side view which shows the positional relationship of the pseudo point light source of FIG. 7A, and an image sensor. It is a figure which shows an example of the information memorize | stored in the memory | storage part which concerns on embodiment. It is a flowchart which shows an example of operation | movement of the image generation system which concerns on embodiment. It is a schematic diagram which shows an example of the relationship between a coordinate and a focal plane. It is a flowchart which shows an example of operation | movement of the imaging device which concerns on embodiment. It is a flowchart which shows an example of operation | movement of the refocus processing part which concerns on embodiment. It is a schematic diagram explaining the specific example of the refocus process which concerns on embodiment. It is a schematic diagram explaining the specific example of the refocus process which concerns on embodiment. It is a schematic diagram explaining the specific example of the refocus process which concerns on embodiment. It is a schematic diagram explaining the specific example of the refocus process which concerns on embodiment. It is a schematic diagram explaining the specific example of the refocus process which concerns on embodiment. 10 is a cross-sectional side view schematically showing an example of a structure of a culture container tray according to Modification 1. FIG. 6 is a top view schematically showing an example of a structure of a culture container tray according to Modification 1. FIG. It is a sectional side view showing typically an example of the structure of the tray for cultivation containers concerning modification 2. 10 is a top view schematically showing an example of a structure of a culture container tray according to Modification 2. FIG.

  An image generation system according to an aspect of the present disclosure includes a first image sensor having first and second light sources, a surface on which the first object is placed, and a plurality of sensor pixels, and a second object. A second image sensor having a surface on which is mounted and a plurality of sensor pixels, and a virtual focal plane located between the first and second light sources and the first and second image sensors And at least one control circuit for generating a focused image corresponding to each of the first and second image sensors, each time the first and second light sources illuminate, the plurality of sensors. A plurality of captured images of the first and second objects are acquired using a luminance value based on light received by a pixel, and the at least one control circuit is (a1) the first and second Select one of the image sensors and (a (A) acquiring a plurality of captured images captured by the selected image sensor; (a3) acquiring information on the focal plane located between the first and second light sources and the selected image sensor; a4) Using the focal plane information and the plurality of photographed images, the luminance values of the sensor pixels corresponding to the plurality of focused pixels constituting the focused image on the focal plane are acquired, and thereby the focal point is acquired. A focused image of the surface is generated, and (a5) the generated focused image of the focal plane is output.

  According to this aspect, the brightness value of the sensor pixel of the image sensor corresponding to the plurality of focused pixels constituting the focused image on the focal plane is acquired using the virtual focal plane information and the plurality of captured images. Is done. Then, using the acquired luminance value of the sensor pixel, a focused image of the focal plane is generated. A captured image of an object on each image sensor is acquired by each of the first image sensor and the second image sensor, and an image sensor to generate a focused image on the focal plane is selected, and the selected image sensor A focused image based on the captured image is generated. Therefore, it is possible to generate a high-quality focused image while improving the generation processing speed of the focused image.

  For example, the at least one control circuit may further acquire and use the position information of the first and second light sources and the position information of the selected image sensor in generating a focused image of the focal plane. Also good. According to this aspect, the process for generating the focused pixel is facilitated.

  For example, the position information of the first and second light sources and the position information of the selected image sensor are in a first coordinate system independent of the positions of the first and second light sources and the position of the image sensor. The focal plane information is based on a second coordinate system based on the selected image sensor, and the at least one control circuit generates the focused image of the focal plane, the first coordinate system and You may perform coordinate conversion between said 2nd coordinate systems.

  According to this aspect, by using the first coordinate system, the position information of the light source and the position information of the image sensor can be fixed information that is not affected by these positions. Therefore, setting of each position information is easy. By using the second coordinate system, the amount of processing required to calculate the brightness value of the focused pixel is reduced. Therefore, the processing speed is improved.

  For example, the at least one control circuit uses the luminance value of each sensor pixel in which the position of the light source, the position of the focused pixel, and the position of the sensor pixel are arranged in a straight line. The brightness value of the focal pixel may be calculated. According to this aspect, the luminance value of the intersection point between the straight line connecting the position of the focused pixel and the light source on the focal plane and the light receiving surface of the image sensor can be applied to the luminance value of the focused pixel. Therefore, each focused pixel of the focused image on the virtual focal plane can reflect the luminance values of a plurality of captured images corresponding to the pixel, and a high-quality focused image of the object is generated. Can do.

  An image generation method according to an aspect of the present disclosure is an image generation method for generating an image of an object located on an image sensor, and (b1) each time each of the first and second light sources is illuminated. Using a luminance value based on light received by a plurality of sensor pixels of each of the first and second image sensors, the first and second objects located on the first and second image sensors, respectively. Acquiring a plurality of captured images, (b2) selecting one of the first and second image sensors, (b3) acquiring a plurality of captured images captured by the selected image sensor, (b4) ) Setting a virtual focal plane located between the first and second light sources and the selected image sensor, and (b5) generating a focused image corresponding to the focal plane, Focal plane information and the plurality of Using the shadow image, the brightness value of the sensor pixel corresponding to the plurality of focused pixels constituting the focused image on the focal plane is obtained, thereby generating the focused image on the focal plane, (b6 ) Output a focused image of the focal plane, and at least one of the (b1) to (b6) is executed by the control circuit.

  A program according to an aspect of the present disclosure is a program to be executed by a computer, and (c1) selects one of the first and second image sensors, and (c2) the first and second images. A plurality of photographed images photographed by the selected image sensor are acquired from a plurality of photographed images of the first and second objects respectively positioned on the sensor, wherein the first and second objects Each time a plurality of captured images of an object illuminate each of the first and second light sources, a luminance value based on light received by the plurality of sensor pixels of each of the first and second image sensors is used. (C3) set a virtual focal plane located between the first and second light sources and the selected image sensor, and (c4) the focus. On the face A sensor pixel corresponding to a plurality of focused pixels constituting the focused image on the focal plane using the focal plane information and the plurality of captured images. To obtain a focused image of the focal plane, and (c5) output the focused image of the focal plane.

  These general or specific aspects may be realized by a recording medium such as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. The apparatus, the method, the integrated circuit, and the computer program Also, any combination of recording media may be realized.

  Hereinafter, an image generation system according to an aspect of the present disclosure will be specifically described with reference to the drawings. It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the scope of the claims. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements. In the following description of the embodiments, expressions with “substantially” such as substantially parallel and substantially orthogonal may be used. For example, “substantially parallel” not only means completely parallel, but also means substantially parallel, that is, including a difference of, for example, several percent. The same applies to expressions involving other “abbreviations”.

(Embodiment)
In the image generation system according to the embodiment, a plurality of pseudo point light sources sequentially illuminate the object on the object located on the image sensor, and a plurality of captured images obtained by photographing the object each time the object is illuminated. Is used to generate an image of an object on a virtual focal plane located between the plurality of pseudo point light sources and the image sensor. The plurality of pseudo point light sources emit diffused light from a sufficiently small surface that can be regarded as a plurality of point light sources formed by an illuminator. The plurality of rays in the diffused light do not intersect each other. Hereinafter, an image corresponding to a virtual focal plane generated using a plurality of captured images is also referred to as a focused image. Generating a focused image of an object corresponding to a virtual focal plane using a plurality of captured images is also referred to as refocus processing. In the refocus processing, a pixel in a virtual focal plane may be obtained using the imaged pixel.

[1-1. Configuration of image generation system]
[1-1-1. Schematic configuration of image generation system]
In the present embodiment, a case will be described in which the image generation system is applied to generation of an image of a cultured cell or the like placed on a culture container in a culture container tray. The target to which the image generation system is applied is not limited to this. The image generation system as described above includes a plurality of image sensors at the bottom of the culture container tray, and an illuminator at the lid of the culture container tray. The culture container is configured such that cultured cells and the like placed on the bottom wall can be observed from below through the bottom wall. Further, the culture container is fixed in the culture container tray by being fitted into a recess or frame having a shape and size matching the outer shape of the culture container, for example. The culture container tray is installed in an incubator in a state in which a culture container containing cultured cells and a medium is fixed with a frame. The culture container as described above is positioned between the image sensor at the bottom of the culture container tray and the illuminator at the lid. The light emitted from the lid illuminator passes through the culture container, the culture medium inside the culture container, the cultured cells, the cultured cell mass, or the cultured tissue inside the culture container and reaches the image sensor. The image sensor performs imaging using transmitted light from the illuminator on the lid. The culture container may be a petri dish type, dish type, or multiwell plate type container.

[1-1-2. Overall configuration of image generation system]
With reference to FIG. 1, the overall configuration of an image generation system 10 according to the embodiment will be described. FIG. 1 is a functional block diagram of an image generation system 10 according to the embodiment. An image generation system 10 illustrated in FIG. 1 includes an imaging device 100A, an image generation device 100B, a storage unit 120, and a display unit 150. The image generation system 10 further specifies a first recording unit 111 that stores information on a predetermined focal plane, a second recording unit 121 that records information on a pixel that has undergone refocus processing, and a focal plane. And an input unit 112 that receives input of designation information to be performed.

[1-1-3. Configuration of photographing apparatus]
First, the configuration of the photographing apparatus 100A will be described with reference to FIGS. FIG. 2 is a functional block diagram illustrating an example of a detailed configuration of the imaging apparatus 100A according to the embodiment. The photographing apparatus 100 </ b> A includes an illuminator 101, an image sensor 102, and a photographing control unit 103. The photographing apparatus 100A acquires a photographed image (photographic image) of an object. Here, the photographing apparatus 100A does not have a focus lens.

  The illuminator 101 includes a plurality of point light sources 101aj (j = 1, 2,..., M). In the present embodiment, the point light source is a light source that emits diffused light from a sufficiently small surface that can be regarded as a point light source.

  The image sensor 102 includes a plurality of image sensors 102ak (k = 1, 2,..., N). The image sensor 102ak acquires a captured image based on the intensity of light detected by a plurality of sensor pixels included in the image sensor 102ak, for example, luminance. The sensor pixel is disposed on the light receiving surface of the image sensor 102ak, and acquires the intensity of light emitted from the plurality of point light sources 101aj. An example of the image sensor 102 is a CMOS (Complementary Metal-Oxide Semiconductor) image sensor or a CCD (Complementary Metal-Oxide Semiconductor) image sensor.

  The imaging control unit 103 controls light irradiation by the illuminator 101 and imaging by the image sensor 102. Specifically, the imaging control unit 103 controls the order in which the plurality of point light sources 101aj of the illuminator 101 emit light, and the time interval at which the plurality of point light sources 101aj emits light. The imaging control unit 103 may be configured by a computer system (not shown) including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read-Only Memory), and the like. The functions of some or all of the components of the imaging control unit 103 may be achieved by the CPU executing a program recorded in the ROM using the RAM as a working memory. In addition, some or all of the functions of the components of the imaging control unit 103 may be achieved by a dedicated hardware circuit. The program may be provided as an application through communication via a communication network such as the Internet, communication according to a mobile communication standard, or the like.

  The object to be imaged is, for example, a plurality of translucent substances arranged directly or indirectly on the surface constituting the light receiving surface of each of the image sensors 102ak. The object is placed on the plurality of sensor pixels of the image sensor 102ak. The surface of the image sensor 102ak on which the object is placed includes the surface above the sensor pixels of the image sensor 102. On the surface of the image sensor 102ak, the plurality of substances may be positioned three-dimensionally overlapping. A specific example of the substance is a cell. In addition, at least the illuminator 101 and the image sensor 102 in the imaging apparatus 100A are included in the culture container tray 20.

  FIG. 3 is a cross-sectional side view schematically showing an example of the structure of the culture container tray 20 according to the embodiment, and shows a cross-sectional view of the culture container tray 20 cut in the vertical direction. Referring to FIG. 3, an example of a tray for a culture container such as a petri dish type or a dish type is shown. FIG. 3 shows an example in which two petri dish culture containers 30a1 and 30a2 are fixed to one culture container tray 20.

  The culture container tray 20 includes a bottom portion 21 and a lid portion 22 that can be coupled and separated from each other. In the following description, the direction from the bottom portion 21 toward the lid portion 22 is referred to as “upward”, and the opposite direction is referred to as “downward”. In the member, a portion located above may be referred to as an upper portion or a top portion, and a portion located below may be referred to as a lower portion or a bottom portion. The bottom 21 includes container position fixing frames 21a1 and 21a2. Each of the container position fixing frames 21a1 and 21a2 is configured such that the petri dish type culture containers 30a1 and 30a2 are fitted in recesses formed inside thereof, thereby fixing the culture containers 30a1 and 30a2 to the bottom portion 21. Image sensors 102a1 and 102a2 are arranged at the bottoms of the container position fixing frames 21a1 and 21a2, respectively. The image sensors 102a1 and 102a2 are arranged at positions facing the bottom walls of the culture vessels 30a1 and 30a2 arranged in the vessel position fixing frames 21a1 and 21a2, respectively. The image sensors 102a1 and 102a2 may be arranged so as to be close to or in contact with the bottom walls of the culture vessels 30a1 and 30a2, respectively. In the present embodiment, an example in which two petri dish culture containers are arranged in one culture container tray 20 will be described, but the number of culture containers is one or more for one culture container tray 20. Any number can be used. Moreover, the culture container arrange | positioned at the tray 20 for culture containers is not limited to a petri dish type. The culture container tray 20 is provided with an image sensor 102 corresponding to each culture container that can be arranged. The illuminator 101 is disposed on the lid portion 22 of the culture container tray 20.

  With reference to FIG. 3, FIG. 4, FIG. 5A and FIG. 5B, further detailed configurations of the culture container tray 20 and the culture container will be described. FIG. 4 is a top view schematically showing an example of the structure of the culture container tray 20 according to the embodiment, and is a view of the bottom 21 of the culture container tray 20 of FIG. 3 as viewed from above. FIG. 5A is a perspective view schematically showing an example of the culture container according to the embodiment, and shows an example of the petri dish type culture containers 30a1 and 30a2. FIG. 5B is a perspective view schematically showing another example of the culture container according to the embodiment, and shows an example of the flask-type culture container 30b.

  Referring to FIG. 5A, only the bottom members of the petri dish type culture containers 30a1 and 30a2 constituted by the lid member and the bottom member are shown. The bottom members of the petri dish type culture vessels 30a1 and 30a2 have a bottomed cylindrical shape, and have an observation glass bottom 31 on the bottom wall. Thereby, the image sensors 102a1 and 102a2 can photograph the object placed on the culture vessels 30a1 and 30a2 through the observation glass bottom 31.

  Referring to FIG. 5B, an example in which the culture container is a culture flask type, that is, a flask type is shown. The flask-type culture vessel 30b has a box shape and has an opening that can be opened and closed on the side. The flask-type culture vessel 30b includes an observation glass bottom 31 on the bottom wall that is the lower side during culture.

  In FIG. 3, the bottomed cylindrical container position fixing frames 21a1 and 21a2 matched to the thin cylindrical petri dish shape are shown as the container position fixing frames of the petri dish type culture containers 30a1 and 30a2. For example, the petri dish shape is a rectangular parallelepiped. In this case, it is desirable that the shape of the container position fixing frame is a bottomed rectangular tube shape that matches the outer shape of the petri dish. In the case of the flask-type culture container 30b shown in FIG. 5B, it is desirable that the container position fixing frame has a shape that matches the bottom wall of the flask-type culture container 30b and its outer shape.

  The container position fixing frames 21a1 and 21a2 cover the entire circumference or part of the side surfaces of the culture containers 30a1 and 30a2, thereby fixing the culture containers 30a1 and 30a2 at predetermined positions on the culture container tray 20. 3 and 4, the container position fixing frames 21a1 and 21a2 each have a planar shape in which the culture containers 30a1 and 30a2 just fit in the bottom portion 21 of the culture container tray 20, and the heights of the culture containers 30a1 and 30a2. This is realized by providing a recess having a depth shallower than that. The planar shape of the recess is a shape when viewed from the lid portion 22 toward the bottom portion 21. Further, the container position fixing frame may be realized by raising a wall-shaped frame from the surface of the bottom portion 21 in addition to providing a recess.

  The illuminator 101 includes a plurality of point light sources arranged so as to form a plane, a curved surface, or the like. In the present embodiment, the plurality of point light sources of the illuminator 101 are a plurality of virtual point light sources, and more specifically, a plurality of pseudo point light sources having a small light source area. The pseudo point light source is configured by a pinhole having a diameter of several μm to several hundred μm, for example. Such a pseudo point light source emits light by irradiating light from a light source such as an illumination device arranged on the opposite side of the irradiation direction with respect to the pinhole through the pinhole. The light emitted by such a pseudo point light source is diffused light. A light source such as an illumination device is disposed behind the pinhole with respect to the image sensors 102a1 and 102a2. In addition, the structure of a some point light source is not limited to the above-mentioned structure, What kind of structure may be sufficient. For example, the plurality of point light sources may be composed of a plurality of light emitting elements, or may be composed of light emitting pixels of a display.

  Referring to FIG. 6, an example of an array of pseudo point light sources 101aj (j = 1, 2,..., M), which are pseudo point light sources of the illuminator 101, is shown. FIG. 6 is a top view schematically showing an example of the arrangement of the illuminators 101 according to the embodiment. FIG. 6 shows a positional relationship between the lid portion 22 of the culture vessel tray 20 including the pseudo point light source 101aj, the culture vessels 30a1 and 30a2, and the image sensors 102a1 and 102a2 of the bottom portion 21. In the present embodiment, the plurality of pseudo point light sources 101aj are arranged in a lattice pattern along the upper wall on the upper wall of the lid portion 22. More specifically, the plurality of pseudo point light sources 101aj are arranged at an equal pitch.

  The relationship between the plurality of pseudo point light sources 101aj and the image sensors 102a1 and 102a2 will be described with reference to FIGS. 7A and 7B. FIG. 7A is a side view schematically showing an example of the arrangement of the illuminators 101 according to the embodiment, and is a schematic view of the illuminators 101 configured as shown in FIG. FIG. 7B is a schematic side view showing the positional relationship between the pseudo point light source 101aj and the image sensors 102a1 and 102a2 in FIG. 7A. As shown in FIG. 7B, in the illuminator 101, a plurality of pseudo point light sources 101aj (j = 1, 2,..., M) are planes parallel to the light receiving surfaces that are the surfaces of the image sensors 102a1 and 102a2. They are arranged at different positions on 101H. The arrangement of the plurality of pseudo point light sources 101aj is not limited to this and may be any arrangement.

  As shown in FIGS. 7A and 7B, the plurality of pseudo point light sources 101aj are directed so as to irradiate light toward the image sensors 102a1 and 102a2, specifically, directed so as to irradiate light downward. Has been. Lights from the plurality of pseudo point light sources 101aj are incident on the sensor pixels on the light receiving surfaces of the image sensors 102a1 and 102a2 from different directions. The incident direction is determined according to the relative position between the positions of the image sensors 102a1 and 102a2 and the position of the pseudo point light source 101aj.

  Each of the plurality of pseudo point light sources 101aj is realized by arranging a light shielding plate having a plurality of pinholes in the vicinity of the LED light source between the LED light source and the image sensors 102a1 and 102a2, for example. The size of the pinhole is limited by the sensor pixel pitch of the image sensors 102a1 and 102a2, the distance between the image sensors 102a1 and 102a2 and the pinhole, and the distance from the image sensors 102a1 and 102a2 at which the focused image is generated. Is done.

  In the present embodiment, the control board or the like that controls the image sensors 102a1 and 102a2 is disposed in the bottom portion 21, but may be disposed outside the bottom portion 21 or away from the bottom portion 21. The control board or the like supplies power to the image sensors 102a1 and 102a2, controls shooting of the image sensors 102a1 and 102a2, and stores information such as memory for storing image information captured by the image sensors 102a1 and 102a2 or information such as image information. May include a communication device for transmitting and receiving. In the present embodiment, the control board or the like that controls the illuminator 101 is disposed in the lid portion 22, but may be disposed outside the lid portion 22 or away from the lid portion 22. The control board or the like performs power supply to a lighting device such as an LED light source of the illuminator 101, and controls the lighting device.

[1-1-4. Detailed Configuration of Image Generation Device]
Next, a detailed configuration of the image generation device 100B will be described. In the present embodiment, the image generation device 100B is realized by at least one control circuit. As illustrated in FIG. 1, the image generation device 100 </ b> B includes a focal plane determination unit 110, a refocus processing unit 130, an image generation unit 140, and an image sensor determination unit 160. Each component of the image generation device 100B may be configured by a computer system (not shown) including a CPU, a RAM, a ROM, and the like. Some or all of the functions of each component may be achieved by the CPU executing a program recorded in the ROM using the RAM as a working memory. In addition, some or all of the functions of each component may be achieved by a dedicated hardware circuit. Each component may be composed of a single element that performs centralized control, or may be composed of a plurality of elements that perform distributed control in cooperation with each other. The program may be provided as an application through communication via a communication network such as the Internet, communication according to a mobile communication standard, or the like.

  The image sensor determination unit 160 is realized by, for example, a control circuit or a processor, and selects and determines an image sensor that generates an image from the plurality of image sensors 102ak. Specifically, the image sensor determination unit 160 determines the image sensor 102ak according to information input from the outside via the input unit 112, for example.

  The focal plane determination unit 110 is realized by, for example, a control circuit or a processor, and determines a virtual focal plane located between the illuminator 101 and the plurality of image sensors 102ak. Specifically, the focal plane determination unit 110 determines a focal plane based on information on a predetermined focal plane recorded in the first recording unit 111, for example. Further, for example, the focal plane determination unit 110 may determine a focal plane according to information input from the outside via the input unit 112.

  The storage unit 120 is realized by, for example, a semiconductor memory or a hard disk drive, and the image captured by the plurality of image sensors 102ak is used as the positional information of the image sensor 102ak used for the imaging and the pseudo point light source 101aj of the illuminator 101. Store with location information. The first recording unit 111 and the second recording unit 121 may also have the same configuration as the storage unit 120.

  FIG. 8 shows an example of contents stored in the storage unit 120. For each image file photographed by the photographing apparatus 100A, the storage unit 120 stores the position information of the image sensor 102ak used for acquiring the image file and the position information of the pseudo point light source 101aj used for acquiring the image file. Is stored in combination. In the example of FIG. 8, the positional information of the pseudo point light source 101aj and the position information of the image sensor 102ak are shown in a coordinate system based on the culture container tray 20 to which the pseudo point light source 101aj and the image sensor 102ak are fixed.

  The coordinate system based on the culture container tray 20 is fixed regardless of the positions of the plurality of pseudo-point light sources 101aj and the plurality of image sensors 102ak, that is, based on components or positions that do not depend on these positions. It is also a coordinate system. Such a coordinate system is formed by, for example, the x- and y-axes in the plane along the bottom wall 21b of the bottom 21 and the surfaces of the image sensors 102a1 and 102a2 shown in FIG. 4, and the z-axis perpendicular to the plane. May be. In the present embodiment, as shown in FIG. 4, the x coordinate and y coordinate of the point P near the corner of the bottom wall 21b are set to x = 0 and y = 0, respectively. Furthermore, the point P is located on a plane passing through the surfaces of the image sensors 102a1 and 102a2, and its z coordinate is z = 0. Note that the point P is located in the vicinity of the lower left corner of the bottom wall 21b close to the image sensor 102a1 in FIG.

  Hereinafter, the position information of the pseudo point light source is also referred to as illumination position information, and the position information of the image sensor is also referred to as image sensor position information. The illumination position information and the image sensor position information are stored together with the file ID of the image file, and are combined through the image data and the file ID. The illumination position information and the image sensor position information may be recorded in a part of the image file (for example, header information).

  The refocus processing unit 130 is realized by, for example, a control circuit or a processor, and is specified for each of a plurality of images, position information of the plurality of image sensors 102ak, position information of the plurality of pseudo point light sources 101aj, and the image sensors 102ak. From the information on the virtual focal plane, the intensity of light for each focused pixel constituting the focused image on the focal plane is calculated. Details of this refocus processing will be described later.

  The image generation unit 140 is realized by, for example, a control circuit or a processor, and generates a focused image on the focal plane from the luminance value for each pixel calculated by the refocus processing unit 130.

  The display unit 150 is realized by a display, and displays the focused image generated by the image generation unit 140. The display unit 150 may be configured by a display panel such as a liquid crystal panel, organic or inorganic EL (Electro Luminescence).

[1-2. Operation of image generation system]
[1-2-1. Overall operation of the image generation system]
Next, the overall operation of the image generation system 10 configured as described above will be described with reference to FIGS. FIG. 9 is a flowchart illustrating an example of the operation of the image generation system 10 according to the embodiment. FIG. 10 is a schematic diagram illustrating an example of the relationship between the coordinates and the focal plane.

(Step S1100)
First, the imaging control unit 103 of the imaging apparatus 100A illuminates the object using the plurality of pseudo point light sources 101aj (j = 1, 2,..., M) of the illuminator 101 in order, and A plurality of images are taken by a plurality of image sensors 102. For example, with all of the plurality of pinholes constituting the pseudo point light source 101aj closed in advance, illumination is performed with an illumination device such as an LED light source on the back of a light shielding plate having a plurality of pinholes, and the pinholes are sequentially arranged one by one The plurality of pseudo point light sources 101aj illuminate in turn. The imaging control unit 103 records the intensity of light reaching the light receiving surfaces of the plurality of image sensors 102ak, specifically the image sensors 102a1 and 101a2, each time the pseudo point light source 101aj illuminates the object. The captured image of the first object 51 (see FIGS. 3 and 7A) on the image sensor 102a1 and the captured image of the second object 52 (see FIGS. 3 and 7A) on the image sensor 102a2 are acquired. The acquired captured images of the first object 51 and the second object 52 illuminate the first object 51 and the second object 52 at the time of shooting, and the position information of the image sensor that acquired each captured image. The information is stored in the storage unit 120 together with the position information of the pseudo point light source 101aj. Here, the positions of the pseudo point light sources 101aj and the positions of the image sensors 102a1 and 102a2 are fixed with respect to the culture container tray 20. Position information of the image sensors 102a1 and 102a2 and the pseudo point light sources 101aj is determined in advance. Details of the photographing process will be described later.

(Step S1200)
The image sensor determination unit 160 selects an image sensor that is a target for generating an image. Specifically, the image sensor determination unit 160 generates an image of the first object 51 including the focused image based on the captured image of the image sensor 102a1, or the focused image based on the captured image of the image sensor 102a2. Whether to generate an image of the second object 52 including. That is, the image sensor determination unit 160 determines whether to generate an image captured by the image sensor 102a1 or to generate an image captured by the image sensor 102a2. For example, the image sensor determination unit 160 receives one of the plurality of image sensors 102ak, specifically one of the image sensors 102a1 and 102a2, based on the designation information that designates the image sensor received from the user by the input unit 112. Select.

(Step S1300)
The focal plane determination unit 110 determines a focal plane corresponding to the selected image sensor 102ak. Specifically, the focal plane determination unit 110 determines the focal plane position and tilt (angle) with respect to the selected image sensor 102ak. For example, the focal plane determination unit 110 may determine the focal plane based on focal plane information corresponding to each predetermined image sensor 102ak stored in the first recording unit 111. Alternatively, the focal plane determination unit 110 may determine the focal plane based on the designation information that designates the focal plane received from the user by the input unit 112. The focal planes corresponding to the plurality of image sensors 102ak may be the same or different among the plurality of image sensors 102ak.

  The focal plane corresponds to a virtual plane on which a focused image is generated. That is, the plurality of pixels included in the focused image of the object on the focal plane correspond one-to-one to the plurality of points on the focal plane.

For example, the focal plane determination unit 110 determines the focal plane using the focal plane angle and position. The angle and position of the focal plane are defined by, for example, the X _k Y _k Z _k space shown in FIG. In FIG. 10, the X _k Y _k plane coincides with the light receiving surface of the image sensor 102ak selected in step S1200. The Z _k axis is orthogonal to the light receiving surface of the image sensor 102ak. At this time, the angle of the focal plane is defined as an angle with respect to the X _k axis and the Y _k axis in the X _k Y _k Z _k space with the center of the light receiving surface of the image sensor 102ak as an origin, for example. The position of the focal plane is defined by the coordinates of the central point of the focal plane.

(Step S1400)
The refocus processing unit 130 performs refocus processing based on a plurality of captured images of the selected image sensor 102ak, position information of the plurality of pseudo point light sources 101aj, and information on a focal plane corresponding to the selected image sensor 102ak. To obtain the brightness of each point on the focal plane. Details of the refocus processing will be described later. By this refocus processing, a focused image corresponding to the focal plane can be generated.

(Step S1500)
The image generation unit 140 generates image data that can be output to a display or the like based on the result of the refocus process performed in step S1400. The image generation unit 140 outputs the generated image data to the display unit 150.

(Step S1600)
The display unit 150 displays the image generated in step S1500.

(Step S1700)
The image sensor determination unit 160 determines whether or not to generate an image captured by an image sensor other than the image sensor 102ak selected in step S1200 among the image sensors included in the image sensor 102. Specifically, for example, when the image sensor 102a1 is selected in step S1200, in step S1700, the image sensor determination unit 160 determines whether to generate an image captured by the image sensor 102a2. For example, the image sensor determination unit 160 determines whether new designation information for designating another image sensor has been received by the input unit 112. If new designation information is received (YES in step S1700), image sensor determination unit 160 determines that generation of an image photographed by another image sensor is requested, and returns to step S1200. In step S1200, the image sensor determination unit 160 selects the image sensor designated by the new designation information. On the other hand, when new designation information is not received (No in step S1700), image sensor determination unit 160 determines that generation of an image photographed by another image sensor is not requested, and ends the process. To do.

[1-2-2. Shooting process]
Here, details of the operation of the photographing apparatus 100A in step S1100 will be described. FIG. 11 is a flowchart illustrating an example of the operation of the imaging apparatus 100A according to the embodiment.

(Step S1101)
The imaging control unit 103, for example, a list of predetermined positions of a plurality of pseudo point light sources 101aj stored in the storage unit 120 or the like, or positions of a plurality of pseudo point light sources 101aj specified by an external input (not shown). With reference to a list (hereinafter, all lists are referred to as a pseudo point light source position list), it is determined whether or not shooting of an object illuminated from the position of each pseudo point light source 101aj is completed. In the following description, the position of the pseudo point light source 101aj is also referred to as a pseudo point light source position. Specifically, the imaging control unit 103 has completed imaging of the object by each image sensor 102ak at the time of illumination by the pseudo point light source 101aj at all the pseudo point light source positions included in the pseudo point light source position list. It is determined whether or not. The object is disposed on the surface of the image sensor 102ak.

  Here, when shooting by all the image sensors 102ak by illumination from all the pseudo point light source positions included in the pseudo point light source position list is completed (yes in step S1101), the shooting control unit 103 proceeds to step S1200. move on. On the other hand, if shooting with illumination from any of the pseudo point light source positions in the pseudo point light source position list has not been completed (no in step S1101), the shooting control unit 103 proceeds to step S1102.

(Step S1102)
The imaging control unit 103 selects a pseudo point light source position that has not been illuminated from a plurality of pseudo point light source positions included in the pseudo point light source position list, and outputs a control signal to the illuminator 101. In the pseudo point light source position list, each pseudo point light source position is indicated by, for example, coordinate values in the xyz space set for the culture vessel tray 20 as shown in FIG. Alternatively, each pseudo point light source position is indicated by a number assigned to each pseudo point light source position. The selection of the pseudo point light source position is performed, for example, in ascending order of the list.

(Step S1103)
The illuminator 101 starts illuminating the object according to the control signal output from the imaging control unit 103 in step S1102. That is, the pseudo point light source 101aj corresponding to the pseudo point light source position selected in step S1102 starts light irradiation.

(Step S1104)
While the plurality of objects arranged on each of the plurality of image sensors 102ak are illuminated by the light emitted from the pseudo point light source 101aj, all of the plurality of image sensors 102ak are from the pseudo point light source 101aj. Each image formed by light transmitted through an object on each surface is acquired.

(Step S1105)
Thereafter, the imaging control unit 103 outputs a control signal to the illuminator 101, and stops illumination of the plurality of objects. The stop of the illumination may not be performed in accordance with a control signal from the imaging control unit 103. For example, the illuminator 101 may measure the time length from the start of illumination and actively stop the illumination when the measured time length exceeds a predetermined time length. Alternatively, after all the image sensors 102ak have finished acquiring images in step S1104, the image sensor 102ak may output a control signal for stopping the illumination to the illuminator 101.

(Step S1106)
Next, the imaging control unit 103 includes image sensor position information indicating the position of each image sensor 102ak, an image acquired by each image sensor 102ak in step S1104, and position information of the pseudo point light source 101aj illuminated in step S1103. Is output to the storage unit 120. The storage unit 120 stores image data, image sensor position information, and pseudo point light source position information in association with each other. The imaging control unit 103 returns to step S1101 after step S1106.

  By repeating the processing from step S1101 to step S1106, all the objects are sequentially irradiated with light from the pseudo point light sources 101aj corresponding to all the pseudo point light source positions included in the pseudo point light source position list. A plurality of image sensors 102ak simultaneously acquire images each time light is irradiated.

[1-2-3. Refocus processing]
Further, details of the operation of the refocus processing unit 130 in step S1300 will be described. FIG. 12 is a flowchart illustrating an example of the operation of the refocus processing unit 130 according to the embodiment. 13 to 17 are schematic diagrams for explaining a specific example of the calculation method of the refocus processing. The operation of the refocus processing unit 130 will be described below with reference to the flowchart of FIG.

(Step S1401)
The refocus processing unit 130 acquires information on the image sensor acquired in step S1200, that is, information specifying one of the image sensors 102ak selected by the image sensor determination unit 160. The information may include, for example, an image sensor number, image sensor position information, and the like.

(Step S1402)
The refocus processing unit 130 acquires the focal plane information determined in step S1300 from the focal plane determination unit 110. The focal plane information includes, for example, a coordinate value at the center of the focal plane and a value indicating the tilt of the focal plane. As shown in FIG. 10, the tilt of the focal plane is represented by, for example, an angle formed by the intersection of the focal plane and the X _k Z _k plane with the X _k axis. Further, for example, the tilt of the focal plane is represented by an angle formed by the intersection of the focal plane and the Y _k Z _k plane with the Y _k axis. The coordinate value of the center of the focal plane is the coordinate value of a point on the focal plane corresponding to the center pixel of the focused image.

For example, referring to FIG. 13, an example of a positional relationship among the object 1000 to be photographed, the pseudo point light sources 101a1 and 101a2 of the photographing apparatus 100A, and the image sensor 102ak is shown. FIG. 13 shows an example of a cross-sectional view of the imaging apparatus 100A and the object 1000 on the X _k Z _k plane. The object 1000 is located between the pseudo point light sources 101a1 and 101a2 and the image sensor 102ak, and is located on the image sensor 102ak. In step S1402, the refocus processing unit 130 acquires information on the focal plane 1100.

(Step S1403)
The refocus processing unit 130 converts the shooting coordinate system used at the time of shooting in step S1100 into a refocus processing coordinate system in order to reduce the amount of calculation related to the refocus processing and improve the processing speed. . The imaging coordinate system in step S1100 is a coordinate system based on the culture container tray 20, and is the xyz coordinate system based on the x-axis, y-axis, and z-axis described with reference to FIG. This coordinate system is used to represent the illumination position information of the pseudo point light source 101aj of the illuminator 101 and the image sensor position information of the image sensor 102ak. For example, as shown in FIG. 4, the coordinate system has a point P near the corner of the culture vessel tray 20 as an origin. Note that the z coordinate (that is, z = 0) of the point P is the same as the z coordinate of the surface position of the image sensor 102ak.

The coordinate system for the refocus processing is a coordinate system based on the position of the image sensor specified in step S1401, and X _k Y _k Z based on the X _k axis, the Y _k axis, and the Z _k axis described with reference to FIG. _k coordinate system. For example, when the image sensor 102a1 is specified in step S1401, the coordinate system based on the image sensor 102a1 is the X ₁ Y ₁ Z ₁ coordinate system. In this coordinate system, for example, as shown in FIG. 4, a corner point P1 on the surface of the image sensor 102a1 is set as the origin. This point P1 is located at the upper left corner of the image sensor 102a1 in FIG. Furthermore, the Z ₁ coordinate of the point P1 (that is, Z ₁ = 0) is the same as the Z ₁ coordinate of the surface position of the image sensor 102a1.

  The image sensor position information of the image sensor 102a1 stored in the storage unit 120 is stored in step S1106, and is, for example, information on the position of a corner point on the surface of the image sensor 102a1. In the present embodiment, as shown in FIG. 4, the position information of the corner point P1 on the surface of the image sensor 102a1 is the image sensor position information of the image sensor 102a1.

Then, in the coordinate transformation, and the point P of the corners of the culture container tray 20 which is the origin of the xyz coordinate _system, X _k Y _{k Z k} coordinate system, the image sensor 102a1 is the origin of the words _{X 1 Y} 1 _Z 1 coordinate system The difference in position from the corner point P1 is used as a coordinate conversion value. In the present embodiment, the point P and the point P1 are at the same height position, and the respective z coordinate and Z ₁ coordinate are zero.

For example, when the unit of two coordinate systems is the pitch of the sensor pixels of the image sensor, as shown in FIG. 8, the coordinates in the xyz coordinate system of the position of the image sensor 102a1 stored in step S1106 are (1200 , 720, 0), and the coordinates in the xyz coordinate system of one position of the pseudo point light source are (1300, 880, 9300). When these coordinates are converted into the X ₁ Y ₁ Z ₁ coordinate system, the coordinates of the position of the image sensor 102a1 are (0, 0, 0), and the coordinates of the pseudo point light source are (100, 160, 9300). It becomes. Similarly, all pseudo point light source position information is converted into the X ₁ Y ₁ Z ₁ coordinate system.

(Step S1404)
The refocus processing unit 130 determines whether or not the refocus processing has been completed for all pixels included in the focused image corresponding to the focal plane acquired in step S1402. Here, the refocus processing means processing from step S1404 to step S1411.

  When the refocus processing has been completed for all the pixels included in the focused image (yes in step S1404), the refocus processing unit 130 ends the refocus processing (proceeds to step S1500).

  If the refocus processing has not been completed for any pixel included in the focused image (no in step S1404), the refocus processing unit 130 continues the refocus processing (proceeds to step S1405).

  The focused image includes a plurality of pixels that are focused pixels. The plurality of pixels included in the focused image correspond one-to-one to the plurality of points on the focal plane. Referring to FIG. 14, an example of a plurality of points 1102a to 1102e on the focal plane 1100 corresponding to a plurality of pixels included in the focused image is shown. A plurality of points 1102a to 1102e on the focal plane 1100 shown in FIG. 14 are points on the object 1000, but points not on the object 1000 may correspond to pixels of the focused image.

(Step S1405)
The refocus processing unit 130 selects one pixel from a plurality of pixels included in the focused image. One pixel selected here is a pixel that has not yet been refocused among a plurality of pixels included in the focused image. Note that the initial value of the pixel value of the focused image is zero.

  For example, the second recording unit 121 illustrated in FIG. 1 stores information on pixels that have already undergone refocus processing in the focused image. After the process in step S <b> 1411 described later, the refocus processing unit 130 records information on the refocused pixel in the second recording unit 121. The refocus processing unit 130 refers to the pixel information recorded in the second recording unit 121 and selects a pixel that has not been refocused yet. Hereinafter, a case where a pixel corresponding to the point 1102a is selected as illustrated in FIG. 15 will be described. A pixel corresponding to the point 1102a is also referred to as a selected pixel.

(Step S1406)
The refocus processing unit 130 determines whether or not the addition processing for all pseudo point light source positions has been completed. Here, the addition processing means processing from step S1406 to step S1411. If the addition process for all pseudo point light source positions has been completed (yes in step S1406), the process of the refocus processing unit 130 returns to step S1404. On the other hand, if the addition process for any pseudo point light source position has not ended (NO in step S1406), the refocus processing unit 130 continues the addition process (proceeds to step S1407).

(Step S1407)
The refocus processing unit 130 selects a pseudo point light source position for which the addition process has not yet been completed from all the pseudo point light source positions used for imaging.

(Step S1408)
The refocus processing unit 130 calculates the position of a point where a straight line passing through the selected pseudo point light source position and the position of the selected pixel in the focal plane intersects the light receiving surface of the image sensor 102ak. For example, referring to FIG. 16, an example is shown in which a straight line 1200 passing through the position of the pseudo point light source 101a1 and the point 1102a corresponding to the selected pixel intersects the light receiving surface of the image sensor 102ak at the intersection 1103a. Hereinafter, the intersection 1103a is also referred to as a target point that is a target point on which the addition process is performed. The target point on the light receiving surface of the image sensor 102 is represented by, for example, coordinate values on the X _k Y _k plane shown in FIG. 10, that is, X _k Y _k coordinates.

(Step S1409)
The refocus processing unit 130 acquires an image corresponding to the selected pseudo point light source position from the storage unit 120. That is, the refocus processing unit 130 acquires an image captured using the pseudo point light source 101aj located at the selected pseudo point light source position from the storage unit 120. Specifically, the refocus processing unit 130 acquires the image stored in the storage unit 120 according to the correspondence relationship between the pseudo point light source position information and the image as illustrated in FIG. For example, the refocus processing unit 130 acquires an image corresponding to the position of the pseudo point light source 101a1 illustrated in FIG.

(Step S1410)
The refocus processing unit 130 determines the position of the target point on the image sensor 102ak calculated in step S1408 in the captured image. Specifically, the refocus processing unit 130 determines the position of the target point in the captured image based on the pixel arrangement of the captured image.

  When the position of the target point in the captured image is the pixel position of the captured image, the refocus processing unit 130 sets the luminance value of the target point in the captured image as the luminance value of the pixel. When the position of the target point in the captured image is an intermediate position between the plurality of pixels, the refocus processing unit 130 performs an interpolation process using the luminance values of the plurality of pixels adjacent to the position of the target point, thereby capturing the captured image. The luminance value of the target point at is calculated. Specifically, the refocus processing unit 130 obtains, for example, the distance between each pixel of a plurality of pixels (for example, four pixels) adjacent to the target point and the target point, and the ratio of the distance between the target point and each pixel. Is multiplied by the luminance value of each pixel and added to obtain the luminance value of the target point in the captured image.

  FIG. 17 is a schematic diagram for explaining the calculation of the luminance value of the target point in step S1410. In FIG. 17, the distances between the four pixels A to D adjacent to the target point and the target point are represented as a, b, c, and d, respectively. In this case, the luminance value Lt of the target point is obtained by the following expression 1.

  Here, La, Lb, Lc, and Ld represent the luminance values of the pixel A, the pixel B, the pixel C, and the pixel D, respectively.

(Step S1411)
The refocus processing unit 130 adds the luminance value of the target point calculated in step S1410 to the luminance value of the selected pixel on the focused image.

  By repeating the processing from step S1406 to step S1411 for each pseudo point light source position, the result of adding the luminance value of the target point in the captured image to the luminance value of the selected pixel is obtained for all pseudo point light source positions. , Calculated as the luminance value of the selected pixel.

  By such addition processing, for each point on the focal plane, a plurality of images formed by light from a plurality of directions that have passed through the point are superimposed on one pixel of the focused image.

  For example, in FIG. 16, the light emitted from the pseudo point light source 101a1 passes through the point 1102a on the focal plane 1100 corresponding to the selected pixel and reaches the target point (intersection 1103a) on the light receiving surface of the image sensor 102. . Therefore, the image of the point 1102a on the focal plane 1100 is included in the position of the target point (intersection 1103a) in the image captured through the illumination of the pseudo point light source 101a1.

  In FIG. 16, the light emitted from the pseudo point light source 101a2 passes through the point 1102a on the focal plane 1100 corresponding to the selected pixel and reaches the target point (intersection 1103b) on the light receiving surface of the image sensor 102. . Therefore, the image of the point 1102a on the focal plane 1100 is included in the position of the target point (intersection 1103b) in the image captured through the illumination of the pseudo point light source 101a2.

  By adding the image (luminance value) at the target point (intersection 1103a) and the image (luminance value) at the target point (intersection 1103b), a plurality of images formed by light from a plurality of directions are combined. It is superimposed on the selected pixel of the focus image.

[Modification 1]
Next, Modification 1 of the embodiment will be described. In the embodiment, the configuration is such that an object placed in one culture vessel is photographed by one image sensor. However, in Modification 1, a plurality of objects are arranged in one culture vessel. In addition, a plurality of objects in one culture vessel are photographed with a plurality of image sensors.

  In this modification, the culture vessel 230 is configured to hold a plurality of sets of culture solutions and cells. The culture vessel 230 may have a configuration such as a multi-well plate, for example. In this case, a well 230ak (k = 1, 2,..., N) that is a plurality of depressions at the bottom of the culture vessel 230. And each of the wells 230ak contains and holds a culture solution and cells. 18 and 19, a culture container tray 220 including a multi-well plate type culture container 230 is illustrated. 18 is a cross-sectional side view schematically showing an example of the structure of the culture container tray 220 according to Modification 1. An example of the culture container tray 220 including the culture container 230 is similar to FIG. FIG. FIG. 19 is a top view schematically showing an example of the structure of the culture container tray 220 according to Modification 1, and is a view of the bottom 221 of the culture container tray 220 of FIG. 18 as viewed from above.

  As shown in FIGS. 18 and 19, one multi-well plate type culture vessel 230 is disposed and fixed in one culture vessel tray 220. The culture container tray 220 includes a bottom 221 and a lid 222. The bottom 221 includes a container position fixing frame 221a and a plurality of image sensors 102ak (k = 1, 2,..., N). The lid 222 includes the illuminator 101 including a plurality of pseudo point light sources 101aj (j = 1, 2,..., M).

  The container position fixing frame 221a fixes the multiwell plate type culture container 230 in the bottom portion 221. The plurality of image sensors 102ak are arranged at the bottom of the container position fixing frame 221a, and are further arranged at positions facing the bottom walls of the wells 230ak of the culture vessel 230 arranged in the container position fixing frame 221a. The image sensor 102ak may be arranged so as to be close to or in contact with the bottom wall of the well 230ak. That is, the image sensor 102ak is arranged according to the number and position of the wells 230ak included in the multi-well plate. Thereby, each image sensor 102ak images a target object such as a cell in each well 230ak. In this modification, one multiwell plate type culture container 230 is fixed in one culture container tray 220. However, a plurality of multiwell plate type culture containers 230 may be arranged. . When a plurality of multi-well plate type culture vessels 230 are arranged, the number of wells contained in each culture vessel 230 may be the same or different. Also in this case, the number and position of the wells of all the culture vessels 230 may match the number and position of the image sensors. In the example shown in FIG. 19, the multi-well plate type culture vessel 230 includes 15 wells 230ak, but the number of wells 230ak may be other than this.

  Even when the culture container tray 220 and the culture container 230 according to Modification 1 are used, the image generation by the imaging and refocusing processing in the image generation system 10 is performed in the same manner as in the embodiment.

[Modification 2]
Next, a second modification of the embodiment will be described. In the first modification, a plurality of objects arranged in one culture vessel are photographed with a plurality of image sensors. In the second modification, a plurality of objects arranged in one culture vessel are used. An object is photographed with one image sensor.

  In this modification, one culture container tray 320 includes a plurality of multi-well plate type culture containers 230. 20 and 21, a culture container tray 320 including two multiwell plate type culture containers 230 is illustrated. FIG. 20 is a cross-sectional side view schematically showing an example of the structure of the culture container tray 320 according to Modification 2. An example of the culture container tray 320 including two culture containers 230 is similar to FIG. FIG. FIG. 21 is a top view schematically showing an example of the structure of the culture container tray 320 according to Modification 2, and is a view of the bottom 321 of the culture container tray 320 of FIG. 20 as viewed from above.

  As shown in FIGS. 20 and 21, in the culture container tray 320 according to this modification, one image sensor is arranged for one multiwell plate type culture container 230. That is, two image sensors 102a1 and 102a2 are arranged. Each of the image sensors 102a1 and 102a2 is disposed to face the bottom walls of all the wells 230ak of one culture vessel 230. The image sensors 102a1 and 102a2 may be arranged to be close to or in contact with the bottom wall of the well 230ak. Therefore, in the example shown in FIG. 21, each of the image sensors 102a1 and 102a2 can photograph an object such as a cell in nine wells 230ak.

  Even when the culture container tray 320 and the culture container 230 according to Modification 2 are used, the image generation by the imaging and refocusing processing in the image generation system is performed in the same manner as in the embodiment. Furthermore, with the above-described configuration, when cultivating small cells or cell masses, a plurality of cells or cell masses can be monitored with one image sensor, and space utilization efficiency is increased.

[2. effect]
As described above, according to the image generation system 10 according to the present embodiment, a plurality of in-focus pixels constituting a focal image in the focal plane using information on the virtual focal plane and a plurality of captured images. The luminance value of the sensor pixel of the image sensor 102ak corresponding to is acquired. Then, using the acquired luminance value of the sensor pixel, a focused image of the focal plane is generated. For example, each of the first image sensor 102a1 and the second image sensor 102a2 acquires a captured image of an object on each image sensor, and an image sensor that should generate a focused image on the focal plane is selected and selected. A focused image based on the image captured by the image sensor is generated. Therefore, it is possible to generate a high-quality focused image while improving the generation processing speed of the focused image.

  In the generation of the focused image of the focal plane, in addition to the focal plane information and the captured image, the position information of the pseudo point light source 101aj and the position information of the selected image sensor are used. Further, the position information of the pseudo point light source 101aj and the position information of the image sensor 102ak are based on the first coordinate system that does not depend on the position of the pseudo point light source 101aj and the position of the image sensor 102ak, and the information on the focal plane is the selected image. Based on the second coordinate system with reference to the sensor. In generating a focused image on the focal plane, coordinate conversion is performed between the two coordinate systems. By using the first coordinate system, the position information of the pseudo point light source 101aj and the position information of the image sensor 102ak can be fixed information that is not affected by these positions. Therefore, setting of each position information is easy. By using the second coordinate system, the amount of processing required to calculate the brightness value of the focused pixel is reduced. Therefore, the processing speed is improved.

  For example, when cultivating many cells using a plurality of culture vessels, a plurality of image sensors 102ak can illuminate a cell or the like placed on each image sensor while one pseudo point light source illuminates. Shoot things at the same time. Then, at the time of generating a focused image, each information is converted into coordinates with respect to the position of each image sensor 102ak as a reference, thereby calculating in a short shooting time similar to that for shooting with one image sensor 102ak. In-focus images of a plurality of objects can be generated by the plurality of image sensors 102ak.

  Further, the luminance value of the target point that is the intersection of the straight line connecting the position of the pixel on the focal plane and the position of the pseudo point light source 101aj and the light receiving surface of the image sensor 102ak can be applied to the luminance value of the pixel. Therefore, each focused pixel of the focused image on the virtual focal plane can reflect the luminance values of a plurality of captured images corresponding to the pixel, and a high-quality focused image of the object is generated. Can do.

(Other embodiments)
As described above, the image generation system according to one or more aspects has been described based on the embodiment, but the present disclosure is not limited to this embodiment. Unless it deviates from the gist of the present disclosure, various modifications conceived by those skilled in the art have been made in this embodiment, and forms constructed by combining components in different embodiments are also within the scope of one or more aspects. May be included.

  A comprehensive or specific aspect of the present disclosure may be realized by a recording medium such as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM, and the apparatus, method, integrated circuit, computer program, and You may implement | achieve with arbitrary combinations of a recording medium.

  For example, each component of the image generation system according to the present disclosure may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be composed of a single element that performs centralized control, or may be composed of a plurality of elements that perform distributed control in cooperation with each other.

  Each component of the image generation system may be a circuit such as an LSI (Large Scale Integration), a system LSI, or the like. A plurality of components may constitute one circuit as a whole, or may constitute separate circuits. Each circuit may be a general-purpose circuit or a dedicated circuit.

  The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components on one chip, and specifically, a computer system including a microprocessor, a ROM, a RAM, and the like. A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program. The system LSI and the LSI may be a field programmable gate array (FPGA) that can be programmed after manufacturing the LSI, and may include a reconfigurable processor that can reconfigure the connection and setting of circuit cells in the LSI. .

  Further, some or all of the components of the image generation system may be configured from a removable IC card or a single module. The IC card or module is a computer system that includes a microprocessor, ROM, RAM, and the like. The IC card or module may include the above-described LSI or system LSI. The IC card or the module achieves its function by the microprocessor operating according to the computer program. These IC cards and modules may have tamper resistance.

  The image generation method according to the present disclosure may be realized by a circuit such as an MPU, CPU, processor, LSI, an IC card, a single module, or the like.

  Further, the processing in the image generation apparatus and the image generation method according to the present disclosure may be realized by a digital signal including a software program or a software program. The program and the digital signal composed of the program are recorded on a computer-readable recording medium such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray (registered). (Trademark) Disc), recorded in a semiconductor memory or the like. The program and the digital signal composed of the program may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like. Further, the program and the digital signal composed of the program may be implemented by another independent computer system by being recorded on a recording medium and transferred, or transferred via a network or the like. .

  INDUSTRIAL APPLICABILITY The present invention can be widely used in an apparatus that generates an image of a cell mass such as a cell in culture or an embryo, and is useful when photographing an object in an incubator.

DESCRIPTION OF SYMBOLS 10 Image generation system 20,220,320 Culture container tray 21,221,321 Bottom part 21a1,21a2,221a Container position fixed frame 22,222 Lid 30a1,30a2 Petri dish type culture container 30b Flask type culture container 230 Multiwell plate type Culture container 100A Imaging device 100B Image generation device 101 Illuminator 101aj Pseudo point light source 102, 102ak Image sensor 103 Imaging control unit 110 Focal plane determination unit 111 First recording unit 112 Input unit 120 Storage unit 121 Second recording unit 130 Refocus processing Unit 140 image generation unit 150 display unit 160 image sensor determination unit

Claims (6)

First and second light sources;
A first image sensor having a surface on which the first object is placed and a plurality of sensor pixels;
A second image sensor having a surface on which the second object is placed and a plurality of sensor pixels;
At least one control circuit for generating a focused image corresponding to a virtual focal plane located between the first and second light sources and the first and second image sensors;
Each of the first and second image sensors uses a luminance value based on light received by the plurality of sensor pixels each time the first and second light sources illuminate, respectively. Acquire multiple captured images of the second object,
The at least one control circuit comprises:
(A1) selecting one of the first and second image sensors;
(A2) acquiring a plurality of captured images captured by the selected image sensor;
(A3) obtaining information on the focal plane located between the first and second light sources and the selected image sensor;
(A4) By using the information on the focal plane and the plurality of captured images, obtaining luminance values of the sensor pixels corresponding to a plurality of focused pixels constituting the focused image on the focal plane, Generate a focused image of the focal plane,
(A5) An image generation system that outputs a generated focused image of the focal plane. The at least one control circuit further acquires and uses position information of the first and second light sources and position information of the selected image sensor in generating a focused image of the focal plane. The image generation system described in 1. The position information of the first and second light sources, and the position information of the selected image sensor are based on the first coordinate system independent of the positions of the first and second light sources and the position of the image sensor,
The focal plane information is based on a second coordinate system based on the selected image sensor,
The image generation system according to claim 2, wherein the at least one control circuit performs coordinate conversion between the first coordinate system and the second coordinate system in generating a focused image of the focal plane. The at least one control circuit uses the brightness value of each of the sensor pixels in which the position of the light source, the position of the in-focus pixel, and the position of the sensor pixel are arranged in a straight line. The image generation system according to any one of claims 1 to 3, wherein a luminance value is calculated. An image generation method for generating an image of an object located on an image sensor,
(B1) Each time each of the first and second light sources is illuminated, the first and second light sources are used with luminance values based on light received by the plurality of sensor pixels of the first and second image sensors. Acquiring a plurality of captured images of the first and second objects respectively located on the second image sensor;
(B2) selecting one of the first and second image sensors;
(B3) acquiring a plurality of captured images captured by the selected image sensor;
(B4) setting a virtual focal plane located between the first and second light sources and the selected image sensor;
(B5) generating a focused image corresponding to the focal plane, and using the focal plane information and the plurality of captured images, a plurality of in-focus images constituting the focused image on the focal plane; By obtaining the brightness value of the sensor pixel corresponding to the pixel, a focused image of the focal plane is generated,
(B6) outputting a focused image of the focal plane;
At least one of the (b1) to (b6) is an image generation method executed by a control circuit. A program to be executed by a computer,
(C1) selecting one from the first and second image sensors;
(C2) obtaining a plurality of photographed images photographed by the selected image sensor from among a plurality of photographed images of the first and second objects respectively positioned on the first and second image sensors; Here, each of the plurality of captured images of the first and second objects is a plurality of sensor pixels of the first and second image sensors each time the first and second light sources are illuminated. Acquired by the first and second image sensors using a luminance value based on the received light,
(C3) setting a virtual focal plane located between the first and second light sources and the selected image sensor;
(C4) generating a focused image corresponding to the focal plane, and using the focal plane information and the plurality of captured images, a plurality of in-focus images constituting the focused image on the focal plane; By obtaining the brightness value of the sensor pixel corresponding to the pixel, a focused image of the focal plane is generated,
(C5) A program for outputting a focused image of the focal plane.

Images