JP2016208104A - Image processing apparatus, image processing method and program for image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily retrieve a light field (LF) image that is focused to a desired browsing position.SOLUTION: An image processing apparatus 100 comprises: an image acquisition part 152 for acquiring the LF image capable of changing a focus position that is a focused position in a depth direction, after imaging, and imaging information including a focus range that is a range of an imaging position of the LF image and a changeable focus position; a position selection part 153 for selecting one or more sampling positions from among a plurality of positions in an imaging space in which an imaging apparatus that captures the LF image can perform imaging; an image selection part 154 for selecting one or more images which can be focused and or which the focus position can be set to a sampling position, from the LF image based on the sampling position and the imaging information; an image generation part 155 for generating a refocus image for which the focus position is set to one or more sampling positions; and a display control part 156 for displaying the refocus image on a display part 130.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for processing changed image data after imaging a focused position.

  In recent years, an imaging device called a light field (hereinafter abbreviated as “LF”) camera has been put into practical use. In this imaging apparatus, a microlens array is arranged on an image sensor, and light incident on the imaging apparatus is divided by the microlens, whereby light information indicating the intensity of light coming from a plurality of directions and a direction indicating the incident direction LF image data including information is generated.

  In Patent Document 1, by performing arithmetic processing based on light ray information and direction information, a plurality of refocus images having different in-depth positions (hereinafter referred to as “focus positions”) are sequentially displayed. An image processing apparatus that makes it easy for a user to search for a desired image is disclosed.

JP 2014-220598 A

  However, when a user searches for an image using a conventional image processing apparatus, it is necessary to check a large number of sequentially displayed refocus images, and it takes a long time to find a desired refocus image. In particular, when a large number of images are captured in order to observe the observation site from various positions and directions, the number of refocus images further increases, making it more difficult to search for a desired refocus image.

  For example, when the imaging device is a capsule endoscope for observing an organ or the like, a large number of captured images (tens of thousands of images) are usually taken from the administration of the capsule endoscope to the subject until it is discharged. Image). For this reason, in the case of a search using a conventional image processing apparatus, it takes a long time for a doctor to search for an image focused on a desired affected part.

  Accordingly, the present invention has been made in view of the above-described points, and it is easy to search for a captured image that is focused on a desired viewing location from among a large number of captured images whose focus position can be changed after imaging. Objective.

  In order to solve the above-described problems, an image processing apparatus according to an embodiment of the present invention captures a plurality of light field images that can be changed after capturing a focus position that is a position in the depth direction in focus, and captures the plurality of light field images. Image acquisition means for acquiring imaging information including a focus range that is a position and a range of focus positions that can be changed, and one or more from a plurality of positions in an imaging space that can be imaged by the imaging device that captures the plurality of light field images And selecting one or more focusable images capable of setting a focus position at the sampling position from the plurality of light field images based on the sampling position and the imaging information. Image selection means for selecting the one or more focusable images selected, Image generating means for generating a refocusing image the focus position is set to the serial one or more sampling positions, and a display control means for displaying the generated the refocused image on the display means, characterized by having a.

  Advantageous Effects of Invention According to the present invention, it is possible to easily search for a captured image that is focused on a desired viewing location from among a large number of captured images whose focus position can be changed after imaging.

It is a schematic diagram of a structure of the imaging device 10 as LF camera. 3 is a schematic diagram showing a positional relationship between a microlens array 12 and each pixel of an image sensor 13. FIG. FIG. 6 is a schematic diagram for explaining a relationship between a traveling direction of incident light rays to a microlens of the microlens array 12 and a recording area of the image sensor 13. It is a schematic diagram for demonstrating the information of the light ray which injects into the image sensor. It is a schematic diagram for demonstrating a refocus calculation process. FIG. 6 is a schematic diagram illustrating a relationship between a difference in incident angle to a microlens and a recording area of the image sensor 13. It is a schematic diagram for demonstrating the adjustment process of a depth of field. 1 is a diagram illustrating a configuration of an image processing system S according to a first embodiment. 1 is a diagram illustrating a configuration of an image processing apparatus 100. FIG. It is a figure which shows an example of a sampling position table. 6 is a diagram showing an example of a display screen displayed on the display unit 130. FIG. It is a flowchart which shows the procedure until it displays a layout image. It is a flowchart which shows the procedure of the process which selects LF image data. It is a flowchart which shows the procedure of a selection process. It is a figure which shows an example of the sampling position table which concerns on 2nd Embodiment. It is a figure which shows an example of the layout image which concerns on 2nd Embodiment. It is a flowchart which shows the procedure of the process which selects LF image data which concerns on 2nd Embodiment. It is a flowchart which shows the procedure of the selection process which concerns on 2nd Embodiment.

[Outline of LF camera]
Prior to the description of the image processing apparatus according to the present embodiment, an outline of the LF camera will be described.
FIG. 1 is a schematic diagram of a configuration of an imaging device 10 as an LF camera. The imaging device 10 includes an imaging lens 11, a microlens array 12, and an image sensor 13. The light from the subject enters the image sensor 13 after passing through the imaging lens 11 and the microlens array 12 constituting the imaging optical system. The image sensor 13 converts the incident light into an electric signal and outputs the converted electric signal. The imaging device 10 generates LF image data that can change the focus position, which is the focused position, based on the electrical signal output from the image sensor 13.

  The imaging lens 11 projects light from the subject onto the microlens array 12. The imaging lens 11 is a replaceable lens that is used by being attached to the main body of the imaging device 10. The user of the imaging device 10 can change the imaging magnification by a zoom operation of the imaging lens 11.

  The microlens array 12 has a plurality of minute lenses (microlenses) arranged in a lattice pattern, and is located between the imaging lens 11 and the image sensor 13. Each microlens included in the microlens array 12 divides the incident light from the imaging lens 11 and makes the plurality of divided light incident on the plurality of pixels of the image sensor 13.

  The image sensor 13 is an image sensor having a plurality of pixels, and detects the intensity of light at each pixel. Light in different directions divided by the microlens corresponding to each pixel is incident on each pixel of the image sensor 13 that receives light from the subject.

  FIG. 2 is a schematic diagram showing the positional relationship between the microlens array 12 and each pixel of the image sensor 13. Each microlens of the microlens array 12 is arranged so as to correspond to a plurality of pixels in the image sensor 13. Light divided by the microlens corresponding to each pixel is incident on each pixel of the image sensor 13, and the image sensor 13 detects the intensity of light from a different direction for each pixel. The imaging device 10 generates light ray information indicating the intensity of light detected at each pixel of the image sensor 13.

  The incident direction of the light beam incident on each pixel of the image sensor 13 through the microlens is determined by the arrangement of a plurality of pixels corresponding to each microlens. Therefore, the incident direction is specified according to the positional relationship between each microlens and each pixel of the image sensor 13, and the imaging apparatus 10 uses, for example, coordinates for specifying the pixel as direction information indicating the specified incident direction. Generate information.

  Light from the subject enters the microlens array 12 in a direction determined according to the distance between the subject and the imaging lens 11. The light from the subject passes through a plurality of microlenses determined according to the distance between the subject and the imaging lens 11 and enters the image sensor 13. As a result, the image focused on the surfaces having different distances from the lens apex surface of the microlens array 12 is each of the plurality of pixels of the image sensor 13 at the position corresponding to the amount of eccentricity from the optical axis of each microlens. It is obtained by combining the output. Therefore, the imaging apparatus 10 can generate image data corresponding to an arbitrary focus position by performing light beam rearrangement and calculation processing (hereinafter referred to as reconstruction) based on the light beam information and the direction information. Here, the focus position is an in-focus position, and the user can obtain refocus image data corresponding to a desired focus position based on the LF image data after shooting.

FIG. 3 is a schematic diagram for explaining the relationship between the traveling direction of incident light to the microlenses of the microlens array 12 and the recording area of the image sensor 13.
An image of the subject by the imaging lens 11 is formed on the microlens array 12, and incident light on the microlens array 12 is received by the image sensor 13 via the microlens array 12. At this time, as shown in FIG. 3, the incident light beam to the microlens array 12 is received at different positions on the image sensor 13 according to the traveling direction determined for each focus position, and the image of the subject is formed for each microlens. Image.

FIG. 4 is a schematic diagram for explaining information on light rays incident on the image sensor 13.
The light rays received by the image sensor 13 will be described with reference to FIG. Here, an orthogonal coordinate system on the lens surface of the imaging lens 11 is (u, v), and an orthogonal coordinate system (x, y) on the imaging surface of the image sensor 13 is used. Further, let F be the distance between the lens surface of the imaging lens 11 and the imaging surface of the image sensor 13. In this case, the intensity of the light beam passing through the imaging lens 11 and the image sensor 13 can be expressed by a four-dimensional function L (u, v, x, y) shown in the drawing. Since the light beam incident on each microlens is incident on a different pixel according to the traveling direction, the image sensor 13 uses the above four-dimensional function L (which holds the traveling direction of the light beam in addition to the positional information of the light beam). Information corresponding to the light intensity represented by u, v, x, y) is recorded.

  Next, refocus calculation processing after imaging will be described. The refocus calculation process is a process for generating refocus image data by reconstructing an image based on the intensity of light received by pixels corresponding to different focus positions. In this specification, a surface at a position in focus in a refocus image is referred to as a refocus surface.

FIG. 5 is a schematic diagram for explaining the refocus calculation process.
As shown in FIG. 5, when the positional relationship between the imaging lens surface, the imaging surface, and the refocus plane is set, the light intensity L ′ (u, v, s) in the orthogonal coordinate system (s, t) on the refocus plane. , T) is expressed as the following equation (1).
Further, the image E ′ (s, t) obtained on the refocus plane is obtained by integrating the intensity L ′ (u, v, s, t) with respect to the lens diameter, and is expressed by the following equation (2). It is expressed in
Therefore, image data corresponding to the refocus plane at an arbitrary focus position can be reconstructed by performing the refocus calculation process based on the equation (2).

  Next, the adjustment process of the depth of field after imaging, that is, the adjustment process of the focus position in the depth direction will be described. When adjusting the depth of field, weighting is performed by multiplying each image data forming the image area assigned to each microlens by a weighting factor before the above refocusing calculation processing. For example, when generating an image having a deep depth of field, the integration process is performed using only information on light rays incident at a relatively small angle with respect to the light receiving surface of the image sensor 13. In other words, light rays having a relatively large incident angle to the image sensor 13 are not included in the integration process by multiplying by a weighting factor of 0 (zero).

FIG. 6 is a schematic diagram for explaining the relationship between the difference in the incident angle to the microlens and the recording area of the image sensor 13, and FIG. 7 is a schematic diagram for explaining the depth-of-field adjustment process. is there.
As shown in FIG. 6, the light beam having a relatively small incident angle to the image sensor 13 is located in a more central region. Therefore, when generating an image with a deep depth of field, as shown in FIG. 7, integration processing is performed using only pixel data acquired at the center of the region (shaded portion in the figure).

By performing such processing, the imaging apparatus 10 can express an image having a deep depth of field as if the aperture stop provided in a general imaging apparatus or the like is limited. The imaging apparatus 10 can also generate pan-focus image data with a deeper depth of field by further reducing the pixel data in the central portion to be used. The imaging device 10 can also adjust the depth of field to be different for each region by changing the position of the pixel data to be used for each region of the LF image data. As described above, the imaging device 10 adjusts the depth of field of the image after shooting based on the focus information including the weighting coefficient of each pixel included in the LF image data, the light ray information, and the direction information, and performs a desired operation. The focus position can be set.
Hereinafter, embodiments of an image processing apparatus, an image processing method, and an image processing program for displaying LF image data on a display will be described.

<First Embodiment>
[Configuration of Image Processing System S]
FIG. 8 is a diagram illustrating a configuration of the image processing system S according to the first embodiment. The image processing system S includes an imaging device 10, a receiver 20, and an image processing device 100. The imaging device 10 is a capsule endoscope having a size of a tablet having a small LF camera, illumination, and a communication unit. The imaging device 10 captures each part in the body cavity T while moving in the body cavity T of the subject to be examined, and generates LF image data.

  The receiver 20 receives LF image data generated by the imaging apparatus 10 via a wireless communication line W such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). In addition, the receiver 20 receives wireless signals transmitted from the imaging device 10 with a plurality of antennas, and the imaging device 10 uses the Gauss-Newton method, for example, based on the received intensities of the received plurality of wireless signals. The position of the imaging device 10 when the LF image is captured (hereinafter referred to as “imaging position”) and the direction in which the imaging device 10 faces when the LF image is captured (hereinafter referred to as “imaging direction”) are calculated. The receiver 20 stores the LF image data received from the imaging device 10 in association with imaging information including imaging position data indicating the imaging position and imaging direction data indicating the imaging direction. The receiver 20 transmits the stored LF image data and imaging information to the image processing apparatus 100.

  The image processing apparatus 100 is, for example, a computer or a smartphone. The image processing apparatus 100 acquires LF image data and imaging information from the receiver 20 via a wired communication line, a wireless communication line, or a storage medium, and displays a refocus image based on the LF image data. The image processing apparatus 100 is focused on a position (hereinafter referred to as “sampling position”) selected as a position to be observed in the body cavity T by a user such as a doctor who diagnoses a test result among many LF image data. A screen for selecting the sampling position is displayed so that the refocus image can be selectively confirmed. Then, the image processing apparatus 100 displays a refocused image that is focused on the sampling position selected by the user.

  For example, the image processing apparatus 100 displays a sampling position designation screen for the user to designate a sampling position. The user specifies, for example, a plurality of sampling positions 201 to 204 shown in FIG. 8 on the sampling position specification screen. When the user designates a plurality of sampling positions 201 to 204, the image processing apparatus 100 selects LF image data that can be focused on the designated sampling positions. Then, the image processing apparatus 100 generates a refocus image by setting the focus position of the selected LF image data so that the sampling position is in focus, and displays the generated refocus image.

  By doing in this way, the image processing apparatus 100 allows the user, such as a lesion site or a bleeding site in the body cavity T, among a large number of refocus images that can be generated based on a large number of LF image data generated by the imaging device 10. Can selectively display a refocused image in the vicinity of a portion to be observed. Therefore, the user does not need to confirm an unnecessary refocus image, and it is easy to search for and browse a refocus image in which a desired portion is focused.

[Configuration of Image Processing Apparatus 100]
FIG. 9 is a diagram illustrating a configuration of the image processing apparatus 100. The image processing apparatus 100 includes a communication unit 110, an operation unit 120, a display unit 130, a storage unit 140, and a control unit 150.

The communication unit 110 is a communication interface for receiving LF image data and imaging information from the receiver 20, and includes, for example, a LAN controller.
The operation unit 120 is a device for a user to input an instruction. For example, the operation unit 120 is a touch panel provided on the display unit 130. The operation unit 120 outputs coordinate information indicating the two-dimensional coordinates of the position touched by the user with a fingertip or a pen tip.

The display unit 130 is, for example, a liquid crystal display, and displays an operation screen for receiving a user operation and a refocus image based on LF image data generated by the imaging device 10.
The storage unit 140 is a storage medium such as a ROM, a RAM, and a hard disk. The storage unit 140 stores an image processing program executed by the control unit 150, LF image data generated by the imaging apparatus 10, and various data generated by the control unit 150. For example, the storage unit 140 stores a sampling position table including position information indicating the sampling position designated by the user.

  The control unit 150 is a CPU, for example. The control unit 150 executes the image processing program stored in the storage unit 140 to thereby perform an operation reception unit 151, an image acquisition unit 152, a position selection unit 153, an image selection unit 154, an image generation unit 155, and a display control unit. It functions as 156.

  The operation reception unit 151 acquires coordinate information corresponding to a position where the touch panel included in the operation unit 120 is touched. Based on the coordinate information, the operation accepting unit 151 identifies the operation content of the user, and notifies the identified operation content to each unit of the control unit 150. For example, the operation reception unit 151 specifies the sampling position selected by the user based on the coordinate information, and notifies the position selection unit 153 of the specified sampling position.

  The image acquisition unit 152 acquires LF image data and imaging information via the communication unit 110 and causes the storage unit 140 to store the acquired LF image data and imaging information. The imaging information includes imaging position data, imaging direction data, and focus range data. The imaging position data is data represented by a distance from a predetermined reference position in the body cavity T, for example. The imaging direction data is data represented by an angle with respect to the reference direction, for example.

  The focus range data is data indicating a range in which the focus position of the LF image data can be changed. The focus range is a range of positions in the optical axis direction (that is, the depth direction) in which the imaging apparatus 10 can focus, and is represented by a distance from the lens of the imaging apparatus 10. The focus range is determined based on the characteristics of the lens and sensor of the imaging device 10 and is specified by the receiver 20 based on the LF image data received from the imaging device 10.

  The position selection unit 153 selects one or more sampling positions from a plurality of positions in an imaging space that can be imaged by the imaging apparatus 10. The imaging space that can be imaged by the imaging device 10 is, for example, a space in the body cavity T in which the imaging device 10 can move. Specifically, the position selection unit 153 causes the display unit 130 to display a sampling position designation screen for designating the sampling position by the user via the display control unit 156 and designates the user via the operation unit 120. Select the sampling position.

  For example, the position selection unit 153 causes the display unit 130 to display a sampling position designation screen including a plurality of sampling position candidates provided at intervals designated by the user. The position selection unit 153 may display a plurality of sampling position candidates at equal intervals as shown in FIG. 8 or may display a plurality of sampling position candidates at different intervals for each region in the body cavity T. . The position selection unit 153 may finely adjust the display position of the sampling position in accordance with a user operation. In FIG. 8, the sampling positions 201 to 204 are shown in a straight line, but the sampling positions may be shown in a dotted or planar shape.

  Further, the position selection unit 153 may acquire information indicating the position of the lesion in the body cavity T from an external device (not shown) via the communication unit 110 and display the position of the lesion on the sampling position designation screen. . In this way, the user can specify the sampling position while being aware of the position of the lesion.

  The position selection unit 153 may determine a sampling position candidate to be displayed on the sampling position designation screen based on the position of the lesion. For example, the position selection unit 153 displays a plurality of sampling position candidates at a first interval at a position within a predetermined range from the lesion position, and at the first interval at a position far from the lesion position from the predetermined range. A plurality of sampling position candidates may be displayed at a second interval larger than that. In this way, the user can select an optimum sampling position near the position of the lesion.

When the designation of the sampling position by the user is completed, the position selection unit 153 generates a sampling position table including position information indicating the selected sampling position, and stores the sampling position table in the storage unit 140.
FIG. 10 is a diagram illustrating an example of the sampling position table. In the sampling position table, as shown in FIG. 10, the sampling ID, the position information, and the development ID are associated with each other. The sampling ID is identification information assigned for each sampling position. The position information is information indicated by a distance from a predetermined reference position in the body cavity T, similarly to the imaging position data included in the imaging information attached to the LF image data. The development ID is an identification ID assigned to LF image data that can be focused on each sampling position, and is generated by the image selection unit 154 and added to the sampling position table.

  The image selection unit 154 can set a focus position at a sampling position selected from among a plurality of LF images based on the sampling position selected by the position selection unit 153 and the imaging information acquired by the image acquisition unit 152. One or more LF images are selected as focusable images. Specifically, the image selection unit 154 can generate a refocus image in which each sampling position indicated by the position information included in the sampling position table generated by the position selection unit 153 is in focus. Select data. When the position selection unit 153 selects a plurality of sampling positions, the image selection unit 154 selects one or more LF image data in association with each sampling position.

  Whether the image selection unit 154 includes the distance between the position where the imaging device 10 captured the LF image and one or more sampling positions selected by the position selection unit 153 within the focus range corresponding to the LF image data. A focusable image is selected based on whether or not. For example, it is assumed that the focus range of LF image data corresponding to an LF image captured at a position 5 cm from the reference position is 0 or more and less than 10 cm. From this LF image data, it is possible to generate a refocus image that is focused on a position that is not less than 5 cm and less than 15 cm from the reference position. Therefore, the image selection unit 154 selects the LF image data if the sampling position is a position that is 5 cm or more and less than 15 cm from the reference position, and if the sampling position is a position that is less than 5 cm or 15 cm or more from the reference position, LF image data is not selected.

  The image selection unit 154 may select a focusable image based on the relationship between the imaging direction indicated by the imaging direction data in the imaging information and the direction of the sampling position with respect to the imaging position. Specifically, the image selection unit 154 selects a focusable image based on whether or not the imaging direction is toward the sampling position. More specifically, the image selection unit 154 determines the image capturing direction associated with the LF image data with respect to a line segment connecting the position of the image capturing apparatus 10 and the sampling position when the LF image is captured. Is within the range, the LF image data is selected as focusable image data. That is, when the angle between the imaging direction and the line segment is within a predetermined range, the LF image data is selected as focusable image data.

  When selecting the LF image data as described above, the image selection unit 154 assigns a development ID to the selected LF image data. In the example shown in FIG. 10, there are three LF image data that can be focused on the sampling position with the sampling ID “1”, and the development IDs “0”, “1”, “2” are included in each LF image data. "Is assigned.

  The image generation unit 155 sets one or more refocused images by setting a focus position at one or more sampling positions selected by the position selection unit 153 with respect to one or more LF images selected by the image selection unit 154. Generate. Specifically, the image generation unit 155 sets the focus position at a position corresponding to the distance between the position where the LF image is captured and the sampling position based on the LF image data selected by the image selection unit 154. Generate a refocused image. For example, when the LF image is captured at a position 5 cm from the reference position and the sampling position is 10 cm from the reference position, the image generation unit 155 refocuses by setting the focus position of the LF image to 10 cm−5 cm = 5 cm. Generate an image. The image generation unit 155 stores the generated refocus image in the storage unit 140 in association with the development ID included in the sampling position table.

  The display control unit 156 generates a layout image in which a plurality of refocus images are arranged, and causes the display unit 130 to display the generated layout image. Specifically, the display control unit 156 acquires one or more refocus images corresponding to the development IDs included in the sampling position table from the storage unit 140. The display control unit 156 generates a layout image by arranging one or more refocus images at a predetermined interval.

  FIG. 11 is a diagram illustrating an example of the display screen 131 displayed on the display unit 130 by the display control unit 156. The display screen 131 includes a layout image display area 132, an operation button display area 133, and a sampling position designation area 134. The display control unit 156 causes the layout image display area 132 to display a layout image including a refocus image corresponding to the sampling position designated in the sampling position designation area 134. In FIG. 11A, a layout image including a refocus image corresponding to the sampling position 201 is displayed. When a plurality of refocus images are included in one layout image, for example, the display control unit 156 displays a layout image in which the refocus images are arranged in the order in which the LF images corresponding to the refocus images are captured.

  In the operation button display area 133, operation images for operations such as “play”, “stop”, “pause”, “next image”, “previous image”, and the like are displayed. The display control unit 156 switches the layout image displayed in the layout image display area 132, for example, at regular intervals. The display control unit 156 may switch the layout image in response to the operation image in the operation button display area 133 being touched. For example, when the operation image for designating the image at the next sampling position 202 is touched among the operation images displayed in the operation button display area 133, the display control unit 156, as shown in FIG. The refocus image stored in the storage unit 140 in association with the development ID corresponding to the sampling position 202 is read and displayed in the layout image display area 132.

[Process in which image processing apparatus 100 displays a layout image]
FIG. 12 is a flowchart illustrating a procedure until the image processing apparatus 100 displays a layout image. Hereinafter, a procedure from when the image processing apparatus 100 acquires LF image data to when a layout image is displayed will be described with reference to FIG.

  First, the image acquisition unit 152 acquires LF image data and imaging information (S11). The image acquisition unit 152 stores the acquired LF image data in the storage unit 140 and notifies the position selection unit 153 of imaging information.

  Subsequently, the position selection unit 153 selects one or more sampling positions from a plurality of sampling positions based on the operation information notified from the operation reception unit 151 (S12). The position selection unit 153 generates a sampling position table including the sampling positions, and stores the sampling position table in the storage unit 140.

  Subsequently, the image selection unit 154 selects LF image data for which a refocus image is to be generated based on the imaging information and the position information in the sampling position table (S13). The image selection unit 154 causes the image generation unit 155 to determine a development ID corresponding to the selected LF image data, and causes the storage unit 140 to store the development ID in association with the LF image data. Thereafter, the image selection unit 154 notifies the image generation unit 155 of a development request including the development ID corresponding to the LF image data for which a refocus image is to be generated.

  Subsequently, the image generation unit 155 reads out the LF image data associated with the development ID included in the development request among the plurality of LF image data stored in the storage unit 140, and based on the read out LF image data. Then, a refocus image is generated (S14). The image generation unit 155 stores the generated refocus image in the storage unit 140. In addition, the image generation unit 155 notifies the display control unit 156 of a refocus image generation completion notification including the development ID.

  When receiving the generation completion notification from the image generation unit 155, the display control unit 156 acquires a refocus image corresponding to the development ID included in the generation completion notification from the storage unit 140. The display control unit 156 generates a layout image including the acquired refocus image and causes the display unit 130 to display the generated layout image (S15).

  Next, the process for selecting the LF image data in step S13 in FIG. 12 will be described in detail. FIG. 13 is a flowchart illustrating a processing procedure for selecting LF image data.

  First, the image selection unit 154 acquires the sampling position table generated by the position selection unit 153 from the storage unit 140 (S21). Subsequently, the image selection unit 154 initializes the variable N to 0 (S22). Here, the variable N corresponds to the sampling ID assigned for each sampling position. Next, the image selection unit 154 adds 1 to the variable N (S23). Here, the variable N after the addition corresponds to the sampling ID of the sampling position to be subjected to the process of selecting the LF image data. The image selection unit 154 determines whether or not the variable N is smaller than the total number of sampling positions included in the sampling position table, so that the process of selecting the LF image data corresponding to the sampling position has not ended. It is determined whether or not an unprocessed sampling position remains (S24).

  When the variable N is smaller than the determined total number of sampling positions (Yes in S24), the image selection unit 154 determines that an unprocessed sampling position remains, and focuses on the sampling position corresponding to the variable N. A process of selecting LF image data that can generate a refocused image is executed (S25). On the other hand, when the variable N has reached the determined total number of sampling positions (No in S24), the image selection unit 154 notifies the image generation unit 155 that the LF image selection processing has ended, and FIG. The processing is shifted to S14 in (S26).

  FIG. 14 is a flowchart showing the procedure of the selection process in step S25 in FIG. 13, that is, the procedure of the process of selecting the LF image data that is the target for generating the refocused image focused on the sampling position corresponding to the variable N. It is. First, the image selection unit 154 initializes the value of the variable M to 1 (S31). The variable M is a variable assigned to each of a plurality of LF image data included in the LF image data group acquired by the image acquisition unit 152, and is, for example, a value corresponding to the order in which the LF images are captured. The LF image data group is, for example, an image set including a plurality of LF image data captured by one intracorporeal imaging.

  Subsequently, the image selection unit 154 determines whether or not the variable M is equal to or less than the total number of LF image data, thereby generating a refocus image in focus at the sampling position corresponding to the variable N. It is determined whether the process of selecting LF image data has been performed on all LF image data (S32). Here, when the value of the variable M corresponding to the determination target LF image data is larger than the total number of acquired LF image data (No in S32), the image selection unit 154 performs the process in S23 of FIG. It is determined that the selection processing of LF image data capable of generating a refocused image focused on the sampling position corresponding to the added variable N is completed, and the process proceeds to S23.

  On the other hand, when the value of the variable M is equal to or less than the total number of acquired LF image data (Yes in S32), the image selection unit 154 reads the LF image data and imaging information corresponding to the variable M from the storage unit 140 ( S33). Subsequently, the image selection unit 154 determines whether or not the imaging direction indicated by the imaging direction data included in the imaging information corresponding to the read LF image data is appropriate (S34). For example, the image selection unit 154, based on the imaging direction data of the imaging information corresponding to the LF image data read from the storage unit 140, the line segment connecting the imaging direction of the imaging device 10 and the imaging device 10 and the sampling position. The magnitude of the difference in angle with the direction is calculated. The image selection unit 154 determines that the imaging direction is appropriate when the magnitude of the difference is equal to or less than a predetermined value.

  If the image selection unit 154 determines that the imaging direction is not appropriate (No in S34), the process returns to S32. On the other hand, when the image selection unit 154 determines that the imaging direction is appropriate (Yes in S34), the image selection unit 154 calculates an imaging distance that is a distance between the position when the imaging device 10 images the LF image and the sampling position ( S35). The imaging distance is represented, for example, by the absolute value of the difference between the imaging position of the imaging device 10 and the position information value in the sampling position table.

  Subsequently, based on the calculated imaging distance, the image selection unit 154 determines whether or not it is possible to generate a refocus image in focus at the sampling position corresponding to the variable N from the LF image data corresponding to the variable M. Determine. Specifically, the image selection unit 154 determines whether the imaging distance corresponding to the sampling position of the variable N is within the focus range of the LF image data corresponding to the variable M (S36). When the imaging distance is out of the focus range (No in S36), the image selection unit 154 returns the process to S32.

  On the other hand, when the imaging distance is within the focus range (Yes in S36), the image selection unit 154 determines the development ID and issues a development request for requesting generation of a refocus image including the determined development ID. It transmits to the production | generation part 155 (S37). In addition, the image selection unit 154 adds the development ID included in the development request to the sampling position table (S38). With the above processing from S32 to S38, the processing for determining whether or not to select LF image data corresponding to the variable M is completed. Therefore, the image selecting unit 154 adds 1 to the variable M (S39). The process returns to S32.

  In the present embodiment, the case where the image selection unit 154 performs selection processing on all LF image data included in the image set has been described. However, the image selection unit 154 uses an estimation algorithm or the like, You may limit LF image data used as the object which performs a selection process. For example, in the process of selecting LF image data for a plurality of sampling positions, the image selection unit 154 has the first sampling position within the focus range, and the second sampling position next to the first sampling position is the focus range. For LF image data not included, it is estimated that the next third sampling position is not within the focus range. Then, the image selection unit 154 does not select the LF image data as the LF image data to be selected at the third sampling position. In this way, the image selection unit 154 can reduce the time required for the selection process.

[Effect in the first embodiment]
As described above, in the image processing apparatus 100 according to the present embodiment, the image acquisition unit 152 acquires LF image data and imaging information, and the position selection unit 153 determines the sampling position based on the operation input from the user. To decide. Then, the image selection unit 154 selects LF image data in which the focus position can be set to the sampling position from the acquired LF image data, the image generation unit 155 generates a refocus image, and the display control unit 156 selects the refocus image. The focus image is displayed on the display unit 130.

  By doing in this way, the image processing apparatus 100 displays a refocus image in focus at the sampling position designated by the user in the body cavity from a large number of LF image data captured at various positions and directions. Can do. For this reason, the image processing apparatus 100 can display, for example, a plurality of refocus images taken from a plurality of different viewpoints focused on a site where a lesion has occurred, on one screen. The user can efficiently interpret an image effective for diagnosis.

<Second Embodiment>
In the first embodiment, the refocus image included in the layout image is generated based on the LF image data selected from the single image set, whereas in the second embodiment, the layout is displayed. The refocus image included in the image is different from the first embodiment in that the refocus image included in the image is generated based on LF image data selected from a plurality of image sets.

  The plurality of image sets are a plurality of LF image data groups that are imaged by the imaging device 10 by a plurality of in-vivo imaging. The image acquisition unit 152 acquires a plurality of LF image data included in the first image set and a plurality of LF image data included in the second image set. Then, the display control unit 156 generates a layout image in which the first refocus image corresponding to the LF image data of the first image set and the second refocus image corresponding to the LF image data of the second image set are arranged. Then, the generated layout image is displayed on the display unit 130. Each image set is assigned an image set ID for identifying the image set. An image set ID may be assigned for each patient.

  FIG. 15 is a diagram illustrating an example of a sampling position table created by the image selection unit 154 according to the second embodiment. The image selection unit 154 adds, to the sampling position table created by the position selection unit 153, a development ID corresponding to the LF image data for which a refocus image is to be generated for each image set. A sampling position table shown in FIG.

  FIGS. 16A to 16D are diagrams illustrating examples of layout images displayed on the display unit 130 by the display control unit 156 according to the present embodiment. The display control unit 156 causes the display unit 130 to display the layout images corresponding to each image set side by side. The layout image in FIG. 16A includes four refocus images that are in focus at the sampling position 201. Among these, the three refocus images displayed in the upper row are refocus images generated based on the LF image data selected from the first image set. One refocus image displayed in the lower row is a refocus image generated based on the LF image data selected from the second image set. The layout images in FIGS. 16B, 16C, and 16D include refocus images that are in focus at the sampling positions 202, 203, and 204, respectively.

  FIG. 17 is a flowchart illustrating a processing procedure for selecting LF image data. S41 to S46 in the flowchart shown in FIG. 17 correspond to S21 to S26 in the flowchart of the first embodiment shown in FIG. 13, and the value of the variable L is initialized to 0 in S42 in FIG. Unlike the flowchart, it is the same in other respects. The variable L corresponds to the image set ID assigned to each image set obtained by imaging the body cavity. The image selection unit 154 uses the value of the variable L to manage whether processing for selecting LF image data has been executed for all image sets.

  FIG. 18 is a flowchart showing the procedure of the selection process in step S45 in FIG. S51 in the flowchart shown in FIG. 18 corresponds to S31 in the flowchart shown in FIG. Further, S54 to S61 in the flowchart shown in FIG. 18 correspond to S32 to S39 in the flowchart shown in FIG. The flowchart shown in FIG. 18 differs from the flowchart shown in FIG. 14 in that S52 and S53 are executed between S51 and S54.

  The image selection unit 154 selects LF images for all image sets by determining whether or not the variable L is smaller than the number of image sets after initializing the variable M to 1 in S51. Is determined (S52). If the image selection unit 154 determines that the variable L is less than the total number of image sets (Yes in S52), the image selection unit 154 adds 1 to the variable L, and then executes the processes after S54 (S53). On the other hand, if the image selection unit 154 determines that the value of the variable L is equal to or greater than the total number of image sets (No in S52), the process proceeds to S46 in FIG. 17 and the selection of the focusable image is completed. The image generation unit 155 is notified, and the process proceeds to S14 in FIG.

[Effects of Second Embodiment]
As described above, when the image processing apparatus 100 according to the second embodiment acquires a plurality of different image sets, the image processing apparatus 100 focuses on the sampling position from the plurality of LF image data included in each image set. Display a refocused image that matches. In this way, for example, when there is a set of images taken in a body cavity taken at different times for a subject, a refocus image focused on a specific sampling position in the body cavity is obtained. Can be generated from LF image data included in each image set and displayed on one screen. For this reason, it becomes easy for a user such as a doctor to compare images taken periodically for the purpose of patient follow-up and the like, and efficient interpretation is possible.

<Third Embodiment>
In the first embodiment and the second embodiment, the example in which the imaging space of the imaging device 10 is a body cavity has been described. However, the imaging space is not limited to this, and is a space on a road or in a building. May be. In this case, the position selection unit 153 of the image processing apparatus 100 causes the display unit 130 to display a sampling position selection screen including a map of the area specified by the user, and selects the sampling position based on the user's instruction.

  Then, the image selection unit 154 selects LF image data in which the selected sampling position is included in the focus range from among a large number of LF image data generated when the user captures a space in the city using the imaging device 10. Then, the image generation unit 155 generates a refocus image based on the selected LF image data. In this way, the user can browse only the image that is focused on the location that he / she wants to browse, for example, from a large number of images taken during the trip. Is preferred.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment, A various deformation | transformation and change are possible within the range of the summary. is there. For example, in the above-described embodiment, the image processing apparatus 100 has been described with respect to an example in which the LF image data and the imaging information are acquired from the receiver 20, but the present invention is not limited thereto. The imaging apparatus 10 may generate imaging information, and the image processing apparatus 100 may acquire LF image data and imaging information from the imaging apparatus 10 wirelessly.

DESCRIPTION OF SYMBOLS 100 ... Image processing apparatus 152 ... Image acquisition part 153 ... Position selection part 154 ... Image selection part 155 ... Image generation part 156 ... Display control part

Claims (15)

Imaging information including a plurality of light field images that can be changed after imaging a focus position that is a focused position in the depth direction, and a focus range that is a range of imaging positions of the plurality of light field images and a focus position that can be changed And an image acquisition means for acquiring
Position selection means for selecting one or more sampling positions from a plurality of positions in an imaging space that can be imaged by an imaging device that captures the plurality of light field images;
Image selecting means for selecting one or more focusable images capable of setting a focus position at the sampling position from the plurality of light field images based on the sampling position and the imaging information;
Image generating means for generating a refocus image in which the focus position is set at the one or more sampling positions for the selected one or more focusable images;
Display control means for displaying the generated refocus image on a display means;
An image processing apparatus comprising: The image selection means selects the focusable image based on whether a distance between the imaging position of the light field image and the one or more sampling positions is included in the focus range. Characterized by the
The image processing apparatus according to claim 1. The image generation unit is configured to set the focus position of the focusable image selected by the image selection unit to a position corresponding to a distance between the imaging position and the one or more sampling positions. Is characterized by generating
The image processing apparatus according to claim 2. The imaging information further includes an imaging direction of the corresponding light field image,
The image selection unit selects the focusable image based on a relationship between the imaging direction and the direction of the one or more sampling positions with respect to the imaging position.
The image processing apparatus according to claim 1. The position selection means selects a plurality of the sampling positions,
The image selection means selects the focusable image for each sampling position,
The image processing apparatus according to claim 1. The display control unit generates a layout image in which a plurality of the refocus images are arranged, and causes the display unit to display the generated layout image.
The image processing apparatus according to claim 1. The image acquisition means acquires a plurality of light field images included in a first image group and a plurality of light field images included in a second image group as the plurality of light field images,
The display control unit generates a layout image in which a first refocus image corresponding to the light field image of the first image group and a second refocus image corresponding to the light field image of the second image group are arranged. And displaying the generated layout image on the display means,
The image processing apparatus according to claim 6. Imaging information including a plurality of light field images that can be changed after imaging a focus position that is a focused position in the depth direction, and a focus range that is a range of imaging positions of the plurality of light field images and a focus position that can be changed And a step of obtaining
Selecting one or more sampling positions from a plurality of positions in an imaging space that can be imaged by an imaging device that captures the plurality of light field images;
Selecting one or more focusable images capable of setting a focus position at the sampling position from the plurality of light field images based on the sampling position and the imaging information;
Generating a refocus image in which the focus position is set at the one or more sampling positions for the selected one or more focusable images;
Displaying the generated refocus image on a display means;
An image processing method comprising: In the step of selecting the focusable image, the focusable image is selected based on whether a distance between the imaging position of the light field image and the sampling position is included in the focus range. It is characterized by
The image processing method according to claim 8. The step of generating the refocus image includes:
Calculating a distance between the imaging position and the one or more sampling positions;
Generating the refocus image in which the focus position is set at a position corresponding to the calculated distance;
It is characterized by having
The image processing method according to claim 9. The imaging information further includes an imaging direction of the corresponding light field image,
The step of selecting the focusable image includes
Determining whether the imaging direction is within a predetermined range from the direction of a line segment connecting the imaging position and the sampling position;
Selecting the focusable image that has been determined that the imaging direction is within the predetermined range;
It is characterized by having
The image processing method according to claim 8. In the step of selecting the sampling position, a plurality of the sampling positions are selected,
In the step of selecting the focusable image, the focusable image is selected for each sampling position.
The image processing method according to claim 8. The step of displaying on the display means includes:
Generating a layout image in which a plurality of the refocus images are arranged;
Displaying the generated layout image on the display means;
It is characterized by having
The image processing method according to claim 8. In the step of acquiring the light field image, a plurality of light field images included in the first image group and a plurality of light field images included in the second image group are acquired as the plurality of light field images,
The step of displaying on the display means includes a layout in which a first refocus image corresponding to the light field image of the first image group and a second refocus image corresponding to the light field image of the second image group are arranged. Generating an image; and
Displaying the generated layout image on the display means;
It is characterized by having
The image processing method according to claim 13. On the computer,
Imaging information including a plurality of light field images that can be changed after imaging a focus position that is a focused position in the depth direction, and a focus range that is a range of imaging positions of the plurality of light field images and a focus position that can be changed And a step of obtaining
Selecting one or more sampling positions from a plurality of positions in an imaging space that can be imaged by an imaging device that captures the plurality of light field images;
Selecting one or more focusable images capable of setting a focus position at the sampling position from the plurality of light field images based on the sampling position and the imaging information;
Generating a refocus image in which the focus position is set at the one or more sampling positions for the selected one or more focusable images;
Displaying the generated refocus image on a display means;
The program for image processing characterized by performing these.