JP2013222383A - Image processing apparatus, program, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that suppresses, when some of images are deleted to perform image summarization processing, the occurrence of an area that is difficult to be observed due to the deletion of the images; a program; and an image processing method.SOLUTION: An image processing apparatus includes: an image sequence acquisition unit 200 that acquires an image sequence having a plurality of images; and a processing unit 100 that performs image summarization processing to delete some of the plurality of images of the image sequence acquired by the image sequence acquisition unit 200 to acquire a summary image sequence. The processing unit 100 selects a reference image and a determination object image for the image summarization processing, sets an observation area on the determination object image, obtains a corresponding area that is an area on the reference image corresponding to the observation area on the basis of deformation information between the reference image and the determination object image, and determines whether or not to delete the determination object image on the basis of at least one of a first feature amount obtained from the corresponding area and a second feature amount obtained from the observation area.

Description

  The present invention relates to an image processing apparatus, a program, an image processing method, and the like.

  When still image capturing is continued in time series at a given time interval, when a subject having a spatial spread is covered by a large number of images, or after each moving image is captured, each moving image is configured. When an image is captured as a still image, a very large amount of images (hereinafter also referred to as image sequences) that are temporally or spatially continuous are acquired. In such a case, there is a high possibility that images that are close in the image sequence (that is, close in time or space) are similar images. The need to check everything is not high. In the first place, it is not uncommon for the number of images to be tens of thousands or more, and checking everything with the user's hand itself is a heavy burden.

  Therefore, there is a demand for deleting a part of the image from the image sequence and summarizing it into an image sequence having a smaller number than the original image sequence (hereinafter, this process is referred to as an image summarization process). For example, Patent Document 1 discloses an image summarization processing method that leaves an image that makes it easy to grasp the contents of an image sequence by extracting an image of a boundary where the scene in the image sequence changes or an image representative of the image sequence. Has been.

JP 2009-5020 A JP 2011-24763 A

  For example, when an image summarization technique is applied to the medical field, it is necessary to suppress the occurrence of an area that cannot be observed by deleting an image from the viewpoint of avoiding oversight of a disease. In particular, an important area such as a lesion area or an abnormal area should not be included in an area where observation is impossible.

  However, if only the image of the boundary where the scene changes is left as in the method of Patent Document 1 or if image summarization is performed from the viewpoint of whether the image sequence after summarization is intuitively easy to see, the image is deleted. This may cause an area that cannot be observed, which is not preferable. Further, since the degree of occurrence of the region that cannot be observed depends on the contents of the image, it is difficult to control the degree of oversight of the disease by the conventional image summarization processing method.

  According to some aspects of the present invention, when image summarization processing is performed by deleting some images, an image processing device, a program, and a program for suppressing the occurrence of an area that is difficult to observe due to image deletion An image processing method or the like can be provided.

  According to an aspect of the present invention, an image sequence acquisition unit that acquires an image sequence having a plurality of images, and a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit. A processing unit that performs image summarization processing to be acquired, wherein the processing unit selects a reference image and a determination target image for the image summarization processing from the plurality of images, and sets an observation region on the determination target image In addition, based on deformation information between the reference image and the determination target image, a corresponding area that is an area on the reference image corresponding to the observation area is obtained, and the first feature obtained from the corresponding area The present invention relates to an image processing apparatus that determines whether or not the determination target image can be deleted based on at least one of an amount and a second feature amount obtained from the observation region.

  In one aspect of the present invention, an observation region is set on the determination target image, a corresponding region corresponding to the observation region is set on the reference image based on the deformation information, and at least one of the feature amount of the observation region and the corresponding region is set. Based on the above, it is determined whether or not the determination target image can be deleted. Therefore, when the corresponding region is not suitable for observation, it is possible to suppress generation of a region unsuitable for observation by processing such as including a determination target image including the observation region in the summary image sequence.

  In the aspect of the invention, the first feature amount is at least one of lightness information, size information, and similarity information with a specific shape of the corresponding region, and the second feature amount is The brightness information of the observation area, the size information, and the similarity information with the specific shape may be at least one.

  Accordingly, it is possible to determine whether or not the determination target image can be deleted using at least one of brightness information, size information, and similarity to a specific shape as a feature amount.

  In the aspect of the invention, the processing unit may perform a first comparison process between the first feature quantity and a first threshold value, and a second comparison between the second feature quantity and a second threshold value. Whether or not the determination target image is to be deleted is determined based on at least one comparison process among a process and a third comparison process between a difference between the first feature quantity and the second feature quantity and a third threshold value. A determination may be made.

  This makes it possible to determine whether or not to delete based on at least one of the process using the first feature quantity, the process using the second feature quantity, and the process using both the first feature quantity and the second feature quantity. Become. In that case, since it can be realized by a comparison process with a given threshold value, the process is easy.

  In the aspect of the invention, the processing unit obtains the brightness information of the corresponding area as the first feature amount based on the pixel value of the pixel of the corresponding area, and the pixel of the observation area. Based on the pixel value, the brightness information of the observation area may be obtained as the second feature amount.

  This makes it possible to obtain brightness information based on the pixel values of the pixels included in the region, and to perform deletion permission / inhibition determination processing using the obtained brightness information.

  In the aspect of the invention, the processing unit may be configured such that the first feature amount that is the lightness information is larger than a given upper limit threshold, or the first feature amount is smaller than a given lower limit threshold. May be determined that the determination target image cannot be deleted.

  Accordingly, it is possible to detect an inappropriate observation situation caused by extreme brightness, such as overexposure or underexposure, and to perform deletion feasibility determination processing according to the detection result.

  In the aspect of the invention, the processing unit obtains a value represented by the size information of the corresponding area as the first feature amount, and calculates a value represented by the size information of the observation area. You may obtain | require as a 2nd feature-value.

  This makes it possible to detect an inappropriate observation situation caused by the size when the size of the corresponding region is extremely small, and to perform deletion permission / inhibition determination processing according to the detection result.

  In the aspect of the invention, the processing unit obtains a value representing the similarity between the corresponding region and a given shape as the first feature amount, and the observation region and the given shape A value representing the degree of similarity may be obtained as the second feature amount.

  As a result, it is possible to detect an inappropriate observation situation caused by the shape, such as when the corresponding region is crushed or distorted, and to perform deletion feasibility determination processing according to the detection result.

  In the aspect of the invention, the processing unit may not delete the determination target image when a difference between the first feature amount and the second feature amount is larger than a given threshold. You may judge.

  This makes it possible to determine whether or not the corresponding region is suitable for observation based on the relative relationship with the observation region.

  In one aspect of the present invention, the processing unit sets a summary candidate image sequence composed of summary images remaining in the summary image sequence among the plurality of images of the image sequence acquired by the image sequence acquisition unit. And setting a deletion candidate image sequence made up of deletion candidate images not included in the summary candidate image sequence among the plurality of images of the image sequence, and the processing unit from the summary candidate image sequence to the reference image And selecting the determination target image from the deletion candidate image sequence, setting the observation region on the determination target image, and based on the deformation information between the reference image and the determination target image, The corresponding area that is an area on the reference image corresponding to the observation area is obtained, and the first feature amount obtained from the corresponding area and the second feature amount obtained from the observation area are reduced. When On the other hand, on the basis, determination may be performed in the delete YES in the determination target image.

  As a result, it is possible to set a summary candidate image sequence and a deletion candidate image sequence as pre-processing, and to perform deletion feasibility determination processing using the first and second feature amounts based on the setting result. Therefore, the deletion permission / inhibition determination process from a different viewpoint can be performed, so that the accuracy of the deletion permission / inhibition determination process can be improved.

  In the aspect of the invention, the processing unit may perform a first deletion permission determination process performed based on at least one of the first feature amount and the second feature amount, and the first deletion permission determination. The image summarization process is performed based on a second deleteability determination process different from the determination process, and the processing unit performs the second processing on the image sequence acquired by the image sequence acquisition unit. Deletability determination processing is performed, and among the plurality of images, the image determined to be undeleteable is set as the summary image, and the image determined to be deleteable is set as the deletion candidate image.

  As a result, a given deletion permission determination process can be performed as the second deletion permission determination process, which is a pre-process. Therefore, it is possible to appropriately set the summary candidate image sequence and the deletion candidate image sequence.

  In the aspect of the invention, the processing unit selects a first image and a second image from the plurality of images of the image sequence acquired by the image sequence acquisition unit, and the first image The coverage of the second image by the first image is calculated based on the deformation information between the second image and the second image, and whether or not the second image can be deleted is calculated based on the coverage. The determination process may be executed as the second deletion permission determination process.

  Thereby, the 2nd deletion permission judgment processing can be performed based on the coverage. Since the method prevents the occurrence of a region that cannot be observed when the second image is deleted, it is possible to appropriately set the summary candidate image sequence and the deletion candidate image sequence.

  In the aspect of the invention, the processing unit selects a first image and a second image from the plurality of images of the image sequence acquired by the image sequence acquisition unit, and the first image A process for determining whether or not the second image can be deleted based on a result of the process using the deformation information between the second image and the process using the structural element corresponding to the region of interest. You may perform as said 2nd deletion permission determination processing.

  Thereby, the 2nd deletion permission decision processing can be performed based on a structural element. Even if the second image is deleted, it is possible to suppress the possibility of missing the attention area on the second image, so that it is possible to appropriately set the summary candidate image sequence and the deletion candidate image sequence. Become.

  In the aspect of the invention, the process using the deformation information is a process of deforming at least a part of one of the first image and the second image using the deformation information. In the process using a structural element, it is determined whether or not the structural element is included in a non-covered area that is a region where the second image is not covered by the first image or a contraction process using the structural element. It may be a process.

  As a result, deformation processing of at least a part of the first image or the second image can be performed as processing using deformation information, and contraction processing or uncovered regions can be processed as processing using a structural element. It is possible to determine whether or not a structural element is included in.

  In the aspect of the invention, the processing unit may be arranged adjacent to the reference image, the determination target image, and the image between the reference image and the determination target image in the image sequence among the plurality of images. The deformation information between the matching images may be obtained, and the deformation information between the reference image and the determination target image may be obtained based on the obtained deformation information between the obtained adjacent images.

  Thereby, the deformation information between the first image and the second image can be calculated based on the deformation information between adjacent images.

  Another aspect of the present invention relates to a program that causes a computer to function as each of the above-described units.

  According to another aspect of the present invention, an image sequence having a plurality of images is acquired, a reference image and a determination target image are selected from the plurality of images in the image sequence, and an observation region is set on the determination target image. In addition, based on deformation information between the reference image and the determination target image, a corresponding area that is an area on the reference image corresponding to the observation area is obtained, and a first feature amount obtained from the corresponding area And determining whether or not the determination target image can be deleted based on at least one of the second feature values obtained from the observation region, and determining whether or not the plurality of images in the image sequence are based on the determination result regarding whether or not deletion is possible. The present invention relates to an image processing method for performing image summarization processing for acquiring a summary image sequence by deleting a part of an image.

1 is a configuration example of an image processing apparatus according to a first embodiment. 6 is a flowchart for explaining image summarization processing according to the first embodiment; The figure explaining the setting method of an observation area | region and a corresponding | compatible area | region. FIGS. 4A to 4D are diagrams for explaining a selection process of a reference image and a determination target image. An example in which brightness information is used as a feature amount. An example using size information as a feature quantity. An example in which the similarity with a given shape is used as a feature amount. 6 is a configuration example of an image processing apparatus according to a second embodiment. 12 is a flowchart for explaining image summarization processing according to the second embodiment. The figure explaining a summary candidate image sequence production | generation process. The structural example of the summary candidate image sequence production | generation part. The example of calculation of the covering area based on deformation information. The other example of the method of calculating | requiring a coverage. The flowchart explaining a summary candidate image sequence production | generation process. The figure explaining the process which calculates | requires a covering area | region based on a some 1st image. FIG. 16A and FIG. 16B are diagrams illustrating a modification example of the selection process of the first image and the second image. FIGS. 17A to 17C are diagrams for explaining the relationship between the shape of the uncovered area and the possibility of missing the attention area. The other example of a summary candidate image sequence production | generation part. The other flowchart explaining a summary candidate image sequence production | generation process. FIG. 20A to FIG. 20E are diagrams for explaining a contraction process by a structural element for an uncovered region. 2 is a basic system configuration example of an image processing apparatus.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the method of this embodiment will be described. When an image sequence composed of a large number of images that are temporally or spatially continuous is acquired, the user uses the image sequence to perform some processing (for example, medical practice such as diagnosis if an endoscopic image sequence). When performing, it is desirable to perform image summarization processing. This is because the number of images included in the image sequence is very large, and it is necessary for the user to make a determination after viewing all of the images. In addition, there is a high possibility that there are images that are similar to each other in the image sequence, and even if all such similar images are checked, the amount of information that can be obtained is limited and is not commensurate with the effort. .

  As a specific example, an image sequence captured using a capsule endoscope can be considered. The capsule endoscope is a capsule-shaped endoscope with a built-in small camera, and takes an image at a given time interval (for example, twice per second). Since the capsule endoscope requires several hours (in some cases, several tens of hours) from taking the clothes to discharging, tens of thousands of captured images are acquired in one examination by one user. In addition, the capsule endoscope stays in the same place or returns in the opposite direction due to the influence of the movement of the living body when moving in the living body. For this reason, in a large number of images, the same subject as that of other images is imaged, and there are many images that are not highly useful in finding lesions.

  In the conventional image summarization processing, an image at a boundary where the scene changes and an image representing an image sequence are extracted. However, with such a method, when deleting an image, the relationship between the subject captured in the image to be deleted and the subject captured in the remaining image is not particularly considered. For this reason, it is possible that the subject captured on the image included in the pre-summarization image sequence is not captured on any image included in the post-summary image sequence.

  This is not preferable particularly in image summarization processing in the medical field. In the medical field, for the purpose, oversight of a region of interest (for example, a lesion) that is a region of interest must be suppressed as much as possible. For this purpose, it is desirable to image the widest possible range in the living body, and in the image summarization process, it should be prevented that a subject range that cannot be observed by deleting a given image is generated.

  On the other hand, a reference image (an image to be left, an image to be a candidate to be left depending on the setting method of the reference image) and a determination target image (an image to be deleted or not) are selected from the image sequence, A technique for performing image summarization processing based on the coverage of the determination target image is effective. Specifically, as shown in FIG. 12, the coverage area is calculated on the determination target image (second image in FIG. 12) by deforming the reference image (first image in FIG. 12). The subject imaged with the reference image corresponds to the subject imaged on the covering area of the determination target image. That is, the range outside the covered area (hereinafter referred to as a non-covered area) in the determination target image is an area that cannot be covered even if the reference image is left when the determination target image is deleted.

  Therefore, the ratio of the coverage area in the determination target image is calculated as the coverage ratio, and by determining whether to delete the determination target image based on the calculated coverage ratio, the degree of occurrence of the subject range that cannot be observed is determined. Control. For example, if the image to be judged is deleted when the coverage is equal to or higher than the threshold, and the image to be judged is not deleted when the coverage is less than the threshold, the degree of occurrence of an area that cannot be covered is controlled according to the threshold setting. it can.

  However, the determination of whether or not the image is to be deleted based on the coverage rate takes into consideration the ease of observation of the subject area on the remaining image even if the subject area captured on the image to be deleted is captured on the remaining image. Can not do it. For example, as shown in FIG. 6, a case is considered in which a subject that has been imaged in a given area on the determination target image is imaged in a very narrow area on the reference image. In the example of FIG. 6, from the viewpoint of coverage, since the subject is captured on the reference image, it is determined that the subject is not missed even if the determination target image is deleted. The target image can be deleted. However, as apparent from FIG. 6, the size of the subject is very small on the reference image, so that it is basically suitable for observation although it depends on the number of pixels (resolution) of the reference image. I can't say that. If the size of the subject is small, there is a risk of missing the subject itself, and even if there is no oversight, detailed observation of the subject is necessary (for example, if the subject is a lesion, the subject is truly lesioned) This is because sufficient observation cannot be performed in the case of making a diagnosis of whether or not it is a part, or when it is a lesioned part, and how much progress is made.

  Therefore, the applicant selects the reference image and the determination target image, and based on the feature amount of the observation region on the determination target image and the feature amount of the corresponding region that is the region on the reference image corresponding to the observation region, A method for determining whether or not to delete a determination target image is proposed. At this time, the corresponding region is obtained by deforming the observation region based on the deformation information between the reference image and the determination target image. That is, the subject area imaged in the observation area and the subject area imaged in the corresponding area correspond (in a narrow sense, coincide).

  Therefore, for example, if the size information (area, etc.) of the region is used as the feature amount, it can be detected that the size of the corresponding region is smaller than the observation region as shown in FIG. It becomes possible to cope with the problem. Specifically, since the subject is collapsed into a narrow range (a range corresponding to the corresponding region) on the reference image and is not suitable for observation, it is determined that the determination target image cannot be deleted and is left in the summary image sequence. What is necessary is just to observe a to-be-photographed object in the state imaged in the wide range (range corresponding to an observation area | region) on the determination object image.

  As an embodiment of the image processing apparatus here, an apparatus including a processing unit 100 and an image sequence acquisition unit 200 as shown in FIG. The image sequence acquisition unit 200 acquires an image sequence having a plurality of images. Then, the processing unit 100 performs image summarization processing that acquires a summary image sequence by deleting some of the plurality of images included in the image sequence acquired by the image sequence acquisition unit 200. Specifically, the processing unit 100 selects a reference image and a determination target image for the image summarization process from a plurality of images, sets an observation region on the determination target image, and between the reference image and the determination target image. Based on the deformation information, a corresponding region that is a region on the reference image corresponding to the observation region is obtained, and at least one of the first feature amount obtained from the corresponding region and the second feature amount obtained from the observation region. Based on the above, it is determined whether or not the determination target image can be deleted.

  Hereinafter, detailed embodiments will be described. First, a basic method of image summarization processing that performs deletion feasibility determination based on the feature amounts of the observation region and the corresponding region in the first embodiment will be described. Further, when the deletion possibility determination based on the feature amounts of the observation area and the corresponding area is the first deletion possibility determination process, the first deletion possibility determination process is combined with a second deletion permission determination process different from that. Thus, the image summarization process may be performed. A specific method will be described in the second embodiment.

2. First Embodiment A basic method of this embodiment will be described. Specifically, an example of the system configuration of the image processing apparatus will be described, the flow of processing will be described using a flowchart, and details of the deletion permission / inhibition determination processing will be described with three examples.

2.1 System Configuration Example FIG. 1 shows a system configuration example of an image processing apparatus according to this embodiment. The image processing apparatus includes a processing unit 100, an image sequence acquisition unit 200, and a storage unit 300.

  The processing unit 100 performs image summarization processing on the image sequence acquired by the image sequence acquisition unit 200 by deleting some of the plurality of images included in the image sequence. The function of the processing unit 100 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program, and the like.

  The image sequence acquisition unit 200 acquires an image sequence that is an object of image summarization processing. In addition to storing the image sequence acquired by the image sequence acquisition unit 200, the storage unit 300 serves as a work area for the processing unit 100 and the like, and its function can be realized by a memory such as a RAM or an HDD (hard disk drive).

  Further, as illustrated in FIG. 1, the processing unit 100 includes a reference image selection unit 1002, a determination target image selection unit 1003, a deformation information acquisition unit 1004, an observation region setting unit 1005, a corresponding region setting unit 1006, An image feature amount calculation unit 1007, a deletion permission determination unit 1008, and a partial image sequence setting unit 1009 may be included. Note that the processing unit 100 is not limited to the configuration of FIG. 1, and various modifications such as omitting some of these components or adding other components are possible. Each of the above-described units is set to explain each subroutine when the image summarization process executed by the processing unit 100 is divided into a plurality of subroutines, and the processing unit 100 does not necessarily include the above-described units. Does not have as.

  The reference image selection unit 1002 selects a reference image from a plurality of images in the image sequence. The determination target image selection unit 1003 selects, as a determination target image, an image different from the reference image among a plurality of images in the image sequence.

  The deformation information acquisition unit 1004 acquires deformation information between two images. Here, the deformation information represents what shape (range) the range captured in one image is captured in the other image. For example, the deformation parameter disclosed in Patent Document 2 Etc. The deformation information acquisition unit 1004 acquires deformation information between the reference image selected by the reference image selection unit 1002 and the determination target image selected by the determination target image selection unit 1003.

  The observation area setting unit 1005 sets a partial area of the determination target image as an observation area. As the observation region, for example, a square region with one side length L may be used. The corresponding area setting unit 1006 deforms the observation area based on the deformation information acquired by the deformation information acquisition unit 1004 to obtain a corresponding area on the reference image.

  The image feature amount calculation unit 1007 calculates the feature amount of the observation region set by the observation region setting unit 1005 and the feature amount of the corresponding region set by the corresponding region setting unit 1006. A specific example of the feature amount will be described later.

  The deletion permission / inhibition determination unit 1008 is based on the observation region feature amount (second feature amount) calculated by the image feature amount calculation unit 1007 and the corresponding region feature amount (first feature amount). Determining whether images can be deleted. Details will be described later.

  When the deletion permission determination unit 1008 determines that the determination target image cannot be deleted, the partial image sequence setting unit 1009 determines a part of the image sequence based on the position in the image sequence of the determination target image at that time. Thus, an image sequence composed of one or more images is set as a partial image sequence.

2.2 Process Flow Next, the flow of the image summarization process of the present embodiment will be described using the flowchart of FIG. When this processing is started, first, an image sequence to be subjected to image summarization processing is acquired (S101). The image sequence is acquired by the image sequence acquisition unit 200, and an RGB 3-channel image arranged in time series can be considered. Alternatively, it may be an image sequence that is spatially continuous, such as an image sequence that is spatially arranged and photographed by imaging devices arranged in a horizontal row. In addition, the image which comprises an image sequence is not limited to RGB 3 channel image, You may use other color spaces, such as a Gray1 channel image.

  The reference image selection unit 1002 selects the first image of the input image sequence (the image sequence acquired in S101 in the first process and the partial image sequence set in S109 described later) as the reference image (S102). ). The reference image selected here is left in the summary image sequence. Note that when the reference image cannot be selected from the input image sequence due to an error or the like (for example, when there is no image in the image sequence), the process ends.

  Then, the determination target image selection unit 1003 selects a determination target image from images included in the input image sequence (S103). If the determination target image is not set, the next image of the reference image (second image in the input image sequence) is selected as the determination target image. If the k-th image in the input image sequence has already been selected as the determination target image, the selection position is shifted by one and the k + 1-th image in the input image sequence is selected as the new determination target image. If the determination target image cannot be selected (for example, if the number of images included in the input image sequence is less than 2 or k + 1), the process ends.

  When the reference image and the determination target image are selected, the deformation information acquisition unit 1004 acquires deformation information between the reference image and the determination target image (S104). Then, an observation area is set on the determination target image (S105). When the process of S105 is performed for the first time after setting the determination target image in S103, for example, the upper left area of the determination target image may be set as the observation area.

  When the observation area is set, the corresponding observation area is obtained by deforming the set observation area based on the deformation information acquired in S104 and projecting it on the reference image (S106). Here, the deformation information may be a non-rigid deformation parameter estimated by the method described in Patent Document 2, and how the subject imaged in the reference image is deformed on the determination target image. (Or vice versa). In other words, the subject imaged in the observation area corresponds to the subject imaged in the corresponding area on the reference image (same in a narrow sense).

  Then, the second feature amount that is the feature amount of the observation region and the first feature amount that is the feature amount of the corresponding region are obtained (S107), and the determination target is based on the obtained first and second feature amounts. It is determined whether or not an image can be deleted (S108). Details of the processing in S107 and S108 will be described later.

  If it is determined in S108 that the determination target image can be deleted, the process returns to S105 to reset the observation area. Specifically, the process of updating the set position of the observation area may be performed. As an example, as shown in FIG. 3, the position of the observation area on the determination target image is moved from the upper left to the lower right. Good. When the observation area is set, the processes of S106 to S108 are executed again.

  In S105, if the observation area has reached the lower right corner, it means that the deletion result determination result in S108 can be deleted in all observation areas set in the determination target image. The result that the image can be deleted is confirmed, and the process returns to S103 to update the determination target image.

  On the other hand, if the determination target image is determined to be unremovable even once in the updating of the observation area, the result that the determination target image cannot be deleted is confirmed, and the partial image sequence setting is performed. The unit 1009 sets a partial image sequence (S109). Specifically, an image sequence composed of a determination target image determined to be unremovable and subsequent images may be set as a partial image sequence. When the partial image sequence is set, the process returns to S102, and the above-described processing is executed using the partial image sequence as the input image sequence.

  FIGS. 4A to 4D illustrate the above image summarization processing. As shown in FIG. 4A, when an image sequence having N images is acquired by the image sequence acquisition unit 200, first, the first image is selected as the reference image, and the second image is selected. Selected as a determination target image. Then, it is determined whether or not the determination target image can be deleted based on the first and second feature amounts.

  If it is determined that the determination target image can be deleted, a new determination target image is selected. Specifically, the position of the determination target image is shifted backward, and the third image is selected as the determination target image as shown in FIG. Then, whether or not the determination target image can be deleted is determined between the reference image and the new determination target image, and the image selected as the determination target image is updated until a determination target image that is determined to be undeleteable is found.

  As shown in FIG. 4C, when it is determined that the second to k−1th images can be deleted and the kth image is determined to be undeleteable, the second to k−1th images are deleted. The image means that it is covered to some extent by the reference image (to the extent that the generation of an area unsuitable for observation can be suppressed), and is therefore deleted and not included in the summary image sequence. On the other hand, the k-th image cannot be sufficiently covered by the reference image, so it is necessary to leave it in the summary image sequence. For this purpose, the k-th image and the subsequent images (k to N-th images) are set as partial image sequences.

  Then, the processes shown in FIGS. 4A to 4C may be repeated for the partial image sequence. Specifically, as shown in FIG. 4D, a partial image sequence made up of N−k + 1 images is set as an input image sequence, and the first image (k-th in FIG. 4C) is the reference image, The second (k + 1th in FIG. 4C) image is processed as a determination target image. The subsequent processing is the same. When it is determined that the determination target image can be deleted, the next image is selected as a new determination target image. Further, if it is determined that the determination target image cannot be deleted, the reference image is left in the summary image sequence, the image determined to be deleteable is deleted, and the image after the determination target image at that time is replaced with a new partial image sequence. Set. Eventually, the process ends when it is determined that all of the last image in the input image sequence can be deleted, or when the determination target image cannot be set because only one image is included in the input image sequence. Will do.

  In the flowchart of FIG. 2, although it has been described that the determination target image cannot be deleted when it is determined that the deletion is impossible even once in S108, the present invention is not limited to this. For example, when a maximum M times (M is an integer equal to or greater than 2) observation area is set on the determination target image, an integer t that satisfies 1 ≦ t ≦ M is set, and it is determined that deletion cannot be performed more than t times in S108. In this case, the determination target image may not be deleted. Also, other methods may be used to determine the final result from the result of the determination of whether or not to delete in S108 performed at most M times. For example, the observation region (or the corresponding region) set in the peripheral portion of the image ), It is possible to perform various modifications such as weighting processing that emphasizes the result in the observation region (or corresponding region) set in the center of the image.

2.3 Deletability Determination Processing Details of the feature amount calculation processing in S107 and the deletion permission determination processing in S108 will be described next. In the following description, it is assumed that the observation area and the corresponding area are already set.

2.3.1 Deletability Determination Based on Lightness Information First, a method using lightness information of an observation area and a corresponding area as a feature amount will be described. Various index values representing the brightness information of the area can be considered. For example, the RGB values of the pixels in the area may be converted into brightness values, and the average value of the brightness values in the area may be used as the feature amount. Note that the method for obtaining the brightness information of the region from the brightness value of each pixel is not limited to the calculation of the average value, and a median value or the like may be used. The calculation of the average value is not limited to the simple average, and a weighted average may be used, or a trim average that excludes an extreme value may be used.

  Various methods for converting the RGB value of each pixel into a lightness value are also conceivable, but here, the largest one of the R pixel value, the G pixel value, and the B pixel value is used as it is as the lightness value. . However, it does not preclude obtaining the brightness value by other methods.

  Next, the deletion feasibility determination process based on the first feature quantity that is the brightness information of the corresponding area and the second feature quantity that is the brightness information of the observation area will be described. Here, it is determined whether the determination target image can be deleted based on two conditions.

The first condition is a condition determined by the upper limit threshold K_over and the lower limit threshold K_under. Specifically, a comparison process between the first feature amount and K_over and K_under is performed. More specifically, when the first feature amount satisfies the following expression (1), it is determined that the determination target image can be deleted, and otherwise, it is determined that deletion is not possible.
K_under ≦ first feature value ≦ K_over (1)

  Here, K_under may be set to a value that makes observation difficult because the area is too dark when the brightness information is smaller than the value, and the case where the first feature value is smaller than K_under. Typically, this indicates a situation where the corresponding area is blacked out and is not suitable for observation.

  On the other hand, when the brightness information is larger than the value, K_over may be set to a value that makes observation difficult due to the area being too bright. The case where the first feature value is larger than K_over Typically, this indicates a situation where the corresponding area is white and is not suitable for observation.

  By performing the determination based on the above formula (1), it is possible to detect that the corresponding area is whiteout or blackout (or close to it), so that the subject imaged in the corresponding area is observed. Is difficult. In this case, since the corresponding region and the observation region are set based on the deformation information, the corresponding region and the observation region have a correspondence relationship with the captured subject region, so that the determination target image is left in the summary image sequence and the observation region on the determination target image is defined. By referencing, appropriate observation becomes possible.

Even if the determination target image is left in the summary image sequence, the subject in question cannot be observed if the observation region is whiteout or blackout on the determination target image, and the determination target image is not changed. There are few benefits to leave. Therefore, not only the determination of the above equation (1) but also the following equation (2) is determined, and the determination target image is deleted when the above equation (1) is not satisfied and the following equation (2) is satisfied. A method may be used in which it is determined that the deletion is impossible and the deletion is possible otherwise.
K_under ≦ second feature amount ≦ K_over (2)

The second condition is a condition determined by a given threshold value K_light with respect to a difference value (in the narrow sense, its absolute value) between the first feature value and the second feature value. Specifically, when the following expression (3) holds, it is determined that the determination target image cannot be deleted.
| First Feature Value-Second Feature Value >>> K_light (3)

  When the absolute value of the difference value between the first feature value and the second feature value is large, as shown in FIG. 5, the brightness of the corresponding area and the brightness of the observation area are greatly different. It is assumed that the preferred brightness for observing the subject varies depending on the type of the subject and the like, and there are cases where a bright image is easier to observe and cases where a dark image is easier to observe. Although depending on the setting of the threshold value K_light, the situation where the above equation (3) is satisfied is assumed that one of the first and second feature values is large and the other is small, which is one of the corresponding region and the observation region. It corresponds to the extreme situation where is bright and one is dark. In other words, since the brightness of the corresponding area is not an intermediate value that can be used for general purposes, it may be difficult to observe the subject in the corresponding area depending on the situation. Therefore, the possibility that appropriate observation can be performed is improved by leaving the determination target image including an observation region having significantly different brightness in the summary image sequence.

  Even when the above equation (3) holds, the corresponding region may be more suitable for observation, and the observation region may be less suitable for observation. However, as described above, what brightness is suitable for observation depends on the situation, and since it is difficult to set in advance, if the above equation (3) holds in this embodiment, the determination target image is summarized. It is assumed that processing to be left in the image sequence is performed. That is, the method of the present embodiment allows an unnecessary determination target image to be left in the summary image sequence depending on the situation.

2.3.2 Deletability Determination Based on Size Information Next, a method of using the size information of the observation area and the corresponding area as the feature amount will be described. The size information corresponds to the area of the region, and may be obtained, for example, by counting the number of pixels included in the region. However, the calculation method of size information is not limited to this, and may be obtained by another method.

A problem when the size information is used as a feature amount is a situation where the size of the corresponding area becomes extremely small and the subject is crushed as shown in FIG. Therefore, here, when the value of the first feature value, which is the size information of the corresponding area, is small, the determination target image is left in the summary image sequence, and the subject collapsed in the corresponding area is observed in the observation area of the determination target image. It can be so. Although it does not preclude performing deletion determination from only the size of the corresponding region (that is, only the first feature amount), here the relative size such as the ratio of the size of the observation region to the size of the corresponding region is set. It shall be determined using. For example, when the following expression (4) is satisfied, it is determined that the determination target image cannot be deleted on the assumption that the size of the corresponding region is small enough to be unsuitable for observation.
(Second feature value / first feature value)> K_area (4)

  Note that the formula used for the determination is not limited to the above formula (4), and may be any formula based on the difference between the first feature value and the second feature value. For example, a difference value between the logarithm of the second feature value and the logarithm of the first feature value may be obtained, and a comparison process between the obtained difference value and a given threshold value may be performed. Alternatively, a difference value between the second feature amount and the first feature amount may be simply obtained, and a comparison process between the obtained difference value and a given threshold value may be performed.

2.3.3 Deletability Determination Based on Similarity with a given Shape Next, a method of using the similarity between the observation area and the given shape of the corresponding area as a feature amount will be described. The degree of similarity to a given shape represents how similar the region is to the given shape. For example, if the given shape is a circle, the circle obtained by the following equation (5) The degree may be used as a feature amount. However, the given shape is not limited to a circle, and the similarity calculation method is not limited to the following equation (5).

The problem when the degree of similarity with a given shape is used as a feature amount is the situation where the subject is collapsed due to the extreme shape of the corresponding region as shown in FIG. For example, in FIG. 7, the corresponding region is a rectangle whose short side is extremely short with respect to the long side, so that the subject is compressed in the short side direction, making observation difficult. Therefore, here, after setting a simple shape (for example, a circular shape or a square shape) suitable for observation as the given shape, the value of the first feature value that is the similarity to the given shape in the corresponding region Is small, the determination target image is left in the summary image sequence so that the subject collapsed in the corresponding region can be observed in the observation region of the determination target image. Although it does not preclude the determination of whether or not deletion is possible only from the similarity of the corresponding area (that is, only the first feature amount), here, the relative ratio of the similarity of the observation area to the similarity of the corresponding area, etc. Judgment shall be made using such information. For example, when the following expression (6) is satisfied, it is determined that the determination target image cannot be deleted, assuming that the shape of the corresponding region has become extreme to an extent that is not suitable for observation.
(Second feature value / first feature value)> K_shape (6)

  The point used for the determination is not limited to the above equation (6), which is the same as the example using the size information. For example, a difference value between the logarithm of the second feature value and the logarithm of the first feature value may be obtained, and a comparison process between the obtained difference value and a given threshold value may be performed. Alternatively, a difference value between the second feature amount and the first feature amount may be simply obtained, and a comparison process between the obtained difference value and a given threshold value may be performed.

2.3.4 Combination of determinations based on a plurality of feature amounts In addition, the above-described deletion permission determination using brightness information, deletion permission determination using size information, and deletion permission determination using similarity to a given shape. Of these, two or more may be used in combination.

  Various methods are conceivable for the combination method in this case, but if the emphasis is on the suppression of the generation of an unsuitable region, the possibility that the determination target image is determined to be undeleteable is increased. . Therefore, when a plurality of types of feature amounts are used, the deletion possibility determination is performed for each feature amount, and the determination target image is deleted when it is determined that the deletion is possible in all the deletion permission determinations. On the other hand, when it is determined that deletion is not possible in at least one of the plurality of deletion permission determinations, the determination target image is not deleted. By doing in this way, it is determined that the corresponding region is determined to be unsuitable for observation from at least one feature amount, and the determination target image is left, so that the target subject can be appropriately observed. It becomes possible to increase.

  However, by making it easy to determine that the determination target image cannot be deleted, the number of images remaining in the summary image sequence after the image summarization process increases, and the effect of reducing the number of images may decrease. Therefore, if importance is attached to the reduction of the number of images, a method that is more likely to determine that the determination target image can be deleted may be used as compared with the above method.

  For example, in the same way as the above method, when deletion is not possible in at least one of the plurality of deletion determinations, even if the determination target image is not deleted, It is conceivable to change the determination condition in the deletion permission / inhibition determination. For example, when the similarity between the size information and a given shape is used in combination, the value of K_area and the value of K_shape may be increased as compared with the case where each is used alone. In this way, since it is easy to determine that deletion is possible in each deletion possibility determination, the number of images included in the summary image sequence after image summarization processing can be determined from two different viewpoints of size and similarity. An excessive increase can be suppressed.

  In the above-described embodiment, as illustrated in FIG. 1, the image processing apparatus includes an image sequence acquisition unit 200 that acquires an image sequence having a plurality of images, and a plurality of images in the image sequence acquired by the image sequence acquisition unit 200. A processing unit 100 that performs image summarization processing for deleting a part of the image and acquiring a summary image sequence. The processing unit 100 selects a reference image and a determination target image for image summarization processing from a plurality of images, sets an observation region on the determination target image, and based on deformation information between the reference image and the determination target image. Then, a corresponding area which is an area on the reference image corresponding to the observation area is obtained. Then, based on at least one of the first feature amount obtained from the corresponding region and the second feature amount obtained from the observation region, it is determined whether or not the determination target image can be deleted.

  Here, the observation region is a region set on the determination target image, and is narrower than the determination target image in a narrow sense. If it is determined whether or not the entire subject imaged in the determination target image is in a state suitable for observation in the reference image, the observation region is selected when a set of the reference image and the determination target image is selected. Needs to be set a plurality of times so as to cover the entire determination target image while changing the position on the determination target image. For example, all the pixels included in the determination target image are included in the observation area at least once, and this sets the observation area while shifting by one pixel unit from the upper left to the lower right as shown in FIG. It is realized by doing. However, since the processing load increases as the number of times of setting the observation area increases, the amount of calculation may be reduced by moving the observation area in units of the length of one side of the observation area.

  In addition, when using size information as a feature amount, a necessary and sufficient region for appropriate observation may be set as an observation region. In this way, since the second feature value represents an appropriate reference value, the size of the corresponding region is suitable for observation by comparing the second feature value and the first feature value. You can determine whether or not. Similarly, if the feature amount is similar to a given shape (here, a shape suitable for observation, such as a circle or a square), the observation region is a shape close to the given shape (in the narrow sense, the same shape) ) May be used. In addition, although it is assumed that the observation area is an area having a certain size and shape in a series of processes, this does not prevent the observation area from being variably set.

  Accordingly, it is possible to determine whether or not the determination target image can be deleted based on the feature amount of the observation region on the determination target image and the feature amount of the corresponding region on the reference image. If the corresponding area is an area obtained by deforming the observation area based on the deformation information, the observation area and the subject imaged in the corresponding area have a correspondence relationship. Therefore, when it is determined that the corresponding area is not suitable for observation based on the feature amount, the observation area in which the corresponding subject (the same in a narrow sense) is imaged is made in a state where it can be browsed even after image summarization processing. This can be realized by a process of including the determination target image in the summary image sequence (determining that the determination target image cannot be deleted).

  The first feature amount may be at least one of brightness information, size information, and similarity information with a specific shape of the corresponding region. In addition, the second feature amount may be at least one of lightness information, size information, and similarity information with a specific shape of the observation region.

  Here, the similarity between a given area and a specific shape is an index value that represents how close the given area is to a specific shape. For example, when a circle is considered as a specific shape, in a regular k-gon (k is an integer of 3 or more) region, the smaller the k, the smaller the similarity, and the larger the k, the greater the similarity. In addition, if a specific shape has symmetry like a circle, the degree of similarity tends to be higher in a region having symmetry than in a region having no symmetry.

  This makes it possible to use at least one of lightness information, size information, and similarity information with a specific shape as a feature amount. In other words, the image processing apparatus according to the present embodiment determines whether or not the subject in the corresponding region is suitable for observation due to one of brightness, size, and shape. The target image is left in the summary image sequence. Specifically, when the brightness is too bright (out-of-white), too dark (blackout), the size is extremely small, or the shape is extreme (when it is crushed or distorted), etc. It is considered as an unsuitable state.

  In addition, the processing unit 100 performs a first comparison process between the first feature quantity and the first threshold, a second comparison process between the second feature quantity and the second threshold, and the first feature quantity. The determination as to whether or not the determination target image can be deleted may be performed based on at least one comparison process among the third comparison processes between the second feature amount difference and the third threshold value.

  Here, the degree of difference represents the degree of difference between the first feature quantity and the second feature quantity, and the difference or ratio, or a value corresponding thereto (for example, the logarithm difference of each feature quantity). Etc.). When the brightness information is used as the feature quantity, a large difference means that one of the corresponding area and the observation area is bright and the other is dark. Further, when the size information is used as the feature amount, the fact that the degree of difference is large represents a situation where one area of the corresponding area and the observation area is large and the other area is small. Similarly, when the degree of similarity with a specific shape is used as a feature quantity, a large difference is that one of the corresponding region and the observation region is a shape close to the specific shape, and the other is from the specific shape. Represents the situation of a distant shape.

  This makes it possible to determine whether or not to delete based on at least one of the first to third comparison processes. The first comparison process is a process that is performed based on the first feature quantity that is the feature quantity of the corresponding area. For example, as shown in the above equation (1), it is determined whether the corresponding area is over-exposure or black-out. It is assumed that it is used in a situation where determination can be made regardless of the state of the observation area. The second comparison process is a process that is performed based on the second feature value that is the feature value of the observation region. For example, as shown in the above equation (2), whiteout or blackout in the observation region is determined. It is assumed that it will be used when Although the second comparison process is not prevented from being used alone, it is desirable to use it together with the first comparison process or the like in consideration of the effect of reducing the number of images after the image summarization process. On the other hand, since the third comparison process is a process based on the difference between the first feature quantity and the second feature quantity, the third comparison process is a process that takes into consideration the states of both the corresponding area and the observation area. By performing the processing, it is possible to determine whether or not deletion can be performed with high accuracy. It should be noted that the comparison between the brightness information of the corresponding region and the size information of the observation region is not useful, and the two feature amounts used in the third comparison process need to have a correspondence relationship. That is, if one feature value is lightness information, the other feature value used in the third comparison process is also lightness information. If one is size information, the other is size information, and one is similar to a specific shape. If it is a degree, the other also becomes a similarity with a specific shape.

  Further, the processing unit 100 obtains the brightness information of the corresponding area as the first feature amount based on the pixel value of the pixel of the corresponding area, and calculates the brightness information of the observation area based on the pixel value of the pixel of the observation area. You may obtain | require as a 2nd feature-value.

  This makes it possible to determine whether or not deletion is possible using the brightness information as the feature amount. The brightness information of the corresponding area and the observation area is obtained based on the calculated brightness for each pixel included in the area. The brightness of the pixel uses the maximum value among the R, G, and B pixel values, but may be obtained by other methods (for example, an average value of the maximum value and the minimum value). In addition, as a method for obtaining the brightness information of the region from the brightness of the pixel, it is only necessary to take the average value of the brightness of all the pixels included in the region, but this also applies to other methods (median, weighted average, trimmed average) Etc.) may be used. As a specific example of the determination as to whether or not deletion is possible based on the brightness information, a process for making the determination target image unremovable when the corresponding region is white or blacked out and is not suitable for observation can be considered. In this way, it is possible to leave a determination target image including an observation region that is assumed to be capturing the same subject as the corresponding region in the summary image sequence, thereby enabling appropriate observation of the subject.

  In addition, the processing unit 100 deletes the determination target image when the first feature amount that is the brightness information is larger than a given upper limit threshold value or when the first feature amount is smaller than a given lower limit threshold value. You may determine that it is impossible.

  Thereby, the process shown by the above formula (1) becomes possible. This is a process assuming that the corresponding region is whiteout or blackout (or close to it). Therefore, when the brightness information is larger than that as the upper threshold, a value is set so that the subject is too bright to be suitable for observation, and when the brightness information is smaller than that as the lower threshold. May be set so that the subject is too dark to be suitable for observation.

  Further, the processing unit 100 may obtain a value represented by the size information of the corresponding area as the first feature amount, and obtain a value represented by the size information of the observation area as the second feature amount.

  This makes it possible to determine whether or not deletion is possible using size information as a feature amount. For example, information corresponding to the area may be used as the size information, and specifically, the number of pixels included in the region may be counted. As a specific example of the determination as to whether or not deletion is possible based on the size information, as shown in FIG. 6, when the area of the corresponding region is very small and the subject is collapsed and is not suitable for observation, the determination target image cannot be deleted. Can be considered.

  In addition, the processing unit 100 obtains a value representing the degree of similarity between the corresponding region and the given shape as the first feature amount, and uses the value representing the degree of similarity between the observation region and the given shape as the second feature. It may be obtained as an amount.

  As a result, it is possible to determine whether or not deletion is possible using the similarity with a specific shape as a feature amount. As the similarity to the specific shape, for example, if the specific shape is a circle, the circularity of the above equation (5) may be used. As a specific example of the determination of whether or not to delete based on the similarity to a specific shape, the shape of the corresponding area is extreme as shown in FIG. 7, and the subject is crushed (in the example of FIG. 7, it is crushed in the vertical direction). Thus, when it is not suitable for observation, a process for making it impossible to delete the determination target image is considered.

  Further, the processing unit 100 may determine that the determination target image cannot be deleted when the degree of difference between the first feature amount and the second feature amount is greater than a given threshold.

  As a result, it is possible to determine whether or not deletion is possible using the feature amounts of both the corresponding area and the observation area. When brightness information is used as a feature amount, a situation where the degree of difference is large is a situation where one of the corresponding area and the observation area is bright and the other is dark. Brightness suitable for observation is not a value that can be determined uniquely, such as the shape and color of the subject of interest, or the color of the background image captured behind the subject of interest, etc. It varies depending on the relationship. If the brightness information of the corresponding area is an intermediate value, there is a possibility that it can be used universally to some extent, but such a situation is unlikely in the situation where the degree of difference is large as described above. In other words, when the degree of difference is large (for example, the above equation (3) holds), the possibility that the corresponding region has brightness that is not suitable for observation must be taken into consideration, and the determination target image cannot be deleted. Good. However, the corresponding region is in a state suitable for observation, and the determination target image including the observation region unsuitable for observation may be left in the summary image sequence by the processing using the above formula (3) or the like. Need attention.

  In addition, when using size information as a feature amount, it is a problem that the area of the corresponding region is small. In this case, it is assumed that the determination target image remains if the area of the observation region is not large enough to be suitable for observation. Is also not effective. Therefore, as a situation where the degree of difference is large, it is preferable to detect that the area of the corresponding region is sufficiently small with respect to the area of the observation region as in the above equation (4). In the process using size information as a feature value, if the size and shape of the set observation region are unchanged, the second feature value is assumed to be unchanged, so a value calculated in advance It is also possible to continue using. In this case, since the second feature value of the above formula (4) is a fixed value, the determination is substantially based on the first feature value in terms of processing. However, the observation area of the present embodiment prevents dynamic setting (for example, when the observation area changes according to a user instruction or when an area corresponding to the attention area detected by image processing is set). Therefore, in such a case, it is necessary to obtain the second feature amount each time.

  In addition, when using a similarity with a specific shape (a shape suitable for observation) is used as a feature quantity, there is a problem that the shape of the corresponding region becomes extreme. In this case, the shape of the observation region If it is not close to a specific shape to some extent, it is not effective even if the image to be judged is left. Therefore, as a situation where the degree of dissimilarity is large, it may be detected that the degree of similarity of the corresponding region is sufficiently small with respect to the degree of similarity of the observation region as in the above equation (6). Also in this case, if the shape of the observation region is not changed, the second feature amount can be obtained in advance. For example, an area having the same shape as a specific shape may be set as the observation area.

  Further, the processing unit 100 obtains deformation information between adjacent images for an image between the reference image and the determination target image in the image sequence among the reference image, the determination target image, and the plurality of images, and the acquired adjacent images. The deformation information between the reference image and the determination target image may be obtained based on the deformation information.

  Thereby, when the reference image and the determination target image are not adjacent to each other, it is possible to calculate by accumulating the deformation information obtained between the adjacent images instead of directly obtaining the deformation information between the two images. Become. The deformation information can be calculated by the method shown in Patent Document 2 or the like, but generally, the process of combining a plurality of pieces of deformation information is much lighter than the process of calculating the deformation information from scratch. For example, if the deformation information is a matrix or the like, the process of obtaining the matrix from two pieces of image information is heavy, but combining a plurality of already obtained matrices (for example, it is only necessary to take the matrix product) ) It will be very easy.

  Note that the image processing apparatus or the like according to the present embodiment may realize part or most of the processing by a program. In this case, a processor such as a CPU executes the program, thereby realizing the image processing apparatus according to the present embodiment. Specifically, a program stored in the information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes of the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

3. Second Embodiment Next, in addition to the deletion feasibility determination using the first and second feature amounts described in the first embodiment (hereinafter referred to as a first deletion propriety determination process), what is it? A method of using different second deletion permission / inhibition determination processes together will be described.

  Specifically, a second deletion possibility determination process is performed on each image of the image sequence acquired by the image sequence acquisition unit 200, and a summary candidate image sequence made up of summary images determined not to be deleted can be deleted. A deletion candidate image sequence made up of deletion candidate images determined to be is set. Thereafter, the first deletion possibility determination process is performed to generate a summary image sequence. At this time, the reference image is selected from the summary candidate image sequence and the determination target image is selected from the deletion candidate image sequence.

  That is, according to the method of the present embodiment, the second deletion permission determination process is first performed as a pre-process, the remaining image and the image to be deleted are provisionally determined, and then the first deletion permission determination process using the temporary result is performed. This is a technique for performing a two-stage process of determining the final result. As a result, it is possible to improve the determination accuracy and the like compared to the case where the image summarization process based on only one of the first deletion permission determination process and the second deletion permission determination process is performed. This is because the second deletion permission determination process uses a process different from the first deletion permission determination process, and thus allows processing from a different viewpoint.

  For example, the determination based on the above-described coverage may be performed as the second deletion permission determination process. Although the image summarization process that guarantees the degree of coverage of the area is possible by the determination based on the coverage rate, an area that is difficult to observe can be generated as shown in FIG. That is, in the second deleteability determination process using the coverage rate, an image that should not be deleted is excessively deleted. Therefore, the first deleteability determination process using the result of the second deleteability determination process is performed. By doing so, it is possible to restore the excessively deleted images, and more appropriate image summarization processing can be performed.

  Hereinafter, an example of the system configuration of the image processing apparatus will be described, the flow of the process will be described using a flowchart, and then the details of the second deletion permission determination process will be described with two examples.

3.1 System Configuration Example FIG. 8 shows a system configuration example of the image processing apparatus according to this embodiment. The partial image sequence setting unit 1009 is removed from the processing unit 100 of FIG. 1, and a summary candidate image sequence generation unit 1001 is added. Detailed description of the same configuration as in the first embodiment will be omitted.

  The summary candidate image sequence generation unit 1001 performs a second deletion possibility determination process on the image sequence acquired by the image sequence acquisition unit 200, and includes a summary candidate image sequence that is a summary image that is an image remaining in the summary image sequence. Is generated. At the same time, among the image sequences acquired by the image sequence acquisition unit 200, a deletion candidate image sequence composed of deletion candidate images that are images not included in the summary candidate image sequence may be set.

  The determination target image selection unit 1003 according to the present embodiment selects a determination target image from images included in the deletion candidate image sequence. Further, the reference image selection unit 1002 sets the summary candidate image sequence according to the position of the determination target image selected by the determination target image selection unit 1003 in the image sequence (the image sequence acquired by the image sequence acquisition unit 200). A reference image is selected from the included images. Details of processing in the reference image selection unit 1002 and the determination target image selection unit 1003 will be described later.

3.2 Processing Flow Next, the flow of image summarization processing according to the present embodiment will be described with reference to the flowchart of FIG. When this processing is started, first, an image sequence to be subjected to image summarization processing is acquired (S201). Hereinafter, the image sequence acquired in S201 is also referred to as an acquired image sequence in order to clearly distinguish it from other image sequences.

  Then, a second deletion possibility determination process is performed on the acquired image sequence, and a summary candidate image sequence and a deletion candidate image sequence are set (S202). A specific example of the second deletion permission determination process in S202 will be described later. For example, as shown in FIG. 10, the third and eighth images are deleted from the 12 images included in the acquired image sequence. When it is determined that other images cannot be deleted, the summary candidate image sequence is composed of the third and eighth images, and the deletion candidate image sequence is composed of the other 10 images. become. Depending on the processing content of the second deletion permission determination process, the deletion permission determination may be performed a plurality of times for one image, but the deletion permission determination here is the end of the second deletion permission determination process. This indicates the final result of the time, and it is not always determined that the image is a deletion candidate image or a summary image immediately because it is determined that the image can be deleted once or cannot be deleted.

  After the second deleteability determination process, a determination target image is selected (S203). Here, the determination target images are selected in order from the top of the deletion candidate image sequence. Therefore, when the process of S203 is performed for the first time, the first image in the deletion candidate image sequence is selected. In the second and subsequent processing of S203, the determination target image update process for selecting the next image in the deletion candidate image sequence is performed on the current determination target image.

  When the determination target image is selected, a reference image is selected from the summary candidate image sequence according to the position of the determination target image in the acquired image sequence (S204). Here, in the acquired image sequence, an image closest to the determination target image is selected as the first reference image from the summary images ahead of the determination target image, and the summary image is behind the determination target image. Of these, the image closest to the determination target image is selected as the second reference image. However, if there is no summary image in front of or behind the determination target image, the corresponding reference image is not selected.

  In the example of FIG. 10, when the first image in the deletion candidate image sequence is selected as the determination target image, there is no summary image ahead of the determination target image in the acquired image sequence, so the first reference image is Not selected. In addition, there are two summary images (third and eighth images in the acquired image sequence) behind the determination target image, and among them, the third image closest to the determination target image (summary candidate image sequence). The first image) becomes the second reference image.

  In the example of FIG. 10, if the third to sixth images in the deletion candidate image sequence are selected as the determination target images, the first reference image becomes the first image in the summary candidate image sequence, and the second reference image The image is the second image in the summary candidate image sequence. Similarly, if the seventh to tenth images in the deletion candidate image sequence are selected as the determination target images, the first reference image becomes the second image in the summary candidate image sequence, and the second reference image is not selected. become.

  Deformation information acquisition processing (S205), observation region selection processing (S206), corresponding region selection processing (S207), image feature amount calculation processing (S208), deletion permission determination processing after the reference image and determination target image are selected. Since (S209) is the same as S104 to S108 in FIG. 2, detailed description thereof is omitted.

  When two reference images are selected, the first deletion permission determination process using the first reference image and the determination target image, and the first deletion permission determination based on the second reference image and the determination target image. Each process is performed, and when it is determined that deletion cannot be performed in both determinations, a process for leaving the determination target image in the summary image sequence as being undeleteable may be performed. This is because even if the corresponding region on one reference image is not suitable for observation, if the corresponding region on the other reference image capturing the same subject is suitable for observation, the determination target image is a summary image. This is because the merit to leave in the queue is small.

  Further, in the present embodiment, a search for an image to be left temporarily is performed by the second deletion permission determination process. Therefore, a process for setting a partial image sequence as in the first embodiment (FIG. 2). (Corresponding to S109) is unnecessary.

3.3 Second Deletability Determination Next, a specific example of the second deletion permission determination process (summary candidate image sequence setting process) performed in S202 of FIG. 9 will be described. Below, two methods, a method using coverage and a method based on structural elements, will be described, but the second deletion permission determination process is not limited to this. For example, the second deletion permission determination process may be performed based on a scene change or the like, and an arbitrary deletion permission determination process (or image summarization process) different from the first deletion permission determination process is applicable.

3.3.1 Second Deletability Determination Based on Coverage First, the second deleteability determination process based on the coverage will be described. Here, the first image and the second image are selected from the acquired image sequence, and the ratio of the second image covered by the first image is defined as the coverage. This process is a process for determining whether or not the area imaged there is sufficiently covered by the first image even if the second image is deleted. Therefore, from the viewpoint of whether it is an image to be left (or its candidate) or an image to be determined whether it can be deleted, the first image has the same meaning as the reference image in the first deletion permission determination process. The second image has the same meaning as the determination target image. Here, different terms are merely used so as not to confuse the image selected in the first deletion permission determination process with the image selected in the second deletion permission determination process.

  FIG. 11 shows a configuration example of the summary candidate image sequence generation unit 1001. The summary candidate image sequence generation unit 1001 includes a first image selection unit 101, a second image selection unit 102, a coverage area calculation unit 103, a coverage rate calculation unit 104, a deletion possibility determination unit 105, and a partial image. A column setting unit 106. The summary candidate image sequence generation unit 1001 is not limited to the configuration of FIG. 11, and various modifications such as omitting some of these components or adding other components are possible. Each of the above-described units is set to explain each subroutine when the second deletion possibility determination process executed by the summary candidate image sequence generation unit 1001 is divided into a plurality of subroutines. The image sequence generation unit 1001 does not have the above-described units as constituent requirements.

  The first image selection unit 101 selects the first image. The second image selection unit 102 selects the second image. Details of the selection method will be described later.

  The covering area calculation unit 103 deforms the first image based on deformation information between the first image and the second image (assumed to be acquired by the deformation information acquisition unit 1004 in FIG. 8). Then, the projected area is obtained by projecting onto the second image. Specifically, as shown in FIG. 12, the covered area corresponds to an area covered by the first image in the second image.

  The coverage calculation unit 104 calculates the coverage based on the calculated coverage region. For example, the coverage can be obtained from the ratio of the area of the covered region to the area of the entire second image as shown in FIG. However, the coverage may be information indicating the degree of coverage of the second image by the first image, and is not limited to the ratio / ratio. For example, as shown in FIG. 13, a plurality of points set on the second image are projected on the first image based on the deformation information, and the first image in the first image corresponding to the number of the plurality of points is projected. The ratio of the points included in (or the number of points included in the first image itself) may be used as the coverage.

  Deletability determination unit 105 determines whether or not the second image can be deleted based on the calculated coverage. Deletability determination based on the coverage rate may be performed by comparing with a given threshold value. If the coverage rate is equal to or greater than the threshold value, the second image can be deleted. If the coverage rate is smaller than the threshold value, the second image can be deleted. It is only necessary to make the image cannot be deleted. Even if the second image is deleted based on the determination as to whether or not deletion is possible based on the coverage, it is guaranteed that the first image can cover an area with a certain degree (corresponding to the threshold) captured in the second image. Is done.

  The partial image sequence setting unit 106 sets a partial image sequence based on the determination result in the deletion permission / inhibition determination unit 105. For example, the same processing as the partial image sequence setting unit 1009 in FIG.

  Next, the flow of the second deletion permission determination process shown in S202 of FIG. 9 will be described using the flowchart of FIG. When this process is started, first, the first image is selected from the image sequence (acquired image sequence) acquired in S201 (S301), and if the first image can be selected, the second image is selected. (S302). Here, the selection method of the first image and the second image may be the same method as that shown in FIGS. 4A to 4D, for example. Specifically, the first image in the input image sequence (acquired image sequence or partial image sequence set in S307) is the first image, and the image one image behind the first image is the first second image. And If it is determined in S306 that the image can be deleted and the process returns to S302, the second image may be updated to the next image.

  When the first image and the second image are selected, deformation information between them is acquired (S303). Based on the deformation information, the first image is deformed to obtain a coverage area (S304), and the coverage is calculated from the coverage area (S305).

  Whether or not the second image can be deleted is determined based on a comparison process between the coverage ratio and the threshold (S306). If it is determined that the second image can be deleted, the process returns to S302 and the second image is updated. If it is determined that the image cannot be deleted, the second image at that time and the image behind the second image are set in the partial image sequence (S307). After setting the partial image sequence, the process returns to S301 and the above-described processing is performed again on the partial image sequence.

  However, the selection method of the first image and the second image is not limited to the above-described method. For example, a plurality of first images may be selected. In this case, if the second image can be covered by any one of the plurality of first images, the second image can be deleted. Therefore, each first image is obtained by deformation as shown in FIG. Processing may be performed using a region corresponding to the union of the regions as a covering region.

  As a method of selecting a plurality of first images, as shown in FIGS. 16 (A) and 16 (B), a total of two first images, one from the front of the second image and one from the back. The method of selecting the image of this can be considered. In this case, an image between the two first images is sequentially selected as the second image. If all the images between the two first images can be deleted, the two first images are used as summary images, and the image between them is used as a deletion candidate image. A second deletion permission / inhibition determination process that guarantees the degree of coverage of the candidate image can be performed.

  However, the second deletion is performed from the viewpoint of reducing the number of summary images (it is possible to increase the possibility that the number of images after the final image summarization processing will decrease depending on the result of the first deletion permission determination process). If the determination process is performed, the condition that all the images between the first image (front) and the first image (back) can be deleted is satisfied, and the first image (front) and the first image are included therein. It is preferable to search for a position where one image (backward) is farthest away. In this case, when the first image (front) is fixed, the kth image is set as the first image (rear) as shown in FIGS. 16 (A) and 16 (B). In this case, all the images in the middle can be deleted. However, when the k + 1th image is set as the first image (backward), k is searched for such that at least one of the images in the middle cannot be deleted. In this case, the partial image sequence setting unit 106 may set the kth image and subsequent images as the partial image sequence.

  In addition, the selection of the first image and the second image can be realized by various methods.

3.3.2 Second Deletability Determination Processing Based on Structural Element Next, the second deletion permission determination processing based on the structural element will be described. Also in this case, the first image and the second image are selected in the same manner as the coverage example, and it is determined whether the second image can be deleted based on the first image.

  Here, processing is performed using a structural element corresponding to a region of interest (a region that is not desired to be overlooked, such as a lesion in the medical field). Specifically, as shown in FIG. 17A, a covered region covered with the first image and a non-covered region that is other than the second image are considered. Then, it is determined whether or not the attention area (for example, FIG. 17C) is completely included in the non-covering area. When the region of interest is completely included in the uncovered region as shown in FIG. 17B, the region of interest captured on the second image (the region of attention captured in its narrow sense in its entirety) is Therefore, the second image cannot be deleted because there is a possibility that the first image may not be captured at all. On the other hand, when the attention area is not completely included in the non-covering area (for example, in the case of FIG. 17A), the position of the attention area is captured at any position on the second image. Since at least a part of the image is captured by the first image, the second image can be deleted. Here, the structural element is used to determine whether or not the attention area is completely included in the non-covering area.

  FIG. 18 shows a configuration example of the summary candidate image sequence generation unit 1001. Compared to FIG. 11, the coverage rate calculation unit 104 is removed, and the structure element generation unit 107 and the attention area miss possibility determination unit 108 are added. Detailed description of the configuration common to FIG. 11 is omitted.

  The covered area calculation unit 103 according to the present embodiment may calculate a covered area and set an area that is not a covered area in the second image as a non-covered area.

  The structural element generation unit 107 generates a structural element used for processing in the attention area miss possibility determination section 108 based on the attention area. As an example, an area having the same shape and size as the attention area may be set, but the present invention is not limited to this.

  When the second image is deleted, the attention area missed possibility determination unit 108 is in a situation where the attention area captured on the second image is not captured in the first image (that is, the situation where the attention area is missed). To determine the possibility. Details will be described later.

  The deletion possibility determination unit 105 determines that the attention area is likely to be overlooked by the attention area overlookability determination section 108, and determines that the second image cannot be deleted. It is determined that the second image can be deleted.

  Next, the flow of the second deletion permission determination process shown in S202 of FIG. 9 will be described using the flowchart of FIG. Since S401 to S404 of this process are the same as S301 to S304 of FIG. 14, detailed description thereof is omitted.

  When the covering area and the non-covering area are calculated in S404, the possibility of missing the attention area is determined based on whether or not the attention area is completely included in the non-covering area (S405). Specifically, the shrinkage process using the structural element is performed on the non-covered area other than the covered area in the second image to determine whether or not there is a remaining area.

  A specific example of the contraction process will be described with reference to FIGS. 20 (A) to 20 (E). As shown in FIG. 20A, the non-covering region is always a closed region, and its boundary can be set. For example, in FIG. 20A, BO1 that is the outer boundary and BO2 that is the inner boundary are set.

  At this time, the contraction process by the structural element is a process of cutting the overlapping area between the non-covering area and the structural element when the reference point of the structural element is set on the boundary of the non-covering area. For example, when a circular region as shown in FIG. 17C is set as a structural element and the reference point is the center of the circle, a circle having a center is drawn on the boundary of the uncovered region, and the circle And a process of excluding a portion where the non-covering region overlaps from the non-covering region. Specifically, as shown in FIG. 20A, a circle centered on a point on the outer boundary BO1 of the non-covering region is drawn, and an overlapping region with the non-covering region (in this case, a half-line indicated by diagonal lines) (Circular area) is excluded.

  Considering that the outer boundary BO1 is discretely processed, the outer boundary BO1 is composed of a plurality of points. Therefore, the above-described processing may be performed for each point of the plurality of points. As an example, as shown in FIG. 20 (A), starting from one point on the boundary, a circle centering on the point on the boundary BO1 is sequentially drawn in a given direction, and the overlapping area with the non-covering area is not covered. What is necessary is just to exclude from a covering area | region.

  As shown in FIG. 17B, there may be one boundary depending on the shape of the non-covered region. In this case, the above-described processing may be performed for the one boundary. In addition, as shown in FIG. 20A, when two boundaries, BO1 and BO2, are considered as the boundary of the non-covering region, the above-described processing is performed for each. Specifically, as shown in FIG. 20B, for the inner boundary BO2, a circle having a center on BO2 is drawn and an overlapping area with the non-covering area is excluded, and this process is performed as BO2. What is necessary is just to repeat about each point which comprises.

  By performing such shrinkage processing, the area of the non-covering region is reduced. For example, when attention is paid to the left part of the non-covered region in FIG. 20A, the non-covering process is performed by BO1 shown in FIG. 20A and BO2 shown in FIG. The covered area is completely deleted and there is no remaining area. On the other hand, when attention is paid to the lower right portion of the non-covered region, as shown in FIG. 20C, a residual region RE remains that is not excluded in both the shrinking process in BO1 and the shrinking process in BO2. . Therefore, the result of performing the contraction process by the structural element on the entire uncovered region here is as shown in FIG. 20D, and the residual region RE is generated.

  Here, the meaning of contraction processing when a circle having a radius r is used as a structural element will be considered. The non-covered area which is a closed area is an area inside a boundary (may be a different boundary such as BO1 and BO2 or a single boundary as shown in FIG. 17B). Can think. By performing the above-described shrinkage processing on this boundary, among the points included in the uncovered area, points within the distance r from the points on the boundary are to be deleted. That is, when a point included in the remaining area that has not been deleted is considered, the distance from the point to an arbitrary point on the boundary is greater than r. Therefore, when a circle having a radius r centering on an arbitrary point on the residual region is drawn, the circumference of the circle does not intersect any boundary. In other words, this represents a situation in which the region of interest represented by a circle with a radius R (= r) is completely contained in the non-covered region with the point in the remaining region as its center. Even when a shape other than a circle (such as a square) is used as a structural element, the basic concept is the same.

  That is, the case where the residual region exists is a case where the region corresponding to the structural element is included in the uncovered region as shown in the lower right of FIG. If there is a region of interest, if the second image is deleted, there is a possibility that the region of interest cannot be observed even if the first image is left. Conversely, when there is no remaining area, as shown in the upper left of FIG. 20E, at least a part of the attention area is included in the covering area, and even if the second image is deleted. At least a part of the attention area can be left in the first image. From the above, the attention area missing possibility determination unit 108 performs the contraction process with the structural element on the uncovered area and outputs whether or not the remaining area exists as a result. It is possible to determine whether or not an image can be deleted.

  Then, the deleteability determination unit 105 determines whether or not the second image can be deleted based on whether or not there is a residual area as a result of the shrinking process (S406). Since the subsequent processing is the same as the example of FIG. 14, detailed description thereof is omitted.

  Even when processing is performed using structural elements, the same points are applicable in that the first and second images can be selected by various methods. Further, the method of determining the possibility of missing the attention area is not limited to the contraction process shown in FIGS. 20 (A) to 20 (E).

  In the present embodiment described above, the processing unit 100 sets a summary candidate image sequence composed of summary images remaining in the summary image sequence among a plurality of images of the image sequence acquired by the image sequence acquisition unit 200, and Among the plurality of images, a deletion candidate image sequence composed of deletion candidate images not included in the summary candidate image sequence is set. Then, the processing unit 100 selects a reference image from the summary candidate image sequence, selects a determination target image from the deletion candidate image sequence, sets an observation region on the determination target image, and sets a space between the reference image and the determination target image. Based on the deformation information, a corresponding area that is an area on the reference image corresponding to the observation area is obtained. Then, the processing unit 100 determines whether or not the determination target image can be deleted based on at least one of the first feature amount obtained from the corresponding region and the second feature amount obtained from the observation region. .

  As a result, as a pre-process of the deletion permission / inhibition determination process (first deletion permission / inhibition determination process) using the first and second feature quantities, the summary candidate image from the image string (acquired image string) as shown in FIG. It is possible to perform processing for setting a sequence and a deletion candidate image sequence. By selecting the reference image from the summary candidate image sequence and selecting the determination target image from the deletion candidate image sequence, it is determined that the summary image is included in the summary image sequence, and is the deletion candidate image really deleted or summarized? Processing for confirming whether or not to leave the candidate image sequence can be performed. Therefore, the summary candidate image sequence and the deletion candidate image sequence setting process may be somewhat inaccurate. For example, all of the plurality of images included in the acquired image sequence may be determined as deletion candidate images except for images that can be clearly determined as summary images. This is because, after that, a highly accurate determination is performed by the first deletion permission determination process, and as a result, there is a possibility that the deletion candidate image is included in the summary image sequence. However, since it is assumed that the summary image is left as it is in the summary image sequence, the summary candidate image sequence is considered in consideration of the final image summary processing result (for example, the effect of reducing the number of images). It is desirable that the deletion candidate image sequence setting process is not performed completely at random, but based on some criteria.

  In addition, the processing unit 100 performs a second deletion permission / prohibition that is different from the first deletion permission determination process and the first deletion permission determination process performed based on at least one of the first feature value and the second feature value. An image summarization process may be performed based on the determination process. Then, the processing unit 100 performs a second deletion permission determination process on the image sequence acquired by the image sequence acquisition unit 200, and sets an image determined to be undeleteable among a plurality of images as a summary image and deletes the summary image. The image determined to be possible is set as a deletion candidate image.

  As a result, a second deletion permission determination process different from the first deletion permission determination process using the first and second feature amounts is performed, and a summary candidate image string and a deletion candidate image string are set based on the result. It becomes possible to do. Accordingly, since some deletion possibility determination processing (image summarization processing) can be performed in the preprocessing, it is possible to increase the setting accuracy of the summary candidate image sequence and the deletion candidate image sequence. In addition, since different methods are used in combination by the first deletion permission determination process and the second deletion permission determination process, it is possible to improve the determination accuracy of the deletion permission determination.

  In addition, the processing unit 100 selects a first image and a second image from a plurality of images in the image sequence acquired by the image sequence acquisition unit 200, and deformation information between the first image and the second image The process of calculating the coverage of the second image based on the first image and determining whether the second image can be deleted based on the coverage is executed as the second deletion permission determination process. May be.

  Here, the coverage is information indicating how many of the subjects imaged on the second image are captured on the first image. Further, the deletion permission / prohibition determination process based on the coverage is, for example, a comparison process with a given threshold. Increasing the threshold value can be expected to improve the suppression effect against the occurrence of an area that cannot be observed by deleting the image. On the other hand, if the threshold value is lowered, the number of images included in the summary image sequence after the summary processing can be reduced. There is a trade-off between improving the suppression effect and reducing the number of images, and control is possible by setting the threshold value. Therefore, it is desirable to set the threshold value appropriately according to the situation.

  Thereby, it is possible to improve the determination accuracy of the determination as to whether or not deletion is possible by using both the processing based on the coverage and the processing based on the first and second feature amounts. Specifically, from the viewpoint of suppressing the occurrence of a subject area that cannot be observed as a result of deleting an image, and from the viewpoint of suppressing the generation of an area that is not suitable for observation (even if imaging is performed). It becomes processing from.

  In addition, the processing unit 100 selects a first image and a second image from a plurality of images in the image sequence acquired by the image sequence acquisition unit 200, and deformation information between the first image and the second image And a process for determining whether or not the second image can be deleted based on the result of the process using the process and the process using the structural element corresponding to the region of interest may be executed as the second deleteability determination process. .

  Here, the process using the deformation information may be a process of deforming at least a part of one of the first image and the second image using the deformation information. The process using the structural element is a process of determining whether or not the structural element is included in a non-covered area that is an area where the second image is not covered by the first image. There may be.

  In addition, the attention area is an area where the priority of observation for the user is higher than other areas. For example, when the user is a doctor and desires treatment, an area in which a mucous membrane portion or a lesion is copied is displayed. Point to. As another example, if the object that the doctor desires to observe is a bubble or stool, the region of interest is a region in which the bubble or stool portion is copied. In other words, the object that the user should pay attention to depends on the purpose of observation, but in any case, in the observation, an area in which the priority of observation for the user is relatively higher than other areas is the attention area.

  Accordingly, it is possible to improve the determination accuracy of the deletion feasibility determination by using both the processing based on the structural element and the processing based on the first and second feature amounts. Specifically, the processing is from the viewpoint of suppressing the possibility that the user misses the attention area, and from the viewpoint of suppressing the generation of an area not suitable for observation. Note that suppression of the possibility of missing the attention area is performed based on the idea that an image that captures at least a part of the attention area is always left in the summary candidate image sequence even if the second image is deleted. .

  The two embodiments 1 and 2 to which the present invention is applied and the modifications thereof have been described above, but the present invention is not limited to the embodiments 1 and 2 and the modifications as they are, The constituent elements can be modified and embodied without departing from the spirit of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described first and second embodiments and modifications. For example, you may delete a some component from all the components described in each Embodiment 1-2 or the modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. In addition, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings. Thus, various modifications and applications are possible without departing from the spirit of the invention.

100 processing unit, 101 first image selection unit, 102 second image selection unit,
103 coverage area calculation unit, 104 coverage rate calculation unit, 105 deletion possibility determination unit,
106 partial image sequence setting unit, 107 structural element generation unit,
108 attention area miss possibility determination unit, 200 image sequence acquisition unit, 300 storage unit,
1001 summary candidate image sequence generation unit, 1002 reference image selection unit,
1003 determination target image selection unit, 1004 deformation information acquisition unit,
1005 observation region setting unit, 1006 corresponding region setting unit,
1007 Image feature amount calculation unit, 1008 Deletion possibility determination unit,
1009 Partial image sequence setting unit,
BO1 outer boundary, BO2 inner boundary, RE residual area

Claims (16)

An image sequence acquisition unit for acquiring an image sequence having a plurality of images;
A processing unit that performs image summarization processing for acquiring a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit;
Including
The processor is
From the plurality of images, select a reference image and a determination target image for the image summarization process,
While setting an observation area on the determination target image, based on deformation information between the reference image and the determination target image, a corresponding area that is an area on the reference image corresponding to the observation area,
An image that determines whether or not the determination target image can be deleted based on at least one of a first feature amount obtained from the corresponding region and a second feature amount obtained from the observation region. Processing equipment. In claim 1,
The first feature amount is:
At least one of lightness information, size information, and similarity information of a specific shape of the corresponding region;
The second feature amount is
An image processing apparatus comprising: at least one of the brightness information, the size information, and the similarity information of the observation area. In claim 2,
The processor is
A first comparison process between the first feature quantity and a first threshold; a second comparison process between the second feature quantity and a second threshold; and the first feature quantity and the second threshold value. An image processing apparatus that determines whether or not the determination target image can be deleted based on at least one comparison process of the third comparison process between the difference degree of the feature amount and the third threshold value. In claim 1,
The processor is
The brightness information of the corresponding area is obtained as the first feature amount based on the pixel value of the pixel of the corresponding area, and the brightness information of the observation area is determined based on the pixel value of the pixel of the observation area. Is obtained as the second feature amount. In claim 4,
The processor is
When the first feature value that is the brightness information is larger than a given upper threshold value, or when the first feature value is smaller than a given lower threshold value, it is determined that the determination target image cannot be deleted. An image processing apparatus. In claim 1,
The processor is
Image processing characterized in that a value represented by the size information of the corresponding region is obtained as the first feature amount, and a value represented by the size information of the observation region is obtained as the second feature amount. apparatus. In claim 1,
The processor is
A value representing the similarity between the corresponding region and the given shape is obtained as the first feature amount, and a value representing the similarity between the observation region and the given shape is obtained as the second feature amount. An image processing apparatus characterized by obtaining as follows. In any of claims 4 to 7,
The processor is
An image processing apparatus, wherein when the degree of difference between the first feature amount and the second feature amount is greater than a given threshold, the determination target image is determined not to be deleted. In any one of Claims 1 thru | or 8.
The processor is
Among the plurality of images of the image sequence acquired by the image sequence acquisition unit, a summary candidate image sequence consisting of summary images remaining in the summary image sequence is set, and among the plurality of images of the image sequence, Setting a deletion candidate image sequence consisting of deletion candidate images not included in the summary candidate image sequence;
The processor is
Selecting the reference image from the summary candidate image sequence, selecting the determination target image from the deletion candidate image sequence,
The observation area is set on the determination target image, and the corresponding area that is an area on the reference image corresponding to the observation area is determined based on the deformation information between the reference image and the determination target image. Seeking
Determining whether or not the determination target image can be deleted based on at least one of the first feature amount obtained from the corresponding region and the second feature amount obtained from the observation region. An image processing apparatus. In claim 9,
The processor is
A first deletion permission determination process performed based on at least one of the first feature value and the second feature value, and a second deletion permission determination process different from the first deletion permission determination process Based on the image summarization process,
The processor is
The second deleteability determination process is performed on the image sequence acquired by the image sequence acquisition unit, and the image determined to be undeleteable among the plurality of images is used as the summary image and can be deleted. The image processing apparatus characterized in that the image determined to be the deletion candidate image. In claim 10,
The processor is
A first image and a second image are selected from the plurality of images of the image sequence acquired by the image sequence acquisition unit;
Based on the deformation information between the first image and the second image, the coverage of the second image by the first image is calculated,
An image processing apparatus that executes a process of determining whether or not the second image can be deleted based on the coverage as the second deleteability determination process. In claim 10,
The processor is
A first image and a second image are selected from the plurality of images of the image sequence acquired by the image sequence acquisition unit;
Determining whether or not the second image can be deleted based on a result of processing using the deformation information between the first image and the second image and processing using a structural element corresponding to a region of interest The image processing apparatus is characterized in that the process of performing is performed as the second deletion permission determination process. In claim 12,
The process using the deformation information is as follows:
A process of deforming at least a part of one of the first image and the second image using the deformation information;
The process using the structural element is as follows:
It is a process for determining whether or not the structural element is included in an uncovered area that is an area where the second image is not covered by the first image, or a contraction process by the structural element. Image processing device. In any one of Claims 1 thru | or 13.
The processor is
For the image between the reference image and the determination target image in the image sequence among the reference image, the determination target image, and the plurality of images, the deformation information between the adjacent images is obtained, and the obtained neighbor An image processing apparatus, wherein the deformation information between the reference image and the determination target image is obtained based on the deformation information between the matching images. An image sequence acquisition unit for acquiring an image sequence having a plurality of images;
As a processing unit that performs image summarization processing for acquiring a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit,
Make the computer work,
The processor is
From the plurality of images, select a reference image and a determination target image for the image summarization process,
While setting an observation area on the determination target image, based on deformation information between the reference image and the determination target image, a corresponding area that is an area on the reference image corresponding to the observation area,
A program for determining whether or not the determination target image can be deleted based on at least one of a first feature amount obtained from the corresponding region and a second feature amount obtained from the observation region. . Obtain an image sequence with multiple images,
A reference image and a determination target image are selected from the plurality of images in the image sequence,
While setting an observation area on the determination target image, based on deformation information between the reference image and the determination target image, a corresponding area that is an area on the reference image corresponding to the observation area,
Based on at least one of the first feature amount obtained from the corresponding region and the second feature amount obtained from the observation region, it is determined whether the determination target image can be deleted,
An image processing method for performing image summarization processing for acquiring a summary image sequence by deleting a part of the plurality of images of the image sequence based on the determination result of the deletion possibility.

Images