WO2019155628A1

WO2019155628A1 - Image processing device, image processing method, and recording medium

Info

Publication number: WO2019155628A1
Application number: PCT/JP2018/004691
Authority: WO
Inventors: 雅弘西光
Original assignee: 日本電気株式会社
Priority date: 2018-02-09
Filing date: 2018-02-09
Publication date: 2019-08-15
Also published as: JPWO2019155628A1; JP6988926B2

Abstract

The present invention provides an image processing device and the like capable of quickly displaying a frame indicating the whole of an irregular object. An image processing device 100 is provided with: an object area candidate generation unit 11 for generating a plurality of object area candidates indicating a position in an image where at least a part of an object to be detected exists; a heatmap generation unit 12 for generating a heatmap corresponding to the position in the image by using the generated object area candidates; and an object area output unit 13 for outputting an object area surrounding the whole of the object to be detected by using the generated heatmap, the object area being constituted of one or more object area candidates.

Description

Image processing apparatus, image processing method, and recording medium

The present invention relates to an image processing apparatus for facilitating confirmation of a detection target object in an image.

There is an image processing method for assisting in finding a detection target object in a moving image. Non-Patent Document 1 and Non-Patent Document 2 disclose examples of this image processing method. For example, this image processing method operates as follows in an apparatus including an object region candidate generation unit and an object region candidate output unit. That is, when an image (in the case of a moving image, each frame image constituting the moving image) is input, the object region candidate generation unit is configured to output a plurality of candidates (in the region where the object to be detected included in the image is considered to exist) Object region candidate) is generated. The object region candidates are rectangles or circles circumscribing the object to be detected, and these are variables indicating the shape of the object (for example, center coordinates (x, y), height (h) and width (w)), and This is expressed by the reliability indicating the content of the detection target object in the object region candidate. The object area candidate output means outputs a rectangle or a circle circumscribing the object to be detected based on the degree of overlap or reliability of each object area candidate from the object area candidates generated by the object area candidate generation means. To do.

In addition, Patent Document 1 discloses a technique related to the present application.

JP 2016-206995 A

In the image processing methods disclosed in Non-Patent Document 1 and Non-Patent Document 2, the detection target object has a regular shape and is an object that can be predicted to some extent. For example, the size, shape, etc. of the detection target object, such as a human face, is regularly predicted to some extent, and the predetermined height (h) and width (w) from the center coordinates (x, y) in the image It is an object that fits in. However, the object to be detected cannot be predicted irregularly in its size, shape, etc., and may not fit within a predetermined height and width preset from the center coordinates in the image (hereinafter referred to as this). Are called irregular objects). For example, it is a non-rigid body that does not fit in a certain shape, such as a lesion inside or outside an organ (such as a tumor), and can be irregularly deformed into various shapes as the disease progresses. In this case, it is difficult to determine and output an object region circumscribing the entire object using the image processing methods disclosed in Non-Patent Document 1 and Non-Patent Document 2. This is because the image processing methods disclosed in Non-Patent Document 1 and Non-Patent Document 2 generate a large number of object region candidates that are at least partially circumscribed with the object in the shape of an irregular object. This is because, generally, one object region candidate is set so as to include the detection target object as much as possible in the region and not to include anything other than the detection target object as much as possible. As a result, For one irregular object, a plurality of object region candidates partially overlap and are generated in succession. For example, if the detection target object is a round shape, a single object region candidate centered on the center of gravity is often sufficient, but if the detection target object is a starfish type, a plurality of object region candidates (centered on the center of gravity) are sufficient. Are often represented by object region candidates covering the vicinity of the center and a plurality of object region candidates covering a plurality of feet included in the starfish type). Further, since the latter object region candidate includes the detection target objects at a high ratio by the above setting, the reliability tends to be high. For these reasons, when the detection target is an irregular object, the image processing methods disclosed in Non-Patent Document 1 and Non-Patent Document 2 circumscribe the entire irregular object (or the detection target object). Rectangle or circle area candidates (including the whole) are difficult to output.

As a problem associated with this, in the image processing methods disclosed in Non-Patent Document 1 and Non-Patent Document 2, a part of a plurality of object region candidates are generated by overlapping one irregular detection target object. For the operator (for example, “doctor” in the case of a lesion) observing the detection target object, it is difficult to visually recognize the detection target object due to the multiple overlapping frames representing the object region candidates. There is a problem that it is difficult to judge the overall shape. Or, when the frame representing the object region candidate does not appropriately represent the detection target object for some reason, there is a problem that the observation by the operator is misleaded to the frame and the detection target object to be discovered is overlooked.

In the technique disclosed in Non-Patent Document 1, since region candidates are output based on the overlap and reliability of each region candidate, the object region candidate circumscribing the detection target object has higher reliability than other object region candidates. Become. However, the candidate object region circumscribing an irregular object contains many non-object regions, which generally results in lower reliability than a partially circumscribing rectangle, and thus accurate. There arises a problem that an object region candidate cannot be extracted.

In the image processing method disclosed in Non-Patent Document 2, a problem is that high-speed image processing is difficult because an object region of an image is generated or regenerated using a neural network that requires a large amount of calculation. is there.

Therefore, in view of the above-described problems, an object of the present invention is to provide an image processing apparatus or the like that displays a frame indicating the entire irregular object at high speed.

In view of the above problems, the image processing apparatus according to the first aspect of the present invention provides:
An object region candidate generation unit that generates a plurality of object region candidates indicating positions where at least a part of the detection target object exists in the image;
Using the generated object region candidate, a heat map generation unit that generates a heat map corresponding to the position in the image,
And an object region output unit that outputs an object region that surrounds the entire detection target object, which is made up of one or more object region candidates, using the generated heat map.

An image processing method according to a second aspect of the present invention includes:
Generating a plurality of object region candidates indicating positions where at least some of the detection target objects exist in the image;
Using the generated object region candidate, generate a heat map corresponding to the position in the image,
Using the generated heat map to output an object region that includes one or more object region candidates and surrounds the entire detection target object.

An image processing program according to a third aspect of the present invention is:
Generating a plurality of object region candidates indicating positions where at least some of the detection target objects exist in the image;
Using the generated object region candidate, generate a heat map corresponding to the position in the image,
An image processing program for causing a computer to output an object region that surrounds the entire detection target object, which is composed of one or more object region candidates, using a generated heat map.

The image processing program may be stored in a non-transitory computer-readable storage medium.

According to the present invention, it is possible to provide an image processing apparatus or the like that displays a frame indicating the entire irregular object at high speed.

1 is a block diagram illustrating a configuration example of an image processing apparatus according to a first embodiment of the present invention. 3 is a flowchart showing an operation in the image processing apparatus according to the first embodiment of the present invention. It is a figure which shows the object area | region candidate in an image. It is a figure which shows the heat map corresponding to an image. It is a figure which shows the object area | region in an image. It is a figure which shows the object area | region in an image with a detection target object. It is a block diagram which shows the structural example of the image processing apparatus concerning the 2nd Embodiment of this invention. It is a flowchart which shows operation | movement in the image processing apparatus concerning the 2nd Embodiment of this invention. It is a figure which shows the heat map and object area | region candidate corresponding to an image. It is a figure which shows the object area | region in an image with a detection target object. It is a block diagram which shows the structural example of the image processing apparatus concerning the 3rd Embodiment of this invention. It is a flowchart which shows operation | movement in the image processing apparatus concerning the 3rd Embodiment of this invention. It is a block diagram which shows the structural example of the information processing apparatus applicable in each embodiment.

In the following embodiment, an image processing apparatus or the like that mainly targets a detection target object having an irregular shape and an unpredictable object region will be described. Of course, the image processing apparatus can also detect an object having a regular shape and a predictable object region. In the following description, an irregular detection target object is a non-rigid body, for example, a lesion (tumor) that occurs inside or outside an organ. In this case, the user of the image processing apparatus is a medical person such as a doctor.

This image processing apparatus is incorporated in, for example, a real-time tumor detection system for endoscopic moving images. When there is a lesion detected in a moving image or still image (hereinafter simply referred to as an image) taken through the endoscope, the doctor can check that all or part of the internal or external organ where the lesion occurs is normal. By observing whether or not there is a mutation, the mutation type, malignancy and progression of the tumor are determined. In addition, when a doctor removes (removes) a tumor, it is necessary to accurately recognize the boundary between a tumor portion and a normal portion in a certain organ. For this reason, the image processing apparatus in the embodiment of the present invention displays a frame circumscribing the entire tumor accurately and at high speed as information indicating the position of the tumor. Thereby, it is possible to assist the diagnosis and treatment by the doctor quickly and accurately.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings schematically show the configuration in the embodiment of the present invention. Furthermore, the embodiment of the present invention described below is an example, and can be appropriately changed within a range in which the essence is the same.

<First Embodiment>
(Image processing device)
As illustrated in FIG. 1, the image processing apparatus 100 according to the first embodiment of the present invention includes an object region candidate generation unit 11, a heat map generation unit 12, and an object region output unit 13.

The object area candidate generation unit 11 generates a plurality of object area candidates indicating positions where at least a part of the detection target object exists in the image. Specifically, the object region candidate generation unit 11 inputs an image captured by an external camera (not shown in FIG. 1), for example, an endoscope camera, and there is a detection target object in the input image. A plurality of region candidates (object region candidates) are generated. As a method for generating a plurality of object region candidates in a region where an object to be detected included in an image is supposed to exist, existing methods described in Non-Patent Document 1, Non-Patent Document 2, and the like may be used. When there are too many object area candidates, the object area candidate generation unit 11 generates some combinations of the object area candidates suitable as the object area candidates, and outputs one of them as an object area candidate (group). May be.

The heat map generation unit 12 generates a heat map corresponding to the position in the image using the generated object region candidate. Specifically, the heat map generation unit 12 inputs one or more images including object region candidates, and generates a heat map related to the image based on the reliability of each object region candidate. Here, the reliability is a value indicating the probability that the detection target object exists in the object region candidate. For example, the degree of overlap of each object region candidate or the content rate of the detection target object in a certain object region candidate, etc. It is calculated based on the degree indicating Further, the heat map refers to a diagram in which a data matrix or the like indicating each value of a section (for example, pixel) in an image is visibly expressed as a color in two dimensions. In the heat map, the reliability score is set to be high in the region where the object exists.

The object region output unit 13 inputs the generated heat map, and outputs an object region (rectangular frame) indicating the entire irregular detection target object in the image based on the heat map. Specifically, the object region output unit 13 includes, for each of a plurality of sections (for example, for each pixel) into which the heat map is divided, an object including a set of sections whose reliability is equal to or higher than a predetermined threshold. A region candidate is output as the object region.

(Operation of image processing device)
The specific operation of the image processing apparatus 100 will be described with reference to the flowchart shown in FIG.

First, in step S101, the image processing apparatus 100 inputs an image to the object region candidate generation unit 11, and generates an object region candidate. A specific operation example of this step will be described below.

If the image processing apparatus 100 is, for example, an apparatus for recognizing a lesion (tumor) in an endoscopic image, the object region candidate generation unit 11 receives an image including a tumor, and for this image, An object region candidate indicating a region including a tumor is generated. As a technique for realizing high-speed object recognition, a technique that uses a rectangle that circumscribes an object to be detected as an object region is adopted. In this case, in the drawing function (Rectangle), the central coordinates (x, y), width (w), height (h), and the reliability (conf ) Shall be used. In this case, a certain nth (n is a positive integer) object region candidate (Rectangle) can be expressed by the function of Equation 1.
_{_{Rectangle n = [x n, y}} n, w n, h n, conf n] ... ( Equation 1)
In this example, the object region is an example of a rectangle surrounded so as to be in contact with the entire outer edge (outer periphery) of the detection target object (hereinafter, this state is also referred to as circumscribing), but this is not limited to a rectangle. Yen may be used. Further, the object region may be enclosed so as to be in contact with the entire inner periphery of the detection target object, instead of circumscribing the object region. The center of gravity of the object may be used as the center coordinate. Further, a variable representing rotation may be added and used.

FIG. 3 shows an example in which a plurality of object region candidates obtained by inputting a plurality of variables (n) to the function of Expression (1) are represented on a heat map. 3 is a rectangular area candidate surrounded by the thick line frame so as to be in contact with the entire detection target object (tumor) (hereinafter, this state is also referred to as circumscribing), and other rectangles indicate a part of the detection target object. This is a rectangular area candidate shown. The rectangular area candidates are set so as to include the detection target object as much as possible in the area and not to include anything other than the detection target object as much as possible.

In step S102, the generated plurality of object region candidates are received, and the heat map generation unit 12 generates a heat map based on the plurality of object region candidates. A specific operation example of this step will be described below.

Each of n input object region candidates, that is, Rectangle ₁ = [x ₁ , y ₁ , w ₁ , h ₁ , conf ₁ ], Rectangle ₂ = [x ₂ , y ₂ , w ₂ , h ₂ , conf _{_{2], ..., Rectangle n =}} [x n, y n, w n, h n, conf n] is confidence conf _1, conf ₂ indicating the likelihood that an object exists in the area candidate, ..., conf _n Is provided. This reliability is associated with the value of each pixel of the heat map in the object region candidate in the image. That is, the value of each pixel of the heat map is set as conf _n = heatmap _{score (i, j)} . Here, (i, j) represents the coordinate position of the pixel in the heat map. Thereby, a heat map having reliability in each pixel constituting the image is generated.

FIG. 4 shows an example of a heat map generated by the heat map generator 12. In this example, the value of the object region candidate is set using the reliability conf _n value of each of the plurality of pixels included in each object region candidate of the heat map. For example, the value of the reliability conf _n of a certain object region candidate is set as the value of a pixel existing in the object region candidate. When there are a plurality of object region candidates, for example, a portion where the object region candidates A and B partially overlap (for example, this portion is referred to as C), the reliability value of the portion C is the object region candidate A And a value obtained by adding the reliability of B.

When the heat map is set so that the higher the pixel value is, the whiter the color is, the value of the pixel of the object region candidate on the lower left side of the image in which many object region candidates are confirmed in FIG. As a result, the image becomes whitish as shown in FIG.

Finally, in step S103, the object area output unit 13 that has acquired the generated heat map selects an object area candidate that circumscribes the entire detection target object from among a plurality of object area candidates, and outputs the object area candidate as an object area.

The operation of the object area output unit 13 will be described in detail. The input heat map image (FIG. 4) has a value indicating the probability (reliability) that an object exists in each pixel. Here, it is determined that an object exists in a pixel whose value is equal to or greater than a predetermined threshold value, and no object exists in a pixel smaller than the threshold value. The threshold value may be set in advance by the designer, or may be set to be changeable by the operator. In this case, the heat map determined that the detection target object exists is as shown in FIG. It is assumed that the detection target object (tumor) exists as shown in FIG. At this time, the object area output unit 13 outputs a frame (frame 1 in FIGS. 5 and 6) surrounding all of the area where the object exists as a frame of the object area. The output image is displayed on an external monitor or the like and viewed by an operator such as a doctor. The output object region is output as a frame (rectangular frame) along the shape of the detection target object.

(Effects of the first embodiment)
The image processing apparatus 100 according to the first embodiment of the present invention can display a frame indicating the entire irregular object at high speed. As a result, it is possible to facilitate the discovery and determination of irregular objects by the operator. This is because the image processing apparatus 100 outputs each object region using a heat map based on the reliability of each object region candidate, and thus correctly outputs the object region circumscribing the entire detection target object. Further, since the object region is output by comparing the pixel value in the heat map generated from the reliability of the object region candidate generated by the object region candidate generating unit 11 with a preset threshold value, the neural network calculation is performed. This is because the object region is output at high speed with a much smaller calculation amount than the amount.

<Second Embodiment>
In the first embodiment of the present invention, the image processing apparatus 100 outputs the object region of the detection target object based on the heat map, but the output of the object region includes a heat map and a corresponding object region candidate. May be used simultaneously. Further, in the process in which the object region candidate generation unit 11 of the first embodiment generates object region candidates, if the number of the object region candidates is too large, it takes a lot of processing time to calculate the combination of the object region candidates. Was. Therefore, in the second embodiment of the present invention, a method of outputting the object region of the detection target object by using the heat map and the object region candidate will be described by simplifying the processing to be generated.

(Image processing device)
As shown in FIG. 7, the image processing apparatus 200 according to the second embodiment of the present invention includes an object region candidate generation unit 21, a heat map generation unit 22, and an object region output unit 23.

The object region output unit 23 uses the object region candidate generated by the object region candidate generation unit 21 and the heat map generated by the heat map generation unit 22 corresponding to the object region candidate (heat map on which the generated object region candidates are superimposed). Then, an object region candidate circumscribing the entire detection target object is selected from among a plurality of object region candidates and output as an object region.

The object region candidate generation unit 21 generates an object region candidate. However, the object region candidate generation unit 21 is different from the first embodiment (object region candidate generation unit 11) when the number of object region candidates is too large. The combination of is not generated. The generated object area candidate outputs the object area candidate to the heat map generation unit 22 and the object area output unit 23.

The heat map generator 22 is the same as that of the first embodiment (heat map generator 12).

(Operation of image processing device)
The specific operation of the image processing apparatus 200 will be described with reference to the flowchart shown in FIG.

Steps S201 and S202 are the same as steps S101 and S102 in FIG. 2, respectively.

In step S203, the object region output unit 23 acquires the object region candidate from the object region candidate generation unit 21 and the heat map from the heat map generation unit 22, and outputs the object region based on these. A specific operation example of this step will be described below.

First, object region candidates (Rectangle _n = [x _n , y _n , w _n , h _n , conf _n ]) generated by the object region candidate generation unit 21, and a heat map ( 4) is input to the object region output unit 23. FIG. 9 shows a plurality of object region candidates (frames) drawn on the heat map (see FIG. 4). The object region output unit 23 outputs each object region candidate (Rectangle ₁ = [x ₁ , y ₁ , w ₁ , h ₁ , conf ₁ ], Rectangle ₂ = [x ₂ , y ₂ , w ₂ , h ₂ , conf _2]. ], ..., Rectangle _n = [x _n , y _n , w _n , h _n , conf _n ])), based on the value (reliability) of each pixel in the heat map corresponding to the pixel in the object area The score of the object region candidate is calculated using a function for tabulation. For example, when the score of the input object region candidate (Rectangle _n ) is the sum of the reliability of each pixel of the heat map corresponding to the object region, a score (Rectangle) _n ) = Σ _{(w, h)} heatmap _{score (i, j)} . Thereafter, the object region output unit 23 compares the score of the object region candidate (sum of reliability of all the pixels in the object region candidate) with a preset threshold value, and the object region candidate having a score equal to or higher than the threshold value. Is output as the object region. The object area output unit 23 extracts object area candidates whose object area candidate score is equal to or greater than a threshold value from the object area candidates, and outputs the area surrounded by the outer edges as the object area. For example, FIG. 10 shows an object region 2c composed of two extracted

object region candidates

2a and 2b. It should be noted that the

object region candidates

2a and 2b may be displayed as they are as long as they are numbers that can be easily visually recognized by the operator (for example, two in FIG. 10), without necessarily being combined into one. Further, the threshold value may be set in advance by the designer, or may be set to be changeable by the operator.

(Effect of 2nd Embodiment)
The image processing apparatus 200 according to the second embodiment of the present invention can display a frame indicating the entire irregular object at high speed. As a result, it is possible to facilitate the discovery and determination of irregular objects by the operator. This is because the image processing apparatus 200 outputs each object region using the heat map score based on the reliability of each object region candidate, and thus correctly outputs the object region circumscribing the entire detection target object. is there. Further, since the object region is output by comparing the pixel value in the heat map generated from the reliability of the object region candidate generated by the object region candidate generating unit 21 with a preset threshold value, the neural network calculation is performed. This is because the object region is output at high speed with a much smaller calculation amount than the amount.
In the second embodiment, the processing of the object region candidate generation unit 21 is simplified in comparison with the first embodiment, so that the entire processing time can be shortened. In the second embodiment, the heat map generation unit 22 needs to perform aggregation processing for calculating the score from the heat map, but the object region output unit 23 configures the object region from the object region candidates using the heat map. By selecting the object region candidate, the processing in the object region candidate generation unit 11 is simplified. For this reason, even when the number of object region candidates is large, it is possible to avoid the processing time of the object region candidate generating unit 11 from being prolonged, and as a result, the processing time as a whole can be shortened.

<Third Embodiment>
In the first and second embodiments, it is mainly assumed that the detection target object is stationary during detection, but the detection target object may move. For example, when the object to be detected is a tumor, the tumor itself may move in conjunction with the movement of the surrounding involuntary muscles without moving. In the real-time tumor detection system in an endoscope moving image that is an example of use of the present invention, the position of a tumor in the image moves as the camera moves in accordance with an endoscope operation by a doctor. Therefore, in the third embodiment, an image processing apparatus capable of outputting an object region at high speed even when a detection target object moves during detection will be described.

(Image processing device)
As shown in FIG. 11, the image processing apparatus 300 according to the third embodiment of the present invention includes an object region candidate generation unit 31, a heat map generation unit 32, and an object region output unit 33. In the present embodiment, it is assumed that the image processed by the image processing apparatus 300 is a moving image (a series of temporally continuous images).

The basic operation of the heat map generation unit 32 is the same as that of the second embodiment, except that the output of the heat map generation unit 32 is the same as that of the heat map generation unit 32 as represented by the dotted arrows in FIG. To be input. In this embodiment, in order to process a moving image, the heat map of the past image produced | generated in the heat map production | generation part 32 is utilized for the heat map production | generation of the present image. The heat map generation unit 32 uses the heat map generated at a past time and the object region candidate corresponding to the current time generated by the object region candidate generation unit 31 in the image showing a certain tumor, and the current time A heat map corresponding to is generated.

The object region candidate generation unit 31 and the object region output unit 33 are the same as the object region candidate generation unit 21 and the object region output unit 23 of the second embodiment.

(Operation of image processing device)
A specific operation of the image processing apparatus 300 will be described with reference to a flowchart shown in FIG.

Step S301 is substantially the same as step S201 in FIG. 8, but the object region candidate generation unit 31 inputs an image to the object region candidate generation unit 31 at predetermined intervals (for example, 1 second). Alternatively, an image may be input when there is a change in the moving image.

In step S302, when the object region candidate generated by the object region candidate generating unit 31 and the heat map of the past image are received, the heat map generating unit 32 generates a heat map based on these. A specific operation example of this step will be described below.

In the present embodiment, the heat map generation unit 32 uses the past heat map obtained by the object region candidate generation unit 31 and the heat map generation unit 32 for the input image at time t as Score (Rectangle _{t, n} ) = Let Σ _{t, i, j} heatmap _{score (t, i, j)} .

Here, for example, each object region candidate generated by the object region candidate generation unit 31 for the image at time t + 1 is displayed.
Rectangle _{t, n} = [x _{t, n} , y _{t, n} , w _{t, n} , h _{t, n} , conf _{t, n} ] (Formula 2)
And the heat map at time t
Score (Rectangle _{t, n} ) = Σ _{t, w, h} heatmap _{score (t, i, j)} (Equation 3)
When using the equation (2) and the equation (3), the heat map at time t + 1 is
Score (Rectangle _{t + 1, n} ) = Σt _{, w, h} heatmap _{score (t, i, j)} + Σt _{+ 1, i, j} heatmap _{score (t + 1, i, j)} … Formula (4)
Can be expressed as

That is, the object region candidate generation unit 31 generates a heat map at the current time based on the object region candidate for the image at the current time and the heat map generated from the image at the past time.

In this example, the case where the heat map of the image with the previous time is simply added is shown, but the present invention is not limited to this. For example, you may use the heat map of the time until the past time n (for example, second). Moreover, when adding, it is not necessary to add each heat map of the past time equivalently, You may give a weight to each time. That is,
Score (Rectangle _{t + 1, n} ) = w _t Σ _{t, w, h} heatmap _{score (t, i, j)} + w _{t + 1} Σ _{t + 1, i, j} heatmap _{score (t + 1, i, j )} (Formula 6)
It is good. The weight w _t at each time may be set as w _t = 1 / t, for example, or may be determined according to the heat map at each time t. For example, the similarity of the heat map image may be set to w _t . As an example of weighting, a similarity indicating a degree of coincidence between a heat map generated in the past and a heat map currently generated may be calculated and used as a weight. In this case, the degree of coincidence increases as there is no movement in the past and the present.

Step S303 is the same as step S203 in FIG.

(Effect of the third embodiment)
The image processing apparatus 300 according to the third embodiment of the present invention can display a frame indicating the entire irregular object at high speed even when the detection target object moves and is captured as a moving image. it can. As a result, it is possible to facilitate the discovery and determination of irregular objects by the operator. This is because the heat map generation unit 32 generates a heat map using the heat map generated by the heat map generation unit 32 at the past time in addition to the object region candidate at the current time, and the object region output unit This is because 33 uses this heat map to select an object area from object area candidates. Furthermore, by giving a weight to each time and performing weighting based on the similarity of the current time, a frame indicating the entire irregular object can be displayed at high speed stably with respect to the movement of the detection target object. . As a result, it is possible to facilitate the discovery and determination of irregular objects by the operator.

In each of the embodiments of the present invention described above, the

image processing apparatuses

100, 200, 300, and 400 may be integrated with an apparatus such as a camera that captures an image, or a camera and an information processing apparatus (such as a personal computer). May be connected to control the overall operation of the image processing on the information processing apparatus side. That is, an application program for image processing on the information processing apparatus side may be used under the following configuration.

(Information processing device)
In each of the embodiments of the present invention described above, some or all of the components in the image processing apparatus shown in FIGS. 1, 7, 11 and the like are arbitrary information processing apparatuses 500 and programs as shown in FIG. It can also be realized using a combination. The information processing apparatus 500 includes the following configuration as an example.

CPU (Central Processing Unit) 501
ROM (Read Only Memory) 502
-RAM (Random Access Memory) 503
A storage device 505 for storing the program 504 and other data
A drive device 507 for reading / writing the recording medium 506
Communication interface 508 connected to the communication network 509
An input / output interface 510 for inputting / outputting data
-Bus 511 connecting each component
Each component of the image processing apparatus in each embodiment of the present application is realized by the CPU 501 acquiring and executing a program 504 that realizes these functions. A program 504 that realizes the functions of each component of the image processing apparatus (for example, the object region candidate generation unit 11, the heat map generation unit 12, and the object region output unit 13) is stored in advance in the storage device 505 or the RAM 503, for example. The CPU 501 reads the data as necessary. Note that the program 504 may be supplied to the CPU 501 via the communication network 509 or may be stored in the recording medium 506 in advance, and the drive device 507 may read the program and supply it to the CPU 501.

There are various modifications to the method of realizing each device. For example, the image processing apparatus may be realized by an arbitrary combination of a separate information processing apparatus and a program for each component. A plurality of components included in the image processing apparatus may be realized by an arbitrary combination of one information processing apparatus 500 and a program.

Also, some or all of the constituent elements of the image processing apparatus are realized by other general-purpose or dedicated circuits, processors, or combinations thereof. These may be configured by a single chip or may be configured by a plurality of chips connected via a bus.

Some or all of the components of the image processing apparatus may be realized by a combination of the above-described circuit and the like and a program.

When some or all of the components of the image processing apparatus are realized by a plurality of information processing apparatuses and circuits, the plurality of information processing apparatuses and circuits may be centrally arranged or distributedly arranged. May be. For example, the information processing apparatus, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client and server system and a cloud computing system.

As mentioned above, although this invention was demonstrated with reference to this embodiment, this invention is not limited to the said embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

The present invention can be applied to applications such as an image processing apparatus that recognizes a non-rigid object having an irregular and unpredictable shape included in an image, and a program for realizing the apparatus on a computer.

100, 200, 300

Image processing device

11, 21, 31 Object region

candidate generation unit

12, 22, 32 Heat

map generation unit

13, 23, 33 Object region output unit 500 Information processing device 501 CPU
503 RAM
504 Program 505 Storage device 506 Recording medium 507 Drive device 508 Communication interface 509 Communication network 510 Input / output interface 511 Bus

Claims

Object region candidate generation means for generating a plurality of object region candidates indicating positions where at least some of the detection target objects exist in the image;
Heat map generating means for generating a heat map corresponding to the position in the image using the generated object region candidate;
An image processing apparatus comprising: an object region output unit configured to output an object region that surrounds the entire detection target object, the object region including one or more object region candidates, using the generated heat map.
The image processing apparatus according to claim 1, wherein the heat map generation unit generates the heat map using a reliability that represents a probability that each object region candidate includes the detection target object.
The object area output means uses, as the object area, an object area candidate including a set of sections whose reliability is equal to or higher than a predetermined threshold for each of a plurality of sections into which the heat map is divided. The image processing apparatus according to claim 1, which outputs the image processing apparatus.
The image processing apparatus according to claim 1, wherein the object area output unit outputs the object area using the heat map on which the generated object area candidates are superimposed.
The heat map generation means uses a heat map generated at a past time and an object region candidate corresponding to the current time generated by the object region candidate generation means in an image showing a certain detection target object, The image processing apparatus according to claim 1, wherein a heat map corresponding to the current time is generated.
The image processing apparatus according to claim 5, wherein the heat map generation unit generates a heat map corresponding to the current time using a weighted heat map corresponding to the past time.
The image processing apparatus according to claim 6, wherein a similarity indicating a degree of coincidence between a heat map corresponding to the past time and a heat map corresponding to the current time is used as the weighting.
The image processing apparatus according to claim 2, wherein the detection target object is a lesion reflected in an endoscopic image.
Generating a plurality of object region candidates indicating positions where at least some of the detection target objects exist in the image;
Using the generated object region candidate, generate a heat map corresponding to the position in the image,
An image processing method for outputting an object region that surrounds the entire detection target object, which is composed of one or more object region candidates, using the generated heat map.
Generating a plurality of object region candidates indicating positions where at least some of the detection target objects exist in the image;
Using the generated object region candidate, generate a heat map corresponding to the position in the image,
A recording medium for storing an image processing program for causing a computer to output an object region surrounding the entire detection target object, which is composed of one or more object region candidates, using the generated heat map .