JP2006013722A - Unit and method for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing unit capable of adaptively changing a region extracted according to the request of a user from an image, and extracting a notable region matching the request of the user. <P>SOLUTION: The image processing unit comprises an image input section 11 for acquiring an image; an instruction input section 12 on an attribute of which the region should be extracted from a user as conditions of the extraction of a notable region; and a region specifying section 13 for specifying the region extracted from the image based on an instruction that the instruction input section 12 has acquired. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method for setting an area for an image, and more particularly to a technique for adaptively controlling the position and shape of an area, a display method, and the like in accordance with a user or system instruction. .

  Generally, extracting an area that is likely to attract human attention (hereinafter referred to as “attention area”), such as an area where some subject exists in an image, or an area that is significantly different in color from other parts. It is effective in many image processing such as subject recognition processing and effective image display.

Conventionally, as a technique for extracting a region of interest from such an image, for example, there is a technique disclosed in Patent Document 1.
The technique disclosed in Patent Document 1 applies two types of smoothing filters having different numbers of pixels used for smoothing to an image, obtains a difference between the generated images, and obtains a portion where the pixel value is positive. Extraction is performed as pixels belonging to the region of interest.
JP 2002-183713 A

However, the method disclosed in Patent Document 1 can be applied only to grayscale images, and cannot be applied to general color images.
In addition, the conventional region extraction method obtains a specific feature amount from an image and determines a region to be extracted based on the feature amount. Therefore, there is a problem that it is rigid with respect to the extraction of the region and cannot adaptively respond to the demands of the user and the system.

  Therefore, the present invention has been made in view of the above-mentioned reasons, and its purpose is to apply a method for determining the degree of attention to each part of an image and a method for determining an extraction region according to a user request. It is an object of the present invention to provide an image processing apparatus and an image processing method that can change the target area and extract a region of interest that meets the user's request.

  In order to solve the above-described problems and achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus that extracts a predetermined region included in an image, and the extraction condition includes: An instruction acquisition unit that acquires instruction information about an attribute, and an area specifying unit that specifies an area to be extracted from the image based on the instruction information acquired by the instruction acquisition unit.

  As a result, an instruction relating to the attribute of the image is acquired from the user or system side as an attention area extraction condition, and the area to be the attention area is specified from the image based on the instruction. The matched area can be extracted.

Here, the instruction acquisition unit analyzes the acquired instruction information, and the region specifying unit changes a region to be extracted according to the analysis result of the instruction information.
According to this configuration, it is possible to realize an image processing apparatus that can adaptively change a region to be extracted according to an extraction condition instructed by a user.

  In addition, the region specifying unit further includes an attention level determination unit that calculates a degree of attention that is a degree of attention for a region included in the image, according to the attention level calculated by the attention level determination unit. Preferably, the region to be extracted is changed, and the attention level determination unit further includes, for the region included in the image, an image feature amount extraction unit that extracts a predetermined image feature amount that is a characteristic of the region. It is more preferable to include an attention level calculation unit that calculates the attention level based on the image feature amount.

  Accordingly, the attention level is calculated from the image feature amount, and the region with the high attention level is extracted from the image as the attention region. Therefore, it is possible to appropriately respond to an ambiguous request from the user.

  The present invention can be realized not only as such an image processing apparatus, but also as an image processing method including steps characteristic of the image processing apparatus. May be realized as a program for realizing the above on a computer such as a personal computer. It goes without saying that such a program can be distributed through a recording medium such as a CD-ROM or a communication medium represented by the Internet.

  As described above, according to the image processing apparatus according to the present invention, an area to be extracted is searched based on an instruction from the user or the system side. Can be acquired automatically.

In addition, since the image is displayed in consideration of the attention area, a more impressive effect can be given to the user than when the image is displayed as it is.
Furthermore, by inputting information about the subject as instruction information relating to the extraction of the attention area, a nearby area where the subject exists can be extracted as the attention area. As a result, when the subject is recognized and displayed, the region to be searched in the image can be narrowed down, and improvement in both accuracy and speed of extracting the subject can be expected.

The best mode for carrying out the present invention will be described below with reference to the drawings.
(Embodiment)
First, an image processing apparatus according to an embodiment of the present invention will be described.

FIG. 1 is a functional block diagram of an image processing apparatus 10 according to the present embodiment.
The image processing apparatus 10 is an apparatus that captures an image in the apparatus and extracts a region of interest included in the captured image in accordance with a request of a user or a system. A specifying unit 13, an image conversion unit 17, a database 18, and an output unit 20 are provided.

  The image input unit 11 is an input interface for taking an image for extracting a region of interest into the apparatus, and is realized by an interface such as USB (Universal Serial Bus) or IEEE (Institute of Electrical and Electronic Engineers) 1394.

  Here, the image captured in the apparatus is not particularly limited, and is an image captured by an imaging apparatus such as a digital still camera, an image obtained from an image reading apparatus such as a scanner, or a computer graphic (CG) image. Any of the image data recorded on the recording medium may be used. Examples of recording media include magnetic media such as a flexible disk, hard disk, and magnetic tape, optical media such as CD-ROM / R / RW, DVD-ROM / RAM, SD memory card (registered trademark), and compact flash ( Examples thereof include semiconductor memory media represented by a registered memory card.

There is no particular limitation on the subject of the image captured in the apparatus. For example, any of a face image with a human face as a subject and a landscape image with a landscape as a subject may be used.
The instruction input unit 12 acquires instruction information, which is an extraction condition indicating what region is desired to be extracted from the captured image, from the user via an input interface such as a keyboard or a mouse. Then, the instruction input unit 12 controls operations of an area specifying unit 13, an image converting unit 17, and an attention area determining unit 19, which will be described later, based on the acquired instruction information.

  Here, the instruction information acquired from the user includes information on the subject such as “parts in which a person's face is reflected” and “parts in front”, for example, “reddish part” and “flashy part”. The information regarding the attribute of the image such as the information regarding the impression that the image has. In addition, the instruction information acquired from the user includes information regarding the display method of the region such as the number, size, and shape of the region to be extracted, or, for example, “Travel photo to thumbnail (reduced version image for list display)” It may also be information relating to processing that the user wants to perform, such as “Creating only a part in which a person is shown” or “Creating only a part where a person is shown”.

  Note that the instruction information acquired from the user is not limited to these examples, and arbitrary information can be used. In addition, the instruction information acquired from the user may be only one type or a plurality of information.

  Here, the instruction input unit 12 may omit an operation of acquiring instruction information from the user via an input interface such as a keyboard or a mouse. In this case, the instruction input unit 12 automatically determines a region of interest by a predetermined method, and controls operations of the region specifying unit 13, the image conversion unit 17, and the region of interest determination unit 19 based on the determination. To do.

  The area specifying unit 13 specifies an area in the image based on a predetermined calculation formula in accordance with instruction information output from the instruction input unit 12 from images (input images) captured by the image input unit 11 in the apparatus. And an attention level determination unit 14 and an attention area determination unit 19. In addition, the area | region which the area | region specific part 13 extracts may be one with respect to one image, or multiple.

  The attention level determination unit 14 is a processing unit that calculates a degree of attention that indicates how much attention is attracted to which part in the image according to the instruction information output from the instruction input unit 12 from the input image. The image feature quantity extraction unit 15 and the attention level calculation unit 16 are provided. Note that the unit for the attention level determination unit 14 to calculate the attention level may be for each pixel of the input image, or for each block in which a plurality of pixels are combined.

The image feature quantity extraction unit 15 obtains a feature quantity of an image that serves as an index of attention to the input image.
Here, as examples of the feature amount obtained by the image feature amount extraction unit 15, the strength of the edge component of each pixel, the coefficient of discrete cosine transform (DCT) or DCT-like transform for each block, the coefficient of wavelet transform, the image Color histogram, distance between the color component of each pixel and the average color component of the entire image or a small area of the image, entropy of pixel values in the small area of the image, distance in the depth direction of each pixel, position of the subject in the image And any feature quantity that can be acquired depending on the type.

  Note that the feature quantity to be obtained may be one type or plural, and the image feature quantity extraction unit 15 determines the type of feature quantity to be obtained and the feature quantity according to the instruction information output from the instruction input unit 12. The parameter for obtaining may be determined.

  For example, in an image in which red apples and orange mandarin oranges are shown as shown in FIG. 2A, the reciprocal of the distance between the designated color and each pixel in the color space is obtained as the attention level. When the user designates red in the instruction input unit 12, since a pixel in the vicinity where an apple is reflected is close in color space (approximates a designated color), a high degree of attention is assigned to this area. As a result, a region as shown in FIG. 2B can be obtained. Similarly, when the user designates orange using the instruction input unit 12, the pixels near the mandarin orange are close in distance on the color space (approximate to the designated color). As a result, an area as shown in FIG. 2C can be obtained.

  The attention level calculation unit 16 determines the attention level based on the feature amount obtained by the image feature amount extraction unit 14. The calculation formula for determining the attention level by the attention level calculation unit 16 can be expressed as the following formula 1.

(Equation 1)
I = f (F1, F2,..., Fn, P1, P2,..., Pm)

  This equation 1 means that the attention degree I can be described by a function having n feature values F1, F2,..., Fn and m parameters P1, P2,. By changing the parameter, it is possible to assign different attention values to the same feature value.

  For example, by giving the position of a specific point as a parameter and attenuating the attention as the distance from the point increases, a relatively high attention value can be assigned around this point. In this way, it is possible to extract a more appropriate region of interest for an image having a rough composition such as a portrait photograph.

  In addition, by giving a predetermined category of an image as a parameter, it is possible to select an effective feature amount in the category or other parameters. For example, in a portrait photograph, by setting the vicinity of the face as the center point of attenuation of attention, a higher attention value can be assigned to the vicinity of the face.

  In addition, the above formula 1 can omit parameters like the following formula 2. In this case, the attention level is uniquely determined for the same feature value.

(Equation 2)
I = f (F1, F2,... Fn)

  Note that the attention level calculation unit 16 may determine a feature amount used for calculation of the attention level from among the feature amounts obtained by the image feature amount extraction unit 15 in accordance with the instruction information output from the instruction input unit 12. The value of may be determined.

  Here, the attention level determination unit 14 may be configured by omitting the attention level calculation unit 16. In this case, the attention level determination unit 14 outputs the feature value value obtained by the image feature amount extraction unit 15 as the attention level to the attention area determination unit 19.

  The attention area determination section 19 is a processing section that determines which part of the input image is displayed in what form, that is, the attention area, based on the attention degree obtained by the attention degree determination section 14. A method of determining a region of interest in the region of interest determination unit 19 will be described below with reference to FIGS.

  For example, with respect to an input image as shown in FIG. 3A, a rectangular area including all pixels having a degree of attention exceeding a predetermined threshold Th can be considered as the attention area. That is, as shown in FIG. 3B, when there are five pixels exceeding the threshold Th (circles whose inside is indicated by diagonal lines), an attention area as shown in FIG. 3C is obtained. become.

  Here, the shape of the region of interest is a rectangle, but it may be any other shape such as a circle or an ellipse. The threshold value Th can be changed as a parameter, whereby the shape of the attention area obtained by the attention area determination unit 18 can be changed.

  In addition, as a method of determining a region of interest, regarding an input image as shown in FIG. 4A, for example, an area having a certain size such as a quarter of the entire input image is considered, and pixels inside the region are determined. It is also conceivable that the position of the attention area is the position of the attention area. That is, as shown in FIG. 4B, a region having a predetermined size is set, the region is moved in the direction of the arrow indicated by a broken line, and the sum of attention is sequentially calculated. Then, the position where the sum of the attention degrees is maximized is determined as the attention area. An example of a region of interest obtained by this determination method is shown in FIG.

  Further, as a method of determining a region of interest, for example, an input image as shown in FIG. 5A may be selected, for example, by selecting a region within a certain range from each pixel having a high degree of attention, and the union of the regions as a region of interest Conceivable. That is, as shown in FIG. 5B, when there are five pixels of high interest (circles indicated by diagonal lines), the sum of regions within a predetermined range centered on each pixel. If the set is an attention area, an attention area as shown in FIG. 5C can be obtained.

As another method for determining the attention area, an area including the pixels having the attention degree of the top n is considered as the attention area.
Next, a method of displaying the attention area in the attention area determination unit 19 will be described below.

  For example, as a method of displaying the attention area, a method of displaying only the inside of the attention area as shown in FIG. By adopting such a display method, it is possible to display the region of interest of the image more emphasized.

  In addition, as another method of displaying the attention area, a method of performing display according to the value of the attention level of the pixel, such as displaying a brighter part of the attention area, or smoothing only inside or outside the attention area Image processing such as image processing and color conversion, displaying images by performing different image processing inside and outside the attention area, zooming in on the attention area, attention area A method of cutting out and combining with another image, a method of giving a visual effect to the image, or a combination of these methods can be considered.

Here, the attention area determination unit 19 may determine a determination method and a display method of the attention area according to the instruction information output from the instruction input unit 12.
For example, as shown in FIG. 6A, as a result of calculating the attention level with respect to the input image, the points A to E shown in FIG. Assume that (A) = 120, I (B) = 105, I (C) = 95, I (D) = 70, and I (E) = 55. Further, it is assumed that the method of determining the attention area is “selecting a rectangular area that includes a point whose attention degree is equal to or higher than the threshold Th”. Here, when the user sets the threshold value Th to 100 with the instruction input unit 12, a rectangular area including two points A and B is obtained as the attention area as shown in FIG. 6B. Similarly, when the threshold Th is set to 50, a rectangular area including all of A to E as shown in FIG. 6C is obtained as the attention area.

The description of the configuration of the image processing apparatus 10 will be continued again.
The image conversion unit 17 is a processing unit that performs a predetermined conversion process on the input image. This image conversion unit 17 performs predetermined conversion processing on the input image in order to perform the processing in the attention level determination unit 14 with high accuracy or efficiency, and converts the converted image, which is the converted image, into a region. Output to the identification unit 13.

  Here, conversion methods performed on the input image include enlargement or reduction, trimming, rotation, color conversion, color correction, luminance correction, smoothing / sharpness processing, conversion processing such as contour extraction, and combinations thereof. is there.

  Note that the result of conversion by the image conversion unit 17 may be overwritten on the input image, or may be stored as a converted image separately from the input image. Alternatively, a plurality of different conversion processes may be performed on one input image to obtain a plurality of different converted images. By extracting feature quantities from a plurality of converted images in this way, more information can be obtained from a single input image.

  For example, consider a case where a converted image is generated by reducing an input image at a plurality of resolutions, and the edge strength of each converted image is obtained as a feature amount. In this case, it is considered that the edge strength when viewed more globally can be obtained from the converted image having a smaller reduction ratio. In this way, a plurality of feature amounts having different meanings can be obtained even when a single feature amount extraction unit is used. As a result, more information can be used for setting the attention area.

  Note that the feature values extracted from the respective converted images may be the same or different. For example, by using a conversion image obtained by performing contour extraction on an input image and a conversion image obtained by performing conversion that emphasizes the color of the subject of interest, information on the shape and color of the subject can be obtained. A region including the subject can be set as a region of interest with higher accuracy than when each feature amount is used alone.

  The database 18 is a database for storing information used for processing for specifying a region of interest, such as information on the type of subject and information for obtaining image feature amounts, and is realized by a storage device such as a hard disk. Is done.

  Since the image processing apparatus 10 includes the database 18, for example, the image feature amount extraction unit 15 identifies a subject in the image based on information on the type of subject recorded in the database 18 and obtains a feature amount. Will be able to. Alternatively, information for obtaining other image feature amounts may be recorded in the database 18 and the image feature amounts may be obtained based on the information.

  Further, the image processing apparatus 10 stores, in the database 18, the type of feature amount obtained by the image feature amount extraction unit 15 and the parameter of the attention degree calculation unit 15 for the instruction information output from the instruction input unit 12. Information for controlling the operation of the attention level determination unit 14 may be recorded, or information for controlling the operation of the attention area determination unit 19 may be recorded. Furthermore, the attention level obtained by the attention level determination unit 14 and the attention area information obtained by the attention area determination unit 19 may be recorded in the database 18.

  The output unit 20 is an interface that outputs an image to a device for presentation to the user in accordance with the attention area obtained by the attention area determination unit 19 and the display method thereof. Here, the output destination device may be a display device such as a CRT or LCD, or a printing device such as a printer.

The processing operation of the image processing apparatus 10 configured as described above will be described below.
FIG. 7 is a flowchart showing a flow of processing for the image processing apparatus 10 to adaptively select a region in response to a user request.

  First, the image processing apparatus 10 captures an image for extracting a region of interest into the apparatus via the image input unit 11 (step S100), and acquires instruction information from the user for the input image via the instruction input unit 12. (Step S101). Here, a case will be described as an example where instruction information “acquire region of interest to create a thumbnail from a photo taken when traveling” is acquired. Note that the type of instruction information is not limited to the above example, and general requests for processing that the user wants to perform can be handled as instruction information.

  Next, the image processing apparatus 10 analyzes the input instruction information in the instruction input unit 12, and determines what region is to be extracted specifically (step S102). For example, in the above example, it is considered that “a region that is approximately ½ the size of the original image and that is likely to include a person” is determined as the target region to be extracted. Note that the correspondence between the instruction information listed here and the area actually extracted is an example, and different target areas may be set for the same type of instruction information.

  Here, instead of analyzing the instruction information in step S102, it is directly selected in step S101, such as “an area that is about half the size of the original image and is likely to contain a person”. Information regarding whether to extract such a region may be acquired from the user. In this case, step S102 can be omitted.

Subsequently, the image processing apparatus 10 sets actual contents and parameters of various processes to be described later according to the information on the target area (step S103). Examples of the parameter setting method include the following methods (1) to (3).
(1) Set a predetermined initial value (2) Express as a function of a parameter that sets the degree of achievement of the target, and set the parameter value that maximizes the degree of achievement (3) Similar from the database Refer to the extraction result for the image and set the parameters at that time. Note that the parameter setting method is not limited to the above example, and other methods may be used.

Then, the image processing apparatus 10 performs a process for determining a region of interest according to the result set in step S103.
First, the image processing apparatus 10 performs image conversion processing as preprocessing in the image conversion unit 17 (step S104). Since the contents of the image conversion are described in the description regarding the image conversion unit 17 described above, the description is omitted here.

  Next, the image processing apparatus 10 obtains the attention level for the converted image in the attention level determination unit 14 (step S105). The details of the processing here are described in the above description regarding the attention level determination unit 14, and are omitted here.

  Thereafter, the image processing apparatus 10 causes the attention area determination unit 19 to extract attention area candidates that are considered to be attention areas based on the obtained attention level (step S106). Details of the processing here are described in the above description of the attention area determination unit 19, and are omitted here.

  Here, the image processing apparatus 10 determines whether the region of interest candidate obtained as described above satisfies the target region condition set in step S102 in the region specifying unit 13 (step S107).

Examples of the determination method in step S107 include the following methods. In the above example, the target area is “an area that is approximately half the size of the original image and that is likely to have a person in it”, so the following two points may be determined.
(1) Size of attention area candidate (2) Ratio of pixels determined to be skin color among pixels in attention area candidate Note that the determination items for the target area shown here are examples, and determination is based on other criteria May be performed.

In addition, as a method for resetting parameters, the following methods can be cited.
(1) Change the parameter in one direction to increase or decrease in order (2) Express the achievement level of the target as a function of the parameter, and change the parameter in the direction to increase the achievement level in the way of optimization calculation Similarly, the parameter resetting method is not limited to this example, and other methods can be used.

  As a result of the determination in step S107, if the condition is satisfied (Yes in step S107), the image processing apparatus 10 determines the attention area candidate extracted by the attention area determination unit 19 as the attention area (step S108), and performs the processing operation. finish.

  On the other hand, when the condition is not satisfied (No in step S107), the image processing apparatus 10 returns to step S103 again and resets the parameters. Here, the image processing apparatus 10 may return to step S101 instead of step S103 and acquire instruction information from the user again according to the number of parameter resets.

Next, an application example of the image input apparatus 10 according to the embodiment of the present invention will be described.
FIG. 8 is a block diagram showing a typical configuration of the image processing apparatus 10 according to the present embodiment, and FIG. 9 is a flowchart showing a processing operation of the image processing apparatus 10 having the above configuration.

  First, the image processing apparatus 10 takes in an image for extracting a region of interest into the apparatus via the image input unit 11 (step S200). At this time, the image processing apparatus 10 initializes the converted image and the counter k that specifies the feature amount extracted from the converted image to “1” (step S201).

  Next, the image processing apparatus 10 acquires and analyzes instruction information from the user for the input image via the instruction input unit 12 (step S202). The instruction information includes information relating to selection of an image feature amount to be extracted or attention level, and information relating to a method of determining and displaying an attention area. For example, as the former information, “parts where a person's face is reflected”, “parts where a round object is reflected”, “reddish part”, “bluish part”, “greenish part”, “bright part” , “Dark part”, “flashy part”, “calm part”, etc. are selected as the latter information “pixels with a degree of attention of 25% or more of the maximum value belong to the attention area”, “rectangular area” ”,“ Display a portion with a high degree of attention brightly ”, and the like.

  The information acquired from the user is information related to the processing that the user wants to perform, such as “create thumbnails (reduced version images for list display) from travel photos” and “cut out only the part where people are shown”. However, the present invention is not limited to these examples, and arbitrary information can be used. Information acquired from the user may be only one type, and a plurality of pieces of information may be acquired. Also good.

Here, examples of user interface screens that the image processing apparatus 10 displays and outputs in order to acquire instruction information from the user are shown in FIGS.
FIG. 10 is a diagram illustrating a first example of a user interface screen.

  The image display device 10 outputs a display screen 210 as shown in FIG. 10 to the display device, and accepts designation of the region extraction mode from the user. Here, the region extraction mode means a method of extracting a region of interest. In the case of the ID photo mode in the figure, attention is high in the vicinity of a person's face without obtaining detailed instruction information from the user. It will be extracted as a region. Here, a state is shown in which a manual mode for acquiring detailed instruction information from the user is selected. Therefore, when the user performs an operation of pressing the next page button 211, the image display device 10 is displayed in FIG. Switch the display to the screen as shown.

FIG. 11 is a diagram illustrating a second example of the user interface screen.
The image display apparatus 10 outputs a display screen 220 as shown in FIG. 11 to the display apparatus, and designates what the subject included in the image is to be focused on, that is, what is the attention area from the user. Accept. Here, a state in which an instruction “colorful place” is input from the user is shown, and when the user performs an operation of pressing the next page button 221, the image display apparatus 10 is displayed in FIG. Switch the display to the screen as shown.

FIG. 12 is a diagram illustrating a third example of the user interface screen.
The image display device 10 outputs a display screen 230 as shown in FIG. 12 to the display device, and accepts designation of the attention area extraction method from the user. Here, a state in which an instruction “to make it look good” is input from the user is shown. When the user performs an operation of pressing the next page button 231, the image display device 10 is displayed in FIG. Switch the display to the screen as shown.

FIG. 13 is a diagram illustrating a fourth example of the user interface screen.
The image display device 10 outputs a display screen 240 as shown in FIG. 13 to the display device, and receives designation from the user regarding the method of displaying the attention area. Here, a state in which an instruction “to be emphasized” is input from the user as text is shown. When the user performs an operation of pressing the region extraction button 241, the image display apparatus 10 analyzes the user instruction information and performs the process of step S203 shown in FIG.

It should be noted that the display examples of the user interface screen shown here are merely examples of display and are not intended to be limited to these.
Hereinafter, the description of the processing operation of the image processing apparatus 10 having the configuration shown in FIG. 8 will be continued with reference to FIG. 9 again.

The image processing apparatus 10 sets actual contents and parameters of various processes to be described later based on the result of analysis in step S202 (step S203). Examples of the parameter setting method include the following methods.
(1) Set a predetermined initial value (2) Express as a function of a parameter for setting the achievement level of the target, and set a parameter value that maximizes the achievement level (3) From the database 18 to the input image Refer to the extraction results for similar images and set the parameters at that time. The parameter setting method is not limited to the above example, and other methods may be used.

Subsequently, in the image processing apparatus 10, the image conversion unit 17 generates a converted image from the input image (step S204). As the image conversion, for example, the following conversion is performed.
(A) Of the RGB components of the input image, an image consisting only of the R component is generated. Similarly, an image consisting only of the G component and an image consisting only of the B component are generated.
(B) Generate an image consisting only of the luminance component of the input image. The luminance component Y is obtained by Equation 3 shown below.

(Equation 3)
Y = 0.299 * R + 0.587 * G + 0.114 * B

(C) The input image is divided into small areas such as 2 * 2, 4 * 4, and an image in which the pixel value of each pixel in the area is replaced with the average value of the pixel values of all the pixels in the area is generated.
(D) An image in which a smoothing filter is applied to the entire image is generated. Here, for example, a 3 * 3 average filter can be applied as the smoothing filter. In the 3 * 3 average value filter, when the pixel value of the target pixel is determined, a new pixel value of the target pixel is determined by performing weighted averaging of the pixel values in the vicinity of itself as shown in Table 1 below.

Examples of other generally known smoothing filters include a Gaussian filter and a median filter.
(E) The resolution of the input image is converted.

  In addition to the conversion methods (a) to (e) described above, the image conversion unit 17 may use an existing image processing method for image conversion. According to the instruction information input in step S102, the image conversion unit 17 Conversion may be performed.

  The image conversion unit 17 repeats these processes while incrementing the counter k until the value of the counter k reaches a predetermined number n of converted images (step S205 and step S206).

Subsequently, the image processing apparatus 10 initializes the value of the counter k to “1” again in the image feature amount extraction unit 15 (step S207), and obtains the image feature amount from each of the generated n converted images (step S207). S208). Here, the image feature quantity extraction unit 15 obtains, for example, the following as feature quantities.
(A) An edge component of an image is obtained for an image composed only of the R component of the input image. Edge components are obtained in the same manner for images consisting only of the G component and images consisting only of the B component. As a method for obtaining the edge component, for example, a 3 * 3 Laplacian filter can be applied. In the 3 * 3 Laplacian filter, when determining the size of the edge component of the target pixel, the neighboring pixels are weighted and added as shown in Table 2a or Table 2b below, thereby adding the edge of the target pixel. Determine the ingredients.

Other commonly known edge extraction methods include a method of applying a filter such as a Sobel filter or a Prewitt filter, or a method of using a wavelet transform. Etc.
(B) An edge component is obtained for an image composed of luminance components of the input image in the same manner as in (a).
(C) The input image is divided into 8 * 8 blocks, and a discrete cosine transform (DCT) is performed on each block.
(D) The Euclidean distance between each pixel of the input image and the pixel value of a specific color is obtained.
(E) The input image is divided into several small areas, and a color histogram representing the frequency distribution of RGB values of pixels belonging to each area is obtained. In step S103, conversion to another color space such as an HSV color system or an HSI color system may be performed, and a color histogram for these pixel values may be obtained.
(F) The entire input image is matched with a template of a subject such as a face, for example, and a score indicating the degree of matching is obtained.

  Note that the image feature quantity extraction unit 15 may extract the feature quantity of the input image using an existing image feature quantity extraction method other than the image feature quantity extraction methods (a) to (f) described above. . Further, all of these feature amounts may be obtained for the input image, or only a necessary feature amount may be obtained based on the instruction information input in step S202.

  Similarly to step S204, the image feature quantity extraction unit 15 repeats these processes while incrementing the counter k until the value of the counter k reaches the number n of converted images (step S209 and step S210).

  Subsequently, the image processing apparatus 10 calculates the attention level in the attention level calculation unit 16 based on the image feature amount thus obtained and the instruction information acquired in Step S202 (Step S211).

  For example, when the instruction information is a “reddish part”, (1) the pixel value of each pixel and (R, G, B) = (255, 0, 0) from the above-described image feature amount. (2) The R component edge strength of the input image is used to assign a high degree of attention to a portion where the change in the red component is large and actually close to red. Can do.

  Similarly, when the instruction information is “flashy part”, the image feature amount is (1) the high frequency component of the DCT coefficient in the 8 * 8 block, (2) the edge strength of the luminance component of the input image, (3) By using the saturation value in the HSV color system of each pixel, it is possible to assign a high degree of attention to a portion where the color / brightness changes drastically and the color is bright.

Note that the correspondence relationship between the instruction information and the image feature amount given here is merely an example, and different image feature amounts can be used for similar instruction information.
Here, when the instruction information does not include information related to the calculation of the attention level, the attention level calculation unit 16 adds a value of each image feature amount to obtain the attention level, for example. The degree of attention is obtained according to the method.

  Thereafter, in the attention area determination unit 19, the image processing apparatus 10 extracts which area of the image is the attention area, that is, a attention area candidate, based on the attention degree obtained in step S211 (step S212).

For example, as a method for determining a region of interest candidate, the following method may be mentioned.
First, the maximum value of the degree of attention assigned to each pixel is set to SMax, and the threshold value Th is determined. However, 0 ≦ Th ≦ 1.

  Next, when the attention level S of each pixel is set, a pixel satisfying the relationship of the following formula 4 is determined as a pixel belonging to the attention area.

(Equation 4)
S ≧ SMax * Th

  Then, the entire rectangle including the entire pixels determined to belong to the attention area is extracted as the attention area candidate.

  FIGS. 3A to 3C are diagrams schematically showing the operation of this extraction method. 3A shows the original image, FIG. 3B shows an example of setting the attention area candidate, and FIG. 3C shows an example in which only the inside of the attention area candidate set is displayed. .

In addition, the attention area determination unit 19 can also extract attention area candidates as follows.
That is, the maximum value of the degree of attention assigned to each pixel is SMax, and the threshold Th is determined. However, 0 ≦ Th ≦ 1.

Next, when the attention degree S of each pixel is set, a pixel satisfying the relationship of the above mathematical formula 4 is determined as a pixel belonging to the attention area.
Then, a circle with a radius r is drawn around each pixel determined to be a pixel belonging to the attention area, and a pixel belonging to one of the circles is extracted as a attention area candidate.

  FIGS. 5A to 5C are diagrams schematically showing the operation of this extraction method. That is, FIG. 5A shows an original image, FIG. 5B shows an example of setting an attention area candidate, and FIG. 5C shows an example in which an attention area candidate to be finally selected is displayed. Yes.

  Note that the attention area determination unit 19 may change the extraction method of the attention area candidate and the image display method according to the instruction information input in step S202. For example, when the instruction information is “display a pixel with a higher degree of attention brighter”, the display is made such that the brightness is higher in the vicinity of the pixel with a higher degree of attention, thereby enabling a more impressive display.

  Thereafter, the image display device 10 determines in the attention area determination unit 19 whether the attention area candidate obtained in step S212 satisfies the user's request, that is, whether a desired area has been obtained ( S213).

When the desired area is obtained (Yes in step S213), the attention area determination unit 19 determines the attention area candidate as the attention area (S214), and ends the processing operation.
On the other hand, if the desired area is not obtained (No in step S213), the process returns to step S203 to set the parameters again.

Examples of the parameter resetting method include the following methods.
(1) Change the parameter in order to increase or decrease (2) Express the achievement level of the target as a function of the parameter, and change the parameter in the direction of increasing the achievement level in the way of optimization calculation The resetting method is not limited to the above example, and other methods can also be used.

  As described above, according to the image display device according to the present embodiment, an instruction relating to the attention area is obtained from the user or the system side and analyzed, and the area to be the attention area is specified from the image based on the instruction. Therefore, it is possible to extract a region of interest that matches the requirements of the user and the system.

In addition, since the user can adaptively change the method for determining the region to be extracted, it is possible to flexibly acquire the region of interest from the image.
Furthermore, since the attention level is calculated from the image feature amount to determine the attention area, it is possible to appropriately respond to an ambiguous request from the user.

The image processing apparatus according to the present invention has been described based on the embodiment, but the present invention is not limited to this embodiment.
For example, in the above embodiment, the attention area determination unit 19 determines the display method in addition to the shape of the attention area, but the output unit 20 may determine the display method of the attention area.

  Further, although the flowchart shown in FIG. 9 has been described as the processing operation of the image processing apparatus 10, the flowchart shown in FIG. 14 may be used. That is, in the flowchart of FIG. 9, after all the converted images are once generated from the input image, the feature amount is extracted for each generated converted image. However, as shown in the flowchart of FIG. The feature amount may be extracted from the converted image each time (Step S304 to Step S307).

Further, the attention level obtained in step S211 in FIG. 9 may be stored in the database 18 and used for another process.
Further, by omitting step S211 in FIG. 9 and the like, the attention area may be determined directly from the image feature value itself.

Also, instead of obtaining the attention level in step S211 in FIG.
Further, an image may be output to a display device such as a display or a printing device such as a printer via the output unit 20 in accordance with the attention area obtained in step S212 in FIG.

Furthermore, by omitting step S213 in FIG. 9 or the like, the attention area obtained by the initially set processing method and parameters may be directly handled as the final output.
It goes without saying that the image display apparatus according to the present invention can be variously modified or modified within the spirit of the present invention without departing from the scope thereof.

  The image processing apparatus and the image processing method according to the present invention are useful as an apparatus and a method for adaptively selecting a region in an image according to user and system requirements, including a digital still camera and the like. It is suitable for an image capturing device, a display device, and the like.

1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. It is the schematic diagram which showed that the value of the attention level was changed according to a user's instruction | indication, and, as a result, an attention area | region changes. It is a figure which shows typically the extraction method of the 1st attention area. It is a figure which shows typically the extraction method of the 2nd attention area. It is a figure which shows typically the extraction method of the 3rd attention area. It is the schematic diagram which changed a threshold value according to a user's instruction | indication, and showed that an attention area changed as a result. 4 is a flowchart showing a flow of processing of the image processing apparatus according to the present embodiment. It is a block diagram which shows the typical structure of the image processing apparatus which concerns on this Embodiment. It is a flowchart which shows the flow of a process of the image processing apparatus shown in FIG. It is a figure which shows the 1st example of a user interface screen. It is a figure which shows the 2nd example of a user interface screen. It is a figure which shows the 3rd example of a user interface screen. It is a figure which shows the 4th example of a user interface screen. It is a flowchart which shows the flow of another process of the image processing apparatus shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Image input part 12 Instruction input part 13 Area | region specific part 14 Attention level determination part 15 Image feature-value extraction part 16 Attention degree calculation part 17 Image conversion part 18 Database 19 Attention area determination part 20 Output part 210,220, 230, 240 Display screen 211, 221, 231 Next page button 241 Area extraction button

Claims (44)

An image processing apparatus for extracting a predetermined area included in an image,
As the extraction condition, instruction acquisition means for acquiring instruction information regarding the attribute of the image;
An image processing apparatus comprising: area specifying means for specifying an area to be extracted from the image based on the instruction information acquired by the instruction acquiring means. The instruction acquisition means analyzes the acquired instruction information,
The image processing apparatus according to claim 1, wherein the area specifying unit changes an area to be extracted according to an analysis result of the instruction information. The region specifying means further includes:
An attention level determination unit that calculates a degree of attention, which is a degree of attracting attention, for the region included in the image,
The image processing apparatus according to claim 1, wherein an area to be extracted is changed according to the attention level calculated by the attention level determination unit. The attention level determination unit further includes:
An image feature amount extraction unit that extracts a predetermined image feature amount that is a characteristic of the region included in the image;
The image processing apparatus according to claim 3, further comprising: an attention level calculation unit that calculates the attention level based on the image feature amount. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a size of an edge component in each pixel of the image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a frequency component of an image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a color distribution of a predetermined region included in the image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a distance in a color space between a color component of each pixel and an average color of the entire image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a distance between a color component of each pixel and an average color of a predetermined area included in the image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is an entropy of a pixel value in a predetermined region included in the image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a distance in a depth direction of each pixel. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a position of a subject included in the image. The image processing apparatus according to claim 4, wherein the predetermined image feature amount is a type of a subject included in the image. The predetermined image feature amount includes the size of the edge component in each pixel of the image, the frequency component of the image, the color distribution of a predetermined area included in the image, and the color space between the color component of each pixel and the average color of the entire image. Distance above, distance between the color component of each pixel and the average color of the predetermined area included in the image, entropy of the pixel value in the predetermined area included in the image, distance in the depth direction of each pixel, subject included in the image The image processing apparatus according to claim 4, wherein the image processing apparatus is a combination of any of the following positions and the types of subjects included in the image. The feature amount extraction unit includes:
The image processing apparatus according to any one of claims 4 to 14, wherein the type of image feature amount to be extracted is changed based on the instruction information acquired by the instruction acquisition unit. The attention level calculation unit
The value of the degree of attention is calculated for each value of the extracted image feature value by changing a parameter related to the value of the image feature value. The image processing apparatus described. The attention level calculation unit
The image processing apparatus according to claim 16, wherein the parameter is changed based on instruction information acquired by the instruction acquisition unit. The attention level calculation unit
The image processing apparatus according to claim 4, wherein the value of the image feature amount is the value of the degree of attention. The image processing apparatus further includes:
The image processing apparatus according to claim 3, further comprising an image conversion unit that performs predetermined conversion on the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is enlargement or reduction of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is trimming to cut out a part of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is rotation of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is a color conversion of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is color correction of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is brightness correction of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is smoothing or sharpness processing of the image. The image processing apparatus according to claim 19, wherein the predetermined conversion is extraction of a contour of a subject included in the image. The predetermined conversion includes enlargement or reduction of the image, trimming to cut out a part of the image, rotation of the image, color conversion of the image, color correction of the image, luminance correction of the image, smoothing of the image The image processing apparatus according to claim 19, wherein the image processing apparatus is a combination of sharpness processing and contour extraction of a subject included in the image. The image conversion means includes
The image is subjected to a plurality of different predetermined conversions to generate a converted image,
The attention level determination unit
The image processing apparatus according to any one of claims 19 to 28, wherein the attention level is determined for each of a plurality of different converted images. The image conversion unit
The image processing apparatus according to any one of claims 19 to 29, wherein a type of conversion applied to the image is changed based on instruction information acquired by the instruction acquisition unit. The region specifying means further includes:
The image processing apparatus according to any one of claims 3 to 30, further comprising an attention area determination unit that identifies the area to be extracted based on the attention level calculated by the attention level determination unit. The attention area determination unit
32. The image processing apparatus according to claim 31, wherein an area including all pixels having the calculated degree of attention exceeding a predetermined threshold is specified as the area to be extracted. The attention area determination unit
32. The predetermined number of pixels are selected in descending order from the pixel having the highest attention value, and an area including all the selected pixels is specified as the area to be extracted. Image processing apparatus. The attention area determination unit
An area having a predetermined size is set at an arbitrary position in the image, and an area at a position where the sum of the attention degrees of pixels in the area is maximized is specified as the area to be extracted. The image processing apparatus according to claim 31. The attention area determination unit
The image processing apparatus according to any one of claims 31 to 34, wherein a method for specifying the region to be extracted is changed based on the instruction information acquired by the instruction acquisition unit. The image processing apparatus further includes:
An area specifying database in which information for specifying the area to be extracted is recorded;
The image processing device according to any one of claims 31 to 35, wherein the attention area determination unit specifies the area to be extracted with reference to the information. The image processing apparatus further includes:
The image processing apparatus according to any one of claims 31 to 36, further comprising an extraction region database for storing information on the region to be extracted that is identified by the attention region determination unit. The image processing apparatus further includes:
A feature amount database in which information for extracting the image feature amount is recorded;
The image processing apparatus according to any one of claims 4 to 37, wherein the image feature amount extraction unit extracts the image feature amount with reference to the information. The image processing apparatus further includes:
An attention level calculation database in which information for calculating the attention level is recorded;
The image processing apparatus according to any one of claims 4 to 38, wherein the attention level determination unit calculates the attention level with reference to the information. The image processing apparatus further includes:
The image processing apparatus according to any one of claims 3 to 39, further comprising an attention level database for storing the attention level value calculated by the attention level determination unit. The image processing apparatus further includes:
The image processing apparatus according to any one of claims 1 to 40, further comprising: an output unit that outputs the image based on information on the region specified by the region specifying unit. An image processing method for extracting a predetermined area included in an image,
An instruction obtaining step for obtaining instruction information regarding the attribute of the image as the extraction condition;
An area specifying step for specifying an area to be extracted from the image based on the instruction information acquired in the instruction acquiring step. A program for an image processing apparatus that extracts a predetermined area included in an image,
An instruction obtaining step for obtaining instruction information regarding the attribute of the image as the extraction condition;
A program for causing a computer to execute an area specifying step for specifying an area to be extracted from the image based on the instruction information acquired in the instruction acquiring step. A computer-readable recording medium on which a program is recorded,
An instruction obtaining step for obtaining instruction information regarding the attribute of the image as the extraction condition;
A computer-readable recording medium in which a program for causing a computer to execute an area specifying step for specifying an area to be extracted from the image is recorded based on the instruction information acquired in the instruction acquiring step. .

Images