JP6949671B2 - Information processing device, image area selection method, computer program, and storage medium - Google Patents

Description

The present invention relates to an information processing device for selecting a predetermined area from an image.

The information processing device may perform a predetermined process on a selected area in the image. Various user interfaces have been proposed as a method for selecting a target area for this purpose. The most common method is to select a bounding box by clicking and dragging a point in the image with a pointing device such as a mouse. In addition to this, a slice tool that cuts out the outline by clicking the outline of the area to be cut out multiple times is also a commonly used method. In each of these methods, the user manually selects the area. In response to these manual area selections, a method of performing automatic / semi-automatic area selection has also been proposed.

Patent Document 1 discloses an image processing device that detects a person's crown and eyes, and automatically adjusts the size of a face region from the detection results to determine a trimming size. The processing of this image processing device is performed by using a bottom-up method specialized in face area selection. Patent Document 2 discloses an image extraction device that selects a region by region growing, which is typical as a bottom-up method. This image extraction device first divides a region having primary features such as background subtraction and optical flow. The image extraction device extracts an object region by connecting similar regions with secondary features such as color components, centering on a region selected from the regions divided by the primary features. Patent Document 3 proposes a graph-based method. In this method, the user roughly specifies the area inside the contour of the area to be selected, and the graph cut is repeated according to the feature distribution inside the specified area to calculate the object area.

On the other hand, the problem of cutting out a semantic area such as a person area, an automobile area, a road area, a building area, and an empty area from an image is being studied. Such a problem is called semantic segmentation, and is expected to be applied to image correction corresponding to the type of object, scene interpretation, and the like. In performing the semantic region division, it is already common to identify the category label for each position of the image in units of small areas (superpixels) instead of units of pixels. The small area is mainly cut out from the image as a small area having similar characteristics. Various methods have been proposed for cutting out small areas with similar characteristics. Non-Patent Document 1 is a typical example of such a method. The category label of the small area is identified by using the internal feature amount or the context feature amount around it. Usually, region identification is performed by training such a locally-based region classifier using various training images. The technique disclosed in Non-Patent Document 2 divides an image into small regions at a plurality of levels, and identifies the small regions at each level by a linear SVM (Support Vector Machine). The category label of each pixel of the image is estimated with the category likelihood at all levels in each pixel as the input of the linear SVM.

Japanese Unexamined Patent Publication No. 2002-152492 Japanese Unexamined Patent Publication No. 9-185720 U.S. Pat. No. 7,660,463

SLIC Superpixels, R.Achanta, A.Shaji, K.Smith, A.Lucchi, EPFL Technical Report, 2010. RGB- (D) Scene Labeling: Features and Algorithms, X.Ren, L.Bo and D.Fox, CVPR2012.

When the area in the image that the user wants to select is large or irregular, it is difficult for the user to accurately select the area. For example, if you specify a bounding box that surrounds a person with both arms outstretched, if you fail to place the first point in the correct position, your arms will break or you will get a bounding box that is too large for the person. Sometimes. Further, cutting out a contour by designating a plurality of points is a very time-consuming work when there are many irregularities on the contour of the area.

In the bottom-up method described above, the area is expanded by the similarity of adjacent areas. Therefore, for example, in an image in which a person running in a red running shirt has a wall close to skin color in the background, it is determined that the arm and the wall are more similar areas than the arm and the running shirt. In this case, it is determined that the arm and the wall are connected to each other, and the person in the image is not selected as one area.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an information processing apparatus capable of selecting a predetermined area from an image by a simple operation.

The information processing apparatus of the present invention includes an image acquisition means for acquiring an image, an area identification means for hierarchically identifying areas in the acquired image into a plurality of categories, a display means for displaying the image, and the display means. According to the initial area setting means for setting the area at a predetermined position as the initial area of the selection area according to the user's operation on the image displayed on the image, and according to the hierarchical category determination result according to the user's predetermined operation. It is characterized by comprising an area control means for expanding and contracting the selected area and updating the selected area.

According to the present invention, a user can select a predetermined area from an image by a simple operation.

Explanatory drawing of information processing apparatus. A flowchart showing the learning process. (A) to (c) are explanatory views of the training image and the area category label data. Explanatory drawing of the area category label. (A) and (b) are flowcharts showing an image area selection process. (A) to (c) are explanatory views of an image area selection process. (A) to (c) are explanatory views of an image area selection process. (A) to (d) are explanatory views of an image area selection process. (A) and (b) are explanatory views of image area selection processing. (A) to (d) are explanatory views of an image area selection process. (A) to (g) are explanatory views of an image area selection process. (A) to (c) are explanatory views of image processing. The flowchart which shows the process of S1600. (A) to (c) are explanatory views of the excursion expansion operation and the excursion reduction operation. (A) to (e) are explanatory views of the excursion expansion operation. The flowchart which shows the process of S1600. (A) to (d) are explanatory views of the contour correction operation. (A) and (b) are explanatory views of the area addition operation.

Hereinafter, embodiments will be described in detail with reference to the drawings.

(First Embodiment)
FIG. 1 is an explanatory diagram of an information processing device that realizes the image area selection device of the present embodiment. The image area selection device has a function for performing an image area selection process for the user to select a desired area from an image, and a learning process for generating in advance an area classifier necessary for performing the image area selection process. It has a function to perform.

The function for performing the image area selection process will be described. This function is realized by the image acquisition unit 1100, the area division unit 1200, the area identification unit 1300, the display unit 1400, the initial area setting unit 1500, the area control unit 1600, and the processing unit 1700. Each function may be realized on the same information processing device, or may be realized by independent modules. As the information processing device, for example, a combination of a personal computer and a monitor, a tablet terminal, a smartphone, or the like can be used. Each function may be realized by executing a computer program implemented in the information processing apparatus on a CPU (Central Processing Unit). Further, each function may be realized by executing hardware or a computer program inside a photographing device such as a camera.

The image acquisition unit 1100 acquires an input image from an external device. The area division unit 1200 divides the input image acquired by the image acquisition unit 1100 into a plurality of small areas. The area identification unit 1300 reads out the area classifier stored in the area classifier storage unit 5200, and estimates the area category of each small area divided by the area division unit 1200. The area classifier storage unit 5200 stores the area classifier generated by the learning process described later. The display unit 1400 is a display device that displays the input image acquired by the image acquisition unit 1100. The user can confirm the input image by the display of the display unit 1400. The initial area setting unit 1500 sets the area at a predetermined position of the input image as the initial area according to the instruction of the user by the predetermined interface. The area control unit 1600 expands / reduces the initial area according to the operation performed by the user, and generates a selected area. The processing unit 1700 performs a predetermined process on the selected area.

The function for performing the learning process for generating the area classifier used in the image area selection process will be described. This function is realized by the learning data acquisition unit 2100, the learning image area division unit 2200, and the area classifier generation unit 2300. Each function may be realized on the same information processing device, or may be realized by independent modules. Each function may be realized by executing a computer program implemented in the information processing apparatus on a CPU (Central Processing Unit).

The learning data acquisition unit 2100 acquires learning data from the learning data storage unit 5100. The learning data storage unit 5100 stores the learning data used in the learning process in advance. The learning data is composed of a plurality of learning images and area category label data to which an area category label for hierarchically defining each pixel of the learning image is assigned. The learning image area division unit 2200 divides each learning image into small areas for the learning data acquired by the learning data acquisition unit 2100. The area classifier generation unit 2300 performs learning processing based on the feature amount of each small area divided by the learning image area dividing unit 2200 and the area category label, and generates an area classifier that identifies the category of the small area. The area classifier generation unit 2300 stores the generated area classifier in the area classifier storage unit 5200. The learning data storage unit 5100 and the area classifier storage unit 5200 are realized by internal or external storage of the information processing device.

Each function used for the image area selection process and each function used for the learning process may be realized on the same information processing device or may be realized by different information processing devices. When the learning process and the image area selection process are realized by separate information processing devices, the area classifier storage unit 5200 may be realized by different storages for each. In that case, the area classifier obtained in the learning process is used by copying or moving it to the storage in the device for the image area selection process.

The learning process and the image area selection process by the image area selection device having the above configuration will be described. FIG. 2 is a flowchart showing the learning process. The learning process is to generate an area classifier used for performing an image area selection process from a trained image prepared in advance. The area classifier generated by learning once is stored in the area classifier storage unit 5200, read from the area classifier storage unit 5200, and reused. Therefore, it is not necessary to perform the learning process each time during the image area selection process.

When the learning process is started, the learning data acquisition unit 2100 acquires the learning data including the learning image and the hierarchically defined area category label data from the learning data storage unit 5100 (S2100). The learning image is specifically image data taken by a digital camera or the like. The number of learning images is N, and the nth learning image is described as a learning image I_n (n = 1 ... N). In the area category label data, a hierarchical area category label is assigned to each pixel of each learning image. The number of layers is L, and the index of the layer is described as index l = 1 ... L. Let M_l be the number of categories defined in the first layer.

FIG. 3 is an explanatory diagram of the training image and the area category label data. In this example, the case where the number of layers L = 5 layers will be described, but the number of layers is not limited to this value. In the learning image 500 shown in FIG. 3A, the corresponding area category label data is shown in layers 510 to 550 of FIG. 3B. Area category labels give the subject categories from coarse to detailed. In the example of FIG. 3B, layer 510 is the coarsest category label, detailed categories are given in the order of layers 520, 530, 540, and layer 550 is the most detailed category definition. Here, the first layer, the second layer, and so on are referred to in order from the coarsest layer 510.

In the first layer 510, category labels of empty 511 and non-empty 512 are assigned. In the second layer 520, the sky 511 in the first layer 510 is inherited as the sky 521, and the non-empty 512 in the first layer 510 is decomposed into the human body 522 and the plant 523. In the third layer 530, the sky 521 is inherited as the sky 531 and the human body 522 is decomposed into the face 532 and the upper body 533, and the plant 523 is decomposed into the flowers 534 and the foliage 535. In the fourth layer 540, the sky 531 is inherited as the sky 541, the upper body 533 is inherited as the upper body 544, the flower 534 is inherited as the flower 545, the foliage 535 is inherited as the foliage 546, and the face 532 is decomposed into the hair 542 and the face 543. .. In the fifth layer 550, the sky 541 is inherited as the sky 551, the hair 542 is inherited as the hair 552, and the upper body 544 is inherited as the upper body 555. The face 543 is decomposed into eyes 553, facial skin 554, and mouth 556, the flower 545 is decomposed into petals 557 and tubular flowers 558, and the stems and leaves 546 are decomposed into leaves 559 and stems 560.

The area category label data in the first layer corresponding to the training image I_n is represented as GT_ (n, l). In addition to the examples in FIG. 3, these semantic area categories are hierarchically defined inclusive relations. FIG. 4 is an explanatory diagram of the area category label of FIG. It goes without saying that the area category label can define various area categories and hierarchy levels in addition to this.

The learning image area dividing unit 2200 divides the acquired learning image into small areas (S2200). The small area is cut out from the training image as a small area having similar characteristics. Several methods have been proposed for dividing the training image into small areas. As a typical example, there is a method as described in Non-Patent Document 1. Further, a region obtained by simply dividing the learning image into rectangles of uniform size may be used as a small region. Further, each pixel of the learning image may be regarded as a small area, and in that case, it is not necessary to perform the division processing in particular. FIG. 3C exemplifies the result of dividing the learning image 500 into small areas.
When R_n small regions are generated as a result of region division for the training image I_n, the total number of small regions for training is R = ΣR_n. The small area of the training image is described as SP_r (r = 1 ... R) with a serial number.

The area classifier generation unit 2300 learns and generates an area classifier for identifying a category of a small area (S2300). The area classifier generator 2300 extracts the area features related to the small area used for learning. The area features extracted here are not limited to the types such as the color average value and the color histogram inside the small area, the position and size of the small area, and the texture features such as LBP (Local Binary Pattern). Further, the region feature may be a context feature based on a line segment or a color distribution around the small region. Further, the convolutional layer may be regarded as a feature extractor by using CNN (Convolutional Neural Network). The region feature extracted from the small region SP_r is referred to as a small region feature x_r.

Assuming that the area category label of the first layer corresponding to the small area SP_r of GT_ (n, l) is c_ (r, l), the teacher vector τ_ (r, l) of the first layer for the small area SP_r is the following equation. It is represented by.

Here, the area category label c_ (r, l) is an index of the category in which the small area SP_r is assigned as the area category label in the first layer. The learning of the region classifier is performed so that the error between the output vector and the teacher vector τ_ (r, l) obtained when the small region feature x_r is input to the discriminant function is small throughout the entire training data. Adjusting the parameters to generate a region classifier. The area classifier takes the small area feature x_r as an input and outputs the score vector f_l (x_r) of the area category in each layer. The score vector f_l (x_r) is an M_l dimension vector. Each element of the score vector f_l (x_r) is a score for each area category, and the score for the cth area category of the first layer is described as f_c (x_r) (c = 1 ... M_l).

There are various possible models of discriminant functions and their learning methods. For example, SVM, multi-layer neural network, logistic regression and the like can be used. Further, when the above-mentioned CNN is used, the fully connected layer can be regarded as a model of the discriminant function, and it is possible to learn including the convolution layer responsible for feature extraction. The model of the discriminant function of the present embodiment and the learning method thereof are not limited by their types. The area classifier generation unit 2300 stores the area classifier obtained by learning in the area classifier storage unit 5200.

The learning process is performed as described above. The information processing device performs the image area selection process using the area classifier obtained by the learning process. FIG. 5 is a flowchart showing the image area selection process. FIG. 5A shows the entire processing of the image area selection processing. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, and FIG. 11 are explanatory views of the image area selection process.

The image acquisition unit 1100 acquires an input image (S1100). FIG. 6A illustrates the input image 100. Various methods can be considered for acquiring the input image 100, but the present embodiment is not limited to the acquisition method. For example, it may be acquired directly from an image pickup device such as a camera, or may be acquired from image data previously stored in a storage such as a hard disk.

The area division unit 1200 divides the acquired input image 100 into small areas (S1200). It is preferable that the learning image region dividing unit 2200 is the same processing as the processing of S2200 in the division processing into small regions performed here. When each pixel is regarded as a small area, it is not necessary to perform the division process in particular. FIG. 6B illustrates the region division result 200 of the input image 100. Let K be the total number of small areas obtained by dividing the input image 100 into areas.

The area identification unit 1300 identifies the area category for each small area of the input image 100 (S1300). The area identification unit 1300 extracts the area features of each small area generated by the area division unit 1200. The region feature to be extracted is, for example, the same type as the region feature extracted from the small region of the learning image by the region classifier generator 2300 in the process of S2300 of FIG. Let x_k be the region feature extracted from the small region SP_k (k = 1 ... K). The area classifier 1300 reads the area classifier obtained by the learning process stored in the area classifier storage unit 5200. The area identification unit 1300 inputs the area feature x_k related to each small area SP_k to the read area classifier f_l (l = 1 ... L). As a result, the area identification unit 1300 generates the score vector f_l (x_k) of each area category in the first layer.

The area identification result for each small area SP_k of the first layer is generated, for example, as a category c_ (k, l) having the maximum score vector f_l (x_k).

The area identification unit 1300 applies the area classifier f_l in all layers l to all the small areas SP_k (k = 1 ... K) to obtain the category c_ (k, l) which is the result of all area identification. When it is done, the processing of S1300 is terminated. FIG. 6 (c) illustrates the region identification results 110, 120, 130, 140, 150. As the most detailed category identification result, the identification result in the fifth layer is shown in the area identification result 150. In this example, mouth 151, hair 152, eyes 153, facial skin 154, arms 155, torso 156, hands 157, crotch 158, legs 159, feet 161, indoor wall 162, furniture 163, outer wall 164, floor 165, and the like. The area is obtained. In the region identification result 140 in the fourth layer, regions such as hair 141, facial skin 142, arms 143, torso 144, crotch 145, legs 146, head 147, and indoor 148 are obtained. In the area identification result 130 in the third layer, areas such as the head 131, 134, the upper body 132, the lower body 133, and the building 135 are obtained. In the area identification result 120 in the second layer, the areas of the artificial object 121, the human body 122, and 123 are obtained. In the area identification result 110 in the first layer, the area of the entire screen is identified as non-empty 111.

The display unit 1400 displays the input image 100 (S1400). The display unit 1400 shall have a graphical user interface for the displayed image, but the present embodiment is not limited to the format of the display unit 1400. The display unit 1400 that displays the input image 100 may be a touch panel or a monitor connected to a personal computer that can use a mouse or a pen tablet. In the following, the display unit 1400 will be described by taking a touch panel used for a tablet or a smartphone as an example.

The initial area setting unit 1500 sets the initial area by designating a predetermined position with respect to the input image 100 displayed on the display unit 1400 (S1500).
The user taps a part of the area to be selected on the input image 100 displayed on the display unit 1400 as illustrated in FIG. 7A. The initial area setting unit 1500 sets the small area S_i including the tapped position as the initial area among all the small areas S_k (k = 1 ... K) obtained by being divided by the area dividing unit 1200. For example, when the tapped position as shown in FIG. 7 (a) is inside the small area 401 corresponding to the right half of the facial skin of the right person shown in FIG. 8 (a), it is shown in FIG. 8 (b). As described above, this small area 401 becomes the initial area 411.

The area control unit 1600 expands or contracts the initial area 411 set by the initial area setting unit 1500 according to a predetermined operation, and acquires a desired area (S1600). FIG. 5B shows the details of the processing by the area control unit 1600.

The area control unit 1600 displays the area (selected area) selected at that time on the display unit 1400 (S1690). The display format of the selected area is not limited to this embodiment. As the selected area, the outline of the selected area may be displayed on the input image as illustrated in FIG. 9 (a), or only the inside of the selected area may be displayed as illustrated in FIG. 9 (b). ..

The area control unit 1600 acquires an operation performed by the user with respect to the displayed selected area (S1610). The area control unit 1600 determines the operation content by the user and determines the next process (S1615). When the operation performed by the user is an area expansion operation (S1615: area expansion), the area control unit 1600 expands the selected area at that time (S1620). When the operation performed by the user is an area reduction operation (S1615: area reduction), the area control unit 1600 reduces the selected area at that time (S1630). When the operation performed by the user is the area selection end operation (S1615: end), the area control unit 1600 ends the area control process. This completes the image area selection process.

7 (b) and 7 (c) explain the area expansion operation and the area reduction operation. Here, the area expansion operation and the area reduction operation are assigned to the up and down slide operations on the display screen. However, the area expansion operation and the area reduction operation are not limited to this. For example, the area expansion operation and the area reduction operation may be performed by the left and right slide operation. When the pressure when pressing the touch panel increases, the area expansion operation may be performed, and when the pressure decreases, the area reduction operation may be performed. You may distinguish or switch the operation by tapping another menu while long tapping. If it is a combination with a keyboard, it may be distinguished or switched by a combination with a special key such as the SHIFT or Ctrl key while long-tapping. When a mouse is used, the area expansion operation and the area reduction operation may be performed by the vertical movement or the left and right movement of the mouse, or may be controlled by the rotation operation of the scroll wheel. The area expansion operation and the area reduction operation may be distinguished or switched by the combination of the mouse click operation and the special key such as the SHIFT key or the Ctrl key. The area selection end operation is, for example, an operation such as releasing the finger from the touch panel or double-clicking with the mouse.

When expanding the selected area in the process of S1620, the area control unit 1600 first adds a small area spatially adjacent to the selected area and in the same category to the selected area in the same layer. , Expand the selection area. When there is no adjacent same category area in the hierarchy, the area control unit 1600 moves the hierarchy up one level and performs the same processing. When one small area is added to the selected area, the area control unit 1600 returns to the process of S1690.

A specific example of how the area is expanded by repeating the process of S1620 is shown below. A case where the area expansion operation is performed on the initial area 411 illustrated in FIG. 8B will be described as an example.
The initial region 411 is included in the region of the facial skin 154 determined to be the facial skin category in the region identification result 150 of the fifth layer of FIG. 6 (c). Of the small areas adjacent to the initial area 411, the small areas determined to be in the same facial skin category are the small areas 402 and 403 shown in FIG. 8A. The region control unit 1600 first selects a region having the closest characteristics to the initial region 411 among these regions. Various features such as color histograms and texture features such as LBP can be considered as the feature amount used for selection, but the feature amount is not limited in this embodiment.

The area control unit 1600 updates the area in which the small area 402 having the closest characteristics to the initial area 411 is combined with the initial area 411 as a new selection area. FIG. 8 (c) illustrates the updated selection area 412. Further, when the area expansion operation is continuously performed, the area control unit 1600 updates the remaining small area 403 combined with the selection area 412 as the selection area. FIG. 8 (d) illustrates the updated selection area 410. When the area is expanded up to this point, all the facial skin areas that can be connected to the initial area in the fifth layer are connected.

If the area expansion operation is further performed here, the area control unit 1600 moves the processing layer up one level, and in this example, to the fourth layer. The facial skin category area in the fifth layer belongs to the face category area in the fourth layer.
Among the face category regions, the small regions 404, 405, and 406, which are the eyes and mouth shown in FIG. 10 (a), are not yet included in the selected region at this point. However, in the fourth layer, it belongs to the face skin 142 of the face category area of FIG. 6C, similarly to the selection area 410. Therefore, the area control unit 1600 combines the small area having the closest characteristics with the selected area 410 among the small areas adjacent to the selected area 410 in the same manner as before. FIG. 10B illustrates an updated selection region 421 by combining a small region 404 with respect to the selection region 410. FIG. 10 (c) illustrates a selection region 422 further coupled with a small region 405. FIG. 10 (d) illustrates a selection region 420 further coupled with a small region 406.

In this way, small areas of the same category in the same hierarchy are connected, and when there are no adjacent small areas of the same category, the next higher layer is moved and the same processing is repeated. , The area can be expanded according to the hierarchical category. 11 (a) to 11 (g) show the connection results in each layer when the area expansion operation is continued from the initial area 411. FIG. 11B illustrates a selection region 410 obtained by connecting the same category regions from the initial region 411 of FIG. 11A in the fifth layer. Further, when the area expansion operation is continued, the face category area illustrated in FIG. 11C is obtained as the selection area 420 in the fourth layer, and the head category area illustrated in FIG. 11D is the selection area in the third layer. Obtained as 430.
In the second layer, by expanding from the head category area, the right person area 440 illustrated in FIG. 11 (e) is obtained as an intermediate result of the person category area. Further expanding, as shown in FIG. 11 (f), a person category area in which the left person area is also connected through the connected hands is obtained as the selection area 450. When the connection is further continued in the first layer, as shown in FIG. 11 (g), an image in which the entire image is combined into one area as a non-empty category area is set as the selection area 460.

When the area is reduced by the processing of S1630, the area control unit 1600 first obtains a small area from the current selection area excluding the category area including the initial area in the layer one level below the current layer. Use as a deletion candidate area. The area control unit 1600 updates the selected area by removing from the selected area a small area having the characteristics most different from the selected area among the deletion candidate areas. When one small area is deleted from the selected area, the area control unit 1600 returns to the process of S1690. A specific example in which the area is reduced by repeating the process of S1630 is shown below.

For example, it is assumed that the area reduction operation is performed on the selected area 420 illustrated in FIG. 10 (d). At this time, the processing layer is the fourth layer, and the processing target category is the face skin 142 of the face category area shown in FIG. 6C. In the next lower layer, that is, the fifth layer, the area including the initial area is the category area of the facial skin 154 illustrated in FIG. 6 (c). The category region of the facial skin 154 including the initial region is the selection region 410 illustrated in FIG. 10 (a), and the regions excluding it are the small regions 404, 405, and 406 in FIG. 10 (a). The area control unit 1600 sets these small areas 404, 405, and 406 as deletion candidate areas, and deletes the small area having the most different characteristics from the selected area 420 from the selected area. As a result, as shown in FIGS. 10 (c) and 10 (b), the selected area 420 is deleted from the portion other than the facial skin category area, so that only the facial skin category area remains like the selected area 410. It will be. When the selection area 410 becomes only the facial skin category area, the processing hierarchy is lowered by one. Here, moving to the fifth layer, the area control unit 1600 continues the same reduction processing with the small areas 401, 402, and 403 shown in FIG. 8A excluding the initial area 411 as deletion candidate areas. If the reduction process is continued without stopping, it is finally possible to return to the state of the initial region 411 illustrated in FIG. 11 (a).

The user may stop the area expansion operation by performing the area selection end operation when a desired area is obtained while performing the area expansion operation and the area reduction operation.
As described above, by controlling the expansion and contraction of the area according to the hierarchical semantic category, the user can easily select the area which is a meaningful mass. The region thus obtained is very useful for various image processing.

FIG. 12 is an explanatory diagram of image processing of the present embodiment. In the example of the image 700 in FIG. 12, if the area including a person is to be cropped, it is as follows. First, the internal area of the person is set as the initial area, and the person area 710 illustrated in FIG. 12A is selected by performing the area expansion / reduction operation. It is easy to calculate the circumscribed rectangle 720 of the selected area illustrated in FIG. 12B. Based on this, cropping can be performed so as to fully include the human body region 730 illustrated in FIG. 12 (c). Similarly, it is easy to optimize the zoom factor with respect to the screen size according to the circumscribed rectangle of the selected area. In this way, if the user can freely expand and reduce the area according to the semantic category as in the present embodiment, the cropping and zoom ranges are automatically calculated for the area desired by the user. It can be done, and there will be no cut-offs or useless margins.

(Second Embodiment)
In the first embodiment, the area is selected by expanding the area designated by the user and gradually connecting the adjacent areas. However, since the semantic area identification result is obtained, it is possible to select the areas of the same category collectively without being limited to the spatial adjacency relationship on the image. In the present embodiment, it is possible to select an area by such an excursion. Since the apparatus configuration of this embodiment is the same as that of the first embodiment shown in FIG. 1, the description thereof will be omitted. Further, since the learning process of this embodiment is the same as the learning process of the first embodiment shown in FIG. 2, the description thereof will be omitted.

The image area selection process of the present embodiment is roughly the same as the image area selection process of the first embodiment shown in FIG. 5A. In the present embodiment, the details of the area control process of S1600 are different from those of the first embodiment. The area control process of S1600 of the present embodiment is a combination of a total of four types of operations, including two types of an excursion expansion operation and an excursion reduction operation in addition to the area expansion operation and the territory reduction operation performed in the first embodiment. It is done in.

FIG. 13 is a flowchart showing the area control process of S1600 of the present embodiment. The processing of S1690, S1610, S1615, S1620, and S1630 is the same as the processing of the first embodiment shown in FIG. 5 (b). In the present embodiment, the area control unit 1600 determines the jumping area expansion operation and the jumping area reduction operation in addition to the area expansion operation and the area reduction operation by the process of S1615 which determines the operation content by the user and determines the next process. I do.

The area control unit 1600 performs the process of S1620 if the operation performed by the user is an area expansion operation, performs the process of S1630 if the operation is an area reduction operation, and the area if the operation performed by the user is an area selection end operation. End the control process. The area control unit 1600 performs the excursion expansion process of S1640 if the operation performed by the user is the excursion expansion operation, and performs the excursion reduction process of S1650 if the operation is the excursion reduction operation.

The area expansion operation and the area reduction operation are the operations described with reference to FIGS. 7 (b) and 7 (c). FIG. 14 is an explanatory diagram of an excursion expansion operation and an excursion reduction operation. FIG. 14A shows an excursion expansion operation. FIG. 14B shows an excursion reduction operation. While the area expansion operation and the area reduction operation are up and down slide operations (see FIGS. 7 (b) and 7 (c)), the excursion expansion operation and the excursion reduction operation are left and right slide operations. Here, the distinction of the slide direction is not limited to the combination described above, as long as the four operations can be distinguished by the combination of the slide directions of up, down, left and right. As shown in FIG. 14C, the excursion expansion operation and the excursion reduction operation may be performed by a tap operation or a double tap operation with another finger in another spatially separated area.
When using a mouse, the area expansion operation, the area reduction operation, the excursion expansion operation, and the excursion reduction operation may be performed by combining the vertical movement or the left / right movement of the mouse, and any of them may be combined with the rotation operation of the scroll wheel. It may be done. Alternatively, the area expansion operation, the area reduction operation, the excursion expansion operation, and the excursion reduction operation may be performed in combination with a special key such as the Alt key or the Tab key.

In the process of S1640, the area control unit 1600 adds a small area of the same category as the selected area at that time to the selected area regardless of the spatial adjacency relationship on the image. When one or a plurality of small areas are added, the area control unit 1600 returns to the process of S1690. A specific example of the processing of S1640 is shown below.

For example, when the excursion expansion operation is performed in the state of FIG. 10 (c), the process of S1640 is performed in the area identification result 140 of the fourth layer of FIG. 6 (c), and the area category being expanded is the face. It is a category. The small area of the same category adjacent to the selected area 422 at that time is only the small area 406 of FIG. Small areas of the same category are candidates for consolidation.

FIG. 15 is an explanatory diagram of the excursion expansion operation. The small areas 407, 408, 409, 411, 421, and 413 that form the face of the left person in FIG. 15A are also candidates for connection of the selected area 422 in addition to the small area 406 of the right person. Of the small areas 407, 408, 409, 411, 421, and 413 that make up the face of the left person, the areas whose features are closest to the selected area 422 are connected.

The area control unit 1600 updates a new selection area including a small area (here, a small area 407) having the closest characteristics to the selection area 422 and an excursion area added to the selection area 422. FIG. 15B illustrates the updated selection area 471. When the small area having the closest characteristics to the selected area 471 is the small area 408, the selected area obtained next is as shown in the selected area 472 illustrated in FIG. 15 (c). By adding the remaining small regions 406, 409, 411, 412, and 413, respectively, the selected region in this hierarchy finally becomes the selected region 470 illustrated in FIG. 15 (d).

When the area expansion operation is performed in the state of FIG. 15D, the expansion process is continued by moving to the next higher layer, here, the third layer. In this case, the area control unit 1600 expands the head category area with respect to the adjacent areas of the right person and the left person. Therefore, in this hierarchy, the selection area 480 illustrated in FIG. 15 (e) is finally obtained. Further, when the area expansion process is continued, the selection area 450 illustrated in FIG. 11F is obtained in the second layer.

In the process of S1650, the area control unit 1600 preferentially deletes a small area from the selected area at the present time with respect to the portion that is not spatially connected to the initial area in the current hierarchy. When the area control unit 1600 deletes one or a plurality of small areas from the selected area, the area control unit 1600 returns to the process of S1690. A specific example of the processing of S1650 is shown below.

Here, a case where the excursion reduction process is performed on the selected area 470 illustrated in FIG. 15D will be described. When the initial area is the area of the right person, the area control unit 1600 removes the small area having the most characteristics with respect to the entire selection area 470 from the small areas in the face area of the left person. When all the small areas on the face of the left person are excluded from the selection area in this way, the selection area 420 illustrated in FIG. 11C remains. When the excursion reduction operation is continued, the area reduction process for the face area of the right person is performed in the same manner as the area reduction process.

By using the excursion expansion process and the excursion reduction process together with the area expansion process and the area reduction process as described above, the user can easily select a favorite area according to the purpose. For example, when it is desired to select the entire body of the right person in the input image 100 illustrated in FIG. 6A, the user selects the internal area of the right person as the initial area. The information processing device continues the area expansion process based on this initial area. When the right person area 440 illustrated in FIG. 11E is obtained, the user performs the area selection end operation, so that the information processing apparatus ends the area selection process.

Further, when it is desired to select only the faces of both the left and right persons, the user taps the inside of the face area of either person to select it as the initial area. When the user performs the excursion expansion operation when the face area of the person is obtained, the information processing apparatus can obtain a selection area including another face area. It is also possible to select two or more person areas by the same operation. Needless to say, it is also possible to simultaneously select, for example, a plurality of automobile areas other than the person area.

(Third Embodiment)
The small area that is the processing unit during the area control of the first embodiment and the second embodiment is not always divided at a desired contour position. For example, if the background of black hair is a dark night view, the hair and background areas may not be divided into one small area. In this embodiment, the contour is modified in such a case to obtain an appropriate small area. Since the apparatus configuration of this embodiment is the same as that of the first embodiment shown in FIG. 1, the description thereof will be omitted. Further, since the learning process of this embodiment is the same as the learning process of the first embodiment shown in FIG. 2, the description thereof will be omitted.

The image area selection process of the present embodiment is roughly the same as the image area selection process of the first embodiment shown in FIG. 5A. In the present embodiment, the details of the area control process of S1600 are different from those of the first embodiment. The area control process of S1600 of the present embodiment includes two types of operations, a contour correction operation and an area addition operation, in addition to the area expansion operation, the territory reduction operation, the excelave expansion operation, and the excelave reduction operation performed in the second embodiment. It is performed by a combination of a total of 6 types of operations including.

FIG. 16 is a flowchart showing the area control process of S1600 of the present embodiment. The processing of S1690, S1610, S1615, S1620, S1630, S1640, and S1650 is the same as the processing of the second embodiment shown in FIG. In the present embodiment, the area control unit 1600 determines the operation content by the user and determines the next process by the process of S1615, in addition to the area expansion operation, the territory reduction operation, the excursion expansion operation, and the excursion reduction operation. Judge the contour correction operation and the area addition operation.

The area control unit 1600 performs the process of S1620 if the operation performed by the user is an area expansion operation, performs the process of S1630 if the operation is an area reduction operation, and the area if the operation performed by the user is an area selection end operation. End the control process. The area control unit 1600 performs the process of S1640 if the operation performed by the user is a jump land expansion operation, and performs the process of S1650 if the operation is a jump land reduction operation. The area control unit 1600 performs the process of S1660 if the operation performed by the user is a contour correction operation, and performs the process of S1670 if the operation is an area addition operation.

The area expansion operation, the area reduction operation, the excursion expansion operation, and the excursion reduction operation are the operations described with reference to FIGS. 7 and 14. FIG. 17 is an explanatory diagram of the contour correction operation. FIG. 18 is an explanatory diagram of the area addition operation.

FIG. 17A shows a region 850 obtained as a result of performing a region expansion operation and a region reduction operation on the displayed input image 800. Here, the area that the user really wants to obtain is represented by the contour 810. The actually obtained region is represented by the contour 820. Therefore, the region between the contour 810 and the contour 820 becomes an unnecessary region.
In FIG. 17B, as a contour correction operation, the user performs a flick operation on an unnecessary area portion of the displayed input image 800 with a finger different from the finger used for area selection. Here, the flick operation is a contour correction operation. In addition to this, the contour correction operation may be a tap operation or a double tap operation on an unnecessary area. The information processing device stores the position of the unnecessary area specified by the user when the contour correction operation is performed as the unnecessary area position.

The area control unit 1600 re-divides the image in the vicinity of the unnecessary area position into small areas by the process of S1660. FIG. 17 (c) illustrates a small region 830 containing a selected unwanted region position. Inside the small area 830, the area control unit 1600 resets the area division parameters so as to further subdivide the small area, and as illustrated in FIG. 17 (d), the small area 830 is subdivided into small areas. Regions 831 and 832 are generated. The area control unit 1600 adds a flag that the small area 832 including the unnecessary area position generated by subdivision does not belong to any category, and excludes it from the selected area as a target of the area expansion and area reduction processing. .. When the area control unit 1600 updates the selected area, the area control unit 1600 returns to the process of S1690.

The area addition operation will be described. FIG. 18A shows the area 950 obtained as a result of performing the area expansion operation and the area reduction operation on the input image 900. Here, the area that the user really wants to obtain is represented by the contour 910. Therefore, a shortage area 920 is generated outside the selected area. In FIG. 18B, as an area addition operation, the user long-tap the insufficient area portion with a finger different from the finger used for area selection. The information processing device stores the position specified by the user when the area addition operation is performed as the additional area position.
By the process of S1670, the area control unit 1600 replaces the category determination result of the small area at the designated additional area position so as to match the category of the selected area, and adds it to the selected area. When the area control unit 1600 updates the selected area, the area control unit 1600 returns to the process of S1690.

By such a process, the user can easily correct the contour by subdividing the small area and add the missing area. Therefore, the information processing apparatus can select a more accurate area with a simple operation even if the small area division or the area recognition is inappropriate.

(Fourth Embodiment)
In the first to third embodiments, it has been described that only one category can be obtained as the area identification result for each small area of the image, but a plurality of categories may be obtained in consideration of the ambiguity of the categories. .. In this embodiment, an image area selection method that allows output of a plurality of categories will be described. The ambiguity of the category refers to a state in which the category to which the category belongs is not uniquely determined for a predetermined area in the image. For example, a log house with a bare tree trunk may be classified as a natural tree or an artificial building as a category. When there is such an area in the training image, a plurality of labels are duplicated as the area category label.

In the present embodiment, in the learning process, the area classifier generation unit 2300 learns the area classifier by the following formula.

Here, C_ (r, l) is a set of category label indexes allocated to allow duplication with respect to the small area r in the first layer. Other than that, the same learning process as in the first embodiment is performed.

In the image area selection process, the area identification unit 1300 acquires the area identification result of each small area k by the following formula.

Here, C_ (k, l) is a set of category indexes that are the identification results for the small area k in the first layer. θ is a predetermined threshold value, and is set, for example, θ = 0.5. As a result, a plurality of class labels may be obtained as identification results depending on the area of the input image.

The area expansion processing, area reduction processing, excelave expansion processing, excelave reduction processing, contour correction processing, and area addition processing described in the first to third embodiments are the initial area setting processing of S1500 by the initial area setting unit 1500. Depends only on the initial area set by. Therefore, even if a plurality of labels are assigned to each small area as described above, the effect described in the first to third embodiments can be obtained by executing the image area selection process by performing the same process. Can be obtained.

(Fifth Embodiment)
In the first to fourth embodiments, the area expansion and the area reduction are performed by adding and deleting a small area to the selected area. The area expansion and area reduction may be performed in pixel units as well as in small area units. In this case, the area control unit 1600 adds the pixel having the highest degree of similarity to the selected area among the pixels of the same category adjacent to the selected area in the process of S1620. As the similarity that can be used at this time, the likelihood of the pixel color luminance value when the color distribution in the selected region is expressed as a mixed Gaussian distribution may be used. Similarly, in the process of S1630, the area control unit 1600 deletes the outermost pixel of the selected area and the pixel having the lowest similarity with the selected area from the selected area.

In each of the above embodiments, the area is set based on the semantic category obtained by learning using prior knowledge, so that an accurate area can be selected as compared with bottom-up area growth. .. In addition, various levels of semantic areas can be selected according to the concept of upper / lower levels of semantic categories. As a result, the user can easily select an area in the image, and can easily perform various operations by the user interface for each area, which is complicated in the conventional point designation.

The present invention supplies a program that realizes one or more functions of each of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (21)

Image acquisition means to acquire images and
An area identification means for hierarchically identifying the area in the acquired image into a plurality of categories, and
A display means for displaying the image and
An initial area setting means for setting an area at a predetermined position as an initial area of a selection area according to a user's operation on an image displayed on the display means.
It is characterized by comprising an area control means for expanding and contracting the selected area according to the hierarchical category determination result according to a predetermined operation of the user and updating the selected area.
Information processing device. The area control means expands the area by combining the areas of the same category adjacent to the selected area in the same layer in accordance with a predetermined first operation of the user to the selected area.
The selected area is reduced by deleting an area that does not include the initial area from the selected area in accordance with a predetermined second operation of the user.
The information processing device according to claim 1. The area control means is characterized in that the first operation and the second operation are distinguished by the direction of a slide operation by a touch panel or a mouse.
The information processing device according to claim 2. The area control means is characterized in that the first operation and the second operation are distinguished by the pressure on the touch panel.
The information processing device according to claim 2. The area control means is characterized in that the first operation and the second operation are distinguished by a combination of a click by a mouse and a special key.
The information processing device according to claim 2. The area control means is characterized in that the first operation and the second operation are distinguished by switching a click operation with a mouse with a special key.
The information processing device according to claim 2. The area control means is characterized in that the first operation and the second operation are distinguished or switched by a combination of a long tap on a touch panel and a special key.
The information processing device according to claim 2. The area control means is characterized in that the first operation and the second operation are distinguished or switched by tapping the touch panel separately from the long tap and the menu.
The information processing device according to claim 2. The area control means is characterized in that the first operation and the second operation are distinguished according to a rotation operation of a scroll wheel in a mouse.
The information processing device according to claim 2. The area control means connects an area in the same category as the selected area including an area not connected to the initial area in the same layer to the selected area in accordance with a predetermined third operation of the user. Extend and
The selected area is reduced by preferentially deleting an area that is not connected to the initial area from the selected area in accordance with a predetermined fourth operation of the user.
The information processing device according to any one of claims 1 to 9. The area control means is characterized in that the third operation and the fourth operation are recognized by a slide operation using a touch panel or a mouse.
The information processing device according to claim 10. The area control means recognizes the third operation and the fourth operation according to the rotation operation of the scroll wheel in the mouse.
The information processing device according to claim 10. The area control means recognizes the third operation and the fourth operation by a tap operation on another area away from the selected area.
The information processing device according to claim 10. The area control means is characterized in that the third operation and the fourth operation are distinguished from the first operation and the second operation by a special key.
The information processing device according to claim 10. The area control means
By designating an unnecessary area from the selected area and dividing the unnecessary area into areas according to a predetermined fifth operation of the user, the contour of the selected area is corrected.
Adding the shortage area to the selection area by designating the shortage area from outside the selection area and matching the category of the shortage area with the category of the selection area according to a predetermined sixth operation of the user. Characteristic,
The information processing device according to any one of claims 1 to 14. The area control means recognizes the fifth operation by tapping or flicking the unnecessary area displayed on the touch panel.
The information processing device according to claim 15. The area control means is characterized in that the fifth operation and the sixth operation are recognized by an operation of clicking the unnecessary area with a mouse.
The information processing device according to claim 15. The area control means recognizes the sixth operation by tapping the insufficient area displayed on the touch panel.
The information processing device according to any one of claims 15 to 17. Information processing device
Steps to get the image and
A step of hierarchically identifying areas in the acquired image into a plurality of categories,
The step of displaying the image and
A step of setting an area at a predetermined position as an initial area of a selection area according to a user's operation on the displayed image, and
Including a step of expanding and contracting the selected area according to the hierarchical category determination result according to a predetermined operation of the user and updating the selected area.
Image area selection method. Computer,
Image acquisition means to acquire images,
Area identification means for hierarchically identifying areas in the acquired image into a plurality of categories,
Display means for displaying the image,
An initial area setting means that sets an area at a predetermined position as an initial area of a selection area according to a user's operation on an image displayed on the display means.
An area control means that expands and contracts the selected area according to the hierarchical category determination result according to a predetermined operation of the user, and updates the selected area.
A computer program to function as. A computer-readable storage medium that stores the computer program of claim 20.

Images