JP6700804B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing device, an information processing method, and a program, and is particularly suitable for use in extracting a color of a subject in a captured image.

  There is a technology related to mixed reality in which information in a virtual space is superimposed on a real space in real time and presented to the user. In mixed reality, a composite image in which a virtual space image (Computer Graphics: CG) according to the position/orientation of the image pickup device is superimposed on the whole or a part of the actual image taken by the image pickup device such as a video camera. Is displayed.

  By extracting a region such as the user's hand from the image in the real space, it is possible to determine whether or not interference occurs between the region in the real space and the CG, and determine whether the region in the real space is in front of or behind the CG. A relationship can be determined. In the following description, a region of a subject such as a user's hand in the captured image will be referred to as a subject region as necessary.

  Methods of extracting the subject area include a method of dividing the area, a method of dynamically extracting the contour, and a method of using the result of learning. In either method, the initial subject area is detected by extracting the color (for example, skin color) of the subject area. Techniques described in Patent Documents 1 to 3 are techniques for extracting the color of the subject region in this manner.

  Patent Literature 1 discloses the following technique. A skin color distribution model is created from skin color data of various people. After that, the presence of a person's face is specified from the input image, and a skin color distribution model is created from the pixel data of the skin color in the specified face area. A modified skin color distribution model is created by weighting and adding these skin color distribution models. Using this modified skin color distribution model, the skin color likelihood corresponding to each pixel of the input image data is obtained. By comparing this skin color likelihood with a threshold value, it is determined whether each pixel data group of the input image data has a skin color.

  In addition, Patent Document 2 discloses the following technique. Color information is extracted from each of the background image that does not include the subject and the background image that includes the subject. The color information is compared with each other, and among the color information in the background image containing the subject, the color information not in the background image not containing the subject is registered as the color information of the subject. Further, by adjusting the color information of the noise region designated by the user from the subject region, the accuracy of detecting the color of the subject region is improved.

  Further, Patent Document 3 discloses that a color table corresponding to a pixel position is created, a color table to be used is designated for each pixel, and a color of each pixel is extracted using the designated color table. ..

JP, 2007-257087, A JP, 2005-228140, A Japanese Patent No. 3118915

  However, in the techniques described in Patent Documents 1 and 2, when the tint changes depending on the position of the pixel, there is a possibility that the color of the subject region may not be correctly extracted. For example, in a captured image with dimming around the camera, the brightness and tint change between the center and the edges. Therefore, if the color information is registered only near the center of the captured image, the number of undetected colors increases at the edge portion of the screen. On the other hand, if color information is registered in the entire screen, color overdetection increases.

On the other hand, in the technique described in Patent Document 3, the registered color can be extracted regardless of the location of the captured image by creating a color table for each pixel of the image. However, since a plurality of color tables are created for each pixel, an enormous memory is required and realization is not easy.
The present invention has been made in view of such a problem, and an object thereof is to make it possible to easily and reliably extract the color of a subject region in a captured image.

A first example of the information processing apparatus of the present invention includes an acquisition unit that acquires an area designated in a captured image of a subject captured by the imaging unit, and color information extracted from each pixel of the captured image , Of the pixels having the color information in the captured image, the number of pixels inside the designated area, and the number of pixels having the color information outside the designated area. Based on at least one of the numbers, the color information sets an effective range that is effective as the predetermined color in the process of extracting the area of the predetermined color from the image captured by the imaging unit. When the effective range setting means and the color information of each pixel of the image captured by the image capturing means are included in the effective range set for each color information by the effective range setting means at the position of the pixel. And a color area extracting means for extracting the pixel as a part of the area of the predetermined color .
A second example of the information processing apparatus of the present invention is an acquisition unit that acquires a region designated in a captured image of a subject captured by the image capturing unit, and a pixel included in the designated region of the captured image. For each color information extracted from, the color information is set as an effective range that is effective as the predetermined color in the process of extracting the area of the predetermined color from the image captured by the imaging unit. Effective range setting means, generating means for generating color information in a new effective range of a number larger than the effective range set by the effective range setting means, and each pixel of an image captured by the image capturing means. When the color information is included in the effective range set for each color information by the effective range setting means at the position of the pixel, the pixel is extracted as a part of the area of the predetermined color. A color area extracting means, wherein the generating means extracts, from each of the effective ranges set by the effective range setting means, one or more colors representing color information corresponding to the effective range, Color information in the new effective range is generated based on the extracted color.

  According to the present invention, it is possible to easily and surely extract the color of a subject region in a captured image.

It is a figure which shows the 1st example of a structure of an image processing system. It is a figure which shows the external structure of an image processing system. It is a flow chart which shows the 1st example of skin color registration processing. It is a flow chart which shows details of processing of Step S303. It is a figure which shows the distribution of the pixel number of skin color, and the distribution of the pixel number of non-skin color. It is a figure explaining a color table. It is a figure which shows the 2nd example of a structure of an image processing system. It is a flow chart which shows processing of an image pick-up position calculation part. It is a figure which shows the example of a display of the information which shows a new imaging region. It is a figure which shows the 3rd example of a structure of an image processing system. It is a flow chart which shows processing of an effective depth acquisition part. It is a figure which shows the 4th example of a structure of an image processing system. It is a flow chart which shows the 2nd example of skin color registration processing. 6 is a flowchart illustrating an example of color information generation processing by a color information interpolating unit. It is a figure which shows the relationship from the distance from a screen center, and a shading coefficient. It is a figure which shows the effective range of color information typically. It is a figure which shows typically the color information before interpolation by a color information interpolation part. It is a flowchart which shows the interpolation process by a color information interpolation part. 7 is a flowchart showing a synthesizing process performed by a color information interpolating unit.

  Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

(First embodiment)
First, the first embodiment will be described.
<Structure>
FIG. 1 is a block diagram showing an example of the configuration of an image processing system. FIG. 2 is a schematic diagram showing an example of the configuration of the image processing system in this embodiment.
As shown in FIGS. 1 and 2, the image processing system includes an imaging unit 100, an input unit 200, a display unit 300, and an information processing device 1000. In the present embodiment, the image capturing unit 100, the input unit 200, and the display unit 300 are communicably connected to the information processing apparatus 1000. The information processing apparatus 1000 also acquires a captured image from the image capturing unit 100. The imaging unit 100 is realized by, for example, a video see-through HMD (Head Mounted Display) or a network camera. The input unit 200 is realized by a mouse or a keyboard. The display unit 300 is realized by a display such as an HMD or a PC monitor.

As shown in FIG. 1, the information processing apparatus 1000 includes an image acquisition unit 1010, a data storage unit 1020, a skin color area setting unit 1030, a pixel information extraction unit 1040, an effective range calculation unit 1050, a skin color registration unit 1060, and an image processing unit. 1070 and an image output unit 1080.
The image acquisition unit 1010 acquires the captured image captured by the imaging unit 100 and stores the acquired image in the data storage unit 1020. The captured image includes a subject such as a hand.

The data storage unit 1020 stores information including virtual object data, a captured image input from the image acquisition unit 1010, and color table data input from the skin color registration unit 1060.
The skin color area setting unit 1030 sets the area input from the input unit 200 as the area where the skin color exists.

The pixel information extraction unit 1040 extracts pixel color information and pixel position from the captured image acquired by the image acquisition unit 1010. Here, the color information is information on a color that does not include brightness, such as CbCr or a ^* b ^* . Color information is not limited as long as it is a method of expressing a color that does not depend on brightness. The pixel position is information indicating the position of the pixel, and is represented by, for example, the distance from the center of the screen. Further, the pixel position may be represented by the identification number of each small area when the area of the screen is divided into a plurality of small areas. Any pixel position may be represented as long as the pixel region can be identified. In the present embodiment, the case where the distance from the center of the screen is the pixel position will be described as an example.

  The effective range calculation unit 1050 calculates an image area (effective range) in which the color information is effective as a flesh color based on the distribution of the number of pixels for each color information (relationship between the number of pixels and the pixel position). For example, the effective range calculation unit 1050 derives the appearance frequency of the color information for each distance from the center of the screen, and sets the distance range with high appearance frequency as the image area (effective range) in which the color information is effective. Is performed for each color information. In this way, the effective range for each color information is obtained. In the following description, a table for registering the effective range for each color information is referred to as a color table as needed.

The skin color registration unit 1060 saves the data calculated by the effective range calculation unit 1050 (the effective range of each color information) in the data storage unit 1020 (color table).
The image processing unit 1070 uses the color table registered by the skin color registration unit 1060 to extract a skin color region from the captured image. The image processing unit 1070 combines the information indicating the extraction result with the captured image to generate a combined image.
The image output unit 1080 displays the combined image generated by the image processing unit 1070 on the display unit 300.

<Processing procedure>
FIG. 3 is a flowchart showing an example of the flow of skin color registration processing in the information processing apparatus 1000.
In step S301, the image acquisition unit 1010 acquires the captured image captured by the imaging unit 100. The image acquisition unit 1010 is realized, for example, by using a video capture card that acquires the captured image obtained by the imaging unit 100.

  Next, in step S302, the skin color area setting unit 1030 specifies the skin color area in the captured image acquired by the image acquisition unit 1010. An area set by the user by operating the input unit 200 (for example, GUI (Graphical User Interface)) can be adopted as the skin color area. Further, a predefined area may be adopted as the skin color area. For example, at least one of displaying a frame on the screen (electronic viewfinder) of the image capturing unit 100 and displaying the captured image and the frame on the display unit 300 is performed so that the skin color appears in the frame. The user may image the subject with various compositions. In this case, the area inside the frame becomes the skin color area.

  Next, in step S303, the pixel information extraction unit 1040 performs color information extraction processing. In the present embodiment, the pixel information extraction unit 1040 extracts color information and pixel positions for all pixels from the captured image acquired by the image acquisition unit 1010. Then, the effective range calculation unit 1050 performs an effective range setting process. In the present embodiment, the effective range calculation unit 1050 calculates, for each color information, an image area (effective range) in which the color information is effective as a skin color, based on the distribution of the number of pixels for each color information. Further, the skin color registration unit 1060 creates a color table and stores it in the data storage unit 1020. As described above, the color information obtained by the effective range calculating unit 1050 and the effective range are registered in the color table in association with each other. Details of step S303 will be described later.

  Next, in step S304, the image processing unit 1070 performs color area extraction processing. In the present embodiment, the image processing unit 1070 uses the color table stored in the data storage unit 1020 to extract the skin color region from the captured image. For example, the image processing unit 1070 extracts color information and a pixel position for each pixel of the captured image, reads an effective range corresponding to the same color information as the extracted color information from the color table, and the extracted pixel position is effective. If it is within the range, the color at the position of the pixel is determined to be a skin color. The image processing unit 1070 combines the information indicating the result extracted as described above with the captured image to generate a combined image. The image output unit 1080 displays this composite image on the display unit 300. The extraction result of the skin color area can be shown by emphasizing the extraction area in the captured image. For example, red may be alpha-blended and superimposed on the pixel to be emphasized, or the pixel to be emphasized may be filled with black. The display method of the extraction result of the skin color area is not limited as long as it is a means capable of displaying the extraction result of the skin color area.

  Next, in step S305, when the user confirms the composite image, the user operates the input unit 200 to instruct whether or not to finish the skin color registration. The information processing apparatus 1000 determines whether to finish the skin color registration based on this instruction. As a result of this determination, when the skin color registration is to be ended, the processing according to the flowchart of FIG. 3 is ended.

On the other hand, when the skin color registration is not completed, the process returns to step S302, and the skin color area setting unit 1030 designates an area different from the skin color area designated so far as the skin color area. Then, the processes of steps S302 to S305 are repeated until it is determined that the skin color registration is completed.
If it is determined in step S305 that the skin color registration is not completed, the process may return to step S301 instead of step S302. In this case, the image capturing unit 100 obtains the captured image in a state in which the position of the subject (hand, etc.) is different from the position in the past captured image.

Next, an example of the details of the process of step S303 of FIG. 3 will be described with reference to the flowchart of FIG.
In step S401, the pixel information extraction unit 1040 performs pixel selection processing. In the present embodiment, the pixel information extraction unit 1040 selects one unselected pixel in the captured image acquired by the image acquisition unit 1010, and acquires the color information and pixel position of the pixel. As described above, in the present embodiment, the case where the distance from the center of the screen is the pixel position will be described as an example.

  Next, in step S402, the effective range calculation unit 1050 performs a region determination process. In the present embodiment, the effective range calculation unit 1050 determines that the pixel position of the pixel selected in step S401 is inside (skin color) or outside (non-skin) of the region (designated region) designated by the skin color region setting unit 1030 in step S302. Skin color).

  As a result of this determination, if the pixel position of the pixel selected in step S401 is within the designated area (skin color), the process proceeds to step S403. In step S403, the effective range calculation unit 1050 performs a relationship derivation process for the skin color area. In the present embodiment, the effective range calculation unit 1050 adds “1” to the number of pixels of the skin color (color information acquired in step S401) at the pixel position (distance from the center of the screen) acquired in step S401. Then, the process proceeds to step S405 described later.

  On the other hand, if the pixel position selected in step S401 is outside the designated area (non-skin color), the process proceeds to step S404. In step S404, the effective range calculation unit 1050 performs a relationship derivation process for a non-skin color area. In the present embodiment, the effective range calculation unit 1050 adds “1” to the number of pixels of the non-skin color (color information acquired in step S401) at the pixel position (distance from the center of the screen) acquired in step S401. Then, the process proceeds to step S405.

  In step S405, the pixel information extraction unit 1040 determines whether or not the processes of steps S401 to S404 have been completed for all the pixels of the captured image acquired by the image acquisition unit 1010. If the result of this determination is that processing has not been completed for all pixels of the captured image acquired by the image acquisition unit 1010, processing returns to step S401. Returning to step S401, the pixel information extraction unit 1040 selects an unselected pixel from the pixels of the captured image acquired by the image acquisition unit 1010. Then, the processes of steps S401 to S405 are repeated until the process is completed for all the pixels of the captured image acquired by the image acquisition unit 1010.

  When the processing is completed for all pixels of the captured image acquired by the image acquisition unit 1010 as described above, the process proceeds to step S406. In step S406, the effective range calculation unit 1050 performs an effective range calculation process. In the present embodiment, the effective range calculation unit 1050 determines an image effective as a skin color based on the number of pixels of skin color/non-skin color at each pixel position (distance from the center of the screen) obtained by the pixel information extraction unit 1040. The area (effective range) is calculated for each color information. Specifically, the effective range calculation unit 1050 compares the number of pixels of skin color/non-skin color at each pixel position, and the number of pixels of skin color corresponding to the distance from the center of the screen is larger than the number of pixels of non-skin color at the distance. The range with a large number may be set as the effective range.

FIG. 5 is a diagram showing an example of the distribution of the number of pixels of skin color (inside the area) and the distribution of the number of pixels of non-skin color (outside the area).
For example, as shown in FIG. 5, when the distribution of the number of pixels (the relationship between the number of pixels and the pixel position in the captured image) is represented by a skin color distribution 501 and a non-skin color distribution 502, the effective range calculation unit 1050 Will calculate an image area (effective range 503) that is effective as a skin color. Since such distributions 501 and 502 are obtained for each color information, the effective range 503 is also obtained for each color information.

However, it is not always necessary to calculate the effective area as described above. For example, a range of pixel positions (distance from the center of the screen) where the number of pixels of skin color is a certain number or more may be set as the effective range. In this case, only the skin color distribution 501 needs to be obtained. In addition, a range in which the number of pixels of skin color is a certain number or more in the range in which the number of pixels of skin color is larger than the number of pixels of non-skin color may be the effective range. Further, a range of pixel positions (distance from the center of the screen) where the number of non-skin-color pixels is a certain number or less may be set as the effective range. In this case, only the non-skin color distribution 502 need be obtained. In addition, a range in which the number of non-skin-color pixels is equal to or less than a certain number in a range in which the number of pixels of flesh-color is larger than the number of pixels of non-skin-color may be the effective range.
The effective range calculation unit 1050 calculates the effective range as described above for each color information.

  Next, in step S407, the skin color registration unit 1060 performs registration processing. In the present embodiment, the flesh color registration unit 1060 saves the effective range calculated by the effective range calculation unit 1050 for each color information in the data storage unit 1020. FIG. 6 is a diagram illustrating an example of the color table created at this time. The color information of the color table is associated with distance information indicating which area in the captured image is the effective range for the color information. For example, each piece of color information in the color table is associated with identification information that identifies the effective range of that color information. The identification information may be numerical data of distance or an identification number of the effective area. In the example shown in FIG. 6, the identification information of the effective range of the color information is the numerical value data of the distance. In FIG. 6, the effective range of the color information 601 is a range of 50 pixels from the center of the screen. The effective range of the color information 602 is a range from 50 pixels away from the center of the screen to 100 pixels away from the center of the screen. Registration of such an effective range in the color table is performed for each of all color information (cells in FIG. 6) in the captured image.

  When a subject region (a region such as a hand) is detected from a captured image in order to superimpose CG on the captured image, the information processing apparatus 1000 acquires color information and a pixel position for each pixel of the captured image, and the color The effective range corresponding to the information is read from the color table. When the pixel position of the pixel is included in the read valid range, the information processing apparatus 1000 determines that the color of the pixel position is a skin color. On the other hand, when the pixel position of the pixel is not included in the extracted effective range, the information processing apparatus 1000 determines that the color of the pixel position is not a skin color. The information processing apparatus 1000 extracts the area of the pixel determined to have the skin color in this way as the subject area (area of the hand or the like) in the captured image.

  As described above, in the present embodiment, the color information and the pixel position of each pixel of the captured image are derived, and whether each pixel is in the skin color region (whether or not it is a skin color) is determined. The distribution 501 of the number of pixels of skin color and the distribution 502 of the number of pixels of skin color are obtained for each color information, and the effective range 503 of each color information is calculated based on these distributions 501 and 502. Therefore, even if there is peripheral dimming, it is possible to accurately extract a subject region such as a hand from a real image without having a large number of color tables. Therefore, in mixed reality, it is possible to synthesize an actual subject region (such as a hand) in the virtual space without a sense of discomfort.

(Second embodiment)
Next, a second embodiment will be described. In the first embodiment, the case has been described as an example where the user determines whether or not to finish the skin color registration. On the other hand, in the present embodiment, an example will be described in which attention is paid to the size of the effective range of each color information and it is automatically determined whether or not the flesh color has been sufficiently registered. As described above, the present embodiment and the first embodiment are mainly different in the method of determining whether or not to finish the skin color registration. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment will be denoted by the same reference numerals as those in FIGS. 1 to 6, and the detailed description thereof will be omitted.

<Structure>
FIG. 7 is a block diagram showing an example of the arrangement of the image processing system according to this embodiment.
As shown in FIG. 7, the image processing system of the present embodiment is the image processing system of the first embodiment shown in FIG. 1 in which an image pickup position calculation unit 2010 is added to the information processing apparatus 2000. .

  The imaging position calculation unit 2010 acquires the effective range for each color information from the data storage unit 1020 (color table), and stores information indicating the position of the subject on the screen in the next imaging in the data storage unit 1020.

<Processing procedure>
FIG. 8 is a flowchart showing an example of the flow of processing of the imaging position calculation unit 2010. The flowchart of FIG. 8 is executed instead of step S304 of FIG.
In step S801, the imaging position calculation unit 2010 acquires the data (color information and effective range) of the color table registered by the skin color registration unit 1060 from the data storage unit 1020.
Next, in step S802, the imaging position calculation unit 2010 calculates the average and variance of the effective range of color information from the data of the color table acquired in step S801.

  Next, in step S803, the imaging position calculation unit 2010 determines whether the variance calculated in step S802 is less than or equal to a threshold value. If the result of this determination is that the variance is less than or equal to the threshold value, it is determined that the registration of the skin color has been sufficiently performed, and it is determined that the skin color registration is completed.

  On the other hand, if the variance is not less than or equal to the threshold, the process proceeds to step S804. In step S804, the imaging position calculation unit 2010 performs position derivation processing and output processing. In the present embodiment, the image pickup position calculation unit 2010 calculates an image pickup region in which the variance is reduced from the average and variance of the effective range of color information, and causes the display unit 300 to display information indicating the image pickup region. FIG. 9 is a diagram showing a display example of information indicating a new imaging region. For example, when the skin color is registered only near the center of the screen (captured image), the effective range of each color information varies. Therefore, as shown in FIG. 9, the imaging area is presented to the user so that imaging is performed at a position where the effective range is out of the average (the position shown by the broken line in FIG. 9).

Then, returning to step S301 in FIG. 3, the user instructs the imaging unit 100 to perform imaging in a state where the subject area (area such as the hand) is located in the imaging area displayed in step S804. As a result, the image capturing unit 100 captures an image of the subject area (such as the hand) with a composition in which the subject (such as the hand) is captured in the image capturing area displayed in step S804, and obtains a captured image.
According to the above, the user does not need to determine whether or not to finish the skin color registration, and more appropriate skin color registration can be performed.

(Third Embodiment)
Next, a third embodiment will be described. In the first and second embodiments, the case where the flesh color is registered based on the color information and the effective area has been described as an example. In this embodiment, in addition to the color information and the effective area, the effective depth from the camera in the physical space is used to register the skin color. As described above, the present embodiment and the first and second embodiments are configured and processed by including, as the data of the color table, the depth from the camera effective in the real space in addition to the color information and the effective area. Mainly different. Therefore, in the description of the present embodiment, the same parts as those in the first and second embodiments will be denoted by the same reference numerals as those in FIGS. 1 to 9, and the detailed description thereof will be omitted.

<Structure>
FIG. 10 is a block diagram showing an example of the arrangement of the image processing system according to this embodiment.
As shown in FIG. 10, the image processing system of the present exemplary embodiment is the image processing system of the first exemplary embodiment shown in FIG. 1 in which an effective depth acquisition unit 3010 is added to the information processing apparatus 3000. ..

  The effective depth acquisition unit 3010 obtains the skin color region of the captured image from the data (effective range for each color information) of the color table held in the data storage unit 1020, and the camera (imaging unit 100) for the obtained skin color region. Calculate the depth from. The depth is the distance between the camera and the subject in the direction perpendicular to the imaging surface of the camera. The effective depth acquisition unit 3010 calculates the depth from the camera for the skin color region from the plurality of captured images, and saves the range from the maximum depth to the minimum depth as the effective depth range in the data storage unit 1020 (color table). To do.

<Processing procedure>
FIG. 11 is a flowchart showing an example of the processing flow of the effective depth acquisition unit 3010. The flowchart of FIG. 11 is executed, for example, between step S303 and step S304 of FIG.
In step S1101, the effective depth acquisition unit 3010 performs area extraction processing. In the present embodiment, the effective depth acquisition unit 3010 uses the color table to extract the skin color region of the captured image acquired by the image acquisition unit 1010.

  Next, in step S1102, the effective depth acquisition unit 3010 performs a depth derivation process. In the present embodiment, the effective depth acquisition unit 3010 derives the depth from the camera (imaging unit 100) for the skin color region extracted in step S1101. The depth from the camera for the skin color region may be derived using, for example, the stereo method, or may be derived using the value of the depth sensor. When the stereo method is used, a stereo image is obtained by the image pickup unit 100. The method of deriving the depth from the camera for the skin color area is not limited as long as the method of detecting the distance between the camera (imaging unit 100) and the skin color area in the real space is used. The effective depth acquisition unit 3010 performs such depth for each of the plurality of captured images.

  Finally, in step S1103, the effective depth acquisition unit 3010 calculates a range between the maximum value and the minimum value of the depth from the camera for the skin color area derived from the plurality of captured images as the effective depth range, and stores the data. It is stored in the section 1020.

  When detecting a subject region (a region such as a hand) from the captured image in order to superimpose the CG on the captured image, the information processing apparatus 1000 acquires color information, a pixel position, and a depth for each pixel of the captured image. Then, the information processing apparatus 1000 reads from the color table an effective range (an image area where the color information is effective as a skin color) and an effective depth range corresponding to the acquired color information. When the pixel position of the pixel is included in the read effective range and the depth at the position of the pixel is included in the read effective depth range, the information processing apparatus 1000 determines the position of the pixel. The color is determined to be a skin color. On the other hand, if not, the information processing apparatus 1000 determines that the color at the position of the pixel is not a skin color. The information processing apparatus 1000 extracts the area of the pixel determined to have the flesh color in this way as a subject area (area such as a hand) in the captured image.

  As described above, in the present embodiment, in addition to the effective range (image area effective as a skin color) corresponding to each color information, the effective depth range is registered, and the subject area is extracted based on both of these. Therefore, it is possible to more reliably extract only the skin color in the subject area (the area such as the hand) registered when the skin color is registered, and it is possible to further reduce the false detection rate of the subject area.

(Fourth Embodiment)
Next, a fourth embodiment will be described. In the first to third embodiments, the case where the effective range for each color information is calculated based on the set skin color area has been described as an example. On the other hand, in this embodiment, color information associated with a valid range is acquired in advance, and interpolation processing is performed on the color information to generate new color information associated with a new valid range. The case will be described as an example. As described above, the present embodiment and the first to third embodiments are mainly different in that color information associated with the effective range in advance is acquired and that interpolation processing is performed on the color information. Therefore, in the description of the present embodiment, the same parts as those in the first to third embodiments will be denoted by the same reference numerals as those in FIGS. 1 to 11, and the detailed description thereof will be omitted.

<Structure>
FIG. 12 is a block diagram showing an example of the arrangement of the image processing system according to this embodiment. In FIG. 12, the color information interpolating unit 4010 generates new color information associated with a new effective range by performing an interpolation process on the color information associated with the effective range in advance. The effective range calculation unit 1210 has a different role from the effective range calculation unit 1050 of the first to third embodiments. The effective range calculation unit 1050 in the first to third embodiments calculates the effective range of each color information from the color information extracted by the pixel information extraction unit 1040 and the pixel position. On the other hand, the effective range calculation unit 1210 in the present embodiment has a role of presetting or calculating the effective range of the color information extracted by the pixel information extraction unit 1040.

<Processing procedure>
FIG. 13 is a flowchart showing an example of the flow of skin color registration processing in the information processing device 4000. The flowchart of FIG. 13 is executed instead of, for example, steps S301 to S304 of FIG.
In step S1301, the color information interpolation unit 4010 acquires the color information associated with the effective range. An example of a method of generating the acquired color information will be described later.
Next, in step S1302, the color information interpolation unit 4010 performs interpolation processing on the color information acquired in step S1301 to generate new color information associated with the new effective range. The interpolation process will be described later.

FIG. 14 is a flowchart showing an example of the flow of a generation process regarding the color information acquired in step S1301.
In step S1401, the effective range calculation unit 1210 sets (acquires or calculates) the effective range of the color information acquired in step S1405 described later. For example, the pixel position in the image is acquired from the input unit 200 as the effective range of the color information. Further, the pixel position indicating the effective range may be set in advance in the data storage unit 1020 and then acquired. In addition, setting information indicating the number of pixels of the image captured by the image capturing unit 100 and the approximate effective range such as “center” and “edge” is acquired from the input unit 200 and the data storage unit 1020, and the setting information is acquired. The pixel position may be calculated from the information as the effective range of the color information.

Next, in step S1402, the effective range calculation unit 1210 calculates the exposure of the image capturing unit 100 based on the effective range calculated in step S1401, and sets the exposure of the image capturing unit 100 via an image capturing control unit (not shown). .. The method of setting the exposure will be described later.
Next, in step S1403, the image acquisition unit 1010 acquires a captured image from the imaging unit 100 and stores the captured image in the data storage unit 1020, as in the first embodiment.

Next, in step S1404, the skin color area setting unit 1030 sets a skin color area, as in the first embodiment.
Next, in step S1405, the pixel information extraction unit 1040 extracts color information of pixels included in the skin color region set in step S1404, as in the first embodiment.
Next, in step S1406, it is determined whether or not the user finishes shooting, and the information is given to the information processing apparatus 4000 via the input unit 200. Alternatively, the number of times of shooting may be set in advance, and the information processing apparatus 4000 may determine whether or not to end shooting, depending on whether or not the number of times of shooting has been performed. If the result of this determination is that shooting should end, then the processing according to the flowchart of FIG. 14 ends.
On the other hand, when the photographing is not ended, the process returns to step S1401 and the processes of steps S1401 to S1406 are repeated until it is determined that the photographing is ended.

  Through the above processing, it is possible to generate a plurality of color information associated with the effective range. Note that the present embodiment is not limited to the case of generating color information according to the flowchart of FIG. 14, and color information may be generated by any method. For example, color information is extracted from an image taken so that the proper exposure is made at the center of the screen and an image taken so that the proper exposure is made at the edge of the screen, and the color information and the edge of the screen whose effective range is the center of the screen are extracted. It may be used as the color information for the effective range. Further, the color information generated outside the information processing device 4000 may be acquired via the input unit 200.

  FIG. 15 is a diagram showing the relationship between the distance from the center of the screen and the shading coefficient. Shading refers to brightness unevenness in the screen, and in the following description, the case where the brightness at each pixel position when the brightness at the screen center is 1.0 is used as the shading coefficient will be described as an example. Hereinafter, with reference to FIG. 15, an exposure calculation method by the effective range calculation unit 1050 will be described. First, the distance from the screen center at the center of the effective range set in step S1210 is calculated, and the shading coefficient s at that distance is calculated. Then, a value obtained by multiplying the proper exposure by s is calculated as the exposure of the image capturing unit 100. The above is the description of the exposure calculation method by the effective range calculation unit 1210.

  FIG. 16 is a diagram schematically showing an example of the effective range of color information. FIG. 16A shows the valid ranges 1601 to 1603 of color information set in step S1401 and the boundary portions 1604 and 1605 thereof. Further, FIG. 16B shows effective ranges 1606 to 1614 of new color information set by the color information interpolation unit 4010. In addition, FIG. 17 is a diagram schematically illustrating an example of the color information set in step S1401. FIG. 17A is a diagram showing the entire color information 1701 set in step S1401. FIG. 17B is a diagram showing color information 1702 corresponding to the effective range 1601 of the color information 1701. FIG. 17C is a diagram showing color information 1703 corresponding to the effective ranges 1602 and 1603.

  When the skin color region of the subject is detected using these color information, when the color information 1702 corresponding to the effective range 1601 and the color information 1703 corresponding to the effective ranges 1602 and 1603 are greatly different, the left and right sides of the boundary portions 1604 and 1605 are detected. The detection results are significantly different in the area. In order to solve this problem, in the present embodiment, interpolation processing is performed on the color information 1702 and 1703, so that the effective range divided into smaller effective ranges 1606 to 1614 in FIG. Generates color information corresponding to.

FIG. 18 is a flowchart showing an example of the flow of interpolation processing in the color information interpolation unit 4010.
In step S1801, the color information interpolation unit 4010 calculates the centroids 1704 and 1705 of the color information 1702 and 1703. For example, when the color information is information shown in the CbCr color space, the coordinates in the CbCr color space are calculated as the center of gravity of the color information.
Next, in step S1802, the color information interpolating unit 4010 arranges the colors of the color information 1702 and 1703 in order of decreasing distance from the centroids 1704 and 1705 calculated in step S1801.

Next, in step S1803, the color information interpolation unit 4010 refers to one effective range from the effective ranges 1606 to 1614. Here, the effective range referred to in step S1803 will be referred to as a reference effective range as necessary.
Next, in step S1804, the color information interpolation unit 4010 acquires the shading coefficient in the effective range referred to in step S1803. The shading coefficient can be acquired from the information indicating the relationship between the distance from the screen center and the shading coefficient described in step S1402 (see FIG. 15). The present embodiment is not limited to the case where a value faithful to the shading of the actual image is acquired as the shading coefficient. For example, the shading coefficient in the effective range corresponding to the center of the screen can be set to 1, and the shading coefficient in the effective range corresponding to the screen edge can be set to 0 (zero).

Next, in step S1805, the color information interpolation unit 4010 calculates the interpolation ratio of the color information 1702 and 1703 based on the shading coefficient acquired in step S1804. For example, when the shading coefficient of the reference effective range is s, the shading coefficient of the effective range 1601 is s1, and the shading coefficient of the effective ranges 1602 and 1603 is s2, the interpolation ratios r1 and r2 of the color information 1702 and 1703 are as follows: ) And equation (2).
r1=(s-s2)÷(s1-s2) (1)
r2=1-r1 (2)

Next, in step S1806, the color information interpolation unit 4010 combines the color information of the effective ranges 1601 and 1602 based on the interpolation ratios r1 and r2 calculated in step S1805, and corresponds to the reference effective range referred to in step S1803. Generate color information.
Next, in step S1807, the color information interpolation unit 4010 determines whether or not all effective ranges 1606 to 1614 have been referenced. As a result of this determination, if all the valid ranges 1606 to 1614 are not referred to, the process returns to step S1803 to refer to the valid ranges that have not yet been referred to among the valid ranges 1606 to 1614. Then, the processes of steps S1803 to S1807 are repeated until all the valid ranges 1606 to 1614 are referenced. When the processes of steps S1803 to S1806 are performed on all the valid ranges 1606 to 1614, the process according to the flowchart of FIG. 18 ends.

FIG. 19 is a flowchart showing an example of the flow of the combining process in step S1806.
In step S1901, the color information interpolating unit 4010 refers to one of the color information 1702 and 1703 before combining.
Next, in step S1902, the color information interpolation unit 4010 extracts a color from the color information referred to in step S1901 in accordance with the interpolation ratios r1 and r2 calculated in step S1805. Hereinafter, a case where the color information 1702 is referred to in step S1901 will be described as an example. The interpolation ratio of the color information 1702 is r1. Therefore, the color information interpolating unit 4010 extracts the number of colors r1 times the total number of colors included in the color information 1702 in the order arranged in step S1802 (from the center of gravity). The same applies to the processing when the color information 1703 is referred to in step S1902. However, r2 is used as the interpolation ratio instead of r1.

  By the above processing, it is possible to extract colors in order from important colors (close to the center of gravity) according to the interpolation ratios r1 and r2. The color extraction processing in this embodiment is not limited to the above. For example, when a color is extracted from certain color information, it may be extracted in order of distance from the center of gravity of the other color information instead of the center of gravity of the color information itself. When the color extraction processing is performed in this way, even if the color information before combining is separated from each other in the color space, it is possible to perform interpolation so as to connect them.

Next, in step S1903, the color information interpolating unit 4010 determines whether or not all of the pre-combination color information 1702 and 1703 has been referenced. As a result of this determination, if all of the pre-combination color information 1702 and 1703 are referenced, the process advances to step S1904.
On the other hand, if all of the pre-combination color information 1702 and 1703 has not been referred to, the process returns to step S1901 and the uncombined color information 1702 and 1703 is referred to. Then, the processes of steps S1901 to S1903 are repeated until all the pre-combination color information 1702 and 1703 have been referenced.
Next, in step S1904, the color information interpolating unit 4010 calculates the color information after composition by taking the logical sum of the colors extracted from the color information 1702 and 1703 in step S1902. By this processing, it is possible to generate the color information interpolated by the interpolation ratio based on the shading coefficient.

  Note that in step S1902, if the colors already extracted from other color information are extracted redundantly, the number of colors of the color information after combining becomes smaller than the number of colors of the color information 1702 and 1703 before combining. .. Therefore, it is desirable to extract the colors so that such overlapping does not occur. However, the present invention is not limited to the case of eliminating such color overlap, and the colors may be extracted in an overlapping manner.

  Also in this embodiment, the registration processing is performed by the skin color registration unit 1060. However, in the present embodiment, the effective range for each color information calculated by the color information interpolating unit 4010, not the effective range for each color information calculated by the effective range calculating unit 1050, is stored in the data storage unit 1020. ..

  In the present embodiment, the case where two pieces of color information are interpolated has been described, but the number of pieces of color information to be interpolated is not limited to this. By interpolating two or more color information, it can be applied to all cases in which more color information is generated. The present invention can also be applied to the case of generating new color information by extrapolation instead of interpolation.

As described above, in the present embodiment, the color information corresponding to the more finely divided area can be calculated by interpolating the color information acquired in advance. As a result, it is possible to reduce a decrease in the detection results that greatly differ between the left and right areas of the boundary of the effective range when the skin color area is detected.
Note that the configurations of the second and third embodiments can also be adopted in this embodiment.

  In each of the above embodiments, the case where the skin color is detected as the color of the subject has been described as an example. However, the color to be detected is appropriately determined according to the subject, and is not limited to the skin color.

  It should be noted that each of the above-described embodiments is merely an example of an embodiment for carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, first, software (computer program) that realizes the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the computer program.

  100: imaging unit, 200: input unit, 300: display unit, 1000: information processing device

Claims (16)

An acquisition unit that acquires an area designated in the captured image of the subject captured by the imaging unit;
For each color information extracted from each pixel of the captured image, the number of pixels in the captured image having the color information , the number of pixels inside the specified region, and the pixel having the color information. That is, based on at least one of the number of pixels outside the designated region, the color information is a process of extracting a region of a predetermined color from an image captured by the image capturing unit, An effective range setting means for setting an effective range that is effective as the predetermined color ;
When the color information of each pixel of the image captured by the image capturing unit is included in the effective range set for each of the color information by the effective range setting unit at the position of the pixel, the pixel is set in advance as described above. An information processing apparatus, comprising: a color area extraction unit that extracts a part of a predetermined color area. The effective range setting unit is, for each color information extracted from each pixel of the captured image, the number of pixels having the color information in the captured image, the number of pixels being inside the designated region. A certain first number and a second number, which is the number of pixels having the color information and located outside the designated region, are set to the center of the imaged images captured by the image capturing unit. Is calculated for each of a plurality of regions defined based on the distance from
The area in which the first number is larger than the second number among the plurality of areas is set as the valid range of the color information for each of the color information. The information processing device described. The effective range setting means further includes
Information indicating the position in the captured image regarding the pixels inside the specified region and the number of pixels, and information indicating the position in the captured image regarding the pixels outside the specified region and at least one of the relationship between the number of pixels, comprising the relationship deriving means to derive for each of the color information,
Based on the relationship derived by the relationship deriving means, the pre-Symbol valid range, information processing apparatus according to claim 2, wherein the deriving for each of the color information. The relationship deriving means, for the pixel that is inside the designated area, the relationship between the information and the number of pixels indicating the position in the captured image, for pixels that are outside of the designated area, the imaging The relationship between the information indicating the position in the image and the number of pixels is derived for each of the color information,
The effective range setting means, for the pixels inside the designated area, the relationship between the information indicating the position in the captured image and the number of pixels, and for the pixels outside the designated area, based on the result of comparing the relationship between the information and the number of pixels indicating the position in the captured image, the information processing according to claim 3, the pre-Symbol valid range, and wherein the deriving for each of the color information apparatus. An acquisition unit that acquires an area designated in the captured image of the subject captured by the imaging unit;
In the process of extracting a region of a predetermined color from the image captured by the image capturing unit, for each color information extracted from the pixels included in the designated region in the captured image. An effective range setting means for setting an effective range that is effective as the predetermined color,
A generating means for generating color information in the set the chromatic Kohan large number of new chromatic Kohan circumference than enclosed by said effective range setting means,
When the color information of each pixel of the image captured by the image capturing unit is included in the effective range set for each of the color information by the effective range setting unit at the position of the pixel, the pixel is set in advance as described above. A color area extracting means for extracting as a part of a predetermined color area ,
The generation means extracts one or more colors representing color information corresponding to the effective range from each of the effective ranges set by the effective range setting means, and based on the extracted colors, It characterized that you generate color information in the new coverage information processing apparatus. The generating means sets, from each of the effective ranges set by the effective range setting means, one or a plurality of colors representing color information corresponding to the effective range, which is set by the effective range setting means. The information processing apparatus according to claim 5, wherein a predetermined number of colors are extracted according to the position of the effective range . The generation unit determines, from each of the effective ranges set by the effective range setting unit, a total number of colors included in the color information as one or a plurality of colors representing color information corresponding to the effective range. 6. The information processing according to claim 5 , wherein the number of colors is extracted by multiplying by a ratio determined based on an index representing the brightness unevenness corresponding to the position of the effective range set by the effective range setting means. apparatus. Furthermore, on the basis of the chromatic Kohan circumference of each of the color information, a position deriving means for deriving a position in the screen of the object in the next captured,
Output means for outputting information indicating the position derived by the position deriving means,
The effective range setting means, when the information indicating the position by the pre-SL output means is output, setting the effective range for each color information extracted from the captured image of a subject captured by subsequently said image pickup means The information processing apparatus according to any one of claims 1 to 7, characterized in that. Wherein the position deriving unit on the basis of the mean and variance of the chromatic Kohan circumference of each of the color information, according to claim 8, characterized in that to derive the position in the screen of the object in the next captured Information processing device. 10. The display device further comprises display means for combining the captured image with the information indicating the area of the predetermined color extracted by the color area extraction means and displaying the combined image on a display device. The information processing apparatus according to any one of 1. Furthermore, for the color of the region where a predetermined extracted by the color region extracting means, the depth from the imaging means to derive for each of the plurality of the captured image, based on the derived depth, the predetermined was color extraction to a valid, the information processing apparatus according to any one of claims 1 to 10, characterized in that it comprises a depth deriving means for deriving a range of depth from the imaging means. The information processing device according to claim 1, wherein the predetermined color is a skin color. Furthermore, on the basis of the specified realm, the information processing apparatus according to any one of claims 1 to 12, characterized in that it comprises combining means for combining the pre-Symbol IMAGING images and CG. An acquisition step of acquiring an area designated in the captured image of the subject imaged by the imaging means ;
For each color information extracted from each pixel of the captured image, the number of pixels in the captured image having the color information , the number of pixels inside the specified region, and the pixel having the color information. That is, based on at least one of the number of pixels outside the designated region, the color information is a process of extracting a region of a predetermined color from an image captured by the image capturing unit, An effective range setting step of setting an effective range that is effective as the predetermined color ;
When the color information of each pixel of the image picked up by the image pickup unit is included in the effective range set for each of the color information by the effective range setting step at the position of the pixel, the pixel is set in advance as described above. An information processing method, comprising: a color area extracting step of extracting as a part of a predetermined color area. An acquisition step of acquiring an area designated in the captured image of the subject imaged by the imaging means;
In the process of extracting a region of a predetermined color from the image captured by the image capturing unit, for each color information extracted from the pixels included in the designated region in the captured image. An effective range setting step of setting an effective range that is effective as the predetermined color,
A generation step of generating color information of the new chromatic Kohan circumference greater number than the chromatic Kohan circumference set by the effective range setting step,
When the color information of each pixel of the image captured by the image capturing unit is included in the effective range set for each of the color information by the effective range setting step at the position of the pixel, the pixel is set to the pixel in advance. And a color area extracting step of extracting as a part of a predetermined color area ,
The generating step extracts one or more colors representing color information corresponding to the effective range from each of the effective ranges set by the effective range setting step, and based on the extracted color, information processing method characterized that you generate color information in the new coverage.   A program that causes a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 13.