JP2021101321A - Information processing device, method and program - Google Patents

Abstract

To precisely extract a foreground from an image.SOLUTION: An information processing device 123 includes an extracting unit 403 and an evaluating unit 404. The extracting unit 403 extracts a foreground candidate region 416 that is a candidate foreground on the basis of an input image 411 and of a background image 413 corresponding to the input image 411. The evaluating unit 404 determines that, in the foreground candidate region, a contour region in which the number of pixels is equal to or greater than a second threshold, the pixels having equal to or smaller than a first threshold of a difference between pixel values of a contour region corresponding to the contour of the foreground candidate region and pixel values of a region corresponding to the contour region in the background image, as a foreground.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing technique, and more particularly to a technique of extracting a main subject from an input image.

As a technique for extracting a main subject from an input image obtained by capturing an image with a camera, foreground background separation, which separates a specific person or object as the main subject as the foreground and the others as the background, is generally widely used. Foreground and background separation methods include a method using features such as edges and corners of a subject, a method using the difference between the pixel values of the background and the foreground, and a method of learning and separating the foreground or background properties.

In the case where a specific object or organism is extracted as the foreground against the background of a part of a building or industrial machine, the area where the foreground exists is restricted, or the color or texture of the area where the foreground appears is different from other areas. There are cases where they are different. When the region where the foreground exists is known in advance, the portion of the image corresponding to the region other than the region can be excluded from the processing target, and the portion of the image corresponding to the region can be treated as the processing target. Further, in Patent Document 1, a candidate object (foreground) is selected by using the similarity between the average pixel value of the peripheral region of the candidate object (foreground candidate region) and the average pixel value of the input image excluding the candidate object. The technology is disclosed.

JP-A-2017-204276

However, in the technique of Patent Document 1, when a background composed of a plurality of colors or a part of the foreground is overlapped with another foreground, a correct result cannot be obtained even if the average pixel value of the surrounding area of the candidate object is obtained, and the input is performed. There was a possibility that the foreground could not be extracted accurately from the image.

The present invention provides a technique for accurately extracting the foreground from an input image.

The image treatment apparatus according to one aspect of the present invention includes an extraction means for extracting a candidate region that is a candidate for the foreground based on an input image and a background image corresponding to the input image, and the candidate region in the candidate region. The number of pixels in the region where the difference between the pixel value of the contour region corresponding to the contour of the above and the pixel value of the region corresponding to the contour region in the background image is equal to or less than the first threshold value is equal to or greater than the second threshold value. It is characterized by having a determination means for determining the contour region as the foreground.

According to the present invention, the foreground can be accurately extracted from the input image.

The figure which shows the configuration example of the image processing system which generates a virtual viewpoint image Diagram showing an example of the internal configuration of the camera adapter The figure which shows the hardware configuration example of an information processing apparatus Block diagram showing an example of logical configuration of an information processing device Diagram showing an example of processing target Flowchart showing the flow of processing executed by the information processing device Flowchart showing the flow of processing executed by the information processing device

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. However, the components described in this embodiment are merely examples, and are not intended to limit the scope of the present invention to them. Moreover, not all of the combinations of components described in the embodiments are essential for the means for solving the problem, and various modifications and changes are possible.

<< Embodiment 1 >>
(System configuration)
FIG. 1 is a diagram showing a configuration example of an image processing system that generates a virtual viewpoint image. The image processing system 100 includes imaging modules 110a to 110z, a database (DB) 250, a server 270, a control device 300, a switching hub 180, and an end user terminal 190. That is, the image processing system 100 has three functional domains, an image acquisition domain, a data storage domain, and an image generation domain. The image acquisition domain includes the imaging modules 110a to 110z, the data storage domain includes the DB 250 and the server 270, and the image generation domain includes the control device 300 and the end user terminal 190.

The control device 300 manages the operating state and controls the parameter setting for each block constituting the image processing system 100 through the network. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard (registered trademark), or may be configured by combining an interconnect Infiniband, an industrial local area network, or the like. Further, the network is not limited to these, and other types of networks may be used.

First, an operation of transmitting 26 sets of captured images of the imaging modules 110a to 110z from the imaging module 110z to the server 270 will be described. The image pickup modules 110a to 110z each have one camera 112a to 112z. In the following, 26 sets of systems from the imaging modules 110a to 110z may not be distinguished and may be simply referred to as “imaging module 110”. Similarly, the devices in each imaging module 110 may be described as "camera 112" and "camera adapter 120". The number of imaging modules 110 is 26 sets, but this is just an example and is not limited to this.

The imaging modules 110a to 110z are connected by a daisy chain. This connection form has the effect of reducing the number of connection cables and labor saving in wiring work in increasing the resolution of captured images to 4K or 8K and increasing the capacity of image data due to the increase in frame rate. The connection form is arbitrary. For example, a star-type network configuration may be obtained in which the imaging modules 110a to 110z are connected to the switching hub 180 and data is transmitted / received between the imaging modules 110 via the switching hub 180.

In the present embodiment, each image pickup module 110 is composed of a camera 112 and a camera adapter 120, but is not limited thereto. For example, it may have a microphone, a pan head, and an external sensor. Further, in the present embodiment, the camera 112 and the camera adapter 120 are separated from each other, but they may be integrated in the same housing. The captured image obtained by the camera 112a in the image pickup module 110a is transmitted to the camera adapter 120b of the image pickup module 110b after the image processing described later is performed on the camera adapter 120a. Similarly, the imaging module 110b transmits the captured image obtained by the camera 112b to the imaging module 110c together with the captured image acquired from the imaging module 110a. By continuing such an operation, 26 sets of captured images are transmitted from the imaging module 110z to the switching hub 180, and then transmitted to the server 270.

In this embodiment, it is assumed that the foreground is evaluated in each camera adapter 120. However, the present invention is not limited to such an aspect, and the server 270 that has received 26 sets of captured images may be configured to generate silhouette images corresponding to the individual captured images.

(Camera adapter configuration)
Next, the details of the camera adapter 120 will be described. FIG. 2 is a functional block diagram showing an example of the internal configuration of the camera adapter 120. The camera adapter 120 includes a network adapter 121, a transmission unit 122, an information processing device 123, and a camera control unit 124.

The network adapter 121 performs data communication with another camera adapter 120, a server 270, and a control device 300. Further, for example, in accordance with the Ordinary Clock of the IEEE1588 standard, the time stamp of the data transmitted / received to / from the server 270 is saved and the time is synchronized with the server 270. In addition, time synchronization with a time server may be realized by another EtherAVB standard or an original protocol. In the present embodiment, a NIC (Network Interface Card) is used as the network adapter 121, but the present embodiment is not limited to this.

The transmission unit 122 controls the transmission of data to the switching hub 180 and the like via the network adapter 121. The transmission unit 122 has a function of performing compression by applying a predetermined compression method, compression rate, and frame rate to the transmitted / received data, and a function of decompressing the compressed data. It also has a function of determining the routing destination of the received data and the data processed by the information processing device 123, and a function of transmitting the data to the determined routing destination. It also has a function of creating a message for transferring image data to another camera adapter 120 or server 270. The message contains meta information of image data. This meta information includes a time code or sequence number at the time of sampling of image imaging, a data type, an identifier of the camera 112, and the like. The image data to be transmitted may be compressed. Further, a message is received from another camera adapter 120, and data information fragmented to a packet size specified by a transmission protocol is restored to image data according to the data type included in the message.

The information processing device 123 performs a process of generating an image of an object, which is a foreground, based on the image data obtained by the camera 112 imaged under the control of the camera control unit 124 and the initialization information. It also performs processing such as dynamic calibration. By generating the foreground by each of the plurality of camera adapters 120, the load on the image processing system 100 can be distributed. Dynamic calibration is a calibration performed during imaging, and is a color correction process for suppressing color variation between cameras and a blur correction process for stabilizing the position of an image against blur caused by camera vibration. (Electronic vibration isolation treatment) etc. are included.

The camera control unit 124 is connected to the camera 112 to control the camera 112, acquire a captured image, provide a synchronization signal, set a time, and the like. For control of the camera 112, for example, setting and reference of imaging parameters (number of pixels, color depth, frame rate, white balance setting, etc.), state information of the camera 112 (during imaging, stopping, synchronizing, error, etc.) Acquisition, start and stop of imaging, focus adjustment, etc.

(Hardware configuration of information processing device)
FIG. 3 is a diagram showing a hardware configuration example of the information processing device 123. The information processing device 123 includes a CPU 1231, a RAM 1232, a ROM 1233, a secondary storage device 1234, an input IF 1235, and an output IF 1236. The components are connected to each other via bus 1237 so that data can be transmitted and received.

The CPU 1231 executes a program stored in the ROM 1233 or the secondary storage device 1234 to control the information processing device 123 in an integrated manner. The RAM 1232 functions as a main memory when the CPU 1231 executes a program, and is used as a temporary storage area. The ROM 1233 stores the control program of the information processing device 123. The secondary storage device 1234 is a storage medium such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores image data, various programs, and the like.

The input IF1235 is an interface corresponding to standards such as USB and IEEE1394, and connects a camera 112, an external device (not shown), an input device (not shown), or the like to the bus 1237. The output IF1236 is an interface corresponding to standards such as DVI and HDMI (registered trademark), and connects a display device (not shown) and the CPU 1231.

The information processing device 123 may have one or more dedicated hardware or GPU (Graphics Processing Unit) different from the CPU 1231. In that case, the GPU or dedicated hardware may perform at least a part of the processing by the CPU 1231. Examples of dedicated hardware include ASICs (application specific integrated circuits) and DSPs (digital signal processors).

The hardware configuration example of the information processing apparatus 123 has been described above. The hardware configuration of the information processing device 123 is not limited to the above configuration. The CPU 1231 may read the program stored in the ROM 1233, the secondary storage device 1234, or the like into the RAM 1232 and execute the program, so that the CPU 1231 functions as each part shown in FIG. 4 to be described later. That is, the information processing apparatus 123 may realize each module shown in FIG. 4 as a software module.

(Logical configuration of information processing device)
FIG. 4 is a block diagram showing a logical configuration example of the information processing apparatus 123 according to the present embodiment. The information processing device 123 functions as the logical configuration shown in FIG. 4 by the CPU 1231 executing the program stored in the ROM 1233 by using the RAM 1232 as the work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 1231, and even if the information processing device 123 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 1231. Good.

In the information processing device 123 of FIG. 4, the background image obtained by capturing the image with the camera 112 in the absence of the foreground of a specific person or object which is the main subject, and the background color of the region from which the foreground is extracted are obtained. The next evaluation is performed using the method of specifying one or more. That is, the information processing apparatus 123 executes an evaluation for determining whether or not the foreground candidate region extracted from the input image is the foreground. In addition, the background image is an image obtained by taking an image without a foreground according to the time zone such as morning, noon, and evening, and the period from January to December, and only the background prepared in advance. It may be an image composed of. Although not specified in this embodiment, the background image is updated by triggering a change in the background, such as every fixed period or every fixed number of frames, in addition to the method of using a predetermined background image. It may be updated by dynamic background update.

The information processing device 123 has an input unit 401, a holding unit 402, an extraction unit 403, and an evaluation unit 404. The input unit 401 receives the image data input to the information processing device 123, including the input image 411 and the initialization information 412. The input image 411 is an image obtained by photographing the subject with the camera 112. The initialization information 412 is information used for the operation of the information processing apparatus 123. The initialization information 412 in the present embodiment is information used in the process shown in FIG. 6 described later, and includes an initial value of the background image 413, a background color 414, and a first threshold value for defining a range of the background color. , A second threshold value used by the evaluation unit 404 as a determination condition for the foreground. It is assumed that the first threshold value and the second threshold value are set in the evaluation unit 404 at the time of initialization of the information processing apparatus 123, respectively. The initial value of the background image 413 is a background image obtained in advance by a camera in a state where there is no foreground of a specific person or object which is the main subject. The background color 414 is a background color of a region for extracting the foreground from the background image 413, is one or more background colors specified in advance, and indicates a pixel value of the background. Part or all of the above-mentioned image data may be data received from the camera 112, data received from an external device, or data received from the secondary storage device 1234. The input image 411 received by the input unit 401 is output to the extraction unit 403. The initialization information 412 received by the input unit 401 is output to the holding unit 402.

The holding unit 402 holds the initialization information 412 input from the input unit 401. The initialization information 412 held in the holding unit 402 includes the initial value of the background image 413, the background color 414, the first threshold value that defines the range of the background color, and the first threshold value used by the evaluation unit 404 as the foreground determination condition. Includes two thresholds. When the background image is updated by the dynamic background update, the background image held by the holding unit 402 is replaced with the updated background image input via the input unit 401. In this case, the background color 414 held by the holding unit 402 is also replaced with the updated background color corresponding to the updated background image, similarly to the background image. The background image 413 held by the holding unit 402 is output to the extraction unit 403. The background color 414, the first threshold value, and the second threshold value held by the holding unit 402 are output to the evaluation unit 404.

The extraction unit 403 uses the input image 411 input from the input unit 401 and the background image 413 input from the holding unit 402 to extract a foreground candidate area 416 indicating a foreground candidate area from the input image 411. To do. The foreground candidate region 416 extracted by the extraction unit 403 is output to the evaluation unit 404.

The evaluation unit 404 extracts the contour from the foreground candidate region 416, acquires the coordinate information (pixels) of the contour of the foreground candidate region 416, and holds the pixels of the background image corresponding to the contour coordinate information as the contour background image 415. Obtained from the background image 413 held by the unit 402. The contour is also referred to as a contour area. Then, the evaluation unit 404 sets the background of the pixels in which the difference in pixel values between the contour background image 415 and the background color 414 is equal to or less than the first threshold value and the designated background color flag is turned on for each of the foreground candidate regions 416. It is counted as the number of color contour pixels. The evaluation unit 404 classifies the foreground candidate region corresponding to the contour background image in which the number of background color contour pixels is equal to or greater than the second threshold value into the foreground. The evaluation unit 404 may explicitly classify the foreground candidate region corresponding to the contour background image in which the number of background color contour pixels is less than the second threshold value into the background and treat it, or leave it as the foreground candidate region. You may handle it with a lower priority than the foreground.

Here, a method of classifying the foreground from the foreground candidate region by the evaluation unit 404 will be specifically described with reference to FIG. FIG. 5A is a diagram showing an example of a background image used in the information processing apparatus 123. FIG. 5B is a diagram showing an example of an input image input to the information processing apparatus 123. The background image of FIG. 5A corresponds to the input image of FIG. 5B. FIG. 5C is a diagram showing an example of a background color contour image corresponding to the foreground candidate region of FIG. 5B. A method of counting the number of background color contour pixels will be described with reference to FIGS. 5 (a) and 5 (b).

In the background image 500 of FIG. 5A, the background area 512 and the background area 510 shown in gray are areas having a color close to the background color designated as the target of foreground extraction, and the background area 511 shown in white is designated. It is assumed that the area is different from the background color. Examples of the background areas 510 and 512 include turf in a stadium. In the background area 512 and the background area 510, the difference between the designated background color (background pixel value) and each pixel value in the background area is equal to or less than the numerical value set as the first threshold value. It is assumed that the input image 504 shown in FIG. 5B includes three regions, a foreground candidate region 501, a foreground candidate region 502, and a foreground candidate region 503. The background color contour pixel 507 shown in FIG. 5C is composed of pixels corresponding to the contour in the foreground candidate region 501 included in the input image 504. Similarly, the background color contour pixel 508 shown in FIG. 5C is composed of pixels corresponding to the contour in the foreground candidate region 502 included in the input image 504. The background color contour pixel 509 shown in FIG. 5C is composed of pixels corresponding to the foreground candidate region 503 included in the input image 504.

The number of pixels of the background color contour pixel 507 corresponding to the foreground candidate area 501 is 4 because the total of 3 pixels in which the outline of the foreground candidate area 501 overlaps the background area 510 and 1 pixel overlapping the background area 512 is 4 pixels. It becomes. Similarly, the number of pixels of the background color contour pixel 508 corresponding to the foreground candidate region 502 is 8 because the contour of the foreground candidate region 502 does not overlap with the background region 512 and the number of pixels overlapping with the background region 510 is 8. Become. The number of pixels of the background color contour pixel 509 corresponding to the foreground candidate region 503 is 1 because the contour of the foreground candidate region 503 does not overlap with the background region 512 and the pixel overlapping with the background region 510 is one pixel.

(Operation in information processing device)
FIG. 6 is a flowchart showing a flow of processing executed by the information processing apparatus 123. The information processing apparatus 123 functions as each part shown in FIG. 4 by executing the program stored in the ROM 1233 by the CPU 1231 using the RAM 1232 as the work memory, and executes a series of processes shown in the flowchart of FIG. It is assumed that the input image 411 is input to the extraction unit 403 from the input unit 401. Further, it is assumed that the initialization information 412 is input to the holding unit 402 from the input unit 401. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 1231, and even if the information processing device 123 is configured so that a part or all of the processes is performed by one or a plurality of processing circuits other than the CPU 1231. Good. Hereinafter, the symbol "S" in the description of each process means a step in the flowchart. Hereinafter, the details of the operation in the information processing apparatus 123 will be described with reference to the flowchart of FIG. The processes of S601 to S606 are included in the process of extracting the foreground candidate from the input image.

In S601, the extraction unit 403 identifies the pixel of interest in the input image 411 input from the input unit 401. The pixel of interest may be specified in raster order or in any other order.

In S602, the extraction unit 403 determines whether or not the pixel of interest specified in S601 satisfies the background condition. For determining whether or not the attention pixel satisfies the background condition, it is known as background subtraction, which is a foreground background separation technique, using the attention pixel of the input image and the background image corresponding to the input image and acquired in advance. Method is used. When the extraction unit 403 obtains a determination result that the pixel of interest satisfies the background condition (YES in S602), the extraction unit 403 shifts the process to S613. In S613, the extraction unit 403 classifies the pixel of interest determined to satisfy the background condition in the process of S602 as the background. After the pixels of interest are classified into the background by the extraction unit 403, the process shifts to S606. On the other hand, when the extraction unit 403 obtains a determination result that the pixel of interest does not satisfy the background condition (NO in S602), the extraction unit 403 shifts the process to S603.

In S603, the extraction unit 403 classifies the pixel of interest that does not satisfy the background condition in the processing of S602 as a foreground candidate. It can be said that the pixel of interest that does not satisfy the background condition in the processing of S602 is extracted as a foreground candidate by the extraction unit 403.

In S604, the extraction unit 403 sets the difference between the pixel value of the background image corresponding to the pixel of interest classified as the foreground candidate in the process of S603 and one or more background colors (background pixel values) specified in advance. It is checked and it is determined whether or not the checked difference is equal to or less than the first threshold value. When the extraction unit 403 obtains a determination result that the difference between the pixel value of the background image corresponding to the pixel of interest and any of the background colors specified in advance is equal to or less than the first threshold value (YES in S604). The process shifts to S605. When the extraction unit 403 obtains a determination result that the difference between the pixel value of the background image corresponding to the pixel of interest and any of the background colors specified in advance is not equal to or less than the first threshold value (NO in S604). S605 is skipped and the process shifts to S606.

In S605, the extraction unit 403 sets the designated color background flag on for the pixels classified as foreground candidates in the processing of S603 and determined to be equal to or less than the first threshold value in the processing of S604. ..

In S606, the extraction unit 403 determines whether or not the foreground candidate extraction process has been completed for the entire input image. When the extraction unit 403 obtains a determination result that the foreground candidate extraction process has not been completed for the entire input image because there are unprocessed pixels that are not specified as the pixels of interest in the input image (NO in S606). , The process is returned to S601. Then, the extraction unit 403 executes the processes S601 to S605 and S613 on the unprocessed pixels that are not specified as the pixels of interest in the input image. On the other hand, the extraction unit 403 obtained a determination result that all the pixels of the input image were specified as the pixels of interest and classified as the foreground candidate or the background, and the extraction process of the foreground candidate was completed for the entire input image. If (YES in S606), the process shifts to S607.

In S607, the extraction unit 403 determines whether or not the input image to be processed has pixels classified as foreground candidates. When the extraction unit 403 obtains a determination result that the input image to be processed does not have pixels classified as foreground candidates, the extraction unit 403 ends this flow. On the other hand, when the extraction unit 403 obtains a determination result that the input image to be processed has pixels classified as foreground candidates, the extraction unit 403 shifts the processing to S608.

In S608, the extraction unit 403 groups the pixels extracted as the foreground candidate for each region to specify the foreground candidate region 416. Then, the extraction unit 403 outputs the specified foreground candidate area 416 to the evaluation unit 404. The evaluation unit 404 counts the pixels of the background image corresponding to the contour extracted from the foreground candidate area 416 as the background color contour pixels for which the designated background color flag is set to ON in S605, and counts the pixels for each foreground candidate area. The number of background color contour pixels is recorded in. The recording destination of the number of background color contour pixels in each foreground candidate region may be any functional unit as long as it can be read out by the evaluation unit 404 in the determination process of S610 described later. By executing the processes S601 to S608 in this way, the input image is classified into three regions, a first region, a second region, and a third region. That is, the first region is an region composed of pixels classified as foreground candidates and with the designated background color flag set to ON. The second area is an area composed of pixels that are classified as foreground candidates but the designated background color flag is not set to on. The third region is an region composed of pixels classified as a background.

The processes of S609 to S612 are included in the process of extracting the foreground from the foreground candidate region.

In S609, the evaluation unit 404 specifies the foreground candidate region of interest. The foreground candidate area of interest may be specified in order from the left end, or may be specified in any other order.

In S610, the evaluation unit 404 indicates whether the number of background color contour pixels recorded in S608 (the number of background color contour pixels in the foreground candidate region of interest) corresponding to the foreground candidate region of interest specified in S609 is equal to or greater than the second threshold value. Judge whether or not. When the evaluation unit 404 obtains a determination result that the number of background color contour pixels in the foreground candidate region of interest is not equal to or greater than the second threshold value (NO in S610), the evaluation unit 404 skips S611 and shifts the process to S612. On the other hand, when the evaluation unit 404 obtains a determination result that the number of background color contour pixels in the foreground candidate region of interest is equal to or greater than the second threshold value (YES in S610), the evaluation unit 404 shifts the process to S611.

In S611, the evaluation unit 404 classifies the foreground candidate region of interest, which is determined by the process of S610 to have the number of background color contour pixels equal to or greater than the second threshold value, into the foreground. It can be said that the foreground candidate region of interest, for which the number of background color contour pixels is determined to be equal to or greater than the second threshold value in the processing of S610, is extracted as the foreground by the extraction unit 403.

In S612, the evaluation unit 404 determines whether or not the foreground extraction process, which is a check for whether or not the foreground is foreground, has been completed for all the foreground candidate regions. When the evaluation unit 404 obtains a determination result that the check for the foreground has not been completed for all the foreground candidate regions (NO in S612), the evaluation unit 404 returns the process to S609. Then, the evaluation unit 404 executes the processes S609 to S612 on the unprocessed foreground candidate region that is not specified as the attention foreground candidate region in the input image. On the other hand, in the evaluation unit 404, all the foreground candidate areas are specified as the attention foreground candidate areas and classified into the foreground or others, and the foreground extraction process is performed for all the foreground candidate areas and the check is completed. When the determination result is obtained (YES in S612), this flow ends.

Here, the determination process of S610 will be specifically described using the foreground candidate region shown in FIG. 5 as an example. In S610, the foreground candidate regions 501, 502, and 503 shown in FIG. 5B are classified as follows according to whether the number of background color contour pixels of the corresponding contour is equal to or greater than the second threshold value. It will be.

When the second threshold value is set to 8, the foreground candidate region 502 corresponding to the background color contour pixel 508 having the background color contour pixel number of 8 is classified into the foreground and extracted. The foreground candidate areas 501 and 503 corresponding to the background color contour pixels 507 and 509, which are less than the second set value, are explicitly classified as the background by the evaluation unit 404, or are treated as the foreground candidate area. It is left behind and treated with a lower priority than the foreground.

When the second threshold value is set to 3, the foreground candidate areas 501 and 502 corresponding to the background color contour pixels 507 and 508 having the background color contour pixels 4 and 8 are classified into the foreground and extracted. It will be. The foreground candidate area 503 corresponding to the background color contour pixel 509 that is less than the second threshold value is explicitly classified as the background by the evaluation unit 404 and treated, or is left as the foreground candidate area and is more than the foreground. It is treated with a lower priority.

As described above, the information regarding the background image corresponding to the contour of the foreground candidate region, the foreground candidate region in which the difference in pixel values between the background colors specified in advance is equal to or less than the first threshold value, and the contour of the foreground candidate region Based on this, the following effect is obtained by determining that the foreground candidate area is the foreground. When the background of the area where the foreground appears has different characteristics from the other areas, the foreground can be extracted accurately. Even if the subject changes due to zooming or moving the camera, the process shown in FIG. 6 can be continuously executed as long as the characteristics of the area regarded as the background are the same. Also, even if there are multiple colors or the brightness changes due to changes in sunshine, the background can be excluded.

Further, by classifying the foreground candidate region in which the number of background color contour pixels is equal to or greater than the second threshold value into the foreground, the foreground candidate region having a large amount of noise and error can be excluded from the foreground.

When the background of the area where the foreground appears has different characteristics from the other areas, the foreground can be extracted accurately from the image. Even if the subject changes due to zooming or moving the camera, continuous operation is possible as long as the characteristics of the area regarded as the background are the same. Also, if there are multiple colors or the brightness changes due to changes in sunshine, the background can be excluded.

When treating the foreground candidate region having a certain size as the foreground, the foreground candidate region that is too large or too small compared to the shape to be extracted by using the method shown in this embodiment is excluded from the processing target. Foreground noise can be reduced. The size is decided to some extent based on the angle of view of the input image (the number of pixels in the width direction and the number of pixels in the height direction), the focal length from the camera to the foreground, the size and shape of the person, etc. Including the case where the foreground is calculated and determined to be mapped with a predetermined (desired) size. The foreground candidate region, which is too large compared to the shape to be extracted, does not exactly match the size of the shape to be extracted, but indicates a region larger than the size of the shape to be extracted plus a few percent margin. The foreground candidate region, which is too small compared to the shape to be extracted, is not an exact match with the size of the shape to be extracted, but is a region smaller than the size of the shape to be extracted minus a few percent margin.

<< Embodiment 2 >>
In the first embodiment, a case where the foreground candidate region in which the number of background color contour pixels is equal to or larger than the second threshold value is classified into the foreground and extracted has been described. In the present embodiment, a case where the foreground candidate region in which the ratio of the contour length of the contour of the foreground candidate region to the number of background color contour pixels is equal to or larger than the third threshold value is classified into the foreground and extracted will be described. The description of the parts common to the first embodiment will be omitted.

(Logical configuration of information processing device)
Next, the logical configuration of the information processing apparatus of the present embodiment is the same as the logical configuration of the information processing apparatus 123 of the first embodiment shown in FIG. 4 except for the evaluation unit 404, and the description thereof will be omitted.

The evaluation unit 404 of the present embodiment extracts the contour from the foreground candidate region 416, acquires the coordinate information (pixels) of the contour of the foreground candidate region 416, and plots the pixels of the background image corresponding to the coordinate information of the contour as the contour background. Obtained from the background image 413 held by the holding unit 402 as the image 415. Then, the evaluation unit 404 sets the background of the pixels in which the difference in pixel values between the contour background image 415 and the background color 414 is equal to or less than the first threshold value and the designated background color flag is turned on for each of the foreground candidate regions 416. It is counted as the number of color contour pixels. The evaluation unit 404 derives and records the ratio of the number of pixels of the contour (contour length) to the number of background color contour pixels corresponding to the contour for each foreground candidate region. The recording destination of the derived ratio may be any functional unit as long as it can be read by the evaluation unit 404 described later. The evaluation unit 404 determines whether or not the ratio of the contour length to the number of background color contour pixels is equal to or greater than the third threshold value. When the evaluation unit 404 obtains a determination result that the ratio of the contour length and the number of background color contour pixels is equal to or greater than the third threshold value, the evaluation unit 404 classifies the corresponding foreground candidate region into the foreground. The evaluation unit 404 may explicitly classify the foreground candidate region corresponding to the ratio in which the ratio of the contour length and the number of background color contour pixels is less than the third threshold value into the background and treat the foreground candidate. It may be left as an area and handled with a lower priority than the foreground.

(Operation in information processing device)
The processing by the information processing apparatus according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing a flow of processing executed by the information processing apparatus of the present embodiment. The information processing apparatus 123 functions as each part shown in FIG. 4 by executing the program stored in the ROM 1233 by the CPU 1231 using the RAM 1232 as the work memory, and executes a series of processes shown in the flowchart of FIG. 7. It is assumed that the input image 411 is input to the extraction unit 403 from the input unit 401. Further, it is assumed that the initialization information 412 is input to the holding unit 402 from the input unit 401. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 1231, and even if the information processing device 123 is configured so that a part or all of the processes is performed by one or a plurality of processing circuits other than the CPU 1231. Good. The same reference numerals are added to the steps having the same processing contents as those in the first embodiment. The processes of S601 to S606 are included in the process of extracting the foreground candidate from the input image.

In S601 to S607 of the present embodiment, the same processing as that of the first embodiment is performed.

Subsequently, in S708, the extraction unit 403 groups the pixels extracted as the foreground candidate for each region to specify the foreground candidate region 416. Then, the extraction unit 403 outputs the specified foreground candidate area 416 to the evaluation unit 404. The evaluation unit 404 uses the number of pixels (contour length) of the contour extracted from the foreground candidate area 416 and the pixels of the background image corresponding to the contour, and the pixels in which the designated background color flag is set to the background color contour pixels. The ratio to the number of background color contour pixels obtained by counting as is recorded for each foreground candidate region. The recording destination of the ratio may be any functional unit as long as it can be read by the evaluation unit 404.

The processes of S609, S710, and S611 to S612 are included in the process of extracting the foreground from the foreground candidate region.

Subsequently, in S609 of the present embodiment, the same processing as that of the first embodiment is performed.

Subsequently, in S710, the evaluation unit 404 determines whether or not the ratio of the contour length and the number of background color contour pixels recorded in S708, which corresponds to the foreground candidate region of interest specified in S609, is equal to or greater than the third threshold value. judge. When the evaluation unit 404 obtains a determination result that the ratio of the contour length and the number of background color contour pixels is not equal to or greater than the third threshold value (NO in S710), the evaluation unit 404 skips S611 and shifts the process to S612. On the other hand, when the evaluation unit 404 obtains a determination result that the ratio of the contour length and the number of background color contour pixels is equal to or greater than the third threshold value (YES in S710), the evaluation unit 404 shifts the process to S611.

Subsequently, in S611 and S612 of the present embodiment, the same processing as in the first embodiment is performed. In S612, the evaluation unit 404 ends this flow when the foreground extraction process is performed on all the foreground candidate regions and a determination result that the check is completed is obtained (YES in S612).

Here, the determination process of S710 will be specifically described using the foreground candidate region shown in FIG. 5 as an example. Before the processing of S710, it is assumed that the ratio of the contour length of each of the foreground candidate regions 501, 502, and 503 shown in FIG. 5B to the number of background color contour pixels is derived by the evaluation unit 404. The ratio of the contour length of the foreground candidate region derived by the evaluation unit 404 to the number of background color contour pixels is recorded in S708.

Since the contour length of the contour of the foreground candidate region 501 is 12 and the number of background color contour pixels corresponding to the contour of the foreground candidate region 501 is 4, the ratio of the contour length of the foreground candidate region 501 to the number of background color contour pixels is , 33.3%. Since the contour length of the contour of the foreground candidate region 502 is 12, and the number of background color contour pixels corresponding to the contour of the foreground candidate region 502 is 8, the ratio of the contour length of the foreground candidate region 502 to the number of background color contour pixels is , 66.6%. Since the contour length of the contour of the foreground candidate region 503 is 1 and the number of background color contour pixels corresponding to the contour of the foreground candidate region 503 is 1, the ratio of the contour length of the foreground candidate region 503 to the number of background color contour pixels is , 100% is derived.

In S710, the foreground candidate regions 501, 502, and 503 shown in FIG. 5B are as follows, depending on whether the ratio of the contour length of the corresponding contour to the number of background color contour pixels is equal to or greater than the third threshold value. It will be classified as. When the third threshold value is set to 50%, the foreground candidate areas 502 and 503 in which the ratio of the contour length of the foreground candidate region to the number of background color contour pixels is 66.6% and 100% are classified into the foreground. It will be extracted. The foreground candidate area 501 corresponding to the background color contour pixel 507 that is less than the third threshold value is explicitly classified as the background by the evaluation unit 404 and treated, or is left as the foreground candidate area and is more than the foreground. It is treated with a lower priority.

As described above, by filtering the foreground candidate area using the ratio of the contour length of the foreground candidate area and the number of background color contour pixels, the extraction range is limited to the specified background color area, and the foreground candidate is used. The foreground can be extracted accurately regardless of the size of the area. In addition, by using the ratio of the contour length to the number of background color contour pixels, even if the zoom is changed during imaging, the foreground can be extracted from the input image as before, without changing the foreground extraction conditions. Can be done.

[Other Embodiments]
In the first embodiment, the process of classifying the foreground candidate region in which the number of background color contour pixels is equal to or greater than the second threshold value into the foreground has been described. In the second embodiment, the process of classifying the foreground candidate region in which the ratio of the contour length and the number of background color contour pixels is equal to or more than the third threshold value into the foreground has been described. The foreground candidate region in which the number of background color contour pixels is equal to or greater than the second threshold value and the ratio of the contour length to the number of background color contour pixels is equal to or greater than the third threshold value may be classified as the foreground. That is, the first extraction process using the second threshold value may be performed, and the extraction result of the first extraction process may be subjected to the second extraction process using the third threshold value. Further, the second extraction process using the third threshold value may be performed, and the extraction result of the second extraction process may be subjected to the first extraction process using the second threshold value.

A case where the second extraction process is performed after the first extraction process will be specifically described by taking the foreground candidate region shown in FIG. 5 as an example. It is assumed that the second threshold value used in the first extraction process is set to 3. It is assumed that the third threshold value used in the second extraction process is set to 50%.

In the first extraction process, the foreground candidate regions 501, 502, and 503 shown in FIG. 5B are next depending on whether the number of background color contour pixels of the corresponding contour is 3 or more of the second threshold value. It will be classified as. The foreground candidate regions 501 and 502 corresponding to the background color contour pixels 507 and 508 having the number of background color contour pixels 4 and 8 are classified into the foreground and extracted.

Subsequently, in the second extraction process, each of the foreground candidate regions 501 and 502 depends on whether the ratio of the contour length of the corresponding foreground candidate region to the number of background color contour pixels is 50% or more of the third threshold value. , Will be classified as follows. Only the foreground candidate region 502 in which the ratio of the contour length of the foreground candidate region to the number of background color contour pixels is 66.6% is classified into the foreground and extracted.

The foreground candidate area 503 corresponding to the background color contour pixel 509 that does not satisfy the conditions of the second and third threshold values or more, and the background that satisfies the conditions of the second threshold value or more but does not satisfy the condition of the third threshold value or more. The foreground candidate region 501 corresponding to the color contour pixel 507 is treated as follows. That is, the foreground candidate areas 501 and 503 are explicitly classified and treated as the background by the evaluation unit 404, or are left as the foreground candidate area and are treated with a lower priority than the foreground. As a result, it is possible to reduce false detection of noise in the region where the background is desired to be extracted. That is, it is possible to reduce false detection of noise in a region that is not desired to be extracted as the foreground.

A case where the first extraction process is performed after the second extraction process will be specifically described by taking the foreground candidate region shown in FIG. 5 as an example. It is assumed that the third threshold value used in the second extraction process is set to 50%. It is assumed that the second threshold value used in the first extraction process is set to 3.

In the second extraction process, each foreground candidate region 501, 502, 503 depends on whether the ratio of the contour length of the corresponding foreground candidate region to the number of background color contour pixels is 50% or more of the third threshold value. It will be classified as follows. The foreground candidate areas 502 and 503 in which the ratio of the contour length of the foreground candidate region to the number of background color contour pixels is 66.6% and 100% are classified into the foreground and extracted.

Subsequently, in the first extraction process, the foreground candidate regions 502 and 503 shown in FIG. 5B depend on whether the number of background color contour pixels of the corresponding contour is 3 or more of the second threshold value. It will be classified as follows. The foreground candidate region 503 corresponding to the background color contour pixel 508 having the background color contour pixel number of 8 is classified into the foreground and extracted.

The foreground candidate region 501 corresponding to the background color contour pixels 507 that does not satisfy the conditions of the third and second threshold values or more, and the background that satisfies the conditions of the third threshold value or more but does not satisfy the conditions of the second threshold value or more. The foreground candidate region 503 corresponding to the color contour pixel 509 is treated as follows. That is, the foreground candidate areas 501 and 503 are explicitly classified and treated as the background by the evaluation unit 404, or are left as the foreground candidate area and are treated with a lower priority than the foreground. As a result, it is possible to reduce false detection of noise in the region where the background is desired to be extracted. That is, it is possible to reduce false detection of noise in a region that is not desired to be extracted as the foreground.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

123 Information processing device 403 Extraction unit 404 Evaluation unit

Claims (9)

An extraction means for extracting a candidate region that is a candidate for the foreground based on the input image and the background image corresponding to the input image.
In the candidate region, the number of pixels in which the difference between the pixel value of the contour region corresponding to the contour of the candidate region and the pixel value of the region corresponding to the contour region in the background image is equal to or less than the first threshold value is the second. A determination means for determining the contour region that is equal to or greater than the second threshold value as the foreground, and
An information processing device characterized by having. The determination means is such that the number of pixels in the contour region which is equal to or less than the first threshold value is equal to or greater than the second threshold value and the number of pixels in the contour region is equal to or less than the first threshold value. The information processing apparatus according to claim 1, wherein the contour region in which the ratio of the number of pixels to the number of pixels is equal to or greater than a third threshold value is determined to be the foreground. An extraction means for extracting a candidate region that is a candidate for the foreground based on the input image and the background image corresponding to the input image.
In the candidate region, the number of pixels in which the difference between the pixel value of the contour region corresponding to the contour of the candidate region and the pixel value of the region corresponding to the contour region in the background image is equal to or less than the first threshold value. A determination means for determining that the contour region in which the ratio of the contour region to the number of pixels is equal to or greater than the third threshold value is the foreground
An information processing device characterized by having. In the determination means, the ratio of the number of pixels in the contour region, which is equal to or less than the first threshold value, to the number of pixels in the contour region is equal to or greater than the third threshold value and equal to or less than the first threshold value. The information processing apparatus according to claim 3, wherein the contour region in which the number of pixels in the contour region is equal to or greater than a second threshold value is determined to be the foreground. The input image is a captured image obtained by imaging with an imaging means.
The information processing apparatus according to any one of claims 1 to 4, further comprising an acquisition means for acquiring the captured image. The information processing apparatus according to any one of claims 1 to 5, further comprising a holding means for holding the background image corresponding to the input image. An extraction step of extracting candidate regions that are candidates for the foreground based on the input image and the background image corresponding to the input image.
In the candidate region, the number of pixels in which the difference between the pixel value of the contour region corresponding to the contour of the candidate region and the pixel value of the region corresponding to the contour region in the background image is equal to or less than the first threshold value is the second. A determination step of determining the contour region that is equal to or greater than the second threshold value as the foreground, and
An information processing method characterized by having. An extraction step of extracting candidate regions that are candidates for the foreground based on the input image and the background image corresponding to the input image.
In the candidate region, the number of pixels in which the difference between the pixel value of the contour region corresponding to the contour of the candidate region and the pixel value of the region corresponding to the contour region in the background image is equal to or less than the first threshold value. A determination step of determining that the contour region in which the ratio of the contour region to the number of pixels is equal to or greater than the third threshold value is the foreground
An information processing method characterized by having. A program that causes a computer to function as each means of the information processing apparatus according to any one of claims 1 to 6.

Images