JP7370769B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to image processing technology.

While the installation of imaging devices for surveillance purposes has become widespread, the importance of protecting the privacy of people appearing in images captured by the imaging devices is increasing. In Patent Document 1, a moving object region is extracted by comparing an input image and a background image, a foreground mask image is generated by converting the extracted moving object region into a predetermined pattern, and the foreground mask image is converted into a background image. A technique is described that generates an output image by superimposing images.

Generally, when generating a background image based on a captured image, if the person in the captured image remains still for a long time, such as by sitting on a chair, the image of the person may be included in the background image over time. . Even if a background image including the person is compared with a captured image including the person, the area of the person is not extracted as a moving object area, and a foreground mask image corresponding to the area of the person is not generated. Since a foreground mask image to be superimposed on a background image that includes the person is not generated, the privacy of the person cannot be protected in the output image that is generated as a result. In order to solve this problem, Patent Document 1 discloses the following contents. In other words, in the initial background image that is the first generated background image, a region where the person is considered to be stationary (suppression region) is specified, and the image in the suppression region of the initial background image is used as a background mask image, and the newly generated background An output image is generated by superimposing the background mask image on the image.

Japanese Patent Application Publication No. 2018-148368

However, in Patent Document 1, if a person is not included in the image of the suppression area in the initial background image and a person is included in the image of the area corresponding to the suppression area on the input image, the person in the initial background image is not included. An output image is generated in which the image of the suppression region is superimposed. That is, in Patent Document 1, although privacy can be protected, an output image that reflects the situation in the captured image is not generated.

Therefore, an object of the present invention is to provide a technique for generating an output image that reflects the situation in a captured image while protecting privacy.

In order to solve the above problems, an image processing device according to the present invention includes, for example, the following configuration. That is, a candidate background image used as at least a part of a background image for extracting a foreground object region from a captured image captured by an imaging means, the luminance value of which is closest to the average value of the luminance values of the captured image . an acquisition means for acquiring a candidate background image having an average value of , and generating an image of a partial region in the background image using the candidate background image acquired by the acquisition means, a generation means for executing a first generation process to generate an image based on the captured image; and an extraction means for extracting a region of the foreground object from the background image generated by the first generation process and the captured image. , a synthesis unit for generating an output image by synthesizing a mask image that abstracts the region of the foreground object extracted by the extraction unit and a predetermined image, and the generation unit is configured to generate a luminance value of the captured image If the difference value between the average value of the background image and the average value of the brightness values of the background image is greater than or equal to a first threshold value, the first generation process is executed, and if the difference value is less than the first threshold value. , the background image is generated using a captured image captured by the imaging unit without using the candidate background image acquired by the acquisition unit.

According to the present invention, it is possible to generate an output image that reflects the situation in a captured image while protecting privacy.

FIG. 1 is a diagram showing an example of a system configuration. FIG. 3 is a diagram showing functional blocks of image processing that generates an output image. FIG. 3 is a diagram for explaining image processing for generating a background image. 7 is a flowchart showing the flow of image processing for generating a background image. 3 is a flowchart showing the flow of image processing to generate an output image. FIG. 3 is a diagram showing functional blocks of image processing that generates an output image. FIG. 3 is a diagram showing the hardware configuration of each device.

Embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the configuration shown in the following embodiments is only an example, and the configuration is not limited to the illustrated configuration.

(Embodiment 1)
FIG. 1 is a diagram showing the system configuration in this embodiment. The system in this embodiment includes an image processing device 100, an imaging device 110, a recording device 120, and a display 130.

The image processing device 100, the imaging device 110, and the recording device 120 are interconnected via a network 140. The network 140 is realized, for example, by a plurality of routers, switches, cables, etc. that comply with communication standards such as ETHERNET (registered trademark).

Note that the network 140 may be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like.

The image processing apparatus 100 is realized, for example, by a personal computer or the like installed with a program for realizing image processing functions described below. Note that in this embodiment, the image processing device 100 and the imaging device 110 are connected via the network 140, but the image processing device 100 and the imaging device 110 may be provided in a single housing.

The imaging device 110 is a device that captures images. The imaging device 110 associates image data of the captured image, information on the imaging time at which the captured image was captured, and identification information that is information for identifying the imaging device 110, and transmits the data to the image processing device 100 via the network 140. or to an external device such as the recording device 120. Note that in the system according to this embodiment, there is one imaging device 110, but there may be a plurality of imaging devices 110.

The recording device 120 records image data of a captured image captured by the imaging device 110, information on the imaging time at which the captured image was captured, and identification information for identifying the imaging device 110 in association with each other. Then, in accordance with a request from the image processing apparatus 100, the recording apparatus 120 transmits the recorded data (captured image, identification information, etc.) to the image processing apparatus 100.

The display 130 is configured with an LCD (Liquid Crystal Display) or the like, and displays the results of image processing by the image processing device 100, images captured by the imaging device 110, and the like. The display 130 is connected to the image processing device 100 via a display cable compliant with communication standards such as HDMI (registered trademark) (High Definition Multimedia Interface).

Further, the display 130 functions as a display means, and displays a captured image captured by the imaging device 110, a result of image processing to be described later, and the like. Note that at least any two or all of the display 130, the image processing device 100, and the recording device 120 may be provided in a single housing.

Note that the results of image processing by the image processing device 100 and the captured images captured by the imaging device 110 are not limited to the display 130 connected to the image processing device 100 via a display cable, but can be displayed on external devices such as the following. It may be displayed on a display included in the device. That is, it may be displayed on a display of a mobile device such as a smartphone or a tablet terminal connected via the network 140.

Next, image processing of the image processing apparatus 100 according to the present embodiment will be described with reference to functional blocks of the image processing apparatus 100 according to the present embodiment shown in FIG. In this embodiment, each function shown in FIG. 2 is realized as follows using a ROM (Read Only Memory) 720 and a CPU (Central Processing Unit) 700, which will be described later with reference to FIG. shall be carried out. Each function shown in FIG. 2 is realized by the CPU 700 of the image processing apparatus 100 executing a computer program stored in the ROM 720 of the image processing apparatus 100.

The communication unit 200 can be realized by an I/F (Interface) 740, which will be described later with reference to FIG. 7, and communicates with the imaging device 110 and the recording device 120 via the network 140. The communication unit 200 receives, for example, image data of a captured image captured by the imaging device 110, and transmits a control command for controlling the imaging device 110 to the imaging device 110. Note that the control command is, for example, a command that instructs the imaging device 110 to capture an image, or a transmission request that requests to transmit a captured image, and specifies the imaging time of the captured image to be transmitted. This includes commands for sending requests, including . Upon receiving the transmission request command, the imaging device 110 transmits image data of a captured image corresponding to the imaging time included in the transmission request to the image processing device 100.

The storage unit 201 can be realized by a RAM (Random Access Memory) 710, an HDD (Hard Disk Drive) 730, etc., which will be described later with reference to FIG. 7, and stores (retains) captured images acquired by the communication unit 200, for example.

The operation reception unit 202 receives operations performed by a user via an input device (not shown) such as a keyboard or a mouse. For example, the operation accepting unit 202 accepts an operation that specifies a partial area on the background image in which the background image is not generated (updated) based on the captured image.

The background management unit 203 stores (retains) a plurality of candidate background images that are used as at least part of a background image for extracting a foreground object region from a captured image. Further, when the operation reception unit 202 receives a user operation specifying a candidate background image, the background management unit 203 generates a plurality of candidate background images from the candidate background image by converting the brightness of the candidate background image. It may also be retained. Further, when the operation reception unit 202 receives a user operation specifying a plurality of candidate background images, the background management unit 203 may hold the plurality of candidate background images. Note that the candidate background image is an image that does not include a specific object (person) to be hidden.

The setting unit 204 sets, for the background image, a partial area in which the background image is not generated (updated) based on the captured image. The setting unit 204 in this embodiment sets a partial area to the background image based on the user's operation for specifying a partial area received by the operation receiving unit 202.

The selection unit 205 selects (acquires) a candidate background image to be used for generating a background image from among the plurality of candidate background images held by the background management unit 203, based on the brightness of the captured image currently being processed. The selection unit 205 according to the present embodiment selects (obtains) candidate background images by, for example, performing the following process. That is, the selection unit 205 executes a process for calculating the average value of the brightness values of each pixel of the candidate background images for each of the plurality of candidate background images. Furthermore, the selection unit 205 calculates the average value of the brightness values of each pixel of the captured image currently being processed. Then, the selection unit 205 compares the average brightness value of the captured image to be processed with each of the average brightness values of the candidate background images, and selects the brightness value of the captured image from among the plurality of candidate background images. A candidate background image having an average value of brightness values closest to the average value is selected (obtained). That is, the selection unit 205 selects (obtains) a candidate background image that is close to the situation of the captured image currently being processed from a plurality of candidate background images.

The background generation unit 206 generates an image of a partial area in the background image using the candidate background image acquired by the selection unit 205, and generates an image of an area other than the partial area in the background image based on the captured image. Execute generation processing. Furthermore, the background generation unit 206 executes a second generation process in which the candidate background image acquired by the selection unit 205 is used as a background image. That is, the second generation process is a process in which the candidate background image acquired by the selection unit 205 is set as the latest background image. Note that a detailed explanation of the background generation unit 206 will be given later.

The extraction unit 207 uses the background image generated by the first generation process or the second generation process in the background generation unit 206 to extract the region of the foreground object from the captured image currently being processed. Specifically, the extraction unit 207 according to the present embodiment compares the captured image to be processed with the background image generated by the first generation process or the second generation process in the background generation unit 206. (background subtraction method), the area of the foreground object is extracted from the captured image to be processed.

Note that as a method for extracting a foreground object region by the extraction unit 207 according to this embodiment, the following process is executed. That is, the extraction unit 207 generates a binary image in which the foreground area indicating the area of difference with respect to the generated background image is set to "1" and the other areas are set to "0". At this time, the extraction unit 207 calculates the difference value between each pixel of the background image and each pixel of the captured image to be processed, and sets "1" representing the foreground region to the pixel for which the calculated difference value is equal to or greater than the threshold value. A binary image is generated by setting "0" when the value is less than the threshold value.

The synthesis unit 208 generates a mask image by abstracting (anonymizing) the region of the foreground object extracted by the extraction unit 207, and generates an output image by synthesizing the mask image and a predetermined image. The combining unit 208 according to the present embodiment combines, for example, the generated mask image and the candidate background image acquired by the selecting unit 205. Further, the synthesizing unit 208 may, for example, synthesize the generated mask image and the background image generated by the first generation process or the second generation process.

Note that as the process of abstracting (anonymizing) the region of the foreground object by the combining unit 208, the combining unit 208 executes the following process, for example. That is, the synthesis unit 208 generates a mask image that is abstracted (anonymized) by filling the foreground object region extracted from the captured image with an arbitrary color (RGB values). Note that the mask image in which the region of the foreground object is abstracted (anonymized) may be, for example, a mosaic image in which the region is subjected to mosaic processing, a blurred image in which the region is subjected to blurring processing, or the like.

The output control unit 209 outputs the output image generated by the composition unit 208 to an external device (recording device 120, display 130, etc.). At this time, the output control unit 209 causes the display 130 to display the output image generated by the composition unit 208, for example.

Next, image processing for generating an output image with privacy protection performed by the image processing apparatus 100 according to the present embodiment will be described in more detail with reference to FIGS. 3 and 4.

FIG. 3 is a diagram for explaining image processing according to this embodiment. First, the process of generating an output image will be described with reference to FIGS. 3(a) to 3(c). A captured image 300a shown in FIG. 3A is a captured image captured by the imaging device 110, in which a person 301 is lying on a bed. Furthermore, a background image 300b shown in FIG. 3(b) is a background image that does not include a person to be hidden. The extraction unit 207 according to the present embodiment extracts the region of the person 301 included in the captured image 300a as a foreground object region by comparing the captured image 300a and the background image 300b. Then, the synthesizing unit 208 generates an output image 300c by synthesizing the mask image 302 obtained by anonymizing the extracted foreground object region with the background image 300b.

As a conventional technique, there is a process of generating a background image based on a plurality of captured images. For example, there is a technique that generates a background image by calculating the average value of each pixel value of captured images of the most recent N frames (N is an integer). Furthermore, there is a technique in which a moving object is detected from a captured image using inter-frame differences, etc., and a background image is generated (updated) using an image of an area other than the moving object in the captured image. In these technologies, for example, when there is a person who remains stationary for a certain period of time, such as a person lying on a bed reflected in the captured image 300a, the image of the person becomes the background, and a background image such as the background image 300d is generated. may be done. At this time, when the captured image 300a and the background image 300d are compared, the person 301 is not extracted as a foreground region, so a mask image corresponding to the person 301 is not generated. That is, since a mask image to be combined with the background image 300d is not generated, the person 301 is displayed in the resulting output image, and privacy protection is no longer achieved.

Therefore, in the image processing according to the present embodiment, a process of generating a background image by the above-described first generation process or second generation process is executed. First, the first generation process will be explained here.

In this embodiment, the output control unit 209 causes the display 130 to display, for example, the first generated (set) background image. Then, the operation reception unit 202 receives a user operation that specifies an area in which a person is expected to remain stationary for a certain period of time in the displayed background image, and the setting unit 204 receives a user operation based on the user operation received by the operation reception unit 202. Set a partial area for the background image. A partial area 303 set by the setting unit 204 is shown in the background image 300e shown in FIG. 3(e).

In the first generation process, images of areas other than the partial area 303 in the background image 300e are generated (updated) based on captured images input from time to time. For example, an image of a region other than the partial region 303 in the background image 300e is generated (updated) by calculating the average value of each pixel of the image corresponding to the region in the captured images of the most recent N frames. On the other hand, in the first generation process, an image of a partial area in the background image 300e is generated (updated) using a candidate background image acquired by the selection unit 205 as being close to the situation of the captured image currently being processed. For example, an image of a partial region in the background image 300e is replaced with an image of a region corresponding to the partial region in the candidate background image acquired by the selection unit 205.

Here, the effect of generating a background image by the first generation process will be explained. First, after the captured image 300a is captured, as the illumination of the area to be imaged is switched from on to off, a captured image 300f in which the area to be imaged becomes dark as shown in FIG. 3(f) is captured. Assume that At this time, instead of the first generation process described above, if a process is executed in which images of areas other than the partial area 303 in the background image are generated (updated) based on the captured image, but images of the partial area 303 are not generated (updated), The following issues arise. That is, after the captured image 300f is captured, if the illumination continues to be off and images similar to the captured image 300f continue to be captured, images of areas other than the partial area 303 are generated (updated) based on the captured image. Therefore, a background image 300g as shown in FIG. 3(g) is generated. At this time, since the image in the partial area 303 is not generated (updated) based on the captured image, the image in the partial area 303 shown in the background image 300g has not changed from the background image 300e. In this case, the extraction unit 207 compares the background image 300g and the captured image 300f, and extracts a region corresponding to the partial region 303 in the captured image 300f as a region of the foreground object. As a result, the synthesis unit 208 generates an output image 300h as shown in FIG. 3(h), and an output image in which it is impossible to determine whether a person is present or not is generated.

However, in the first generation process according to the present embodiment, as described above, the image of the partial area 303 in the background image 300e is generated (updated) using the candidate background image acquired by the selection unit 205. Specifically, first, a candidate background image 300k having a brightness close to that of the captured image 300f is acquired from among a plurality of candidate background images held in advance. Then, the image of the partial area 303 on the background image 300e is updated (generated) using the image of the area corresponding to the partial area 303 in the selected candidate background image 300k, so that the background shown in FIG. Image 300i is generated. In this case, the extraction unit 207 can extract the person 301 in the captured image 300f as a foreground object region by comparing the captured image 300f and the background image 300i. Then, the synthesis unit 308 generates a mask image 305 in which the region of the foreground object extracted by the extraction unit 207 is abstracted (anonymized). Further, the combining unit 308 can generate an output image 300j as shown in FIG. 3(j) by combining the generated mask image 305 and the background image 300i. In the output image 300j, the presence of a person can be seen while protecting privacy so that the identity of the person in the captured image cannot be identified, that is, the situation of the captured image is reflected.

As explained above, by extracting the region of the foreground object from the captured image using the background image generated by the first generation process, the privacy of the person in the captured image is protected while the situation of the captured image is It is possible to generate an output image that reflects the

Next, the second generation process will be explained. The second generation process is a process in which the candidate background image acquired by the selection unit 205 is set as the latest background image without considering the partial area. For example, assume that the current background image is the background image 300b shown in FIG. 3(b), and that the captured image to be currently processed has shifted from the captured image 300a to the captured image 300f. At this time, the selection unit 205 acquires a candidate background image 300k from the plurality of candidate background images as a candidate background image that has a brightness close to the captured image currently being processed. In the second generation process, the candidate background image 300k acquired by the selection unit 205 is set as the latest background image. In this case, the extraction unit 207 can extract the person 301 in the captured image 300f as a foreground object region by comparing the captured image 300f with the candidate background image 300k, which is the current background image. Then, the synthesis unit 308 generates a mask image 305 in which the region of the foreground object extracted by the extraction unit 207 is abstracted (anonymized). Further, the combining unit 308 can generate an output image 300j as shown in FIG. 3(j) by combining the generated mask image 305 and the candidate background image 300k.

Note that the background generation unit 206 in this embodiment performs the first generation process or the first generation process based on the difference between the average value of the brightness values of the current background image and the average value of the brightness values of the captured image currently being processed. 2 Execute generation processing. Specifically, the background generation unit 206 calculates the difference value between the average value of the brightness values of each image of the current background image and the average value of the brightness values of each pixel of the captured image currently being processed. Then, for example, the background generation unit 206 executes the first generation process when the difference value is greater than or equal to the first threshold and less than the second threshold, and when the difference value is greater than or equal to the second threshold, the background generation unit 206 executes the second generation process. Execute. Note that the second threshold is a larger value than the first threshold. In the first generation process, images of regions other than the partial regions in the background image are generated (updated) based on a plurality of captured images inputted from time to time. However, if the brightness of the area to be imaged changes significantly, and the brightness value differs greatly between the currently processed image and the current background image, the brightness of the current image will be the same as that of the latest background image. It may take some time for the changes to be reflected. Therefore, the background generation unit 206 in this embodiment sets the difference between the average luminance value of the current background image and the average luminance value of the current captured image to be processed to be equal to or larger than a second threshold value, which is larger than the first threshold value. If so, the second generation process is executed. By doing so, it becomes possible to generate a background image that is close to the current situation of the captured image in a short time.

Next, image processing by the image processing apparatus 100 according to the present embodiment will be described in more detail with reference to the flows shown in FIGS. 4 and 5. By executing the process of the flow shown in FIG. 4, the first generation process or the second generation process can be executed. Furthermore, by executing the process shown in the flow shown in FIG. 5, it is possible to generate an output image using the generated background image.

Note that the processing of the flows shown in FIGS. 4 and 5 is executed by the functional blocks shown in FIG. shall be taken as a thing. Further, it is assumed that the processing of the flow shown in FIG. 4 is started or ended, for example, in accordance with an instruction from a user.

First, the processing of the flow shown in FIG. 4 will be explained. In S401, the background generation unit 206 acquires an initial background image that is an initial background image. As the initial background image, for example, a captured image selected by the user from among the plurality of captured images displayed on the display 130 by the output control unit 207 may be used as the initial background image. Furthermore, prior to the processing of the flow shown in FIG. 4, the background generation unit 206 generates an initial background image based on a plurality of captured images that are captured without a specific object (such as a person) to be hidden. It's okay. At this time, the background generation unit 206 may generate the initial background image by, for example, calculating the average of each pixel value of the plurality of captured images.

Next, in S402, the communication unit 200 acquires a captured image to be processed. Note that in this embodiment, the communication unit 200 acquires a captured image to be processed from the imaging device 110 via the network 140, but it may also acquire a captured image to be processed from the recording device 120, for example. .

Next, in S403, the selection unit 205 calculates the average value of the brightness values of the captured images. Specifically, the selection unit 205 calculates, for example, the average value of the brightness values of each pixel of the captured image currently being processed.

Next, in S404, the selection unit 205 acquires a candidate background image to be used for generating a background image from among the plurality of candidate background images held by the background management unit 202, based on the brightness of the captured image currently being processed. . Specifically, the selection unit 205 executes, for each of the plurality of candidate background images, a process of calculating the average value of the brightness values of each pixel of the candidate background images. Then, the selection unit 205 compares the average brightness value of the captured image to be processed with each of the average brightness values of the candidate background images, and selects the brightness value of the captured image from among the plurality of candidate background images. Obtain a candidate background image with an average value of brightness values closest to the average value. That is, the selection unit 205 acquires a candidate background image that is close to the situation of the captured image currently being processed from a plurality of candidate background images. Note that the average value of the brightness values of each of the candidate background images is calculated in advance, and the background management unit 203 stores the candidate background images in association with the average value of the brightness values of the candidate background images. You can also do this. Then, the selection unit 205 compares the average value of the captured image to be processed with each of the average values of the luminance values of the candidate background images, which are the luminance values held by the background management unit 203, and calculates the luminance value of the captured image. Alternatively, the candidate background image closest to the average value of .

Next, in S405, the background generation unit 206 calculates a difference value between the average value of the brightness values of the current background image and the average value of the brightness values of the captured image currently being processed. Specifically, the background generation unit 206 calculates the difference value between the average value of the brightness values of each image of the current background image and the average value of the brightness values of each pixel of the captured image currently being processed. When the process of S405 is executed for the first time after the process of the flow shown in FIG. Calculate the difference value.

In S406, the background generation unit 206 determines whether the difference value of the brightness values calculated in S405 is greater than or equal to the first threshold. If the difference value of the brightness values calculated in S405 is greater than or equal to the first threshold (Yes in S406), the process proceeds to S407, and if the difference value of the brightness values calculated in S405 is less than the first threshold value (No in S406), the process moves to S410. In S407, the background generation unit 206 determines whether the difference value between the brightness values calculated in S405 is equal to or greater than the second threshold. If the difference value of the brightness values calculated in S405 is greater than or equal to the second threshold (Yes in S407), the process moves to S408, and if the difference value of the brightness values calculated in S407 is less than the second threshold value. (No in S407), the process moves to S409.

In S408, the background generation unit 206 executes a second generation process in which the candidate background image acquired by the selection unit 205 is used as a background image. That is, the second generation process is a process in which the candidate background image acquired by the selection unit 205 is set as the latest background image. The background management unit 203 holds the background image generated by the second generation process as the latest background image.

In S409, the background generation unit 206 generates an image of a partial area in the background image using the candidate background image acquired by the selection unit 205, and generates an image of an area other than the partial area in the background image based on the captured image. Execute first generation processing to generate. The background management unit 203 holds the background image generated by the first generation process as the latest background image.

In S410, processing to generate an output image is executed. Here, the process of generating an output image will be described in more detail with reference to the flow shown in FIG. In S501 shown in FIG. 5, the extraction unit 207 uses the current background image to extract a foreground object region from the captured image currently being processed. Next, in S502, the synthesis unit 208 generates a mask image in which the foreground object region extracted by the extraction unit 207 is abstracted (anonymized). Note that the synthesis unit 208 according to the present embodiment generates a mask image that is abstracted (anonymized) by filling the region of the foreground object extracted from the captured image with an arbitrary color (RGB values). Note that the mask image in which the region of the foreground object is abstracted (anonymized) may be, for example, a mosaic image in which the region is subjected to mosaic processing, a blurred image in which the region is subjected to blurring processing, or the like. Next, in S503, the combining unit 208 generates an output image by combining the mask image generated in S502 and a predetermined image. Note that the composition unit 208 generates an output image by, for example, composition of the mask image generated in S502 and the candidate background image acquired by the selection unit 205 in S404. Note that the combining unit 208 combines, for example, the mask image generated in S502 and the latest background image (the background image generated by the first generation process, the background image generated by the second generation process, or the initial background image). The output image may be generated by combining the two.

After the processing in S503, the process moves to S411, and in S411, the output control unit 209 outputs the output image generated in S410 to an external device (display 130, recording device 120, etc.). Note that the output control unit 209 may display the output image generated in S410 on the display 130. Next, in S412, if there is no instruction to end (No in S412), the process moves to S402, and in S402, the communication unit 200 acquires a captured image to be processed next. On the other hand, if there is an instruction to end at S412 (Yes at S412), the process shown in FIG. 4 ends.

Note that in this embodiment, in S406, the background generation unit 206 determines whether the difference value of the brightness values calculated in S405 is equal to or greater than the first threshold, and if it is less than the first threshold, the process proceeds to S410. However, the present invention is not limited to this. For example, the background generation unit 206 executes the first generation process when the difference between the average luminance value of the captured image to be processed and the average luminance value of the current background image is equal to or greater than a predetermined threshold. However, if the difference value is less than a predetermined threshold, the following process may be performed. That is, if the difference value between the calculated average values of brightness values is less than a predetermined threshold, the background generation unit 206 may generate a background image using the captured image without using the acquired candidate background image. good. In this case, the background generation unit 206 generates the background image by, for example, using a plurality of captured images of the most recent N frames (N is an integer) and taking the average of the pixel values of each of the plurality of captured images. Good too. Note that at this time, the background generation unit 206 may use a known technique as long as it generates a background image using the captured image.

As described above, the image processing apparatus according to the present embodiment executes the following process as the first generation process. That is, in the first generation process, the image processing device generates (updates) an image of a partial region in the background image in which a person is considered to remain stationary for a long time using the candidate background image, and The image is generated (updated) using the captured image. Furthermore, in the second generation process, the image processing apparatus according to the present embodiment sets the candidate background image acquired by the selection unit 205 as the latest background image. By doing this, even if a person remains still for a long time, the foreground area corresponding to the person can be appropriately extracted, and as a result, the current situation of the captured image is appropriately reflected. It is possible to generate an output image that protects the privacy of the person concerned.

(Embodiment 2)
In the first embodiment, the setting unit 204 sets an area specified by the user as an area where a person is expected to remain stationary for a long time in the background image as a partial area. The setting unit 204 according to the present embodiment sets a partial region in the background image based on the detection result of detecting a specific object. Note that parts that are different from Embodiment 1 will be mainly described, and the same or equivalent components and processes as in Embodiment 1 will be given the same reference numerals, and redundant explanation will be omitted.

Image processing by the image processing apparatus 100 according to this embodiment will be described below with reference to FIG. In this embodiment, each function shown in FIG. 6 is realized as follows using a ROM 720 and a CPU 700, which will be described later with reference to FIG. Each function shown in FIG. 6 is realized by the CPU 700 of the image processing apparatus 100 executing a computer program stored in the ROM 720 of the image processing apparatus 100.

As shown in FIG. 6, the image processing apparatus 100 according to this embodiment further includes a detection unit 600 in addition to the functions shown in FIG. The detection unit 600 according to this embodiment detects a specific object from a background image or a captured image. The specific object is, for example, an object such as a chair or a bed on which a person is expected to remain stationary (position) for a long period of time. The detection unit 600 according to the present embodiment detects a specific object included in a background image or a captured image by performing processing such as pattern matching using a matching pattern (dictionary). Note that other known techniques may be used as a method for detecting a specific object, not just pattern matching. For example, the detection unit 600 detects a bed as a specific object from the captured image, and acquires information on the position and size of the bed. Specifically, when detecting a bed as a specific object, the detection unit 600 sets a circumscribed rectangle surrounding the bed, and calculates the coordinates of the upper left vertex of the circumscribed rectangle, the horizontal size and the vertical size of the circumscribed rectangle. Get the size of. It is assumed that the image size of the captured image and the background image are the same, and the coordinates indicating the positions on the captured image and the background image are XY coordinates with the upper left end point of each of the captured image and background image as the origin. do.

Then, the setting unit 204 according to the present embodiment sets a partial area for the background image based on the position and size of the specific object detected by the detection unit 600. For example, when the detection unit 600 detects a bed as a specific object, the setting unit 204 sets a circumscribed rectangle of the bed as a partial area. Note that the setting unit 204 may set an area obtained by expanding the circumscribed rectangle of the specific object detected by the detection unit 600 by a predetermined magnification as the partial area.

As described above, in the image processing apparatus 100 according to the present embodiment, a partial area is set for the background image based on the position of the specific object detected by the detection unit 600, so that the user who sets the partial area load can be reduced.

(Other embodiments)
Next, with reference to FIG. 7, the hardware configuration of the image processing apparatus 100 for realizing each function of each embodiment will be described. Note that although the hardware configuration of the image processing device 100 will be described in the following description, it is assumed that the recording device 120 and the imaging device 110 are also realized by a similar hardware configuration.

The image processing device 100 in this embodiment includes a CPU 700, a RAM 710, a ROM 720, an HDD 730, and an I/F 740.

The CPU 700 is a central processing unit that centrally controls the image processing apparatus 100. RAM 710 temporarily stores computer programs executed by CPU 700. Further, the RAM 710 provides a work area used by the CPU 700 when executing processing. Further, the RAM 710 functions, for example, as a frame memory or as a buffer memory.

The ROM 720 stores programs for the CPU 700 to control the image processing apparatus 100 and the like. The HDD 730 is a storage device that records image data and the like.

The I/F 740 communicates with external devices via the network 140 according to TCP/IP, HTTP, or the like.

In addition, in the description of each embodiment mentioned above, an example will be described in which the CPU 700 executes the processing, but at least a part of the processing of the CPU 700 may be performed by dedicated hardware. For example, the process of displaying a GUI (GRAPHICAL USER INTERFACE) or image data on the display 130 may be executed by a GPU (GRAPHICS PROCESSING UNIT). Further, the process of reading the program code from the ROM 720 and expanding it to the RAM 710 may be executed by a DMA (DIRECT MEMORY ACCESS) functioning as a transfer device.

Note that the present invention can also be implemented by a process in which one or more processors read and execute a program that implements one or more of the functions of the embodiments described above. The program may be supplied to a system or device having a processor via a network or a storage medium. The present invention can also be implemented by a circuit (eg, an ASIC) that implements one or more of the functions of the embodiments described above. Further, each part of the image processing apparatus 100 may be realized by the hardware shown in FIG. 7, or may be realized by software.

Note that another device may have one or more functions of the image processing device 100 according to each embodiment described above. For example, the imaging device 110 may have one or more functions of the image processing device 100 according to each embodiment. Note that the above-described embodiments may be combined, for example, any combination of the above-described embodiments may be implemented.

Although the present invention has been described above along with the embodiments, the above embodiments are merely examples of implementation of the present invention, and the technical scope of the present invention is interpreted to be limited by these embodiments. It's not a thing. That is, the present invention can be implemented in various forms without departing from its technical idea or main features. For example, a combination of each embodiment is also included in the disclosure content of this specification.

100 Image processing device 110 Imaging device 200 Communication unit 201 Storage unit 202 Operation reception unit 203 Background management unit 204 Setting unit 205 Selection unit 206 Background generation unit 207 Extraction unit 208 Combination unit 209 Output control unit 600 Detection unit

Claims (12)

A candidate background image used as at least a part of a background image for extracting a foreground object region from a captured image captured by an imaging means, the average of luminance values closest to the average of the luminance values of the captured image . an acquisition means for acquiring a candidate background image having a value ;
A first generation process of generating an image of a partial area in the background image using the candidate background image acquired by the acquisition means, and generating an image of an area other than the partial area in the background image based on the captured image. a generating means to execute;
Extracting means for extracting a region of a foreground object from the background image generated by the first generation process and the captured image;
composing means for generating an output image by composing a mask image that abstracts the region of the foreground object extracted by the extraction means and a predetermined image ;
The generation means executes the first generation process when the difference between the average brightness value of the captured image and the average brightness value of the background image is greater than or equal to a first threshold, and If the value is less than the first threshold, the background image is generated using a captured image captured by the imaging device without using the candidate background image obtained by the obtaining device. Processing equipment. Further comprising a holding means for holding a plurality of candidate background images used as at least a part of the background image,
The acquisition means is characterized in that the acquisition means acquires a candidate background image to be used for generating the background image from a plurality of candidate background images held by the holding means, based on the brightness value of the captured image taken by the imaging means. The image processing device according to claim 1. 3. The acquisition means acquires a candidate background image to be used for generating the background image based on the brightness value of the captured image and the brightness value of each of the plurality of candidate background images. The image processing device described. The generating means is configured to generate the above-mentioned image based on the difference value between the average brightness value of the captured image and the average brightness value of the background image, and when the difference value is equal to or more than the first threshold value and less than the second threshold value, When the first generation process is executed or the difference value is equal to or greater than the second threshold, the candidate background image acquired by the acquisition means is set as the background image in the partial area and areas other than the partial area. The image processing apparatus according to any one of claims 1 to 3 , wherein the image processing apparatus executes the second generation process. The image processing apparatus according to any one of claims 1 to 4, further comprising a setting means for setting the partial area for the background image. 6. The image processing apparatus according to claim 5 , wherein the setting unit sets the partial area for the background image based on a user's operation of specifying an area. further comprising detection means for detecting a specific object from the captured image,
6. The image processing apparatus according to claim 5 , wherein the setting means sets the partial area for the background image based on the position of a specific object detected by the detection means. The image processing device according to any one of claims 1 to 7 , wherein the combining unit generates an output image by combining the mask image and the candidate background image acquired by the acquiring unit. . The image processing apparatus according to any one of claims 1 to 7, wherein the combining unit generates an output image by combining the mask image and the background image generated by the generating unit. 10. The image processing apparatus according to claim 1, further comprising output control means for displaying the output image generated by the combining means on a display means. A candidate background image used as at least a part of a background image for extracting a foreground object region from a captured image captured by an imaging means, the average of luminance values closest to the average luminance value of the captured image . an obtaining step of obtaining a candidate background image having a value ;
A first generation process of generating an image of a partial area in the background image using the candidate background image acquired in the acquisition step, and generating an image of an area other than the partial area in the background image based on the captured image. A generation process to be performed;
an extraction step of extracting a foreground object region from the background image generated by the first generation process and the captured image;
a synthesis step of generating an output image by synthesizing a mask image that abstracts the region of the foreground object extracted in the extraction step and a predetermined image;
In the generation step, if the difference between the average brightness value of the captured image and the average brightness value of the background image is equal to or greater than a first threshold, the first generation process is executed, and the difference If the value is less than the first threshold, the background image is generated using the captured image captured by the imaging means without using the candidate background image obtained in the obtaining step. Processing method. A program for causing a computer to function as each means of the image processing apparatus according to claim 1 .

Images