JP7418074B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to image processing technology.

In recent years, surveillance cameras have become widely installed. A system using such a surveillance camera is useful in that it can perform crime prevention, marketing analysis, and service improvement by specifying the movements of the photographed person. On the other hand, in such a system, it is important that the privacy of the imaged person is protected.

Patent Document 1 discloses a technique for superimposing a humanoid icon whose color varies depending on whether or not a person detected from the image is moving on a background image that has been captured in advance.

International Publication No. 2017/141454

However, in Patent Document 1, when an object that is not a person is detected to be a person, a humanoid icon corresponding to the object may be superimposed on the background image. That is, when an object that is not actually a person is detected as a person, an inappropriate output image may be generated in which a humanoid icon corresponding to the object is superimposed on a background image.

Therefore, an object of the present invention is to provide a technique that makes it possible to generate an image in which an appropriate area in a captured image is hidden.

An image processing device according to one aspect of the present invention includes an extraction unit that extracts a foreground region from a captured input image, a detection unit that detects a specific object from the input image, and a specific object that is detected by the detection unit. a setting means for setting a specific area, at least a part of which is a curved area, based on the position of the foreground area other than the specific area set by the setting means; generating means for generating an output image in which a silhouette image that abstracts the foreground region in the specific area set by the setting means is superimposed on the predetermined image without superimposing it on the image; and a determining means for determining, and the setting means sets a specific area whose horizontal width is reduced by a predetermined magnification for a person located in the crowded area determined by the determining means.

According to the present invention, it is possible to provide a technique that makes it possible to generate an image in which an appropriate area in a captured image is hidden.

FIG. 1 is a diagram showing an example of a system configuration. FIG. 2 is a diagram showing functional blocks of an image processing device. 3 is a flowchart showing the flow of processing for generating an output image. FIG. 3 is a diagram for explaining processing for generating an output image. FIG. 3 is a diagram for explaining the shape of a human region. FIG. 3 is a diagram for explaining processing for generating an output image. FIG. 2 is a diagram showing functional blocks of an image processing device. FIG. 3 is a diagram for explaining processing for generating an output image. FIG. 3 is a diagram for explaining processing for generating an output image. FIG. 2 is a diagram showing functional blocks of an image processing device. 3 is a flowchart showing the flow of processing for generating an output image. FIG. 3 is a diagram for explaining processing for generating an output image. 3 is a flowchart showing the flow of processing for generating an output image. FIG. 3 is a diagram showing the hardware configuration of each device.

Embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the configuration shown in the following embodiments is only an example, and the configuration is not limited to the illustrated configuration.

(Embodiment 1)
FIG. 1 is a diagram showing the system configuration in this embodiment. The system in this embodiment includes an image processing device 100, an imaging device 110, a recording device 120, and a display 130.

The image processing device 100, the imaging device 110, and the recording device 120 are interconnected via a network 140. The network 140 is realized, for example, by a plurality of routers, switches, cables, etc. that comply with communication standards such as ETHERNET (registered trademark).

Note that the network 140 may be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like.

The image processing apparatus 100 is realized, for example, by a personal computer or the like installed with a program for realizing image processing functions described below.

The imaging device 110 is a device that captures images. The imaging device 110 associates image data based on the captured image, information on the imaging time at which the image was captured, and identification information that is information for identifying the imaging device 110 and sends the data to the image processing device 100 and the like via the network 140. The information is transmitted to an external device such as the recording device 120. Note that in the system according to this embodiment, there is one imaging device 110, but there may be a plurality of imaging devices 110.

The recording device 120 records image data of an image captured by the imaging device 110, information on the imaging time at which the image was captured, and identification information for identifying the imaging device 110 in association with each other. Then, in accordance with a request from the image processing apparatus 100, the recording apparatus 120 transmits the recorded data (image, identification information, etc.) to the image processing apparatus 100.

The display 130 is configured with an LCD (Liquid Crystal Display) or the like, and displays the results of image processing by the image processing device 100, images captured by the imaging device 110, and the like. The display 130 is connected to the image processing device 100 via a display cable compliant with communication standards such as HDMI (registered trademark) (High Definition Multimedia Interface).

Further, the display 130 functions as a display means, and displays images captured by the imaging device 110, a setting screen related to image processing, etc. to be described later. Note that at least any two or all of the display 130, the image processing device 100, and the recording device 120 may be provided in a single housing.

Note that the results of image processing by the image processing device 100 and the images captured by the imaging device 110 are not limited to the display 130 connected to the image processing device 100 via a display cable, but can be displayed on external devices such as the following. It may be displayed on a display that has. That is, it may be displayed on a display of a mobile device such as a smartphone or a tablet terminal connected via the network 140.

Next, image processing of the image processing apparatus 100 according to the present embodiment will be described with reference to functional blocks of the image processing apparatus 100 according to the present embodiment shown in FIG.

In this embodiment, each function shown in FIG. 2 is realized as follows using a ROM (Read Only Memory) 1420 and a CPU (Central Processing Unit) 1400, which will be described later with reference to FIG. shall be carried out. Each function shown in FIG. 2 is realized by the CPU 1400 of the image processing apparatus 100 executing a computer program stored in the ROM 1420 of the image processing apparatus 100. Note that in the following description, it is assumed that the specific object is a person.

The communication unit 200 can be realized by an I/F (Interface) 1440, which will be described later with reference to FIG. 14, and communicates with the imaging device 110 and the recording device 120 via the network 140. The communication unit 200 receives, for example, image data of an image captured by the imaging device 110, and transmits a control command for controlling the imaging device 110 to the imaging device 110. Note that the control command includes, for example, a command that instructs the imaging device 110 to capture an image.

The storage unit 201 can be implemented by a RAM (Random Access Memory) 1410, an HDD (Hard Disk Drive) 1430, etc., which will be described later with reference to FIG. 14, and stores information and data related to image processing by the image processing apparatus 100. For example, the storage unit 201 stores information regarding the position of a person detected from an image.

The display control unit 202 causes the display 130 to display images captured by the imaging device 110, a setting screen for making settings related to image processing according to the present embodiment, information indicating the results of image processing, and the like. For example, the display control unit 202 causes the display 130 to display an output image generated by a generation unit 207, which will be described later. The operation reception unit 203 receives an operation performed by a user via an input device (not shown) such as a keyboard or a mouse.

The extraction unit 204 extracts a foreground region from the captured input image. The extraction unit 204 extracts a foreground region by comparing the input image received by the communication unit 200 and the background image. Note that the background image is an image that does not include a specific object (in this embodiment, a person) whose privacy is to be protected (hidden object).

The extraction unit 204 extracts a foreground region by comparing the input image received by the communication unit 200 and the background image. For example, the extraction unit 204 generates a binary image in which the foreground region indicating a difference with respect to the background image is set to "1" and the other regions are set to "0". In this case, the extraction unit 204 calculates a difference value between each pixel of the background image and each pixel of the input image, and sets "1" representing the foreground region to a pixel for which the calculated difference value is greater than or equal to a threshold value. , if it is less than the threshold value, a binary image can be generated by setting "0". Note that the extraction unit 204 may identify the foreground region using not only the background difference but also other methods.

The detection unit 205 detects a specific object from the image. For example, the detection unit 205 detects a specific object from the image using a matching pattern (dictionary). Note that when detecting a person from an image, improvement in detection accuracy can be expected by using both the matching patterns used when the person is facing forward and when the person is facing sideways. For example, a matching pattern for matching an image of a human body facing the front (back) and a matching pattern for matching an image of a person facing sideways are held, and the matching pattern is stored based on the installation state of the imaging device 110 or the user's designation. You can use both.

Further, the matching pattern may be prepared from other angles such as from an oblique direction or from above. Furthermore, when detecting a person, it is not necessarily necessary to prepare matching patterns (dictionaries) that indicate the characteristics of the whole body, but rather to prepare matching patterns for parts of the person such as the upper body, lower body, head, face, and feet. You can. Note that the detection unit 205 only needs to have a function of detecting a person as a specific object from an image, and is not limited to pattern matching processing.

In the following description, it is assumed that the detection unit 205 in this embodiment detects the upper body of a person from an image using a matching pattern of the upper body of the person. Then, the detection unit 205 identifies an image area of the whole body of the detected person from the area of the image (image area) showing the upper body of the person. For example, the detection unit 205 sets an image area obtained by enlarging the vertical size of an image area showing the upper body of a person by a predetermined magnification (for example, 2 times) downward in the vertical direction of the image as an image area of the whole body of the person. Identify. Note that the position of the person detected by the detection unit 205 is the coordinate of the center of gravity in the image area of the whole body of the person, with the upper left end point of the image as the origin.

Based on the position of the specific object detected by the detection unit 205 in the image, the setting unit 206 sets an area (specific area) having a shape corresponding to the specific object. The setting unit 206 in this embodiment sets a specific area, which is an area having a shape corresponding to the person, based on the position of the person in the image detected by the detection unit 205. For example, the setting unit 206 sets the specific area so as to surround the image area of the whole body of the person detected by the detection unit 205. Note that in the following description, the person area will be explained as a specific area set for a detected person.

The generation unit 207 generates a silhouette image that abstracts the foreground area in the person area set by the setting unit 206. In other words, the generation unit 207 excludes foreground areas other than the specific area set by the setting unit 206 from among the foreground areas extracted by the extraction unit 204 from the concealment target (abstraction target). Then, the generation unit 207 generates an abstracted (concealed) silhouette image by filling the foreground area in the person area set by the setting unit 206 with, for example, an arbitrary color (RGB values). Note that the silhouette image that abstracts the foreground region of the human region may be, for example, an image with texture, a mosaic image that has been subjected to mosaic processing, a blurred image that has been subjected to blurring processing, or the like.

Then, the generation unit 207 generates an output image in which a silhouette image, which is an abstraction of the foreground area in the person area set by the setting unit 206, is superimposed on a predetermined image. Note that the generation unit 207 in this embodiment superimposes the silhouette image on the background image, but it may also be superimposed on a display image that is an image prepared in advance for display purposes, or the communication unit 207 may superimpose the silhouette image on the background image. may be superimposed on the image (input image) received by the user.

Next, image processing by the image processing apparatus 100 according to the present embodiment will be described in more detail with reference to FIGS. 3 and 4. FIG. 3 is a flowchart showing the flow of image processing by the image processing apparatus 100 according to the present embodiment. Further, FIG. 4 is a diagram for explaining image processing of the image processing apparatus 100 according to the present embodiment.

Note that by executing the flow shown in FIG. 3, it is possible to generate an output image in which a silhouette image, which is an abstraction of the foreground region in the set human region, is superimposed on the background image. Note that the processing of the flow shown in FIG. 3 is started or ended, for example, according to an instruction from a user. It is assumed that the processing in the flowchart shown in FIG. 3 is executed by the functional blocks shown in FIG. 2, which are realized by the CPU 1400 of the imaging device 110 executing a computer program stored in the ROM 1420 of the imaging device 110.

First, in S301, the communication unit 200 receives an image (input image) captured by the imaging device 110. FIG. 4A is a diagram showing an example of an input image received by the communication unit 200. The image shown in FIG. 4(a) includes a door 400 that can be opened and closed, and people 401 and 402.

Next, in S302, the extraction unit 204 acquires a background image. Note that the extraction unit 204 may obtain, for example, a preset background image. Further, the extraction unit 204 may obtain a background image generated based on input images of the past N frames from the current frame. For example, the generation unit 207 generates the background image by finding the average value of each pixel value of the input image for the most recent five frames from the current frame. Then, the extraction unit 204 may acquire the background image generated by the generation unit 207. The image shown in FIG. 4(b) is a diagram showing an example of the background image acquired by the extraction unit 204 in S302, and the difference from the image shown in FIG. It is.

Next, in S303, the extraction unit 204 extracts a foreground region from the captured input image. The extraction unit 204 in this embodiment extracts a foreground region by comparing the input image received by the communication unit 200 with a background image. FIG. 4(c) is an example of an input image captured at a temporally different timing from that of FIG. 4(a). 4B differs from FIG. 4B in the presence or absence of persons 401 and 402 and in the open/closed state of the door 400. Note that FIG. 4C shows an example in which the state of the light source around the door 400 changes depending on whether the door 400 is opened or closed. The area 403 shown in FIG. 4(c) is an area where the pixel value changes significantly with respect to the background image shown in FIG. It shows.

The foreground regions 404 and 405 shown in FIG. 4(d) are extracted by the extraction unit 204 in S303 by comparing the input image shown in FIG. 4(c) and the background image shown in FIG. 4(b). This is the foreground area. In the example shown in FIG. 4D, the foreground area 404 is a foreground area corresponding to the person 401, and the shape of the person is clear because there are no other foreground areas around it. On the other hand, the foreground area corresponding to the person 402 is blended into the foreground area 405 corresponding to the area 403 with large changes in pixel values occurring around the person 402, making it difficult to distinguish the shape and position of the person 402. be.

Note that the image shown in FIG. 4E is an output image in which a silhouette image obtained by abstracting the foreground region extracted by the extraction unit 204 in S303 is superimposed on a background image. As shown in FIG. 4E, in the output image in which a silhouette image that simply abstracts the extracted foreground region is superimposed on the background image, it is difficult to understand the situation of the environment imaged by the imaging device 110. It may happen.

Next, in S304, the detection unit 205 detects a person from the input image using the matching pattern (dictionary). In the example shown in FIG. 4, the detection unit 205 detects a person 401 and a person 402 from the input image shown in FIG. 4(c).

Next, in S305, the setting unit 206 sets a person area, which is an area having a shape corresponding to the person, based on the position of the person detected by the detection unit 205. A person area 406 shown in FIG. 4F is a person area set by the setting unit 206 to surround the whole body of the person 401 based on the position of the person 401 detected by the detection unit 205. Further, a person area 407 shown in FIG. 4F is a person area set by the setting unit 206 to surround the whole body of the person 402 based on the position of the person 402 detected by the detection unit 205. Note that the shapes of the human regions 406 and 407 shown in FIG. 4(f) are elliptical, and are set by the setting unit 206 at a size corresponding to the size of each detected human. Although the shape of the person regions 406 and 407 is elliptical, for example, the shape of the region may include at least a first shape corresponding to the head of the person and a second shape corresponding to the moving body of the person. good. Note that the shape of the person area may be such that it completely encloses the whole body of the detected person, or may be such that a portion of the person protrudes. For example, the shape of the person area may be such that it covers only a part of the person, such as the head or upper body of the person. Note that a detailed description of the shape of the person area set by the setting unit 206 will be given later with reference to FIG.

Next, in S306, the generation unit 207 generates a silhouette image in which the foreground area in the person area set by the setting unit 206 is abstracted. The image shown in FIG. 4(g) is a silhouette image obtained by abstracting the foreground area in each of the human areas 406 to 407 set by the setting unit 206 with respect to the foreground area shown in FIG. 4(d). As shown in FIG. 4G, in the person area 406, the only foreground area that exists inside is the foreground area 404 corresponding to the person 401. Therefore, even if a silhouette image is generated by abstracting the foreground region 404, the shape of the difference region 404 is extracted as is as the shape of the silhouette image. On the other hand, in the person region 407, since a difference region exists around the person 402, it is difficult to generate a silhouette image that abstracts the detailed shape of the person 402. However, a silhouette image is generated in which the foreground area 405 in the person area 407 corresponding to the position and size of the person 402 is abstracted. By doing so, it is possible to generate a silhouette image that allows a rough understanding of the position, size, etc. of the person 402.

Next, in S307, the generation unit 207 generates an output image in which a silhouette image, which is an abstraction of the foreground area in the person area set by the setting unit 206, is superimposed on the background image. The diagram shown in FIG. 4(h) is a diagram illustrating an example of an output image in which the silhouette image generated in S306 is superimposed on the background image. A silhouette image 408 shown in FIG. 4(h) is a silhouette image in which the foreground region in the human region 406 is abstracted. Further, a silhouette image 409 shown in FIG. 4(h) is a silhouette image in which the foreground region in the human region 407 is abstracted. As shown in FIG. 4H, in areas where there are few environmental changes such as changes in brightness in the image due to opening and closing of the door 400, a silhouette image that reflects the shape of the actually existing person is displayed as shown in the silhouette image 408. It is possible to generate Therefore, the state of the imaging environment can be grasped in detail. On the other hand, in areas where there are many environmental changes, it is possible to generate a silhouette image that corresponds to the position and size of the person, as shown in silhouette image 409. It is possible to grasp the status of whether it exists or not.

Next, in S308, the display control unit 202 outputs the output image generated by the generation unit 207. Note that in this embodiment, the display control unit 202 causes the display 130 to display the output image generated by the generation unit 207.

Next, in S309, if the user instructs to end the process (Yes in S309), the process ends. On the other hand, if there is no instruction from the user to end the process (No in S309), the process returns to S301, and the communication unit 200 receives the next frame image (input image).

As explained above, the image processing of the image processing apparatus 100 according to the present embodiment sets a person area based on the detected position of the person, and uses a silhouette image that abstracts the foreground area of the set person area as a background image. Generates an output image superimposed on the image. By doing so, it is possible to provide a technique that makes it possible to generate an image in which an appropriate area of a captured image is hidden.

Next, with reference to FIG. 5, the shape of the human region set by the setting unit 206 will be described. In the above description with reference to FIG. 4, the elliptical shape 500 shown in FIG. 5 is used as an example of the human regions 406 and 407 set by the setting unit 206, but the present invention is not limited to this. The shape of the person area may be a simple figure such as a rectangle 501 or a trapezoid 502 shown in FIG. 5, for example.

Further, the shape of the person region may include a first shape corresponding to at least the head of the person and a second shape corresponding to the torso of the person. For example, the shape of the person region may be a shape such as shape 503 that includes a circular first shape 510 corresponding to the head and a triangular second shape 511 corresponding to the torso of the person. Further, the shape of the human region is a shape that includes a circular first shape 520 corresponding to the head and a second shape 521 having a part of an ellipse corresponding to the torso of the person, such as shape 504. Good too. Further, the shape of the human region may be a shape that includes a circular first shape 530 corresponding to the head and a substantially rectangular second shape 531 corresponding to the torso of the person, as in the shape 505.

Note that, as shown in shapes 503 to 505, which are examples of the shape of the human region, as described above, the shape includes a first shape that imitates the head of a person and a second shape that imitates the torso of the person. Yes, but not limited to this. For example, the shape of the human region may include a first shape that imitates the head of a person, a second shape that imitates the torso of the person, and a third shape that imitates the lower body of the person. You can.

Note that the shape of the person area in this embodiment may be a shape without a curve, such as a rectangle 501 or a trapezoid 502 shown in FIG. The shape may have a partial curve. In this way, by using a human region having a partially curved shape, it is possible to generate a silhouette image that more closely resembles the shape of a human.

By using a person region closer to the shape of a person, such as shapes 503 to 505 shown in FIG. 5, it is possible to generate a silhouette image closer to the shape of the person. For example, in FIG. 5H, the silhouette image 409 corresponding to the person 402 has an elliptical shape, but for a person detected in an area with a large foreground area, it is possible to use a person area that is closer to the shape of the person. , it becomes possible to present the presence of a person to the user more easily.

As explained above, the image processing of the image processing apparatus 100 according to the present embodiment sets a person area based on the detected position of the person, and uses a silhouette image as a background image that is an abstraction of the foreground area of the set person area. Generate an output image superimposed on the image. By doing this, it is possible to generate and display a silhouette image based on the position of the photographed person, making it possible to display the presence of a person in an easy-to-understand manner while ensuring privacy. . Further, even if a large foreground region is extracted so as to cover a person appearing in the image, it is possible to display an image that makes it easy to understand the state of the imaging environment. Furthermore, in the process of detecting a person, even if an object (for example, a poster including a person) that is not originally a detection target (hidden target) is detected, a silhouette image will not be generated unless a foreground region exists. Therefore, it is possible to suppress generation of an output image that may mislead the user, and to generate an output image that more closely reflects the actual situation. As described above, according to the image processing of the image processing apparatus 100 according to the present embodiment, it is possible to provide a technique that makes it possible to generate an image in which an appropriate area of a captured image is hidden.

(Embodiment 2)
This embodiment describes an embodiment that makes it easier for users to understand the presence of people in images while protecting privacy even in cases where there are many people in a crowded place such as a station or shopping mall. do. Hereinafter, parts that are different from Embodiment 1 will be mainly described, and the same or equivalent components and processes as Embodiment 1 will be denoted by the same reference numerals, and redundant explanation will be omitted.

Image processing according to this embodiment will be described below with reference to FIG. 6. FIG. 6 is a diagram for explaining image processing according to this embodiment. FIG. 6A shows an example of an input image captured by the imaging device 110. A person 600 in FIG. 6A is a person who exists alone outside an area 660 (crowded area) crowded with a plurality of people. Further, a person 601 indicates a plurality of people existing in the area 660. Shadow 602 is a shadow area created by a person in area 660.

FIG. 6(b) shows an example of a background image used in this embodiment. FIG. 6(c) shows a foreground region extracted by the extraction unit 204 by comparing the input image shown in FIG. 6(a) and the background image shown in FIG. 6(b). In the example shown in FIG. 6C, a foreground region 603 corresponds to a person 600 existing alone. Further, a foreground region 604 shown in FIG. 6C indicates a foreground region corresponding to a plurality of people existing in the region 660 and a shadow region 602 created by the plurality of people.

FIG. 6(d) is a diagram showing a person area set by the setting unit 206 based on the position of the person detected by the detection unit 205 with respect to the input image shown in FIG. 6(a). A person area 605 that is set based on the position of a person 600 that exists alone does not overlap with other person areas. On the other hand, the plurality of person areas 606 corresponding to the plurality of persons 601 existing in the area 660 (crowded area) overlap with other person areas. Here, in the same manner as the image processing described in Embodiment 1, a silhouette image is generated by abstracting the foreground area in the person area shown in FIG. 7(d), and an output image is obtained by superimposing the generated silhouette image on the background image. Assume that you want to generate . An example of the output image generated in this manner is shown in FIG. 6(e). In FIG. 6E, a silhouette image 607 corresponding to a person 600 existing alone can be displayed while distinguishing the person. However, in silhouette images 608 corresponding to multiple people 601 existing in a crowded area, an output image in which the silhouette images overlap is generated, making it difficult to understand the situation of the people in the captured image. Become.

Therefore, in order to make it easier to understand the situation of the people in the image even in a situation where the image is crowded with a plurality of people, the image processing apparatus 100 according to the present embodiment performs the following process, for example. That is, the generation unit 207 according to the present embodiment changes the display mode of the silhouette image, which is an abstraction of the foreground area in the person area, for each person area set by the setting unit 206. As a method of varying the display form of the silhouette image, for each person area set by the setting unit 206, the color of the foreground area in the person area may be different, or the type of texture given to the person area may be different. There is a way to do it. However, the method is not limited to the above method as long as it can be made distinguishable for each person area.

In addition, when the human areas overlap each other as shown by the plurality of human areas in the crowded area 660 as described above, when changing the display format for each human area, it is necessary to It is preferable to display silhouette images with priority. Note that, for example, the generation unit 207 generates an output image in which a silhouette image corresponding to a person area located on the front side of the image is located on the front side with respect to a silhouette image corresponding to a person area located on the back side of the image. By generating, priority display is performed.

Specifically, first, the detection unit 205 identifies the position of the detected person's feet. For example, the detection unit 205 identifies the lowest point in the image area of the detected person's whole body as the position of the person's feet. Then, the generation unit 207 preferentially displays a silhouette image corresponding to a person closer to the imaging device 110, for example, a person whose feet are located further down in the vertical direction of the image. Note that the silhouette image corresponding to a person is a silhouette image that abstracts the foreground area in the person area set for the person. Note that the generation unit 207 outputs an output in which a silhouette image corresponding to a person whose feet are located further down is located on the front side, with respect to a silhouette image corresponding to a person whose feet are located higher in the vertical direction of the image. Generate an image.

Here, the processing of the generation unit 207 according to the present embodiment will be described with reference to the flow shown in FIG. 3. In S306, the generation unit 207 according to the present embodiment generates a silhouette image for each person area set by the setting unit 206, while changing the display mode of the silhouette image that abstracts the foreground area in the person area. .

FIG. 6F shows an example of an output image in which the silhouette image generated by the processing of the generation unit 207 according to the embodiment described above is superimposed on the background image. As shown in FIG. 6(f), each silhouette image is identified not only in a silhouette image 609 corresponding to a single person 600 but also in a silhouette image 610 corresponding to multiple persons 601 existing in a crowded area. possible output images.

As described above, the generation unit 207 according to the present embodiment changes the display mode of the silhouette image, which is an abstraction of the foreground area in the person area, for each person area set by the setting unit 206. In this way, even in a situation where the image is crowded with multiple people, the privacy of the person included in the image is protected, and the situation of the person included in the image can be easily understood.

(Embodiment 3)
The generation unit 207 according to the second embodiment generates a foreground area in the person area for each set person area in order to make it easier to understand the situation of the person in the image even in a situation where the image is crowded with multiple people. The display mode of abstracted silhouette images was changed.

The generation unit 207 according to the present embodiment performs the following processing in order to make it easier to understand the situation of the person in the image even in a situation where the image is crowded with a plurality of people. That is, the generation unit 207 according to the present embodiment generates a display mode of a silhouette image in which the foreground region of the human region is abstracted for each set human region based on the position of a crowded region crowded with a plurality of people in the image. Make the difference. The processing of the generation unit 207 according to this embodiment will be described below with reference to FIG. Note that parts that are different from Embodiments 1 and 2 will be mainly explained, and components and processes that are the same or equivalent to Embodiments 1 and 2 will be denoted by the same reference numerals, and redundant explanation will be omitted.

FIG. 7 is a diagram showing functional blocks of the image processing apparatus 100 according to this embodiment. In addition, each function shown by the functional block of FIG. 7 shall be implement|achieved as follows using ROM1420 and CPU1400 in this embodiment. Each function shown in FIG. 7 is realized by the CPU 1400 of the image processing apparatus 100 executing a computer program stored in the ROM 1420 of the image processing apparatus 100.

The processes of the communication unit 200, storage unit 201, display control unit 202, operation reception unit 203, extraction unit 204, detection unit 205, and setting unit 206 in FIG. 7 are the same as those in Embodiment 1, so description thereof will be omitted.

A determining unit 708 shown in FIG. 7 determines a crowded area in an image. For example, the determining unit 708 determines a crowded area in the image based on the number of people detected by the detecting unit 205. In this case, the determination unit 708 divides the entire area of the image into a plurality of divided areas, counts the number of people detected in each divided area, and selects the divided areas where the number of people is equal to or greater than a threshold as a crowded area. Determine as.

Further, the determining unit 708 determines a crowded area, for example, based on the position and size of the foreground area extracted from the image by the extracting unit 204. Specifically, the determining unit 708 may determine that the foreground area extracted by the extracting unit 204 is a crowded area when the size of the foreground area is equal to or larger than a threshold value. Note that one foreground region is defined as a region formed by connecting adjacent pixels among pixels that are "1" indicating a foreground region in a binary image. Then, when a plurality of foreground regions are extracted from the image, the determination unit 708 compares the size of the foreground region with a threshold value for each of the plurality of foreground regions to determine which foreground region is a crowded region. do.

Further, the determining unit 708 determines a crowded area in the image based on, for example, the position and size of the foreground area extracted from the image by the extracting unit 204 and the number of people detected from the image by the detecting unit 205. You can. Specifically, the determination unit 708 determines whether the size of the foreground region extracted from the image is equal to or greater than a first threshold, or the number of people detected when the image is divided into a plurality of areas is equal to or greater than a second threshold. area is determined to be a congested area.

Then, the generation unit 207 according to the present embodiment generates a silhouette image that abstracts the foreground area of the person area for each of the person areas set for the person located in the crowded area in the image determined by the determination unit 708. Make the display mode different. For example, the determination unit 708 determines an area 660 crowded with people shown in FIG. 6 as a crowded area. In this case, the generation unit 207 displays, for each person area set for a person located in the area 660 determined to be a crowded area by the determination unit 708, a silhouette image that abstracts the foreground area of the person area. Make the aspects different. The method of varying the display mode of the silhouette image, which is an abstraction of the foreground area in the person area, for each person area is the same as that described in Embodiment 2, so a description thereof will be omitted. Note that, for example, when the overlapping ratio, which is the ratio at which the detected person's whole body image area and the crowded area overlap, is equal to or higher than a threshold, the determination unit 708 determines that the person is a person located in the crowded area. do.

As described above, the generation unit 207 according to the present embodiment abstracts the foreground area in the person area for each set person area based on the position of the crowded area in the image determined by the determination unit 708. The display mode of silhouette images is made different. In this way, even in a situation where the image is crowded with multiple people, the privacy of the person included in the image is protected and the situation of the person included in the image can be easily understood.

(Embodiment 4)
The generation unit 207 according to the second embodiment generates a foreground area in the person area for each set person area in order to make it easier to understand the situation of the person in the image even in a situation where the image is crowded with multiple people. The display mode of abstracted silhouette images was changed.

The image processing apparatus 100 according to the present embodiment performs the following process in order to make it easier to understand the situation of the person in the image even in a situation where the image is crowded with a plurality of people. That is, the setting unit 206 of the image processing apparatus 100 according to the present embodiment sets human regions having different shapes based on the crowded region of the people included in the image. The processing of the setting unit 206 according to this embodiment will be described below with reference to FIG. 7. Note that parts that are different from Embodiments 1 to 3 will be mainly explained, and components and processes that are the same or equivalent to Embodiments 1 to 3 will be given the same reference numerals, and redundant explanation will be omitted.

FIG. 8 is a diagram illustrating the shape of a person area that is set when congestion occurs. A person area 800 shown in FIG. 8 is a person area set by the setting unit 206 for a person 801 existing alone in a non-congested area, and is a person area of a size that fits the whole body of the person 801. be. Note that, for convenience, the human region in this embodiment has the shape of an ellipse 500 shown in FIG. 5, but is not limited to this. For example, the person area to be set may have a shape other than the ellipse 500 shown in FIG. 5.

Here, assuming that the extraction unit 204 extracts an image area of the whole body of the person 801 in the person area 800 as a foreground area, the generation unit 207 generates the silhouette image 802 by abstracting the foreground area. As shown in the silhouette image 802, in a silhouette image in which the foreground area is abstracted in a person area set for a person in a non-congested area, the shape of the person is relatively easy to understand.

On the other hand, a person area 810 is a person area set for a person 812 among a person 811 and a person 812 existing in a crowded area (area 660 shown in FIG. 6, etc.), and has the same shape as the person area 800. It is. Assuming that the whole body image area of the person 812 and the whole body image area of the person 811 are extracted by the extraction unit 204 as foreground areas, the generation unit 207 generates a foreground image area in the person area 810 set for the person 812. A silhouette image 813 is generated by abstracting the area. Here, an example is shown in which a part of the person 811 located behind the person 812 enters the person area 810, and as a result, part of the person 811 is generated as a silhouette image. As shown in the silhouette image 813, in a person area set for a person in an area where congestion has occurred, a silhouette image that abstracts the foreground area will display an unnatural shape as the shape of the person. It may become difficult to see depending on the user.

The person area 820 is a person area set by the setting unit 206 for the person 822 among the plurality of people 821 and 822 existing in the crowded area. Furthermore, the human region 820 is an example of a human region whose horizontal width is reduced by a predetermined magnification compared to the human region 800 and the human region 810. Assuming that the extraction unit 204 extracts the whole body image area of the person 821 and the whole body image area of the person 822 as foreground areas, the generation unit 207 generates a foreground image area in the person area 820 set for the person 822. A silhouette image 823 is generated by abstracting the region.

For a person 822 existing in a crowded area, such as the person area 820, the setting unit 206 according to the present embodiment sets a person area whose horizontal width is reduced by a predetermined magnification. For example, the determining unit 708 shown in FIG. 7 determines a crowded area in an image, and the setting unit 206 sets a predetermined width in the horizontal direction for each person existing in the crowded area determined by the determining unit 708. Set the human area reduced by the magnification of . At this time, for example, assume that the determination unit 708 divides the image into multiple areas, counts the number of people detected in each area, and determines an area where the number of people exceeds a threshold as a crowded area. . In this case, the setting unit 206 sets a person area prepared in advance as in the first embodiment for a person detected in an area that has not been determined as a crowded area. On the other hand, for a person detected in an area that is not determined to be a crowded area, the setting unit 206 sets a person area whose horizontal width is reduced by a predetermined magnification.

Further, assume that the determining unit 708 determines a crowded area based on the position and size of the foreground area extracted from the image by the extracting unit 204. Specifically, assume that the determining unit 708 determines the foreground area extracted by the extracting unit 204 as a crowded area when the size of the foreground area is equal to or larger than a threshold value. In this case, for a person located in a foreground area having a size equal to or larger than the threshold value (for example, a person positioned to be included in the foreground area), the setting unit 206 reduces the person area by a predetermined magnification in the horizontal direction. Set.

Further, the case where the determining unit 708 determines the crowded area in the image based on the position and size of the foreground area extracted from the image by the extracting unit 204 and the number of people detected from the image by the detecting unit 205 is assumed. Suppose. Specifically, the determination unit 708 determines, for example, if the size of the foreground region extracted from the image is equal to or greater than a first threshold, or if the number of people detected when the image is divided into a plurality of areas is equal to or greater than a second threshold. Assume that the area , is determined to be a congested area. In this case, the setting unit 206 sets a person area whose horizontal width is reduced by a predetermined magnification for the person detected in the thus determined crowded area.

As explained above, the setting unit 206 according to the present embodiment sets a predetermined magnification in the horizontal direction for a person (such as the person 822 shown in FIG. 8) existing in a crowded area in the image determined by the determination unit 708. Set the human area reduced by the amount. A silhouette image obtained by abstracting the foreground region of the human region set in this way may become a silhouette image in which the original shape of the human figure is removed. However, such a silhouette image can be prevented from becoming a silhouette image in which part of another person located near the person concerned is mixed into the silhouette image. In addition, by performing the same processing on each of multiple people existing in a crowded area, it is possible to easily understand each person in the image individually even if the area is crowded with multiple people. Can be done.

(Embodiment 5)
In the second embodiment, in order to make it easier to understand the situation of the person in the image even in a situation where the image is crowded with multiple people, a silhouette image is created in which the foreground area of the person area is abstracted for each set person area. An embodiment in which the display mode of is changed has been described.

The generation unit 207 in this embodiment changes the display mode of a silhouette image that abstracts the foreground area in the person area set for the person, depending on the movement status of the person in the image.

The processing of the image processing apparatus 100 according to this embodiment will be described below. Note that parts that are different from Embodiments 1 to 4 will be mainly explained, and components and processes that are the same or equivalent to Embodiments 1 to 4 will be denoted by the same reference numerals, and redundant explanation will be omitted.

In addition to the process of detecting a person from an image, the detection unit 205 according to the present embodiment also tracks a person included in the image. For example, when the determination unit 708 detects the same person from the image of the frame of interest as the person detected from the image of the frame one or more frames before the current frame (current frame), the determination unit 708 associates the people in each frame with each other. Attach. That is, a person is tracked between images of a plurality of temporally close frames.

As a method for the detection unit 205 to determine that the person is the same across multiple frames of images, for example, if the predicted movement position of the person and the detected person position are within a certain distance using the movement vector of the detected person. There is a way to prove that they are the same person. Further, the detection unit 205 may use the color, shape, size (number of pixels), etc. of the person to associate people with high correlation between images of a plurality of frames. In this way, the detection unit 205 is not limited to a specific method as long as it can determine that the person is the same person across multiple frames of images and track the person.

The determination unit 708 according to the present embodiment determines the movement status (movement status) of the person tracked by the detection unit 205. Note that the moving status of the person includes, for example, the moving speed of the person, and whether or not the person is moving. Here, it is assumed that the determination unit 708 determines the moving speed of the person tracked by the detection unit 205. In this case, the determination unit 708 determines the moving speed of the person tracked by the detection unit 205 based on the frame rate at which the imaging device 110 captures the image and the change in the position of the person tracked by the detection unit 205 in the image.

Further, it is assumed that the determination unit 708 determines whether or not the person tracked by the detection unit 205 is moving. In this case, the determination unit 708 compares the position of the person tracked by the detection unit 205 in the current frame with the position of the person in the previous frame, and calculates the moving distance of the person. do. The determination unit 708 then compares the calculated movement distance with a threshold value and determines whether the person has moved. Specifically, the determining unit 708 determines that the person is moving if the calculated moving distance is equal to or greater than the threshold, and determines that the person is not moving if the calculated moving distance is less than the threshold. It is determined that

Further, as another method for determining whether or not a person has moved, a method based on inter-frame differences of a plurality of input images may be used. For example, the determination unit 708 determines whether or not the person has moved based on the person area and the size of the moving object area specified by the inter-frame difference. Specifically, the determination unit 708 determines that the person is moving, for example, if the overlapping ratio, which is the ratio at which the person area set for the person and the moving object area overlap, is equal to or higher than a threshold value, the determination unit 708 determines that the person is moving. judge. On the other hand, if the overlap ratio between the person area set for the person and the moving body area is less than the threshold, the determination unit 708 determines that the person is not moving. Note that, as a method for identifying the moving object region, a foreground region extracted by the extraction unit 204 using a background image may be identified as the moving object region. Note that the process of determining whether or not a person has moved may be performed for each divided region obtained by dividing the image into a plurality of regions.

The generation unit 207 according to the present embodiment changes the display mode of a silhouette image in which the foreground area in the person area set for the person is abstracted based on the movement status of the person determined by the determination unit 708. For example, the generation unit 207 changes the display mode of a silhouette image that abstracts the foreground area in the person area set for the person based on the moving speed of the person determined by the determination unit 708. Further, for example, the generation unit 207 changes the display mode of the silhouette image that abstracts the foreground area in the person area set for the person based on whether the person has moved as determined by the determination unit 708.

Note that, for example, as a method of varying the display mode of a silhouette image in which the foreground region in the human region is abstracted depending on the movement situation, the following processing may be performed. That is, when filling the foreground area in the person area set for a moving person with a predetermined color, the generation unit 207 uses different colors among warm colors depending on the moving speed. . On the other hand, when the generation unit 207 fills in the foreground area in the person area set for a person who is not moving with a predetermined color, the generation unit 207 displays the color using different colors among the cool color hues. In this way, general color types (warm colors, cool colors, etc.) are assigned depending on the presence or absence of movement, and furthermore, colors are assigned to each general color type depending on the speed of movement. You can do it like this. Note that the method of varying the display mode of the silhouette image that abstracts the foreground region in the human region is not limited to a specific method, and for example, the method of varying the display mode of the silhouette image as described in Embodiment 2 can be used. You can use

As described above, the generation unit 207 of the image processing device 100 according to the present embodiment displays a silhouette image that abstracts the foreground area of the person area set for the person, depending on the movement status of the person in the image. Make the aspects different. In this way, by changing the display mode of the silhouette image depending on the movement status of the person, it becomes possible to more easily understand the behavior of the person in the image.

(Embodiment 6)
In each of the above embodiments, the privacy of the people included in the image is protected by changing the display form of each person even when there are overlapping people as described with reference to FIG. At the same time, the situation of the person included in the image can be easily understood.

In this embodiment, when people overlap, it is possible to understand the situation of the person included in the image while protecting the privacy of the person included in the image, in a state where each person can be more clearly distinguished. A method for generating an output image that is easy to use will be explained. Note that parts that are different from Embodiments 1 to 5 will be mainly described, and components and processes that are the same or equivalent to Embodiments 1 to 5 will be given the same reference numerals, and redundant explanation will be omitted.

FIG. 9 is a diagram for explaining image processing related to this embodiment. FIG. 9A shows an example of an input image captured by the imaging device 110, in which people 900 to 903 are shown as subjects.

FIG. 9(b) shows an example of a background image used in this embodiment. A person area 990 shown in FIG. 9(c) is a person area in this embodiment, and is large enough to accommodate the person detected by the detection unit 205. 206.

FIG. 9D is an output image in which the silhouette image generated by the generation unit 207 is superimposed on the background image through the image processing according to the first embodiment. Specifically, the setting unit 206 sets a person area 990 for a person detected from the input image shown in FIG. 9(a). Then, the generation unit 207 generates an output image by superimposing a silhouette image obtained by abstracting the foreground area in the person area 990 set by the setting unit 206 on the background image shown in FIG. 9(b). The output image generated by the generation unit 207 in this manner is the image shown in FIG. 9(d).

Here, FIG. 9(d) shows an example of an output image in which overlapping silhouette images 905 to 907 of people are displayed in different display formats using different colors. From this, as shown in FIG. 9(d), there are not only silhouette images 904 corresponding to a single person 900, but also silhouette images 905 to 907 corresponding to multiple persons 901 to 903 existing in a crowded area. It is possible to generate an output image in which each silhouette image can be identified.

When displaying the output image shown in FIG. 9D on the display 130, depending on the display performance of the display 130, similar colors may become visually indistinguishable. In this case, in an image that includes many people, it may be difficult to see the difference in color between the overlapping people. Furthermore, in the case where the display format is explicitly changed based on the difference in color, it is necessary to use combinations of colors that are visually significantly different, so it is necessary to set a restriction on the number of colors that can be used. However, if there are a large number of people in the image compared to the number of available colors, the number of usable colors may be insufficient and the same color may be applied to overlapping people, making it impossible to distinguish them. be. Therefore, in this embodiment, an embodiment will be described in which it is possible to more clearly distinguish between people.

FIG. 9(e) is a diagram showing a suitable example of an output image of this embodiment. In FIG. 9(e), in addition to the process of making the display form different for each person area, only the outer edge of the person area, which overlaps with the foreground area, is displayed in a different display form from the inside of the person area. indicate.

The person area 990 in this embodiment is larger than the actual size of the person to be detected. Therefore, for example, when setting a person area 990 for a person 900 detected from an image, the foreground area extracted from the person 900 and the outer edge of the person area 990 set for the person 900 do not overlap. On the other hand, when setting a human region 990 for each of the persons 901 to 903 that overlap, the foreground region extracted from each of the persons 901 to 903 and the human region 990 for each of the persons 901 to 903 are set. There is a line that overlaps with the outer edge. A line 908 shown in FIG. 9(e) is a line where the foreground region extracted from the persons 901 to 903 overlaps with the outer edge of the human region 990 set for the person 902 (hereinafter referred to as an overlapping line). Furthermore, a line 909 is an overlapping line where the foreground regions extracted from the persons 901 to 903 overlap with the outer edge of the human region 990 set for the person 903. The generation unit 207 in this embodiment makes the colors of lines 908 and 909, which are overlapping lines, different from the color of the silhouette image. For example, the generation unit 207 sets the color of the line 908 and the line 909 to black, and sets the color of the silhouette images 904 to 907 to a color other than black. In this way, by explicitly indicating the lines 908 and 909 where the persons overlap, even when a plurality of persons overlap, it is possible to easily distinguish between the persons.

Note that although the person area 990 in this embodiment has been described as being large enough to accommodate the person in the image, the present invention is not limited to this. For example, assume that in a state where there are many people, a person area is set whose horizontal width is reduced by a predetermined magnification, as described in the fourth embodiment. In such a case, even if the person 900 does not overlap, if a person area whose horizontal width is reduced by a predetermined magnification is set, the foreground area extracted from the person 900 and the person area There is an overlapping line that overlaps with the outer edge. The generation unit 207 in this embodiment displays the overlapping line in a color different from that of the silhouette image. For example, the generation unit 207 sets the overlapping line to black and the silhouette image to a color other than black. Note that the display format for the overlapping lines may be the same for each person (person area), or may be different for each person (person area). Alternatively, the same display form may be applied to persons who can be classified into the same specific group, such as whether or not each person moves and the direction of movement, and a different display form may be used for each group.

Furthermore, in the above description, an example has been shown in which the internal color is changed for each human region, and only the overlapping lines at the outer edge of the human region are displayed in a different manner. However, if it is sufficient to distinguish the overlapping lines at the outer edge of the human area, it is not necessary to change the internal color for each human area, and the same color may be used. Alternatively, by some classification, the same color may be applied to the inside of the person area for people belonging to the same group, and the display format may be different for each group.

(Embodiment 7)
Conventionally, there are cases in which a user wants to understand whether or not he or she is likely to board a train from a still image taken of a station platform or the like. For example, in the event of an accident or disaster, there is a demand to confirm whether the platform or station is overflowing with people and whether it is possible to board the train even if you head to the station, or whether you will not be able to board the train. In such a case, it may be difficult to determine whether or not the person can board the train based on a single still image that includes a silhouette image that is an abstraction of the person included in the image.

Even if there are many people in an image, if the user can determine that many people are moving, the user can determine that they are likely to get on the train, but from a single still image, it is difficult to see that many people are moving. The user cannot tell if this is the case. Therefore, the user cannot determine whether it is likely that he or she will be able to board the train. On the other hand, a user can determine whether a person is moving from a single still image by changing the color of the silhouette image depending on whether the person is moving or not. However, when there are many people in an image, motion detection fails because the people overlap, and as a result, it is not possible to determine whether the people are moving, and the user has to judge whether or not they can get on the train. It may not be possible. Therefore, the present embodiment aims to generate an output image that can easily determine whether a person is likely to board a train at a station or the like, even if it is a single still image that abstracts a person. The image processing in this embodiment will be described below, but the differences from Embodiments 1 to 6 will be mainly explained, and the same or equivalent components and processes as in Embodiments 1 to 6 will be given the same reference numerals. , duplicate explanations will be omitted.

First, image processing by the image processing apparatus 100 according to this embodiment will be described with reference to the functional blocks shown in FIG. In this embodiment, each function shown in FIG. 10 is realized as follows using a ROM 1420 and a CPU 1400, which will be described later with reference to FIG. 14. Each function shown in FIG. 10 is realized by the CPU 1400 of the image processing apparatus 100 executing a computer program stored in the ROM 1420 of the image processing apparatus 100.

The determining unit 708 determines the movement of an object in images continuously captured by the imaging device 110. For example, the determination unit 708 determines the amount of motion in the image based on the inter-frame difference between input images that the communication unit 200 continuously receives. For example, the determination unit 708 calculates a difference value between each pixel value of the previous frame and each pixel value of the current frame, and determines that the pixel for which the calculated difference value is equal to or greater than a threshold is a moving pixel, and Let the sum of the number of pixels be the amount of motion. The amount of motion may be determined by determining the amount of motion in the entire image. Alternatively, the image may be divided into a plurality of regions, and the amount of motion may be determined for each divided region. Furthermore, as the content to be determined, instead of the amount of motion, the presence or absence of motion may be simply determined as described in the fifth embodiment. For example, the determination unit 708 tracks an object included in an image and calculates the distance traveled by the object from the previous frame to the current frame. The determining unit 708 then compares the calculated moving distance with a threshold value, and determines that the object is moving if the moving distance is greater than or equal to the threshold value, and determines that the object is not moving if the moving distance is less than or equal to the threshold value.

Further, the determination unit 708 determines the movement of the specific object, for example, by tracking the position of the specific object detected by the detection unit 205 between input images continuously received by the communication unit 200. When determining motion, for example, the determination is performed for all specific objects appearing in the image. Alternatively, the motion is determined for the representative object selected for each predetermined image position, and is reflected in the motion of other specific objects appearing around the representative object. Alternatively, the image is divided into a plurality of regions, one or more representative objects are selected within the divided regions, the movement is determined, and this is reflected in the movement of other specific objects shown within the divided regions. By reflecting the movement of the representative object on the movement of specific surrounding objects in this way, it is possible to reduce tracking and movement determination processing.

Based on the determination result of the determination unit 708, the adjustment unit 1000 adjusts the number of specific objects (abstraction targets) to be abstracted into a silhouette image among the specific objects detected by the detection unit 205. For example, if the amount of movement in the entire image obtained from the determination result of the determination unit 708 is equal to or greater than a predetermined value, not all of the people detected by the detection unit 205 are abstracted, but only some of the people are abstracted. is the object of abstraction. In other words, the adjustment unit 1000 reduces the number of people to be abstracted.

As a reduction method, for example, an upper limit is set based on the amount of movement, and if the number of people detected by the detection unit 205 exceeds the upper limit, the number of people to be used as silhouette images is reduced up to the upper limit. reduce. The method for determining the upper limit number is to reduce the upper limit number as the amount of movement increases. Note that if the amount of motion is below a predetermined value, no upper limit may be set.

Further, as other preferable examples of the method for reducing specific objects to be abstracted in the adjustment unit 1000, other methods such as reducing at a predetermined ratio or uniformly reducing to a predetermined number may be applied. . Furthermore, as a method for determining objects that are not to be abstracted (non-abstracted objects) in the adjustment unit 1000, it is preferable to determine the objects that are not to be abstracted so that the positions of the objects in the image are dispersed as much as possible. As a method for determining objects to be de-abstracted, for example, representative objects are set in order from objects in the foreground of the image that are considered to have a large visual impact to objects in the distance, and the object closest to the representative object is selected. Become a non-abstract object. By repeating this process until the number of objects to be abstracted is reduced, it is possible to reduce the number of objects to be abstracted so that objects in the image are thinned out on average. Alternatively, as a method for determining objects to be reduced, for example, a plurality of specific objects having close distances between the objects are searched, and an object to be non-abstracted is determined from among the plurality of objects having close distances. By repeating this process until the number of reductions is reached, non-abstract objects are selected from areas with high object density in the image, so the objects to be abstracted are selected to reduce the bias in the density of objects in the image. It becomes possible to reduce the number.

Furthermore, as a method for reducing objects to be abstracted or determining objects to be non-abstracted in the adjustment unit 1000, for example, the image is divided into a plurality of regions and the above-described processing is applied to each divided region. In this way, it is possible to reduce the number of objects detected in a specific image area differently from other image areas, and it is possible to adjust the number of objects to be abstracted depending on the imaging scene. . Furthermore, the method of adjusting the number of objects in the adjustment unit 1000 is not limited to the above-mentioned method, and may be realized by other methods that provide the same effect, such as randomly determining the number using random numbers.

The generation unit 207 generates a silhouette image in which a region corresponding to the specific object to be abstracted is abstracted based on the result of the adjustment by the adjustment unit 1000. In this embodiment, the generation unit 207 generates a silhouette image in which a foreground area in a specific area set for a specific object to be abstracted is abstracted. The generation unit 207 then generates an output image by superimposing the generated silhouette image on the background image.

Next, image processing by the image processing apparatus 100 according to this embodiment will be described in more detail with reference to FIGS. 11 and 12. FIG. 11 is a flowchart showing the flow of image processing by the image processing apparatus 100 according to this embodiment. Further, FIG. 12 is a diagram for explaining image processing of the image processing apparatus 100 according to the present embodiment.

Note that by executing the flow shown in FIG. 11, an output image is generated in which a silhouette image in which the foreground region in the set human region is abstracted is superimposed on the background image after adjusting the number of abstraction targets. be able to. It is assumed that the processing of the flow shown in FIG. 11 is started or ended, for example, according to an instruction from a user. Note that the processing in the flowchart shown in FIG. 11 is executed by the functional blocks shown in FIG. 2, which are realized by the CPU 1400 of the imaging device 110 executing a computer program stored in the ROM 1420 of the imaging device 110.

First, in S1101, the communication unit 200 receives an image (input image) captured by the imaging device 110. FIG. 12A is a diagram showing an example of an input image received by the communication unit 200. The image shown in FIG. 12(a) includes a plurality of people.

Next, in S1102, the extraction unit 204 acquires a background image. FIG. 12(b) is a diagram showing an example of a background image acquired by the extraction unit 204 in S1102, and the difference from the image shown in FIG. 12(a) is the presence or absence of people 1201 to 1209.

Next, in S1103, the extraction unit 204 extracts a foreground region from the captured input image. The extraction unit 204 in this embodiment extracts a foreground region by comparing the input image received by the communication unit 200 with a background image. FIG. 12(c) is an image including a foreground region extracted by the extraction unit 204 in S1103 by comparing the input image shown in FIG. 12(a) and the background image shown in FIG. 12(b). It is. The foreground region shown in FIG. 12(c) is shown in black.

Next, in S1104, the detection unit 205 detects a person from the input image using a matching pattern (dictionary). In the example shown in FIG. 12, the detection unit 205 detects people 1201 to 1209 from the input image shown in FIG. 12(a).

Next, in S1105, the determination unit 708 determines the amount of movement of the person detected from the plurality of input images that are received continuously. Here, if an image of a scene where a person is normally walking at a station or the like is captured, the movement of the person is detected. On the other hand, if entry is restricted to a station, many people will form a line without being able to move, so the detection unit 205 will only detect the movement of people less than a predetermined amount. In this way, the amount of movement in the image is assumed to be information indicating the possibility that the flow of people is stagnant (retention possibility information).

Next, in S1106, the adjustment unit 1000 adjusts the number of persons to be abstracted among the detected persons 1201 to 1209, based on the movement of the persons determined in S1105. For example, if the amount of movement determined in S1105 is equal to or greater than a predetermined value, the adjustment unit 1000 does not set all of the people detected by the detection unit 205 as abstraction targets, but selects some of the people as abstraction targets. do.

As a specific example of the adjustment content performed by the adjustment unit 1000, when there is a low possibility that stagnation has occurred, in order to make it easier to visually understand that there is sufficient space for a person to move in the output image described later. , the number of people to be abstracted is reduced. By doing so, it is possible to reflect in the output image, which will be described later, that although congestion has occurred, there is no stagnation state.

On the other hand, if there is a high possibility that stagnation has occurred, the number of people to be abstracted is not reduced in order to make the actual situation easier to visually understand in the output image described later. Alternatively, the reduction rate is changed depending on the possibility that stagnation has occurred. Here, even if the determining unit 206 determines that there is a high possibility that stagnation is occurring, it does not necessarily mean that congestion is actually occurring. However, even in this case, the number of people to be abstracted is not reduced, and it becomes possible to reflect an actual situation in which no congestion occurs in the output image, which will be described later.

Next, in S1107, the setting unit 206 determines the shape corresponding to the person based on the position and size of the person detected by the detection unit 205 and further adjusted by the adjustment unit 1000. Set the person area which is the area of . That is, the setting unit 206 sets a person area for the person who is the abstraction target.

Next, in S1108, the generation unit 207 generates a silhouette image that abstracts the foreground area in the person area set by the setting unit 206.

Next, in S1109, the generation unit 207 generates an output image in which the silhouette image generated in S1108 is superimposed on the background image. The image shown in FIG. 12(d) is an example of an output image generated by the generation unit 209 when the adjustment unit 1000 does not adjust the number of people, and the movement of the person detected by the detection unit 205 is This corresponds to cases where there are fewer. On the other hand, FIG. 12(e) is an example of an output image generated by the generation unit 207 when the adjustment unit 1000 adjusts the number of people. Compatible.

As shown in FIG. 12E, when people are thinned out as a result of adjusting the number of people to be abstracted, there are many gaps between the people in the silhouette image. As a result, from the image shown in FIG. 12(e), it becomes possible to display an appearance that is not very crowded. Although the image shown in FIG. 12(e) does not reflect the actual number of people detected, it is possible to present to the viewer in a form that is easy to understand at a glance that people can move in the imaging area. For example, it can be easily determined that the situation is not such that it is impossible to enter the station.

On the other hand, as shown in FIG. 12(d), when the number of people to be abstracted is adjusted and the people are not thinned out, the gaps between the people in the silhouette image are detected by the detection unit 205. Decisions will be made based on the results. Here, even if the adjustment unit 1000 does not change the number of people, if the number of people within the imaging range is small, the number of people detected by the detection unit 205 is small. A silhouette image with many gaps will be generated. On the other hand, if there are many people within the imaging range, the number of people detected by the detection unit 205 will increase, and as a result, a silhouette image with fewer gaps between the people will be generated. When a viewer is presented with an image in which there are few gaps between people, the viewer can confirm the appearance of crowding from the silhouette image.

Next, in S1110, the display control unit 202 outputs the output image generated by the generation unit 207. Note that in this embodiment, the display control unit 202 causes the display 130 to display the output image generated by the generation unit 207.

Next, in S1111, if the user instructs to end the process (Yes in S1111), the process ends. On the other hand, if there is no instruction from the user to end the process (No in S1111), the process returns to S1101, and the communication unit 200 receives the next frame image (input image).

In the above explanation, the foreground region is extracted by the background subtraction method, and the output image is generated by superimposing a silhouette image, which is an abstraction of the foreground region in the human region, on the background image, but the present invention is not limited to this. do not have. For example, edges obtained by performing edge extraction processing on the input image and cutting out edge components based on the position of the person detected by the detection unit 205 may be used as the silhouette image. Alternatively, the outline of the area of the person detected by the detection unit 205 may be used as the silhouette image. Alternatively, a filled-in image obtained as a result of filling in the inner region of the outline may be used as the silhouette image. Alternatively, an icon indicating the presence of the person may be superimposed at a position in the background image corresponding to the position of the person in the input image detected by the detection unit 205. In other words, an icon indicating the presence of a person may be used as a silhouette image. In addition, any method for generating a silhouette image may be applied as long as it is applicable to adjusting the number of people.

Furthermore, when applying the above example, in addition to performing the same processing on the entire input image, it is also necessary to divide the input image into multiple regions and adjust the number of people to be abstracted differently for each divided region. You may also do this.

As described above, according to the image processing device according to the present embodiment, it is possible to change the appearance of the image that the viewer sees depending on the amount of movement of the person. As a result, when a crowded situation occurs where people cannot move, it becomes possible to present an image showing the congestion. On the other hand, if the situation is crowded but people are moving, such as getting on and off the train, it is possible to present an image that shows the situation in which people can move. By doing so, in this embodiment, even if it is a single still image in which a person is abstracted, it is possible to generate an output image that can easily determine whether a person is likely to board a train at a station or the like. Please note that the way people move in an image varies depending on the location and installation method of the camera that captured the image, so even if the same amount of movement is obtained, different measures may be required depending on the location of the camera or station. There is. Therefore, in addition to the movement of the person determined in S1105, the determination may be made based on a predetermined criterion, and the adjustment in S1106 may be performed based on the determination result.

(Embodiment 8)
In the seventh embodiment, the detected movement of a specific object is used as residence possibility information, and the number of specific objects to be abstracted is adjusted based on the residence possibility information, and then the silhouette image is An example of generation is shown. In this embodiment, an example will be described in which the number of specific objects is adjusted using information other than the movement of specific objects as residence possibility information, and a silhouette image is generated. Hereinafter, parts that are different from Embodiments 1 to 7 will be mainly described, and components and processes that are the same or equivalent to Embodiments 1 to 7 will be denoted by the same reference numerals, and redundant explanation will be omitted.

BACKGROUND ART In places such as platforms, stations, museums, art galleries, shopping malls, stadiums, and other facilities where people gather, for some reason, people may rush in and make it difficult to use the facilities smoothly. In such a case, information (usage information) indicating that smooth use has become difficult is issued. Specific examples of usage information include information on the operation of trains and other means of transportation (whether or not there will be delays, delay times, estimated time until operations resume, etc.), information on whether or not there are restrictions on entry to platforms, stations, museums, and other facilities, Examples include information on waiting times to enter the facility. If the usage information is sent out, it may not be possible to use it smoothly, so there is a possibility that stagnation will occur. In particular, if information indicating that waiting times will be longer is sent out, the degree of congestion will increase over time, which may eventually lead to people staying behind. Therefore, in this embodiment, an example is shown in which the adjustment unit 1000 performs adjustment using the usage information as retention possibility information.

Image processing according to this embodiment will be described below with reference to FIG. 13. Note that the processing from S1101 to S1104 shown in FIG. 13 is the same as that described in FIG. 11, and therefore the description thereof will be omitted. In S1300 in FIG. 13, communication unit 200 receives the above usage information. Note that, in S1300, the communication unit 200 in this embodiment receives operation information as usage information. Next, in S1301, the adjustment unit 1000 sets some of the detected persons 1201 to 1209 as abstraction targets based on the operation information received in S1300, thereby increasing the number of people to be abstracted. Make adjustments. Note that the processing from S1107 to S1111 shown in FIG. 13 is the same as that described in FIG. 11, and therefore the description thereof will be omitted.

Note that not only are there various types of usage information, but even the same usage information may require different responses depending on the location where the camera is installed or the station. For example, even if information about the same delay time is received, the congestion situation that occurs differs depending on the station. Therefore, the usage information received in S1300 may be determined based on predetermined criteria, and the adjustment in S1301 may be performed based on the determination result. As a result, it becomes possible to generate a silhouette image based on the results of performing different adjustments from the same usage information.

Examples of adjustments based on usage information in this embodiment include the following method. In other words, if the usage information is operational information such as train delays, or information indicating that admission is being restricted, it will be abstracted so that the congestion is reflected in the silhouette image. Avoid adjusting the number of people. On the other hand, if the usage information has not been issued or the information indicates normal times, the number of people will be adjusted. Furthermore, if the usage information includes the delay time or the expected time until operation resumes, the longer the delay time or the expected time, the higher the possibility of congestion, so the reduction rate by adjusting the number of people will be smaller. I will make it happen. In this way, if there is a high possibility that congestion will occur where people cannot move, a silhouette image will be generated that reflects the actual congestion, and if the congestion is such that people can move, it will be possible to create a silhouette image that reflects the actual congestion. It is possible to generate a silhouette image that allows easy movement. In addition, in this embodiment, an example was shown in which the number of people to be abstracted is adjusted based on the above-mentioned usage information, but the present invention is not limited to this. For example, in addition to the usage information, the movement of the person's object described in S1105 of FIG. 11 in the seventh embodiment may also be considered.

(Embodiment 9)
In the seventh and eighth embodiments, a method of reducing the number of abstraction targets has been described as an adjustment process performed by the adjustment unit 1000 in S1106 or S1301. In this embodiment, another adjustment method will be described.

When congestion occurs and the density of people increases, the number of people overlapping each other increases, so depending on the method of person detection performed in S1104, it may be difficult to capture human characteristics, and the number of detected people may decrease. . For example, when using a matching pattern, a person area to be matched may be hidden. In this case, as a result of the extraction of the foreground region performed in S1103 in the plurality of consecutive input images, the number of people detected decreases even though the area of the foreground region does not decrease. Therefore, if the number of detected people decreases even though the foreground area does not decrease between multiple consecutive input images, it is determined that there is a possibility of congestion that makes it difficult to detect people, and abstract Adjust to increase the number of people targeted. By doing this, it is possible to generate a silhouette image that shows a situation where people are so crowded that they cannot move, even though there are people whose detection fails. Become.

A specific example of determining whether to make adjustments to increase the number of people to be abstracted may be as follows. That is, when the foreground region has a predetermined area or more, the adjustment unit 1000 adjusts the number of people to be abstracted when the detected number of people decreases by a predetermined amount or more compared to the ratio of the detected number of people in past images to the foreground region. It may be determined that the adjustment should be made in the direction of increasing the value. Note that the number of people detected and the ratio of the foreground area at the time of this determination may be determined dynamically at the time of execution, or the results measured in advance may be fixedly held.

Further, in the above example, the foreground region is used for determination, but the present embodiment is not limited to this. For example, edges in the background image and input image are detected, and the magnitude of change occurring in the image is determined based on the degree of similarity between the detected edges. If the number of people detected is decreasing even though the magnitude of this change is greater than a predetermined value, adjustments may be made to increase the number of people to be abstracted. Any other method may be used as long as it is capable of capturing temporal changes in the image.

In addition, when increasing the number of people to be abstracted, the method of inserting people is to insert the person at a position where a change occurs in the image and to fill the space between the detected people. It is a good idea to decide. Also, the size of the person to be added should be determined based on the size of the people detected in the surrounding area, but at that time, the size of the person may change greatly depending on the position due to the influence of the imaging angle of view. This change may be taken into consideration. Furthermore, regarding the number of abstraction targets to be increased, it is preferable to increase the number of abstraction targets until the aforementioned gaps between detected persons are filled. Alternatively, the number of people may be increased to a level that visually indicates that congestion has occurred, such as increasing the number of people to the maximum number of people detected within a predetermined time, or increasing the number of people until a predetermined number of people exceeds the maximum number of people. Various methods are applicable.

After adjusting the number of people as described in this embodiment, by generating the silhouette image as described in the previous embodiment, the image allows the viewer to understand the congestion situation in a general manner under various situations. It becomes possible to generate.

(Other embodiments)
Next, with reference to FIG. 14, the hardware configuration of the image processing apparatus 100 for realizing each function of each embodiment will be described. Note that although the hardware configuration of the image processing device 100 will be described in the following description, it is assumed that the recording device 120 and the imaging device 110 are also realized by the same hardware configuration.

The image processing device 100 in this embodiment includes a CPU 1400, a RAM 1410, a ROM 1420, an HDD 1430, and an I/F 1440.

The CPU 1400 is a central processing unit that centrally controls the image processing apparatus 100. RAM 1410 temporarily stores computer programs executed by CPU 1400. Further, the RAM 1410 provides a work area used when the CPU 1400 executes processing. Further, the RAM 1410 functions, for example, as a frame memory or as a buffer memory.

The ROM 1420 stores programs and the like for the CPU 1400 to control the image processing apparatus 100. The HDD 1430 is a storage device that records image data and the like.

The I/F 1440 communicates with external devices via the network 140 according to TCP/IP, HTTP, or the like.

In addition, in the description of each embodiment mentioned above, an example will be described in which the CPU 1400 executes the processing, but at least a part of the processing of the CPU 1400 may be performed by dedicated hardware. For example, the process of displaying a GUI (GRAPHICAL USER INTERFACE) or image data on the display 130 may be executed by a GPU (GRAPHICS PROCESSING UNIT). Further, the process of reading the program code from the ROM 1420 and expanding it to the RAM 1401 may be executed by a DMA (DIRECT MEMORY ACCESS) functioning as a transfer device.

Note that the present invention can also be implemented by a process in which one or more processors read and execute a program that implements one or more of the functions of the embodiments described above. The program may be supplied to a system or device having a processor via a network or a storage medium. The present invention can also be implemented by a circuit (eg, an ASIC) that implements one or more of the functions of the embodiments described above. Further, each part of the image processing apparatus 100 may be realized by the hardware shown in FIG. 14, or may be realized by software.

Note that in each of the embodiments described above, the specific object that is the object of concealment (object of abstraction) is described as a person, but the present invention is not limited to this. For example, if it is an object that is not visible in the background image and is useful for understanding the situation in the imaging range, the object is detected in the same way as human detection, and the above processing is applied to create a silhouette image. An output image may be generated and superimposed on the background image. Objects useful for understanding the situation include various objects, such as large travel bags that tend to get in the way, and barricades set up at construction sites to indicate that entry is prohibited.

Note that another device may have one or more functions of the image processing device 100 according to each embodiment described above. For example, the imaging device 110 may have one or more functions of the image processing device 100 according to each embodiment. Note that the above-described embodiments may be combined, for example, any combination of the above-described embodiments may be implemented.

Although the present invention has been described above along with the embodiments, the above embodiments are merely examples of implementation of the present invention, and the technical scope of the present invention is interpreted to be limited by these embodiments. It's not a thing. That is, the present invention can be implemented in various forms without departing from its technical idea or main features. For example, a combination of each embodiment is also included in the disclosure content of this specification.

100 Image processing device 110 Imaging device 200 Communication unit 201 Storage unit 202 Display control unit 203 Operation reception unit 204 Extraction unit 205 Detection unit 206 Setting unit 207 Generation unit 708 Judgment unit 1000 Adjustment unit

Claims (13)

Extracting means for extracting a foreground region from the captured input image;
detection means for detecting a specific object from the input image;
Setting means for setting a specific area, at least a part of which is a curved area, based on the position of the specific object detected by the detection means;
The foreground area other than the specific area set by the setting means is not superimposed on a predetermined image corresponding to the input image, but a silhouette image is created by abstracting the foreground area in the specific area set by the setting means. , generating means for generating an output image superimposed on the predetermined image;
determining means for determining a crowded area in the input image,
The image processing apparatus is characterized in that the setting means sets a specific area whose horizontal width is reduced by a predetermined magnification for a person located in the crowded area determined by the determination means. The determining means determines that among divided regions obtained by dividing the entire area of the input image into a plurality of divided regions, a divided region in which the number of specific objects detected by the detecting means is equal to or greater than a threshold is a crowded region. The image processing device according to claim 1. The image processing apparatus according to claim 1, wherein the determining unit determines that the foreground area extracted by the extracting unit is a crowded area when the size of the foreground area is equal to or larger than a threshold value. The generating means may vary the display mode of the silhouette image, which is an abstraction of the foreground area in the specific area, for each specific area set by the setting unit for a person located in the congested area determined by the determining unit. The image processing apparatus according to any one of claims 1 to 3, characterized in that: 5. The image processing device according to claim 1, wherein the predetermined image is a background image used to extract the foreground region. The image processing apparatus according to claim 5 , wherein the extraction means extracts the foreground region by comparing a captured input image and the background image. 7. The extraction means extracts the foreground region by comparing a difference value calculated between each pixel of the input image and each pixel of the background image with a threshold value. The image processing device described in . The image processing apparatus according to any one of claims 1 to 7, wherein the specific object is a person. 9. The shape of the specific area set by the setting means is a shape including a first shape corresponding to at least the head of the person and a second shape corresponding to the torso of the person. The image processing device described. The generating means sets a display mode of an outer edge of the specific area that overlaps with the foreground area extracted by the extracting unit to be different from a display mode of a silhouette image in which the foreground area in the specific area is abstracted. The image processing apparatus according to any one of claims 1 to 9. The generation means generates a silhouette image that abstracts the foreground region in a specific region for some specific objects among the plurality of specific objects detected by the detection means, based on the amount of movement in the input image. 11. The image processing apparatus according to claim 1, wherein the image processing apparatus generates an output image in which the predetermined image is superimposed on the predetermined image. an extraction step of extracting a foreground region from the captured input image;
a detection step of detecting a specific object from the input image;
a setting step of setting a specific region, at least a part of which is a curved region, based on the position of the specific object detected in the detection step;
The foreground area other than the specific area set in the setting step is not superimposed on a predetermined image corresponding to the input image, but a silhouette image is created by abstracting the foreground area in the specific area set in the setting step. , a generation step of generating an output image superimposed on the predetermined image, and a determination step of determining a crowded area in the input image,
The method for controlling an image processing apparatus is characterized in that the setting step sets a specific area whose horizontal width is reduced by a predetermined magnification for a person located in the crowded area determined in the determining step. A program for causing a computer to function as each means of the image processing apparatus according to claim 1.

Images