JP2024035625A - Image processing device, image processing method - Google Patents

Abstract

An object of the present invention is to provide a technique for improving the accuracy of analysis of a specific region in a captured image and reducing the time required for the analysis. SOLUTION: A captured image is divided into a plurality of divided regions based on the distribution of human head sizes in the captured image. When a plurality of human bodies are located in a frame of a specific area in the captured image, each estimated head region of the plurality of human bodies is specified based on the distribution, and the head regions of the plurality of human bodies are included. Find the containing area. A first region included in the inclusive region among the plurality of divided regions and a second region not included in the inclusive region among the plurality of divided regions are displayed in a distinguishable manner. [Selection diagram] Figure 5

Description

The present invention relates to image processing technology.

In recent years, an intrusion detection system has been proposed that detects a human body intruding into a specific area (hereinafter referred to as an intrusion area) by photographing a predetermined area with an imaging device and analyzing the photographed image. An example of a use case for such an intrusion detection system is the detection of a person approaching a monitored object.

When detecting an intrusion, if the human body is hidden by an object to be monitored or an obstacle, this will affect image analysis and may prevent an accurate determination of intrusion. Therefore, by using image analysis to detect the head of a human body instead of the whole body, and estimating the foot position from the detection results of the head, we can detect intrusion based on the foot position even if the human body is hidden. is possible. Furthermore, there is a technique that reduces the time required for analysis processing by limiting the objects to be analyzed, rather than analyzing the entire image. For example, Patent Document 1 discloses a technique for enlarging a region set by a user based on size information of an object to be detected and performing detection processing on the region.

Japanese Patent Application Publication No. 2020-135609

In intrusion detection as well, by limiting the objects of analysis, the time required for analysis processing can be shortened. However, when analyzing the intrusion area to reduce the time required for analysis processing, the foot position is estimated from the head detection result, so if the foot is within the intrusion area but the head is outside the intrusion area, Intrusions may be overlooked. Further, in the technique disclosed in Patent Document 1, the area to be analyzed is enlarged based on the size information of the object, but the above-mentioned oversight cannot be avoided by only enlargement. The present invention provides a technique for improving the accuracy of analysis of a specific region in a captured image and reducing the time required for the analysis.

One aspect of the present invention provides a dividing unit that divides a captured image into a plurality of divided regions based on a distribution of head sizes of human bodies in the captured image, and a plurality of human bodies located in a frame of a specific region in the captured image. a processing means for determining a head region of each of the plurality of human bodies estimated when It is characterized by comprising a display control means for displaying a first area included in the inclusive area and a second area that is not included in the inclusive area among the plurality of divided areas in a distinguishable manner. do.

According to the present invention, it is possible to improve the accuracy of analysis of a specific region in a captured image and to reduce the time required for the analysis.

FIG. 1 is a block diagram showing an example of a system configuration. FIG. 1 is a block diagram showing a configuration example of an imaging device 110. 1 is a block diagram showing an example of a functional configuration of an imaging device 110. FIG. 1 is a block diagram showing an example of a hardware configuration of a server 130. FIG. FIG. 2 is a block diagram showing an example of a functional configuration of a server 130. FIG. The figure which shows the example of division of a captured image. (a) is a diagram showing an example of a captured image 701, (b) is a diagram showing an analysis region, and (c) is a diagram showing an example of display of the captured image. 5 is a flowchart of processing performed by the server 130. FIG. 2 is a block diagram showing an example of a functional configuration of a server 130. FIG. 5 is a flowchart of processing performed by the server 130. (a) is a diagram showing an example of a captured image 1101, (b) is a diagram showing an analysis region, and (c) is a diagram showing an example of display of the captured image.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

[First embodiment]
In this embodiment, a system for detecting intrusion of a human body into a specific area in a captured image will be described. As shown in the block diagram of FIG. 1, the system according to this embodiment includes an imaging device 110 and a server 130, and is configured such that the imaging device 110 and the server 130 can communicate with each other via a network 120. There is.

First, the imaging device 110 will be explained. The imaging device 110 is an imaging device such as a network camera, a notebook PC with a camera, a tablet terminal device with a camera, or a smartphone, and captures moving images or periodically or irregularly captures still images. When capturing a moving image, the imaging device 110 uses each frame of the moving image as a captured image, processes the captured image, and/or transmits the captured image to the server 130 via the network 120. On the other hand, when capturing still images regularly or irregularly, the imaging device 110 uses the still images as captured images, processes the captured images, and/or sends the captured images to the server 130 via the network 120. Send to.

In this embodiment, the imaging device 110 will be described as having a configuration capable of executing various types of arithmetic processing. However, the present invention is not limited to this. For example, it is assumed that an external device such as a PC (personal computer) connected to the imaging device 110 is caused to perform the arithmetic processing, and the imaging device 110 operates according to the result of the arithmetic processing. Also good. Further, the imaging device 110 may have a zoom function and a pan head for controlling panning and tilting.

Next, the server 130 will be explained. The server 130 is a computer device such as a PC, a tablet terminal device, or a smartphone.

Next, the network 120 will be explained. The network 120 includes a plurality of routers, switches, cables, etc. that meet communication standards such as, for example, Ethernet (registered trademark). In this embodiment, the network 120 may be any network that enables communication between the imaging device 110 and the server 130, and may be constructed in any size and configuration, and according to a compliant communication standard. For example, the network 120 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), or the like. Further, the network 120 may be configured to enable communication using a communication protocol that conforms to the ONVIF (Open Network Video Interface Forum) standard, for example. However, the present invention is not limited thereto, and the network 120 may be configured to enable communication using other communication protocols, such as a proprietary communication protocol.

Note that in FIG. 1, the number of imaging devices 110 and servers 130 is set to one in order to simplify the following explanation. However, the number of imaging devices 110 and servers 130 is not limited to one, and is not limited to a specific number.

Next, a configuration example of the imaging device 110 will be described using the block diagram of FIG. 2. In the following, it is assumed that each functional unit in FIG. 2 is implemented by hardware, but for example, some or all of the functions of the image processing unit 202 and the arithmetic processing unit 203 are implemented by software (computer program). You may do so. In that case, the software is stored in a memory (not shown) of the imaging device 110 and executed by a controller (not shown) of the imaging device 110, thereby realizing the functions of the corresponding functional units.

The imaging unit 201 includes a lens unit for forming an image of light, an image sensor that converts the light imaged by the lens unit into an analog signal, and an A/D conversion unit that converts the analog signal into a digital signal (image signal). It has a department. The lens section has a zoom function that adjusts the angle of view, an aperture function that adjusts the amount of light, and the like. The image sensor has a gain function that adjusts sensitivity when converting light into an analog signal. These functions are adjusted based on the setting values notified from the image processing unit 202.

The image processing unit 202 includes an image processing engine, its peripheral devices, and the like. Peripheral devices include, for example, RAM (Random Access Memory), drivers for each I/F (interface), and the like. The image processing unit 202 performs image processing such as development processing, filter processing, sensor correction, and noise removal on the image signal acquired from the imaging unit 201 to generate a captured image. Further, the image processing unit 202 can notify the setting value to the lens unit and the image sensor, and can perform exposure adjustment so that a captured image with an appropriate exposure can be obtained.

The arithmetic processing unit 203 includes one or more processors such as a CPU or MPU, memory such as a RAM or ROM, drivers for each I/F, and the like. Note that CPU is an acronym for Central Processing Unit, MPU is for Micro Processing Unit, RAM is for Random Access Memory, and ROM is for Read Only Memory. The arithmetic processing unit 203 performs various operational controls in the imaging device 110.

The distribution unit 204 includes a network distribution engine and peripheral devices such as a RAM and an ETH PHY module. The ETH PHY module is a module that executes Ethernet physical (PHY) layer processing. The distribution unit 204 converts the captured image acquired from the image processing unit 203 and the processing result by the arithmetic processing unit 203 into a format that can be distributed to the network 120, and outputs the converted data to the network 120.

An example of the functional configuration of the imaging device 110 will be described using the block diagram of FIG. 3. In the following, it is assumed that each of the functional units in FIG. It may also be implemented using a program. In that case, the software is stored in the storage unit 303 and executed by the control unit 305, thereby realizing the functions of the corresponding functional units.

The imaging control unit 301 executes control to cause the imaging unit 201 to image the surrounding environment. The signal processing unit 302 performs the above image processing on the image signal acquired from the imaging unit 201 to generate a captured image. The signal processing unit 302 encodes the captured image, for example. If the captured image is a still image, the signal processing unit 302 encodes the still image using an encoding method such as JPEG (Joint Photographic Experts Group). Further, if the captured image is each frame of a moving image, the signal processing unit 302 processes H. Encoding is performed using an encoding method such as H.264/MPEG-4 AVC (hereinafter referred to as "H.264") or HEVC (High Efficiency Video Coding). Further, the signal processing unit 302 encodes the captured image using, for example, an encoding scheme selected by the user via an operation unit (not shown) of the imaging device 110 from among a plurality of preset encoding schemes. may be converted into The storage unit 303 is a memory for temporarily storing data. The communication unit 304 communicates with the server 130 via the network 120. The control unit 305 controls the overall operation of the imaging device 110.

Next, an example of the hardware configuration of the server 130 will be described using the block diagram of FIG. 4.

The processor 401 is a processing unit such as a CPU, MPU, or GPU, and executes various processes using computer programs and data stored in the RAM 402 and ROM 403. As a result, the processor 401 controls the overall operation of the server 130, and also executes or controls various processes described as the processes performed by the server 130.

RAM 402 has an area for storing computer programs and data loaded from ROM 403 and HDD (hard disk drive) 404, and an area for storing data received from imaging device 110 via I/F 405. Further, the RAM 402 has a work area used by the processor 401 to execute various processes. In this way, the RAM 402 can provide various areas as appropriate.

The ROM 403 stores configuration data for the server 130, computer programs and data related to starting the server 130, computer programs and data related to the basic operations of the server 130, and the like.

The HDD 404 stores an OS (operating system), computer programs and data for causing the processor 401 to execute or control various processes described as processes performed by the server 130 . Computer programs and data stored in the HDD 404 are loaded into the RAM 402 as appropriate under the control of the processor 401 and are subject to processing by the processor 401.

I/F 405 is a communication interface for performing data communication with imaging device 110 via network 120.

The operation unit 406 is a user interface such as a keyboard, a mouse, a touch panel, etc., and can be operated by a user to input various instructions to the processor 401. Note that the operation unit 406 may be a microphone, in which case the operation unit 406 can collect the voice uttered by the user, and the processor 401 can receive a corresponding instruction by recognizing the voice.

The display unit 407 has a liquid crystal screen or a touch panel screen, and can display processing results by the processor 401 in the form of images, characters, and the like. Note that the display unit 407 may be a projection device such as a projector that projects images and characters.

The processor 401, RAM 402, ROM 403, HDD 404, I/F 405, operation section 406, and display section 407 are all connected to a system bus 408. Note that the configuration shown in FIG. 4 is only an example of a hardware configuration applicable to the server 130, and can be modified/changed as appropriate.

An example of the functional configuration of the server 130 is shown in the block diagram of FIG. In the following, it is assumed that each functional unit shown in FIG. 5 is implemented by software (computer program). This computer program is stored in the HDD 404, and is read out to the RAM 402 as necessary under the control of the processor 401. Then, when the processor 401 executes the computer program read into the RAM 402, the functions of the corresponding functional units are realized. However, one or more of the functional units shown in FIG. 5 may be implemented in hardware.

Although the functional units in FIG. 5 will be described below as the main body of processing, in reality, the functions corresponding to the functional units are realized by the processor 401 executing a computer program corresponding to the functional units.

Next, a process performed by the server 130 to detect an intrusion of a human body into a designated area designated in an image captured by the imaging device 110 will be described according to the flowchart of FIG. 8.

In step S801, the setting unit 504 obtains the head size distribution (head size distribution setting value) stored in the HDD 404. The head size distribution is generated in advance by the following process and stored in the HDD 404.

The image acquisition unit 501 acquires a captured image transmitted from the imaging device 110 via the network 120. In order to simplify the explanation, the explanation below will be based on the assumption that an unencoded captured image is transmitted from the imaging device 110.

Then, the distribution acquisition unit 503 generates a distribution of head sizes assumed at each position on the captured image acquired by the image acquisition unit 501 as a head size distribution. For example, the user operates the operation unit 406 to specify head sizes at various positions on the captured image. For a position where no head size is specified in the captured image, the distribution acquisition unit 503 estimates the head size at the position by interpolation from the head size specified in the vicinity of the position. Furthermore, when the human head is shown at various positions in the captured image, the distribution acquisition unit 503 calculates the size of the head at each position (for example, the size of a rectangle that includes the head). . For a position in the captured image for which the head size has not been determined, the distribution acquisition unit 503 estimates the head size at the position by interpolation from the head size determined in the vicinity of the position.

The method for estimating the head size by interpolation assumes that, for example, when the head size at the position (x, y) on the captured image is s, s can be expressed by x, y, and one or more unknown parameters. do. For example, assume that s=ax+by+c. In this example, the unknown parameters are a, b, and c. The distribution acquisition unit 503 uses a set of head positions and sizes specified by the user or a set of face (head) positions and sizes detected by face detection to calculate the unknown parameters, for example, by using the set of head positions and sizes specified by the user. It can be determined by statistical processing such as multiplication.

In this way, the distribution acquisition unit 503 generates information representing the head size at each position on the captured image as a head size distribution. Then, the distribution acquisition unit 503 stores the head size distribution generated in this manner in the HDD 404. In step S801, the setting unit 504 acquires the head size distribution generated in this way and stored in the HDD 404.

The acquisition unit 505 also acquires human body size information (set value of human body size information) from the HDD 404, which is information for obtaining the whole body size of the human body from the head size of the human body. By multiplying the head size (vertical size) of a human body by a prescribed vertical magnification, the vertical size of a rectangle (human body frame) that includes the human body can be obtained. Furthermore, by multiplying the head size (horizontal size) of a human body by a prescribed horizontal magnification, the horizontal size of a rectangle (human body frame) that includes the human body can be obtained. The acquisition unit 505 acquires this vertical magnification and horizontal magnification as human body size information. The human body size information may be information input by the user by operating the operation unit 406, or may be determined from the average head size and average height of the human body based on the angle of the imaging device. The method for obtaining the information is not limited to a specific method.

Further, the determining unit 506 acquires from the HDD 404 a setting value that defines a “designated area designated as an area (intrusion area) where intrusion of a human body is detected” in the captured image. The setting value may be, for example, a setting value that defines a designated area specified on the captured image by the user operating the operation unit 406, or a setting value that defines an area where a specific object is detected from the captured image. It may be.

Here, there are cases where the designated area is set based on the "position of the feet of the human body intruding into the intrusion area", and in such cases, conventionally, intrusion into the designated area is accurately determined as described above. That is difficult. In contrast, in the present embodiment, a region that takes into account the head region of a human body is obtained as a transformation region, and intrusion into the specified region is determined only for the person whose head region is detected from the transformation region. Thereby, the intrusion determination can be performed with higher accuracy and faster than conventional methods.

In step S802, the setting unit 504 divides the captured image acquired from the imaging device 110 by the image acquisition unit 501 into a plurality of divided regions based on the head size distribution acquired in step S801 (the captured image is divided into a plurality of divided regions). (setting the area). In machine learning models, there may be a range of high detection accuracy relative to the ratio of detection targets to input images. By dividing the captured image into a plurality of divided regions based on the head size distribution and inputting the divided regions to the head detection unit 502, it can be expected to obtain results with high detection accuracy.

As a method of dividing a captured image into multiple divided regions, for example, assuming that the change in head size in the horizontal direction (x direction) is negligibly small in the head size distribution, The captured image is divided into a plurality of divided regions based on the change in head size in the y direction). FIG. 6A shows an example in which a captured image is divided in the y direction. In FIG. 6A, the change in head size in the x direction is negligibly small, and the head size changes in three stages in the y direction. Therefore, in FIG. 6A, a divided area 605 that includes the topmost head 602, a divided area 606 that includes the middle head 603, and a divided area 607 that includes the bottommost head 604 are set. There is. The divided regions 605, 606, and 607 are arranged so that the ratio occupied by the head 602 in the divided region 605, the ratio occupied by the head 603 in the divided region 606, and the ratio occupied by the head 604 in the divided region 607 are all the same. size is determined. Then, as shown in FIG. 6(b), by arranging each of the divided regions 602, 603, and 604 in the x direction, the uppermost region row in which regions having the size of the divided region 602 are arranged in the x direction, and the divided region 603 A middle region row in which regions having the same size are arranged in the x direction, and a bottom region row in which regions having the size of the divided region 604 are arranged in the x direction are obtained. At this time, the area that protrudes outside the captured image may be set so that it overlaps with other areas and falls within the captured image. Then, each area shown in FIG. 6(b) is defined as a divided area, and the captured image 601 is divided into divided areas.

In step S803, the determining unit 506 specifies (determines) a divided region from which the head is to be detected as an analysis region from among the plurality of divided regions obtained in step S802. First, the determining unit 506 determines the head of the human body that is estimated when the human body is located in the frame of the designated area (the area defined by the "setting value defining the designated area" acquired in step S801) in the captured image. The head area is determined based on the head size distribution. This process will be explained using FIG. 7(a) as an example.

FIG. 7A shows an example of a captured image 701 that is divided into a plurality of divided regions. The determining unit 506 determines the head region 707 when the human body 703 stands upright with position P1 of the upper left corner of the designated region 702, which is an example of the frame of the designated region 702, as the position of the feet. Further, the determining unit 506 determines the head region 710 when the human body 706 stands upright with position P2 of the lower left corner of the designated region 702, which is an example of the frame of the designated region 702, as the foot position. Further, the determining unit 506 determines the head region 708 when the human body 704 stands upright with the position P3 of the upper right corner of the designated region 702, which is an example of the frame of the designated region 702, as the foot position. Further, the determining unit 506 determines the head region 709 when the human body 705 stands upright with position P4 at the lower right corner of the designated region 702, which is an example of the frame of the designated region 702, as the foot position.

More specifically, the determining unit 506 sets the divided region Rn to which the position Pn (n=1, 2, 3, 4) belongs as the target divided region, and calculates the head size S corresponding to the target divided region from the head size distribution. get. Then, the determining unit 506 enlarges the rectangle of the head size S horizontally by the "horizontal magnification indicated by the human body size information" and expands (extends) the rectangle downward by the "vertical magnification indicated by the human body size information". Obtained as a human body frame. Then, the determining unit 506 aligns the lower side of the human body frame with the position Pn, and then determines the head area in the human body frame (which can be specified by the inverse calculation of the above calculation for determining the human body frame from the rectangle of head size S). It is specified as "the head region of the human body when the human body stands upright with position Pn as the position of the feet."

Note that if the human body frame protrudes upward from the target divided area when the lower side of the human body frame is aligned with the position Pn, the determining unit 506 divides the divided area one above the target divided area into the target divided area. The above process is performed again as a region to obtain a new human body frame. Then, the determining unit 506 aligns the lower side of the newly obtained human body frame with the position Pn, and then defines the head region of the human body frame as "the head of the human body when the human body stands upright with position Pn as the position of the feet". specified as “part area”.

Note that there are various methods for determining the estimated head area of a human body when the human body is located in the frame of a designated area in a captured image based on the head size distribution, and the method is not limited to the above method. .

Then, the determining unit 506 determines a containing area that includes the head areas 707 to 710 as a conversion area 711. For example, the determining unit 506 determines as the converted area 711 an area obtained by converting the designated area 702 so as to include the head areas 707 to 710. The transformation includes, for example, one or more of processing such as translation, enlargement, reduction, and rotation.

Then, as shown in FIG. 7B, the determining unit 506 determines (specifies) a divided area included in the conversion area 711 among the divided areas in the captured image 701 as an analysis area 712. In FIG. 7B, if even a part of the divided area is included in the conversion area 711, the divided area is set as the analysis area. However, for example, a divided region whose proportion included in the transformed region 711 is equal to or higher than a specified ratio may be used as the analysis region, and the method of specifying the analysis region based on the divided region and the transformed region is not limited to a specific method.

The determining unit 506 then generates information indicating the position and size of each divided area in the captured image, information indicating the analysis area among the divided areas (for example, identification information of the analysis area among the identification information of each divided area). ), is generated as area setting information.

In step S804, the image acquisition unit 501 acquires the captured image transmitted from the imaging device 110 via the network 120. Then, the display control unit 509 specifies each divided area in the captured image from the above area setting information, and further specifies an analysis area among the identified divided areas from the above area setting information. Then, the display control unit 509 displays the captured image on the display unit 407, and at this time, displays the analysis area and the non-analysis area (divided areas other than the analysis area) in a distinguishable manner. A display example of the captured image is shown in FIG. 7(c).

FIG. 7C is a diagram showing a display example of a GUI (graphical user interface) 713 for displaying a captured image. The display control unit 509 performs display of such a GUI and controls the display of the GUI in response to user operations.

In FIG. 7C, the analysis region 712 in the captured image 799 acquired in step S804 is displayed in a different color from the non-analysis region 714. Note that there are various methods for displaying the analysis area and the non-analysis area in a distinguishable manner, and the display method is not limited to a specific one. For example, you can display the frame of the analysis area with a dotted line and the frame of the non-analysis area with a solid line, or you can display the frame of the analysis area with a filled color and the frame of the non-analysis area without filling it. You can do it like this. Whether or not the frames of the divided regions (analysis region 712 and non-analysis region 714) are displayed can be switched depending on the user's operation on the button 720. When the user operates the operation unit 406 and instructs “ON” on the button 720, the frame of the divided area is displayed, and when the user operates the operation unit 406 and instructs “OFF” on the button 720, the frame of the divided area is displayed. Not done.

Further, in the captured image 799, a frame of the conversion area 711 is displayed. Whether or not the frame of the conversion area 711 is displayed can be switched depending on the user's operation on the button 721. When the user operates the operation unit 406 and instructs “ON” on the button 721, the frame of the conversion area 711 is displayed, and when the user operates the operation unit 406 and instructs “OFF” on the button 721, the frame of the conversion area 711 is displayed. is not displayed.

Furthermore, the frame of the designated area 702 (the area defined by the "setting value defining the designated area" acquired in step S801) is displayed overlappingly on the captured image 799. Furthermore, the captured image 799 includes human bodies 715, 716, and 717.

Note that the configuration of the GUI shown in FIG. 7(c) is an example, and the configuration is not limited to this. For example, an operation section that can set the above-mentioned setting values may be provided, or a display section that can display information regarding each of the analysis area and non-analysis area may be provided.

In step S805, the head detection unit 502 detects the head of the human body from the analysis region (the analysis region specified in step S804) in the captured image acquired by the image acquisition unit 501 in step S804. In this embodiment, the head detection unit 502 detects the head of a human body from an image using a machine learning model that has been trained in advance to be able to detect the head of a human body from an image. Detection of the head using a machine learning model can be realized, for example, by applying the technology described in the following document.

J. Redmon, A. Farhadi, “YOLO9000: Better Faster Stronger”, Computer Vision and Pattern Recognition (CVPR) 2016.
When the head detection unit 502 detects a human head from the analysis area, it outputs information indicating the position and size of a rectangle that includes the head as head detection information. For example, the head detection unit 502 may output the position of the upper left corner of a rectangle that includes the head and the vertical and horizontal sizes of the rectangle as head detection information, or the position of the upper left corner and the lower right corner of the rectangle. The position may be output as head detection information.

Furthermore, in a machine learning model, the size of input data is determined in advance. Therefore, when a captured image with different input data sizes is given, the head detection unit 502 resizes or deforms the captured image using any method such as the generally known bicubic method. .

In step S806, the intrusion determination unit 507 expands the rectangle having the position and size indicated by the head detection information in the horizontal direction by the "horizontal magnification indicated by the human body size information", and downwardly enlarges the rectangle having the position and size indicated by the head detection information by "the vertical magnification indicated by the human body size information". A rectangle expanded (extended) by '' is found as the human body frame. Then, the display control unit 509 displays the human body frame determined by the intrusion determination unit 507.

In the example of FIG. 7C, the human bodies 715, 716, and 717 are included in the analysis area 712. Therefore, in step S805, the heads of the human bodies 715 and 716 are detected, and in step S806, the human body frame 718 of the human body 715 and the detection frame 719 of the human body 716 are determined. Therefore, in step S806, a human body frame 718 of the human body 715 and a human body frame 719 of the human body 716 are displayed. Since the human body 717 is not included in the analysis area 712, the head of the human body 717 is not detected, and therefore the human body frame of the human body 717 is not determined, so the human body frame of the human body 717 is not displayed.

Then, the intrusion determination unit 507 determines whether or not the obtained human body frame has intruded into the designated area (presence or absence of intrusion). In the present embodiment, the intrusion determination unit 507 determines that "intrusion has occurred" when part or all of the lower side of the human body frame enters the designated area. This makes it possible to determine intrusion based on the position of the human body's feet. Note that the method for determining intrusion is not limited to the above method; for example, it may be determined that an intrusion has occurred when even a portion of the human body frame enters the designated area, or it may be determined that the entire human body frame enters the designated area. It may be determined that there has been an "intrusion" if the

As a result of such determination, if it is determined that there has been an intrusion, the process proceeds to step S808 via step S807, and if it is not determined that there has been an intrusion, the process proceeds to step S807. The process then proceeds to step S809.

In step S808, the display control unit 509 causes the display unit 407 to display information indicating that a human body has entered the designated area, in order to notify that the human body has entered the designated area. For example, the display control unit 509 may display a character string indicating that a human body has entered the designated area. Further, the display control unit 509 may display a human body frame whose lower side partially falls within the designated area as the target human body frame, and display the target human body frame and the non-target human body frame in a distinguishable manner. Further, the display control unit 509 may display information related to the head of the human body corresponding to the target human body frame (human body information such as gender and age estimated by the head detection unit 502 when detecting the head). .

Note that the notification method to the effect that a human body has entered the designated area is not limited to display, and notification may be made by a method such as sound or vibration. For example, the communication unit 510 may transmit a message to other devices to the effect that a human body has entered the designated area. Furthermore, when it is determined that the human body frame has entered the designated area, the communication unit 510 may turn on/blink a light emitting body such as an LED.

In step S809, the control unit 512 determines whether to end the process according to the flowchart of FIG. 8. For example, if the user operates the operation unit 406 and inputs an operation instruction to end the process, the control unit 512 determines to end the process according to the flowchart of FIG. On the other hand, unless such an operation instruction is input, it is not determined that the process according to the flowchart of FIG. 8 is to be terminated.

As a result of such determination, if it is determined that the process according to the flowchart of FIG. 8 should be terminated, the process according to the flowchart of FIG. 8 is terminated. On the other hand, if it is not determined that the process according to the flowchart of FIG. 8 should be ended, the process proceeds to step S810.

In step S810, the control unit 512 determines whether one or more of the various setting values described above has been updated. For example, if the user operates the operation unit 406 to update one or more of the various setting values described above, the control unit 512 determines that there is an “updating”, and selects which setting value. If the information has not been updated, it is determined that there is no update.

As a result of such determination, if one or more of the various setting values described above has been updated, the process advances to step S801. On the other hand, if none of the various setting values described above has been updated, the process advances to step S804.

As described above, according to the present embodiment, the head area of only the person included in the analysis area is specified, and intrusion determination is performed based on the head area with the position of the feet as a reference, so that intrusion of a person may not be overlooked. It is possible to further reduce the time required to detect an intrusion while suppressing the intrusion.

<Modified example>
In the first embodiment, a case was described in which intrusion into a specified area of a human body included in the analysis area is detected, but the use of the analysis area is not limited to detecting intrusion into a specified area, but can be applied to various analysis processes. It is. For example, an embodiment may be considered in which the number of human bodies passing through a designated area is counted by counting the number of heads included in the analysis area. We are also considering an embodiment in which information on the human body passing through a designated area is collected by collecting information on the head included in the analysis area (human body information such as gender and age estimated when the head is detected). obtain.

[Second embodiment]
Differences from the first embodiment will be explained below, and unless otherwise mentioned below, it is assumed that the second embodiment is the same as the first embodiment. In the first embodiment, intrusion determination is performed based on the foot of the human body by determining whether a part of the lower side of the human body frame has entered the designated area. However, if you want to detect an intrusion caused by an action such as peeking, it would be more appropriate to set the analysis area based on the head and detect the intrusion of the head. In this embodiment, setting of an analysis area and intrusion determination according to the site and conditions for intrusion determination will be described.

An example of the functional configuration of the server 130 according to this embodiment is shown in the block diagram of FIG. In FIG. 9, functional parts similar to those shown in FIG. 5 are given the same reference numerals, and a description of the functional parts will be omitted. The configuration shown in FIG. 9 has an acquisition section 901 and an acquisition section 902 added to the configuration shown in FIG. 5.

The acquisition unit 901 acquires the setting value of the intrusion determination part. The setting value of the intrusion determination part is a value indicating a part of the human body frame, such as "upper side" or "lower side," and is a value indicating a part of the human body frame that is the target of intrusion determination. Note that the setting value of the intrusion determination part may be a value indicating a portion having a certain width with respect to the "upper side" and the "lower side", such as "upper" and "lower", respectively. Note that the method of setting the setting value of the intrusion determination part is not limited to a specific setting method.

For example, a rectangle representing a human body frame is displayed on the display unit 407. Then, when the user operates the operation unit 406 to specify the upper side of the rectangle, “upper side” is set as the setting value of the intrusion determination part, and the user operates the operation unit 406 to specify the lower side of the rectangle. In this case, "lower side" is set as the setting value of the intrusion determination part.

For example, an image representing a human body is displayed on the display unit 407. When the user operates the operation unit 406 to specify the head in the image, “upper side” is set as the setting value for the intrusion determination part, and the user operates the operation unit 406 to specify the foot in the image. If specified, "lower side" is set as the setting value for the intrusion determination part.

The setting value of the intrusion determination part set by such a setting method is stored in the HDD 404. The acquisition unit 901 then acquires the setting value of the intrusion determination part stored in the HDD 404.

The acquisition unit 902 acquires the setting value of the intrusion determination condition. The set value of the intrusion judgment condition is a value that indicates how much the intrusion judgment part must enter the specified area to be judged as "intrusion", and the value that the intrusion judgment part must satisfy in order to judge that "intrusion" has occurred. This is a value that indicates a condition. In this embodiment, the intrusion determination condition can be set to "part" or "all." When "partial" is set as the intrusion determination condition, the intrusion determination unit 507 determines that there is an "intrusion" when even a part of the intrusion determination part enters the designated area. On the other hand, when "all" is set as the intrusion determination condition, the intrusion determination unit 507 determines that there is "intrusion" when all of the intrusion determination parts are included in the designated area. Note that the method of setting the intrusion determination condition is not limited to a specific setting method.

For example, the user may operate the operation unit 406 to set the intrusion determination conditions, or the corresponding intrusion determination conditions may be set depending on the type of intrusion detection processing. Further, for example, the intrusion determination condition may be specified as a percentage. In this case, the intrusion determination unit 507 determines that there is an "intrusion" when the proportion of the portion of the intrusion determination part that has entered the specified area exceeds the intrusion determination condition (ratio).

Next, a process performed by the server 130 to detect an intrusion of a human body into a designated area in an image captured by the imaging device 110 will be described according to the flowchart of FIG. 10. In FIG. 10, the same step numbers as the processing steps shown in FIG. 8 are given the same step numbers, and a description of the processing steps will be omitted.

In step S1001, in addition to the processing in step S801, the acquisition unit 901 acquires the setting value of the intrusion determination part, and the acquisition unit 902 acquires the setting value of the intrusion determination condition. In step S1002, the determining unit 506 determines, as the analysis region, a divided region from which the head is to be detected, among the plurality of divided regions obtained in step S802. When “lower side” is set as the setting value of the intrusion judgment part and “part” is set as the setting value of the intrusion judgment condition, the determining unit 506 performs the same as in the first embodiment. to determine the analysis area.

If "upper side" is set as the setting value for the intrusion determination part and "part" is set as the setting for the intrusion determination condition, the determining unit 506 determines the analysis area as follows. do.

First, the determining unit 506 determines the head of the human body that is estimated when the human body is located in the frame of the designated area (the area defined by the "setting value defining the designated area" acquired in step S801) in the captured image. The head area is determined based on the head size distribution. This process will be explained using FIG. 11(a) as an example.

FIG. 11A shows an example of a captured image 1101 that is divided into divided regions. The determining unit 506 acquires the head size corresponding to the divided area including the position P1 of the lower left corner of the specified area 1102, which is an example of the frame of the specified area 1102, from the head size distribution. The determining unit 506 then enlarges the obtained rectangle of the head size horizontally by the "horizontal magnification indicated by the human body size information" and expands (extends) the rectangle downward by the "vertical magnification indicated by the human body size information". is determined as a human body frame. The determining unit 506 then sets human body frames 1111 and 1112 that are in contact with the position P1. The upper left corner of the human body frame 1111 is in contact with the position P1, and the upper right corner of the human body frame 1112 is in contact with the position P1.

Similarly, the determining unit 506 obtains the head size corresponding to the divided region including the upper left corner position P2 of the specified region 1102 from the head size distribution. Then, the determining unit 506 enlarges the rectangle of the obtained head size horizontally by the "horizontal magnification indicated by the human body size information" and expands (extends) the rectangle downward by the "vertical magnification indicated by the human body size information". is determined as a human body frame. The determining unit 506 then sets human body frames 1103 and 1104 that are in contact with the position P2. The upper left corner of human body frame 1104 is in contact with position P2, and the upper right corner of human body frame 1103 is in contact with position P2.

Similarly, the determining unit 506 obtains the head size corresponding to the divided region including the upper corner position P3 of the specified region 1102 from the head size distribution. Then, the determining unit 506 enlarges the rectangle of the obtained head size horizontally by the "horizontal magnification indicated by the human body size information" and expands (extends) the rectangle downward by the "vertical magnification indicated by the human body size information". is determined as a human body frame. The determining unit 506 then sets human body frames 1105 and 1106 that are in contact with the position P3. The upper left corner of human body frame 1106 is in contact with position P3, and the upper right corner of human body frame 1105 is in contact with position P3.

Similarly, the determining unit 506 obtains the head size corresponding to the divided region including the upper right corner position P4 of the specified region 1102 from the head size distribution. Then, the determining unit 506 enlarges the rectangle of the obtained head size horizontally by the "horizontal magnification indicated by the human body size information" and expands (extends) the rectangle downward by the "vertical magnification indicated by the human body size information". is determined as a human body frame. The determining unit 506 then sets human body frames 1107 and 1108 that are in contact with the position P4. The upper left corner of human body frame 1108 is in contact with position P4, and the upper right corner of human body frame 1107 is in contact with position P4.

Similarly, the determining unit 506 obtains the head size corresponding to the divided region including the lower right corner position P5 of the designated region 1102 from the head size distribution. Then, the determining unit 506 enlarges the rectangle of the obtained head size horizontally by the "horizontal magnification indicated by the human body size information" and expands (extends) the rectangle downward by the "vertical magnification indicated by the human body size information". is determined as a human body frame. The determining unit 506 then sets human body frames 1109 and 1110 that are in contact with the position P5. The upper left corner of human body frame 1109 is in contact with position P5, and the upper right corner of human body frame 1110 is in contact with position P5.

Then, the determining unit 506 determines the head regions in each of the human body frames 1103 to 1112 in the same manner as in the first embodiment, and determines a containing region that includes the head regions of the human body frames 1103 to 1112 as a conversion region 1113. For example, the determining unit 506 obtains, as the conversion area 1113, an area obtained by converting the specified area 1102 in the same manner as in the first embodiment so as to include the head area of the human body frames 1103 to 1112. Then, as shown in FIG. 11B, the determining unit 506 determines (specifies) a divided area included in the conversion area 1113 among the divided areas in the captured image 1101 as an analysis area 1114.

Note that when "lower side" is set as the setting value of the intrusion determination part and "part" is set as the setting value of the intrusion determination condition, the determining unit 506 determines the analysis area as follows. It's okay.

The determining unit 506 determines the human body frame corresponding to the position of each vertex of the specified region 1102 in the same manner as in the first embodiment, and determines the human body frame that is in contact with the position (the lower left corner of one human body frame is in contact with the position). (and the lower right corner of the other human body frame is in contact with this position). Then, similarly to the first embodiment, the determining unit 506 sets the rectangle of the head size in each human body frame as "the head region of the human body corresponding to the human body frame". Then, the determining unit 506 determines, as a transformation region, an inclusive region that includes the head region corresponding to all the human body frames, and determines (specifies) the divided region included in the transformation region among the divided regions in the captured image as an analysis region. .

Note that if "lower side" is set as the setting value for the intrusion detection part and "all" is set as the setting value for the intrusion judgment condition, the entire human body frame is set to be included in the specified area. , and thereafter, the analysis area is determined in the same manner as in the first embodiment. In the example of FIG. 7A, the human body frame is set so that the lower edge of the human frame of the person 706 overlaps the lower edge of the specified area 702, and the entire human body frame is included in the specified area 702. Further, the human body frame is set so that the lower side of the human frame of the person 705 overlaps the lower side of the specified area 702 and the entire human body frame is included in the specified area 702. Further, the human body frame is set so that the upper side of the human frame of the person 703 overlaps the upper side of the specified area 702 and the entire human body frame is included in the specified area 702. Further, the human body frame is set so that the upper side of the human frame of the person 704 overlaps the upper side of the specified area 702 and the entire human body frame is included in the specified area 702. After setting the human body frame, the analysis area is determined in the same manner as in the first embodiment.

Note that if "Top Side" is set as the setting value for the intrusion detection part and "All" is set as the setting value for the intrusion judgment condition, the entire human body frame is set to be included in the specified area. , and thereafter, the analysis area is determined in the same manner as in the first embodiment. In the example of FIG. 11A, the human body frame is set so that the lower sides of the human body frames 1109 to 1112 overlap the lower side of the specified area 1113, and the entire human body frames 1109 to 1112 are included in the specified area 1113. . Further, the human body frames are set so that the entire human body frames 1103 to 1108 are located near the corresponding vertices and are included in the designated area 1113. After setting the human body frame, the analysis area is determined in the same manner as in the first embodiment.

As described above, in this embodiment, the estimation is performed when a plurality of human bodies are located in the frame portion of the specified area in the captured image with the arrangement according to the portion to be determined in the frame that includes the human body and the conditions for the determination. The head regions of each of the plurality of human bodies are acquired based on the head size distribution. Then, an inclusive region that includes the head regions of the plurality of human bodies is determined as a transformed region, and divided regions included in the transformed region are specified as analysis regions.

In this embodiment as well, in step S804, like the first embodiment, the display control unit 509 displays the analysis area and the non-analysis area in a distinguishable manner. A display example of the captured image is shown in FIG. 11(c).

FIG. 11C is a diagram showing a display example of the GUI 1115 for displaying a captured image. The display control unit 509 performs display of such a GUI and controls the display of the GUI in response to user operations.

In FIG. 11C, the analysis region 1114 in the captured image 1199 acquired in step S804 is displayed in a different color from the non-analysis region 1116. Further, in the captured image 1199, a frame of the conversion area 1113 is displayed. Furthermore, the frame of the designated area 1102 (the area defined by the "setting value defining the designated area" acquired in step S801) is displayed overlappingly on the captured image 1199. Furthermore, the captured image 1199 includes human bodies 1117 and 1118.

The display/non-display of the divided area frame can be switched using the button 720, as in the first embodiment, and the display/non-display of the conversion area frame can be switched using the button 721, as in the first embodiment. can.

In the example of FIG. 7C, the human body 1117 of the human bodies 1117 and 1118 is included in the analysis region 1114, so in step S805, the head of the human body 1117 is detected, and in step S806, the human body 1117 is included in the analysis region 1114. The human body frame 1119 of 1117 is calculated. Therefore, in step S806, a human body frame 1119 of the human body 1117 is displayed. Since the human body 1118 is not included in the analysis area 1114, the head of the human body 1118 is not detected, and therefore the human body frame of the human body 1118 is not determined, so the human body frame of the human body 1118 is not displayed.

Furthermore, a part of the human body frame 1119 may be highlighted based on the intrusion determination site. In FIG. 11C, the upper side of the human body frame 1119 is highlighted because "upper side" is set as the setting value for the intrusion determination part.

In step S807, the intrusion determination unit 507 performs intrusion detection based on the set value of the intrusion determination condition as described above. In step S808, the display control unit 509 may perform the above-mentioned highlighted display, or may display the setting values of the intrusion determination part and the intrusion determination condition on the display unit 407.

As described above, according to the present embodiment, by setting the intrusion determination part and the intrusion determination condition, it becomes possible to set an analysis area and perform intrusion determination according to each situation.

<Modified example>
When the user operates the operation unit 406 to set a designated area, the display control unit 509 performs the above processing to obtain a transformation area, and if a part of the transformation area protrudes from the captured image, the display control unit 509 calculates the transformation area. The user may be notified by displaying this on the display unit 407 or the like.

[Third embodiment]
In the first embodiment and the second embodiment, the image processing device 130 detects the intrusion of a person into the specified area. In this case, the image processing device 130 transmits the setting value of the designated area, area setting information, the setting value of the intrusion determination part, the setting of the intrusion determination condition, etc. to the imaging device 110. The imaging device 110 determines the analysis region in the captured image captured by itself from the region setting information received from the image processing device 130, and determines the designated region of the human body included in the analysis region (the designated region received from the image processing device 130). (can be determined from the set value).

Further, in the first embodiment and the second embodiment, the human body frame is assumed to be rectangular, but the shape of the human body frame is not limited to a rectangle, but may be an ellipse that encompasses the human body, or may be an ellipse that encompasses the human body. It may be a shape that follows the shape. In this case, the "upper side" and the "lower side" of the human body frame may be respectively referred to as the "upper part (the upper region of the human body frame)" and the "lower part (the lower region of the human body frame)".

In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method/destination/source/storage location, etc. used in each of the above embodiments and modifications are explained in detail. This is provided as an example for purposes of implementation, and is not intended to be limiting.

Further, a part or all of the embodiments and modifications described above may be used in combination as appropriate. Further, a part or all of the embodiments and modifications described above may be selectively used.

(Other embodiments)
The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention of this specification includes the following image processing device, image processing method, and computer program.

(Item 1)
dividing means for dividing the captured image into a plurality of divided regions based on the distribution of the size of the human body's head in the captured image;
When a plurality of human bodies are located in a frame of a specific area in the captured image, each estimated head region of the plurality of human bodies is specified based on the distribution, and the head regions of the plurality of human bodies are included. processing means for determining an inclusive area;
Display control means for displaying a first region included in the inclusive region among the plurality of divided regions and a second region not included in the inclusive region among the plurality of divided regions in a distinguishable manner; An image processing device comprising:

(Item 2)
The processing means specifies, based on the distribution, a head region of each of the plurality of human bodies that is estimated when a lower part of each frame of the plurality of human bodies is located in a frame of a specific region in the captured image. , the image processing apparatus according to item 1, wherein the image processing apparatus calculates an inclusive area that includes head areas of the plurality of human bodies.

(Item 3)
The processing means is configured to determine the area to be determined when a plurality of human bodies are located in a frame portion of a specific area in the captured image in an arrangement according to a portion to be determined in a frame that includes a human body and conditions for the determination. The image processing apparatus according to item 1, wherein head regions of each of a plurality of human bodies are acquired based on the distribution, and an inclusive region that includes the head regions of the plurality of human bodies is determined.

(Item 4)
The image processing apparatus according to any one of items 1 to 3, wherein the processing means converts the specific area to obtain the inclusive area.

(Item 5)
5. The image processing device according to item 4, wherein the transformation includes one or more of translation, enlargement, reduction, and rotation.

(Item 6)
moreover,
When the specific area is set for the captured image in response to a user operation, the inclusive area corresponding to the specific area is determined, and if a part of the determined inclusive area protrudes from the captured image, 6. The image processing device according to any one of items 1 to 5, further comprising means for notifying the user.

(Item 7)
moreover,
The image according to any one of items 1 to 6, further comprising an analysis processing means that detects a head of a human body from the first region and performs analysis processing on the specific region based on the head. Processing equipment.

(Item 8)
Item 7, wherein the analysis processing means detects a head of a human body from the first area, and determines whether or not the human body has entered the specific area based on the head. Image processing device.

(Item 9)
Furthermore,
9. The image processing apparatus according to item 8, further comprising a notification means for notifying that there has been an intrusion when the intrusion has occurred.

(Item 10)
An image processing method performed by an image processing device, the method comprising:
a dividing step in which the dividing means of the image processing device divides the captured image into a plurality of divided regions based on the distribution of human head sizes in the captured image;
The processing means of the image processing device specifies, based on the distribution, a head region of each of the plurality of human bodies estimated when the plurality of human bodies are located in the frame of the specific region in the captured image, and a processing step of determining an inclusive region that includes head regions of a plurality of human bodies;
The display control means of the image processing device selects a first region of the plurality of divided regions that is included in the inclusive region, and a second region of the plurality of divided regions that is not included in the inclusive region; An image processing method comprising: a display control step for displaying identifiably.

(Item 11)
A computer program for causing a computer to function as each means of the image processing apparatus according to any one of items 1 to 9.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

130: Image processing device 501: Image acquisition unit 502: Head detection unit 503: Distribution acquisition unit 504: Setting unit 505: Acquisition unit 506: Determination unit 507: Intrusion determination unit 509: Display control unit 510: Communication unit 512: Control Department

Claims (11)

dividing means for dividing the captured image into a plurality of divided regions based on the distribution of the size of the human body's head in the captured image;
When a plurality of human bodies are located in a frame of a specific area in the captured image, each estimated head region of the plurality of human bodies is specified based on the distribution, and the head regions of the plurality of human bodies are included. processing means for determining an inclusive area;
Display control means for displaying a first region included in the inclusive region among the plurality of divided regions and a second region not included in the inclusive region among the plurality of divided regions in a distinguishable manner; An image processing device comprising: The processing means specifies, based on the distribution, a head region of each of the plurality of human bodies that is estimated when a lower part of each frame of the plurality of human bodies is located in a frame of a specific region in the captured image. , the image processing apparatus according to claim 1, wherein an inclusive region that includes head regions of the plurality of human bodies is determined. The processing means is configured to determine the area to be determined when a plurality of human bodies are located in a frame portion of a specific area in the captured image in an arrangement according to a portion to be determined in a frame that includes a human body and conditions for the determination. 2. The image processing apparatus according to claim 1, wherein head regions of each of a plurality of human bodies are acquired based on the distribution, and an inclusive region that includes the head regions of the plurality of human bodies is determined. The image processing apparatus according to claim 1, wherein the processing means converts the specific area to obtain the inclusive area. The image processing apparatus according to claim 4, wherein the transformation includes one or more of translation, enlargement, reduction, and rotation. moreover,
When the specific area is set for the captured image in response to a user operation, the inclusive area corresponding to the specific area is determined, and if a part of the determined inclusive area protrudes from the captured image, The image processing apparatus according to claim 1, further comprising means for notifying. moreover,
The image processing apparatus according to claim 1, further comprising analysis processing means for detecting a head of a human body from the first region and performing analysis processing on the specific region based on the head. 8. The analysis processing means detects a head of a human body from the first area, and determines whether or not the human body has entered the specific area based on the head. image processing device. Furthermore,
9. The image processing apparatus according to claim 8, further comprising a notification means for notifying that the intrusion has occurred when the intrusion has occurred. An image processing method performed by an image processing device, the method comprising:
a dividing step in which the dividing means of the image processing device divides the captured image into a plurality of divided regions based on the distribution of human head sizes in the captured image;
The processing means of the image processing device specifies, based on the distribution, a head region of each of the plurality of human bodies estimated when the plurality of human bodies are located in the frame of the specific region in the captured image, and a processing step of determining an inclusive region that includes head regions of a plurality of human bodies;
The display control means of the image processing device selects a first region of the plurality of divided regions that is included in the inclusive region, and a second region of the plurality of divided regions that is not included in the inclusive region; An image processing method comprising: a display control step for displaying identifiably. A computer program for causing a computer to function as each means of the image processing apparatus according to claim 1.

Images