JP2018022340A - Image processor, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of accurately estimating the number of specific objects in an image by combining plural person estimation methods.SOLUTION: The image processor includes: first estimation means that estimates the number of specific objects in a first area by carrying out a series of detection processing of the specific object on a first area of an image; second estimation means that estimates the number of the specific objects in a second area by carrying out a series of regression processing to estimate the number of the specific objects in each of predetermined areas on the second area of the image; and integration means that integrates estimation result by the first estimation means and estimation result by the second estimation means.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an information processing method, and a program.

In recent years, there has been proposed a system that measures the number of persons in an image by photographing a predetermined area with an imaging device and analyzing the photographed image. Such a system is expected to be used to eliminate congestion at an event and to guide evacuation during a disaster by detecting congestion in a public space and grasping the flow of people at the time of congestion.
As a method for measuring the number of persons in such an image, there is a method for counting the number of persons detected by the human body detection means (Patent Document 1). Hereinafter, this method is referred to as “detection-based number estimation method”. Further, a method (Non-Patent Document 1) for directly estimating the number of people appearing in a predetermined area of an image using a recognition model obtained by machine learning has been proposed. Hereinafter, this method is referred to as “regression-based number estimation method”.

JP 2007-201556 A

Hiroo Ikeda, Ryoma Ohami, Hiroyoshi Miyano. Increased accuracy of people estimation based on crowd patch learning using CNN. FIT, 2014

The detection-based number estimation method can estimate the number of people with high accuracy when people are present sparsely and appear in a predetermined size or more. However, when humans are present at a high density and most of the human body is hidden, or when the human body appears smaller than a predetermined size, the accuracy of human body detection means deteriorates, so the accuracy of estimating the number of people also deteriorates. There are challenges.
The regression-based number-of-people estimation method can estimate the number of people with higher accuracy than the detection-based number of people estimation method when a person is present at a high density or appears smaller than a predetermined size. On the other hand, the regression-based number estimation method is inferior to the detection-based number estimation method when people are sparsely present and appear in a predetermined size or more.
It is an object of the present invention to estimate the number of specific objects in an image with higher accuracy by combining a plurality of person estimation methods.

  The present invention provides first estimation means for estimating the number of specific objects in the first region by performing a specific object detection process on the first region of the image, and a second region of the image. On the other hand, a second estimation unit that estimates the number of specific objects in the second region by performing a regression process that estimates the number of specific objects for each predetermined region, and a result of estimation by the first estimation unit And integration means for integrating the estimation results by the second estimation means.

  According to the present invention, it is possible to estimate the number of specific objects in an image with higher accuracy by combining a plurality of person estimation methods.

It is a figure which shows an example of the hardware constitutions of an image processing apparatus. It is a figure which shows an example of a function structure of an image processing apparatus. It is a flowchart which shows an example of the image process by an image processing apparatus. It is a figure explaining the method to set a detection area and a regression area. It is a figure which shows the example of the detected human body. It is a figure which shows the example of a division | segmentation into the small area | region of a regression area, and a person estimation example. It is a figure which shows an example of the small area | region divided | segmented at the time of regression-based number estimation. It is a figure which shows the example of the display method. It is a figure which shows the structure in which a human body detector and a regression device share the same convolution layer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
FIG. 1 is a diagram illustrating an example of a hardware configuration of the image processing apparatus 100.
The image processing apparatus 100 includes a CPU 10, a memory 11, a network I / F 12, a display device 13, and an input device 14 as a hardware configuration. The CPU 10 governs overall control of the image processing apparatus 100. The memory 11 stores data, programs, etc. used by the CPU 10 for processing. The input device 14 is a mouse, a button, or the like, and inputs a user operation to the image processing device 100. The display device 13 is a liquid crystal display device or the like, and displays a result of processing by the CPU 10. The network I / F 12 is an interface that connects the image processing apparatus 100 to a network. When the CPU 10 executes processing based on the program stored in the memory 11, the functional configuration of the image processing apparatus 100 shown in FIG. 2 described later and the processing shown in the flowchart of FIG. 3 described later are realized.

FIG. 2 is a diagram illustrating an example of a functional configuration of the image processing apparatus 100.
The image processing apparatus 100 includes an image acquisition unit 101, a region determination unit 102, a first estimation unit 103, a second estimation unit 104, a number integration unit 105, and a display unit 106 as functional configurations.
The image acquisition unit 101 acquires an image that is a target for estimating the number of people.
The area determination unit 102 determines an area for performing detection-based number estimation and an area for performing regression-based number estimation. Hereinafter, a region where detection-based number estimation is performed is referred to as a detection region, and a region where regression-based number estimation is performed is referred to as a regression region.
The first estimation unit 103 performs detection base number estimation for the detection area. Detection-based person estimation is an example of a specific object detection process. A person in the image is an example of a specific object.
The second estimation unit 104 performs regression-based number estimation for the regression region. The regression-based person estimation is an example of a regression process that estimates the number of specific objects for each predetermined area.
The number integration unit 105 integrates the number of people estimation result by the first estimation unit 103 and the number of people estimation result by the second estimation unit 104.
The display unit 106 displays the number estimation result integrated by the number integration unit 105 on the display device 13.

FIG. 3 is a flowchart illustrating an example of image processing by the image processing apparatus 100.
In step S <b> 201, the image acquisition unit 101 acquires an image that is a target for estimating the number of people. The image acquisition unit 101 may acquire an image from a solid-state imaging device such as a CMOS sensor or a CCD sensor, or may acquire an image from a memory 11 such as a hard disk.
In S202, the region determination unit 102 determines a detection region and a regression region. Any method may be used to determine the region.
For example, the first method is a method in which the region determination unit 102 sets both the detection region and the regression region as the entire image.
The second method is a method in which the area determination unit 102 explicitly sets both the detection area and the area where the number of persons is estimated based on a setting operation by the user. The area determination unit 102 determines both the detection area and the area where the number of persons is to be estimated based on, for example, the user's area setting operation via the display device 13 and the input device 14.

The third method is a method in which the region determination unit 102 sets the detection region and the regression region according to the size of the human body. For example, the area determination unit 102 can set an area in which a human body appears in a first predetermined size or more as a detection area and an area in which the human body appears in a second predetermined size or less as a regression area. . FIG. 4A shows an example of an image picked up by an image pickup device installed higher than a predetermined height. A gray object represents a human body. When the imaging device is installed obliquely, as shown in FIG. 4, the near-side human body close to the imaging device is large and the far-side human body away from the imaging device appears small. An area 310 in FIG. 4B is an example of a detection area. A region 320 in FIG. 4C is an example of a regression region.
The size of the human body on the image may be explicitly given to the region determination unit 102 by the user using the input device 14. For example, when the user teaches the average size of the human body at a plurality of points on the image, the region determination unit 102 estimates the average size of the human body at any point on the image by interpolation. be able to.
The region determination unit 102 may automatically estimate the size of the human body on the image by statistical processing. For example, the region determination unit 102 performs human body detection processing on the entire image for a predetermined learning image group, and acquires a set of human body frames indicating the positions of the human bodies. When the size of the human body frame at the coordinates (x, y) on the image is s, it is assumed that s can be expressed by x, y and one or more unknown parameters. For example, assume that s = ax + by + c. In this example, the unknown parameters are a, b and c. The region determination unit 102 can obtain an unknown parameter by a statistical process such as a least square method using a set of human body frames acquired from a predetermined learning image group. By this method, the region determination unit 102 can also estimate the size of the human body in a region where human body detection fails because a person appears small.

The fourth method is a method in which the region determination unit 102 sets a detection region and a regression region according to the density of people. For example, the region determination unit 102 can set a region where the human congestion is a predetermined value or less as a detection region and a region where the human congestion is a predetermined value or more as a regression region. Human congestion can be estimated by any method. For example, the area determination unit 102 can estimate the density according to the ratio of the number of pixels determined to have a moving object with respect to the moving object area obtained from the background difference. Further, for example, the region determination unit 102 can perform motion vector detection in a predetermined range and estimate the density according to the sum of the lengths of the motion vectors.
The determination of the area is not limited to these. For example, the region determination unit 102 may combine the third method and the fourth method, and may determine the region in consideration of both the size of the human body and the density of people. Further, the region determination unit 102 may determine the detection region for the entire screen and determine the regression region using at least one of the third method and the fourth method. The region determination unit 102 may set the detection region and the regression region by arbitrarily combining the methods from the first method to the fourth method. Further, the region determination unit 102 may dynamically change the region according to the time zone and the situation.

In S203, the 1st estimation part 103 performs a human body detection using the various methods of known pattern recognition and machine learning with respect to the detection area | region set in S202. Here, the human body detection refers to a process of specifying the position of a predetermined part such as the face or the whole or a part of the human body. In the following, it is assumed that a human body detector that detects a human head position is used. For example, the first estimation unit 103 extracts a plurality of local patterns using the matching pattern over the entire image while changing the size of the image, and calculates each local feature amount. And the 1st estimation part 103 determines whether it is a human body from the sum total of the result of having weighted the local feature-value. FIG. 5 shows an example of a detected human body. A rectangle 401 represents a human body frame obtained by the human body detection process.
Here, the first estimation unit 103 uses the estimated size of the human body obtained in S202 to limit the change range of the image size, thereby reducing the number of verifications of the verification pattern, and the amount of calculation May be reduced.
Next, the first estimation unit 103 counts the number of detected human bodies to obtain an estimated number of people.

In S204, the second estimation unit 104 directly estimates the number of people using the known regression-based number estimation method for the regression region set in S202. As an example, a method using a regressor that takes a small image of a certain fixed size as input and outputs the number of people shown in the small image will be described. The second estimation unit 104 previously learns the regressor based on a known machine learning method such as support vector machine or deep learning, using a large number of small images with known numbers as learning data. When estimating the number of people, the second estimation unit 104 divides the regression region of the input image into small regions that are units for estimating the number of people. Then, the second estimating unit 104 estimates the number of each small region by resizing each small region to a fixed size to form a small image and using each small image as an input to the regression unit. FIG. 6A shows an example of dividing the regression area into small areas and an example of estimating the number of people. In this example, the second estimation unit 104 divides the regression region 320 into small regions 510 having the same size. Further, in FIG. 6A, the number of persons estimated by using a regressor for each small area 510 is represented numerically inside the small area. The number of persons estimated by the second estimation unit 104 is not necessarily an integer, and may be a real number. The second estimating unit 104 may handle the real number by rounding it to an integer by rounding off, or may handle the real number as it is.
The second estimation unit 104 can also improve the accuracy of the regressor by adding a restriction so that the ratio between the size of the small region and the size of the person is substantially constant. The second estimation unit 104 uses the estimated value of the human body size obtained in S202, so that small areas of different sizes on the image are obtained so that the size of the human body is substantially the same as the size of the small area. Can take. FIG. 6B shows an example of dividing the regression area. In this example, the regression area 320 is divided into small areas 520 having different sizes. Each of the divided small areas 520 is resized to be transformed into a fixed-size small image and input to the regressor.
Non-patent document 1 can be referred to for regression-based number estimation.

In S <b> 205, the number integration unit 105 integrates the number estimation result by the first estimation unit 103 and the number estimation result by the second estimation unit 104.
When there is no overlap between the detection region and the regression region, the number integration unit 105 adds up the number of people estimation result by the first estimation unit 103 and the number of people estimation result by the second estimation unit 104, thereby integrating Can be obtained.
An example of integration when there is an overlap between the detection area and the regression area will be described in detail below.
The number integration unit 105 sets the detection base number estimation result as an estimated number after integration for an area where only the detection area is set in the input image.
In addition, the number integration unit 105 sets the result of regression-based number estimation as an estimated number after integration for an area in which only the regression area is set in the input image.
Further, the number integration unit 105 uses an arbitrary expression that uses the result of estimation of the number of people for an area in which both the detection area and the regression area are set in the input image. Calculate the estimated number of people. For example, the number integration unit 105 is an expression that takes a total value of estimated persons, an expression that takes a smaller one of estimated persons, an expression that takes a larger one of estimated persons, an expression that takes an average of estimated persons, or an arbitrary An expression that takes a weighted average by weights can be used.
As an example, FIG. 7 shows a small area 601 divided at the time of regression-based number estimation. It is assumed that the result of the regression-based number estimation for the small area 601 is six. Also, it is assumed that the number of human body frames whose center of gravity is included in the small region 600 among the human body frames obtained by the detection-based number estimation is 4. The human body frame is shown at 611, 612, 613, and 614 in FIG. The number integration unit 105 regards the estimated number of detection base number estimates for the small area 601 as four.
When the number integration unit 105 takes the smaller of the two estimated numbers, the estimated number after integration is four. When the number integration unit 105 takes the larger of the two estimated numbers, the estimated number after integration is six. When the number integration unit 105 takes the average of the two estimated numbers, the estimated number after integration is five. When the number integrating unit 105 weights and averages the two estimated number of persons, the regression-based number estimation result and the detection-based number estimation result with a weight of 3: 5, the estimated number of persons after integration is 4.75. Become.

  The method of integrating the estimated number of people is not limited to these. For example, the number integration unit 105 may take a weighted average with different weights depending on the position of the image. More specifically, the number integration unit 105 increases the weight of the detection base number estimation result at a position where the human body appears larger than a predetermined size, and sets the regression base number at a position where the human body appears smaller than the predetermined size. The weight of the estimation result may be larger than a predetermined weight. For example, the number integration unit 105 may take a weighted average with different weights according to the density. More specifically, the number integration unit 105 increases the weight of the detection base number estimation result at a point where the density is smaller than a predetermined value, and sets the weight of the regression base number at a point where the density is higher than a predetermined value. The weight of the estimation result may be larger than a predetermined value.

In S206, the display unit 106 displays the number estimation result integrated by the number integration unit 105 on the display device 13 or the like.
First, the display unit 106 displays the number of people estimation result after the integration of the entire image. At this time, the display unit 106 may clearly indicate the detection region and the regression region set in S202 on the image. The display unit 106 may display the number of people estimation result in the detection region estimated in S203 and the number of people estimation result in the regression region estimated in S204.
Further, the display unit 106 may perform more detailed display so that the estimated number of people can be known for each position of the image. An example of the method is shown below.
The first display method is a method in which a number of human body frames equal to the estimated number of persons after integration are superimposed on the image. This will be described with reference to FIG. For example, consider a case where the estimated number of people in the small region 601 in FIG. In this case, since the estimated number of persons based on the detection-based number of persons estimation for the small area 601 is four, the human body frame is insufficient for one person. The display unit 106 can select and display a random position in the small area 601 and superimpose the human body frames for the shortage (here, the frame 701), thereby superimposing and displaying the same number of human body frames as the estimated number of people on the image. . At this time, by estimating the size of the human body frame by the method described in S202, the display unit 106 can superimpose a natural size human body frame. On the contrary, when the detection base number estimation result exceeds the estimated number after integration, the display unit 106 may perform a process of selecting and deleting a human body frame at random as many times as necessary.

The second display method is a method of displaying the estimated number of people for each small area divided when the regression-based number of people is estimated. This will be described with reference to FIG. For example, consider a case where the estimated number of people in the small region 601 in FIG. In this case, the display unit 106 displays the number of people estimated to exist in the small area 601 in a superimposed manner on the small area 601. Further, the display unit 106 may perform visually different image processing on each region according to the number of people. For example, the display unit 106 may display an area with a larger number of people than other areas so that the color is darker (for example, red). The processing of the second display method is an example of performing display processing according to the number of specific objects existing in a predetermined area for each predetermined area.
The third display method is a method of performing different display depending on the location of the image. For example, the display unit 106 uses the first display method in an area where the human body frame is displayed larger than a predetermined size, and uses the second display method in other areas. The third display method process is an example of performing different display processes on an image according to the position of the image.
The fourth display method is a method of performing different display depending on the number of people. For example, the display unit 106 uses the first display method in an area where the estimated number of persons is smaller than a predetermined value, and uses the second display method in other areas. The process of the fourth display method is an example in which different display processes are performed on an image according to the number of specific objects in the image.
By using the third display method or the fourth display method, for example, a region as shown in FIG. 8B is displayed in a region where people are crowded, and people are not crowded. The display as shown in FIG. 8A can be performed on the area. Therefore, the user can quickly grasp how many people are in a densely populated area.

<Second Embodiment>
Both the human body detector used for detection-based person estimation and the regressor used for regression-based person estimation may be learned using an existing deep learning technique. At this time, the CPU 10 may be configured such that the human body detector and the regressor share the same convolution layer. A configuration example is shown in FIG.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium. It can also be realized by a process in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was explained in full detail, this invention is not limited to the specific embodiment which concerns. In the above-described embodiment, description has been made by taking a person as an example of estimation. However, the configuration and information processing of the above-described embodiment are used to estimate the number of arbitrary specific objects by replacing a human with an arbitrary specific object such as a non-human animal, a car such as a car, a bicycle, or a microorganism. be able to.

As described above, according to each of the above-described embodiments, a region to which the detection-based number estimation method and a region to which the regression-based number estimation method is applied are automatically determined, and the respective results are integrated to obtain a single number estimation method. It is possible to estimate the number of people with higher accuracy than when using.
Therefore, the number of specific objects in the image can be estimated with higher accuracy by combining a plurality of person estimation methods.

10 CPU
11 Memory 100 Image processing apparatus

Claims (26)

First estimation means for estimating the number of specific objects in the first region by performing a specific object detection process on the first region of the image;
Second estimation means for estimating the number of specific objects in the second region by performing regression processing for estimating the number of specific objects for each predetermined region with respect to the second region of the image;
Integration means for integrating the result of estimation by the first estimation means and the result of estimation by the second estimation means;
An image processing apparatus.   The image processing apparatus according to claim 1, further comprising region determination means for determining the entire image as the first region and the second region.   The image processing apparatus according to claim 1, further comprising an area determination unit that determines the first area and the second area based on a user setting operation.   The image processing apparatus according to claim 1, further comprising an area determination unit that determines at least one of the first area and the second area based on a size of a specific object of the image.   The area determination means estimates the size of the specific object, and determines at least one of the first area and the second area based on the estimated size of the specific object. The image processing apparatus described.   The image processing apparatus according to claim 1, further comprising an area determination unit that determines at least one of the first area and the second area based on a density of specific objects of the image.   The area determination means estimates the density based on a moving object or a motion vector, and determines at least one of the first area and the second area based on the estimated density. Item 7. The image processing apparatus according to Item 6.   The image processing apparatus according to claim 1, wherein the integration unit integrates the result of estimation by the first estimation unit and the result of estimation by the second estimation unit by summing them up.   The integration unit calculates the result of estimation by the first estimation unit and the result of estimation by the second estimation unit when there is no overlapping area between the first region and the second region. The image processing apparatus according to claim 8, wherein the image processing apparatuses are integrated by summing up.   In the case where there is an overlapping area between the first area and the second area, the integration means displays the result of estimation by the first estimation means for the area of only the first area. The number of specific objects after integration, and for the area of only the second area, the result of estimation by the second estimation means is the number of specific objects after integration, and for the overlapping area 8. The integration is performed by obtaining the number of specific objects after integration based on a result of estimation by the first estimation unit and a result of estimation by the second estimation unit. Image processing apparatus.   The integration unit calculates the number of specific objects after integration by summing the estimation result by the first estimation unit and the estimation result by the second estimation unit for the overlapping region. The image processing apparatus according to claim 10 to be obtained.   The integration unit sets the overlapping region to a smaller value based on the estimation result by the first estimation unit and the estimation result by the second estimation unit after the integration. The image processing apparatus according to claim 10, wherein the number of specific objects is calculated.   The integration means sets the overlapping region to a larger value based on the estimation result by the first estimation means and the estimation result by the second estimation means after the integration. The image processing apparatus according to claim 10, wherein the number of specific objects is calculated.   The integrating means takes an average based on the result of estimation by the first estimating means and the result of estimation by the second estimating means for the overlapping region, and the specific object after the integration The image processing apparatus according to claim 10, wherein the number is calculated.   The integration means takes the weighted average based on the result of the estimation by the first estimation means and the result of the estimation by the second estimation means for the overlapping region, and The image processing apparatus according to claim 10, wherein the number of specific objects is obtained.   The image processing apparatus according to claim 15, wherein the integration unit changes the weighting based on a position of the image or a density of a specific object.   The image processing apparatus according to claim 1, further comprising a display unit configured to display a number of specific objects of the image obtained as a result of integration by the integration unit.   The image processing apparatus according to claim 17, wherein the display unit further displays the first area and the second area.   The image processing apparatus according to claim 17 or 18, wherein the display unit further displays a result of estimation by the first estimation unit and a result of estimation by the second estimation unit.   The image processing apparatus according to claim 17, wherein the display unit displays a frame of the number of specific objects existing in the predetermined area for each of the predetermined areas so as to be superimposed on the predetermined area.   The image processing apparatus according to claim 17, wherein the display unit superimposes and displays the number of specific objects existing in the predetermined area for each predetermined area.   The image processing apparatus according to claim 17, wherein the display unit performs display processing according to the number of specific objects existing in the predetermined area for each predetermined area.   The image processing apparatus according to claim 17, wherein the display unit performs different display processing on the image according to the position of the image or the number of specific objects.   The image processing apparatus according to claim 1, wherein the specific object is one of a person, an animal other than a person, a microorganism, and a car. An information processing method executed by an image processing apparatus,
A first estimation step of estimating the number of specific objects in the first region by performing a specific object detection process on the first region of the image;
A second estimation step of estimating the number of specific objects in the second region by performing a regression process for estimating the number of specific objects for each predetermined region with respect to the second region of the image;
An integration step of integrating the estimation result of the first estimation step and the estimation result of the second estimation step;
An information processing method including:   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 24.

Images