JP2021086391A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To set an appropriate estimation area in order to increase the accuracy of estimating the number of specific objects.SOLUTION: Detection means executes detection processing for detecting a specific object from a captured image. Holding means holds object information indicating position and size of the specific object on the image. Tracking means tracks the same specific object on the basis of detection processing on each of the images captured at different times. Setting means sets a plurality of estimation areas on the captured images on the basis of object information corresponding to the tracked specific object. Estimation means executes estimation processing for estimating the number of specific objects included in each of the estimation areas.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image processing technique.

In recent years, a system has been proposed in which a predetermined area is imaged by an imaging device and a person in the image is counted by analyzing the captured image. Such a system is expected to be used for eliminating congestion at an event and guiding evacuation in the event of a disaster by detecting congestion in a public space and grasping the flow of people at the time of congestion.

Non-Patent Document 1 discloses a method of directly estimating the number of people appearing in a predetermined estimation area of an image by using a recognition model obtained by machine learning. Hereinafter, this method will be referred to as a regression-based estimation method.

Hiroo Ikeda, Ryoma Oami, Hiroyoshi Miyano. Higher accuracy of number estimation based on crowd patch learning using CNN. FIT, 2014

In the regression-based estimation method, in order to improve the accuracy of estimating the number of specific objects, it is necessary to set an estimation area having a size proportional to the size of the specific objects displayed on the image. At this time, when a plurality of estimation regions are set for the image captured by the imaging device, the user confirms the size of a specific object displayed at a plurality of different positions on the image, and the plurality of estimation regions are set for the image. Can be considered. However, since the specific object is not always located at a plurality of different positions in the image confirmed by the user, it is not possible to appropriately set an estimation area having a size proportional to the size of the specific object appearing at a plurality of different positions on the image. There was something.

Therefore, an object of the present invention is to set a more appropriate estimation area in order to improve the accuracy of estimating the number of specific objects.

In order to solve the above problems, the image processing apparatus of the present invention has the following configurations. That is, the detection means for executing the detection process for detecting the specific object from the image captured by the imaging means, the holding means for holding the object information indicating the position and size of the specific object on the image, and the imaging means. Based on the detection process for each of a plurality of images captured at different times, the tracking means for tracking the same specific object and the object information corresponding to the same specific object tracked by the tracking means are used. It has a setting means for setting a plurality of estimation regions for an image captured by the imaging means, and an estimation means for executing an estimation process for estimating the number of the specific object included in each of the plurality of estimation regions. ..

According to the present invention, a more appropriate estimation region can be set in order to increase the accuracy of estimating the number of specific objects.

It is a figure which shows an example of a system configuration. It is a figure which shows the functional block of an image processing apparatus. It is a figure for demonstrating the process of estimating geometry information. It is a figure for demonstrating the process of estimating geometry information. It is a figure for demonstrating the process of setting an estimation area. It is a figure for demonstrating the estimation process with respect to the estimation area. It is a flowchart which shows the flow of the process of setting an estimation area and the estimation process for the estimation area. It is a figure for demonstrating the process of estimating geometry information. It is a figure for demonstrating the process of estimating geometry information. It is a figure for demonstrating the process of estimating geometry information. It is a flowchart which shows the flow of the process of setting an estimation area. It is a figure which shows the hardware configuration of each apparatus.

Hereinafter, embodiments according to the present invention will be described with reference to the accompanying drawings. The configuration shown in the following embodiments is only an example, and is not limited to the illustrated configuration.

(Embodiment 1)
FIG. 1 is a diagram showing a system configuration according to the present embodiment. The system in this embodiment includes an image processing device 100, an image pickup device 110, a recording device 120, and a display 130.

The image processing device 100, the image pickup device 110, and the recording device 120 are connected to each other via the network 140. The network 140 is realized from a plurality of routers, switches, cables and the like conforming to a communication standard such as Ethernet (registered trademark).

The network 140 may be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless Lan), a WAN (Wide Area Network), or the like.

The image processing device 100 is realized, for example, by a personal computer or the like in which a program for realizing an image processing function described later is installed. The image pickup device 110 is a device for capturing an image. The image pickup device 110 associates the image data of the captured image, the information of the image capture time at which the image was captured, and the identification information which is the information for identifying the image pickup device 110, and records the image processing device 100 and the recording via the network 140. It is transmitted to an external device such as the device 120. In the system according to the present embodiment, the number of image pickup devices 110 is one, but there may be a plurality of image pickup devices 110.

The recording device 120 records the image data of the image captured by the imaging device 110, the information of the imaging time at which the image was captured, and the identification information for identifying the imaging device 110 in association with each other. Then, in accordance with the request from the image processing device 100, the recording device 120 transmits the recorded data (image, identification information, etc.) to the image processing device 100.

The display 130 is composed of an LCD (Liquid Crystal Display) or the like, and displays the result of image processing of the image processing device 100, an image captured by the image pickup device 110, and the like. The display 130 is connected to the image processing device 100 via a display cable compliant with a communication standard such as HDMI (registered trademark) (High Definition Multimedia Interface).

In addition, the display 130 functions as a display means, and displays an image captured by the image pickup apparatus 110, a result of image processing described later, and the like. At least any two or all of the display 130, the image processing device 100, and the recording device 120 may be provided in a single housing. Further, the image processing device 100 and the image pickup device 110 may be provided in a single housing. That is, the image pickup apparatus 110 may have the functions and configurations of the image processing apparatus 100 described later.

The result of image processing of the image processing device 100 and the image captured by the image pickup device 110 are not limited to the display 130 connected to the image processing device 100 via the display cable, and for example, the following external device It may be displayed on the display of. That is, it may be displayed on the display of a mobile device such as a smartphone or tablet terminal connected via the network 140.

Next, the image processing of the image processing apparatus 100 according to the present embodiment will be described with reference to the functional blocks of the image processing apparatus 100 according to the present embodiment shown in FIG. In the case of the present embodiment, each function shown in FIG. 2 is realized as follows by using a ROM (Read Only Memory) 1202 and a CPU (Central Processing Unit) 1200, which will be described later with reference to FIG. It shall be done. Each function shown in FIG. 2 is realized by executing a computer program stored in the ROM 1202 of the image processing device 100 by the CPU 1200 of the image processing device 100.

The communication unit 200 can be realized by an I / F (Interface) 1204 described later with reference to FIG. 12, and communicates with the image pickup device 110 and the recording device 120 via the network 140. The communication unit 200 receives, for example, image data of an image captured by the image pickup device 110, or transmits a control command for controlling the image pickup device 110 to the image pickup device 110. The control command includes, for example, a command for instructing the image pickup apparatus 110 to capture an image.

The storage unit 201 can be realized by a RAM (Random Access Memory) 1201 or an HDD (Hard Disk Drive) 1203, which will be described later with reference to FIG. 12, and stores information and data related to image processing by the image processing device 100. The storage unit 201 holds object information indicating the position and size information on the image of the specific object detected by the detection process by the detection unit 204 described later.

The output control unit 202 outputs an image captured by the image pickup device 110, information indicating the result of image processing, and the like to an external device, and displays the image on the display 130. The external device to which the information is output by the output control unit 202 includes, for example, another image processing device (not shown) or a recording device 120. The operation reception unit 203 receives an operation performed by the user via an input device (not shown) such as a keyboard or a mouse.

The detection unit 204 executes a detection process for detecting a specific object in the image by using a method different from the regression-based estimation method. The detection unit 204 in the present embodiment detects a specific object from an image by performing processing such as pattern matching using, for example, a collation pattern (dictionary). Then, the storage unit 201 accumulates object information indicating the position and size of the specific object on the image each time a specific object is detected in the image.

When detecting a person as a specific object from an image, the person may be detected from the image by using a plurality of matching patterns such as a matching pattern when the person is facing forward and a matching pattern when the person is facing sideways. Good. By executing the detection process using a plurality of collation patterns in this way, improvement in detection accuracy can be expected.

As a collation pattern, a collation pattern may be prepared when a specific object is viewed from another angle such as from an oblique direction or from an upward direction. In addition, when detecting a person as a specific object, it is not always necessary to prepare a collation pattern (dictionary) showing the characteristics of the whole body, and a collation pattern for a part of the person such as the upper body, lower body, head, face, and legs. May be prepared.

The tracking unit 205 tracks the same specific object based on detection processing for each of a plurality of images captured at different times by the imaging device 110. The tracking unit 205 in the present embodiment performs the following processing on the image captured by the image pickup apparatus 110, for example. That is, when the same specific object as the specific object detected by the detection unit 204 from the image of the frame one or more before the current frame exists in the image of the current frame, the tracking unit 205 identifies the specific object in each frame. Associate objects with each other. That is, the same specific object is tracked between images for a plurality of frames that are close in time. A detailed description of tracking the same specific object by the tracking unit 205 will be described later.

The first estimation unit 206 estimates the size of the specific object for each position on the image captured by the image pickup apparatus 110 based on the object information corresponding to the same specific object tracked by the tracking unit 205. The object information corresponding to the same specific object here indicates the object information accumulated by the storage unit 201 each time the same specific object is detected by the detection unit 204. In the following description, the information indicating the size of the specific object for each position on the image estimated by the first estimation unit 206 is used as the geometry information.

The setting unit 207 makes a plurality of estimates for the image captured by the image pickup device 110 according to the geometry information estimated by the first estimation unit 206 based on the object information corresponding to the same specific object tracked by the tracking unit 205. Set the area.

The second estimation unit 208 uses the regression-based estimation method to perform an estimation process for estimating the number of specific objects included in each of the plurality of estimation regions in the image set by the setting unit 207. In the regression-based estimation method, an image is captured by the image pickup apparatus 110 by using a regression device (learned recognition model) that inputs a small image of a certain fixed size S and outputs the number of specific objects in the small image. Estimate the number of specific objects in the estimation area on the image. In training the regressionr, a large number of small images of fixed size S in which the position of a specific object is known are prepared, and the regressionr is learned based on the machine learning method using the small images as training data. At this time, in order to improve the estimation accuracy of the number of specific objects, it is desirable that the ratio of the size of the small image (fixed size S), which is the learning data, to the size of the specific object reflected in the small image is substantially constant. Then, the second estimation unit 208 makes a small image obtained by resizing the image of the estimation area to a fixed size S for each of the plurality of estimation areas, and inputs the small image to the regression device to "in the estimation area". "Position of a specific object" is obtained as the output from the regression device. At this time, the number of positions of the specific object in the estimated area is the number of the specific objects in the estimated area.

Further, when the setting unit 207 sets a plurality of estimation regions for the image, the ratio of the size of the estimation region to the size of the specific object in the estimation region is reflected in the size of the small image which is the learning data and the small image. It is desirable that the ratio r to the size of a specific object is substantially the same. By setting the estimation area for the image so as to approach the environment of the learning data in this way, the estimation accuracy of the number of specific objects included in the estimation area can be further improved.

Therefore, the setting unit 207 in the present embodiment executes the following processing based on the object information corresponding to the same specific object tracked by the tracking unit 205. That is, the setting unit 207 sets a plurality of estimation regions for the image according to the geometry information estimated based on the object information accumulated for the same specific object tracked by the tracking unit 205. At this time, the setting unit 207 sets a plurality of estimation regions for the image so that the ratio between the size of the estimation region and the size of the specific object included in the estimation region becomes the ratio r corresponding to the learning data according to the geometry information. Set.

The counting unit 209 acquires the counting result by adding up the number of specific objects estimated in the estimation process by the second estimation unit 208 for each of the plurality of estimation regions set for the captured image. The output control unit 202 outputs information indicating a counting result obtained by adding up the estimated number of specific objects for each of the estimation regions to an external device (display 130 or the like).

Here, the image processing according to the present embodiment will be described more specifically with reference to FIG. In the following description, a specific object whose number is to be estimated will be described as a person, but the number is not limited to the person. For example, the specific object may be various vehicles traveling on a road or the like, parts or products flowing on a conveyor in a factory, or an animal or the like.

FIG. 3 is a diagram showing how the same person on the image is tracked by the tracking unit 205. The tracking unit 205 in the present embodiment executes tracking of the detected person by determining that the persons detected by the detection unit 204 are the same over the images of a plurality of frames. The following method can be used as a method for determining that the persons detected over the images of a plurality of frames by the tracking unit 205 are the same. For example, the tracking unit 205 is a method of determining that the person is the same person if the predicted movement position of the person and the position of the detected person are within a certain distance by using the movement vector of the detected person. Further, the tracking unit 205 may associate a person with a high correlation between images of a plurality of frames by using the color, shape, size (number of pixels), and the like of the person. As described above, the tracking unit 205 is not limited to a specific method as long as it can perform a process of determining that the person is the same person over a plurality of frames of images and tracking the person.

The detection unit 204 in the present embodiment detects a person's area (hereinafter referred to as a person's area) on the image by a pattern matching method using a matching pattern, and the tracking unit 205 is detected over a plurality of frames of the image. Track the person area of the same person. Here, returning to the description of FIG. 3, the object information 302 shown in FIG. 3 is information indicating the position and size of the person 301 detected by the detection unit 204 from the first image on the image. Further, the object information 303 is information indicating the position and size of the person region of the person 301 detected in the second image captured after the first image on the image. Further, the object information 304 is information indicating the position and size of the person area of the person 301 detected in the third image captured after the second image on the image.

The position on the image of the detected person area in the object information 302 to 304 shall be indicated by the X coordinate and the Y coordinate at the center position of the person area when the upper left end point of the image is the origin. Further, the size of the detected person area on the image in the object information 302 to 304 indicates the length of the person area in the vertical direction (Y-axis direction) of the image. As shown in FIG. 3, the storage unit 201 holds and accumulates object information corresponding to the person 301 that is tracked over the images of a plurality of frames. That is, the storage unit 201 stores the object information 301 to 304 of the same person 301 detected over the images of a plurality of frames.

The first estimation unit 206 is a geometry information indicating the size of a person appearing at an arbitrary position on an image based on the object information of the person accumulated by the storage unit 201 when the same person is tracked over images of a plurality of frames. To estimate. Geometry information is given from an arbitrary position (x, y) on the image as the size f (x, y) of an average person appearing at that position. It is assumed that f (x, y), which is the size of a person at an arbitrary position on an image, can be represented by, for example, x, y and one or more parameters. For example, assume that f (x, y) = ax + by + c. In this example, the unknown parameters are a, b and c. At this time, the first estimation unit 206 can obtain an unknown parameter by statistical processing such as the least squares method by using the object information of the person accumulated by the storage unit 201. The parameters of f (x, y) are estimated by existing optimization methods such as the least squares method and Newton's method.

Here, an example of geometry information will be described with reference to FIG. In the description with reference to FIG. 4, for simplification of the explanation, the size of the person does not change in the horizontal direction (x-axis direction) on the image, and the size of the person in the vertical direction (y-axis direction) on the image. Suppose there is a change in. The first estimation unit 206 has object information 302 corresponding to the person 301 detected from the first image, object information 303 corresponding to the person 301 detected from the second image, and the person 301 detected from the third image. Geometry information is estimated based on the object information 304 corresponding to. In FIG. 4, the position and size of the person 301 on the image shown by each of the object information 302 to 304 are plotted. Then, the first estimation unit 206 estimates the geometry information shown by the curve as shown in FIG. 4 by the least squares method, Newton's method, or the like using the object information 302 to 304.

When the following conditions are satisfied, the first estimation unit 206 may estimate the geometry information based on the object information of the same person accumulated by the storage unit 201. That is, the first estimation unit 206 may estimate the geometry information based on the object information accumulated for the same person whose distance on the image tracked by the tracking unit 205 is equal to or greater than a predetermined value. Further, when the distance from the first position on the locus of the same person tracked by the tracking unit 205 to the second position on the locus is equal to or greater than a predetermined value, the first estimation unit 206 is accumulated for the same person. Geometry information may be estimated based on the object information. Further, when the number of times that the same person tracked by the tracking unit 205 is detected at different positions on the image reaches a predetermined value, the first estimation unit 206 has a geometry based on the object information accumulated for the same person. The information may be estimated.

Here, a process of setting a plurality of estimation regions by the setting unit 207 in the present embodiment will be described with reference to FIG. 5 (a) to 5 (c) are views showing a state in which a plurality of estimation regions are set by the setting unit 207 for the image 500 captured by the image pickup apparatus 110. In the example shown in FIG. 5, as the size of the person at an arbitrary position on the image indicated by the geometry information, the size of the person is substantially the same in the horizontal direction of the image 500, and the size of the person is from the lower part to the upper part in the vertical direction of the image 500. The size of the person shall decrease accordingly. First, as shown in FIG. 5A, the setting unit 207 in the present embodiment sets a plurality of estimation regions 501 along the lower end of the image 500. At this time, the setting unit 207 estimates so that the ratio between the size of the estimation area 501 and the size of the person indicated by the geometry information at the coordinates of the lower end in the estimation area 501 is substantially the same as the ratio r corresponding to the learning data. Region 501 is set. Next, as shown in FIG. 5B, the setting unit 207 sets a plurality of estimated regions 502 along the upper ends of the plurality of estimated regions 501. At this time, the setting unit 207 estimates so that the ratio between the size of the estimation area 502 and the size of the person indicated by the geometry information at the coordinates of the lower end in the estimation area 502 is substantially the same as the ratio r corresponding to the learning data. Region 502 is set. Next, as shown in FIG. 5C, the setting unit 207 sets a plurality of estimation regions 503 along the upper ends of the plurality of estimation regions 502. At this time, the setting unit 207 estimates so that the ratio between the size of the estimation area 503 and the size of the person indicated by the geometry information at the coordinates of the lower end in the estimation area 503 is substantially the same as the ratio r corresponding to the learning data. A plurality of areas 503 are set. As described above, in the setting unit 207 in the present embodiment, the ratio of the size of the estimated region to the size of the specific object in the estimated region is the size of the small image which is the learning data and the size of the specific object reflected in the small image. The estimation area is set for the image so as to be substantially the same as the ratio r of. By setting the estimation area for the image so as to approach the environment of the learning data in this way, the estimation accuracy of the number of specific objects included in the estimation area can be further improved. In the above description with reference to FIGS. 5A to 5B, the estimation area is set in order from the lower end of the image, but the estimation area is not limited to this, and the estimation area may be set from another position.

The second estimation unit 208 uses a small image obtained by resizing the image of the estimation area to a fixed size S for each of the plurality of set estimation areas, and inputs the small image to the regressionr learned based on machine learning. Therefore, the "position of the person in the estimated area" is obtained as the output from the regression device. At this time, the number of positions of the person in the estimated area indicates the number of people included in the estimated area, and the number of positions of the person in the estimated area may be an integer or after the decimal point. It may be a real value including the value of. FIG. 6 is a schematic diagram showing the result of estimation processing by the second estimation unit 208 for each of the plurality of small regions set by the setting unit 207 for the image 600. As shown in FIG. 6, the numerical value 602 included in the estimation area 601 indicates the number of persons estimated with respect to the estimation area 601. The counting unit 209 acquires 12.1 people, which is the total number of people estimated in the estimation process by the second estimation unit 208 for each of the plurality of estimation regions set for the captured image. .. The output control unit 202 generates an output image by superimposing the plurality of estimation areas 601 and the numerical value 602 which is information indicating the result of the estimation processing on the plurality of estimation areas 601 on the image 600, and the generated output image. Is output to an external device (display 130). At this time, the output control unit 202 may display the generated output image on the display 130.

Next, with reference to FIG. 7, the image processing in the present embodiment will be described in more detail. By executing the flow shown in FIG. 7A, a plurality of estimation regions can be set for the image. Further, by executing the flow shown in FIG. 7B, it is possible to estimate the number of people included in the image by executing the estimation process for each of the plurality of estimation areas set for the image. .. The flow processing shown in FIG. 7A shall be started or ended according to, for example, an instruction by the user. Then, the flow processing shown in FIG. 7B is executed after the flow processing shown in FIG. 7A is executed and a plurality of estimation regions are set. The processing of the flowchart shown in FIG. 7 is executed by the functional block shown in FIG. 2 realized by executing the computer program stored in the ROM 1202 of the image processing device 100 by the CPU 1200 of the image processing device 100. ..

First, the flow processing shown in FIG. 7A will be described. In S701, the communication unit 200 acquires an image of one frame in the moving image captured by the imaging device 110 as an image to be processed. The communication unit 200 may acquire the image to be processed from the image pickup device 110 or the storage device 120 via the network 140, or may acquire the image to be processed from the storage unit 201 of the image processing device 100. Good.

Next, in S702, the detection unit 204 executes a detection process for detecting a person in the image. The detection unit 204 in the present embodiment detects a person by performing processing such as pattern matching using a collation pattern (dictionary).

Next, in S703, the storage unit 201 accumulates object information indicating the position and size of the person on the image each time a person is detected in the image.

Next, in S704, the tracking unit 205 tracks the person detected by the detection unit 204. The tracking unit 205 assigns an ID to each person to be tracked. For example, the tracking unit 205 assigns an ID “1” as a new ID to a person newly detected by the detection unit 204. It is assumed that the ID "1" is assigned to the person detected by the detection unit 204 from the image of the frame one or more before the frame of the image to be processed at present. In that case, if the detection unit 204 also detects the person from the image of the current frame, the tracking unit 205 also assigns the ID "1" to the person. In this way, the tracking unit 205 tracks the same person over images of a plurality of frames.

Next, in S705, the first estimation unit 206 determines whether the same person tracked by the tracking unit 206 satisfies a predetermined condition. For example, the first estimation unit 206 determines whether the distance on the image of the same person tracked by the tracking unit 205 is equal to or greater than a predetermined value. Then, when it is determined that the distance on the image is equal to or greater than a predetermined value (Yes in S705), the process transitions to S706. On the other hand, when it is determined that the distance on the image is less than a predetermined value (No in S705), the transition to S701 occurs, and the communication unit 200 processes the image of the next frame in the moving image captured by the image pickup device 110. Get as an image.

Further, in S705, the first estimation unit 206 determines whether the distance from the first position on the locus of the same person tracked by the tracking unit 205 to the second position on the locus is equal to or greater than a predetermined value. You may. In this case, when it is determined that the distance from the first position, which is an arbitrary position on the locus, to the second position, which is an arbitrary position on the locus, is equal to or greater than a predetermined value (Yes in S705). Transition to S706. On the other hand, when it is determined that the distance from the first position, which is an arbitrary position on the locus, to the second position, which is an arbitrary position on the locus, is not equal to or greater than a predetermined value (No in S705), the process proceeds to S701. Transition. Then, the communication unit 200 acquires the image of the next frame in the moving image captured by the imaging device 110 as the image to be processed. In this way, the processes of S701 to S704 are repeated until the same person tracked by the tracking unit 205 satisfies a predetermined condition, and the storage unit 201 obtains the object information of the same person each time the same person is detected. Accumulate.

Then, in S706, the first estimation unit 206 provides geometry information indicating the size of the person appearing at an arbitrary position on the image based on the object information accumulated by the storage unit 201 for the same person tracked by the tracking unit 205. presume.

Next, in S707, the setting unit 207 takes an image based on the geometry information so that the ratio between the size of the estimated region and the size of the specific object in the estimated region is substantially the same as the ratio r corresponding to the learning data. A plurality of estimation regions are set for the image captured by the device 110.

Next, the flow processing shown in FIG. 7B will be described. First, in S771, the communication unit 200 acquires an image of one frame in the moving image captured by the imaging device 110 as an image to be processed. The communication unit 200 may acquire the image to be processed from the image pickup device 110 or the storage device 120 via the network 140, or may acquire the image to be processed from the storage unit 201 of the image processing device 100. Good. Further, the communication unit 200 may acquire a still image captured by the image pickup apparatus 110 as an image to be processed.

Next, in S772, the setting unit 207 acquires the information of the plurality of estimation areas set in S707, and sets the plurality of estimation areas for the image to be processed at present. Next, in S773, the second estimation unit 208 executes an estimation process for estimating the number of people for each of the plurality of estimation areas set for the image to be processed at present.

Next, in S774, the measurement unit 209 counts by adding up the number of specific objects estimated in the estimation process by the second estimation unit 208 for each of the plurality of estimation regions set for the captured image. Get the result. Next, in S775, the output control unit 202 outputs information indicating the counting result obtained by adding up the estimated number of specific objects for each of the estimation regions to an external device (display 130 or the like). Next, in S776, if there is no termination instruction by the user (No in S776), the transition to S771 occurs, and the communication unit 200 processes the image of the next frame in the moving image captured by the image pickup device 110. Get as an image. On the other hand, when the user gives an instruction to end (Yes in S776), the process shown in FIG. 7B ends.

The flow process shown in FIG. 7A may be executed, for example, when the imaging direction or zoom magnification indicated by the pan and tilt of the imaging device 110 changes. When the imaging range (determined by the imaging direction and the zoom magnification), which is the range to be imaged by the imaging device 110, changes, the size of the person at an arbitrary position on the image, that is, the geometry information changes. Therefore, the flow processing shown in FIG. 7A may be executed according to the change in the imaging range of the imaging apparatus 110.

In the image processing apparatus 100 of the present embodiment, the geometry information is estimated based on the object information accumulated in the process of tracking the same person by the tracking unit 205, but the present invention is not limited to this. For example, in the image processing device 100 of the present embodiment, the tracking unit 205 tracks each of a plurality of persons, and estimates the geometry information based on the object information accumulated in the process of tracking each of the plurality of persons. You may. For example, based on the object information 803 to 808 shown in FIG. 8, the first estimation unit 206 estimates the geometry information. Specifically, the first estimation unit 206 is based on the object information 803 to 805 accumulated in the process of tracking the person 801 and the object information 806 to 808 accumulated in the process of tracking the person 802. Estimate geometry information. In FIG. 9, the position and size of the person 801 on the image shown in the object information 803 to 805 corresponding to the person 801 and the position and size of the person 802 on the image shown by the object information 806 to 808 corresponding to the person 802 are plotted. Has been done. Then, the first estimation unit 206 uses the object information 803 to 808 to estimate the geometry information shown by the curve as shown in FIG. 9 by the least squares method, Newton's method, or the like. In the description shown in FIG. 9, for simplification of the explanation, the size of the person does not change in the horizontal direction (x-axis direction) on the image, and the size of the person in the vertical direction (y-axis direction) on the image. Suppose there is a change in. As described above, in the image processing apparatus 100 of the present embodiment, the tracking unit 205 tracks each of the plurality of persons, and estimates the geometry information based on the object information accumulated in the process of tracking each of the plurality of persons. You may do so.

As described above, in the present embodiment, the image processing apparatus 100 estimates geometry information which is information on the size of a specific object at an arbitrary position on the image based on the object information accumulated for the same tracked person. .. Then, the image processing device 100 sets a plurality of estimation regions for the image so that the ratio between the size of the estimation region and the size of the specific object included in the estimation region becomes the ratio r corresponding to the learning data according to the geometry information. .. By doing so, the estimation area can be set more appropriately, and as a result, the accuracy of estimating the number of specific objects in the estimation area can be improved. (Embodiment 2)
In the present embodiment, a detection area is set for the image, and geometry information is provided based on the object information corresponding to the specific object detected in the detection area and the object information corresponding to the same specific object tracked. Estimate and set a plurality of estimation areas according to the geometry information. The parts different from those of the first embodiment will be mainly described, and the same or equivalent components and the processes as those of the first embodiment are designated by the same reference numerals, and duplicate description will be omitted. Further, as in the first embodiment, the specific object will be described as a person in the following description, but the present invention is not limited to the person. For example, the specific object may be various vehicles traveling on a road or the like, parts or products flowing on a conveyor in a factory, or an animal or the like.

Here, first, the processing of the image processing apparatus 100 in the present embodiment will be described with reference to FIG. As shown in FIG. 7A, the setting unit 207 in the present embodiment sets the detection area 1004 for the image. The detection area 1004 set by the setting unit 207 is a part of the image and may be at an arbitrary position. Further, the setting unit 207 sets the detection area 1004 for the image according to a user operation for designating a part of the area on the image, for example.

Further, the tracking unit 205 in the present embodiment executes tracking of the detected person by determining that the persons detected by the detection unit 204 are the same over the images of a plurality of frames. In the example shown in FIG. 10, the tracking unit 205 determines that the person 1001, the person 1011 and the person 1021 are the same person, and performs tracking. In the following description, the person 1001, the person 1011 and the person 1021 will be described as a specific person A. The person 1001 indicates the position and size in which the person A is detected in the first image. Further, the person 1011 indicates the position and size in which the person A is detected from the second image captured after the first image. Further, the person 1021 indicates the position and size in which the person A is detected from the third image captured after the second image.

In addition, the first estimation unit 206 in the present embodiment determines whether the number of times a person has been detected by the detection unit 204 in the detection area 1004 on the image set by the setting unit 204 has reached a predetermined value. Further, the first estimation unit 206 determines whether the same person tracked by the tracking unit 206 satisfies a predetermined condition. As a predetermined condition here, for example, the first estimation unit 206 determines whether the distance on the image of the same person tracked by the tracking unit 205 is equal to or greater than a predetermined value. Further, as a predetermined condition, the first estimation unit 206 determines, for example, whether the distance from the first position on the locus of the same person tracked by the tracking unit 205 to the second position on the locus is equal to or greater than a predetermined value. You may try to do it.

Then, the first estimation unit 206 detects from the image when it is determined that the number of times the person is detected in the detection area 1004 has reached a predetermined value and the same person who has been tracked satisfies the predetermined condition. Geometry information is estimated based on the object information accumulated for the person. Here, the processing of the first estimation unit 206 will be specifically described with reference to FIG. 10B. Person 1002 in FIG. 10B indicates the position and size of a specific person B detected in the detection area 1004. Further, the person 1003 in FIG. 10B indicates the position and size of a specific person C detected in the detection area 1004. In the first estimation unit 206 of the present embodiment, the person B and the person C detected in the detection area 1004 are other than the detection area 1004 based on the degree of change in the size of the person A tracked by the tracking unit 205 on the image. Estimate the size of the image in the area on the image.

The first estimation unit 206 executes the following processing, for example. Based on the object information 1001a and 1011a, the first estimation unit 206 increases the position of the person 1011 (person A) from x1 to x2 on the x-axis with respect to the person 1001 (person A), and y1 to y2 on the y-axis. It is determined that the size has increased c times as the size increases to b times. From this, the first estimation unit 206 generates virtual object information 1012b which is virtual object information from the object information 1002b of the person 1002 (person B), and virtual object information from the object information 1003C of the person 1003 (person C). Generate 1013c. Specifically, the x-coordinate x2'shown by the virtual object information 1012b is a value obtained by multiplying the x-coordinate x1'shown by the object information 1002b by a, and the y-coordinate y2'shown by the virtual object information 1012b is It is a value obtained by multiplying y1'of the y coordinate indicated by the object information 1002b by b. Further, size2', which is the size indicated by the virtual object information 1012b, is a value obtained by multiplying size1' by c.

Similarly, the x-coordinate x2'' indicated by the virtual object information 1013c is a value obtained by multiplying the x-coordinate x1'' indicated by the object information 1003c by a, and the y-coordinate y2'' indicated by the virtual object information 1013c is , The value is b times the y1'' of the y coordinate indicated by the object information 1003c. Further, the size2 ″, which is the size indicated by the virtual object information 1013c, is a value c times the size1 ″.

Further, in the first estimation unit 206, the person 1021 (person A) is d times from x1 to x3 on the x-axis with respect to the person 1001 (person A), and is e-folded from y1 to y2 on the y-axis. Therefore, it is determined that the size has increased f times. Then, the first estimation unit 206 generates the virtual object information 1022b from the object information 1002b and generates the virtual object information 1023c from the object information 1003c in the same manner as the above-mentioned process for generating the virtual object information 1012b and the virtual object information 1013c. To do.

As described above, the first estimation unit 206 is a virtual case where at least one person detected in the detection area 1004 appears in a region other than the detection area 1004 based on the change in size of the tracked person A on the image. Estimate virtual object information, which is typical object information. Then, the first estimation unit 206 estimates the geometry information based on the object information accumulated for the same person in the process of tracking the same person and the generated virtual object information. At this time, as in the first embodiment, it is assumed that f (x, y), which is the size of a person at an arbitrary position on the image, can be represented by, for example, x, y and one or more parameters. For example, assume that f (x, y) = ax + by + c. In this example, the unknown parameters are a, b and c. At this time, the first estimation unit 206 uses the object information accumulated for the same person in the process of tracking the same person and the generated virtual object information to set an unknown parameter, for example, the least squares method or the like. It can be obtained by statistical processing. In this way, the function of f (x, y) is estimated as the geometry information.

Next, the image processing of the image processing apparatus 100 in the present embodiment will be described with reference to the flow shown in FIG. By executing the flow processing shown in FIG. 12, a plurality of estimation regions can be set more appropriately for the image. The processing of the flowchart shown in FIG. 12 is executed by the functional block shown in FIG. 2 realized by executing the computer program stored in the ROM 1202 of the image processing device 100 by the CPU 1200 of the image processing device 100. ..

First, in S701, the communication unit 200 acquires an image of one frame in the moving image captured by the imaging device 110 as an image to be processed. The communication unit 200 may acquire the image to be processed from the image pickup device 110 or the storage device 120 via the network 140, or may acquire the image to be processed from the storage unit 201 of the image processing device 100. Good.

Next, in S1101, the setting unit 207 sets a detection area for the image to be processed. At this time, for example, based on the operation of designating the detection area on the image received by the operation reception unit 203, the setting unit 207 sets the detection area 1004 as shown in FIG. 10 for the image. Since the processes of S702 to S705 are the same as those described in the first embodiment, the description thereof will be omitted. In S1102, the first estimation unit 206 determines whether the number of times a person is detected in the detection area set by the setting unit 207 is equal to or greater than the threshold value. When it is determined that the number of times a person is detected in the detection area is equal to or greater than the threshold value (Yes in S1102), the transition to S1103 is performed. On the other hand, when it is determined that the number of times a person is detected in the detection area is less than the threshold value (No in S1102), the transition to S701 occurs, and the communication unit 200 moves to the next frame in the moving image captured by the image pickup device 110. Image is acquired as the image to be processed. In this way, the processes S701 to S704 are repeated until the same person tracked by the tracking unit 205 satisfies the predetermined condition and the number of times the person is detected in the detection area exceeds the threshold value, and the detected person is detected. Object information is accumulated about.

Then, in S1103, the first estimation unit 206 generates virtual object information when the person detected in the detection area appears in a place other than the detection area 1004, based on the change in size on the image of the same person tracked. To do. For example, the first estimation unit 206 generates virtual object information 1012b and virtual object information 1022b for the person 1002 detected in the detection area 1004. Further, the first estimation unit 206 generates virtual object information 1013c and virtual object information 1023c for the person 1003 detected in the detection area 1004.

Next, in S1104, the first estimation unit 206 generates the object information accumulated in the process of tracking the same person on the image, the object information accumulated about the person detected in the detection area, and the generated virtual. Geometry information is estimated from the object information. Next, in S707, the setting unit 207 sets a plurality of estimation regions for the image based on the geometry information estimated in S1104.

As described above, the image processing device 100 in the present embodiment generates object information accumulated for a person detected in a part of an image and object information accumulated in the process of tracking the same person. Geometry information is estimated from the virtual object information. Then, the image processing device 100 sets a plurality of estimation regions for the image based on the estimated geometry information. As a result, the geometry information can be estimated more accurately, so that the estimation area can be set more appropriately, and as a result, the accuracy of estimating the number of specific objects in the estimation area can be increased.

(Other embodiments)
Next, with reference to FIG. 12, the hardware configuration of the image processing apparatus 100 for realizing each function of each embodiment will be described. Although the hardware configuration of the image processing device 100 will be described in the following description, it is assumed that the recording device 120 and the image pickup device 110 are also realized by the same hardware configuration.

The image processing device 100 in this embodiment includes a CPU 1200, a RAM 1201, a ROM 1202, an HDD 1203, and an I / F 1204.

The CPU 1200 is a central processing unit that controls the image processing device 100 in an integrated manner. The RAM 1201 temporarily stores a computer program executed by the CPU 1200. The RAM 1201 also provides a work area used by the CPU 1200 to execute processing. Further, the RAM 1201 functions as, for example, a frame memory or a buffer memory.

The ROM 1202 stores a program or the like for the CPU 1200 to control the image processing device 100. HDD 1203 is a storage device for recording image data and the like.

The I / F 1204 communicates with an external device via the network 140 according to TCP / IP, HTTP, or the like.

In the description of each of the above-described embodiments, an example in which the CPU 1200 executes the processing will be described, but at least a part of the processing of the CPU 1200 may be performed by dedicated hardware. For example, the process of displaying a GUI (GRAPHICAL USER INTERDEFACE) or image data on the display 130 may be executed by a GPU (GRAPHICS PROCESSING UNIT). Further, the process of reading the program code from the ROM 1202 and expanding it into the RAM 1201 may be executed by a DMA (Direct Memory Access) that functions as a transfer device.

The present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-described embodiment. The program may be supplied to a system or device having a processor via a network or storage medium. The present invention can also be realized by a circuit (for example, an ASIC) that realizes one or more functions of the above-described embodiment. Further, each part of the image processing apparatus 100 may be realized by the hardware shown in FIG. 7 or by software.

In addition, another device may have one or more functions of the image processing device 100 according to each of the above-described embodiments. For example, the image pickup apparatus 110 may have one or more functions of the image processing apparatus 100 according to each embodiment. It should be noted that each of the above-described embodiments may be combined, for example, any combination of the above-described embodiments may be carried out.

Although the present invention has been described above with the embodiments, the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention is construed in a limited manner by these. It's not a thing. That is, the present invention can be implemented in various forms within a range that does not deviate from the technical idea or its main features. For example, a combination of the respective embodiments is also included in the disclosure contents of the present specification.

Claims (19)

A detection means that executes a detection process that detects a specific object from an image captured by the imaging means,
A holding means for holding object information indicating the position and size of the specific object on the image, and
A tracking means that tracks the same specific object based on the detection process for each of a plurality of images captured at different times by the imaging means.
A setting means for setting a plurality of estimation regions for an image captured by the imaging means based on the object information corresponding to the same specific object tracked by the tracking means.
An image processing apparatus comprising: an estimation means for executing an estimation process for estimating the number of the specific object included in each of the plurality of estimation regions. The holding means accumulates the object information of the specific object each time the specific object is detected from the image.
The setting means is characterized in that a plurality of estimation regions are set for an image captured by the imaging means based on the object information accumulated for the same specific object tracked by the tracking means. The image processing apparatus according to claim 1. The setting means sets a plurality of estimation regions for an image captured by the imaging means based on the object information corresponding to the same specific object whose distance tracked by the tracking means is equal to or greater than a predetermined value. The image processing apparatus according to claim 1 or 2. When the distance from the first position in the locus of the same specific object tracked by the tracking means to the second position in the locus is a predetermined value or more, the setting means corresponds to the same specific object. The image processing apparatus according to claim 1 or 2, wherein a plurality of estimation regions are set for an image captured by the imaging means based on the object information. The setting means is based on the object information corresponding to the specific object detected by the detection process in the detection region on the image and the object information corresponding to the same specific object tracked by the tracking means. The image processing apparatus according to any one of claims 1 to 4, wherein a plurality of estimation regions are set for an image captured by the imaging means. Further, it has a determination means for determining whether or not the number of times the specific object is detected in the detection region on the image has reached a predetermined value.
When the determination means determines that the number of times the specific object has been detected in the detection region has reached the predetermined value, the setting means has the object information corresponding to the specific object detected in the detection region. The claim is characterized in that a plurality of estimation regions are set for an image captured by the imaging means based on the object information corresponding to the same specific object tracked by the tracking means. The image processing apparatus according to 5. The image processing according to any one of claims 1 to 6, wherein the estimation means estimates the number of the specific objects included in the estimation region by using a trained model for the specific object. apparatus. The image processing apparatus according to any one of claims 1 to 7, further comprising an output means for outputting information indicating the result of the estimation processing for each of the plurality of estimation regions. The image processing apparatus according to any one of claims 1 to 8, wherein the specific object is a person. A detection step that executes a detection process to detect a specific object from an image captured by an imaging means,
A holding step of holding object information indicating the position and size of the specific object on the image, and
A tracking step of tracking the same specific object based on the detection process for each of a plurality of images captured at different times by the imaging means.
A setting step of setting a plurality of estimation regions for an image captured by the imaging means based on the object information corresponding to the same specific object tracked in the tracking step.
An image processing method comprising an estimation step of executing an estimation process for estimating the number of the specific object included in each of the plurality of estimation regions. In the holding step, each time the specific object is detected from the image, the object information of the specific object is accumulated.
The setting step is characterized in that a plurality of estimation regions are set for an image captured by the imaging means based on the object information accumulated for the same specific object tracked by the tracking step. The image processing method according to claim 10. In the setting step, a plurality of estimation regions are set for an image captured by the imaging means based on the object information corresponding to the same specific object whose tracking distance is equal to or greater than a predetermined value in the tracking step. The image processing method according to claim 10 or 11, wherein the image processing method is performed. In the setting step, when the distance from the first position in the locus of the same specific object tracked in the tracking step to the second position in the locus is a predetermined value or more, it corresponds to the same specific object. The image processing method according to claim 10 or 11, wherein a plurality of estimation regions are set for an image captured by the imaging means based on the object information. Based on the object information corresponding to the specific object detected by the detection process in the detection region on the image in the setting step and the object information corresponding to the same specific object tracked in the tracking step. The image processing method according to any one of claims 10 to 13, wherein a plurality of estimation regions are set for an image captured by the imaging means. Further, it has a determination step of determining whether or not the number of times the specific object is detected in the detection region on the image has reached a predetermined value.
When it is determined in the determination step that the number of times the specific object is detected in the detection region has reached the predetermined value, the object information corresponding to the specific object detected in the detection region in the setting step. The claim is characterized in that a plurality of estimation regions are set for an image captured by the imaging means based on the object information corresponding to the same specific object tracked in the tracking step. 14. The image processing method according to 14. The image processing according to any one of claims 10 to 15, wherein in the estimation step, the number of the specific objects included in the estimation region is estimated by using the trained model for the specific object. Method. The image processing method according to any one of claims 10 to 16, further comprising an output step of outputting information indicating the result of the estimation processing for each of the plurality of estimation regions. The image processing method according to any one of claims 10 to 17, wherein the specific object is a person. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 10 to 18.

Images