JP6889088B2 - Object distribution estimation system - Google Patents

Description

The present invention relates to an object distribution estimation system that transmits a time-series photographed image in which a space in which a predetermined object such as a person can exist is sequentially photographed and estimates the distribution of the object from the transmitted photographed image.

In spaces where congestion may occur, such as event venues, it is necessary to take measures such as allocating a large number of security guards in areas where congestion is occurring in order to prevent accidents. Therefore, by arranging surveillance cameras at various places in the venue, estimating the distribution of people from the captured image, and displaying the estimated distribution, it is possible to facilitate the grasp of the congestion situation by the observer.

One of the methods for estimating the distribution of people is to scan the captured image with a classifier that has learned the characteristics of the image at the time of congestion of people in advance. For example, in the crowd analysis device described in Patent Document 1 below, the person density is estimated using a classifier machine-learned for each person density using a learning image in which a crowd having a person density exceeding the lower limit of density is captured in advance. It is stated that by doing so, the occurrence of the crowd is determined.

Japanese Unexamined Patent Publication No. 2017-068598

Event venues and the like are generally vast, and a large number of images taken by a large number of surveillance cameras are acquired every moment and become an estimation target. Therefore, when connecting a large number of these surveillance cameras to an image analysis center or the like and performing estimation in a unified manner, it is strongly required to suppress the communication cost related to the transmission of captured images from the surveillance cameras to the image analysis center or the like. ..

Here, in the area where people are crowded in the captured image, since the movement of people is slow, it is difficult for a sudden change in the congestion situation to occur, and the change that occurs in a short time in the display of the distribution of people is minute.

However, in the prior art, the entire captured image is transmitted every time the captured image is input regardless of the presence or absence of a congested area, which causes a wasteful communication cost.

The present invention has been made in view of the above problems, and a time-series photographed image in which a space in which a predetermined object can exist is sequentially photographed is transmitted while suppressing communication costs, and an object is transmitted from the transmitted photographed image. It is an object of the present invention to provide a distribution estimation system for estimating the distribution of.

(1) the object distribution estimation system according to the present invention, the image capturing apparatus for capturing a space in which a predetermined object may be present at a predetermined frame rate, by analyzing the shooting image to estimate the distribution of the object in the space A system in which a distribution estimation device and a relay device that relays the captured image from the imaging device to the distribution estimation device are connected via a network, and a plurality of the distribution estimation devices are set in the captured image. The distribution is estimated by obtaining the degree of congestion of the object in each of the local regions by the analysis of each local region, and the relay device estimates the distribution of the captured image in each of the local regions and the distribution of the local region. It relays at a frame rate according to the degree of congestion obtained by the apparatus.

(2) In the object distribution estimation system according to (1) above, the relay device captures the captured image in each of the local regions, and the higher the degree of congestion obtained by the distribution estimation device for the local region, the lower the frame. It can be configured to relay at a rate.

(3) In the object distribution estimation system according to the above (1) and (2), the relay device responds to the lowest degree of congestion in the block for each block consisting of two or more of the local regions. It can be configured to relay at a different frame rate.

(4) In the object distribution estimation system according to the above (1) to (3), the distribution estimation device captures the characteristics of each density image obtained by photographing the space in which the object exists at the density at a predetermined density. Using the learned density estimator, the density of the object representing the degree of congestion can be obtained for each local region.

According to the present invention, since an image in a region where the degree of congestion of the imaged object is high is transmitted at a lower frame rate, a time-series photographed image in which a space in which a predetermined object such as a person can exist is sequentially photographed is transmitted. It is possible to estimate the distribution of objects with high reliability from the transmitted captured image while suppressing the communication cost and transmitting the object.

It is a block diagram which shows the schematic structure of the object distribution estimation system which concerns on embodiment of this invention. It is a functional block diagram of the photographing apparatus in the object distribution estimation system which concerns on embodiment of this invention. It is a functional block diagram of the relay device in the object distribution estimation system which concerns on embodiment of this invention. It is a functional block diagram of the distribution estimation apparatus in the object distribution estimation system which concerns on embodiment of this invention. It is a functional block diagram of the display device in the object distribution estimation system which concerns on embodiment of this invention. It is a schematic flow chart of the operation of the relay device in the object distribution estimation system which concerns on embodiment of this invention. It is a schematic flow chart of the operation of the distribution estimation apparatus in the object distribution estimation system which concerns on embodiment of this invention. It is a schematic flow chart of the density estimation process for each block in a distribution estimation device. It is a schematic diagram which shows the processing example at a certain time t of the object distribution estimation system which concerns on embodiment of this invention. It is a schematic diagram which shows the processing example at time t + 1 of the object distribution estimation system which concerns on embodiment of this invention. It is a schematic diagram which shows the processing example at time t + 2 of the object distribution estimation system which concerns on embodiment of this invention. It is a schematic diagram which shows the processing example at time t + 3 of the object distribution estimation system which concerns on embodiment of this invention. It is a schematic diagram which shows the processing example at time t + 4 of the object distribution estimation system which concerns on embodiment of this invention.

Hereinafter, the object distribution estimation system 1 according to the embodiment of the present invention (hereinafter referred to as the embodiment) will be described with reference to the drawings. The object distribution estimation system 1 estimates the distribution of people in a space (surveillance space) in which a person can exist by sequentially analyzing captured images taken at predetermined time intervals, and estimates the estimation result to the observer. indicate.

[Configuration of object distribution estimation system 1]
FIG. 1 is a block diagram showing a schematic configuration of the object distribution estimation system 1. The object distribution estimation system 1 includes a photographing device 2, a relay device 3, a distribution estimation device 4, and a display device 5. For example, the object distribution estimation system 1 includes a plurality of photographing devices 2-1a, 2-1b, ..., 2-2a, 2-2b, ... As the photographing device 2, and a plurality of relay devices 3-1 as the relay device 3. , 3-2, ... Included.

Each of the plurality of photographing devices 2 is a so-called network camera, and is connected to a predetermined communication network (hereinafter referred to as a network) such as the Internet, and sets different target spaces in the monitoring space at a predetermined frame rate, that is, A photographed image is generated by photographing at a predetermined time interval (shooting cycle), and the photographed image is sequentially transmitted to the distribution estimation device 4 and the display device 5. Hereinafter, the unit of time engraved in the shooting cycle is set to 1 hour.

For example, each of the plurality of photographing devices 2 is installed on a pole installed at the event venue with a field of view overlooking the target space, and is connected to the Internet by wire or wirelessly. The field of view may be fixed or may be changed according to a prior schedule or external instructions via the Internet. For example, the photographing device 2 photographs the target space with a photographing cycle of 1 second to generate a color image. Further, the photographing device 2 may generate a monochrome image instead of the color image.

Each of the plurality of relay devices 3 is a server computer for relay, and is connected to the above network to communicate between the photographing device 2 and the distribution estimation device 4 and the display device 5, and the distribution estimation device 4 and the display device. The communication with 5 is relayed.

Specifically, each relay device 3 transmits a captured image from the photographing device 2 to the distribution estimation device 4, transmits the captured image from the photographing device 2 to the display device 5, and transmits the captured image from the distribution estimation device 4 to the display device 5. The transmission of the object distribution of is relayed.

Further, each relay device 3 has a filter function for the captured image, and reduces the communication cost by thinning out a part of the captured image addressed to the distribution estimation device 4. Hereinafter, the captured image addressed to the distribution estimation device 4 after filtering is also referred to as an estimation image, and in order to distinguish it from this, the captured image addressed to the display device 5 relayed to the relay device 3 is also referred to as a display image.

Specifically, each relay device 3 relays the estimation image to the distribution estimation device 4, and the degree of congestion is determined based on the degree of congestion of each local region in the estimation image analyzed by the distribution estimation device 4. Images in the higher local region are relayed at a lower frame rate, that is, at longer time intervals, and images in the local region, which are less congested, are relayed at a higher frame rate, that is, at shorter time intervals. The time interval corresponding to the frame rate for transmitting these local regions is referred to as a transmission interval. For example, the degree of congestion is set in three stages, the transmission interval of the image in the region with the highest degree of congestion is 4 hours, the transmission interval of the image in the region with the lowest degree of congestion is 1 hour, and the transmission interval of the image in the region with the intermediate degree of congestion is 1 hour. The transmission interval of is 2 hours.

The transmission interval changes from moment to moment due to changes in the congestion status of the target space, and is asynchronous between the local regions set in the captured image. In order to control the transmission interval in real time, every time the degree of congestion in the local area is estimated, the transmission interval according to the degree of congestion is set as the transmission waiting time for the image in the local area. Each time an image for estimation is acquired, the transmission waiting time is counted down, and the image of the local area where the transmission waiting time is 0 is transmitted (the image of the local area where the transmission waiting time is not 0 is thinned out and transmitted). Is controlled.

Therefore, each relay device 3 receives the setting of the transmission waiting time addressed to its own device from the distribution estimation device 4, and performs filtering based on the received setting.

The connection relationship between the plurality of photographing devices 2 and the plurality of relay devices 3 may be fixed or dynamically changed, but during the period illustrated in the present embodiment, the photographing devices 2-1a , 2-1b are connected to the relay device 3-1 and the photographing devices 2-2a and 2-2b are connected to the relay device 3-2.

The distribution estimation device 4 is a server computer for image processing, is connected to the above network, and is installed in, for example, an image analysis center in a remote location away from the event venue. The distribution estimation device 4 captures an estimation image captured by the photographing devices 2-1a, 2-1b, ..., 2-2a, 2-2b, ... And filtered by the relay device 3-1, 3-2, ... For each local region of the received estimation image, the degree of congestion of people (objects) in the local region is analyzed, and the information of the object distribution as the analysis result is transmitted to the display device 5.

Further, the distribution estimation device 4 sets the transmission interval in each local region according to the degree of congestion in each local region, and updates and updates the transmission waiting time set based on the transmission interval according to the passage of time. The information on the transmission waiting time is transmitted to the relay device 3 to which the corresponding photographing device 2 is connected.

The display device 5 is a terminal device such as a PC (Personal Computer) equipped with a monitor, is connected to the above network, and is installed in, for example, a monitoring center in an event venue. The display device 5 receives the display image captured by the photographing devices 2-1a, 2-1b, ..., 2-2a, 2-2b, ..., And the information on the object distribution analyzed by the distribution estimation device 4. Is received, the information of the object distribution corresponding to the image is synthesized with the display image, and the composite image is displayed on the monitor. By visually recognizing the displayed composite image, the observer grasps the point where congestion is occurring in the monitoring space, and takes measures such as dispatching or increasing the number of guards to the point.

[Functions of each device constituting the object distribution estimation system 1]
Hereinafter, the functions of each device constituting the object distribution estimation system 1 will be described with reference to FIGS. 2 to 5. Here, the plurality of photographing devices 2 and the plurality of relay devices 3 have the same functions although the connection relationship and some settings such as the target space are different.

FIG. 2 is a functional block diagram of the photographing device 2. As described above, the photographing devices 2-1a, 2-1b, ..., 2-2a, 2-2b, ... Have common functions to each other, and the photographing device 2 shown in FIG. 2 is any one of them. The photographing device 2 includes an image acquiring means 20 and a photographed image transmitting means 21.

The image acquisition means 20 includes an image pickup element such as a CCD and optical components such as a lens. The image acquisition means 20 of each photographing device 2 photographs the target space assigned to the photographing device 2 at the frame rate described above to generate a photographed image, and sequentially outputs the generated photographed image to the photographed image transmitting means 21. ..

The captured image transmitting means 21 includes a communication interface circuit with a network, a memory, a clock circuit, and the like. The camera ID and the address of the distribution estimation device 4 assigned in advance for each photographing device 2 are stored in the memory in advance. In addition, the current time is clocked by the clock circuit. The captured image transmitting means 21 assigns a camera ID and a current time (shooting time) to the captured image input from the image acquiring means 20, sets the address of the distribution estimation device 4 as a destination, and sends the captured image to the network. To do.

FIG. 3 is a functional block diagram of the relay device 3. As described above, the relay devices 3-1, 3-2, ... Have the same function as each other, and the relay device 3 shown in FIG. 3 is any one of them. The relay device 3 includes a computing device such as a CPU (Central Processing Unit), a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and a communication interface circuit with a network. , The captured image storage means 31, the waiting time receiving means 32, the waiting time storage means 33, the estimation image transmitting means 34, the display image transmitting means 35, and the like. Programs and various setting values are stored in the memory in advance, and the arithmetic unit performs various processes according to these, sends and receives various information via the network in cooperation with the communication interface circuit, and communicates with the memory. It operates as each means by inputting and outputting various information.

The captured image receiving means 30 of each relay device 3 receives the captured image transmitted to the distribution estimation device 4 and the captured image transmitted to the display device 5 from the photographing device 2 connected to the relay device 3. Then, each image is stored in the captured image storage means 31.

The waiting time receiving means 32 of each relay device 3 receives the transmission waiting time information transmitted from the distribution estimation device 4 to the relay device 3, and stores the information in the waiting time storage means 33. The transmission waiting time information is the transmission waiting time of each local region in the corresponding captured image. As will be described later, the local region is a pixel, and the transmission waiting time is set in block units each consisting of two or more pixels. That is, the transmission waiting time information includes the camera ID assigned to the corresponding captured image and the transmission waiting time of each block of the corresponding captured image. Incidentally, in the present embodiment, the connection relationship between the relay device 3 and the photographing device 2 is fixed, and the waiting time storage means 33 of each relay device 3 basically corresponds to the photographing device 2 connected to the relay device 3. The information of the transmission waiting time is stored. When the photographing device 2 connected to the relay device 3 is dynamically changed, for example, the waiting time storage means 33 of each relay device 3 may store the information about all the photographing devices 2. ..

The estimation image transmitting means 34 reads the transmission waiting time information from the waiting time storage means 33, and reads the captured image addressed to the distribution estimation device 4 corresponding to the information from the captured image storage means 31, and among the images. An estimation image consisting of only the pixel values of the pixels belonging to the block having the transmission waiting time of 0 is generated. Then, the camera ID and the shooting time assigned to the original captured image are assigned to the estimation image, the address of the distribution estimation device 4 is set as the destination, and the estimation image is transmitted to the network.

The display image transmitting means 35 reads the captured image addressed to the display device 5 from the captured image storage means 31 and sends it to the network.

FIG. 4 is a functional block diagram of the distribution estimation device 4. The distribution estimation device 4 includes an arithmetic unit such as a CPU, a memory such as a ROM and a RAM, and a communication interface circuit with a network, and includes an image receiving means 40 for estimation, a density estimator storage means 41, and a density estimation means 42. It operates as an object distribution storage means 43, a transmission interval setting means 44, a waiting time storage means 45, a waiting time transmission means 46, an object distribution transmission means 47, and the like. Programs and various setting values are stored in the memory in advance, and the arithmetic unit performs various processes according to these, sends and receives various information via the network in cooperation with the communication interface circuit, and communicates with the memory. It operates as each means by inputting and outputting various information.

The estimation image receiving means 40 receives an estimation image which is a captured image sent from the photographing device 2 to the distribution estimation device 4 and filtered by the relay device 3 on the way, and outputs the image to the density estimation means 42. To do.

The density estimator storage means 41 is an estimated density calculation function that learns the image features of each density image obtained by photographing a space in which an object (person) exists at the density at a predetermined density, and obtains the feature amount of the image. When input, the estimated value (estimated density) of the density of the object captured in the image is calculated, and the information of the estimator (density estimator) that outputs the calculated estimated density is stored in advance. That is, parameters such as the coefficient of the estimated density calculation function are stored in advance as information of the density estimator.

The density estimation means 42 estimates the distribution of objects in the target space by obtaining the degree of congestion of the objects in each of the local regions by analyzing each of a plurality of local regions set in the captured image. Specifically, the density estimation means 42 extracts features for density estimation for an arbitrary local region in the estimation image input from the estimation image receiving means 40, and also estimates the density from the density estimator storage means 41. The density in the local region is estimated by reading out the instrument and inputting the extracted features into the density estimator. By performing this estimation at a plurality of positions in the captured image, the distribution of objects in the target space can be obtained. The density estimation means 42 stores the coordinates of the local region and the estimated density in the object distribution storage means 43 in association with the camera ID and the shooting time assigned to the estimation image, and stores the estimated density in the transmission interval setting means 44. input.

The estimated density estimated here is information that is partially thinned out corresponding to the image for estimation, but the object distribution storage means 43 stores the estimated density over the entire local region in combination with the past estimated density. It will be in the state of being. The density estimation means 42 stores the newly estimated estimated density in the object distribution storage means 43, then reads out the estimated density of the latest shooting time of each local region from the object distribution storage means 43 for each camera ID, and the camera ID. And the latest shooting time is given in the entire local area and output to the object distribution transmitting means 47.

The processing of density estimation and the density estimator will be specifically described.

In the present embodiment, each pixel in the captured image is set as a local region. The density estimation means 42 sets a window for density estimation at the position of each pixel of the estimation image, and extracts a feature amount from the estimation image in each window. The feature quantity is a GLCM (Gray Level Co-occurrence Matrix) feature.

It is desirable that the areas in the target space photographed by each window are the same size. That is, preferably, the density estimation means 42 reads out the camera parameters of the photographing device 2 stored in advance from the camera parameter storage means (not shown), and images the captured images at arbitrary pixels by homography conversion using the camera parameters. After transforming the estimation image so that the areas in the target space are the same size, the windows are set and the features are extracted. Each window can be an area having the same shape and size as the density image described later.

The density estimator can be realized by a discriminator that discriminates a multi-class image, and can be a discriminant function learned by a multi-class SVM (Support Vector Machine) method.

Density, for example, there is no human "Background" class is 0 people / m higher than ² is two / m ² or less "low density" class, higher than two / m ² 4 persons / m ² or less It can be defined as 4 classes of "medium density" class and ^{"high density" class higher than 4 people / m 2.}

The estimated density is a value given in advance to each class and is a value output as a result of distribution estimation. In this embodiment, the values corresponding to each class are described as "background", "low density", "medium density", and "high density".

That is, the density estimator applies the multi-class SVM method to the features of a large number of images (density images) belonging to each of the "background" class, "low density" class, "medium density" class, and "high density" class. This is an identification function for distinguishing the density image of each class from other classes, which is obtained by learning. The parameters of the discriminant function derived by this learning are stored as a density estimator. The feature amount at the time of learning is the same type as the feature amount extracted by the density estimation means 42, and is a GLCM feature.

The density estimation means 42 acquires the estimated density, which is the output value, by inputting each of the feature quantities extracted corresponding to each pixel into the density estimator. When the feature amount is extracted by deforming the estimation image, the density estimation means 42 transforms the density distribution into the shape of the original estimation image by homography conversion using camera parameters.

The object distribution is a set of estimated densities obtained in this way for each pixel of the captured image. Here, in the local region (highly congested region) where the estimated density is in the “high density” class, the movement of people is slowed down due to congestion, and a sudden change in the degree of congestion is unlikely to occur. Therefore, the highly congested area has little influence on the judgment of the observer even if it is not analyzed at short time intervals. On the other hand, in the local region (low congestion region) where the estimated density is the "background" class and the "low density" class, it is possible for a person to run and a sudden change in the degree of congestion can occur. Therefore, it is desirable to analyze the low-congestion area at short time intervals and present the state of change to the observer in sequence. Further, the speed of movement of a person in a local region (medium-congested region) whose estimated density is in the "medium-density" class is considered to be an intermediate speed between a high-congested region and a low-congested region.

Therefore, the object distribution estimation system 1 suppresses the communication cost of density estimation while ensuring the reliability of the density estimation result by transmitting and receiving images at longer transmission intervals in the local region where the estimated density is higher.

Therefore, the transmission interval setting means 44 sets the transmission interval of each local region according to the estimated density estimated by the density estimation means 42, and relays the image of each local region at the transmission interval set in the local region. Send to 3.

For example, the transmission interval setting means 44 sets the transmission interval of the local region whose estimated density is the “high density” class to 4 hours, and the transmission interval of the local region whose estimated density is the “medium density” class to 2 hours. , The transmission interval of the local region whose estimated density was the "background" class and the local region whose estimated density was the "low density" class is set to 1 time.

Specifically, in order to control the transmission interval in real time, the transmission interval setting means 44 writes, for example, the transmission interval for each local area as the transmission waiting time in the waiting time storage means 45, and makes a new estimation. The transmission waiting time is reduced each time an image is acquired.

Further, among the high-congested areas, the vicinity of the boundary with the medium-congested area or the low-congested area is more likely to change in the degree of congestion than the other areas. Similarly, in the medium-congested region near the boundary with the low-congested region, the degree of congestion is more likely to change than in the other regions. Therefore, the transmission interval setting means 44 sets the transmission interval according to the lowest density in the block for each block each consisting of two or more local regions. The block is set in advance, and the transmission interval setting means 44 and the density estimation means 42 share the setting. The size of the block should be about one or two objects (one or two people). This size is the smallest unit that causes a change in the degree of congestion. By forming blocks according to the size of the object in this way, it is possible to prevent an update delay near the boundary and prevent unnecessary frequent updates other than near the boundary.

The transmission waiting time set by the transmission interval setting means 44 may be information in which some blocks are thinned out corresponding to the estimation image, but the waiting time storage means 45 has a transmission waiting time set in the past. Combined with the time, the transmission waiting time over all blocks is stored. The transmission interval setting means 44 reads the transmission waiting time of all blocks from the waiting time storage means 45 after storing the new transmission waiting time in the waiting time storage means 45, and assigns the camera ID to the estimation image. And the shooting time is given and output to the waiting time transmission means 46.

The waiting time transmission means 46 transmits the transmission waiting time information input from the transmission interval setting means 44 to the network by setting the address of the relay device 3 corresponding to the camera ID given to the information as the destination. ..

The object distribution transmitting means 47 transmits the object distribution information input from the density estimation means 42 to the network by setting the address of the display device 5 as the destination.

FIG. 5 is a functional block diagram of the display device 5. The display device 5 includes a monitor such as a liquid crystal display, a computing device such as a CPU, a memory such as a ROM and a RAM, and a communication interface circuit with a network, and includes a display image receiving means 50, an object distribution receiving means 51, and an image. It operates as a synthesis means 52, an object distribution display means 53, and the like. Programs and various setting values are stored in the memory in advance, and the arithmetic unit performs various processes according to these, sends and receives various information via the network in cooperation with the communication interface circuit, and cooperates with the monitor. It operates as each means by displaying various information and inputting / outputting various information to / from the memory.

The display image receiving means 50 receives a display image which is a captured image sent from the photographing device 2 to the display device 5 and relayed by the relay device 3, and outputs the image to the image synthesizing means 52.

The object distribution receiving means 51 receives the information of the object distribution sent from the distribution estimation device 4 to the display device 5 and relayed by the relay device 3, and outputs the information to the image synthesizing means 52.

The image synthesizing means 52 synthesizes the display image input from the display image receiving means 50 and the object distribution information input from the object distribution receiving means 51 that have the same camera ID and shooting time. Then, the composite image is output to the object distribution display means 53.

At that time, the image synthesizing means 52 has, for example, red for the pixel value of the local region estimated to be the "high density" class, yellow for the pixel value of the local region estimated to be the "medium density" class, and "low density" class. A distribution image in which the estimated pixel value of the local region is set to green and the pixel value of the estimated local region in the "background" class is set to black is generated and transparently combined with the display image.

The object distribution display means 53 displays the composite image input from the image synthesis means 52 on the monitor.

[Operation of object distribution estimation system 1]
The operation of the object distribution estimation system 1 will be described with reference to the flow charts of FIGS. 6 to 8.

When the object distribution estimation system 1 is activated, in the photographing device 2, the image acquisition means 20 photographs the target space at a predetermined imaging cycle, and the captured image transmitting means 21 sequentially transfers the captured images to the distribution estimation device 4 and the display device 5. Send to the address. Further, when the field of view is changed in the photographing device 2, the photographing device 2 notifies the distribution estimation device 4 that the field of view has been changed.

FIG. 6 is a schematic flow chart of the operation of the relay device 3. The captured image transmitted by the photographing device 2 is relayed by the relay device 3. When the captured image receiving means 30 of the relay device 3 receives the captured image from an arbitrary photographing device 2 (when “YES” in step S30), the captured image receiving means 30 stores the captured image in the captured image storage means 31 (step S31). .. On the other hand, when no captured image is received from any of the photographing devices 2 (in the case of "NO" in step S30), step S31 is skipped.

Further, the distribution estimation device 4 transmits updated transmission waiting time information based on the estimation image of the previous time to the relay device 3. When the waiting time receiving means 32 of the relay device 3 receives the transmission waiting time information to which the camera ID is assigned for the arbitrary photographing device 2 from the distribution estimation device 4 (when “YES” in step S32), the said The information is updated by overwriting the information with the transmission waiting time information having the same camera ID stored in the waiting time storage means 33 (step S33).

When the transmission waiting time information is updated, the estimation image transmitting means 34 of the relay device 3 refers to the captured image storage means 31 and is given the same camera ID as the updated information, and from the information. Also confirms whether or not the captured image addressed to the distribution estimation device 4 and the captured image addressed to the display device 5 to which a new shooting time is assigned have been received (step S34).

When the relay device 3 has already received the corresponding captured image (when “YES” in step S34), the estimation image transmitting means 34 receives an updated transmission waiting time from the captured image addressed to the distribution estimation device 4. Only the pixels of the block whose transmission waiting time is 0 are extracted from the information of the above to generate an estimation image, and the generated estimation image is transmitted to the distribution estimation device 4 (step S35).

In this embodiment, as will be described later, the distribution estimation device 4 is used to count down the transmission waiting time. Therefore, in order to notify the distribution estimation device 4 of the passage of one time of the transmission waiting time synchronized with the reception of the captured image from the photographing device 2, the relay device 3 determines the presence or absence of the block in which the transmission waiting time is 0 in step S35. Regardless, the estimation image is transmitted. Here, the estimation image when there is no block having a transmission waiting time of 0 is substantially empty data that does not include the pixel value of the captured image, and is simply data for notifying the distribution estimation device 4 of the passage of time. It becomes.

Further, the display image transmission means 35 of the relay device 3 transmits the display image to the display device 5 (step S36).

When the updated transmission waiting time information is not received from the distribution estimation device 4 for any of the photographing devices 2 (when “NO” in step S32), the processes of steps S33 to S36 are skipped. To. Further, when the photographed device 2 for which the updated transmission waiting time information has been received has not yet received a photographed image newer than the information (in the case of "NO" in step S34), steps S34 to S36 The process is skipped.

When the above processing is completed, the relay device 3 returns the processing to step S30 and repeats the above-mentioned operations of steps S30 to S36.

FIG. 7 is a schematic flow chart of the operation of the distribution estimation device 4. The distribution estimation device 4 executes a loop process including steps S50 to S58.

First, it is confirmed whether or not the transmission interval setting means 44 is at the time of initial activation and whether or not the field of view has been changed (step S50). That is, the transmission interval setting means 44 refers to the waiting time storage means 45, and if the transmission waiting time information is not stored, the transmission interval setting means 44 determines that it is the first start-up. Further, the transmission interval setting means 44 confirms the notification from the photographing device 2 via the relay device 3, and if the notification for changing the field of view is received, it is determined that the field of view has been changed.

When the image is started for the first time, or when the field of view is changed in the photographing device 2 (when “YES” in step S50), the transmission interval setting means 44 initializes the transmission waiting time information (step S51, S52). The initialization process is performed on the transmission waiting time information of all the photographing devices 2 at the time of the first startup. On the other hand, in the case of changing the field of view of the photographing device 2, the initialization process is performed only on the photographing device 2.

In the initialization process, the transmission interval setting means 44 generates transmission waiting time information in which the transmission waiting time of all blocks in the captured image is initialized to 0 [time] for the photographing device 2 to be initialized (step S51). The waiting time transmission means 46 transmits the transmission waiting time information to the relay device 3 (step S52). Further, the transmission waiting time information is stored in the waiting time storage means 45. By the way, as soon as the first start-up or the change of the field of view is detected, the transmission waiting time information initialized to 0 is transmitted to the relay device 3, so that the captured image after the start-up or the change of the field of view is promptly used as an estimation image. It can be transmitted to the distribution estimation device 4 (step S35 in FIG. 6).

If the field of view is not changed at the time of initial activation (NO in step S50), the processes of steps S51 and S52 are skipped, and the values already stored in the waiting time storage means 45 are maintained.

Next, it is confirmed whether or not the estimation image receiving means 40 receives the estimation image which is the captured image transmitted from the photographing device 2 and filtered by the relay device 3 as needed (step S53). ..

When the estimation image is received (when "YES" in step S53), the estimation image receiving means 40 inputs the estimation image to the density estimation means 42, and the density estimation means 42 into which the estimation image is input. Sequentially sets each block as a block of interest and executes the loop processing of steps S54 to S56. That is, assuming that the block having the transmission waiting time of 0 is the block for which the update timing of the estimated density has arrived, the process of updating the estimated density is performed for the block.

FIG. 8 is a schematic flow chart of the process S55 relating to the density estimation for each block. The process S55 will be described with reference to FIG.

As described above, the estimation image includes only the pixel value information of the pixel belonging to the block in which the transmission waiting time is 0 among the pixel values constituting the image captured by the photographing device 2, and includes information on the pixel values of the other pixels. Pixel values are omitted images. The data of the estimation image can obtain high compression efficiency by omitting the pixel value of the block whose transmission waiting time is not 0, and can improve the transmission efficiency from the relay device 3 to the distribution estimation device 4. it can. Various well-known methods can be used for data compression of the estimation image.

In the present embodiment, when the estimation image receiving means 40 receives the estimation image data from the relay device 3, the estimation image receiving means 40 develops the data into an image having the same two-dimensional pixel array as the captured image output by the photographing device 2. The developed image is output to the density estimation means 42. In the developed image, only the pixels belonging to the block having the transmission waiting time of 0 have effective pixel values.

The density estimation means 42 confirms whether or not the transmission waiting time is 0 by referring to the waiting time storage means 45 for the attention block sequentially set in step S54 of FIG. 7 (step S80).

If the transmission waiting time of the attention block is 0 (when “YES” in step S80), the density estimation means 42 estimates the density of each pixel in the attention block in the estimation image, and estimates at that pixel. By overwriting and storing the density in the object distribution storage means 43, the density in each pixel in the attention block is updated (step S81).

That is, the density estimation means 42 sets a window based on the position of each pixel in the estimation image, extracts the feature amount in the window, and reads out the density estimator from the density estimator storage means 41. The feature amount extracted at the position of each pixel is input to the density estimator, and the estimated density in the pixel is obtained as the output value of the density estimator.

Further, the density estimation means 42 outputs the estimated density of each pixel in the block of interest to the transmission interval setting means 44. Based on the input estimated density, the transmission interval setting means 44 determines the update interval, which is the time until the next update of the estimated density for the block of interest, and the update interval is newly added to the waiting time storage means 45. Set as the transmission waiting time.

Specifically, the transmission interval setting means 44 confirms whether or not the estimated density input for each pixel in the block of interest includes a value indicating a "background" class or a "low density" class (step). S82). Then, when the attention block includes the "background" class or the "low density" class (when "YES" in step S82), the transmission interval setting means 44 pays attention stored in the waiting time storage means 45. The block transmission waiting time is updated to 1 [time] (step S83).

On the other hand, when the attention block does not include the "background" class or the "low density" class (when "NO" in step S82), the transmission interval setting means 44 has the "medium density" class in the input estimated density. It is confirmed whether or not the value indicating the above is included (step S84). When the attention block includes the "medium density" class (when "YES" in step S84), the transmission interval setting means 44 sets the transmission waiting time of the attention block stored in the waiting time storage means 45 to 2. Update to [Time] (step S85).

Further, when the attention block does not include the "medium density" class (when "NO" in step S84), the transmission interval setting means 44 assumes that the attention block contains only the "high density" class and stores the waiting time. The transmission waiting time of the attention block stored in the means 45 is updated to 4 [time] (step S86).

If the transmission waiting time of the attention block is not 0 (when “NO” in step S80), the processing of steps S81 to S86 for the attention block is omitted.

When the density estimation process S55 for the block of interest is completed in this way, the process proceeds to step S56 of FIG. 7, and the density estimation means 42 confirms whether or not all the blocks have been processed. If there is an unprocessed block (“NO” in step S56), the density estimation means 42 returns the process to step S54, sets the next block as the block of interest, and continues the loop process.

On the other hand, when all the blocks have been processed (when "YES" in step S56), a new transmission waiting time is transmitted to the relay device 3. Here, the transmission waiting time set with the update of the estimated density in step S56 is essentially counted down with the passage of time according to the subsequent shooting cycle of the shooting device 2. However, in the present embodiment, the distribution estimation device 4 counts down the transmission waiting time and transmits the transmission waiting time information to the relay device 3, and in order to simplify this process, the distribution estimation device 4 Then, the transmission waiting time is counted down without waiting for the reception of the next estimation image. That is, the transmission interval setting means 44 updates the transmission waiting time of each block set in step S55 and stored in the waiting time storage means 45 by 1 [time], and the waiting time transmitting means 46 updates the transmission waiting time. The updated transmission waiting time information is transmitted to the relay device 3 (step S57).

Further, the density estimation means 42 reads the estimated density of all pixels from the object distribution storage means 43 as an object distribution, outputs the object distribution to the object distribution transmission means 47, and the object distribution transmission means 47 is input from the density estimation means 42. The object distribution is transmitted to the display device 5 (step S58).

Note that steps S54 to S58 are omitted until a new estimation image is received (in the case of "NO" in step S53).

When the above processing is completed, the operation of the distribution estimation device 4 is returned to step S50, and the processing of steps S50 to S58 is repeated.

In the display device 5, the display image receiving means 50 receives the display image, which is a captured image transmitted from the photographing device 2 to the display device 5 and relayed by the relay device 3, and outputs the display image to the image combining means 52. .. Further, the object distribution receiving means 51 receives the information of the object distribution transmitted from the distribution estimation device 4 to the display device 5 and relayed by the relay device 3, and outputs the information to the image synthesizing means 52.

Then, in the display device 5, the image synthesizing means 52 converts the information of the object distribution into an image having a color value corresponding to the estimated density of each pixel, and the converted image corresponds to the object distribution. It is transparently combined with the display image and output to the object distribution display means 53.

[Processing example]
9 to 13 show how the transmission waiting time stored in the waiting time storage means 45 is updated by the processing over 5 hours from the time t to the time (t + 4), and the density of the blocks whose update timing has arrived. It is a schematic diagram which illustrates the state of being estimated, and the state of updating the estimated density stored in the object distribution storage means 43.

The analysis waiting time information 100, 102, 110, 112, 120, 122, 130, 132, 140, 142 schematically describes the arrangement of blocks in the two-dimensional coordinates corresponding to the captured image and the transmission waiting time of each block. It is shown as an image. In the image, each of the plurality of rectangles arranged two-dimensionally in a matrix represents a block, and the numerical value in the rectangle represents the transmission waiting time of the block represented by the rectangle.

Here, among the information 100, 110, 120, 130, 140 shown as "transmission waiting time (at the time of image acquisition)" in FIGS. 9 to 13, the information at any time T is basically the distribution estimation device 4. This is information stored in the waiting time storage means 45 of the distribution estimation device 4 and the waiting time storage means 33 of the relay device 3 when the estimation image at time T is received (step S53 in FIG. 7). On the other hand, the information 102, 112, 122, 132, 142 shown as "transmission waiting time (after density estimation)" specifically waits for transmission after the loop processing related to density estimation in steps S54 to S56 shown in FIG. 7 is completed. It represents the time and reflects the update result in the processing (steps S83, S85, S86) in step S55.

The density estimation results 101, 111, 121, 131, 141 shown as "density estimation results (updated portion)" in FIGS. 9 to 13 show the estimated densities newly obtained at each time in two dimensions corresponding to the captured images. It is schematically shown as an image in coordinates. The estimated density is the density updated in the process (step S81) in step S55 shown in FIG. 7, and only the portion of the block group corresponding to the captured image whose density is updated is the density estimation in FIGS. 9 to 13. It is shown as the resulting image.

Specifically, the "density estimation result (updated portion)" corresponds to the block in which the transmission waiting time is 0 in the information 100, 110, 120, 130, 140 shown as "transmission waiting time (at the time of image acquisition)". The density estimation results 101, 111, 121, 131, 141 shown as "" are generated.

In the images of the density estimation results in FIGS. 9 to 13, the white areas are pixels in the "background" class, the shaded areas are pixels in the "low density" class, and the horizontal lines are in the "medium density" class. Pixels and shaded areas represent pixels of the "high density" class.

As described with reference to FIG. 8, a new update interval is set for the update portion of the estimated density for which the density estimation results 101, 111, 121, 131, 141 are generated, and the result is reflected in the “waiting for transmission”. Information 102, 112, 122, 132, 142 shown as "time (after density estimation)" is generated.

The density estimation results 103, 113, 123, 133, and 143 shown as "density estimation results (synthesis results)" in FIGS. 9 to 13 correspond to the estimated densities stored in the object distribution storage means 43, respectively, corresponding to the captured images2. It is schematically shown in a dimensional image. Of the density estimation results 103, 113, 123, 133, and 143, those at arbitrary time T have the estimated density generated one time before the time and stored in the object distribution storage means 43, and the density estimation of the updated portion. The result 101, 111, 121, 131, 141 is generated by synthesizing the one generated at the time T, and is stored in the object distribution storage means 43. For example, the density estimation result 113 at time t + 1 is generated by overwriting the density estimation result 111 of the updated portion at time t + 1 with the density estimation result 103 at time t + 1.

Hereinafter, the processing at each time of time t to t + 4 will be described more specifically.

FIG. 9 relates to processing at time t. The captured image at time t is acquired immediately after the first activation or the change of the field of view, so that the transmission waiting time of all the blocks is initialized to 0 immediately before that (step S51 in FIG. 7), and the transmission waiting time is set to the relay device 3. It is transmitted (step S52 in FIG. 7) and stored in the waiting time storage means 33. As a result, the relay device 3 transmits an image of all blocks as an estimation image to the distribution estimation device 4, and the distribution estimation device 4 refers to the waiting time storage means 45 to reach the update timing of the estimation density for all blocks. As a result, the density is estimated for each pixel in all blocks as shown in the image of the density estimation result 101 based on the received estimation image.

Corresponding to the density estimation result 101, a new transmission waiting time is set according to the estimated density for all blocks, and transmission waiting time information 102 is generated. Specifically, the transmission waiting time of a block containing "background" class or "low density" class pixels is 1, and the transmission waiting time of a block containing other "medium density" class pixels is 2. The blocks other than the above are said to consist of pixels of only the "high density" class, and the transmission waiting time is updated to 4.

Further, the density estimation result 101 is generated for all blocks, and the estimated density of all blocks in the density estimation result 103 is updated with the information of the density estimation result 101.

FIG. 10 relates to processing at time t + 1. The transmission waiting time information 110 when the captured image at time t + 1 is acquired is obtained by subtracting the transmission waiting time of all blocks of the transmission waiting time information 102 by 1 in the process at time t (step S57 in FIG. 7). Is generated. As a result, the transmission waiting time of 63 blocks out of all 120 blocks becomes 0, and the relay device 3 transmits the estimation image including the 63 blocks to the distribution estimation device 4. Then, the distribution estimation device 4 estimates the density of these 63 blocks on the assumption that the update timing has arrived, and as a result, the density estimation result 111 is obtained.

Then, the transmission waiting time is updated for the 63 blocks whose density is estimated out of the transmission waiting time information 110, and the transmission waiting time information 112 is generated. Specifically, the transmission waiting time of 62 blocks including the pixels of the "background" class or the "low density" class out of the 63 blocks is 1, and the transmission waiting time of the other blocks including the pixels of the "medium density" class is 1. The transmission waiting time has been updated to 2. Incidentally, i-th row from the top, expressed from left j-th column block and B _ij, 1 block transmission latency is set to 2 is B _27. In the density estimation result 111, _{a horizontal line area indicating pixels of the "medium density" class is obtained in addition to B 27} , but some blocks in the area are in the "low density" class shown by the shaded area. The transmission waiting time is set to 1 because it includes the pixels of.

Further, the 63 blocks of the density estimation result 103 are updated with the information of the density estimation result 111 to generate the density estimation result 113.

FIG. 11 relates to processing at time t + 2. The transmission waiting time information 120 when the captured image at time t + 2 is acquired is obtained by subtracting the transmission waiting time of all blocks of the transmission waiting time information 112 by 1 in the process at time t + 1 (step S57 in FIG. 7). Is generated. As a result, the transmission waiting time becomes 0 for 91 blocks out of all 120 blocks, and the update timing arrives. Correspondingly, the relay device 3 transmits the estimation image including the 91 blocks to the distribution estimation device 4. Then, in the distribution estimation device 4, a density estimation result 121 composed of the 91 blocks is generated as an update portion of the estimated density, and the transmission waiting time is updated for the 91 blocks based on the updated estimated density, and the transmission waiting time is transmitted. Information 122 is generated. Specifically, the transmission waiting time of 64 blocks including "background" class or "low density" class pixels out of the 91 blocks is 1, and 27 blocks including "medium density" class pixels other than that. The transmission waiting time has been updated to 2. Further, the 91 blocks of the density estimation result 113 are updated with the information of the density estimation result 121 to generate the density estimation result 123.

The processing at time t + 3 shown in FIG. 12 and the processing at time t + 4 shown in FIG. 13 are also performed in the same manner, and the transmission waiting time at the time of image acquisition from the transmission waiting time updated at the previous time and stored in the waiting time storage means 45 is performed in the same manner. Is generated and transmitted to the relay device 3, the relay device 3 transmits only the pixels of the block whose update timing has arrived to the distribution estimation device 4 as an estimation image, and the distribution estimation device 4 transmits only the blocks whose update timing has arrived. Update the estimated density. Then, based on the result, the transmission waiting time is updated and the synthesis result of the estimated density of the entire captured image is generated.

As a result of such processing, the processing cost of density estimation for 144 blocks is reduced in 5 hours. On the other hand, in the density estimation results 103, 113, 123, 133, and 143, changes in the congestion situation are obtained with high accuracy.

[Modification example]
(1) In the above embodiment, an example in which the object to be detected is a human is shown, but the object to be detected is not limited to this, and the object to be detected may be a vehicle, an animal such as a cow or a sheep, or the like.

(2) In the above embodiment and its modifications, the density estimator learned by the multi-class SVM method is illustrated, but instead of the multi-class SVM method, a decision tree type random forest method and multi-class AdaBoost ( Various density estimators such as the density estimator learned by the AdaBoost) method or the multiclass logistic regression method can be used.

Alternatively, it can be a density estimator using an identification type CNN (Convolutional Neural Network).

(3) In the above-described embodiment and each modification thereof, the classes of densities other than the background estimated by the density estimator are set to 3 classes, but the classes may be further divided. In that case, the transmission interval is set at a finer stage corresponding to the classification.

(4) In the above-described embodiment and each modification thereof, a density estimator classified into multiple classes is illustrated, but instead of this, a regression-type density estimator that returns a density value (estimated density) from a feature amount. It can also be. That is, with a density estimator that has learned the parameters of the regression function for obtaining the estimated density from the features by the ridge regression method, the support vector regression method, the random forest method of the regression tree type, or the Gaussian Process Regression. can do.

Alternatively, it can be a density estimator using a regression type CNN.

In these cases, the range of the estimated density that is output as a continuous value instead of the value of the density class is set in association with the update interval.

(5) The transmission interval shown in the above embodiment and each modification thereof is an example, and depends on the shooting conditions such as the range setting of the estimated density, the speed of the detection target, the shooting cycle of the shooting device 2, and the angle of view. It can be another value suitable for those conditions.

(6) In the above-described embodiment and each modification thereof, the GLCM feature is exemplified as the feature amount learned by the density estimator and the feature amount used for the density estimation, but these are local binary patterns (instead of the GLCM feature). It can be various features such as Local Binary Pattern (LBP) feature, Haar-like feature, HOG (Histograms of Oriented Gradients) feature, brightness pattern, or GLCM feature and among them. It is also possible to use a combination of a plurality of features.

(7) In the above-described embodiment and each modification thereof, an example in which the scanning interval of the density estimation means 42 on the captured image is set to each pixel is shown, but the scanning interval is set to every two or more pixels. It is also possible to open the space.

(8) In the above-described embodiment and each modification thereof, an example is shown in which the density estimation means 42 exemplified as a means for estimating the degree of congestion estimates the distribution of an object by estimating the density of the object as the degree of congestion. However, as a process of estimating the degree of congestion, the distribution of objects can be estimated by analyzing the complexity of the image. For example, the captured image is divided into regions for adjacent pixels whose colors are similar to each other, and the divided regions having overlap with the blocks are counted for each of the blocks described above, and the higher the count value, the higher the complexity. Alternatively, frequency analysis is performed for each block of captured images, and the higher the average frequency, the higher the complexity. Then, the relationship between the complexity and the degree of congestion is determined through an experiment in advance (the higher the complexity, the higher the degree of congestion), and the degree of congestion corresponding to the obtained complexity is used as an estimated value for each block.

1 Object distribution estimation system, 2 Imaging device, 3 Relay device, 4 Distribution estimation device, 5 Display device, 20 Image acquisition means, 21 Captured image transmission means, 30 Captured image receiving means, 31 Captured image storage means, 32 Waiting time reception Means, 33, 45 Waiting time storage means, 34 Estimating image transmitting means, 35 Displaying image transmitting means, 40 Estimating image receiving means, 41 Density estimator storage means, 42 Density estimating means, 43 Object distribution storage means, 44 Transmission interval setting means, 46 waiting time transmission means, 47 object distribution transmission means.

Claims (4)

A photographing apparatus for photographing a space in which a predetermined object may be present at a predetermined frame rate, and distribution estimation device for estimating a distribution of the object in the space by analyzing the shadow image shooting, the distribution estimation device from the imaging device The relay device that relays the captured image is an object distribution estimation system connected via a network.
The distribution estimation device estimates the distribution by obtaining the degree of congestion of the object in each of the local regions by analyzing each of a plurality of local regions set in the captured image.
The relay device is an object distribution estimation system characterized in that the captured image in each of the local regions is relayed at a frame rate corresponding to the degree of congestion obtained by the distribution estimation device for the local region. The relay device according to claim 1, wherein the relay device relays the captured image in each of the local regions at a lower frame rate as the degree of congestion obtained by the distribution estimation device for the local region increases. Object distribution estimation system. According to claim 1 or 2, the relay device relays each block composed of two or more of the local regions at a frame rate corresponding to the lowest degree of congestion in the block. The described object distribution estimation system. The distribution estimation device expresses the degree of congestion for each local region by using a density estimator that learns the characteristics of each density image obtained by photographing a space in which the object exists at the density at a predetermined density. The object distribution estimation system according to any one of claims 1 to 3, wherein the density of the object is obtained.

Images