WO2019082652A1

WO2019082652A1 - Image sensor, person detection method, program, and control system

Info

Publication number: WO2019082652A1
Application number: PCT/JP2018/037693
Authority: WO
Inventors: 榎原　孝明; 禎敏齋藤
Original assignee: 株式会社東芝; 東芝インフラシステムズ株式会社
Priority date: 2017-10-25
Filing date: 2018-10-10
Publication date: 2019-05-02
Also published as: JP7002912B2; JP2019080177A

Abstract

According to an embodiment, this image sensor is equipped with an imaging unit, an extraction unit, an identification unit, and a detection unit. The imaging unit images a target space to acquire image data. The extraction unit extracts a motion feature amount from the image data. The identification unit identifies a motion type on the basis of the motion feature amount. The detection unit detects a person in the target space on the basis of the result of identification.

Description

Image sensor, person detection method, program and control system

Embodiments of the present invention relate to an image sensor, a person detection method, a program and a control system.

A recent image sensor has a CPU (Central Processing Unit) and a memory, and can be said to be an embedded computer with a lens. It also has advanced image processing functions, and can analyze the captured image data to calculate, for example, the presence or absence of people, or the number of people.

The interframe difference method is known as one of the methods of detecting moving objects from image data. The principle is that a background image to be a reference is stored in advance, a change in luminance from the background image is evaluated for each pixel, and a person or the like is detected from the result.

Patent No. 5099904 specification

The inter-frame difference method is prone to errors under different conditions from the scene in which the reference background image was taken. That is, it has a weak point that a change in luminance of the background image is erroneously detected as a person. Therefore, in the related art, false detection is prevented by selecting a background image for each scene.

However, since the scene changes depending on the state of the lighting apparatus (on / off, illuminance, etc.) and the state of external light (intensity, angle of insertion, etc.), preparing a background image for each scene consumes a lot of resources. For example, storing background images of all scenes in a space with many lights (such as an office) enlarges the size of the storage capacity and also the number of steps required for adjustment. It is even more unrealistic to store the background during lighting control of all scene combinations, in terms of the size of the storage capacity and the number of adjustment steps. Furthermore, since ambient light is affected by the weather, season, etc., an annual adjustment period is required to store background images of all scenes. As described above, it is difficult to detect a moving object without error as long as it relies on a change in luminance.

Therefore, an object of the present invention is to provide an image sensor, a person detection method, a program and a control system capable of preventing false detection due to a change in luminance.

According to an embodiment, the image sensor comprises an imaging unit, an extraction unit, an identification unit, and a detection unit. The imaging unit captures an image of a target space to obtain image data. The extraction unit extracts a motion feature amount from the image data. The identification unit identifies a motion type based on the motion feature amount. The detection unit detects a person in the target space based on the identification result.

FIG. 1 is a schematic view showing an example of a building management system provided with an image sensor according to the embodiment. FIG. 2 is a diagram illustrating the appearance in the floor of the building. FIG. 3 is a diagram showing an example of a communication network in a building. FIG. 4 is a block diagram showing an example of the image sensor according to the embodiment. FIG. 5 is a diagram showing an example of the flow of data in the image sensor according to the first embodiment. FIG. 6 is a diagram for explaining processing in the motion extraction unit 33a. FIG. 7 is a diagram showing an example of the movement type. FIG. 8 is a diagram for explaining the process in the motion identification unit 33b. FIG. 9 is a flowchart showing an example of the processing procedure in the image sensor shown in FIG. FIG. 10 is a diagram showing an example of the flow of data in the image sensor according to the second embodiment. FIG. 11 is a diagram showing an example of the flow of data in the image sensor according to the third embodiment. FIG. 12 is a diagram showing an example of the flow of data in the image sensor according to the fourth embodiment.

The image sensor can acquire various information as compared with a human sensor, a light sensor, an infrared sensor, and the like. If a fisheye lens or an ultra-wide-angle lens is used, an area that can be photographed by one image sensor can be enlarged, and distortion of the image can be corrected by calculation processing. It is also possible to give the image sensor a learning function.

FIG. 1 is a schematic view showing an example of a building management system provided with an image sensor according to the embodiment. In FIG. 1, the lighting device 1, the air conditioner 2, and the image sensor 3 are provided for each floor of the building 100 and are communicably connected to the control device 40. The control device 40 on each floor is communicably connected to a building monitoring device 50 provided, for example, in a building management center via the in-building network 500. As a communication protocol of the in-building network 500, Building Automation and Control Networking protocol (BACnet (registered trademark)) is representative.

The building monitoring device 50 can be connected to the cloud computing system (cloud) 200 via, for example, a Transmission Control Protocol / Internet Protocol (TCP / IP) -based communication network 600. The cloud 200 includes a server 300 and a database 400, and provides services related to building management.

As shown in FIG. 2, the lighting device 1, the outlet of the air conditioner 2, and the image sensor 3 are disposed, for example, on the ceiling of each floor. The image sensor 3 captures an image captured within the field of view to acquire image data. This image data is processed in the image sensor 3 to generate environmental information and / or personal information. The lighting device 1 and the air conditioner 2 can be controlled using these pieces of information.

The image sensor 3 processes image data to obtain environmental information and personal information. The environment information is information on the environment of the space (zone) of the imaging target. For example, the environmental information is information indicating the illuminance and temperature of the office. Person information is information on humans in the target space. For example, the personal information is information indicating the presence or absence of a person (referred to as presence / absence), the number of people, the behavior of the person, the amount of activity of the person, and the like.

Each of the small areas obtained by dividing the zone into a plurality of areas is called an area. For example, environmental information and personal information can be calculated for each area. In the embodiment, walking / staying as one of personal information will be described. The walking / staying is information indicating whether a person is walking or staying at one place.

FIG. 3 is a diagram showing an example of a communication network in the building 100. As shown in FIG. In FIG. 3, the lighting device 1, the air conditioner 2, and the image sensor 3 are connected in a daisy chain shape via a signal line L. Among them, for example, one image sensor 3 is connected to the in-building network 500 via the gateway (GW) 7-1. As a result, all the lighting devices 1, the air conditioners 2, and the image sensor 3 are connected to the building monitoring device 5 via the in-building network 500.

Each image sensor 3 is connected to the in-building network 500 via a LAN (Local Area Network) 10, a hub (Hub) 6 and a gateway (GW) 7-2. Thereby, the image data, the environment information and the person information acquired by the image sensor 3 are transmitted to the control device 4, the display device 11 and the building monitoring device 5 via the in-building network 500 independently of the signal line L.

Furthermore, each image sensor 3 can communicate with each other via the LAN 10.
The control device 4 generates control information for controlling the lighting device 1 and the air conditioner 2 based on the environment information and the person information sent from the image sensor 3. The control information is sent to the lighting device 1 and the air conditioner 2 via the gateway 7-1 and the signal line L.
The display device 11 visually displays environment information and person information acquired from the image sensor 3 or various information acquired from the building monitoring device 5.

Furthermore, the wireless access point 8 is connected to, for example, the gateway 7-2. As a result, the notebook computer 9 or the like having a wireless communication function can access the image sensor 3 via the gateway 7-2.

FIG. 4 is a block diagram showing an example of the image sensor 3 according to the embodiment. The image sensor 3 includes a camera unit 31 as an imaging unit, a memory 32, a processor 33, and a communication unit 34. These are connected to one another via an internal bus 35.

The camera unit 31 includes a fisheye lens 31a, an aperture mechanism 31b, an image sensor 31c, and a register 30. The fisheye lens 31a captures a space (target space) in the office floor in a view looking down from the ceiling and forms an image on the image sensor 31c. The light quantity from the fisheye lens 31a is adjusted by the diaphragm mechanism 31b. The image sensor 31c is, for example, a CMOS (complementary metal oxide semiconductor) sensor, and generates, for example, a video signal at a frame rate of 30 frames per second. This video signal is digitally encoded and output as image data.

The register 30 stores camera information 30a. The camera information 30a is, for example, information on the camera unit 31 such as the state of the auto gain control function, the value of gain, exposure time, or information on the image sensor 3 itself.

The memory 32 is a semiconductor memory such as SDRAM (Synchronous Dynamic RAM) or a non-volatile memory such as EPROM (Erasable Programmable ROM), and a program 32 b for causing the processor 33 to execute various functions according to the embodiment The image data 32a acquired by the unit 31 is stored. The memory 32 further stores mask setting data 32c and a motion dictionary 32d.

The mask setting data 32 c is data used to distinguish an area to be image-processed and an area not to be image-processed in the field of view captured by the camera unit 31. The mask setting data 32 c can be set to each image sensor 3 via the communication unit 34 from the notebook computer 9 (FIG. 3), for example.
The motion dictionary 32d is data in a table format in which motion feature quantities and motion types are associated with each other, and can be generated by, for example, a method such as machine learning.

The processor 33 loads and executes the program stored in the memory 32 to implement various functions described in the embodiment. The processor 33 is, for example, a large scale integration (LSI) that includes a multi-core CPU (central processing unit) and is tuned to execute image processing at high speed. The processor 15 can also be configured by an FPGA (Field Programmable Gate Array) or the like. An MPU (Micro Processing Unit) is also one of the processors.

The communication unit 34 is connectable to the signal line L and the LAN 10, and mediates exchange of data with a communication partner including the building monitoring device 5, the notebook computer 9, and the other image sensor 3.

The processor 33 includes, as processing functions according to the embodiment, a motion extraction unit 33a, a motion identification unit 33b, a person detection unit 33c, a sensitivity setting unit 33d, and a camera information acquisition unit 33e. In the motion extraction unit 33a, the motion identification unit 33b, the person detection unit 33c, the sensitivity setting unit 33d, and the camera information acquisition unit 33e, the program 32b stored in the memory 32 is loaded into the register of the processor 33, and Accordingly, it can be understood as a process generated by the processor 33 performing arithmetic processing. That is, the program 32 b includes a motion extraction program, a motion identification program, a person detection program, a sensitivity setting program, and a camera information acquisition program.

The motion extraction unit 33a performs image processing on the image data 32a stored in the memory 32 according to a predetermined algorithm to extract a motion feature amount. For example, it is possible to calculate a motion feature amount by tracing a change in luminance of a frame included in image data for each pixel and analyzing its time series. For example, feature quantities such as histograms of oriented gradients (HOG) feature quantities, contrast, resolution, S / N ratio, and color tone are known. In addition, a luminance gradient direction co-occurrence histogram (Co-occurrence HOG: Co-HOG) feature, a Haar-Like feature, and the like are also known as a feature. The motion extraction unit 33a particularly extracts a motion feature that indicates the motion of the object in the field of view.

The motion identification unit 33 b identifies the motion type of the object by, for example, rule-based identification processing or machine learning identification processing based on the extracted motion feature amount.

The human detection unit 33c detects a human in the target space based on the result of the motion identification.
The sensitivity setting unit 33 d sets the sensitivity of the motion identification unit 33 b to identify the motion type. The set value of the sensitivity is input, for example, from the notebook computer 9 (FIG. 3) via the communication unit 34.

The camera information acquisition unit 33 e acquires the camera information 30 a from the register 30 of the camera unit 31. Next, several embodiments will be described based on the above configuration.

First Embodiment
FIG. 5 is a diagram showing an example of the flow of data in the image sensor according to the first embodiment. In FIG. 5, the image data 32a acquired by the camera unit 31 is temporarily stored in the memory 32, and then sent to the motion extraction unit 33a. The motion extraction unit 33a performs image processing on the image data 32a to extract a motion feature amount representing the type of motion from the image data 32a.

In general, motion features can be calculated for each image frame. Besides, as shown in FIG. 6A, it is also possible to divide the image frame or the target space into a plurality of small regions (blocks) and calculate the motion feature amount for each block. It is also possible to calculate the motion feature quantity for each middle area (area) including a plurality of blocks. Ultimately, it is possible to calculate motion features for each of the pixels that make up an image frame.

Usually, the unit for extracting the motion feature amount and the calculation range related to the extraction of the motion feature amount are set to be the same, but may be different as shown in FIG. For example, when a block indicated by hatching in FIG. 6B is to be processed, an area of a bold line frame surrounding the block may be set as an operation range related to extraction of a motion feature. Furthermore, for example, the mask area may be set from the notebook computer 9 and the mask area may be excluded from the calculation target. By limiting the area for extracting the motion feature amount in this manner, the amount of operation processing can be reduced, and the processing time can be shortened.

Returning to FIG. 5, the motion feature quantity extracted by the motion extraction unit 33a is passed to the motion identification unit 33b. The motion identification unit 33 b identifies a motion type in the image data 32 a based on the motion feature amount. For example, as shown in FIG. 7, movement types relating to a person can be roughly classified into (office work) and (walking), and furthermore, three types of movement types can be identified including (other than that). These indicate unique motion feature quantities. Thus, the motion identification unit 33 b can extract motion feature amounts for each of a plurality of motion types.

In FIG. 7, the motion types detected by the change in luminance are roughly classified into moving and non-moving objects. Lighting control (lighting on / off, light control), change in daylight, and movement of a screen such as a display / projector / television all show movement feature quantities as non-moving objects. These items all cause changes in the brightness of the image data, and may be detected as motion even though they themselves do not move. For example, a screen saver of a personal computer screen or a motion of a fan of a fan corresponds to this.

The motion is roughly divided into a person and a non-person. Among these, non-persons (objects that are not persons) are distinguished, for example, from those having periodicity in motion and those having no periodicity. The office work and walking of the person show their respective motion feature quantities, and the motion having feature quantities that do not fall under this is classified as (other than that). As described above, since an object detected in the target space indicates a unique motion feature amount, it can be used to improve the accuracy of human detection.

As shown in FIG. 8A, for example, it is assumed that a feature amount indicating (person walking) is detected in three blocks (hatched areas). Then, the motion extraction unit 33a passes the result of motion identification (person walking) in these blocks to the person detection unit 33c (FIG. 5). The person detection unit 33c determines that a person has been detected for the block, as shown in FIG. 8 (b). The communication unit 34 in FIG. 5 is a signal line L as a communication network or LAN 10 as a result of detection of a person by the person detection unit 33c, processing result of the motion extraction unit 33a and the motion identification unit 33b, processing data, parameters, and the like. Send to Thus, the data and information can be shared with the other image sensors 3, the building monitoring device 5, the notebook computer 9 and the like via the in-building network 500 and the like.

FIG. 9 is a flowchart showing an example of the processing procedure in the image sensor 3 shown in FIG. In FIG. 5, when the image sensor 3 acquires image data by the camera unit 31 (step S1), the image sensor 3 stores the image data in the memory 32 (step S2).

Next, the image sensor 3 performs image processing on the image data 32a in the memory 32 to extract a motion feature amount (step S3). The extracted motion feature may be stored in the memory 32. Next, the image sensor 3 identifies a motion type based on the extracted motion feature amount (step S4). Next, the image sensor 3 detects a person in the target space based on the result of motion identification (step S5). The result of the human detection obtained in this step is communicated with another image sensor via the communication unit 34 (step S6) and shared.

As described above, in the image sensor according to the first embodiment, for example, the motion extracting unit 33a that extracts the motion feature amount based on the luminance change of the image data, and the motion of the target based on the extracted motion feature amount And a motion identification unit 33b for identifying the type. Then, when a movement type corresponding to a movement of a person is detected, the presence of a person in the target space is detected.

The existing technology that relies on the inter-frame difference method is vulnerable to changes in the scene, so it has to prepare a large number of reference background images. If the background image is not accumulated enough, the expected accuracy can only be obtained in a very limited environment, eg with only one illumination, no windows and no ambient light. Furthermore, for example, at home, there is a fear that movements other than the person, such as the shaking of a television screen or a curtain, may be detected as the movement of the person.

On the other hand, in the first embodiment, by combining the results of the motion identification and determining whether the identified motion type is the motion of a person, illumination control, brightness change due to daylight, paper of a printer in an office, etc. It has been made to prevent false detection of non-human subjects such as screen savers and fans as human figures. Furthermore, by setting a monitoring camera so that the target space is viewed from directly above with a fisheye lens, it is possible to facilitate identification of office work / walking.

From these, it is possible to provide an image sensor, a person detection method, a program, and a control system that can prevent erroneous detection due to a change in luminance.

Second Embodiment
FIG. 10 is a diagram showing an example of the flow of data in the image sensor according to the second embodiment. The motion dictionary 32d of the image sensor shown in FIG. 10 includes a plurality of motion identification data (motion identification 1, motion identification 2,...). Each motion identification data is prepared in advance, for example, for each motion type shown in FIG.

The sensitivity setting unit 33 d sets, for each motion type, the sensitivity for identifying the motion type by the motion identification unit 33 b. For example, the sensitivity of identification can be varied for each movement type by individually setting the threshold value related to movement identification for each movement identification data.

The sensitivity of motion type identification may be set in units of the entire area of the captured image. Alternatively, for example, the target space may be variably set for each of a plurality of divided areas. That is, the sensitivity of the identification of the motion type can be set for each area divided in the mesh shape of FIG. Furthermore, the setting can be made in units of pixels or blocks in more detail.

Also, the sensitivity of identification may be variably set based on the time series of the identification result for each motion type. That is, the position at which the moving object is detected in the previous state and the sensitivity around it are increased, so that the position of the moving object detected last time and its surroundings are easily identified as the "moving object" in the determination in the next identification. If there is no moving object, it is easy to distinguish it as "non-moving object".

Also, the sensitivity of identification for each motion type can be variably set according to the state of the target space. For example, in the image sensor installed near the window on the south side, it is easy to detect a change in daylight in the daytime time zone. Therefore, for an image sensor in such an environment, the identification error can be reduced by, for example, increasing the sensitivity for identifying "daylight change" during the daytime period by setting from the notebook computer 9 .

Furthermore, external information such as weather acquired from the communication unit 34 may be referred to. For example, in an image sensor installed near the south side window, the sensitivity of the "daylight change" identification is increased in the daytime time zone and when the weather is fine. Thus, the sensitivity can also be set to change depending on the environment or time zone.

As described above, in the second embodiment, the sensitivity of identification can be variably set for each of a plurality of motion types. As a result, it is possible to improve the accuracy of the motion identification and further enhance the effect of preventing false detection due to a change in luminance.

Third Embodiment
FIG. 11 is a diagram showing an example of the flow of data in the image sensor according to the third embodiment. In FIG. 11, the motion identification unit 33 b acquires camera information 30 a from the register 30 of the camera unit 31. Thereby, the identification of the movement type and the function such as the automatic gain control of the camera unit 31 can be linked. That is, the motion identification unit 33b identifies the motion type based on the extracted motion feature amount and the camera information 30a.

For example, when the gain, the exposure time, or the like changes, the sensitivity of identification of “lighting control” and / or “daylight change” is increased to make it easy to identify these motion types. This also improves the accuracy of motion identification, and can further enhance the effect of preventing false detection due to a change in luminance.

Fourth Embodiment
FIG. 12 is a diagram showing an example of the flow of data in the image sensor according to the fourth embodiment. The memory 32 of the image sensor shown in FIG. 12 includes a motion dictionary 32 d-1 including a plurality of motion identification data (motion identification 1, motion identification 2,...), And a countermeasure dictionary 32 d for coping with undetection / false detection. -Remember that. The countermeasure dictionary 32d-2 also includes a plurality of motion identification data (motion identification 1, motion identification 2,...). Each motion identification data is prepared in advance, for example, for each motion type in FIG.

In FIG. 12, when the non-detection / false detection occurs, the motion identification unit 33 b determines motion identification data (a parameter of motion identification) in the memory 32 to identify the scene as a specific motion type in the subsequent identification. Change / rule / dictionary etc.)

For example, the movement of the undetected person is identified as the “person”, the erroneously detected illumination change is identified as the “illumination change”, and the paper feed / fan of the erroneously detected printer is identified as the “non-person”; The motion identification data is backpropagated to identify the screen saver animation as "non-moving". By doing this, it is possible to reduce the frequency of occurrence of non-detection / false detection, and thus to improve the accuracy of human detection.

Note that information (motion identification data) such as parameters / rules / dictionaries related to motion identification can be copied between the plurality of image sensors 3 by the communication unit 34. For example, the adjustment man-hour can be shortened by copying the movement identification of the individual that becomes the base for which adjustment has been completed for a certain property to another image sensor. Furthermore, the motion identification data can also be modified by online learning.

Other Embodiments
The sensitivity setting unit 33d can set the number and / or time of image frames to be referred to by the motion extraction unit 33a in motion extraction processing fixed or variable. For example, when the frame rate of the camera unit 31 changes, the time is fixed and the number of sheets is variable. Further, for example, when it is identified as walking in the previous state, it is identified with a small number of sheets, and when it is identified as office work in the previous state, it is identified with a large number of sheets. Furthermore, even if the reference number is fixed by default, parameter settings can be changed from the outside by the notebook computer 9 or the like.

The sensitivity setting unit 33 d may select the type of image data to be referred to by the motion extraction unit 33 a at the time of motion extraction processing from an original image / average image / edge image or the like. Only one type of image data may be referenced, or a plurality of types of image data may be referenced. Furthermore, even if the type of the reference image is fixed by default, the type of the reference image can be changed from the outside by the notebook computer 9 or the like.

As described above, according to the above embodiments, it is possible to provide an image sensor, a person detection method, a program, and a control system capable of preventing false detection due to a change in luminance.

The present invention is not limited to the above embodiment. For example, not only the result of motion identification by the motion identification section 33b but also processing results such as inter-frame difference / background difference / human shape recognition are given to the human detection unit 33c, and these information are comprehensively combined to obtain a human It may be detected.

The unit of person detection may be an image unit or an area unit. Also, as shown in FIG. 8B, detection may be performed in individual person units. Furthermore, coordinates on the image of a person may be acquired as a result of person detection.

In the third embodiment, the motion identification unit 33 b and the camera unit 31 are linked. Instead of this, the motion extraction unit 33a and the camera unit 31 may be linked. By doing this, it is possible to extract the motion feature amount in consideration of gain / exposure time and the like, and it is possible to improve the accuracy of human detection.

Further, in the fourth embodiment, the items of the motion type may be increased to correspond to the specific scene and the specific motion type. Alternatively, the motion dictionary for identification itself may be increased, such as a motion dictionary of a person who has become undetected, a dictionary of erroneously detected illumination changes, or the like.

While the embodiments of the present invention have been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be implemented in other various forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. This embodiment and its modifications are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

Claims

An imaging unit for imaging the target space to acquire image data;
An extraction unit that extracts a motion feature from the image data;
An identification unit that identifies a motion type based on the motion feature amount;
An image sensor comprising: a detection unit that detects a person in the target space based on a result of the identification.
The image sensor according to claim 1, wherein the identification unit can extract a motion feature amount for each of a plurality of motion types.
The image sensor according to claim 2, further comprising a setting unit configured to set a sensitivity of identification for each of the plurality of motion types.
The image sensor according to claim 3, wherein the setting unit variably sets the sensitivity for each of a plurality of areas obtained by dividing the target space.
The image sensor according to claim 3, wherein the setting unit variably sets the sensitivity based on a result of identification for each of the motion types.
The image sensor according to claim 3, wherein the setting unit variably sets the sensitivity in accordance with a state of the target space.
And an acquisition unit configured to acquire information on the imaging unit.
The image sensor according to claim 1, wherein the identification unit identifies the movement type based on the movement feature amount and information on the imaging unit.
The image sensor according to claim 1, further comprising a communication unit that transmits the result of the detection of the person to a communication network.
The image sensor according to claim 1, further comprising a setting unit for variably setting the number of frames of the image data to be referred to by the extraction unit.
The image sensor according to claim 1, further comprising a setting unit for selecting a type of image data to be referred to by the extraction unit.
A person detection method executed by a computer that captures an image of a target space and acquires image data,
Extracting the motion feature quantity from the image data by the computer;
Identifying the motion type based on the motion feature amount by the computer;
And D. the computer detecting the person in the target space based on the result of the identification.
In a computer of an image sensor that captures an object space and acquires image data,
Extracting a motion feature from the image data;
Identifying a motion type based on the motion feature amount;
A program for executing a process of detecting a person in the target space based on a result of the identification.
The image sensor according to any one of claims 1 to 10, which images a target space;
A control system, comprising: a control device configured to control an apparatus provided in the target space based on a result of detection of a person in the target space by the image sensor.