JP7254570B2 - Image sensor system, computing device, sensing method, and program - Google Patents

Description

Embodiments of the present invention relate to image sensor systems, computing devices, sensing methods, and programs.

A recent image sensor is equipped with a CPU (Central Processing Unit) and memory, and can be said to be a so-called built-in computer with a lens. It also has advanced image processing functions, and can analyze captured image data and calculate the presence or absence of people, or the number of people, for example. Image sensors of this type are being used in combination with building management systems to improve comfort and livability and promote energy conservation.

Republished WO2013/187047 Patent No. 5903634 JP 2017-215694 A Republished WO2010/013572 JP 2011-193187 A

Existing sensors can only detect the brightness of a specific location or the presence or absence of people, so it was necessary to install sensors for each application, such as lighting control and air conditioning. As a result, the wiring becomes complicated, and construction work is required each time, resulting in an increase in engineering work. There is a demand for a system that utilizes the capabilities of an image sensor and has increased availability.

Accordingly, it is an object of the present invention to provide an image sensor system, computing device, sensing method, and program with improved availability.

According to an embodiment, an image sensor system includes a plurality of image sensors for sensing an object and a computing device in communication with the image sensors. Each image sensor includes an imaging section and an image processing section. The imaging unit acquires image data by imaging the assigned area. The image processing unit performs image processing on the image data to detect motion information in the area. The computing device comprises an acquisition unit, a storage unit, and an integration processing unit. The acquisition unit acquires motion information detected by each image sensor. The storage unit stores map data indicating a correspondence relationship between an area assigned to each image sensor and a map of the target space. The integration processing unit integrates the acquired motion information based on the map data to calculate motion information in the target space.

FIG. 1 is a schematic diagram illustrating an example of an image sensor system including an image sensor according to an embodiment. FIG. 2 is a diagram showing an example inside a building floor. FIG. 3 is a block diagram showing an example of the image sensor system shown in FIG. FIG. 4 is a block diagram illustrating an example of an image sensor according to the embodiment; FIG. 5 is a flow chart showing a basic processing procedure of the image sensor 3. As shown in FIG. FIG. 6 is a diagram showing an example of data flow in the image sensor 3. As shown in FIG. FIG. 7 is a block diagram showing an example of the management terminal 200 according to the embodiment. FIG. 8 is a diagram showing an example of the relationship between the sensor coordinate system and the reference coordinate system. FIG. 9 is a flowchart showing a basic processing procedure of the management terminal 200. As shown in FIG. FIG. 10 is a diagram for explaining the motion vector mapping process in the management terminal 200. As shown in FIG. FIG. 11 is a diagram for explaining motion vector synthesis when areas overlap. FIG. 12 is a diagram for explaining connection of motion vectors detected in time series. FIG. 13 is a diagram for explaining motion vector interpolation. FIG. 14 is a diagram for explaining motion vector correction. FIG. 15 is a diagram for explaining that the position of a person can be predicted based on motion vectors. FIG. 16 is a diagram showing an example of a heat map based on the motion vector analysis result. FIG. 17 is a diagram showing an example of a color map based on the motion vector analysis result. FIG. 18 is a diagram showing an example of data flow in the management terminal 200. As shown in FIG.

An image sensor can acquire a wider variety of information than a human sensor, a light sensor, an infrared sensor, or the like. By using a fish-eye lens, an ultra-wide-angle lens, or the like, the area that can be captured by a single image sensor can be expanded, and image distortion can be corrected by calculation processing. Image sensors are also known that have a function of masking an area in the field of view that is not desired to be sensed, and have a learning function.

[composition]
FIG. 1 is a schematic diagram illustrating an example of an image sensor system including an image sensor according to an embodiment. In FIG. 1, a lighting device 1, an air conditioner 2, and an image sensor 3 are provided, for example, on each floor of a building 10, and are connected to a controller 4 so as to be communicable. The controller 4 on each floor is communicably connected to, for example, a building management center BEMS (Building Energy Management System) server 5 via an intra-building network 500 . A typical communication protocol for the intra-building network 500 is the Building Automation and Control Networking protocol (BACnet (registered trademark)). In addition, protocols such as DALI, ZigBee (registered trademark), and ECHONET Lite (registered trademark) are also known.

The BEMS server 5 can be connected to a cloud computing system (cloud) 600 via a TCP/IP (Transmission Control Protocol/Internet Protocol) based communication network 300, for example. The cloud 600 includes a database 70 and a server 80, and provides services related to building management and the like.

As shown in FIG. 2, lighting equipment 1, outlets of air conditioning equipment 2, and image sensor 3 are arranged on, for example, the ceiling of each floor. Each image sensor 3 is assigned an area to be sensed. In FIG. 2 it is shown that areas A1 and A2 each cover approximately half of the floor. All areas together can cover the floor of the target space. As indicated by hatching in the drawing, the areas assigned to different image sensors 3 may partially overlap (overlapping portion).

The image sensor 3 acquires image data by photographing an image captured within its field of view. This image data is processed by the image sensor 3, and personal information, environment information, or the like is calculated. That is, the image sensor 3 detects (senses) an object in the object space and calculates a feature amount related to this object.

The environment information is information about the environment of the target space (zone), such as the illuminance and temperature of the floor. Person information is information about a person in the target space. For example, the number of people in an area, the behavior of people, the amount of activity of people, presence or absence indicating the presence or absence of people, or walking stay indicating whether people are walking or staying in one place. For example.

In recent years, it has been studied to ensure human comfort and safety by controlling the lighting equipment 1 and the air conditioning equipment 2 based on such information in a human living environment. It is also possible to calculate environmental information and person information for each small area obtained by dividing a zone into a plurality of areas. This small area may be associated with the allocated area for each image sensor 3 .

In the embodiment, attention is paid to motion information indicating the motion of the object. A typical example of motion information is a motion vector that represents the magnitude and direction of motion of an object. Alternatively, a scalar value, such as a quantity indicative of the magnitude of an object's movement or an amount indicative of the direction of an object's movement, can also be understood as motion information.

FIG. 3 is a block diagram showing an example of the image sensor system shown in FIG. In FIG. 3, under the control of the BEMS server 5, there are an air conditioning controller 41 that controls the air conditioning equipment 2, an elevator controller 42 that controls the elevator, a security controller 43 that controls the security system, a disaster prevention controller 44 that controls the disaster prevention system, and lighting. A lighting controller 45 that controls the equipment 1 (1-1, 1-2) is connected via an intra-building network 500. FIG.

Also, a management terminal 200 and gateways 71 and 72 are connected to the intra-building network 500 so as to be able to communicate with the BEMS server 5 . The gateway 71 accommodates a plurality of image sensors 3 (3-1 to 3-n) as subordinates. The image sensors 3-1 to 3-n are connected by a sensor network 101 in a unicursal manner (crossover wiring). Similarly, the image sensors 3 - 1 to 3 - m are accommodated under the gateway 72 via the sensor network 102 .

EtherCAT (registered trademark), for example, is known as a protocol for the sensor networks 101 and 102 . The sensor network 101 connects the image sensors 3-1 to 3-n and the gateway 71 so as to be able to communicate with each other. The sensor network 102 connects the image sensors 3-1 to 3-m and the gateway 72 so as to be able to communicate with each other. Each gateway 71, 72 has a protocol conversion function between the sensor networks 101, 102 and the intra-building network 500, respectively. Thereby, the BEMS server 5 and the management terminal 200 can also communicate with the image sensors 3-1 to 3-n and 3-1 to 3-m via the gateways 71 and 72, respectively.

FIG. 4 is a block diagram showing an example of the image sensor 3 according to the embodiment. The image sensor 3 includes a camera section 31 as an imaging section, a memory 32 , a processor 33 and a communication section 34 . These are connected to each other via an internal bus 35 . By having the memory 32 and the processor 33, the image sensor 3 functions as a computer.

The camera unit 31 includes a fisheye lens 31a, an aperture mechanism 31b, an image sensor 31c, and a register 30. The fish-eye lens 31a catches the area in the field of view looking down from the ceiling and forms an image on the image sensor 31c. The amount of light from the fisheye lens 31a is adjusted by a diaphragm mechanism 31b. The image sensor 31c is, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor, and generates a video signal with a frame rate of, for example, 30 frames per second. This video signal is digitally encoded and output as image data.

The register 30 stores camera information 30a. The camera information 30a is information about the camera unit 31, such as the state of the auto gain control function, gain value, and exposure time, or information about the image sensor 3 itself.
For example, the processor 33 controls parameters that affect the sensing sensitivity and accuracy, such as the frame rate, the gain value of the image sensor 31c, and the exposure time.

The memory 32 is a semiconductor memory such as SDRAM (Synchronous Dynamic RAM) or a non-volatile memory such as NAND flash memory or EPROM (Erasable Programmable ROM). 32a and image data 32b acquired by the camera unit 31 are stored. Further, the memory 32 stores dictionary data 32c and motion vector data 32d.

The dictionary data 32c is tabular data in which sensing items and feature amounts are associated with each other, and can be generated by a technique such as machine-learning. For example, by pattern matching processing using the dictionary data 32c, it is possible to identify the detection target (person, etc.) in the area.

The motion vector data 32d is data indicating a motion vector in an area, generated by the image processing section 33a of the processor 33. FIG. The motion vector data 32d are accumulated in the memory 32, for example, for a predetermined period (one day, one week, one month, etc.).

In addition, the memory 32 may store mask setting data and the like. The mask setting data is used to distinguish an area for image processing (target area for image processing) and an area for non-image processing (non-target area for image processing) in the field of view captured by the camera unit 31 .

The processor 33 loads and executes programs stored in the memory 32 to implement various functions described in the embodiments. The processor 33 is, for example, a multi-core CPU (Central Processing Unit) and is an LSI (Large Scale Integration) tuned to execute image processing at high speed. The processor 15 can also be configured with an FPGA (Field Programmable Gate Array) or the like. An MPU (Micro Processing Unit) is also one of the processors.

The communication unit 34 is connectable to the sensor network 101 (102), and mediates the exchange of data with a communication partner (the BEMS server 5, the management terminal 200, or another image sensor 3, etc.). The communication interface may be wired or wireless. Any topology such as a star type or a ring type can be applied to the topology of the communication network. The communication protocol may be a general purpose protocol or an industrial protocol. A single communication method may be used, or a plurality of communication methods may be combined.

In particular, the communication unit 34 receives sensing data from the image sensor 3, processing results of the processor 33, processing data, parameters, image data, motion vector data, accumulated past motion vector data, processed motion vector data, statistics, Analysis data, output data, parameters, dictionary data, firmware, etc. are transmitted and received via the sensor network 101 (102) as a communication network. As a result, the above data and information can be shared with other image sensors 3, BEMS servers 5, management terminals 200 and the like via the intra-building network 500 and the like.

By the way, the processor 33 includes an image processing section 33a and a control section 33b as processing functions according to the embodiment. The image processing unit 33a and the control unit 33b load the program 32a stored in the memory 32 into the registers of the processor 33, and the processor 33 executes arithmetic processing as the program progresses. , can be understood. That is, the program 32a includes an image processing program and a control program.

The image processing unit 33a performs image processing on the image data 32b to detect a motion vector (optical flow) in the target area. The motion vector data thus obtained is stored in the memory 32 (motion vector data 32d).

When analyzing the motion vector, it is also possible to use feature amounts unique to the image data. The features include histograms of oriented gradients (HOG), contrast, resolution, S/N ratio, and color tone. A co-occurrence histogram (Co-occurrence HOG: Co-HOG) feature amount, a Haar-Like feature amount, and the like are also known as feature amounts.

In addition, the image processing unit 33a senses any one or a plurality of sensing items among sensing items including presence/absence, number of people, amount of activity, illuminance, staying on foot, and the motion vector. These multiple sensing items can also be set for each area. The image processing unit 33a can sense each sensing item for each block obtained by further dividing the area into a plurality of blocks, integrate the sensing results for each block, and obtain the sensing result for each area.

The control unit 33b varies the sensing sensitivity according to the notification from the management terminal 200. FIG. For example, when notified that a person is moving from an area of another image sensor to its own area, the control unit 33b gives a control signal to the camera unit 31 to increase the sensing sensitivity from the default state. . There are methods to increase the sensing sensitivity, such as increasing the gain of the image sensor, lowering the threshold value related to object detection, and increasing the frame rate.

FIG. 5 is a flow chart showing a basic processing procedure of the image processing section 33a shown in FIG. In FIG. 5, image data 32b acquired by the camera section 31 (step S1) is temporarily stored in the memory 32 (step S2), and then sent to the image processing section 33a. The image processing unit 33a performs image processing on the image data 32b to perform presence/absence determination (step S3), number estimation (step S4), activity amount estimation (step S5), illuminance estimation (step S6), walking retention determination (step S7) and motion vector calculation (step S8) are executed, and these items are sensed from the image data 32b.

All of these sensing items may always be sensed at the same time. Or you may sense individually as needed. The cycle of processing for each item may be the same for all items in synchronization with the frame cycle, for example, or may be different for each item.

As for the presence/absence of a person, it is possible to determine the presence/absence of a person by extracting a person area using, for example, background difference/inter-frame difference/person recognition. If the zone is divided into areas (area division), it is possible to determine whether or not there is a person area for each area, and to estimate the presence/absence of each area. Furthermore, it is possible to provide an erroneous detection suppression function by illumination variation determination or the like.

As for the number of people, the number of people can be estimated by extracting a person area using, for example, background difference/inter-frame difference/person recognition, and detecting individuals from the person area. If there is area division, the location of individuals can be estimated and the number of people can be estimated for each area.

As for the amount of activity, it is possible to estimate the amount of activity by extracting a motion area using inter-frame differences or the like. The amount of activity may be classified into a plurality of stages such as no/small/medium/large, for example. Alternatively, an index such as METs (Metabolic equivalents) may be used to represent the amount of activity. Furthermore, if there is an area division setting, it is possible to extract a motion region for each area and estimate the amount of activity for each area.

The illuminance can be estimated based on the image data 32b and camera information such as gain and exposure time. Camera information ( 30 a ) can be obtained from the register 30 of the camera section 31 . If there is an area division setting, the image can be divided into areas, and the illuminance of each area can be estimated from the divided images and camera information.

With regard to walking stagnation, a moving region can be extracted using an inter-frame difference or the like, and walking stagnation can be determined. If there is a setting for area division, it is possible to extract a motion region for each area and estimate walking stagnation for each area.

A motion vector can be calculated by subjecting the image data 32b to image processing and using an algorithm such as a block matching method or a gradient method. The calculated motion vector is stored in the memory 32 (motion vector data 32d).

As described above, sensing data of each sensing item can be obtained. The communication unit 34 communicates these sensing data and various parameters related to item calculation to the management terminal 200 via the sensor network 101 (102) (step S10).

FIG. 6 is a diagram showing an example of data flow in the image sensor. In FIG. 6, the image data 32b acquired by the camera section 31 is temporarily stored in the memory 32 and then sent to the image processing section 33a of the processor 33. FIG. The image processing unit 33a performs image processing on the image data 32b to generate motion vector data. The obtained motion vector data 32 d are stored in the memory 32 .

The communication unit 34 transmits various data including the generated motion vector data 32 d to the management terminal 200 and the BEMS server 5 via the sensor network 101 ( 102 ), the gateway 71 (or 72 ), and the intra-building network 500 .

FIG. 7 is a functional block diagram showing an example of the management terminal 200 shown in FIG. The management terminal 200 as an example of a computing device is a computer including a processor 250 such as a CPU or MPU (Micro Processing Unit), a ROM (Read Only Memory) 220 and a RAM (Random Access Memory) 230 . The management terminal 200 further includes a storage unit 240 such as a hard disk drive (HDD), an optical media drive 260, a communication unit 270, and a monitor 210 that displays various information.

The ROM 220 stores basic programs such as BIOS (Basic Input Output System) and UEFI (Unified Extensible Firmware Interface), various setting data, and the like. RAM 230 temporarily stores programs and data loaded from storage unit 240 .

Optical media drive 260 reads digital data recorded on a recording medium such as CD-ROM 280 . Various programs executed by the management terminal 200 are recorded on, for example, a CD-ROM 280 and distributed. The program stored in this CD-ROM 280 is read by the optical media drive 260 and installed in the storage section 240 .

The communication unit 270 has a function for communicating with the image sensor 3, the BEMS server 5, etc. via the intra-building network 500. FIG. Various programs to be executed by the management terminal 200 can also be downloaded from a server via the communication unit 270 and installed in the storage unit 240, for example. It is also possible to download the latest program from the cloud 600 (FIG. 1) via the communication unit 270 and update the installed program. Also, the management terminal 200 can download and transmit dictionary data to the image sensor 3 via the intra-building network 500 and the gateways 71 and 72 .

Storage unit 240 stores motion vector data 240b, statistical analysis data 240c, design data 240d, and map data 240e in addition to program 240a executed by processor 250. FIG.

Motion vector data 240b are motion vectors collected from image sensors 3-1 to 3-n, 3-1 to 3-m. The image sensors 3-1 to 3-n and 3-1 to 3-m transmit motion vectors detected in a coordinate system (called a sensor coordinate system) for each image sensor. The management terminal 200 that collects this transforms the motion vector of each sensor into a motion vector in the reference coordinate system. As the reference coordinate system, for example, an XY coordinate system that determines the position on the floor map can be used. Both the motion vector data immediately after collection and the converted motion vector data may be included in the motion vector data 240b.

The motion vector data that is constantly generated may be stored in the storage unit 240, or the time and period may be designated and the motion vector data for only that period may be stored. Also, motion vector data from all image data may be accumulated, or motion vector data from any selected image data may be accumulated.

The statistical analysis data 240c is generated by statistically analyzing the motion vector data 240b accumulated in the storage unit 240. FIG.

The design data 240d is data including the installation position in the target space of each of the image sensors 3-1 to 3-n and 3-1 to 3-m, the orientation direction of the camera section 31 (or the fisheye lens 31a), and the like. . That is, the design data 240d is data used to uniquely determine the correspondence between each position coordinate on the floor map and the pixels forming the image data of each image sensor 3 .
The map data 240e is data indicating the correspondence relationship between the assigned area for each image sensor 3 and the map of the target space.

The processor 250 executes an OS (Operating System) and various programs. In addition, the processor 250 includes, as processing functions according to the embodiment, an acquisition unit 250a, a map creation unit 250b, an integration processing unit 250c, an analysis unit 250d, a motion vector processing unit 250e, a prediction unit 250f, a notification unit 250g, and a data processing unit. A portion 250h is provided.

These functional blocks can be understood as processes that the program 240a is loaded into the RAM 230 and generated during execution of the program. In other words, the program 240a operates the management terminal 200, which is a computer, as an acquisition unit 250a, a map creation unit 250b, an integration processing unit 250c, an analysis unit 250d, a motion vector processing unit 250e, a prediction unit 250f, a notification unit 250g, and a data processing unit 250h. operate as

The acquiring unit 250a acquires motion vector data detected by each of the image sensors 3-1 to 3-n and 3-1 to 3-m. The acquired motion vector data is stored in the storage unit 240 (motion vector data 240b).
The map creator 250b creates map data 240e based on the design data 240d. That is, the map creation unit 250b creates the map data 240e based on data such as the installation position and orientation of each image sensor 3, wiring, communication order, and the like.

FIG. 8 is a diagram showing an example of the relationship between the sensor coordinate system and the reference coordinate system. Taking the image sensors 3A, 3B, 3C, and 3D as an example, it is assumed that each allocated area is composed of 4×4=16 blocks. In FIG. 8, it is assumed that there is a relationship of floor>area>block in order of size.

In FIG. 8, the image sensors 3A, 3B, 3C, and 3D detect motion vectors in their assigned areas in their respective sensor coordinate systems, and notify the management terminal 200 of them. The management terminal 200 integrates the notified motion vectors and calculates a motion vector on the floor map for 4×4×4=6 blocks, which is the sum of all areas. This calculated motion vector is expressed in the reference coordinate system.

Since the positions (X, Y) of the image sensors 3A, 3B, 3C, and 3D in the reference coordinate system and the mounting orientation θ are recorded in the design data 240d, coordinate conversion can be performed based on this information. Each pixel of the image data captured by the image sensor 3 can be represented by the distance and angle from the center of the image in each image sensor 3 . If the installation position and installation orientation of the image sensor 3 on the map in the reference coordinate system are known, the position of each pixel of the image sensor 3 can be associated with the position on the map. The corresponding position of each pixel may be stored in the map data 240e, or may be calculated each time.

The integration processing unit 250c integrates motion vector data for each image sensor 3 based on the map data 240e to calculate a motion vector in the target space. That is, the integration processing unit 250c creates map correspondence data in which the motion vector data of the plurality of image sensors 3 are associated with the floor map based on the map data 240e. Note that motion vector integration may include processing such as interpolation or correction based on statistical processing as well as simply connecting a plurality of motion vectors.

In addition, the integration processing unit 250c performs processing for aligning the motion vectors detected by the respective image sensors in overlapping portions of the allocated areas of the image sensors. For example, two motion vectors may be combined equally at 1:1, or weighted to one of them. The integration processing unit 250c synthesizes motion vectors from individual image sensors so as to be optimal as a whole, and connects the entire map as a target. As a result, motion vectors can be combined while matching the entire floor map.

The analysis unit 250d statistically analyzes the motion vector data 240b accumulated in the storage unit 240 and generates statistical analysis data 240c. Statistical analysis can determine various events across the floor from local motion vectors. Either real-time data or stored data may be used for statistical analysis. Analysis methods include analyzing the distribution of motion vectors, performing heat map analysis by superimposing motion vectors, and estimating a trajectory by superimposing motion vectors. For example, events such as congestion, crowd behavior, layout estimation, queue estimation, office usage rate, and factory operation status can be obtained quantitatively. In addition, it is possible to estimate and analyze the moving speed of the target, its change, the moving distance, and the like. Furthermore, you may estimate the operating conditions, such as the facilities of a factory.

The motion vector processing unit 250e, for example, refers to the statistical analysis data 240c to supplement or correct motion vectors on the floor map. For example, the motion vector processing unit 250e corrects the motion vector in order to cope with image distortion caused by using a fisheye lens.

The prediction unit 250f predicts the movement of an object (such as HUMAN in FIG. 8) on the floor based on the statistically analyzed data 240c.
As a result of the prediction by the prediction unit 250f, when it is predicted that a person will move across areas, the notification unit 250g notifies the image sensor responsible for the destination area. For example, in FIG. 8, when a person in the area of the image sensor 3D is predicted to move to the area of the image sensor 3B, the notification unit 250g notifies that "a person is moving from the area of the image sensor 3D." , to the image sensor 3B. That is, the image sensor 3B in charge of the destination area (area of the image sensor 3B) is notified that a person is moving from the source area (area of the image sensor 3D). The image sensor 3B that has received the notification performs control such as varying the sensing sensitivity as necessary.

In addition, the notification unit 250g transmits the calculated motion vector data 240b to the BEMS server 5 via the intra-building network 500, and notifies the system administrator of various information, for example. Also, when an event such as congestion in the target space, disturbance in the flow of people, or abnormal movement of equipment is detected, the notification unit 250g notifies the administrator to that effect via the communication network. .

The data processing unit 250h processes either or both of the motion vector data 240b and the statistical analysis data 240c to generate output data. For example, the data processing unit 250h generates data (display image, etc.) for displaying the movement of people on the floor, the degree of congestion, or the line of flow in a color map. The generated data is transmitted to the BEMS server 5 or the like via the intra-building network 500 .

[Action]
FIG. 9 is a flowchart showing a basic processing procedure of the management terminal 200. As shown in FIG. 9, the management terminal 200 acquires motion vector data individually detected by each image sensor 3 from each image sensor 3 (step S20), and accumulates them in the storage unit 240 (step S21). Next, the management terminal 200 maps the accumulated motion vectors to a common coordinate system by the integration processing unit 250c (step S22).

10(a) and 10(b) are motion vectors detected by different image sensors. The management terminal 200 integrates these motion vectors based on the map data 240e to calculate motion vector data (FIG. 10(c)) corresponding to the floor map.

As shown in FIGS. 11(a) and 11(b), when there is overlap in the allocated areas of different image sensors, the integration processing unit 250c aligns the motion vectors detected by the respective image sensors. Synthesize (FIG. 11(c)).

FIGS. 12(a), (b), and (c) show that time-series-changing motion vectors are detected in this order. When such a motion vector is observed, the management terminal 200 statistically analyzes the motion vectors on the time axis of multiple image sensors, and as shown in FIG. Connect the motion vectors as

Furthermore, in the flowchart of FIG. 9, the management terminal 200 interpolates and/or corrects local motion vectors from motion vectors of the entire map based on the statistical analysis data 240c (step S23).
Even if the motion vector between FIGS. 13(a) and (c) is lost as shown in FIG. 13(c), for example, by using the statistical analysis data 240c, as shown in FIG. Motion vectors can be interpolated.

Focusing on FIGS. 14(a) and 14(c), it can be seen that the motion vectors in FIG. 14(b) contain errors because they do not satisfy the time-series relationship. Therefore, the management terminal 200 corrects the motion vector using, for example, the statistical analysis data 240c, and calculates the corrected motion vector shown in FIG. 14(d).

When the motion vector for the entire map through interpolation and correction is obtained, the management terminal 200 statistically analyzes the motion vector for the entire map and creates statistical analysis data. Based on this statistical analysis data, the management terminal 200 determines various events occurring on the floor (step S24).

As shown in FIG. 15, the management terminal 200 predicts the destination position (or area) of the person based on the motion vector on the entire map. The determined event and predicted position are notified to the BEMS server 5, the administrator, etc. via the intra-building network 500 and the communication network. Furthermore, the management terminal 200 creates a visualization image of motion vectors on the entire map as shown in FIG. 16 (step S25).

FIG. 16 is a diagram showing an example of an image (heat map) showing the distribution, direction, density, etc. of motion vectors in different colors. For example, areas where people pass frequently (dense motion vectors) are shown in red, areas where people do not pass very often (sparse motion vectors) are yellow, and intermediate areas are shown in orange (color map). You can grasp the degree of congestion and traffic lines at a glance. Of course, different colors may be used according to the length (magnitude) and direction of the motion vector. This kind of heat map can be applied, for example, to office floors, airports or train station concourses.

By utilizing the results of motion vector analysis, it is possible to grasp the current state of offices, stores, public facilities, factories, etc., and make improvements. For example, the following two examples are typical.
(1) Examples of office improvements
Analyze the utilization rate of a conference room, etc., and change areas with low utilization rates to other uses. Estimate the occupancy rate of free-address offices, areas of people with multiple jobs, etc., and change unnecessary areas to other uses. Change the flow line to a smart layout.
(2) Examples of store improvements
Display ads in high-traffic areas. Arrange related products according to the flow line.

Another conceivable application is to grasp the congestion status of a cafeteria in almost real time and display it on the user's smartphone.
As shown in FIG. 17, if the congestion status is color-coded for each area of a predetermined space (dining room, etc.) and released as a heat map that can be accessed via the Web, it is possible to improve convenience for users.

FIG. 18 is a diagram showing an example of data flow in the management terminal 200. As shown in FIG. In FIG. 18, the integration processing unit 250c reads and integrates map data 240e accumulated in the storage unit 240 to generate motion vector data 240b on the map. The analysis unit 250d analyzes the motion vector data 240b, and the motion vector processing unit 250e performs motion vector interpolation and correction processing. The data processing unit 250h also creates data such as heat maps, color maps, movement of people based on motion vectors, congestion conditions, flow lines, etc., and transmits the data to the BEMS server 5 or the like via the communication unit 270 .

[effect]
As described above, in this embodiment, motion vectors sensed by a plurality of image sensors 3 are collected/accumulated in the management terminal 200, and integrated motion vectors in the reference coordinate system are calculated. In addition, the motion vectors on the floor map thus obtained are statistically analyzed to generate statistical analysis data in the target space. This makes it possible to estimate the flow line, usage rate, degree of congestion, etc. in the entire space.

Further, in the embodiment, information obtained by motion vector analysis is notified to an administrator or the like. For example, it is possible to notify the administrator of information such as the occurrence of congestion, a disordered flow of people, and abnormal movement of facilities.
Also, control instructions may be generated from the analysis results to control various devices. For example, it can be applied to lighting control/air-conditioning control based on presence/absence/walking/estimated illuminance, air-conditioning control based on estimated number of people/activity, elevator operation control based on designated number of people, or security door opening/closing control.
For these reasons, according to the embodiments, it is possible to provide an image sensor system, a computing device, a sensing method, and a program with improved availability.

In addition, this invention is not limited to the said embodiment. For example, various functions implemented in the management terminal 200 as a computing device may be implemented in the gateways 71 and 72 or the BEMS server 5 .

In addition, when a person's movement is predicted, it is notified to the image sensor in charge of the destination area, and the image sensor determines the control amount such as sensing sensitivity. Alternatively, the management terminal 200 may calculate the control amount for the image sensor of the notification destination, and notify the image sensor of the obtained control amount.

While embodiments of the invention have been described, the embodiments are provided by way of example and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 1... Lighting equipment, 2... Air conditioner, 3, 3-1~3-m, 3-1~3-n... Image sensor, 3A, 3B, 3C, 3D... Image sensor, 4... Controller, 5... BEMS server , 10... building, 15... processor, 30... register, 30a... camera information, 31... camera section, 31a... fisheye lens, 31b... aperture mechanism, 31c... image sensor, 32... memory, 32a... program, 32b... image data, 32c dictionary data 32d motion vector data 33 processor 33a image processing unit 33b control unit 34 communication unit 35 internal bus 41 air conditioning controller 42 elevator controller 43 security controller , 44... Disaster prevention controller, 45... Lighting controller, 70... Database, 71, 72... Gateway, 80... Server, 101, 102... Sensor network, 200... Management terminal, 210... Monitor, 220... ROM, 230... RAM, 240 Memory unit 240a Program 240b Motion vector data 240c Statistical analysis data 240d Design data 240e Map data 250 Processor 250a Acquisition unit 250b Map creation unit 250c Integration processing unit , 250d... analysis unit, 250e... motion vector processing unit, 250f... prediction unit, 250g... notification unit, 250h... data processing unit, 260... optical media drive, 270... communication unit, 300... communication network, 500... intra-building network , 600 . . . cloud.

Claims (17)

a plurality of image sensors for sensing an object;
a computing device in communication with the image sensor;
The image sensor is
an imaging unit that captures an image of the assigned area and acquires image data;
an image processing unit that performs image processing on the image data to detect motion information in the area ;
A control unit that varies the sensitivity of the sensing ,
The computing device
an acquisition unit that acquires motion information detected by each of the image sensors;
a storage unit that stores map data indicating a correspondence relationship between the areas assigned to each of the image sensors and a map of the target space;
an integration processing unit that integrates the acquired motion information based on the map data to calculate motion information in the target space ;
an accumulation unit that accumulates the acquired motion information;
an analysis unit that statistically analyzes the accumulated motion information to generate statistical analysis data;
a prediction unit that predicts the movement of the target based on the statistical analysis data;
a notification unit that notifies an image sensor, to which the destination area of the target is assigned, of the movement of the target based on the result of the prediction;
The control unit
An image sensor system that varies the sensitivity of the sensing according to the notification . 2. The image sensor system according to claim 1, wherein said control unit controls at least one of a frame rate, a gain value of said imaging unit, and an exposure time to vary said sensing sensitivity. 2. The image sensor system according to claim 1 , wherein said computing device further comprises a motion information processor that complements and/or corrects said motion information based on said statistical analysis data. Based on the statistical analysis data, the analysis unit analyzes at least one of flow line, utilization rate, congestion degree, crowd behavior, layout estimation, queue estimation, office usage rate, and factory operation status in the target space. 2. The image sensor system of claim 1 , wherein the image sensor system estimates 2. The image sensor system according to claim 1 , wherein said analysis unit creates a color map indicating at least one of distribution, direction, and density of said motion information in different colors based on the analysis result of said motion information. 2. The image sensor system according to claim 1 , wherein said computing device further comprises a data processing unit that processes at least one of said motion information and said statistical analysis data to generate output data. The storage unit
storing design data including the installation positions of the plurality of image sensors and the orientation direction of the imaging unit;
2. The image sensor system according to claim 1, wherein said computing device further comprises a map generator that generates said map data based on said design data. The integration processing unit
2. The image sensor system of claim 1, wherein in overlapping portions of areas assigned to different image sensors, motion information sensed by the different image sensors are aligned with each other to calculate motion information in the object space. The integration processing unit
2. The image sensor system of claim 1, wherein the motion information detected by different image sensors is connected in chronological order to calculate motion information in the object space. 2. The image sensor system of claim 1, wherein the motion information is a quantity indicative of magnitude of motion of the object in the object space. 2. The image sensor system of claim 1, wherein the motion information is a quantity indicative of the direction of motion of the object in the object space. 2. The image sensor system of claim 1, wherein the motion information is a motion vector containing magnitude and direction of motion of the object in the object space. 2. The image processing unit according to claim 1, wherein said image processing unit senses a plurality of sensing items among sensing items including presence/absence in said target space, number of people, amount of activity, illuminance, staying on foot, person recognition, attribute identification and said movement information. An image sensor system as described. 6. The image sensor system according to claim 5, wherein said analysis unit color-codes the congestion state for each area of said target space and publishes said color map as a heat map accessible via the Web. A computing device capable of communicating with a plurality of image sensors for sensing objects and detecting motion information in an assigned area,
an acquisition unit that acquires motion information detected by each of the image sensors;
a storage unit that stores map data indicating a correspondence relationship between the areas assigned to each of the image sensors and a map of the target space;
an integration processing unit that integrates the acquired motion information based on the map data to calculate motion information in the target space;
an accumulation unit that accumulates the acquired motion information;
an analysis unit that statistically analyzes the accumulated motion information to generate statistical analysis data;
a prediction unit that predicts the movement of the target based on the statistical analysis data;
A computing device comprising: a notification unit that notifies an image sensor to which the destination area of the target is assigned, of the movement of the target based on the result of the prediction. A computer implemented sensing method comprising:
A process in which the computer acquires motion information detected by each of a plurality of image sensors that sense the target and sense motion information in the assigned area;
The computer calculates motion information in the target space by integrating the obtained motion information based on map data indicating a correspondence relationship between the areas assigned to each of the image sensors and a map of the target space. process and
the computer storing the acquired motion information;
a process in which the computer statistically analyzes the accumulated motion information to generate statistical analysis data;
a process in which the computer predicts the movement of the target based on the statistical analysis data;
the computer notifying an image sensor assigned a destination area for the object to move the object based on the result of the prediction;
and the image sensor varying the sensitivity of the sensing in response to the notification. A computer capable of communicating with multiple image sensors that sense objects and detect motion information in assigned areas,
obtaining sensed motion information for each of the image sensors;
a step of calculating motion information in the target space by integrating the acquired motion information based on map data indicating a correspondence relationship between the areas assigned to each of the image sensors and a map of the target space;
accumulating the obtained motion information;
a step of statistically analyzing the accumulated motion information to generate statistical analysis data;
predicting the movement of the object based on the statistically analyzed data;
and notifying an image sensor to which the destination area of the target is assigned to move the target based on the result of the prediction.

Images