JP2022141274A - Image sensor, sensing method, and program - Google Patents

Abstract

To provide an image sensor, a sensing method and a program, which allow for obtaining space usage.SOLUTION: An image sensor 3 is provided, comprising an image capturing unit 31 configured to obtain image data by capturing an image of a target space, a memory 32 for storing the image data, and a processor 33 configured to perform image processing on the stored image data to compute a space usage rate, which is an indicator of an extent of usage of the target space by a moving object at each location.SELECTED DRAWING: Figure 4

Description

Embodiments of the present invention relate to image sensors, sensing methods, and programs.

A recent image sensor is equipped with a CPU (Central Processing Unit) and memory, and can be said to be a so-called built-in computer with a lens. It also has advanced image processing functions, and can analyze the captured image data and calculate the presence/absence of people moving within the field of view, or the number of people. An object that can be detected by moving within the field of view of the image sensor is hereinafter referred to as a moving object. For example, living objects such as humans and animals, and non-living moving objects (vehicles, etc.) are moving objects.

Japanese Patent No. 6616521 JP 2019-200715 A JP 2012-146022 A Japanese Unexamined Patent Application Publication No. 2013-137701 Japanese Patent Application Laid-Open No. 2003-109001

The people flow data is an index that converts the degree of movement of moving bodies into data. By using people flow data, it is possible to create information that is effective for visual confirmation of people (heat map), by expressing places where there is a lot of movement of people in dark colors and places where there is little movement in light colors. . In other words, people flow data is data obtained by accumulating how individual people move, and it can be said that people are the subject of discussion.

From a different point of view, there is an increasing need to quantify the extent to which a moving object uses space. For example, it is considered to create a more comfortable space by calculating the degree to which a person stays in place as a numerical value and applying that value to an air conditioning or lighting control system. However, there is no known technology that enables numerical calculation of the degree to which a moving object uses space. With existing technology, only indices such as density and degree of congestion are known, and indices that can be directly input to devices and can be called space utilization rates of moving objects cannot be obtained.

Therefore, an object is to provide an image sensor, a sensing method, and a program capable of obtaining the utilization rate of space.

According to an embodiment, an image sensor comprises an imaging unit, a storage unit, and a processor. The imaging unit acquires image data by imaging a target space. The storage unit stores image data. The processor performs image processing on the stored image data to calculate a space usage rate, which is an index indicating the extent to which the moving object uses the target space for each location.

FIG. 1 is a schematic diagram illustrating an example of an image sensor system including an image sensor according to an embodiment. FIG. 2 is a diagram showing an example of floors in a building. FIG. 3 is a block diagram showing an example of the image sensor system shown in FIG. FIG. 4 is a block diagram showing an example of the image sensor 3 according to the embodiment. FIG. 5 is a diagram showing an example of data flow in the image sensor 3. As shown in FIG. FIG. 6 is a flow chart showing an example of the processing procedure of the image sensor 3 according to the embodiment. FIG. 7 is a diagram showing an example of an image within the field of view of the imaging section 31. As shown in FIG. FIG. 8 is a schematic diagram showing an example of space usage data. FIG. 9 is a schematic diagram showing an example of automatic area division based on the seating rate. FIG. 10 is a schematic diagram showing an example of a layout map. FIG. 11 is a diagram showing an example of regions divided based on motion distribution. FIG. 12 is a diagram showing attributes of a target space divided into regions.

An image sensor can acquire a wider variety of information than a human sensor, a light sensor, an infrared sensor, or the like. By using a wide-angle lens such as a fisheye lens, the area that can be captured by a single image sensor can be expanded. Image distortion can be corrected by computational processing. Image sensors are also known that have a function of masking an area in the field of view that is not desired to be sensed, and have a learning function.

A recent image sensor is equipped with a CPU (Central Processing Unit) and memory, and can be said to be a so-called built-in computer with a lens. It also has advanced image processing functions, and can analyze captured image data and calculate the presence or absence of people, or the number of people, for example. Some have communication functions, and have a high affinity with IoT (Internet of Things) technology. Furthermore, it is also expected to be an edge device for big data analysis. It is important to process image data captured by a fisheye camera and obtain highly accurate sensing data. An example of a solution using an image sensor is described below.

[Constitution]
FIG. 1 is a schematic diagram illustrating an example of an image sensor system including an image sensor according to an embodiment. In FIG. 1, lighting equipment 1, lighting controller 4, and image sensors 3 (3-1 to 3-n) are provided, for example, on each floor of building 400, and are connected to gateway device 7 so as to be communicable. The gateway device 7 on each floor is communicably connected to, for example, a building management center BEMS (Building Energy Management System) server 5 via an intra-building network 8 .

The BEMS server 5 can be connected to a cloud computing system (cloud) 100 via a TCP/IP (Transmission Control Protocol/Internet Protocol) based communication network 10, for example. The cloud 100 includes a server 200 and a database 300, and provides services related to building management and the like.

FIG. 2 is a diagram showing an example of floors in the building 400. As shown in FIG. As shown in FIG. 2, the lighting equipment 1, the outlets of the air conditioners 2, and the image sensor 3 are arranged on, for example, the ceiling of each floor. Each image sensor 3 is assigned an area to be sensed. In FIG. 2 it is shown that areas A1 and A2 each cover approximately half of the floor. All areas together can cover the floor of the target space. As indicated by hatching in the drawing, the areas assigned to different image sensors 3 may partially overlap.

The image sensor 3 looks down on the assigned area and catches it within the field of view at an angle of view, and acquires image data by capturing an image within the field of view. This image data is processed by the image sensor 3 to generate various sensing data such as human information or environmental information. That is, the image sensor 3 generates sensing data related to the target space.

The environment information is information about the environment of the target space (zone), such as the illuminance and temperature of the floor. Person information is information about a person in the target space. Examples of personal information include the number of people in an area, behavior of people, amount of activity of people, presence/absence indicating the presence or absence of people, and the like.

In recent years, it has been studied to improve the comfort and safety of people in living environments by controlling lighting equipment and air conditioning equipment based on such information. It is also possible to calculate environmental information and person information for each small area obtained by dividing a zone into a plurality of areas. This small area may be associated with the allocated area for each image sensor 3 .

In the embodiments, an image sensor capable of introducing a quantity called space utilization and calculating and outputting this quantity will be described. Here, the space usage rate is an index indicating the degree (usage rate) of the space that the moving body uses. For example, the space utilization rate is an index that indicates the degree to which people stay in each place.

Using the space may mean, for example, sitting at a seat and doing office work. Alternatively, it may be possible to move through passages or stairs in the space. Alternatively, it may be walking around the sales floor in a store or the like.

FIG. 3 is a block diagram showing an example of the image sensor system shown in FIG. In FIG. 3, a gateway device 7, a management terminal 20, a BEMS server 5, and a lighting controller 4 are connected to an intra-building network 8 to form a building system. A typical communication protocol for the intra-building network 8 is the Building Automation and Control Networking protocol (BACnet (registered trademark)). In addition, protocols such as DALI, ZigBee (registered trademark), and ECHONET Lite (registered trademark) are also known.

The gateway device 7 accommodates a plurality of image sensors 3 (3-1 to 3-n) as subordinates. The image sensors 3-1 to 3-n are connected by a sensor network 9 in a unicursal manner (crossover wiring). The sensor network 9 connects the image sensors 3-1 to 3-n and the gateway device 7 so as to be able to communicate with each other. EtherCAT (registered trademark), for example, is known as a protocol for the sensor network 9 .

The gateway device 7 converts communication protocols between the sensor network 9 and the in-building network 8 . This allows the BEMS server 5 and the management terminal 20 to mutually communicate with the image sensors 3-1 to 3-n. The BEMS server 5 acquires sensing data from the image sensors 3-1 to 3-n and controls the air conditioning and lighting of the building 400. FIG.

The management terminal 20 individually accesses the image sensors 3-1 to 3-n to input commands and perform various settings. The lighting controller 4 controls the lighting equipment 1 based on control from the BEMS server 5 .

FIG. 4 is a block diagram showing an example of the image sensor 3 according to the embodiment. The image sensor 3 includes an imaging section 31 , a memory 32 , a processor 33 and a communication section 34 . These are connected to each other via an internal bus 35 . Image sensor 3 with memory 32 and processor 33 is a computer.

The imaging unit 31 is a so-called fish-eye camera, and includes a fish-eye lens 31a as a wide-angle lens, a diaphragm mechanism 31b, an image sensor 31c, and a register 30. The fish-eye lens 31a captures the area in the view looking down from the ceiling (downward angle of view) and forms an image on the image sensor 31c. The amount of light from the fisheye lens 31a is adjusted by a diaphragm mechanism 31b. The image sensor 31c is, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor, and generates a video signal with a frame rate of, for example, 30 frames per second. This video signal is digitally encoded and output as original image data. This image data is stored in the memory 32 (image data 32b).

The register 30 stores camera information 30a. The camera information 30a is information related to the imaging unit 31, such as the state of the auto gain control function, frame rate, gain value, and exposure time, or information related to the image sensor 3 itself.

The memory 32 is a semiconductor memory such as SDRAM (Synchronous Dynamic RAM), or a non-volatile memory such as NAND flash memory or EPROM (Erasable Programmable ROM). The memory 32 stores a program 32a for causing the processor 33 to function, image data 32b acquired by the imaging section 31, parameters 32c, and an algorithm 32d.

The processor 33 is, for example, a multi-core CPU (Central Processing Unit) and is an LSI (Large Scale Integration) tuned to execute image processing at high speed. The processor 15 can also be configured by an MPU (Micro Processing Unit), an FPGA (Field Programmable Gate Array), or the like. The processor 33 executes a program 32a read into an internal register (not shown) to realize various functions described in the embodiments.

The communication unit 34 can be connected to the sensor network 9, and mediates the exchange of data with a communication partner (the gateway device 7, the management terminal 20, another image sensor 3, or the BEMS server 5, etc.). The communication interface may be wired or wireless. Arbitrary topologies such as line wiring, ring wiring, and tree wiring can be applied to the topology of the communication network. The communication protocol may be a general purpose protocol or an industrial protocol. A single communication method may be used, or a plurality of communication methods may be combined. The object of communication may be either real-time data or stored data. Furthermore, it is also possible to select constant communication, on-demand communication, or the like.

In particular, the communication unit 34 receives sensing data from the image sensor 3, processing results of the processor 33, processing data, map information, parameters related to image analysis, various images (heat map, etc.) used as a basis for calculating the space utilization rate, Various images such as original images and processed images, various parameters such as space usage rate calculation algorithms and layout maps, or various parameters for other events (dictionary data, etc.) are transmitted and received via the sensor network 9 as a communication network. . As a result, the data and information can be shared with the gateway device 7, other image sensors 3, the BEMS server 5, the management terminal 20, and the like via the intra-building network 8 and the like.

By the way, the processor 33 includes a space usage data generation unit 33a, a space usage rate calculation unit 33b, and an area setting unit 33c as processing functions according to the embodiment. These functional blocks can be understood as processes generated by the program 32a stored in the memory 32 being loaded into the registers of the processor 33 and the processor 33 performing arithmetic processing as the program progresses.

The space usage data generation unit 33a performs image processing on the image data 32b in the memory 32, and detects areas in which people exist (person areas) for each of a plurality of frames. The human region is accumulated over multiple frames. Then, the space use data generation unit 33a generates space use data as an average value between frames for a predetermined accumulation time, for example. Here, the accumulation time can be set to any value from the management terminal 20, for example, and registered in the parameter 32c. Space usage data may be represented as array data, or may be represented as an image such as a heat map.

As image processing, techniques such as background subtraction, inter-frame subtraction, template matching, machine learning, or deep learning can be applied. The unit of the person region can be arbitrarily set, for example, in pixel units, block units, area units, the entire image, or a mixture thereof. This can also be arbitrarily changed from the management terminal 20, for example, and the set value is registered in the parameter 32c.

A parameter is an index for associating the field of view of the imaging unit 31 with an area on the real space, and can be stored in a plurality of types in the embodiment. Examples of parameters include numerical values such as the ceiling height and detection range of the target space, a floor layout map, and a layout map divided into areas. It is appropriately read and used by the processor 33 .

The space usage rate calculator 33b calculates the space usage rate based on the space usage data generated above. Space utilization can be defined in various ways. That is, various space utilization rates can be calculated based on different algorithms (functions). Individual algorithms are stored in memory 32 (algorithm 32d) and read out by processor 33 for use. Which algorithm is used to calculate the space usage rate can be arbitrarily set from the management terminal 20, for example.

For example, a unit of calculation can be cited as an example of different definitions of space utilization. In other words, it is possible to define whether the space usage rate is calculated in units of the entire image or in units of individual areas. The area may be, for example, each seat, or may be defined as a unit in which a plurality of areas are grouped together. Information such as how to set the area can be arbitrarily set from the management terminal 20, for example, and registered in the parameter 32c.

Also, the denominator and numerator in the calculation formula can be set to different amounts to define a plurality of space usage rates. For example, if the denominator is the area on the image or the area on the real space, at least two types of space usage rates can be defined. Furthermore, the denominator may be defined as the area of the entire image or area in which a person has existed in the past. In doing so, it is also possible to define a reference time such as the average value of the past day, the average value of the past week, the average value of the past month, the average value of each season, or the average value of a year. . The reference time to the denominator may be the same as or different from the accumulated time when calculating space utilization. The amount applied to the denominator and the reference time can be arbitrarily set from the management terminal 20, for example, and registered in the parameter 32c.

As the numerator, for example, the area of the person region can be applied. One can also apply one or more conditionings to the person and define the numerator as the area of one or more person regions. Examples of conditioning include, for example, magnitude of movement (person working in an office, person walking, etc.). In addition, posture (person standing, sitting, etc.), clothing (person wearing uniform, person wearing suit, etc.), attributes (gender, age), etc. can also be used as examples of conditioning. can. Furthermore, by specifying an individual within the field of view and using the area of each individual as a numerator, it is possible to calculate the space usage rate of the individual.

Furthermore, the area setting unit 33c of the processor 33 sets the area for calculating the space usage rate to the target space. For example, it is possible to set an area for each chair on the office floor, and define the space usage rate as the amount (seating rate) indicating how long people are sitting in each chair. A technique such as semantic segmentation, for example, can be applied to the setting of the area.

FIG. 5 is a diagram showing an example of data flow in the image sensor 3. As shown in FIG. In FIG. 5, the image data 32b acquired by the imaging unit 31 is temporarily stored in the memory 32 and then transferred to the space use data generation unit 33a of the processor 33. FIG. The space usage data generation unit 33a
Space usage data is generated from the image data and passed to the space usage rate calculator 33b.

The space usage rate calculator 33b refers to the memory parameter 32c and the algorithm 32d to calculate the space usage rate from the space usage data. The calculated space usage rate is transmitted from the communication unit 34 to the gateway device 7 or the like via the sensor network 9 and directly used as basic data for air conditioning control and lighting control.

[Action]
Next, the operation of the above configuration will be described.
FIG. 6 is a flow chart showing an example of the processing procedure of the image sensor 3 according to the embodiment. In FIG. 6, the image sensor 3 stores the image data 32b acquired by the imaging unit 31 (step S1) in the memory 32 and accumulates them (step S2). The stored image data 32b is passed to the space usage data generator 33a.

The space usage data generation unit 33a generates space usage data based on the set parameters (step S3), and passes it to the space usage rate calculation unit 33b. The space usage rate calculator 33b applies a predetermined calculation algorithm (calculation formula) to the space usage data to calculate the space usage rate (step S4). The calculated space usage rate is transmitted from the communication unit 34 to the BEMS server 5 via the gateway device 7 and used as basic data for air conditioning control and lighting control.

FIG. 7 is a diagram showing an example of an image within the field of view of the imaging section 31. As shown in FIG. FIG. 7 shows a one-frame image captured at a specific point in time, and captures the location of a person at this point in time. Some are seated, some are walking. The imaging unit 31 accumulates time-series image data (frames) in the memory 32 . Image data over a period of time is then processed to generate space usage data reflecting, for example, the duration of a person's presence for each cell.

FIG. 8 is a schematic diagram showing an example of space usage data. The period of stay of a person is graphically shown for each cell (square) set in the target space. In FIG. 8, for example, the maximum value is 60 minutes, and the sitting time is visualized with, for example, 4 ranks until the length of stay reaches 10 minutes. By visualizing the sitting time from the accumulated image data and further analyzing it, for example, a value such as ([Seating rate] from 9:00 to 10:00 on DD, MM, 2021 is 48%) is calculated. be. This [seat rate] is an example of the space usage rate.

FIG. 9 is a schematic diagram showing an example of automatic area division based on the seating rate. The region setting unit 33c (FIG. 4) sets a region for calculating the space usage rate in the target space. For example, by setting the area to be excluded from the calculation target (mask area) as the target space, you can set the space usage rate to be calculated only for the other area (the area to be calculated). can do.

Moreover, it is also possible to further divide the unmasked area (area to be processed) into a plurality of areas and set a plurality of areas. It is also possible to divide an image into areas by coloring it like a color map.

Furthermore, it is also possible to set the area division from the area-divided layout map. FIG. 9 shows an example of this, in which an area with a relatively high [seat rate] is set as a desk area, and an area with a relatively low [seat rate] is set as an aisle area. The desk area is indicated by dots, and the aisle area is indicated by diagonal hatching. In this way, it is possible to automatically perform region division based on the motion distribution of the person.

FIG. 10 is a schematic diagram showing an example of a layout map. The layout map is downloaded from, for example, the BEMS server 5 and stored in the memory 32 of the image sensor 3 in advance.
FIG. 11 is a diagram showing an example of regions divided based on motion distribution. Assume that, for example, a rectangular area is detected in the target space by human motion analysis. By comparing this area with the layout map, the attributes of the area (desk, passage, etc.) can be detected.

FIG. 12 is a diagram showing attributes of a target space divided into regions. As shown in FIG. 12, the desk area and the passage area can be automatically recognized and set only by the sensing processing of the image sensor 3 itself. The set area is notified to the BEMS server 5 and the like by the communication unit 34 .

As described above, the area setting unit 33c (FIG. 4) automatically sets the area for calculating the space usage rate in the target space. In addition to the human movement analysis described above, semantic segmentation, for example, can also be used to automatically segment areas into desks/floors/walls, and the like. Alternatively, the lighting may be turned on and off according to the lighting section of the lighting, and the automatic area division may be performed based on the result obtained by sensing the illuminance distribution at that time. Each area can also be set by an administrator's operation using the management terminal 20. FIG.

[effect]
As described above, in this embodiment, from the image data acquired by the image sensor 3, the space usage data is calculated as the basis for calculating the space usage rate. The space utilization rate is calculated as data for In other words, the image sensor itself can calculate not only the space usage data such as the color map, but also the control amount that can be applied directly to the device to be controlled.

In this way, it becomes possible for the image sensor to directly generate a control amount for device control without the need for intervening computational resources such as a server. This makes it possible to dramatically increase the degree of freedom in air conditioning control and lighting control in buildings.

Further, in the embodiment, when calculating the space utilization rate, a plurality of parameters and a plurality of calculation formulas (algorithms) are registered in the image sensor 3 in advance, and the space utilization rate under different parameters and algorithms is calculated. made it possible to calculate Since this is done, it is possible to calculate the space usage rate according to the characteristics of the target space and building. For example, by using trends in space usage rates by month, season, etc., it is possible to make use of this to improve the equipment layout of the office.

From these, according to the embodiment, it is possible to provide an image sensor, a sensing method, and a program capable of obtaining the utilization rate of the space. As a result, for example, it is possible to output an index indicating the degree to which a person stays for each location, and it is possible to provide an image sensor that is highly compatible with device control.

In addition, this invention is not limited to the said embodiment. For example, the imaging unit 31 is not limited to a fish-eye lens, and it is also possible to use another type of wide-angle lens such as a so-called panorama lens using a mirror surface.

Also, the object to be detected is not limited to a person, and may be an object other than a person. For example, if you use a background dictionary for detecting and identifying the background in an office, a desk dictionary for detecting desks, a chair dictionary for detecting chairs, or a dictionary for detecting fixtures and multifunction devices, only people can be detected. Instead, it can detect various objects.

While embodiments of the invention have been described, the embodiments are provided by way of example and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 1... Lighting installation, 2... Air conditioner, 3 (3-1-3-n)... Image sensor, 4... Lighting controller, 5... BEMS server, 7... Gateway apparatus, 8... Building network, 9... Sensor network, DESCRIPTION OF SYMBOLS 10... Communication network 15... Processor 20... Management terminal 30... Register 30a... Camera information 31... Imaging part 31a... Fish-eye lens 31b... Diaphragm mechanism 31c... Image sensor 32... Memory 32a... Program, 32b image data 32c parameter 32d algorithm 33 processor 33a space usage data generator 33b space usage rate calculator 33c area setting unit 34 communication unit 35 internal bus 100 ... cloud, 200 ... server, 300 ... database, 400 ... building.

Claims (11)

an imaging unit that captures an image of a target space and acquires image data;
a storage unit that stores the image data;
and a processor that performs image processing on the stored image data to calculate a space usage rate, which is an index indicating the extent to which a moving object uses the target space for each location. The processor
a space usage data generation unit that performs image processing on the image data to generate space usage data that indicates the distribution of the moving object in the target space;
2. The image sensor according to claim 1, further comprising a space usage rate calculator that calculates said space usage rate based on said space usage data. The storage unit stores a plurality of parameters that associate the field of view of the imaging unit with an area in real space,
3. The image sensor according to claim 2, wherein said space usage rate calculator calculates said space usage rate based on at least one of said plurality of parameters. The storage unit stores a plurality of algorithms that express the space utilization rate with different definitions,
3. The image sensor according to claim 2, wherein said space usage rate calculator calculates said space usage rate based on one of said plurality of algorithms. 2. The image sensor according to claim 1, wherein said processor further comprises an area setting unit that sets an area for calculating said space usage rate in said target space. further comprising a communication unit that accepts access via a communication network,
6. The image sensor according to claim 5, wherein said area setting unit receives setting of said target space by access via said communication network. 6. The image sensor according to claim 5, wherein said area setting unit automatically sets said object space by an automatic area division algorithm. 2. The image sensor according to claim 1, further comprising a communication unit that sends said calculated space utilization rate to a communication network. 2. The processor as set forth in claim 1, wherein said space utilization rate and at least one sensing item among sensing items including presence/absence in said target space, number of people, amount of activity, illuminance, and staying on foot. Image sensor as described. A sensing method executed by the processor of an image sensor comprising an imaging unit that captures an image of a target space and acquires image data, a storage unit, and a processor,
the processor storing the image data in the storage unit;
the processor image processing the stored image data to generate space usage data indicative of a distribution of moving objects in the object space;
A sensing method, wherein the processor calculates a space usage rate, which is an index indicating the extent to which the moving body uses the target space for each location, based on the space usage data. 10. A program for causing a computer comprising an imaging unit that captures an image of a target space and acquires image data, and a processor to function as the image sensor according to any one of claims 1 to 9.

Images