JP7353875B2 - Sensor system and sensing method - Google Patents

Description

Embodiments of the present invention relate to sensor systems and sensing methods.

Image sensors as IoT (Internet of Things) devices are attracting attention. Recent image sensors are equipped with a memory and a processor (CPU (Central Processing Unit), MPU (Micro Processing Unit), etc.), and can be said to be an embedded computer with a lens. It is also highly compatible with AI (Artificial Intelligence) technology, and can analyze captured image data to calculate the presence/absence of people, as well as the number of people. For example, there is a literature that explains obstacle detection using chamfer distance. A method has also been proposed in which lighting is controlled by checking a dictionary to confirm that a moving object detected by an image sensor is a person.

A technique is known in which lighting is controlled by checking a dictionary to confirm that a moving object detected by an image sensor is a person. A method is also known in which a target person is imaged from two directions and the person is identified by comparing the images with features of images obtained in advance by imaging from at least three directions.

A method using HOG (Histograms of Oriented Gradients) is available as a method capable of extracting feature information with high discriminative ability for a target moving object. This method, called brightness gradient direction histogram, is a method in which a candidate region is divided into a plurality of blocks and the brightness gradient direction of each block is created into a histogram. By using only the presence or absence of a gradient, it is said to be less susceptible to changes in brightness and robust to changes in shape. Furthermore, a method called CoHOG (Co-occurrence HOG), which has improved shape recognition ability, has also been developed.

Japanese Patent Application Publication No. 2010-9847 Unexamined Japanese Patent Publication No. 2016-1447 JP 2017-169768 Publication Special table 2015-504616 publication

Lakshitha Dantanarayana*, Gamini Dissanayake, Ravindra Ranasinge, C-LOG: A Chamfer distance based algorithm for localization in occupancy grid-maps, CAI Transactions on Intelligence Technology 1 (2016) 272e284, http://www.journals.elsevier.com/ cai-transactions-on-intelligence-technology/ Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, INRIA Rh.one-Alps, 655 avenue de l'Europe, Montbonnot 38334, France

One of the challenges related to image recognition using image sensors and machine learning is continuous learning using teacher data (training data). In other words, it is no longer difficult to create training data after the image sensor is installed at the site. This is because creating training data requires a large amount of manual work. Therefore, there has been a problem in that it is difficult to improve the performance by accumulating learning of data for recognition/judgment (hereinafter referred to as a determination model) installed in the image sensor.

Therefore, an object is to provide a sensor system and a sensing method that can continuously improve performance.

According to the embodiment, the sensor system includes a plurality of image sensors disposed in a building, an analysis section, and a state determination section. The analysis unit analyzes image data captured by the image sensor and obtains moving object information about a target moving object. The state determination unit determines the state of the target moving object from the moving object information based on a determination model generated by machine learning using teacher data.

FIG. 1 is a diagram showing an example of a sensor system 1 according to an embodiment. FIG. 2 is a functional block diagram showing an example of the central server 30. As shown in FIG. FIG. 3 is a flowchart illustrating an example of the processing procedure of the system according to the embodiment. FIG. 4 is a flowchart illustrating an example of the processing procedure of the system according to the embodiment. FIG. 5 is a flowchart illustrating an example of the processing procedure of the system according to the embodiment. FIG. 6 is a flowchart showing an example of the process in step S102. FIG. 7 is a flowchart showing an example of the process in step S114. FIG. 8 is a flowchart showing an example of the process in step S109. FIG. 9 is a flowchart showing an example of the process in step S109. FIG. 10 is a diagram illustrating an example of a truth table that can be used to evaluate a determination model. FIG. 11 is a graph showing an example of the results obtained when the classification of FIG. 10 is applied to the determination model. FIG. 12 is a diagram showing an example of a PR curve. FIG. 13 is a diagram showing an example of a PR curve. FIG. 14 is a diagram showing an example of applying the truth table to an actual system. FIG. 15 is a diagram showing an example of a moving object imaged by an image sensor. FIG. 16 is a diagram illustrating an example of a moving object imaged by an image sensor. FIG. 17 is a diagram illustrating an example of a moving object imaged by an image sensor. FIG. 18 is a diagram showing an example of a moving object imaged by an image sensor. FIG. 19 is a diagram showing an example of a flow line. FIG. 20 is a diagram showing an example of a flow line. FIG. 21 is a diagram showing an example of a flow line. FIG. 22 is a diagram showing an example of a flow line. FIG. 23 is a functional block diagram showing an example of the image sensors 10a to 10f.

Next, embodiments of the invention will be described with reference to the drawings.
The need for solutions using image sensors is increasing year by year. For example, from the images taken, we can quickly identify the unexpected departure of a wandering elderly person from a nursing home, a person who suddenly has difficulty walking due to poor health, an inebriated person, a lost child in a public place, or a person whose behavior is suspicious. There is a desire to detect moving objects (people) that exhibit unusual behavior. There is also a desire to understand the flow lines of vehicles such as forklifts operating in warehouses, even if they are the same moving objects. As described above, there are various needs for image recognition technology, and there are high expectations for image sensors and sensor systems using image sensors.

If the target moving object (person, vehicle, etc.) has a transmitter such as a mobile phone or smartphone, it can be detected using GPS (Global Positioning System) or the like. However, this method cannot be applied when the user does not have a transmitter or indoors. In this case, the flow line will be grasped through calculation processing using a determination model using, for example, an image sensor installed in the building.

Learning for improving decision models is broadly classified into unsupervised learning and supervised learning. Unsupervised learning is effective when the rules and target state are clear, such as in Shogi or Go. For example, a winning state is clear, such as capturing the opponent's ball or having a large area (territory) covered by one's own team. If the goal is clear, it will be possible to determine whether the decision model has been improved. This method is also called reinforcement learning, and is a method of learning through repeated trial and error from the results of actions on the environment. Incidentally, a model obtained by machine learning using data is called a learned model. The determination model according to the embodiment is also an example of a trained model.

Clustering is also a type of unsupervised learning. Clustering is a method that discovers and learns regularities from given data, and can extract the essential structure and type behind the data. Since it is a trial-and-error approach, the decision model can only be improved after a large amount of data has been accumulated. Since it tends to be highly dependent on data and produces rather arbitrary results, it may be difficult to handle and does not necessarily lead to the expected results immediately.

In a case where a target moving object is continuously monitored, which is assumed in the embodiment, an approach that uses supervised learning to reproduce the expected judgment model output is efficient and effective.

Teacher data is needed to improve a decision model when it is impossible to know whether the decision model has been improved unless someone tells it. Teacher data is data in which the correct answer is known in advance. By using training data, it is possible to determine whether or not the estimation result based on a judgment model using certain data matches the actual result. Such teacher data can be used as training data, and if it is a sample that has not been studied in the past, it can be used as test data as a means to know the current accuracy of the judgment model.

In the application described in the embodiment, data such as movement of people in a facility, building, etc., tracing of flow lines, or the number of people in a room is collected using an image sensor. The future number of people in a certain room can be effectively predicted by understanding the flow lines to adjacent rooms. These collected data do not have target values, nor do they actively influence the environment, such as controlling the flow of people according to a judgment model. Therefore, supervised learning is the most effective way to improve the decision model in such cases. That is, this is a method of learning which class should be identified using data (teacher data) for which the correct answer is known in advance, and improving the judgment accuracy of the judgment model.

If such data that can be used as training data can be collected on-site, it is possible to create and optimize a decision model that locally fits the captured image. In the embodiment, a technique will be described that allows automatic/semi-automatic improvement of a determination model to be continued using pseudo training data even after an image sensor is installed at a site.

For example, when grasping the flow line of a target moving object using a group of installed image sensors, if the target individual or the social or organizational group to which the individual belongs can be identified, this may be used as training data. For example, if the person who enters the building is an employee who commutes to work every day, it is safe to judge that the person's flow line information is that of an ordinary person. In order to identify an individual or a group to which an individual belongs to a target moving object, information collected from the card information carried by the person or, for example, a facial recognition judgment model based on deep learning, can be used to match registered information. , makes a judgment. Incidentally, there are currently many open source facial recognition recognition models available.

If it is possible to correlate information that identifies an individual or the group to which the individual belongs in this way with flow line information obtained from a group of image sensors installed at the site, supervised flow line information (i.e., whether it is an ordinary person or It is possible to obtain on-site data (data that is known in advance). In other words, even after the image sensor is installed at the site, it is possible to continue to collect effective training data and continue to improve the judgment model that determines whether a person is an ordinary person based on the flow line.

Note that a group means a social group to which an individual belongs. For example, there are organizational affiliations and classifications that are mainly collected from mobile electronic information, and affiliation groups that are estimated from information such as age group, gender, and clothing obtained from the biological information of the target moving object.

Furthermore, a truth rate may be added to a determination model that determines whether a series of movements of a target moving object can be used as training data. The truth rate is a quantity that expresses the truth or falsity between a predicted class based on already collected image data and an actual class as a ratio. If it is judged from the truth/false rate that the target sequence of actions has a high probability of being misrecognized, it can be decided not to use that part as training data.

[Embodiment]
In the embodiment, attention is paid to moving object information representing one feature of a target moving object. Moving object information is an example of information obtained by analyzing image data acquired by an image sensor. A typical example of moving object information is flow line information obtained by tracing the position of target information. Furthermore, information indicating the magnitude and direction of movement of a target moving object (motion vector), information indicating the direction of movement and its speed (velocity vector), etc. are also examples of moving object information. Furthermore, scalar values such as a quantity indicating the magnitude of movement of a target moving body and a quantity indicating the direction of movement of the target can also be understood as moving body information. Further, living body/device information (person information, forklift information), etc. collected by the image sensor system is also an example of moving body information.
Furthermore, shape information representing the shape of a stopped target moving object is also an example of moving object information. As shape information, HOG (Histogram of Oriented Gradients) and CoHOG (Co-occurrence Histogram of Oriented Gradients) are known.

(composition)
FIG. 1 is a schematic diagram showing an example of a sensor system 1 according to an embodiment. The sensor system 1 is a system for realizing monitoring of the target space 4 using sensors. The target space 4 may be, for example, a building such as a building, a manned space such as a store or an office, or an unmanned space such as a factory where equipment is brought in/out, or a server room. The three-dimensional shape of the target space 4 is not limited to a rectangular parallelepiped, but may be a space of any shape in which a sensor is installed. Alternatively, the sensor system 1 may be a multi-story, substantially hollow building including the entire building in which the sensor system 1 is installed.

The target space 4 is divided into areas a to f, for example. Sensor monitoring of areas a to f is realized by corresponding image sensors 10a to 10f and communication units 20a to 20f, respectively. For example, the central server 30 collects and processes information sent from each communication unit 20, thereby realizing air conditioning control in the entire target space 4. The image sensor 10 and the communication unit 20, which are distinguished by common letters (a to f), correspond to the areas a to f. Hereinafter, when areas a to f are not distinguished, they will simply be referred to as "area".

The sensor system 1 includes image sensors 10a to 10f, communication units 20a to 20f, and a central server 30. The communication units 20a to 20f respectively control image sensors 10a to 10f that are distinguished by common letters (a to f). Hereinafter, when the image sensors 10a to 10f are not distinguished, they are referred to as the image sensor 10, and when the communication units 20a to 20f are not distinguished, they are referred to as the communication unit 20.

Each communication unit 20 and central server 30 are connected via a network 5 and configured to be able to communicate with each other. The network 5 is a communication path for transmitting/collecting data from image sensors arranged throughout the building, other information collecting means, existing systems, etc., and controlling each sensor. The network 5 may be, for example, Ethernet (registered trademark), which is a typical wired LAN, or PLC (Power Line Communication). Communication between each communication unit 20 and the central server 30 may be realized by wireless communication. The central server 30 performs overall control of each communication unit 20 based on measurement data acquired from each communication unit 20 .

A face authentication sensor 7a and an entrance/exit gate 8a are installed in an area 6 adjacent to the target space 4. The face authentication sensor 7a is an example of a biological information sensor, and the entrance/exit gate 8a is an example of electronic information reading means. Both are examples of existing systems. The face authentication sensor 7a collects biological information of the target moving object, that is, face information. The entrance/exit gate 8a collects electronic information unique to the target moving object individual or the (social) group to which the target moving object belongs. For example, employee cards and entry/exit cards provide unique electronic information. This information is transmitted to the central server 30 or nearby (for example, the image sensor 10c) via communication paths 9a and 9b forming a network.

The administrator input information acquisition unit 11a collects information when the administrator/user inputs whether or not the collected image information can be used as teacher data. The collected information is transmitted to the central server 30 from the communication unit 21a. This information can be used as one material for determining whether the feature amount/flow line information obtained from the captured image information can be used as teacher data. In this case, the central server 30 is part of the existing system and also has a function of collecting/controlling information on the sensor system 1.

The amount of information from sensors used to identify individuals, such as face recognition sensors and facial recognition sensors, is usually much larger (several times or more) than the amount of data for tracing moving objects using image sensors installed on the ceiling. It is. Therefore, there are cases in which image sensors such as face recognition sensors that require large amounts of data processing are placed throughout the facility/building (Case 1), and cases in which face recognition sensors are placed only in specific locations such as Area 6. In other cases, the total amount of resources required (hardware resources, software resources, network traffic, etc.) vary greatly. The embodiment corresponds to case 2 and can be configured with fewer communication/computation resources.

Various methods can be considered for collecting biological information and information on portable electronic media. for example,
(A) The target moving object belongs to a group that can pass through the door 23 where visitors are restricted.
(B) Whether or not there is a target moving object in a specific location 25 of the mesh-like divided area 24 corresponding to the image sensor 10f. This is information generated by an action such as taking a seat at one's own seat.
(C) Whether or not the lighting in this specific location was turned on/off.
(D) Whether or not the target moving object has accessed the information terminal 26 such as a mobile PC. Such.
There are various methods for identifying an individual or a group to which an individual belongs. Most of the information is specific locations within the facility, and becomes information that helps determine whether or not the feature amount information of the target moving object can be used as training data via the map information of the facility.

FIG. 2 is a functional block diagram showing an example of the central server 30. As shown in FIG. Image sensors 10a to 10f included in the sensor system collect environmental information (sensor data) such as room temperature and illuminance. The sensor data is transmitted to the central server 30 from the communication units 20a to 20f connected to each of the image sensors 10a to 10f, and is received by the communication unit 31.

Information input by the administrator/user into the administrator input information acquisition section 11a is transmitted to the central server 30 via the communication section 21a. The face authentication sensor 7a and/or the entrance/exit gate 8a are examples of electronic information reading means. Both are examples of existing systems. The face authentication sensor 7a collects biological information of the target moving object, that is, face information. Information on the target moving object collected by the entrance/exit gate 8a is transmitted from the communication section 22a to the central server 30 via the identification information acquisition section 12a.

The administrator input information acquisition unit 11a, the face authentication sensor 7a, and the entrance/exit gate 8a are examples of existing systems, and each of them transmits the collected data to the central server 30. The existing system is an example of a data collection system.

In the embodiment, information that can be collected through activities that have the original purpose, such as passing through a security gate, moving to a predetermined location, and using facility equipment, which are the daily or normal actions of the target moving object, is collected using existing equipment. Included as collected data from the system.

Sensor data acquired by the sensor system and collected data collected by the existing system are collected in the central server 30. The central server 30 receives these data via the communication unit 31. Among the sensor data collected from the sensor system, the feature amount extraction section 32 of the control section 46 extracts information necessary for creating a determination model from the sensor data.

After extraction, the determination unit 33 determines the image data. The determining unit 33 determines whether the image data satisfies predetermined conditions and whether the data can be sent to the teacher data determining unit 40. At the time of determination, the image data 34a and feature amount 34b stored in the storage section 34, and the dictionary data 35a of the storage section 35 may be used.

After extracting the feature amount, the feature amount and image data may be stored in the storage unit 34 as image data 34a and feature amount 34b. These controls and processes are executed by the CPU 36. The information obtained from the image sensor in this way is sent to the teacher data determination section 40.

On the other hand, the collected data collected by the existing system is sent to the central server 30 and sent to the control unit 46 via the communication unit 31 within the central server 30. If the collected data is data from the face authentication sensor 7a, it is sent to the feature amount extraction unit 32, where the feature amount is extracted. The determining unit 33 determines whether the data satisfies predetermined conditions or whether the data may be sent to the teacher data determining unit 40. When making a determination, the image data 34a, the feature amount 34b, and the dictionary data 35a may be used. Further, the extracted feature amount and image data may be stored in the storage unit 34 as image data 34a and feature amount 34b. These controls and processes are performed by the CPU 36.

If the collected data is data from the entry/exit gate 8a, unlike biometric information, there is no need to extract feature amounts. Therefore, for this type of data, the determination unit 33 determines whether the information is usable. This control and processing is performed by the CPU 36. This information is sent to the teacher data determination section 40.

The teacher data determining unit 40 associates sensor data (image data, etc.) from the sensor system with collected data from the existing system using the information linking unit 41. The criteria for association includes time information such as data time stamps, and/or location information such as the location of the sensor and the imaging range, as well as information such as card reading operations, imaged operations, etc. It is possible to refer to the motion information of the target moving object.

The associated data is determined by the acceptance/rejection determining unit 42 as to whether or not it can be employed as teacher data. When making the determination, comparison may be made with teacher data already stored in the teacher data database (DB) 44. As a result of the determination, if it is determined that the data is "teacher data", it is saved in the teacher data DB 44.

A series of these controls is performed by the CPU 43, which controls the central server 30 in an overall manner. In FIG. 2, the CPU 43 that controls the acceptance/rejection determination unit 42 is shown to also control the entire cooperative system, including the functions of each block diagram shown in FIG.

Furthermore, the central server in FIG. 2 includes an existing system DB 37, personal information DBs 38 and 39, and a notification/destination DB 45.

3 to 5 are flowcharts showing an example of the processing procedure of the system according to the embodiment. In FIG. 3, the teacher data determination unit 40 selects a target moving object in step S101. If there is no target moving object, the following procedure will not start. Next, the teacher data determining unit 40 determines the presence or absence of another identification information in step S102.

Step S102 includes a process of confirming whether the moving body information of the target moving body and another identification information are issued from the same individual. That is, it is determined based on time, for example, whether data is collected at the same time as the imaging time of the moving object information. Next, it is determined based on location, for example, whether data is collected at the same or close location. Furthermore, it is determined based on the motion information of the moving object, for example, whether or not the passing time from the passing image of the entrance/exit gate matches the electronic information collection time from the card.

In step S102, (Yes), that is, if different identification information for the same moving object is obtained, in the next step S103, the teacher data determination unit 40 determines that the target moving object can be used as teaching data based on the different identification information. Determine whether or not. If it can be used as teacher data using another identification information (Yes), the teacher data judgment unit 40, in step S104, uses information (data) from the sensor system and information (data) from the existing system. ). Next, in step S105, the teacher data determining unit 40 temporarily stores the data that has been subjected to the association processing. Note that in the case of (No) in step S102 or (No) in step S103, the flow chart of FIG. 5 is reached.

The next steps S106 and S107, as well as steps S112 and S113, are related to notification of normality/abnormality diagnosis of the target moving object. These procedures are an effective means of obtaining training data when data is biased during actual operation, such as when the amount of data for abnormality diagnosis is small.

For example, consider a case where the ratio of normal diagnosis to abnormal diagnosis during actual operation is, for example, 10,000:1. At the time when moving object information indicating an abnormal value that may become valuable training data is obtained, it is possible to immediately notify the notification target/destination to alert them. Subsequent tracking and monitoring of moving objects can be performed manually, or automatic processing that allocates sensor system and data processing system resources to focus on target moving objects that show abnormal values enables real-time tracking with high temporal and spatial resolution. This makes it possible to collect more effective teacher data. In addition, prompt notification is necessary for facility management/operation when a movement that requires immediate attention is detected, such as a behavioral pattern of a suspicious person, a malfunction of automated driving equipment, or a person becoming unsteady or unconscious. It is also a measure.

In step S106, the teacher data determining unit 40 determines whether notification of normal/abnormal diagnosis of the target moving object is required. "Normal" here means normal behavior of a normal person. "Abnormal" is when the movement trajectory of the target moving object does not follow the normal range of movement, such as automatic driving equipment, wandering people, lost children, suspicious people, people who become sick while walking, and drunk people. This refers to cases in which people have characteristics that are different from those of ordinary people. Many of these can be diagnosed using flow line information, which is time-lapse information, obtained from shape information and velocity vector information of the target moving object.

If the diagnosis result requires notification to a predetermined administrator or destination, the result is (Yes) in step S106, and the teacher data judgment unit 40 performs "notification to notification target/destination" in step S107. . If the notification is not necessary (No), the process of step S107 is not performed and the process of the next step S108 (FIG. 4) is performed. Note that even after it is determined in step S103 that the target moving object can be used as training data, the target moving object is tracked in the embodiment. Then, for example, after taking into consideration the flow line information of the target moving object synthesized from image data from multiple image sensors, it is determined whether "the flow line has no errors and is within the range that can be used as training data" in step S108. judge.

If (Yes) in step S108 of FIG. 4, the teacher data determination unit 40 performs registration/learning as teacher data in the next step S109. Note that a method may be adopted in which the newly collected teacher data is used to check whether the judgment model is improved when compared with a predetermined index value, and then registered as the teacher data. By using the teacher data including the degree of effectiveness for improving the judgment model in this way, it becomes possible to order the degree of effectiveness of the teacher data.

In step S108, the teacher data determining unit 40 checks whether the data satisfies a predetermined threshold value as the teacher data. Here, it is confirmed that the flow line obtained from the collected feature values of the target moving object can be drawn with one stroke, and that it has passed a predetermined time and distance. If the data can withstand as teaching data, the result is (Yes).

The teacher data determination unit 40 performs registration/learning processing as teacher data in step S109. The registered information includes normal/abnormal diagnosis results based on a judgment model for current image recognition. The information storage destination is a recording medium in which past teacher data/test data is stored (step S110). In step S108, if the current flow line information cannot be used as teacher data (No), the process for the target moving object is ended (END).

If (No) in step S102 of FIG. 3, it means that there is no other identification information for the target moving object before data collection is completed, so the teacher data determination unit 40 temporarily stores the data in step S111. . Next, in step S112 of FIG. 5, the teacher data determining unit 40 determines whether the target moving object is normal or abnormal and determines whether notification is necessary. It is expected that the judgment model used for this judgment will be able to make judgments with higher accuracy as training data from actual sites accumulates. The same applies to "determination of normality/abnormality of target moving object and determination of notification necessity" in step S106.

If the result of step S112 in FIG. 5 is that notification is required (Yes), the teacher data determining unit 40 notifies the notification target person/destination in step S113. If the result of step S112 is that notification is not required (No), the processing procedure moves to step S114. The teacher data determining unit 40 determines (Is there a normal/abnormal determination input after data collection?) in step S114.

This is the case when new information is added during or after data collection. The normality/abnormality determination of the target moving object may be determined by a human being who visually inspects the data or images, or by a determination model installed in a separate computer, such as a computer with richer hardware/software resources. You can also enter the results. If the result of this determination is input (Yes), the processing procedure returns to step S103 in order to determine whether or not it can be used as teacher data. If such other identification information is not input, the result is (No) in step S114, and the series of processing for the target moving object ends (END).

FIG. 6 is a flowchart showing an example of the process in step S102. When the process in step S102 is started (START), the teacher data determining unit 40 determines in step S102b, "Was the other identification information confirmed to be that of the target moving object?" Confirmation criteria include (1) time stamp, (2) location, and (3) movement/flow line confirmation using image data.

(1) When using time stamps, by comparing the times at which the image sensor and the identification information acquisition unit 12a each start collecting/obtaining information on a moving object, it is determined that both information is from the same moving object. do.

(2) In the case of location, the target moving object is identified based on the positional relationship between the location of the identification information acquisition unit 12a and the image sensor closest thereto. In FIG. 1, the location where the face authentication sensor 7a or the entry/exit gate 8a is placed and the closest image sensor 10c for moving body tracing are located closest to each other and are used.

(3) In the movement/flow line confirmation using image data, the image sensor It can be confirmed whether the moving object imaged by 10c and the moving object information obtained by another recognition information obtaining means are the same individual. Of course, in order to make it more certain that the object is the target moving object, it is possible to identify whether it is the target moving object by combining (1), (2), and (3).

If it is confirmed in step S102b that the other identification information is that of the target moving object, that is, in the case of (Yes), the teacher data determination unit 40 determines (Is the information from the biological information of the target moving object?) in step S102c. to judge. Biometric information can be obtained through many means, including facial recognition, voice recognition, fingerprints, glow, and vein patterns. If it is biometric information, the result is (Yes) in step S102c, and the teacher data determining unit 40 compares the collected biometric information or information obtained by processing the collected biometric information with the DB in step S102e. The comparison destination is the individual's unique information DB (biometric information) that is stored and registered separately (step S102g).

After the comparison, the teacher data determination unit 40 determines "degree of match determination with DB OK>threshold" in step S102f. If it exceeds the threshold value (Yes), it is determined that another identification information is that of the target moving object. That is, a Yes determination is made in step S102 (FIG. 3) that means "another identification information exists" (step S102i), and the processing procedure jumps to step S103 in FIG. 3.
On the other hand, if (No) in step S102f, the two pieces of information do not match. That is, a No determination is made in step S102 (FIG. 3) that means "there is no other identification information" (step S102k), and the processing procedure jumps to step S111 in FIG. 5 via FIG. 3.

Note that in step S102c, if the other identification information is not biometric information (No), the teacher data determining unit 40 determines whether the other identification information is unique electronic information of the target moving object or the group to which the target moving object belongs in the next step S102d. Make a judgment. Similarly to step S102c, if the other identification information is electronic information (Yes), check with the information registered in advance in the personal unique information DB (electronic media information) in "Verification with DB". (Step S102h). After the comparison, if there is a match in step S102j, "matching degree determination with DB>threshold", the result is (Yes). If YES in step S102j, a YES determination is made (step S102i), and the processing procedure jumps to step S103 in FIG. 3. If the threshold value is not met in step S102j, it is determined that the electronic information cannot be identified (No), the determination is No (step S102k), and the processing procedure jumps to step S111 in FIG. 5 via FIG. 3. do.

Further, in step S102d, if the electronic information is not collatable, the result is (No), this also becomes a No determination (step S102k), and the processing procedure jumps to step S111 in FIG. 3. This means that neither the biological information nor the electronic information obtained as separate identification information could identify the individual of the target moving body or the group to which the individual belongs. The group to which the target object (individual) belongs refers to a social group, including belonging to an organization or company related to the target facility, or entering the facility as a visitor. A group that can be estimated in advance whether the flow line information is normal or abnormal based on statistics or accumulated data.

FIG. 7 is a flowchart showing an example of the process in step S114. In step S114b, the teacher data determining unit 40 loads the settings if the administrator/user has input the settings. The administrator/user determines the setting values in the first process, and then, if automatic processing is set, the extraction of teacher data will continue without any further operation. Even if automatic processing is selected once, if you wish to change it afterwards, for example, by operating a key while loading the setting values, automatic processing will be interrupted and you can return to the manual input screen according to the flow. To simplify the explanation, below, operations will be described at the time of initial setting or when automatic setting is canceled by key input.

When the set value is loaded in step S114b, the teacher data determination unit 40 determines whether the administrator and/or a predetermined destination has been notified in step S114c. Determine. Here, it is determined whether a moving object that is a warning target has been observed during the observation period and a notification to that effect has been received.

If there is no notification, it means that there is no warning target moving object, and the processing procedure moves to step S114d, ``Should the determination status be displayed in order of the abnormal value for the input period?''. If there is a notification in step S114c, (Yes), the teacher data determining unit 40 determines whether to display the notified list in step S114h. If (Yes) here, the process of "displaying the notification list" is performed in step S114i, and the process moves to the next step S114e.

If (No) in step S114h, the teacher data determining unit 40 performs the process of step S114d. If the notification has not been made in step S114c, the result is (No), and the process of step S114d is performed. If (Yes) in step S114d, the process moves to step S114e after performing step S114j "list display for a predetermined period".

Even in the case of (No) in step S114d, the teacher data determination unit 40 performs the determination in step S114e "Is there any data to be selected as a teacher data candidate?". If (Yes) here, the teacher data judgment unit 40 allows the administrator/user to check the displayed candidates in step S114k "Input teacher data candidates in list for predetermined period". to select candidate data.

Next, the teacher data determining unit 40 determines in step S114f, "Is there a threshold value to be set at the time of automatic selection?", and determines whether or not to perform step S114l, "input threshold value." Next, the teacher data judgment unit 40 determines whether or not to perform similar processing if the data satisfies the threshold in step S114g, "Do you want to automatically process the same flow for a predetermined input period?" . If the answer is No here, a YES determination is made (step S114n), and "another identification information exists" in FIG. 3 is established, and the processing procedure jumps to step S103 (FIG. 3). If YES in step S114g, a predetermined setting value is input (step S114m), and then the process similarly jumps from the YES determination to step S103.
Note that if the result in step S114e is (No), there is no teacher data, so the determination is No (step S114o). With the No determination in step S114, the processing procedure ends (END in FIG. 5).

FIGS. 8 and 9 are flowcharts showing an example of the process in step S109. Here, the storage/registration process in step S109 will be described with respect to a procedure in which the administrator/user interactively decides data to be registered in the DB, and a process in which the process is automatically processed.

In FIG. 8, when the procedure is started, the teacher data determining unit 40 loads the default value or the setting value set by the administrator/user in step S109b "load setting value". Next, the teacher data determination unit 40 displays a list of teacher data that could be collected during the current period in step S109c "Display list of teacher data to be targeted this time."

Next, the teacher data judgment unit 40 determines whether or not to use the collected teacher data as training data for improving the judgment model in step S109d "Do you want to perform local fitting of the judgment model using the additional teaching data?" to judge. If it is to be used (Yes), the processing procedure moves to step S109e. If the training data is not used as training data (No), the teacher data judgment unit 40 makes a judgment in step S109k as to whether to evaluate the performance of the determination model as the teacher data.

That is, the obtained teacher data can be used as training data, or as test data used to evaluate the current judgment model. Even data used as test data can of course be reused as training data.

Note that in step S109d, if local fitting (learning) is to be performed using the collected teacher data (Yes), the processing procedure moves to step S109e "Have you selected the teacher data applied to improve the judgment model?". Here, the teacher data determining unit 40 selects the teacher data to be used from among the collected teacher data. If at least one data is selected (Yes), the process moves to step S109f. The teacher data judgment unit 40 displays the evaluation value of the current judgment model and the collected teacher data in step S109f "Display the judgment model performance evaluation value and the amount of improvement before and after local fitting (improved when the amount of improvement > 0"). Compare the evaluation value of the changed judgment model after adding and learning.

For this comparison, separately prepared teacher data that is not used for learning may be used. If the amount of improvement is a positive value (amount of improvement>0), it indicates that the addition of the determination model had an effect of improvement. In step S109g, the teacher data determining unit 40 determines, "Since the amount of improvement is >0, should the local fit result be reflected and the version of the determination model be updated?" If the amount of improvement is <0, the default is (No) and the process moves to step S109m.

If the amount of improvement is >0 and the determination model is to be improved, the result is (Yes). A positive threshold value may be provided, and the determination model may be automatically improved (updated) only when the amount of improvement>threshold value.

If the determination model is to be upgraded (Yes), the teacher data judgment unit 40 adds the collected teacher data to the DB in step S109h "Update teacher data/test data DB". Next, the teacher data judgment unit 40 implements the upgraded judgment model in step S109i "locally fitted judgment model implementation process".

The teacher data determining unit 40 determines in step S109o in FIG. 9, "In order to follow the current processing flow, do you want to save it as a setting and perform automatic processing from next time?" If automatic processing is to be performed, the answer is (Yes), and in step S109p "Save current processing flow as setting values", if the flow is the same as the current flowchart, the setting values will be set so that automatic processing can be performed from next time. Save as.

If the flows are different, such as when there is no collected data, you can save each possible pattern as a setting value by saving it as a setting value after inputting it. This allows the process to be performed automatically without any operation by the administrator/user. To cancel automatic processing, the administrator/user inputs a specific key during step S109b "Load setting value" or step S109c "Display list of current target teacher data" in FIG. It may also be possible to cancel automatic processing.

On the other hand, in the case of (No) in step S109d of FIG. 8, that is, if local fitting is not performed, the teacher data determination unit 40 determines in step S109k of FIG. ?/Do you want to end?". Here, the teacher data determining unit 40 determines whether or not to test the current determination model using the currently collected teacher data.

If performance evaluation is to be performed (Yes), the teacher data judgment unit 40 uses the collected teacher data as test data to evaluate the current judgment model in step S109i "Display performance evaluation value of current judgment model". Evaluate.

In step S109e (No), that is, if no training data for improving the judgment model is selected, the teacher data judgment unit 40 performs the "data addition to training data/test data DB" step S109m in FIG. Make a decision as to whether to do so or not. If (Yes) here, the teacher data determination unit 40 adds the collected teacher data to the DB by "updating the teacher data/test data DB" in step S109n. When the addition is completed, the processing procedure moves to step S109o.

In step S109g of FIG. 8, if (No), that is, the version of the determination model is not to be performed, the process moves to step S109m of FIG. If data is not added to the DB in step S109m (No), the process moves to step S109o. In this way, it is possible to automate the process using the default settings, or when re-executing the steps once followed, it is also possible to perform the automation process by reflecting the manual settings and selections.

Information related to the target moving object is collected by the face authentication sensor 7a and/or the entrance/exit gate 8a, and is transmitted from the communication unit 22a to the central server 30 via the identification information acquisition unit 12a. In the embodiment, information that can be collected through activities that have the original purpose, such as passing through a security gate, moving to a predetermined location, and using facility equipment, which are the daily or normal actions of the target moving object, is collected using existing equipment. Included as information from the system. Further, the administrator/user inputs information to the administrator input information acquisition section 11a, and the information is transmitted to the central server 30 via the communication section 21a. At least until automatic processing/semi-automatic processing is set, information on the target moving object is input manually, independently from the sensor system and the existing system.

With reference to FIGS. 10 to 12, evaluation indicators for determining how good a determination model is will be described. For example, based on the truth table shown in FIG. 10, it can be determined that if an action is likely to be detected as an error, it will not be adopted as training data. Furthermore, the degree of improvement of the determination model can be evaluated using this evaluation index.

FIG. 10 shows an example in which the classes predicted by the determination model are classified into two, ie, positive and negative. Then, depending on the actual class, it is divided into "true" cases and "false" cases. "True" is when the prediction was correct, and "false" is when the prediction was wrong. Classifying grades in this way provides a clue to the estimation accuracy of the model. There are four classifications: TP, FN, FP, and TN.

(TP) The actual class has been correctly estimated for the predicted class (positive: P), and is true. This case is written as TP.
(FP) The actual class is incorrectly estimated for the expected class (Positive: P) and is False. This case is written as FP.
(TN) The actual class has been correctly estimated for the predicted class (negative: N), and is true. This case is written as TN.
(FN) The actual class is incorrectly estimated for the expected class (Negative: N) and is False. This case is expressed as FN.

If the action (behavior) of the target moving object belongs to one of these four categories, if it is below a separately given threshold, it is possible to perform operations such as not applying this judgment model as the estimation reliability is low. becomes.
Action of target moving object (TP, FP, TN, FN) > Threshold (TP/FP/TN/FN)
If: Apply the model.
Action of target moving object (TP, FP, TN, FN) < Threshold (TP/FP/TN/FN)
If: Do not apply the model.

FIG. 11 is a graph showing an example of the results obtained when the classification of FIG. 10 is applied to the determination model. The performance of the decision model can be evaluated according to the classification shown in FIG. For example, assume that the truth values for a sample with a research population of 100 are as shown in FIG. At this time, there are various methods for making the determination. The two most basic methods are as follows.

Precision rate: Indicates what percentage of cases were determined to be positive among cases that were actually positive.
Precision = TP/(FP + TP)
Recall rate: Represents the percentage of data that was predicted to be positive out of the actual positive data.
Recall rate (recall)= TP/(FN + TP)
These values change depending on the boundary value for determining Yin and Yang, that is, the threshold value to be set. There are two main ways of change:
(1) A state where the precision rate is high and the recall rate is low. This state is a state in which there is little waste, but there are many missed decisions.
(2) A state where the precision rate is low and the recall rate is high. This state is a state in which there are few missed decisions, but there is a lot of wasteful determination.
In this way, precision and recall are in a trade-off relationship. Which one is suitable depends on the application. The choices are whether you should point it out, even if you make a mistake, or whether you should be as accurate as possible, even if you omit it.

Note that if the population is biased, it is not necessarily easy to use with both precision and recall. As shown in FIG. 11, both precision and recall are often difficult to use. Therefore, in practice, a method using the F value may be considered.
F value = 2x (precision rate x recall rate) / (precision rate + recall rate)
The F value is expressed by the above formula.

For example, as shown in FIG. 13, model evaluation based on the break-even point (BEP) using a PR curve can be considered. Furthermore, there is a method of determining the quality of the model by determining the BEP value from the PR curve.

In other methods, since precision and recall are in a trade-off relationship, a PR (Precision-Recall) curve is drawn and a BEP (Break Even Point) is calculated from it. To draw a PR curve, it is sufficient to plot how the recall rate and precision rate change while changing the set threshold of the judgment model. In the process of plotting, a threshold value at which the recall rate is maximum, a threshold value at which the precision rate is maximum, and a point at which the recall rate and the precision rate match, that is, the BEP value, are determined. This is a method of determining the quality of the model based on this BEP value.

By setting a threshold value for the BEP value, it is possible to adopt a method of making a judgment based on the model after confirming that it is a good model. By using these evaluation criteria, it is possible to quantitatively understand whether or not the determination model has been improved. In addition to the PR curve, there is RoC (Receiver Operator Characteristic) as an index for determining the performance of machine learning. When analyzing data where the number of positive and negative cases is biased, that is, the data is highly distorted, the PR curve is suitable when there are many negative cases. There are various other evaluation indicators, which are used to judge the degree of improvement of the judgment model.

In the application of the embodiment, cases where the movement line of the target moving object shows a movement line that is not normal, such as a lost child, a suspicious person, a person in poor health, etc. ) is considered negative. If the majority is negative, it is possible to apply a method of determining the quality of the judgment model from the PR curve.

12 and 13 are diagrams showing examples of PR curves. FIG. 12 shows the calculated recall rate and precision rate in a table format. An example is shown in which the values predicted by the judgment model for 10 data points are arranged in descending order, and the recall rate and precision rate are calculated according to the formula. The predicted value of 1.00 is added to the table and calculated using the formula. For the same judgment model, the recall rate and precision rate are calculated when the predicted value is used as the threshold. This calculation requires actual truth values. In other words, it is necessary to use training data.

FIG. 13 is a diagram showing an example of a PR curve. When the table of FIG. 12 is plotted as a graph, the curve of FIG. 13 is obtained. In addition to the BEP value, the area surrounded by a to k including the origin may be used as an index for evaluating the performance of the determination model.

FIG. 14 is a diagram showing an example of applying the truth table to an actual system. For example, consider the truth table for leaving and sitting. Suppose that when a person is seated (positive) and a person leaves a seat as negative, false positives (FP) occur 40 times out of 100 trials in which it is determined that the person is not seated even though he or she is seated (positive). On the other hand, suppose that there were 20 cases in which it was determined that the user was seated correctly when the user took a seat (positive). For example, when calculating the precision in this case,
Seated precision = TP/(FP + TP) = 20/(20 + 40) = 0.33
becomes.

On the other hand, it is assumed that the case where it is determined that the user has not left the seat even though the user has left the seat (FN) occurs 10 times. The correct decision to leave the seat (TN) occurred 30 times. Therefore, the precision rate for leaving seats is
Away precision rate = TN/(FN + TN) = 30/(10 + 30) = 0.75
becomes.

Similarly, when calculating recall and F-measure,
Seated recall = TP/(FN + TP) = 20/(10 + 20) = 0.66
Away recall rate = TN/(FP+TN) = 30/(40 + 30) = 0.43
Seated F value = 2 x (precision rate x recall rate) / (precision rate + recall rate)
= 2 x (0.33 x 0.66) / (0.33 + 0.66) = 0.44
Away F value = 2x (0.75x 0.43) / (0.75+0.43) = 0.54
becomes.

For example, when the F value threshold is set to 0.50, the seated F value is 0.44, which causes a large false recognition and cannot be used as training data. The absentee F value exceeds the threshold, which means that it satisfies the conditions for inclusion as teacher data.

15 to 18 are diagrams showing examples of moving objects imaged by an image sensor. FIG. 15 shows an example of a moving object imaged by an image sensor placed on the ceiling of a building. For example, it may be considered that the image is captured by the image sensor 10c in FIG. FIG. 16 shows an example of an image captured using the face authentication sensor 7a of FIG. 1. Comparing the two, it can be seen that the amount of information obtained also differs greatly depending on the imaging direction.

FIG. 17 is an example of a case where a moving object is determined to be normal when a flow line is traced using one or more image sensors. In reality, an image similar to that shown in FIG. 15 is obtained. For example, if it is known that the moving object exhibiting the behavior shown in FIG. 17 is a staff member in a building, flow line information collected as teacher data is used to improve the determination model as the flow line of a person behaving normally.

FIG. 18 shows an example of a case where it is determined that there is an abnormality. Based on the information on the movement line that has been grasped, a list of possible elderly people wandering or lost children is created. The facility manager confirms whether there were any such visitors in the building on that day, and enters that information from the terminal. If there is a relevant person and flow line information has been obtained, the collected flow line information is used as the flow line of the person who behaves abnormally and as training data to improve the judgment model.

19 to 22 are diagrams showing examples of flow lines.
The flow lines in FIG. 19 are drawn by arranging the flow lines that the target moving object traced through the imaging spaces of the image sensors 10a, 10b, and 10c in FIG. The target moving object is moving approximately in a straight line, and within this range it can be determined that the moving object is operating normally.

The flow line in FIG. 20 is the flow line followed by the target moving object in the imaging space of the image sensors 10a to 10f in FIG. Since it is still moving in a straight line, it is determined that it is within the normal operating range.

FIG. 21 is an example of tracing the imaging spaces 10a to 10f, but the movement is not linear, but is a flow line that is likely to be judged as abnormal. However, this target moving object traces the locations marked with "○" in the diagram once. Therefore, it is inferred that this target moving object was interested in the position of the "○" mark. If the target moving object had been identified by means of identification, such as information on an employee ID card, for example, the expiration date of the fire extinguisher placed at the "〇" mark would have been periodically checked. can be inferred.

FIG. 22 shows an example of tracing the imaging spaces 10a to 10f. Here, it is assumed that the information for identifying this moving object is only the information on the visitor card, and the details of the individual are unknown. Also, assume that the information on the visitor card shows that the user has never entered this facility in the past. The flow line goes back and forth in area 10b. In 10d and 10f, the movement does not seem to have a clear intention, such as drawing an arc. The travel time was longer than average, and as a result, it returned to 10a, making it hard to believe that there was a clear destination. In this case, it can be determined that the flow line is abnormal, regardless of the presence or absence of additional information for identifying the moving object.

As described above, various applications can be considered by combining information from existing systems and flow line information. For example, consider detecting a lost child using an image sensor at a theme park. The age group of the target moving object can be determined from the information obtained from the pass card at the entrance/exit gate. If infants, elementary school students, or the elderly, who are likely to get lost, are included, it is possible to narrow down the target moving objects to be monitored with priority. In addition, if there are multiple people passing through the entrance/exit gate, but the target moving object is alone for a predetermined period of time, there is a high possibility that the object has gotten lost. to decide. This allows notification to be sent to the facility monitor. In addition, it is possible to collect information for designing the store's flow line.

As a solution, from among the data collected during image sensor operation, individuals or groups belonging to individuals, which can be used as training data, are collected using separate information (for example, card information used when entering and exiting, information on the number of people who come to work every day, etc.). One possible method would be to obtain the information from a facility employee (because it is unlikely that the child would act as a lost child) and link it to the motion information collected by sensors. For example, the other information may be in the form of a facility manager inputting information that there were no lost children today at the end of the day. Another method is to increase the reliability using other information, etc., and input the candidate data that can be used as training data into the judgment model for recognition/judgment, thereby increasing the evaluation value of the judgment model based on the truth table. It is also possible to judge based on the presence or absence of improvement. In this way, the collected training data allows automatic/semi-automatic continuous improvement of the decision model.

With the field of view of one image sensor, the range to be traced is narrow, and in order to make more accurate judgments, it is necessary to collect image information from multiple image sensors. In this case, the images collected by each image sensor are sent to a central server, where the flow line of the same person captured by multiple image sensors is traced in chronological order. As a result, in a sufficiently long tracing distance, it can be determined that the target moving object is, for example, a lost child by comparing it with a dictionary of movement lines that determine whether it is normal or abnormal.

There are methods to constantly detect faces and collect images and compare them with separately registered dictionary information to identify the target moving object, and to estimate emotions such as tension, pleasure, and displeasure from minute differences in facial expressions. . Data collection is usually possible with one image sensor.

One option is to have all the cameras in the facility capture images at a resolution that allows facial recognition, but for the sake of personal information protection, it is difficult to constantly perform facial recognition because the person being photographed must be able to see the camera within their field of view. You may feel uncomfortable. It is unreasonable in terms of spatial arrangement to always place an image sensor in the direction of movement of a target moving object and continuously photograph its face. In order to improve the accuracy of shape recognition of moving objects, including face recognition, it is necessary to learn a large amount of dictionary information and install a large number of image sensors that take into account the orientation of the moving object being photographed. .

In order to satisfy the desire to find moving objects (people) that behave in unusual ways in photographed images, it is possible to consider a method of understanding the movement line of the target moving object over a wide range. In this case, in order to improve the accuracy of judgment based on the flow line from the captured image data, there is an issue that requires local fitting, that is, adjustment of the judgment model including preprocessing of the image data captured at the installation site. be. However, it has not been possible to make such adjustments on-site, as it requires manpower and large amounts of data.

Once the flow line of the target moving object can be accurately grasped, the flow of people can be grasped, and when setting the room temperature of a room such as an office, it is possible to not only grasp the current number of people in the room, but also to check the surrounding position. By taking into account information about when the person is moving, it becomes possible to grasp the number of people who are expected to come to the room after that.

For these reasons, according to the embodiment, learning can be continued even after installation at the site. Therefore, according to the embodiment, it is possible to provide a sensor system and a sensing method that can continuously improve performance.

Note that this invention is not limited to the above embodiment. For example, the image sensor may include a functional block such as a feature extraction unit depending on the capabilities of a processor (CPU or MPU) included in the image sensor.

FIG. 23 is a functional block diagram showing an example of image sensors 10a to 10f (collectively designated by the reference numeral 10). The image sensor 10 includes a camera section 310 as an imaging section, a memory 320, a processor 330, and a communication section 20. These are connected to each other via an internal bus 350. By including the memory 320 and the processor 330, the image sensor 10 functions as a computer.

The camera section 310 includes a fisheye lens 31a, an aperture mechanism 31b, an image sensor 31c, and a register 300. The fisheye lens 31a captures the area in its field of view looking down from the ceiling, and forms an image on the image sensor 31c. The amount of light from the fisheye lens 31a is adjusted by an aperture mechanism 31b. The image sensor 31c is, for example, a CMOS (complementary metal oxide semiconductor) sensor, and generates a video signal at a frame rate of, for example, 30 frames per second. This video signal is digitally encoded and output as image data.

Register 300 stores camera information 30a. The camera information 30a is information regarding the camera section 310, such as the state of the auto gain control function, gain value, and exposure time, or information regarding the image sensor 3 itself.
For example, parameters that affect the sensitivity and accuracy of sensing, such as the frame rate, the gain value of the image sensor 31c, and the exposure time, are controlled by the processor 330.

The memory 320 is a semiconductor memory such as an SDRAM (Synchronous Dynamic RAM), or a non-volatile memory such as a NAND flash memory or an EPROM (Erasable Programmable ROM), and stores a program for causing the processor 330 to execute various functions related to the embodiment. 32a and image data 32b acquired by the camera unit 310. Furthermore, the memory 320 stores dictionary data 32c and a determination model 32d.

The dictionary data 32c is data in a table format in which sensing items and feature amounts are associated with each other, and can be generated by, for example, a method such as machine learning. For example, by pattern matching processing using the dictionary data 32c, it is possible to identify a detection target (such as a person) in the area.

The determination model 32d is a learned model generated by machine learning in the central server 30. The decision model 32d is sent from the central server 30 and stored in the memory 320.

In addition, mask setting data and the like may be stored in the memory 320. The mask setting data is used to distinguish between an area to be image-processed (a target area for image processing) and an area not to be image-processed (a non-target area for image processing) in the field of view captured by the camera unit 310.

The processor 330 loads and executes programs stored in the memory 320, thereby realizing various functions described in the embodiments. The processor 330 is, for example, an LSI (Large Scale Integration) that includes a multi-core CPU (Central Processing Unit) and is tuned to perform image processing at high speed. The processor 330 can also be configured with an FPGA (Field Programmable Gate Array) or the like. MPU (Micro Processing Unit) is also one of the processors.

The communication unit 20 is connectable to the network 5 and mediates data exchange with a communication partner (central server 30, other image sensor, etc.). The communication interface may be wired or wireless. Any topology such as a star type or a ring type can be applied to the communication network. The communication protocol may be a general purpose protocol or an industrial protocol. A single communication method may be used, or a plurality of communication methods may be combined.

In particular, the communication unit 20 receives sensing data from the image sensor 10, processing results from the processor 330, processed data, parameters, image data, motion vector data, accumulated past motion vector data, processed motion vector data, and statistics. Analysis data, output data, parameters, dictionary data, firmware, etc. are transmitted and received via the network 5 as a communication network.

By the way, the processor 330 includes a feature extraction section 33a, a moving object determination section 33b, and a flow line information calculation section 33c as processing functions according to the embodiment. The feature extraction unit 33a, the moving object determination unit 33b, and the flow line information calculation unit 33c load the program 32a stored in the memory 320 into the register of the processor 330, and the processor 330 performs arithmetic processing as the program progresses. It can be understood as a process generated by execution. That is, the program 320a includes a feature amount extraction program, a moving object determination program, and a flow line information calculation program.

The feature extraction unit 33a performs image processing on the image data 32b to calculate feature information of the target moving object in the target area. The obtained feature amount information may be stored in the memory 320 in addition to being sent to the central server 30. The moving object determination unit 33b determines what the target moving object is (for example, a person, a moving object such as a forklift, an automatic guided vehicle, etc.) based on the feature amount information. The flow line information calculation unit 33c calculates flow line information of the target moving object.

Note that the feature extraction unit 33a can also sense presence/absence, number of people, amount of activity, illuminance, walking retention, motion vectors, and the like. These multiple sensing items can also be set for each area. The feature extraction unit 33a can also obtain sensing results for each area by sensing each sensing item for each block that further divides the area into a plurality of blocks, and integrating the sensing results for each block.

Although an embodiment of the invention has been described, this embodiment is presented by way of example and is not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. This embodiment and its modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

Additional notes are listed below.
[1]
The system of the embodiment is
a sensor system including a sensor that collects motion information of a moving body;
an information collection system that collects electronic information about a moving object within a sensing range of the sensor system;
a calculation unit that executes a judgment model for judgment;
The system includes an information linking unit that links moving body information and electronic information and determines whether or not the moving body information is to be used as training data with correct answers for a determination model.
According to the system [1], by linking the moving object information (person information and forklift information) collected by the sensor system with the information obtained from the information collection system, information about the moving object or the group to which the moving object belongs can be obtained. Additional information can be obtained. Using the additional information, it is possible to judge whether or not it can be regarded as teaching data with correct answers.
[2]
The system of the embodiment is the system of [1], further comprising means for determining whether or not to modify the judgment model based on a comparison between the result of calculating the teaching data using the judgment model and the correct answer. It is.
According to the system [2], when training data with correct answers is obtained, it can be determined whether or not to modify the judgment model. Further, when the judgment model is modified, the judgment model can be improved.
[3]
In the system of [2], the system of the embodiment corrects the judgment model using the teaching data as a method for determining whether to use the teaching data with correct answers or determining whether to correct the judgment model. When implementing this, the system uses part of the training data as test data to determine whether the judgment model is closer to the desired state using an index value calculated based on the resulting truth rate. be.
According to the system in [3], by using training data as test data, the truth rate of the judgment model can be obtained, and based on this, precision rate, recall rate, F value, etc. can be obtained as index values. , a BEP value is obtained as an index value based on the truth rate obtained by changing the threshold value of the judgment model. Using these indicators, it is possible to judge whether the decision model has improved. This makes it possible to judge whether the data can be adopted as training data or whether it is better to further modify the judgment model.
[4]
The system of the embodiment is the system according to any one of [1] to [3], wherein the moving object information includes shape information of the moving object and velocity vector information indicating the moving direction and speed of the moving object. be.
According to the system [4], the orientation of the moving object is determined from the moving direction of the moving object. With this, it is possible to cope with the fact that the shape information of the moving object changes greatly depending on the direction of the moving object. In addition, when two moving objects are photographed overlapping each other, it is possible to distinguish between a single individual and multiple individuals based on velocity vector information or flow line information (time-series cumulative addition of velocity vector information). There are also many. Therefore, by using the shape information and velocity vector information of the moving object as a set, the identification/identification of the moving object becomes easier. This provides moving object information of a moving object with a clear identity, so to speak, and has the effect of making it possible to select more reliable training data.
[5]
The system of the embodiment is a system in which the sensor of any one of the systems [1] to [3] is an image sensor that images a moving object.
According to the system [5], moving body information about a moving body is collected by an image sensor.
[6]
In the system of the embodiment, in any one of [1] to [4], electronic information is stored in a recording medium of a portable device for entering and leaving a facility.
According to the system [6], daily entry/exit management information such as employee identification cards, visitor cards, smartphones, and embedded RFIDs can be used as electronic information.
[7]
In the system of the embodiment, in any one of [1] to [4], electronic information is stored in a recording medium of an information device for personal use or assigned to a group to which the individual belongs.
According to the system [7], information obtained from PCs, tablets, smartphones prepared for individuals or groups to which individuals belong, or control panels installed in facilities can be used as electronic information. I can do it. That is, information acquired from a device to which access is restricted can be made into electronic information.
[8]
The system of the embodiment is the system according to any one of [1] to [4], in which electronic information is transmitted as information from an individual or a group to which the individual belongs, based on operation history information of equipment installed at a specific location. It is a recognition system.
According to the system in [8], for example, based on the turning on/off of lights in a specific place or room, the opening/closing of a door, and the operation history information of appliances and devices in a specific place in the facility, the Groups can be identified.
[9]
In the system according to any one of [1] to [4], the system of the embodiment can detect that a moving object has arrived at a specific location in the information collection system by obtaining information about the individual of the moving object or the group to which the individual belongs. It is a system.
According to the system of [9], for example, the individual or the group to which the individual belongs can be identified based on the fact that the moving object arrives and stops at a location where it can be assumed that the individual is seated in the assigned seat (specific location).
[10]
In the system of the embodiment, in any one of [1] to [4], the information cooperation unit detects a time difference between the sensor system and the information collection system, and collects the time stamp of the moving object information and the electronic information. This system further includes a function to correct the time difference between the information and the time stamp.
According to the system [10], since the time difference between the sensor system and the information collection system is corrected, it becomes easy to identify the target moving object.
[11]
In the system of [4], the system of the embodiment determines that the moving direction of the target moving object is the front direction if the velocity vector information is faster than the threshold value.
According to the system in [11], the moving direction of the target moving object can be determined from the velocity vector information, and if the speed is greater than or equal to a threshold value, it is determined that the moving direction is in the front direction of the target moving object. Therefore, information for determining whether or not it can be used as teaching data can be obtained.
[12]
In the system of [11], the system of the embodiment calculates the shape information change rate at which the shape information becomes a threshold for the orientation of the target moving object at a time after the timestamp of the shape information and velocity vector information when it is determined that the direction is the front direction. If the distance does not exceed , it is determined that the target moving object is maintaining its front direction.
According to the system in [12], if the rate of change in shape information does not exceed a threshold, it is considered that the posture is maintained. By determining whether the orientation of the target moving object is maintained, information for determining whether or not it can be used as teacher data can be obtained.
[13]
In the system of the embodiment, in any one of [1] to [4], the sensor has a built-in calculation unit.
According to the system of [13], it is possible to timely process the determination model determined within the sensor without uploading data to a central server. Thereby, it is possible to quickly determine whether or not to use the data as teacher data.
[14]
In the system of [4], the system of the embodiment forms a flow line by accumulating velocity vector information of a moving object, and determines whether or not to modify the judgment model based on whether or not a specific pattern matches the flow line. decide.
According to the system of [14], it is possible to determine whether or not the movement is characteristic and requires notification to the administrator by forming a movement line from the velocity vector information of one or more sensors. I can do it. It also has the advantage of serving as reference information for extracting training data in the case of abnormal values.
[15]
The system of the embodiment is the system according to any one of [1] to [4], in which the determination model is further based on the truth rate between the predicted class based on past data and the actual class. , determine whether or not it can be used as training data.
According to the system in [15], if the judgment accuracy of the judgment model based on the truth rate in data collected in the past is low, the data is not used as training data. This can be used to improve the decision model. For example, in the case of the action of sitting on a chair, if there is a high rate of misjudgment in which a person is judged to be seated even though the person is not actually seated, then this action can be interpreted as training data. Not hiring. On the other hand, in the case of an action of leaving the seat, the erroneous determination rate of determining that the person is still present even though the person has left the seat is considered to be low, and is therefore used as a candidate for teacher data or test data. In this way, there is an effect that highly accurate training data can be selected and collected.
[16]
In the system of the embodiment, in any one of [1] to [4], the sensor is an image sensor, and each of the image sensors is
an imaging unit that acquires image data;
an extraction unit that extracts moving object information from image data;
a processing unit using a judgment model for image identification;
a determination unit that determines a target moving object based on the moving object information;
Equipped with
[17]
In the system of the embodiment [4], the shape information is calculated by HOG (Histogram of Oriented Gradients) or CoHOG (Co-occurrence Histogram of Oriented Gradients).
According to the system of [17], HOG/CoHOG-based image processing information can be used as shape information.
[18]
In the system of the embodiment, in any one of [1] to [4], the determination model is created based on deep learning using a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network).
According to the system in [18], a decision model for image processing is created using CNN or DNN. Thereby, it is possible to determine with higher accuracy whether or not the data can be used as teaching data.
[19]
In the system of the embodiment, in any of the systems [1] to [4], the electronic information does not have an abnormal value exceeding a predetermined threshold for a certain period of time, or an abnormal value occurs at a specific date and time. By confirming and inputting the information, the information is confirmed as teacher data.
According to the system of [19], as a means for identifying a moving object, for example, an administrator/user checks whether an abnormal value has occurred during or after collecting moving object information. Thereby, the moving object information can be determined as teacher data or test data according to the intention of the administrator or the user.
[20]
The system of the embodiment uses data determined as teacher data in any of the systems [1] to [4] as training data or test data for a determination model.
According to the system of [20], the obtained teacher data can be used as training data for learning the decision model or as test data for evaluating the performance of the decision model. This makes it possible to achieve automation/semi-automation of learning.
[21]
The system of the embodiment includes any one of [1] to [3],
One or more pieces of data determined as training data are used for learning the determination model. Furthermore, it is determined whether or not to save as teacher data together with the evaluation value of the learned judgment model based on the truth rate.
According to the system in [21], after checking whether the judgment model improves when compared using index values, it is registered as training data, including the degree of effectiveness in improving the judgment model. You can decide whether to save it as training data or not. The teacher data stored in this way can be ordered in terms of effectiveness.
[22]
In the system of any one of [1] to [4], the system of the embodiment determines whether or not to change a predetermined process (preprocessing) before processing with the judgment model based on the moving object information and the electronic information. to decide.
According to the system in [22], it is possible to use training data in an actual operational environment to check whether the judgment accuracy improves by changing various parameters of preprocessing, and to achieve local fit preprocessing. . This is because a large part of the accuracy of the determination model depends on the quality of the preprocessing of the collected images. This has the effect of improving the quality of subsequent teacher data and making collection/judgment easier.
Note that preprocessing is an image processing process before inputting to the judgment model, and includes, for example, (1) color (RGB) or monochrome (black and white), (2) image data format (storage format), (3) ) Transparent data processing, (4) Trimming, resizing, (5) Rotation/reversal processing, (6) Tone conversion/color reversal processing, (7) Gamma coefficient adjustment/brightness adjustment, (8) Threshold processing (binarization) ), (9) masking, and (10) noise removal.
[23]
In the system of the embodiment, in any one of [1] to [4], the sensor in the sensor system is an image sensor, and the image sensor uniquely determines an imaging area divided into a plurality of cells. When the target moving object is reflected in at least one cell,
From the shape information and velocity vector information captured in one cell, calculate the amount of movement of the moving object from one cell from the velocity vector information,
While calculating the rate of change in the shape information before and after the movement, it is compared with the allowable rate of change in the shape information determined in advance from the relative position information of one cell in the imaging area and the amount of movement,
Is it possible to use a judgment model as training data that determines that the shape information, including the orientation, posture, clothing, and attachments of the target moving object, has changed significantly when the rate of change in shape information exceeds the allowable rate of shape information? It is included as part of the decision model that determines whether
According to the system of [23], since the image sensor has a relatively wide coverage area, even if the same moving object is photographed at the periphery and at the center, the shape information will be different. For example, in the case of a fisheye lens-type imaging camera placed on the ceiling, the peripheral cells will show the face or back side, and the top of the head will be seen directly below the camera. These three types of shape information are different. Therefore, the amount of change in the shape information of the same moving object varies depending on which cell among the plurality of cells divided by the image sensor is used to capture the image and in which direction and how far the object moves. The shape change rate for the same moving object is determined as a function of relative position information and movement amount of the cell, and is determined in advance as the allowable change rate of the shape information. This makes it possible to determine whether the shape information of the target moving object has changed significantly. For example, it is possible to check whether or not the data can be used as training data that can determine changes such as falls and stupor based on changes in the orientation of the target moving object.

DESCRIPTION OF SYMBOLS 1... Sensor system, 2... Case, 3... Image sensor, 4... Target space, 5... Network, 6... Area, 7... Biometric information sensor, 7a... Face authentication sensor, 8a... Entry/exit gate, 9a... Communication path, 10a to 10f (10)...Image sensor, 11a...Administrator input information acquisition unit, 12a...Identification information acquisition unit, 16...CPU, 20...Communication unit, 20a to 20f (20)...Communication unit, 21a...Communication unit, 22... Communication department, 23... Door, 24... Divided area, 25... Location, 26... Information terminal, 30... Central server, 30a... Camera information, 31... Communication department, 31a... Fisheye lens, 31b... Aperture mechanism, 31c... Image Sensor, 32...Feature amount extraction unit, 32a...Program, 32b...Image data, 32c...Dictionary data, 32d...Determination model, 33...Determination unit, 33a...Feature amount extraction unit, 33b...Moving object determination unit, 33c...Flow line Information calculation unit, 34...Storage unit, 34a...Image data, 34b...Feature amount, 35...Storage unit, 35a...Dictionary data, 36...CPU, 40...Teacher data judgment unit, 41...Information linkage unit, 42...Acceptance/rejection determination Unit, 43...CPU, 44...Teacher data database, 46...Control unit, 300...Register, 310...Camera unit, 320...Memory, 320a...Program, 330...Processor, 350...Internal bus, 37...Existing system DB, 38 ...Personal information DB, 44...Teacher data DB, 45...Notification/destination DB, a to f...Area.

Claims (5)

Multiple image sensors installed in the building,
an analysis unit that analyzes image data captured by the image sensor to obtain moving object information of a target moving object;
a state determining unit that determines the state of the target moving body from the moving body information based on a determination model generated by machine learning using training data ;
an information linking unit that cooperates with the data collection system of the building and associates the collected data collected by the data collection system with the image data based on time information;
an acceptance/rejection determination unit that determines whether or not the moving object information can be adopted as teaching data with correct answers for the machine learning, based on the collected data;
A sensor system comprising: Each of the image sensors includes:
an imaging unit that acquires the image data;
a feature extraction unit that extracts feature information of the target moving object from the image data;
a moving object determination unit that determines the target moving object based on the feature amount information;
a flow line information calculation unit that calculates flow line information of the target moving object;
The sensor system according to claim 1, further comprising: a transmitting section that transmits the flow line information to the state determining section. Each of the image sensors includes:
further comprising a storage unit that stores the determination model,
The sensor system according to claim 2, wherein the moving object determination unit determines the state of the target moving object from the flow line information based on the determination model. The acceptance/rejection determination unit calculates a performance evaluation index value of the determination model, and determines whether or not the moving object information can be adopted as training data for the machine learning based on the performance evaluation index value. The sensor system according to item 1. A sensing method applied to a sensor system equipped with a plurality of image sensors installed in a building, the method comprising:
a process in which a computer analyzes image data captured by each image sensor to obtain moving object information of a target moving object;
a step in which the image sensor determines the state of the target moving body from the moving body information based on a determination model generated by machine learning using teacher data;
a step in which the computer cooperates with the data collection system of the building and associates the collected data collected by the data collection system with the image data based on time information;
A sensing method including a step in which the computer determines whether or not the moving body information can be employed as teaching data with correct answers for the machine learning, based on the collected data.