WO2015189893A1

WO2015189893A1 - Individual identification device, and individual identification method

Info

Publication number: WO2015189893A1
Application number: PCT/JP2014/065232
Authority: WO
Inventors: 健河本
Original assignee: 株式会社日立製作所
Priority date: 2014-06-09
Filing date: 2014-06-09
Publication date: 2015-12-17

Abstract

An individual identification device (1) comprises: an intra-image movement extraction unit (24) which, for each person-region extracted from video data from a camera (2), extracts an intra-image movement quantity graph (12c) representing movement, within an image, of the person-region; a sensing correction unit (23) which, on the basis of a camera distance graph (13b), corrects sensing data representing the movement of each person as measured from sensor terminals (1) held by each person; and an individual identification unit (26) which, on the basis of the degree of similarity between the extracted intra-image movement quantity graphs (12c) and the corrected sensing data, maps the person-region in the video data with a holder of the sensor terminal (1).

Description

Individual identification device and individual identification method

The present invention relates to an individual identification device and an individual identification method.

The process of extracting an area representing a person from a captured image of a fixed point observation camera and distinguishing the person area from another person area is referred to as a “recognition” process. In this recognition processing, the movement of the time-series human area can be tracked, so that, for example, it can be seen that a person shown on the left side of the screen at 10:10 moved to the right side of the screen at 10:20.

Further, the process of associating the human areas individually extracted by the recognition process with the personal information registered in advance is called “identification” process. By this identification processing, for example, the person shown on the left side of the screen at 10:10 can be identified as Mr. A with a membership number of 1234. That is, by performing the recognition process and the identification process in order, it is possible to know who is in the photographed image of the camera and what kind of movement it is.

First, there is a method in which identification processing is performed only with a photographed image of the camera. A method for extracting an area representing a person from an image and identifying which image area corresponds to which ID from the face information and the appearance information associated with each ID and the similarity of the extracted image area is there. The advantage of this method is that a single camera can be used for wide-ranging identification, and facilities such as factories, warehouses, and stations have long been equipped with fixed-point monitoring cameras, so they can be introduced at low cost. is there.

However, it is difficult to accurately identify a person in an environment where many people move around in length and breadth, because the person cannot be accurately identified unless the person's face or figure is clearly reflected. In addition, in order to perform identification, it is necessary to hold information on the face and the appearance of the ID from the beginning. However, it took a lot of time to acquire such information, and there were not a few people who felt resistance to the fact that their own information about their face and appearance was held in the database, and it was difficult for them to be accepted by society.

Therefore, in Patent Document 1, as an example of having the subject cooperate with the identification process, the similarity between the motion information measured from the acceleration sensor held by each infant and the motion information of the human area extracted from the captured image of the camera An identification process for associating an infant in an image with an ID based on a combination with a high is described.

JP 2004-96501 A

In a conventional identification processing system that performs matching of motion information such as Patent Document 1, an error occurs in motion information extracted from a captured image of the camera due to the capturing characteristics of the camera. For example, a subject with a short distance from the camera installation position to the subject position (camera distance) appears large in the photographed image, so even a minute movement of the subject is reflected in the photographed image.
On the other hand, since a subject with a long camera distance appears small in the captured image, the minute movement of the subject may not be detected as an image.

Therefore, regardless of the camera distance, the motion information from the acceleration sensor that can detect a fine motion and the motion information that cannot detect a fine motion from a captured image of a subject with a long camera distance, It may be erroneously determined that the motion information is not similar.

In order to make the camera distance the same distance regardless of the current position of the subject, for example, a mode in which the camera is installed on a very high ceiling (overhead position) and an overhead image from the ceiling is used. Conceivable. However, in many facilities such as stations and stores, the ceiling is not so high, and it is difficult to secure an ideal shooting location.

Accordingly, the main object of the present invention is to improve the accuracy of individual identification in a system that performs individual identification processing based on motion information from a sensor and motion information from a captured image, regardless of the position and orientation of the camera. And

In order to solve the above-described problem, the individual identification device of the present invention includes:
For each human region extracted from the video data of the camera, an intra-image movement extraction unit that extracts intra-image movement amount data indicating movement in the image of the human region;
A sensing correction unit that corrects sensing data indicating movement of each person measured from a sensor terminal held by each person based on a camera distance from the installation position of the camera to the position of the person shown as the person area When,
Based on the similarity between the extracted movement amount data in the image and the corrected sensing data, the individual identification unit that associates the human region in the video data with the gripper of the sensor terminal, It is characterized by having.
Other means will be described later.

According to the present invention, in a system that performs individual identification processing based on motion information from a sensor and motion information from a captured image, the accuracy of individual identification can be improved regardless of the position and orientation of the camera.

It is the block diagram (FIG. 1 (a)) which shows the individual identification system regarding one Embodiment of this invention, and the block diagram (FIG.1 (b)) which shows a sensor terminal. It is a block diagram which shows the individual identification apparatus regarding one Embodiment of this invention. Explanatory drawing (FIG. 3 (a)) which shows the relationship between the camera and space position regarding one Embodiment of this invention, the image | video data storage part which stores the image | video image | photographed with the camera of Fig.3 (a), and its image | video It is explanatory drawing (FIG.3 (b)) which shows the depth map storage part which stores the depth map allocated to. A sensing table (FIG. 4A), a human area table (FIG. 4B), an individual similarity table (FIG. 4C), and an individual identification table (FIG. and d)). It is a flowchart which shows the main process (FIG.5 (a)) performed by the individual identification apparatus regarding one Embodiment of this invention, and the person area extraction process (FIG.5 (b)) performed by a person area extraction part. . FIG. 6 relating to an embodiment of the present invention is a flowchart showing similarity evaluation processing executed by a similarity evaluation unit or the like. Sensing graph before correction (FIG. 7 (a)), camera distance graph (FIG. 7 (b)), sensing graph before correction (FIG. 7 (c)), and intra-image movement amount graph according to an embodiment of the present invention It is explanatory drawing which shows (FIG.7 (d)). FIG. 8 shows a first stage (FIG. 8A), a second stage (FIG. 8B), and a third stage (FIG. 8C) of the automatic generation processing of the depth map storage unit according to the embodiment of the present invention. It is explanatory drawing. FIG. 9 relating to an embodiment of the present invention is an explanatory diagram showing a worker management screen. FIG. 10A is an explanatory diagram illustrating an example in which a directional display is used as a display device according to an embodiment of the present invention, and FIG. 10A is a screen diagram illustrating display contents visible to each person from the display device of FIG. (FIG. 10B). An explanatory view (FIG. 11 (a)) showing a state of shooting from the camera on the ceiling relating to the first comparative example, an explanatory view (FIG. 11 (b)) showing an image from the camera of FIG. 11 (a), and FIG. It is explanatory drawing (FIG.11 (c)) which shows the individual identification result based on the image | video of (b). Explanatory drawing (FIG. 12 (a)) which shows a mode that it image | photographs from the camera turned to the horizontal direction regarding a 2nd comparative example, and explanatory drawing (FIG.12 (b)) which shows the image | video from the camera of Fig.12 (a). FIG. 13 is an explanatory diagram (FIG. 12C) showing the individual identification result based on the video of FIG.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1A is a configuration diagram showing an individual identification system. The individual identification system is configured by connecting a sensor terminal 1, a camera 2, a base station 8, and an individual identification device 9 via a network.
The sensor terminal 1 is attached to the subject's arm or the like, and detects the subject's operation and state such as acceleration of the arm as biological information. The sensor terminal 1 wirelessly transmits the acceleration data detected at a predetermined cycle as sensing data to the base station 8 via the antenna.
The base station 8 transfers the received sensing data to the individual identification device 9. Note that the sensing data communication path may be transferred directly to the individual identification device 9 via a USB (Universal Serial Bus) connection or the like (without going through the base station 8) if wired communication is possible. Furthermore, the sensing data may be transmitted sequentially from the sensor terminal 1, or the sensing data accumulated in the sensor terminal 1 may be transmitted every predetermined period.

The camera 2 is a video camera that takes an image of the subject. The captured video data is transmitted to the individual identification device 9 through wireless communication or wired communication. Note that the video data may be transmitted sequentially, or may be transmitted at predetermined intervals after being stored in the video server. In the following, an example in which the camera 2 is installed at a fixed point is shown, but for example, the camera 2 may be a handy camera carried by a person or a wearable camera.

The individual identification device 9 receives sensing data corresponding to the movement of the target person from each sensor terminal 1 attached to each person, receives video data captured from each camera 2, and also stores the received data. In addition, the sensor terminal 1 that the person photographed in the video data is individually identified, and the result is displayed on the display device 91.

The individual identification device 9 is connected to a display device 91 that displays various information and an input device 92 that allows various information to be input by a user's operation via an I / F (Interface) 96 (or individual identification). (Built in the device 9).
Although the display device 91 is configured as a display terminal such as a liquid crystal display or a CRT display, an output device such as a printer device or an image file output device may be used instead of the display device 91.
The input device 92 is an input device such as a keyboard and a mouse. The display device 91 and the input device 92 may be a single device having both functions, such as a touch panel display.

The individual identification device 9 is configured as a computer including a processor 93, a memory 94, and a recording device 95. The recording device 95 records various programs for realizing each processing unit to be described later in FIG. 2 and various data handled by the various programs, such as a hard disk drive, a CD-ROM drive, and a flash memory. . Various programs and various data tables may be divided and recorded in a plurality of recording devices 95.
The processor 93 implements various functions by reading various programs recorded in the recording device 95 into the memory 94 and executing them.

FIG. 1B is a configuration diagram (a schematic view seen from the front and a cross-sectional view seen from the side) showing a bracelet type sensor node as an example of the sensor terminal 1. Such bracelet-type sensor nodes are described, for example, in the literature “Tanaka T, et al.,“ Life Microscope: ContinuousCondaily-activity recording system with tiny wireless sensor ”, 5th International Conferene on Networked Sensing Systems, 2008". .

The sensor terminal 1 includes a case 11 for storing a sensor and a control device, and a band 12 for attaching the case 11 to an arm of a human body.
Inside the case 11 is stored a substrate 10 provided with a microcomputer, a sensor 6 and the like. The sensor 6 is, for example, an acceleration sensor that measures three-axis accelerations of XYZ, and measures the movement of a human body (living body) that is worn. The sensor 6 may include a pressure-sensitive sensor or a capacitance sensor, and may output whether or not the user is wearing the sensor terminal 1 as a wearing state.

As another example of the sensor terminal 1, any sensor terminal 1 may be used as long as it can measure a user's movement. For example, a wearable camera with a built-in acceleration sensor or a mobile device such as a smartphone with a built-in acceleration sensor may be used. The acquired sensing data need not be acceleration data, and suggests user activities such as angular velocity sensor data, geomagnetic data, audio data, video data, radio wave intensity data, device operation history data, and Doppler sensor data. Any data is acceptable.

It should be noted that the sensor terminal 1 that has only a sensor function for measuring the user's movement can be introduced at a lower cost than using a self-position estimation terminal such as three-point positioning. First, since the operation of the self-position estimation terminal needs to receive a radio wave of a beacon separately installed in a building, the accuracy is lowered depending on the environment due to a decrease in radio wave intensity due to shielding. In order to improve accuracy, it is necessary to increase the number of beacons, but it takes a lot of cost to install a large number of beacons in the environment, and to provide power supply and network connection infrastructure.

FIG. 2 is a configuration diagram showing the individual identification device 9. The individual identification device 9 includes a sensing table 11a, a pre-correction sensing graph 11b storage unit, a post-correction sensing graph 11c storage unit, a video data storage unit 12a, a human region table 12b, and an intra-image movement amount graph 12c. Storage unit, depth map storage unit 13a, camera distance graph 13b storage unit, individual similarity table 17, individual identification table 18, scalarization processing unit 20, human region extraction unit 21, camera distance An extraction unit 22, a sensing correction unit 23, an in-image movement extraction unit 24, a similarity evaluation unit 25, and an individual identification unit 26 are included.
Hereinafter, an outline of each component of the individual identification device 9 will be described. Details of these components will be clarified in the description after FIG.

In the sensing table 11a, sensing data received from the sensor terminal 1 and a sensor ID for identifying the sensor terminal 1 are stored in association with each other. In addition, although illustration was abbreviate | omitted, by associating the information (for example, access authority information to a room, the balance of a fare prepaid card, etc.) regarding the subject with the subject ID identified from the sensor ID, Information about the subject who has been individually identified can also be displayed on the display device 91.

The scalarization processing unit 20 converts the sensing data of the sensing table 11a into a scalar, thereby converting it into a user motion amount having a scalar value that does not depend on the orientation of the sensor terminal 1 that changes with the passage of time. It outputs to the sensing graph 11b before correction | amendment.
The sensing correction unit 23 changes from the pre-correction sensing graph 11b to the post-correction sensing graph 11c based on the camera distance described in the camera distance graph 13b (the distance from the camera 2 in the human area in the video at each time point). to correct.

The video data storage unit 12 a stores video data of the camera 2.
The human area extraction unit 21 reads the video data in the video data storage unit 12a, assigns a different ID to each human area recognized from the video data at each time point, and stores the assignment result in the human area table 12b. .
The intra-image movement extraction unit 24 extracts the movement amount in each image constituting the video data as a scalar value for each human region in the human region table 12b, and stores the extraction result in the intra-image movement amount graph 12c. To do.

The depth map storage unit 13a stores a depth map input by an administrator or the like. The depth map is data corresponding to the position in the image and the camera distance corresponding to the position, and details will be described later with reference to FIG.
The camera distance extraction unit 22 refers to the depth map of the depth map storage unit 13a, specifies the camera distance of the person area from the position in the image of the person area for each person area in the person area table 12b, The specified result is output to the camera distance graph 13b.
Alternatively, the camera distance extraction unit 22 does not refer to the depth map, for example, a predetermined function that makes the camera distance closer as the area of the human area is larger (the area of the human area is inversely proportional to the square of the camera distance. The camera distance may be specified from the human region in the image based on the function to be performed.
The similarity evaluation unit 25 evaluates the similarity of both graphs by comparing the corrected sensing graph 11c, which is a time-series scalar value, with the intra-image movement amount graph 12c.
The individual identification unit 26 associates a combination having a high similarity (a combination of sensing data and in-image human area data) as an evaluation result of the similarity evaluation unit 25 and outputs the result to the individual identification table 18.

FIG. 3A is an explanatory diagram showing the relationship between the camera 2 and the spatial position. Since a worker exists on the right side of the passage in the distribution warehouse and is within the camera field of view of the camera 2, the worker is shown as a human area in the photographing data. The worker's spatial position (X, Y, Z) and straight line vector will be described later.
The camera 2 may image not only a warehouse but also any facility or environment where people come and go. For example, by capturing an image of a traffic system (station, platform, train, etc.) and causing the individual identification device 9 to individually identify each passenger shown as the person's area, data relating to each passenger (account balance, etc.) can be specified. it can.
Therefore, it is possible to construct a complete walk-through type traffic system in which the toll is automatically deducted from the balance, and a person who cannot be individually identified can be found as a passenger who has no ticket.

FIG. 3B illustrates a video data storage unit 12a that stores video captured by the camera 2 in FIG. 3A and a depth map storage unit 13a that stores a depth map assigned in the video. FIG.
In the video data in the video data storage unit 12a, a worker who exists at the spatial position (X, Y, Z) in FIG. 3A is photographed in the upper right of the image.

The depth map in the depth map storage unit 13a is data indicating the distance from each point on the space represented by each pixel of the captured data to the mapping element of the camera 2 for each camera 2.
For example, the lower side of the image shows a floor at a position close to the camera, so a close distance (for example, 3 m) is stored. The center side of the image indicates a slightly far distance (for example, 5 m), and the upper side of the image indicates a far distance (for example, 10 m).

If the environment surrounding the camera 2 changes (for example, when a new pillar is added in the warehouse or the position where the camera 2 faces changes), the depth map also needs to be updated. The means for updating the depth map may be manually input by the system administrator only once for each camera 2 (or whenever the environment changes significantly), or in information such as a sensor (for example, a laser distance sensor). May be constructed by manually inputting the angle of the camera 2 with respect to the floor, or by measuring the angle of the camera 2 with respect to the vertical based on information such as a sensor (for example, an acceleration sensor). May be built.
The update timing of the depth map may be explicitly performed by the system administrator when the environment changes, or may be automatically updated, for example, at a predetermined cycle (every day or the like).

The straight line vector in FIG. 3A is a vector indicating the direction of the worker's spatial position (X, Y, Z) as seen from the position of the camera 2. This straight line vector uses the camera's warehouse position and orientation information and information about the camera's optical system (specifically, the camera's specification data such as the lens distortion model and angle of view). It can be obtained for each internal position (x, y). In addition, what is necessary is just to let a user input manually the input data for calculating | requiring a linear vector at the time of environment construction, for example.
Then, the individual identification device 9 uses the linear vector of the worker as the camera distance indicated by the depth map at the position (x, y) in the image starting from the position of the camera 2 (zero distance) (for example, in FIG. 3B). By tracing only 10 m), the worker's spatial position (X, Y, Z) can be obtained. Therefore, the individual identification device 9 can identify which worker is present at which spatial position (X, Y, Z) in the warehouse.

FIG. 4A is a configuration diagram showing the sensing table 11a. In the sensing table 11a, for each sensor ID that is an identifier of the sensor terminal 1, acceleration data acquired by the sensor terminal 1 is stored in association with the acquisition time.
In the present embodiment, since the sensor terminal 1 is distributed one by one for each individual identification target person (such as a warehouse worker), the sensor ID may be used as an ID for identifying the target person. Function. Alternatively, when the sensor ID and the target person ID do not correspond 1: 1, if the data for associating both IDs is prepared separately, the target person ID can be uniquely identified from the sensor ID.

The acceleration data is data consisting of three axes (x, y, z). The sensor terminal 1 may be a sensor that detects only the motion (relative acceleration value) of the subject from which the gravitational acceleration component has been excluded in advance, or the motion including the motion of the subject and the gravitational acceleration component. A sensor may be used. When the latter sensor is used, in order to extract only the movement of the subject necessary for the individual identification process, the process of removing the gravitational acceleration component from the detected value of the sensor is performed as a pre-process as shown below.
The sensor terminal 1 approximates each vertical acceleration component by calculating a moving average with respect to (x, y, z) including the gravitational acceleration component, and subtracts the average value from the (x, y, z) acceleration. Thus, the relative acceleration values (dx, dy, dz) in the (x, y, z) axial directions are calculated.

FIG. 4B is a configuration diagram showing the human area table 12b. The human area table 12b includes, for each camera ID that is an identifier of the camera 2, a human area ID assigned to each human area extracted from the captured video of the camera 2, and a detection time of the human area ID (the detection source video is The time when the image was taken) and the position in the image (center coordinate x, center coordinate y) where the person area of the person area ID exists are stored in association with each other. Instead of the center point (tummy position) of the human area, the lowest coordinate point (toe position) of the human area may be used as the position of the human area.
Here, the person area ID is information that can not uniquely identify an individual alone and recognizes only whether the individual person shown in the image is the same person or a different person.

FIG. 4C is a configuration diagram showing the individual similarity table 17. The individual similarity table 17 stores the similarity of the combination of the sensor ID of the sensing table 11a and the human area ID of the human area table 12b, and the latest time of the period in which the similarity between the two is calculated in association with each other. Is done.

FIG. 4D is a configuration diagram showing the individual identification table 18. The result of associating the sensor ID selected based on the similarity in the individual similarity table 17 with the human region ID and the “time” in the individual similarity table 17 are stored in association with each other.
For example, in the individual similarity table 17, there are three human area ID candidates (3-004, 3-005, 3-006) corresponding to the sensor ID “0x4014”. The human region ID (3-004) having the highest similarity among the two candidates is associated as a result of individual identification. In other words, at the time “2014-03-28 15: 42: 01.191”, the person shown as the human area (3-004) at the coordinates (45,130) in the image has the sensor ID “0x4014”. Identified as an employee.

The individual identification unit 26 may acquire all combinations of sensor IDs and human area IDs from the individual similarity table 17 or only combinations that have changed from the previous individual identification process. Also good.
Then, the individual identification unit 26 refers to the similarity for each acquired combination, arranges the items in descending order of similarity, and discards combinations whose similarity is equal to or less than a threshold (for example, 0.5). Increase identification accuracy.
Furthermore, the individual identification unit 26 adopts combinations that have not yet adopted the sensor ID and the human area ID in descending order of similarity, and writes the adoption results to the individual identification table 18. Thereby, all the similarity of the combination of sensor ID and person area ID is relatively high, and the combination without a cover can be selected. As another implementation method, combination optimization may be performed using an algorithm such as Simulated Annealing so that the average similarity of the selected combination is maximized.
Here, the individual identification unit 26 stores the sensor ID as an abnormal value (for example, “null”) in the individual identification table 18 for a human region where the corresponding sensor could not be found. This “null” person area ID is also useful information because it may be a suspicious person who is not registered as a member, for example.

FIG. 5A is a flowchart showing main processing executed by the individual identification device 9. The execution timing of this flowchart may be, for example, every predetermined period, when communication with the sensor terminal 1 occurs, when input occurs via the input device 92, or when the individual identification device 9 Or when the execution of another application running on another device is completed.
Furthermore, each processing step of this flowchart or each flowchart described later may be executed due to the end of the previous step, or may be executed every predetermined cycle (for example, 1 second).

In the data acquisition process of S11, the individual identification device 9 stores the sensing data received from the sensor terminal 1 in the sensing table 11a, and stores the video data received from the camera 2 in the video data storage unit 12a.
In the human region extraction process of S12, the human region extraction unit 21 reads the video data in the video data storage unit 12a, assigns a different ID to each human region recognized from the video data at each time point, and displays the assignment result. Store in the human area table 12b.
In the similarity calculation process of S13, the similarity evaluation unit 25 calculates the similarity between the motion information of the sensing data stored in S11 and the motion information of the human area stored in S12.
In the individual identification process of S14, the individual identification unit 26 associates the combinations having high similarity in S13 with each other and outputs the result to the individual identification table 18. The process of S14 may be executed immediately after the execution of S13 is completed, or may be executed every arbitrary time (for example, 5 minutes).
In the result display process of S15, the individual identification device 9 causes the display device 91 to display the contents of the individual identification table 18. At this time, in addition to the result of individual identification (who is shown in which image), supplementary information (such as the name of the person shown) of each individual identified may be displayed together (for details) , Later in FIG. 9).

FIG. 5B is a flowchart showing details of the human area extraction process (S12) executed by the human area extraction unit 21. The human area extraction unit 21 reads the image captured from the specific camera 2 at a specific time from the video data storage unit 12a, and then executes the following processing.

In S101, the human region extraction unit 21 reads an image at a specific time corresponding to a specific camera ID from the video data storage unit 12a. The camera ID here may be, for example, the lowest numbered identifier in which an unprocessed image exists. Further, the time here may be the oldest unprocessed image from the images acquired from the camera. Furthermore, the previous image taken with the same camera as the read image is read simultaneously.

In S102, the human region extraction unit 21 calculates a difference image between the image read in S101 and the previous image in time series. For example, for each pixel of both images, the difference image calculation processing is performed by first converting the color information (R, G, B) originally represented by the intensity of the three primary colors into a scalar value v, and then using the same coordinates of both images. A method for calculating the absolute difference (| v1−v2 |) between the two calculated scalar values (v1, v2) is mentioned.

There are various ways to convert to a scalar value, but it may be converted to gray scale based on an equation such as “v = 0.299 * R + 0.587 * G + 0.144 * B”. Any conversion formula can be used as long as the structure of the formed shape is not easily lost. The difference image calculated by the processing described above evaluates the difference from the image at the previous time, and an image representing the amount of change of each pixel at the time is calculated. Is shown at a higher value.
In addition, the difference image calculation process does not take a difference from the previous image, but may take a difference from a background image separately prepared in advance, for example, for example, the past 10 seconds (an arbitrary period such as) You may take the difference with the average of all the images image | photographed between.

In S103, the human region extraction unit 21 binarizes depending on whether each pixel of the difference image calculated in S102 exceeds a threshold value, and converts it into an image including only 1 and 0 as values. At this time, the threshold value may be arbitrarily set (for example, 0.2), for example, an average value of each pixel of the difference image calculated over a long term (such as one day) is calculated, and the average value is set as the threshold value. May be set as By this step, a binary image is generated in which a moving object such as a person is displayed as 1 and otherwise is displayed as 0. Note that the human region extraction unit 21 may perform noise removal as the pre-processing or post-processing of S103.

In S104, the human region extraction unit 21 extracts human regions each representing a moving object from the binary image calculated in S103. In the human region extraction process, for example, when pixels having a value of “1” are adjacent to each other, a path that is included in the same object and passes only a pixel of “1” between pixels having a value of “1” Is a process of grouping “1” pixels so that they belong to different objects. By this processing, a plurality of moving bodies (for example, humans) reflected in the screen are identified as different pixel areas. Here, the coordinate information of the center regarding the region, the coordinate points of the upper left corner and the lower right corner, and the coordinate points of all pixels constituting the region are held.
Note that the human region extraction unit 21 may extract objects using, for example, a recognizer of Haar-like features, instead of the human region extraction processing exemplified in S101 to S104.

In S105, the human area extraction unit 21 assigns a human area ID to each object area extracted in S104. Here, for example, for each person area extracted in the time unit immediately before the camera, an area having the largest overlapping area among the person areas extracted this time is assigned, and the newly extracted area is The area ID of the allocated old person area is adopted.

By S105, it is expected that the same person can be continuously given the same ID over a continuous time. In addition, a new ID (for example, an ID that has never been used retroactively in the past) is assigned to a new person area that is not associated with the old person area (that is, a person area that represents a person photographed for the first time at this time). Issue. Although a simple implementation example of ID management has been described above, a method (Particle Filter or the like) that is resistant to disturbances such as object occlusion may be used.
In S106, the human area extraction unit 21 stores the human area assigned with the area ID in S105 in the human area table 12b.

FIG. 6 is a flowchart showing details of the similarity calculation process (S13) executed by the similarity evaluation unit 25 and the like. This process may be executed, for example, at the timing when the human region extraction process (FIG. 5B) relating to a certain time is completed and sensing data up to that time is collected from the sensor terminal 1. It may be executed every time (for example, 5 minutes).

The purpose of the processing in FIG. 6 is to compare the sensor data acquired by a certain sensor with the operation of a certain human area and evaluate the similarity, so that the person who reflects the human area has the sensor. It is to measure the degree of possibility. For that purpose, the similarity (likelihood) when the person region is assumed to have the sensor is quantified.

In S201 to S207, the similarity evaluation unit 25 executes a loop that sequentially selects so as to cover combinations of the sensor IDs included in the sensing table 11a and the human region IDs included in the human region extraction unit 21. Hereinafter, the combination selected in this loop is referred to as an “evaluation pair”.
In S202, the similarity evaluation unit 25 reads a record related to the sensor ID of the evaluation pair from the sensing table 11a, and reads a record related to the human area ID of the evaluation pair from the human area table 12b. The similarity evaluation unit 25 may read records of evaluation pairs by referring to time information and narrowing down to data within a predetermined period in the past (for example, the past 10 minutes).

In S203, the similarity evaluation unit 25 acquires the corrected sensing graph 11c from the sensing correction unit 23.
Therefore, first, the scalarization processing unit 20 converts the acceleration data (dx, dy, dz) of the sensing table 11a read in S202 into a scalar value, whereby the sensing graph before correction in FIG. 11b is obtained. The conversion formula of the scalar value is, for example, the square root of (square of dx + square of dy + square of dz).
The pre-correction sensing graph 11b shows a time-series motion amount (acceleration) for each user of the sensor terminal 1. For example, it can be seen that the user moves small (in small increments) in the first period of FIG. 7A and moves greatly in the second period.
From the sensing graph 11b before correction, it can be seen that the user repeats the activity and the stop, and the large vibration is repeated at a high cycle during the activity. For example, since a periodic vertical movement is performed when a person walks, a periodic movement is observed from acceleration data.

Next, the camera distance extraction unit 22 refers to the depth map of the depth map storage unit 13a, and for each human region in the human region table 12b read in S202, the human region is determined from the position in the image of the human region. The camera distance is specified, and the specified result is output to the camera distance graph 13b in FIG.
In FIG. 7B, the user starts from about 5 m from the camera first, moves to about 13 m in the first period, and stops still. After that, the user returns to the place of about 3 m, stays at the place of about 3 m in the second period after standing still, and is active.

The sensing correction unit 23 then converts the pre-correction sensing graph 11b to the post-correction sensing graph 11c based on the camera distance described in the camera distance graph 13b (the distance from the camera 2 of the human area in the video at each time point). To correct.
The correction process is, for example, a process that lowers the passband of the low-pass filter as the camera distance is longer (in other words, strongly corrects so as to remove more high-frequency components). As a method for realizing the low-pass filter, there is a method using a moving average. For example, when the distance is 0 m, the window width is averaged at 0 seconds, and when the distance is more than an arbitrary distance (for example, 20 m), the window width is set to an arbitrary width (for example, 1 second) at a certain time The corrected acceleration data is set to the average of the acceleration data for the window width before and after the time.

For example, as shown in FIG. 7 (c), when the camera is moved away from the camera as in the first period, correction is strongly applied (p corrected acceleration is obtained by a moving average of wide window widths from p1 to p2). The high frequency components (small movements) that existed in the first period of FIG. 7A are removed by correction. On the other hand, when the distance is close to the camera, there is little correction on the acceleration data (the correction acceleration of q is obtained by a moving average with a narrow window width from q1 to q2), and therefore exists in the second period of FIG. The high frequency components (minor movements) that have been retained remain to some extent even in FIG.

The intent of the sensing correction unit 23 is to reproduce the phenomenon that “the movement appears dull when moving away from the camera” in the sensing data. The low-pass filter can reproduce the above phenomenon with a relatively high accuracy even though the amount of calculation is small, but it may reproduce how the acceleration appears as an image more precisely by performing a stricter correction.
The process of S203 has been described above.

As S <b> 204, the similarity evaluation unit 25 acquires the intra-image movement amount graph 12 c from the intra-image movement extraction unit 24.
Therefore, the intra-image movement extraction unit 24 quantifies the operation of the human area on the image for each human area stored in the human area table 12b read in S202. Assuming that the position in the image of the human area at a certain time is a coordinate point (x, y), the acceleration (accX, accY) on the image can be obtained by performing differentiation twice for each axis. Then, the intra-image movement extracting unit 24 calculates the square root of (square of accX + square of accY), and the result is taken as the vertical axis of the intra-image movement amount graph 12c of FIG. ).
Note that, as described with reference to FIG. 12, the motion on the video looks different when the same motion is performed near the camera and when the same motion is performed far from the camera. That is, in the intra-image movement amount graph 12c, rather than “how the user actually moved”, “how the user appeared to move in the video of the camera” is shown.

As S205, the similarity evaluation unit 25 calculates the similarity between the corrected acceleration of the corrected sensing graph 11c acquired in S203 and the human area movement acceleration of the intra-image movement amount graph 12c acquired in S204 (similarity of evaluation pairs). To evaluate. For this purpose, the similarity evaluation unit 25 simply calculates a correlation coefficient between the corrected acceleration and the human area movement acceleration at each time, or a coefficient equivalent thereto. For example, when the correlation coefficient is calculated, a value close to 1 is obtained if both data (the shape of the graph) are similar.
In consideration of the time difference of the clock between the sensor terminal 1 and the camera 2, the time difference of the corrected sensing graph 11c or the intra-image movement amount graph 12c is slid to the left or right to correct the time difference. The degree of similarity may be evaluated.
As S206, the similarity evaluation unit 25 stores the similarity of the evaluation pair calculated in S205 in the individual similarity table 17.

Hereinafter, the automatic generation process of the depth map storage unit 13a will be described with reference to FIG.
In the first stage shown in FIG. 8A, the initial state starts from a depth map in which all depths are fixed (for example, 3 m). From the corrected sensing graph 11c corrected based on the depth map storage unit 13a, the person A who is actually at a camera distance of 3 m can be found (successfully identified), but at a position of 5 m. The person B and the person C at a position of 10 m cannot be found because the correction amount is not appropriate.
In FIG. 8, a portion that is not updated as the initial value in the depth map storage unit 13a is surrounded by a dashed ellipse, and an updated portion is surrounded by a solid ellipse.

FIG. 8B is an explanatory diagram showing a second stage of the automatic generation process of the depth map storage unit 13a.
The individual identification device 9 tracks the person A found in FIG. 8A, the acceleration data of the person A (sensing graph 11b before correction), and the apparent acceleration in the image of the person A (intra-image movement amount graph). 12c is regarded as the corrected sensing graph 11c), and the camera distance graph 13b is estimated by inverse calculation.
That is, the sensing correction unit 23 corrects the pre-correction sensing graph 11b to the post-correction sensing graph 11c with the correction amount based on the camera distance graph 13b, but the pre-correction sensing graph 11b and the post-correction sensing graph 11c are input. Then, the correction amount based on the camera distance graph 13b can be calculated backward.
As a result, since the camera distance of the person A who has moved to the upper side than FIG. 8A is found to be 5 m, the initial value (3 m) of the depth map storage unit 13 a is updated at 5 m.

FIG. 8C is an explanatory diagram showing a third stage of the automatic generation process of the depth map storage unit 13a. The depth map is gradually created by repeating the same process as in FIG. 8B and following the trajectory of the person A (3 m → 5 m → 10 m). Note that the individual identification device 9 can follow the trajectory of the person B by changing the initial value from 3 m to 5 m after following the trajectory of the person A.

FIG. 9 is an explanatory diagram showing a worker management screen 1200 displayed on the display device 91. The worker management screen 1200 is displayed, for example, when a display request from a user is received via the input device 92. Note that a browser may be adopted as an application that runs on the individual identification device 9, or an application that runs alone may directly display the worker management screen 1200.

This worker management screen 1200 can grasp the work progress of each worker if the worker who moves in the warehouse is known. On the other hand, when the person who could not be recognized (that is, walking in the warehouse but there is no record of the ID on the server) or when it is written that the ID should not be in that place If it can be discovered, it can be used for security.

The worker management screen 1200 is a screen that displays the position of each worker currently in the warehouse and the state of the worker selected on the screen, and a map area 1201 that displays a map in the warehouse and the position of the worker. And a worker information area 1202 for displaying detailed information on the selected worker, and a video display area 1203 for displaying the current video for the selected worker.

The map area 1201 is an area for displaying a shelf or an obstacle in the target warehouse, the spatial position (X, Y, Z) of each worker described in FIG. The information on shelves and obstacles may be displayed by inputting a design document and CAD (Computer Aided Design) data at the time of warehouse design, or may be manually input by a user.
In the vicinity of the icon indicating each worker, the sensor ID (for example, “0x2480”) of the sensor terminal 1 held by the worker is also displayed. Further, from the icon indicating each worker, the locus of the position in the past arbitrary period (for example, 5 minutes) is displayed as a line. In this way, by displaying the map area 1201, how many people the user (for example, warehouse manager) concentrates on which area in the warehouse, and each worker is moving efficiently, You can see at a glance.

In addition, when a specific worker (for example, “0x2486”) is selected via the input device 92 in the worker information area 1202, the current state (work progress status or current state) of the selected worker is selected. (Location) may be displayed.
As for the progress status of the work, the data of the warehouse information management system associated with each worker in advance may be referred to, or the progress may be calculated according to the distance to the specific destination. The current location may be displayed as coordinates, or the nearest shelf number may be displayed so that the user can easily see at a glance. By displaying the worker information area 1202, detailed information can be presented to the user.

Further, in the video display area 1203, the latest image in the video data storage unit 12a for the selected worker (for example, “0x2486”) is displayed. This latest image can be acquired by the following procedure.
(Procedure 1) The sensor ID is specified from the worker ID.
(Procedure 2) Referring to the individual identification table 18, a human region ID is acquired from the sensor ID.
(Procedure 3) With reference to the person area table 12b, the camera ID of the camera that has captured the person area and its time are specified from the person area ID.
(Procedure 4) Video data at the specified camera ID and time is acquired from the video data storage unit 12a.
(Procedure 5) Based on the center coordinates of the human area table 12b, the area indicated by the human area and its periphery (for example, up to twice the area of the human area) are cut out from the acquired video data.
Thereby, the operation | work which looks for a desired person from the surveillance camera video group which was difficult until now by this can be implement | achieved easily.
In addition, the individual identification device 9 prepares in advance information on a movement permission range indicating a spatial position where entry is permitted for each worker ID and a spatial position where the entry is not permitted, and work outside the movement permission range. When an employee is detected, a warning screen may be displayed on the screen, or an email may be automatically transmitted to the administrator.

FIG. 10A is an explanatory diagram illustrating an example in which a directional display is used as the display device 91. This directional display is installed as a public display and displays different contents depending on the viewing location. The directional display is configured, for example, as a display that shows different things for each place using any of shutter glasses, lenticular lenses, and polarization displays described in Japanese Patent Application Laid-Open No. 2014-41355.
As the camera 2 and the individual identification device 9, for example, using Kinect (registered trademark) which is a video processing system for video games, each person's personal ID and each person's position (for example, eye position) And identify. According to the identification result, it is assumed that the customer U1 exists on the left side of the display device 91, the employee U2 exists on the front side, and the executive U3 exists on the right side.

FIG. 10B is a screen diagram showing display contents visible to each person from the display device 91 of FIG.
Only customer “Uri” with low secrecy can be viewed by customer U1, and employee U2 can also browse “Business Division” with slightly higher secrecy in addition to “Uri of Product”. In addition to the “Uri of the product, the division in charge”, the executive U3 is also able to browse the “confidential sales forecast for the current term”.
As described above, when the directional display is applied to the presentation, the content displayed on the slide can be changed depending on the affiliation and authority. For example, the directional display may show a password only to a person with a specific personal ID and show another person in a state where the password display location is concealed.

In this way, the individual identification device 9 is based on who is where (the contents of the individual identification table 18) and what is shown to whom (display data associated in advance for each individual ID or individual attribute). Then, where and what to show (display contents of the directional display) are determined and output to the display device 91.
In addition, by installing directional displays on the floors and walls of distribution warehouses, alerts and navigation can be shown only to specific workers.
In addition, directional displays may show signboard advertisements for each person depending on their interests, etc., or they may show subtitles or change the granularity of information depending on gender, age, or understandable language. Alternatively, the display color of the directional display installed as the interior of the room may be changed according to preference.

In the present embodiment described above, the similarity evaluation unit 25 compares the intra-image movement amount graph 12 c extracted from the video data from the camera 2 and the corrected sensing graph 11 c obtained by correcting the motion data from the sensor terminal 1. As a result, the individual similarity table 17 is created. Here, in the correction process from the pre-correction sensing graph 11b to the post-correction sensing graph 11c, the sensing correction unit 23 determines that the subject camera distance is short according to the camera distance graph 13b extracted from the video data from the camera 2. By reducing the correction and increasing the correction when the camera distance is long, it is possible to create apparent motion information from the sensing data as indicated by the intra-image movement amount graph 12c.
Therefore, since both the corrected sensing graph 11c and the intra-image movement amount graph 12c are apparent motion information viewed from the camera, it is expected that the same person is correctly associated, and the success rate of individual identification is improved. .

In order to describe in detail the effect of the correction process by the sensing correction unit 23 of the present embodiment, the following comparative example illustrates an individual identification process when correction is not performed (corresponding to the pre-correction sensing graph 11b).
FIG. 11A is an explanatory diagram showing a state of shooting from a ceiling camera. There are three people (person 1, person 2, person 3) in the environment, and a camera installed on the ceiling photographs the situation from directly above. In addition, the curve described with each person as the starting point indicates the trajectory that each person has moved.
FIG.11 (b) is explanatory drawing which shows the image | video from the camera of Fig.11 (a). The moving body detection means detects a human region (objects A, B, C) from the camera video.
FIG.11 (c) is explanatory drawing which shows the individual identification result based on the image | video of FIG.11 (b). The acceleration data from the acceleration sensor held by each person (person 1, person 2, person 3) in FIG. 11A is transmitted to the individual identification server as needed. This acceleration data oscillates finely because when a person walks, the acceleration periodically changes.

In addition, the individual identification server calculates the movement amount of the moving body in the image by continuously following the movement in the image in time series for the three objects extracted from the image in FIG.
Then, the individual identification server evaluates the similarity between the acceleration data graph and the motion amount graph of the moving object in the image, and adopts the objects in the image in order, for example, in order from the combination with the highest similarity. (For example, object C) and an acceleration sensor (for example, person 1) that the object will have are associated with each other. This association is referred to as individual identification.
In the comparative example of FIG. 11 described above, although the individual identification of three persons is successful, the camera distance is equalized regardless of the position of the three persons. Therefore, as shown in FIG. Must be used. Therefore, although a certain identification rate can be obtained, the introduction cost of the system is increased.

FIG. 12 is an explanatory view showing a state of shooting from a camera directed in the horizontal direction, unlike FIG.
FIG. 12A shows a case where a person who is an object of individual identification is photographed with a camera obliquely or from the side.
FIG. 12B is an explanatory diagram showing an image from the camera of FIG.
FIG.12 (c) is explanatory drawing which shows the individual identification result based on the image | video of FIG.12 (b).

Here, the individual identification between the person 2 and the object A and the individual identification between the person 3 and the object B are successful, but the individual identification between the person 1 and the object C is greatly different from each other in the shape of the graph. , Fail. The reason for this failure is that the person 1 as the object C is located far from the camera, so that the movement amount of the moving body in the image appears smaller than the actual acceleration data. In addition, the amount of movement of the moving object in the image appears to be a large movement on the image when the object moves at a right angle to the direction of the camera, but when the object moves along the direction of the camera (approaching the camera or moving away from the camera). This is because the movement is reflected in a small movement.

For this reason, in the movement data of the object C, the oscillation operation due to the periodic movement of walking is disappeared, and only rough acceleration at the time of starting and stopping is reflected in the data. Of course, even if the correlation between the acceleration data of the person 1 and the movement amount data of the object C is not obtained, a high value is not obtained. As a result, identification of person 1 fails.
In the comparative example of FIG. 12 described above, the introduction cost of the system can be suppressed because a camera directed in the horizontal direction may be used. However, individual identification of one of the three people fails because the camera distance is long.

In both the comparative example of FIG. 11 and the comparative example of FIG. 12, since the correction of sensing data is not performed, there is a problem of a decrease in the identification rate due to the camera distance. On the other hand, the individual identification device 9 according to the present embodiment corrects the sensing data, so that the identification rate does not decrease even for a subject with a long camera distance, and is freed from physical restrictions on the camera position. Therefore, the introduction cost of the system can be suppressed.

In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described.
Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.
Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment. Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit.
Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor.

Information such as programs, tables, and files for realizing each function is stored in memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC (Integrated Circuit) card, an SD card, a DVD (Digital Versatile Disc), etc. Can be placed on any recording medium.
Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

1 Sensor terminal 2 Camera 8 Base station 9 Individual identification device 11a Sensing table 11b Sensing graph before correction 11c Sensing graph after correction 12a Video data storage unit 12b Human area table 12c Intra-image movement amount graph 13a Depth map storage unit 13b Camera distance graph 17 Individual similarity table 18 Individual identification table 21 Human region extraction unit 22 Camera distance extraction unit 23 Sensing correction unit 24 In-image movement extraction unit 25 Similarity evaluation unit 26 Individual identification unit 91 Display device 92 Input device 93 Processor 94 Memory 95 Recording device 96 I / F

Claims

For each human region extracted from the video data of the camera, an intra-image movement extraction unit that extracts intra-image movement amount data indicating movement in the image of the human region;
A sensing correction unit that corrects sensing data indicating movement of each person measured from a sensor terminal held by each person based on a camera distance from the installation position of the camera to the position of the person shown as the person area When,
Based on the similarity between the extracted movement amount data in the image and the corrected sensing data, the individual identification unit that associates the human region in the video data with the gripper of the sensor terminal, An individual identification device characterized by comprising:
The sensing correction unit corrects the sensing data recorded in time series by passing through a low-pass filter, and lowers the pass band in the low-pass filter as the camera distance is farther from the camera. The individual identification device according to claim 1.
The individual identification device further determines the camera distance of the human area based on a predetermined function that makes the camera distance closer as the area of the human area in the video data of the camera is larger. The individual identification device according to claim 1 or 2.
The individual identification device further refers to a depth map in which the camera distance corresponding to the pixel position is previously associated with each pixel position in the video data of the camera, and The individual identification device according to claim 1, wherein the camera distance of the human region is obtained from a pixel position.
The individual identification device further includes the individual identification unit from the installation position of the camera based on the installation position and installation direction of the camera and the in-image position of the human area associated with the individual identification unit. A linear vector heading to the position of the person holding the sensor terminal associated with the sensor terminal is obtained, and the position where the obtained linear vector is traced by the camera distance is determined from the installation position of the camera. The individual identification device according to claim 1, wherein the individual identification device is specified as a spatial position of a grasping person.
The individual identification device further acquires display data to be shown to a person holding the sensor terminal associated by the individual identification unit with reference to display data to be shown for each person prepared in advance, and The individual identification device according to claim 5, wherein display control is performed to display the acquired display data toward a specified person's spatial position.
The individual identification device has an intra-image movement extraction unit, a sensing correction unit, and an individual identification unit,
The intra-image movement extraction unit extracts, for each human area extracted from the video data of the camera, intra-image movement amount data indicating movement in the image of the human area,
The sensing correction unit is a sensing data indicating a movement of each person measured from a sensor terminal held by each person based on a camera distance from an installation position of the camera to a position of the person shown as the person area. To correct
The individual identification unit associates the human region in the video data with the gripper of the sensor terminal based on the similarity between the extracted intra-image movement amount data and the corrected sensing data. The individual identification method characterized by attaching.