JP7187655B1 - OBJECT TRACKING METHOD, PROGRAM, SYSTEM AND RECORDING MEDIUM - Google Patents

Abstract

An object of the present invention is to enhance the identification function by complementing the identification processing with the feature information of the tracked object when the identification based on the position information is difficult. A tracked object (P) is imaged to obtain image information (If) and depth information (Di), and position information including three-dimensional coordinate values representing the tracked object is obtained by the image information and the depth information. and acquiring the location information in time series; and identifying the tracked object using the location information. During tracking of the tracked target, a step of comparing the current position information and the latest position information in time series to calculate the moving distance of the tracked target, and if the moving distance is within a threshold, the current tracked target and the most recent tracked object. [Selection drawing] Fig. 5

Description

The present disclosure relates, for example, to a tracking technique using a portable multifunction device or the like to track an object, such as a person, being tracked on a camera image acquired by the portable multifunction device.

It is already known to detect an object to be tracked from a camera image and track this object on the image. Identification of the object between consecutive images is essential for this object tracking.

Regarding this object tracking, using a deep learning discriminator based on deep learning and a multi-channel feature amount including the feature amount by the deep learning discriminator, particle tracking is performed according to the distance between the multi-channel feature position information and the particle position information It is known to include a particle filter function unit that tracks an object by applying filter likelihood evaluation (for example, Patent Document 1).

Regarding human tracking, feature information for extracting recognition targets is selected from image information acquired by multiple cameras, and feature information effective for image recognition processing is selected from multiple feature information based on importance information and reliability information. is known (for example, Patent Document 2).

Regarding the identification of a person, images are taken from different directions to acquire relative orientation relationships between input area images, and characteristics of registration area images included in a group of registration area images obtained by taking images from at least three directions and the input area. It is known to compare features of images to determine whether they are the same person (for example, Patent Document 3).

For identification of a person, face angle range determination processing is performed to associate the face angle range data representing the face angle range of the face image data with the face image data. It is possible to determine a shot and perform best shot determination processing for deleting face image data other than that, and furthermore, create person management data by aggregating face image data determined to be the best shot, and identify a person. known (for example, Patent Document 4).

JP 2019-153112 A JP 2011-60024 A JP 2016-1447 A JP 2016-157165 A

To track a person on the camera image, the tracked object must be recognized in each frame. The current frame and the previous frame should be compared to identify the tracked object. Thus, it is essential to identify the tracked object between frames.

If there are multiple people in the frame, people overlap or touch each other in the image, and if the person to be tracked moves out of the angle of view of the camera and then re-enters the angle of view, the skeleton of the person will change. It becomes difficult to identify the tracked object by contrasting information only. In other words, in image comparison between frames, if the amount of information required for identification is insufficient, the tracked object cannot be identified and is lost.

If the object to be tracked is a person, facial recognition can be used. For this face recognition, it is necessary to compare the registered face information of the target person and the obtained face information, and to compare feature amounts. For this purpose, it is necessary to register information necessary for comparison, such as registration of a plurality of front images, as registration information of the tracking target.

However, it is cumbersome to acquire and register facial images from persons whose tracked subjects are naturally behaving, and to continuously identify the tracked subjects by contrasting the facial information. Of course, it is impossible to obtain registration information in advance for unspecified tracking targets, and if the tracking target is a pedestrian, an athlete, a care recipient, etc., and its movement is different, the information necessary for tracking is not available. difficult to obtain.

The inventor of the present disclosure obtains image information and depth information in time series to obtain position information including coordinate values in the world space (three-dimensional space), identifies with this position information, We have found that identification using feature information enables highly accurate tracking of objects such as people with various movements.

Therefore, in view of the above problems or findings, the object of the present disclosure is to acquire position information of a tracked target including coordinate values in time series using at least two-dimensional image information and depth information, and to determine a moving target being tracked. The object is to identify the object and realize high-speed and high-precision object tracking.

Another object of the present disclosure is to enhance the identification function by supplementing the identification process with the feature information of the tracked object if identification based on position information is difficult.

In order to achieve the above object, according to one aspect of the object tracking method of the present disclosure, image information and depth information are acquired in time series from a tracked object by imaging, and three-dimensional coordinate values are obtained from the image information and the depth information. obtaining position information in time series, including orientation information of the tracked object and feature information of the face area of the tracked object from the image information in time series, and converting the obtained feature information using the orientation information classifying and storing in a database; identifying the tracked target using the position information ; and, if the tracked target cannot be identified by the acquired image information and the position information based on the depth information, the image information. obtains the orientation information of the tracked object and the feature information of the face area from the database, and compares the obtained feature information with the feature information classified by the orientation information in the database to identify the tracked object. and a step .

In this object tracking method, further, during tracking, comparing the current position information of the tracked object with the most recent position information in time series to calculate the movement distance of the tracked object; and identifying the tracked object, if any.

In this object tracking method, if there is an overlap between the tracked object being tracked and the image of another tracked object, or an overlap between the bounding box of the tracked object and the bounding box of the other tracked object, A step of identifying the tracked object by comparing the feature information immediately before the overlap with the feature information at the time of cancellation of the overlap may be included.

In this object tracking method, the steps of acquiring face region information in time series from the image information, acquiring at least eye or ear region information from the face region information, obtaining the orientation information, classifying the feature information according to the orientation information and storing the characteristic information in the database ; Acquiring the orientation information of the tracking target, acquiring the feature information of the face region from the image information, and comparing the acquired feature information with the feature information classified by the orientation information in the database . and identifying the tracked object.

This object tracking method may further include the step of dividing the bounding box to form a grid of multiple rows and multiple columns, and acquiring the position information specified by the coordinate values of the grid.
This object tracking method may further include the step of dividing the bounding box to form a grid of multiple rows and multiple columns, and obtaining the part information specified by the coordinate values of the grid.

In order to achieve the above object, according to one aspect of the program of the present disclosure, there is provided a program executed by a computer, which obtains image information and depth information in time series by imaging from a tracked object, and obtains the image information and the depth a function of acquiring position information including three-dimensional coordinate values from the information in time series; A function of classifying feature information according to the direction information and storing it in a database, a function of identifying the tracked target using the position information, and identifying the tracked target with the position information based on the obtained image information and the depth information. If identification is not possible, the orientation information of the tracked object and the feature information of the face area are obtained from the image information, and the obtained feature information and the feature information classified by the orientation information of the database are compared. , and the function of identifying the tracked object are performed by the computer.

In this program, further, during tracking, a function of comparing the current position information and the latest position information of the tracked object in time series to calculate the movement distance of the tracked object, and if the movement distance is within a threshold , and the function of identifying the tracked object may be performed by the computer.

In this program, if an overlap occurs between the tracked object being tracked and the image of another tracked object, or between the bounding box of the tracked object and the bounding box of the other tracked object, the overlapping The computer may execute a function of identifying the tracked object by comparing the feature information immediately before the overlap with the feature information at the time of cancellation of the overlap.

This program further comprises a function of acquiring face area information in time series from the image information, acquiring at least eye or ear part information from the face area information, and the orientation of the tracked target using the part information. a function of acquiring information, classifying the feature information according to the orientation information and storing the feature information in the database ; Acquiring the orientation information of the target, acquiring the feature information of the face region from the image information, and comparing the acquired feature information with the feature information classified by the orientation information in the database , and identifying the tracked object may be performed by the computer.

In this program, the computer may further execute a function of dividing the bounding box to form a grid of multiple rows and multiple columns, and acquiring the position information specified by the coordinate values of the grid. .
In this program, the computer may further execute a function of dividing the bounding box to form a grid of multiple rows and multiple columns, and acquiring the region information specified by the coordinate values of the grid.

In order to achieve the above object, according to one aspect of the object tracking system of the present disclosure, an image information acquisition unit that acquires at least two-dimensional image information representing a tracked object in time series, and depth information of the tracked object that a depth information acquisition unit that acquires serially, a feature information acquisition unit that acquires direction information and feature information of the tracked object from the image information, a database that classifies and stores the acquired feature information according to the direction information; Obtaining position information of the tracked target using the image information and the depth information, identifying the tracked target using the position information, and determining the orientation of the tracked target from the image information when the position information cannot be used to identify the tracked target. an identification processing unit that acquires information and the feature information of a face area, compares the acquired feature information with the feature information classified by the orientation information in the database, and identifies the tracking target; and an information presenter for presenting tracking information along with image information representing the object.

In order to achieve the above object, according to one aspect of the object tracking system of the present disclosure, the object tracking system includes at least the image information acquisition unit, the depth information acquisition unit, the feature information acquisition unit, and the identification processing unit, and tracks a tracked object. It comprises two or more devices that output information, and a server that acquires the tracking information from each device and extends the tracking angle or tracking range of the tracked target to track the tracked target.

In order to achieve the above object, according to one aspect of a recording medium of the present disclosure , there is provided a recording medium storing the program or storing the database.

According to the present disclosure, any of the following effects can be obtained.
(1) Since 2D image information obtained from the tracked object and depth information are used to obtain 3D positional information, this positional information can be used to improve the recognition rate and accuracy of tracked objects such as people. This enables high-precision and high-speed identification of the tracked object, increases the reliability of tracking, and enhances the tracking function.

(2) A tracking target such as a moving person can be tracked by a portable device equipped with a camera and a light detection/ranging unit. .

(3) Objects to be tracked For example, it can be used to track various objects such as players in motion, care recipients, people entering and exiting facilities, and passers-by.

FIG. 1 is a diagram showing an example of an object tracking system according to the first embodiment. FIG. 2A is a view of the back portion of the portable multifunction device, and FIG. 2B is a view of the front portion of the portable multifunction device. FIG. 3 is a diagram showing the hardware of the portable multifunction device. FIG. 4 is a diagram showing an example of a tracking information database. FIG. 5 is a flow chart showing a processing procedure for object tracking. FIG. 6 is a flow chart showing a process of acquiring position information of a tracked object. FIG. 7 is a flow chart showing bounding box processing. FIG. 8A is a diagram showing the grid points from the bounding box, and FIG. 8B is a diagram showing the bounding box and depth information. 9A is a diagram showing image information, FIG. 9B is a diagram showing depth information, and FIG. 9C is a diagram showing a synthesized image. FIG. 10 is a flow chart showing a processing procedure for orientation determination. FIG. 11 is a diagram showing the relationship between acquisition sites and orientations. FIG. 12 is a flow chart showing identification processing I. FIG. 13A is a diagram showing image information during tracking, FIG. 13B is a diagram showing image information including bounding box overlapping states, and FIG. 13C is an image information including bounding box separation states. It is a figure which shows. FIG. 14A is a diagram showing a case where the moving distance of the bounding box is equal to or less than the threshold, and FIG. 14B is a diagram showing a case where the moving distance of the bounding box exceeds the threshold. FIG. 15A is a diagram showing a tracked object frame-in, FIG. 15B is a diagram showing a tracked object frame-out, and FIG. 15C is a diagram showing a tracked object re-frame-in. be. FIG. 16 is a flowchart showing identification processing II. FIG. 17 is a diagram showing an object tracking system according to the second embodiment. FIG. 18 is a diagram illustrating increased tracking range with two devices. FIG. 19 is a diagram illustrating extended tracking range and compound tracking with four devices.

[First embodiment]
FIG. 1 shows an object tracking system according to the first embodiment. The configuration shown in FIG. 1 is an example, and the present disclosure is not limited to such a configuration.

This object tracking system 2 identifies a tracked target by identification processing I based on position information acquired from image information and depth information of the tracked target, and identifies the tracked target using feature information acquired from image information when identification processing I fails to identify the target. Identify in Treatment II. This object tracking system 2 is, for example, Apple Inc. (Apple Inc.) products, but are not limited to the illustrated portable multifunction devices.

This object tracking system 2 includes a processing section 4, a camera 6, a light detection/ranging section 8, an information presentation section 10, and the like. The processing unit 4 manages information processing for tracking a tracked object by information processing, and control of various functional units such as the camera 6 and the light detection/ranging unit 8 .

Camera 6 is an example of an imaging unit of the present disclosure. The camera 6 captures an image of an area including the tracked object under the control of the processing unit 4, and outputs continuous, for example, two-dimensional image information in time series.

The light detection/ranging unit 8 is an example of a light detection/ranging function unit installed in a portable multifunction device. The light detection/distance measurement unit 8 outputs depth information acquired in synchronization with the imaging by the camera 6 under the control of the processing unit 4 . The light detection/ranging unit 8 is composed of a light detection/ranging unit such as a LIDAR (Light Detection and Ranging) scanner. This lidar scanner unit scans a laser beam that irradiates a tracked target, receives the reflected light from the tracked target, and obtains depth information representing the distance between the tracked target and the light irradiation point, that is, the depth in chronological order. can do.

The information presenting unit 10 is composed of an image presenting unit such as an LCD (Liquid Crystal Display), and presents an image including a tracked object and tracking information.

<Processing unit 4>
The processing unit 4 includes a tracking control unit 12, an image information acquisition unit 14, a depth information acquisition unit 16, a bounding box processing unit 18, a tracking information database generation unit 20, a position information processing unit 22, a feature information processing unit 24, and a state information processing unit. Functional units such as a unit 26, an identification processing unit 28, an identification information presenting unit 30, and a link processing unit 32 are included.

For example, in order to track a moving tracked target, the tracking control unit 12 controls acquisition of position information and feature information from the tracked target, controls identification of the tracked target based on the position information, and obtains feature information when identification based on the position information is not possible. It governs identification control by

The image information acquisition unit 14 acquires image information including a tracked object from the camera 6 . The image information acquisition unit 14 acquires two-dimensional image information in time series from the camera 6 under the control of the tracking control unit 12 and provides the image information to the tracking information database generation unit 20 .

The depth information acquisition unit 16 acquires the depth information acquired by the light detection/ranging unit 8 in synchronization with the imaging by the camera 6 in chronological order.

The bounding box processing unit 18 executes processing such as tracking target detection, bounding box acquisition function, division function, grid point generation function, and coordinate value acquisition function. The skeleton of the tracked target is detected from the image information, and the tracked target is specified. An API (Application Programming Interface) function included in an existing OS (Operating System) installed in the device generates a bounding box for the image information of the tracked object, so this bounding box can be obtained. The bounding box is, for example, a rectangular bounding box that represents the region containing the upper body of the tracked subject. This bounding box division function divides the bounding box into multiple compartments with multiple row lines and column lines, and generates grid points g at the intersections of the row lines and column lines. Along with acquiring the coordinate values of this grid point g, the depth information is referenced to select position information that is effective for specifying the tracked object.

The tracking information database generation unit 20 generates a tracking information database (DB) 66 (FIG. 4), and stores image information, depth information, bounding box information, grid point information, position information, and features acquired or generated by the processing unit 4. Information necessary for tracking information such as information is stored in the tracking information DB 66 in frame units, for example. This tracking information DB 66 is used to identify the tracked object.

The position information processing unit 22 synthesizes the two-dimensional image information provided from the image information acquisition unit 14 and the depth information provided from the depth information acquisition unit 16, and uses the synthesized image and the grid points g described above. to generate position information including coordinate values for identifying the tracked object.

The feature information processing unit 24 acquires the feature information of the tracked object and the orientation information, which is an example of the classification information of the feature information, from the image information, determines the orientation of the face of the tracked object, and categorizes the feature information into the orientation of the face. . Under the control of the tracking control unit 12, the feature information is obtained by obtaining information on the face area of the tracking target from the image information and obtaining the feature information from this face area. This feature information is, for example, feature information that can be obtained from face image information excluding parts such as eyes and ears. Direction information is classification information such as "front", "leftward", and "rightward" determined by acquisition parts such as eyes and ears. Identification processing II based on face recognition identifies a tracked target according to the amount of feature amounts included in feature information acquired from the tracked target.

The state information processing unit 26 monitors the bounding boxes to be tracked, and obtains state information such as overlap between bounding boxes, moving distance of bounding boxes, re-frame-in after frame-out of bounding boxes. By acquiring this state information, when the tracked object cannot be identified by the position information, the feature information of the tracked object in the frame immediately before that and the feature information of the tracked object in the frame immediately after the indefinite state is resolved are acquired.

The identification processing unit 28 executes identification processing I using the position information when the tracking target can be identified using the position information, and when the tracking target cannot be identified using the position information, the tracking target features Perform identification processing II using information. Specifically, under the control of the tracking control unit 12, the identification processing unit 28 acquires the coordinate values obtained in time series for each frame of the image information in time series, and identifies the tracking target in frame units. That is, if the movement distance of the tracked object is equal to or less than a predetermined threshold value, for example, 0.5 m, the tracked object is identified and identified as the same object. On the other hand, if the movement distance of the tracked object is equal to or greater than the threshold, identification processing I based on position information including coordinate values is avoided, and identification processing II based on face authentication is performed. The threshold for the movement distance may be set according to the accuracy of depth information that can be acquired by the depth information acquisition unit 16 .

The identification information presenting section 30 generates a tracking display for the tracked object being tracked and presents it to the information presenting section 10 . This tracking display is an example of tracking information representing tracking states such as tracking and identification of a tracked object. In this tracking display, an image representing the identified tracked object is displayed, for example, in the same color, and is set to be distinguishable from other tracked objects. While the bounding box is an information processing concept for identifying the tracked object, this tracking display is displayed on the tracked object being tracked, for example, the head, unlike the bounding box.

The link processing unit 32 links with an external server 36 via the network 34 to complement tracking processing of the tracked object.

In this first embodiment, the object tracking system 2 includes the link processing unit 32, the network 34 and the server 36, but the link processing unit 32, the network 34 and the server 36 are removed from the object tracking system 2. Object tracking system 2 may consist of portable multifunction device (hereinafter simply referred to as “device”) 38 alone.

<Device 38>
FIG. 2A shows the back side of the device 38. FIG. This device 38 is an example of an object tracker of the present disclosure.

An information acquisition section 41 is set on the back of the apparatus main body 40 of the device 38, and the information acquisition section 41 is provided with the camera 6 and the light detection/distance measurement section 8. FIG.

FIG. 2B shows the front of the device 38. FIG. A display screen portion 46 of a display 45 is installed on the front surface of the device body 40 of the device 38 , and a touch panel 48 is installed on the display screen portion 46 . This touch panel 48 is an example of the operation input unit 56 . The operation input unit 56 includes interface devices such as a keyboard device and a mouse (not shown).

Image information If is displayed on the display screen section 46 as an example. This image information If is moving image information being tracked, and includes a plurality of tracked objects P-1 and P-2 and tracking displays T-1 and T-2. Tracking display T-1 is overlaid on the head of tracked subject P-1 to represent the identification state, and tracking display T-2 is overlaid on the head of tracked subject P-2 to represent the identification state. Each tracking display T-1, T-2 is presented in a different display color, and the tracking targets P-1, P-2 can also be distinguished by the tracking displays T-1, T-2. In terms of information processing, a bounding box B-1 is formed for the tracked object P-1 and a bounding box B-2 is formed for the tracked object P-2, but these are not presented on the image. Hereinafter, when the tracked targets P-1 and P-2 are not specified, they are simply referred to as tracked target P, and when the tracked display is not specified, they are simply referred to as tracked display T, and similarly, Bounding boxes B-1 and B-2 are also simply referred to as bounding box B when the tracking target is not specified.

<Hardware of Device 38>
FIG. 3 shows an example of the hardware of device 38 . The device 38 includes the processing unit 4, the camera 6, the light detection/ranging unit 8, the display 45, the touch panel 48, and the like.

The processing unit 4 includes a processor 58 , a storage unit 60 , an input/output unit (I/O) 62 and a communication unit 64 . The processor 58 executes various programs such as an OS and an object tracking program stored in the storage unit 60, and controls the above-described functional units.

The storage unit 60 is an example of the recording medium of the present disclosure, and stores various information such as an OS, object tracking program, tracking information DB 66, and the like. Storage elements such as ROM (Read-Only Memory) and RAM (Random-Access Memory) are used for the storage unit 60 . The RAM constitutes a work area, a framework, and the like for information processing.

The I/O 62 inputs and outputs information under the control of the processor 58 . The I/O 62 is connected with the camera 6, the light detection/distance measurement unit 8, the display 45, the touch panel 48, etc. as information input means.

A display device other than the display 45 may be used for the information presentation unit 10 . An operation input device other than the touch panel 48 may be used as the operation input unit 56 .

The communication unit 64 communicates with the server 36 via the network 34 under the control of the processor 58, and exchanges information necessary for object tracking through linkage.

<Tracking information DB 66>
FIG. 4 shows the tracking information DB 66 that stores tracking information used in the object tracking system 2. As shown in FIG. This trace information DB 66 includes trace object files 67-1, 67-2, . . . , 67-n generated for each detected trace object.

. . , 67-n includes an image information section 68, a depth information section 70, a bounding box section 72, a grid point section 74, a position information section 76, and a feature information section 78. , a classification information section 80, an orientation information section 82, a state information section 84, an identification information section 86, and a history information section 88 are included.

The image information section 68 stores two-dimensional image information acquired from the tracking target P by the camera 6 in time series. The depth information section 70 stores the depth information acquired from the tracking target P by the light detection/ranging section 8 in chronological order. Bounding box information obtained from image information is stored in the bounding box portion 72 . The grid point portion 74 stores the position information of the grid points generated in the bounding box. The position information section 76 stores position information calculated from the image information acquired from the tracked object and the depth information. The feature information section 78 stores the feature information of the tracked object obtained from the image information. The classification information section 80 stores classification information of image information used for face recognition. The classification information section 80 includes position information for left eye, right eye, left ear, and right ear.

The orientation information section 82 stores feature information classified by orientation information representing the orientation of the face. The orientation information section 82 has a front section 82-1, a left section 82-2, a right section 82-3, and an undefined section 82-4. The front portion 82-1 stores the feature information classified as the front. The leftward portion 82-2 stores the characteristic information classified into the leftward direction. The rightward portion 82-3 stores feature information classified in the rightward direction. The undefined portion 82-4 stores feature information that is not classified into any category.

The state information section 84 stores the state information of the bounding box. A movement distance information section 84-1 and an overlap information section 84-2 are set in this state information. The movement distance information section 84-1 stores movement distance information between bounding boxes, frame-out information, and the like.

The overlap information section 84-2 stores overlap information of bounding boxes and the like. The identification information section 86 stores identification processing I, identification processing II, tracking display color information indicating the identification result, identification category information, and the like in the processing for identifying the tracking target. The history information section 88 stores history information such as tracked object identification history and track record.

<Procedure for object tracking>
FIG. 5 shows the procedure of object tracking using device 38 . This processing procedure is an example of an object tracking method or an object tracking program using the object tracking system of the present disclosure. In FIG. 5, S indicates a function or process step, and the number attached to S indicates the order of steps (the same applies to the flow charts shown in FIGS. 6, 7, 10, 12 and 16). .

As shown in FIG. 5, this processing procedure includes acquisition of image information and depth information (S101), determination of start of tracking mode (S102), acquisition and storage of position information of the tracked object (S103), orientation information of the tracked object. - Acquisition and storage of characteristic information (S104), determination of identification (S105, S106, S107, S108, S109), presentation of tracking information (S110), and the like are included.

Acquisition of Image Information and Depth Information (S101): When the device 38 is activated, an image acquisition mode for acquiring image information is activated. In this image acquisition mode, the camera 6 and the light detection/ranging unit 8 are activated. The camera 6 acquires image information in time series under the control of the tracking control unit 12, and the light detection/ranging unit 8 acquires depth information of the tracked object in time series.

The image information is, for example, two-dimensional image information obtained by imaging with the camera 6 aimed at the tracked object based on the user's operation. Therefore, this image information includes the single or multiple tracked objects as well as the background and the like.

The depth information is information representing the depth acquired by the light detection/ranging unit 8 directed toward the tracking target together with the camera 6 based on the user's operation. If the image information is two-dimensional, the depth is distance information representing the depth direction with respect to the image information.

Determining whether to start tracking mode (S102): In this embodiment, the tracking mode for acquiring image information and tracking the tracked object are set separately. Initiation of this tracking mode requires that initiation conditions be met. The start condition is acquisition of start information in the image acquisition mode. For example, a touch on bounding box B displayed on the tracked object is sensed and used as starting information to initiate tracking mode.

Acquisition and storage of position information of tracked object (S103): In object tracking of the present disclosure, a tracked object is identified and position information including coordinate values of the tracked object within bounding box B is obtained. To acquire this position information, a tracked target is detected from image information acquired by the camera 6, two-dimensional information representing this tracked target is combined with depth information acquired from the tracked target, and a three-dimensional image representing the tracked target is synthesized. Positional information including coordinate values, which is information, is acquired in chronological order. This position information is recorded and saved in the tracking information DB 66 together with image information and depth information.

Acquisition and storage of orientation information and feature information of tracked object (S104): For object tracking according to the present disclosure, orientation information and feature information of a tracked object are used in addition to position information of the tracked object. For this feature information, if the tracking target is a person, orientation information and feature information of the person are obtained from the face image. Orientation information is information representing the orientation of a person's face, and is used to categorize feature information. For example, three patterns of front, left, and right are set for the face direction. Categorization of feature information is a unit concept for specifying feature information for comparison, and feature information in a face image is classified in units of face orientation, and recorded and saved in the tracking information DB 66 .

Identification Judgment (S105, S106, S107, S108, S109): The identification judgment is a judgment as to whether or not the moving tracking targets are the same. This identification determination includes identification processing I as the first processing and identification processing II as the second processing. Identification processing I is identification using the position information of the tracked object, and identification processing II is identification using the feature information of the tracked object. In other words, when a determination is made based on the three-dimensional coordinate values, which is the determination based on the position information of the tracked object, when identification cannot be performed based on the coordinate values due to the presence of an indeterminate frame, identification processing II, which is processing that complements identification processing I, is performed. Includes face recognition identification.

Even if image information and depth information can be acquired when identification by coordinate values is not possible, for example: (1) When bounding box B-1 overlaps another bounding box B-2 , (2) when the movement distance between the bounding box of the current frame and the bounding box of the previous frame is greater than or equal to the threshold, and (3) when the tracked object moves out of the imaging angle range and returns to the angle of view. included.

If the bounding boxes overlap, the tracked object is identified based on the bounding box, so if the bounding boxes overlap, the tracked object will be lost.

Even if the bounding boxes do not overlap, if the moving distance of the bounding box of the current frame from the bounding box of the previous frame is equal to or greater than the threshold, the precision of the coordinate values is lost, so identification is not performed.

Further, when the tracking target is out of the imaging range of the camera 6, image information and depth information cannot be obtained. Therefore, even if the position information is obtained from the tracked target within the imaging range, the tracked target cannot be identified by comparing the position information obtained from the image information and depth information of the tracked target that has entered the imaging range again.

When the tracked object cannot be identified by the position information in this way, the tracked object is identified by using the feature information. In other words, if the tracking target can be identified by the coordinate values, the identification using the feature information is omitted.

Therefore, this identification judgment (S105) includes identification judgment in identification processing I based on position information including coordinate values (S106), identification judgment in identification processing II based on feature information (S107), and tracking target identification (S108). , indefinite determination (S109). In identification determination (S106) in identification processing I, it is determined whether identification is possible in identification processing I. If identification is possible in identification processing I, the tracking target is identified only by position information (S108).

If identification cannot be performed by identification processing I (NO in S106), it is determined whether identification is by identification processing II (S107). If it can be identified in the identification process II (YES in S107), the tracking target is identified (S108). On the other hand, if it cannot be identified in the identification process II (NO in S107), the tracking target is determined to be undefined (S109), and the process returns to S103. That is, if it cannot be identified even by the characteristic information, the process returns to S103.

Presentation of Tracking Information (S110): If the tracked object is identified (S108), the tracking display is presented together with the tracking information of the tracked object, ie, the image information of the tracked object being tracked (S110), and the process returns to S103. The tracking display of the identified tracked objects is displayed with different colors for each tracked object. That is, the tracking representation of the tracked object being identified is presented in the same color, and is presented with a different coloring than the tracking representations of other tracked objects. In other words, the tracking display of the tracked object for which identification has been lost should be made different from the coloring during identification, and the state may be presented.

<Acquisition of location information>
FIG. 6 shows a processing procedure for acquiring position information. This processing procedure is a subroutine for obtaining and storing the position information of the tracked object (S103 in FIG. 5).

This processing procedure includes acquisition of image information and depth information (S201), detection of a tracked object and acquisition of a bounding box (S202), acquisition of coordinate values of the tracked object (S203), storage of coordinate values (S204), It includes determination of end of value acquisition (S205) and processing of the next frame (S206).

Acquisition of image information and depth information (S201): This acquisition of image information and depth information is acquisition after the start of the tracking mode (S102).

Detection of tracked object P and acquisition of bounding box (S202): The detection of the tracked object P and acquisition of the bounding box are performed by detecting the tracked object from image information obtained from the camera 6 under the control of the tracking control unit 12. Get the coordinates of the tracked target. If the object to be tracked is a person, the person is detected from the image information using the human detection function (for example, Vision Framework - Request Human Detection) installed in the OS, and the bounding box, which is two-dimensional information, is obtained.

Acquisition of coordinate values of tracked target P (S203): In this acquisition of coordinate values of tracked target P, position information is acquired as three-dimensional information including coordinate values representing tracked target P with bounding box B as a unit. To obtain this position information, for example, as shown in FIG. 9, image information, which is two-dimensional information, and depth information are synthesized, and coordinate values are obtained from the synthesized image (three-dimensional coordinate information) obtained by this synthesis. Get in series. In other words, by this processing, it is possible to acquire position information representing the tracked object P within the bounding box with high accuracy.

Saving coordinate values (S204): The acquired coordinate values are stored and saved in the tracking information DB 66 under the control of the tracking control unit 12. FIG. This coordinate value may be stored, for example, in units of frames of image information.

Judgment of end of acquisition of coordinate values (S205): The end of acquisition of coordinate values is determined by the control of the tracking control unit 12 as to whether or not the acquisition of coordinate values in the frame is completed (S205). , S203 and S204 are repeated (NO in S205).

Processing of the next frame (S206): When the tracking control unit 12 determines that the acquisition of the coordinate values in the frame has ended (YES in S205), the processing of the next frame is started (S206), and the processing of S201 to S206 is performed. are executed in chronological order.

<Processing of Bounding Box B>
FIG. 7 shows the bounding box B processing. This processing procedure includes processing for dividing the bounding box B displayed on the tracked object and acquisition of position information of the tracked object, and is a subroutine of S202 of the position information acquisition processing (FIG. 6).

This processing procedure includes acquisition of the width w and height h of the bounding box B (S301), division of the bounding box (S302), generation of the grid point g (S303), determination of the position of the grid point g (S304, S305), Acquisition of coordinate values of grid point g (S306), check of reliability of coordinate values (S307), determination of reliability (S308), storage of coordinate values of grid point g (S309), determination of end of processing (S310), Determination of the number of coordinate values (S311), acquisition of coordinate values of the center Bn (S312), setting of position information (S313), determination of unknown position information (S314), and the like are included.

Acquisition of width w and height h of bounding box B (S301): Width w and height h of bounding box B are obtained from bounding box B acquired by bounding box processing unit 18 under the control of tracking control unit 12. and height h.

Division of bounding box B (S302): Under the control of the tracking control unit 12, the bounding box processing unit 18 divides the width direction into eight and the height into eight by a plurality of row lines and column lines as shown in FIG. 8A. Divide the direction into 8 and divide the bounding box B into 64 partitions. This division number is an example, and the present disclosure is not limited to this division number. Assuming that the width w in the X-axis direction of the bounding box B is w=x, and the height h in the Y-axis direction is h=y, the size of the division frame can be expressed in Steps by Equations 1 and 2. can be represented.

xStep=w/8 (Formula 1)
yStep=h/8 (Formula 2)

Generation of Grid Points g (S303): By dividing the bounding box B in this way, grid points g can be generated by dividing the bounding box B into 8×8, as shown in FIG. 8A. That is, the grid point g is the intersection of row lines and column lines.

Position determination of grid point g (S304): Each position is determined for each created grid point g. It is determined whether or not the position of the grid point g is below the center Bn of the bounding box B by +yStep as a threshold.

Position determination of grid point g (S305): If the position of grid point g is equal to or less than the threshold, that is, if it is below +yStep from the center Bn of bounding box B (YES in S305), the process proceeds to S306, and the position of grid point g is If it is above +yStep from the center Bn of the bounding box B (NO in S305), the process proceeds to S304.

Acquisition of coordinate values of grid point g (S306): For grid point g located below +yStep from center Bn of bounding box B, three-dimensional coordinate values are acquired. For this coordinate acquisition, the bounding box B is the area surrounding the upper body including the face area to be tracked. Therefore, the grid point g above the center Bn of the bounding box B is above the shoulder and includes parts other than the body, and there is a possibility that coordinate values other than the tracking target will be acquired. On the other hand, since the grid point g below the center Bn of the bounding box B is the central portion of the object to be tracked, that is, the central portion of the body, there is a possibility that the effective three-dimensional coordinate values of the grid point g can be obtained. high. Therefore, a grid point g below the center Bn of the bounding box B is acquired and used as position information of the tracked object.

Reliability Check of Coordinate Values (S307): The reliability of the coordinate values of the grid point g is checked. For this check, as shown in FIG. 8B, the depth information Di of the tracked object P that overlaps the grid point g is referred to. Depth information Di has low reliability low (=Depth-value accuracy in which the framework is less confident.) and medium reliability medium (=Depth-value accuracy in which the framework is moderately confident.) and high reliability (=Depth-value accuracy in which the framework is fairly confident.). Therefore, each grid point g can be reliably checked for depth information.

Determination of Reliability of Grid Point g (S308): Regarding the reliability of grid point g, a threshold is set to determine whether or not the reliability is high, and the determination is performed. That is, if the reliability of the grid point g is high (YES in S308), the process proceeds to S309, and if the reliability of the grid point g is low or medium (NO in S308) ), and returns to S305.

Saving coordinate values (S309): If the reliability of the grid point g is high (YES in S308), the acquired three-dimensional coordinate values of the grid point g are registered in the tracking information DB 66 as position information. save.

Processing End Determination (S310): The bounding box processing unit 18 determines whether or not the processing of all acquired grid points g is completed (S310). If the processing of all grid points g has not been completed (NO in S310), the processing of S305 to S310 is continued, and all grid points g are processed. Then, when the processing of all the grid points g is completed (YES in S310), the process proceeds to S311.

Determining the Number of Coordinate Values (S311): The bounding box processing unit 18 determines whether the number of coordinate values of the stored grid point g is equal to or greater than a threshold. For example, 2×2 is set as a constant value for the threshold of the number of coordinate values. In this case, if the number of coordinate values of the stored grid point g is equal to or greater than the threshold value of 2×2 (YES in S311), the process proceeds to S312. If there is (NO in S311), the process proceeds to S314.

Acquisition of coordinate values of the center (S312): If the number of coordinate values of the stored grid point g is equal to or greater than the threshold=2×2 (YES in S311), the bounding box processing unit 18 obtains the tracking Recognize as an object. As a result of this recognition, the bounding box processing unit 18 acquires the coordinate values of the center Bn of the bounding box B. FIG.

In this case, the processing for acquiring the coordinate values of the center Bn is, for example,
Mid X and Mid Y is the midpoint of the X and Y edges of screen bounds
MidX = screenBoundsd.MidX
MidY = screenBoundsd.MidY
Center = (Mid X, Mid Y + yStep)
, the value of the center Bn of the bounding box B is obtained.

Setting of position information (S313): The bounding box processing unit 18 sets the acquired coordinate values of the center Bn as the position information of the tracked object, and ends this process.

Determination of unknown position information (S314): If the number of stored coordinate values of the grid point g is less than the threshold=2×2 (NO in S311), the coordinate values of the grid point g are not saved, so the position information of the tracked object is determined. = is processed as unknown, and in this case, the tracked object has unknown position information.

<Synthesis of image information and depth information, acquisition of position information>
FIG. 9A shows two-dimensional image information. When the tracking target P is imaged by the camera 6, image information If representing the tracking target P is obtained as shown in A of FIG. This image information If is recorded in the tracking information DB 66 under the control of the tracking control unit 12 .

B of FIG. 9 shows the depth information Di of the tracked object P. FIG. When the tracking target P is measured under the same conditions by the light detection/distance measuring unit 8, depth information Di representing the imaging target is obtained. The depth information Di is stored from the upper left to the lower right of the screen (frame) in pairs of the distance (floating point) from the light detection/ranging unit 8 to the tracked object P and the reliability. When the distance is visualized, it is obtained as a grayscale image (binarized information), and the closer the distance, the darker it is, and the farther the brighter it is displayed. In other words, the depth information Di is binary information representing depth in terms of contrast, and the tracking target P can be obtained as grayscale information in the same way as the image information If.

C of FIG. 9 shows the composite image Cm. For example, if the depth information Di is superimposed on the image information If to generate the composite image Cm, the depth information Di is added to the image information If, and the composite image Cm visually has depth. Since the bounding box B in the image information If is presented in this composite image Cm, three-dimensional coordinate values are obtained as position information representing the tracked object P for each bounding box.

<Conversion from two-dimensional image information to three-dimensional coordinate values>
This conversion from two-dimensional image information to three-dimensional coordinate values is called coordinate value conversion. For this coordinate transformation,
A) Get minimum and maximum screen coordinates (2D) of grid point g B) Get world coordinates (3D) C) Width w and height of bounding box B with valid grid point g as apex Acquisition of h D) Includes processing such as acquisition of world coordinates (wx, wy, wz) from screen coordinates (sx, sy). These processes will be described below.

Obtain the minimum and maximum screen coordinates (two-dimensional) for the obtained grid point g in A).
screen. min = minimum value of grid point g (X, Y)
screen. max = maximum value of grid point g (X, Y)

In B) obtaining the world coordinates (three-dimensional) of the grid point g, the world coordinates (three-dimensional) corresponding to the bounding box B are obtained.
world. min = minimum value of grid points (X, Y, Z)
world. max = maximum value of grid points (X, Y, Z)

To obtain the width w and height h of the bounding box B whose vertex is the effective grid point g in C), the width and height of the rectangle whose vertex is the effective grid point g are obtained using the world coordinate values. get.
screen. w=screen. max. x-screen. min. x
screen. h=screen. max. y-screen. min. y
world. w=world. max. x-world. min. x
world. h=world. max. y-world. min. y

D) Acquisition of world coordinates (wx, wy, wz) from screen coordinates (sx, sy) is obtained by obtaining world coordinates (wx, wy, wz) from screen coordinates (sx, sy) of an arbitrary point. , converted using the following formula:
wx=sx*(world.w/screen.w)
wy=sy*(world.h/screen.h)
wz = DepthBuffer [sx + sy * screen width]

<Characteristic information of tracked object P>
Feature information can be acquired from the image information of the tracked object P. FIG. If the tracking target is a person, this feature information can be obtained from face information, and the tracking target can be identified using the feature amount included in this feature information.

The face information includes orientation information representing the orientation of the face. If the feature information acquired from the face information is classified according to the direction information, the identification process can be speeded up and the accuracy can be improved. The orientation of the face can be determined based on information on the parts of the head, such as information on the positions of the left eye, right eye, left ear, and right ear.

<Acquisition of part information and classification of feature information>
FIG. 10 shows face orientation determination logic and feature information categorization processing. This processing procedure indicates a subroutine of S104 (FIG. 5). This processing procedure includes acquisition of feature information and part information based on acquisition of face information, determination of orientation, and classification of feature information.

In this processing procedure, the left eye is the eye seen on the left side of the display screen, the left ear is the ear seen on the left side of the display screen, the right eye is the eye seen on the right side of the display screen, and the right ear is the eye seen on the right side of the display screen. This is the ear that can be seen on the right side of the screen. It is different from the left and right of the tracked object P itself. The same applies to the orientation information table 100 shown in FIG.

In this processing procedure, processing for acquiring position information of the left eye, right eye, left ear, and right ear as part information from the face image of the tracking target P is executed (S401). In this process, when the position information of both the left ear and the right ear is acquired (S402), it is determined that the face orientation of the tracking target P is "front" (S403), and the feature information is categorized into the front ( S404).

After S402, when both the left eye and right eye position information are acquired (S405), and the left ear position information can be acquired and the right ear position information cannot be acquired (S406), the tracking target P face direction=“left facing” (S407), and the feature information is categorized as left facing (S408).

After S405, when the position information of the right ear can be acquired but the position information of the left ear cannot be acquired (S409), it is determined that the face orientation of the tracking target P is "right facing" (S410), and the feature information is categorized as facing right. (S411).

After S405, if the position information of both the right ear and the left ear cannot be acquired (S412), it is determined that the face orientation of the tracking target P is "front" (S403), and the feature information is categorized as "front" (S404). .

After S405, when the position information of the left ear can be acquired, but the position information of the right ear and the right eye cannot be acquired (S413), it is determined that the face orientation of the tracking target P is "leftward" (S407), and the feature information is turned leftward. (S408).

After S405, when the position information of the right ear can be acquired, but the position information of the left ear and the left eye cannot be acquired (S414), it is determined that the face orientation of the tracking target P is "right facing" (S410), and the feature information is set to the right facing direction. (S411).

After S405, if the position information of the left eye, right eye, left ear, and right ear cannot be acquired (S415), it is determined that the face orientation of the tracking target P is "unknown" (S416), and the feature information is not categorized.

<Determination of orientation of tracked object P>
FIG. 11 shows an orientation information table 100 indicating acquired parts and orientation information.

In this orientation information table 100, an acquisition part information section 102 and an orientation information section 104 are set. The acquired part information section 102 stores the position information from the center position of each acquired part for the moving tracking target P. FIG. A left eye portion 102-1, a right eye portion 102-2, a left ear portion 102-3, and a right ear portion 102-4 are set in this acquired region information section 102. FIG.

Position information of the left eye of the tracked object P is stored in the left eye section 102-1. Position information of the right eye of the tracked object P is stored in the right eye section 102-2. Position information of the left ear of the tracking target P is stored in the left ear section 102-3. Position information of the right ear of the tracking target P is stored in the right ear section 102-4.

The orientation information section 104 stores orientation information representing the orientation of the tracked object P determined by a combination of acquired parts.

a) When Face Direction is Front If the position information of the left ear and right ear can be acquired, the orientation of the tracked object P is determined to be "front". Similarly, if position information for the left eye and right eye can be obtained, the orientation of the tracked object P is determined to be "front" even if position information for the left ear or right ear cannot be obtained.

b) When the face is facing left If the position information of the left eye and the right eye can be obtained, and the position information of the left ear can be obtained, it is determined that the tracking target P is facing left. Similarly, even if the position information for the left eye and the right eye cannot be acquired, if the position information for the left ear can be acquired, the direction of the tracked object P is determined to be "leftward."

c) When the face is directed to the right If the position information of the left eye and the right eye can be acquired, and the position information of the right ear can be acquired, the orientation of the tracked object P is determined to be "right." Similarly, even if the position information for the left eye and the right eye cannot be acquired, if the position information for the right ear can be acquired, the direction of the tracked object P is determined to be "facing right."

d) Orientation = Unknown The orientation of the tracked object P is determined to be "unknown" if position information for the left eye, right eye, left ear, and right ear cannot be acquired.

<Identification processing of tracking target P>
Tracking of the tracked target P requires identification processing of the tracked target P that moves. This identification processing includes identification processing I using position information acquired from the tracked target P and identification processing II using feature information acquired from the tracked target P. FIG.

In this embodiment, identification processing I using position information is performed first, and if identification is not possible in identification processing I, identification processing II using feature information is performed to speed up and improve the accuracy of identification processing. Realize

When tracking a moving tracked object P, the state of the tracked object P changes moment by moment. One of these changing states is the overlap between the bounding boxes B-1 and B-2. If the bounding box B-1 overlaps another bounding box B-2, the tracked object P will be lost.

During tracking of the tracked object P, the moving distance M of the bounding box B changes according to its moving speed. If the movement distance M increases, the position information changes, and the tracked object P is lost in this case as well.

The tracking range of the tracked object P is limited. Even when the tracking target P leaves this tracking range and returns to the tracking range again, it cannot be identified only by the position information of the tracking target P. In other words, the tracked object P is lost even when the frame is out.

In this way, when the tracking target P cannot be identified only by the position information, the identification process II is executed using the feature information of the tracking target P obtained in advance. That is, in this embodiment, the identification process I is complemented by the identification process II, thereby speeding up the identification process and achieving high accuracy.

<Identification processing I by position information>
FIG. 12 shows identification processing I based on position information. This processing procedure is a subroutine of S105 of the processing procedure shown in FIG.

This processing procedure includes acquisition of position information of tracked object P (S501), overlap check of bounding box B (S502), overlap determination (S503), indefinite determination (S504), calculation of movement distance M (S505), movement distance M determination (S506), tracking target P identification (S507), check end determination (S508), and the like are included.

Acquisition of position information of tracked object P (S501): In identification process I, the tracking control unit 12 uses the position information obtained from the tracked object P to identify it.

Overlap check of bounding box B (S502): During this identification, it is checked whether or not the bounding box B attached to each tracked object P overlaps. Overlapping of bounding boxes B is a state in which two or more bounding boxes B are in contact or superimposed on the image.

Overlap Determination (S503): If there is overlap between the bounding boxes B (YES in S503), the process proceeds to S504. If the bounding boxes B do not overlap (NO in S503), the process proceeds to S505.

Indeterminate determination (S504): If the bounding boxes B overlap (YES in S503), the tracking control unit 12 determines that each bounding box B is indeterminate, and identifies the corresponding tracked object P by the identification process I. Release and terminate this process. That is, in this case, the process proceeds to identification processing II based on feature information.

Calculation of movement distance M (S505): If there is no overlap between the bounding boxes B (NO in S503), the movement distance M of the bounding box B of the current frame and the bounding box B of the previous frame is calculated (S505). .

Determination of moving distance M (S506): For example, 0.5 m is set as the threshold Mth for the calculated moving distance M, and if it is 0.5 m or less (YES in S506), the process proceeds to S507. If the movement distance M exceeds the threshold Mth (NO in S506), the process proceeds to S504, and the tracking control unit 12 determines that the bounding box B in which the movement distance M exceeds the threshold Mth is indefinite. In other words, the identification by the identification process I for the corresponding tracked object P is cancelled, and this process ends. In this case, identification is determined by identification processing II based on feature information.

Identification of tracked target P (S507): If the calculated moving distance M is equal to or less than the threshold value Mth (YES in S506), the tracked target P is identified based on the position information. If the movement distance M is equal to or less than the threshold value Mth, it is determined that the tracking targets P are the same and identified.

Determination of check completion (S508): The tracking control unit 12 determines whether or not the processing of all bounding boxes B has been completed in the processing of S501 to S507 (S508). If the processing of all bounding boxes B has not been completed (NO in S508), the processing of S501 to S507 is continued, and the processing of bounding boxes B is repeated by the number of bounding boxes. When the process for all bounding boxes B is completed (YES in S508), the process proceeds to identification determination (S105 in FIG. 5) and ends this process.

<Overlap Judgment Operation of Bounding Box B and Acquisition of Feature Information>
FIG. 13A shows image information during tracking. In this image information, there are three tracked objects P-1, P-2, and P-3. has a bounding box B-2 and a tracking representation T-2.

FIG. 13B shows the image information being tracked following FIG. 13A. This image information represents the overlapping state of the bounding boxes B-1 and B-2. In other words, the tracked object P-1 overlaps the tracked object P-2, and the bounding boxes B-1 and B-2 overlap to form one bounding box Bx. Similarly, the tracking displays T-1 and T-2 are overlapped and merged into one tracking display Tx. In this case, both the bounding boxes B-1 and B-2 are in the previously described indeterminate state. Therefore, feature information is obtained from the image information of each of the tracking targets P-1 and P-2 from the frame immediately before the overlap.

FIG. 13C shows the image information being tracked following FIG. 13B. This image information shows the separation of the bounding boxes B-1, B-2 and the tracking representations T-1, T-2 from the overlap. That is, the tracked object P-1 moves away from the tracked object P-2, the bounding boxes B-1 and B-2 are separated, and the tracking representations T-1 and T-2 are separated as well. In other words, the position information of both of the tracking targets P-1 and P-2 can be obtained by canceling the overlapping state. However, immediately after the indefinite state is resolved, the feature information is acquired from the image information of each of the tracking targets P-1 and P-2 from the frame immediately after the overlapping is canceled for identification by the feature information.

<Determination Operation of Moving Distance M of Bounding Box B and Acquisition of Characteristic Information>
FIG. 14A shows a case where the moving distance M of the bounding box B is shorter than the threshold value Mth (M≦Mth).

During tracking of the tracked object P, the tracking control unit 12 monitors the moving distance M of the tracked object P-1. If the moving distance M satisfies M≤Mth, the tracked object P-1 is determined to be the same and identified.

B of FIG. 14 shows a case where the moving distance M of the bounding box B-1 is longer than the threshold Mth (M>Mth).

When the movement distance M of the tracked target P-1 exceeds the threshold Mth (M>Mth) during tracking of the tracked target P-1, the tracking control unit 12 determines that the accuracy of identifying the tracked target P based on the position information decreases. to stop identification based on location information. In this case, for identification by the feature information, the feature information of the tracked object P-1 is obtained from the image information of the frame immediately before the transition to the undefined state.

<Determination operation of frame-out and acquisition of feature information>
FIG. 15A shows image information during tracking. This image information shows the movement of one tracked object P-1 in frame F1.

FIG. 15B shows the case where the tracked object P-1 is out of frame during tracking. This frame-out means that the tracked object P-1 moves out of the frame F2 representing the tracking range as indicated by the arrow. In this case as well, the state is set to an indefinite state, and the identification based on the position information is stopped. Therefore, for identification based on feature information, the feature information of the tracking target P-1 is obtained from the image information of the frame immediately before the transition to the undefined state.

FIG. 15C shows the case where the tracked object P-1 re-frames-in. This re-in-frame means that the object to be tracked P-1 moves from outside the frame F3 and re-enters the frame F3, as indicated by the arrow. Also in this case, for identification based on feature information, feature information of the tracking target P-1 is obtained from the image information of the frame immediately after re-frame-in.

<Identification processing II by feature information>
FIG. 16 shows identification processing II based on feature information. This processing procedure is a subroutine of S105 of the processing procedure shown in FIG.

This processing procedure includes determination of an indeterminate bounding box B (S601), determination of a previous frame (S602), identification of a new bounding box B (S603), acquisition of image information of the face area (S604), orientation information and feature Acquisition of information (S605), comparison of feature information (S606), determination of matching of feature information (S607), identification of bounding box B (S608), and the like are included.

Determination of Indefinite Bounding Box B (S601): Identification processing II based on feature information is premised on the existence of an indefinite bounding box. Therefore, the identification processing unit 28 determines whether or not there is a bounding box B that is indefinite in the current frame. If there is no indefinite bounding box B (NO in S601), this process is terminated and the process returns to identification judgment 1 (S105: FIG. 5). If there is a bounding box B that is indefinite (YES in S601), the process proceeds to S602.

Judgment of previous frame (S602): It is judged whether or not there is a bounding box B that is indeterminate in the previous frame. If there is no bounding box B which was made indefinite in the previous frame (NO in S602), the process proceeds to S603.

Identification of a new bounding box (S603): If there is no bounding box B which was made indeterminate in the previous frame (NO in S602), the bounding box B in the current frame is set as a new bounding box B (S603), and this process is performed. , and returns to the identification judgment (S105: FIG. 5). Also, if there is a bounding box B that is indeterminate in the previous frame (YES in S602), the process proceeds to S604.

Acquisition of image information of face area (S604): If bounding box B which is determined to be indefinite exists (YES in S602), image information of face area is acquired from the image information in bounding box B. FIG.

Acquisition of Orientation Information and Feature Information (S605): Orientation information and feature information are acquired from the acquired image information (face image).

Comparison of feature information (S606): The acquired feature information and the feature information of the bounding box B which was made indefinite in the previous frame are compared with each other, which are categorized for each direction information.

Match determination of feature information (S607): It is determined whether or not the feature information matches. to return to the identification judgment (S105: FIG. 5). The match determination of the feature information may be performed based on, for example, the number of matching feature amounts.

Identification processing of bounding box B (S608): The identification processing unit 28 identifies the tracking target P by identifying the bounding box B of the current frame with the bounding box B of the previous frame, ends this processing, and makes an identification judgment (S105). : Return to Fig. 5).

<Effects of the first embodiment>
According to this first embodiment, one of the following effects can be obtained.
(1) Since the tracking target P is identified using the two-dimensional image information and the depth information obtained from the tracking target P and the position information including the coordinate values, the tracking target P can be identified with high accuracy.

(2) If the tracking target P cannot be identified by the identification processing I using the position information, the identification processing II is performed using the feature information acquired from the image information of the tracking target P, so the identification processing II complements the identification function. and the identification accuracy of the tracked object P can be improved.

(3) By dividing the bounding box and thinning out the position information, the processing information can be lightened, and the identification processing can be speeded up and the processing load for identification can be reduced.

(4) The tracking status of the tracked object can be presented by the tracking display. Further, the tracking display T can express the identification state of the tracked object during tracking by coloring. By confirming the change in coloration of the tracking display T, it is possible to easily recognize whether the identification is in the identification state or the identification failure, and whether the tracking is in progress or the tracking has failed.

[Second embodiment]
FIG. 17 shows an object tracking system 2 according to a second embodiment. In FIG. 17, the same parts as in FIG. 3 are denoted by the same reference numerals.

This object tracking system 2 includes a plurality of devices 38-1, 38-2, . . . , 38-n, as shown in FIG. -n are connected to a server 36 by wire or wirelessly through a network 34;

Tracking information obtained by devices 38-1, 38-2, . Used for tracking P-2, . . . , Pn.

<Effects of Second Embodiment>
According to this second embodiment, one of the following effects can be obtained.
(1) The tracking range can be expanded.

FIG. 18 shows the extension of the tracking range by two devices 38-1, 38-2. If the angle of view of the device 38-1 is A-1 (=tracking range) and the angle of view of the device 38-2 is A-2 (=tracking range), then a plurality of tracking targets P-1, P-2, . , P-7, the tracking targets P-1 and P-2 can be tracked at the angle of view A-1, and the tracking targets P-3 and P-4 outside the angle of view A-1 are at the angle of view A-2. can be tracked with Also, the tracking targets P-5, P-6, and P-7 can be tracked at the angle of view A-2. Therefore, the angles of view A-1 and A-2 are enlarged, and the tracking targets P-1, P-2, . . . , P-7 can be tracked. Since these pieces of tracking information are collected by the server 36, it is possible to expand the tracking range and complement the tracking.

(2) The tracking range can be widened and the tracking range can be enclosed.
FIG. 19 illustrates extended tracking range and compound tracking with four devices 38-1, 38-2, 38-3, 38-4. In this case, in a tracking area 106 such as a waiting room of a hospital, a number of tracked objects, such as tracked objects P-11, P-12, . -24, ..., P-51, P-52, ..., P-54 are staying.

For this tracking area 106, four devices 38-1, 38-2, 38-3, 38-4 are arranged surrounding this tracking area 106. FIG. If the angle of view of device 38-1 is A-11, the angle of view of device 38-2 is A-12, the angle of view of device 38-3 is A-13, and the angle of view of device 38-4 is A-14, , a plurality of angles of view A-11, A-12, A-13, and A-14, and a tracking area 106 is set as a tracking range.

, P-14, P-21, P-22, . . . , P-24, . , P-51, P-52, . . . , P-54 can all be tracked.

, P-14, P-21, P-22, . . . , P-24, . 51, P-52, . can be done quickly.

[Other embodiments]
(1) In the above embodiment, identification processing I and identification processing II are included, and identification processing II is executed when identification processing I fails to identify a tracking target, and identification processing II complements identification processing I. On the other hand, the identification process I and the identification process II may be executed simultaneously, and if identification can be made by one of them, the identification process II may be preceded. Alternatively, the identification process I and the identification process II may be executed simultaneously to identify the tracking object by both identifications.

(2) In the above embodiment, when an overlap occurs between the bounding box of the tracked object and the bounding box of another tracked object, the feature information immediately before the overlap and the feature information at the time of cancellation of the overlap Although the process of identifying the tracked target by comparing it with the information is performed, the same process may be performed when an overlap occurs between the images of the tracked target being tracked and another tracked target.

(3) In the above embodiment, a person was exemplified as the tracking target P, but it can also be used to track objects other than humans. It can also be used for tracking objects such as robots, mobile objects, and tree felling.

(4) Other information processing terminals may be used for the device 38, and the tracking program of the present disclosure is not limited to a specific existing program.

(5) During tracking, the tracking information may be presented with the tracking target P and the tracking display T colored differently from the background color.

(6) The object tracking system 2 shown in FIG. A system that separately includes the functional unit such as the distance unit 8 and the processing unit 4 may be used, or a system having different installation locations may be used.

As described above, the present disclosure has been described with respect to the most preferred embodiments. The disclosure is not limited to the above description. Various modifications and changes can be made by those skilled in the art based on the gist described in the claims or disclosed in the detailed description. It goes without saying that such variations and modifications are included within the scope of the present disclosure.

According to the present disclosure, identification is performed using position information including coordinate values using two-dimensional image information and depth information obtained from a tracked target, and identification is performed using feature information of the tracked target obtained from the image information. In addition, by dividing the bounding box, it is possible to speed up identification and reduce the processing load by reducing the weight of the tracking information.

P tracking target B bounding box T tracking display 2 object tracking system 4 processing unit 6 camera 8 light detection/ranging unit 10 information presentation unit 12 tracking control unit 14 image information acquisition unit 16 depth information acquisition unit 18 bounding box processing unit 20 tracking Information database generation unit 22 Position information processing unit 24 Feature information processing unit 26 State information processing unit 28 Identification processing unit 30 Identification information presentation unit 32 Coordination processing unit 34 Network 36 Server 38 Device 40 Apparatus body 41 Information acquisition unit 45 Display 46 Display screen Section 48 Touch panel 56 Operation input section 58 Processor 60 Storage section 62 Input/output section (I/O)
64 communication unit 66 tracking information database 67-1, 67-2, . Classification information part 82 Orientation information part 82-1 Front part 82-2 Left part 82-3 Right part 82-4 Indefinite part 84 State information part 84-1 Movement distance information part 84-2 Overlap information part 86 Identification information part 88 History Information section 100 Orientation information table 102 Acquisition part information section 102-1 Left eye section 102-2 Right eye section 102-3 Left ear section 102-4 Right ear section 104 Orientation information section 106 Tracking area

Claims (15)

a step of acquiring image information and depth information in time series by imaging from a tracked object, and acquiring position information including three-dimensional coordinate values in time series from the image information and the depth information;
a step of acquiring direction information of the tracked object and feature information of the face area of the tracked object from the image information in time series, classifying the acquired feature information according to the direction information, and storing the acquired feature information in a database;
identifying the tracked object using the location information;
If the tracked target cannot be identified by the acquired image information and the position information based on the depth information, the direction information of the tracked target and the feature information of the face region are acquired from the image information, and the acquired feature information and comparing the feature information classified by the orientation information in the database to identify the tracked object;
An object tracking method, comprising: Further, during tracking, the step of comparing the current location information and the latest location information of the tracked object in time series to calculate the moving distance of the tracked object;
identifying the tracked object if the distance traveled is within a threshold;
2. The object tracking method of claim 1, comprising: Furthermore, if an overlap occurs between the tracked object being tracked and the image of another tracked object, or between the bounding box of the tracked object and the bounding box of the other tracked object, the feature just before the overlap 3. The object tracking method according to claim 1, further comprising a step of identifying a tracked object by comparing the information with the feature information at the time when the overlap is eliminated. a step of acquiring facial region information from the image information in chronological order, and acquiring at least eye or ear region information from the facial region information;
a step of acquiring the orientation information of the tracked object using the part information, classifying the feature information according to the orientation information, and storing the classified information in the database;
obtaining the orientation information of the tracked target using the part information obtained from the image information when the tracked target cannot be identified from the position information, and obtaining the feature information of the face region from the image information; comparing the obtained feature information with the feature information classified by the orientation information in the database to identify the tracked object;
The object tracking method according to any one of claims 1 to 3, comprising: 4. The object tracking method according to claim 3, further comprising the step of dividing said bounding box to form a grid of multiple rows and multiple columns, and acquiring said position information specified by coordinate values of said grid. 5. The object tracking method according to claim 4, further comprising the step of dividing the bounding box to form a grid of multiple rows and multiple columns, and acquiring the part information specified by the coordinate values of the grid. A program executed by a computer,
A function of acquiring image information and depth information in time series by imaging from a tracked object, and acquiring position information including three-dimensional coordinate values in time series from the image information and the depth information;
a function of acquiring orientation information of the tracked object and feature information of the face area of the tracked object from the image information in time series, classifying the acquired feature information according to the orientation information, and storing the acquired feature information in a database;
the ability to identify the tracked object using the location information;
If the tracked target cannot be identified by the acquired image information and the position information based on the depth information, the direction information of the tracked target and the feature information of the face region are acquired from the image information, and the acquired feature information and a function of comparing the feature information classified by the orientation information in the database to identify the tracked object;
by the computer. Furthermore, during tracking, a function of comparing the current location information of the tracked object and the latest location information in chronological order to calculate the moving distance of the tracked object;
A function of identifying the tracked object if the movement distance is within a threshold;
8. The program according to claim 7, for executing by said computer. Furthermore, if an overlap occurs between the tracked object being tracked and the image of another tracked object, or between the bounding box of the tracked object and the bounding box of the other tracked object, the feature just before the overlap 9. The program according to claim 7 or 8, for causing the computer to execute a function of identifying a tracked object by comparing the information with the feature information at the time when the overlap is eliminated. Furthermore, a function of acquiring face region information in time series from the image information and acquiring at least eye or ear region information from the face region information;
a function of acquiring the orientation information of the tracked target using the part information, classifying the feature information according to the orientation information, and storing the feature information in the database;
obtaining the orientation information of the tracked target using the part information obtained from the image information when the tracked target cannot be identified from the position information, and obtaining the feature information of the face region from the image information; a function of comparing the acquired feature information with the feature information classified by the orientation information of the database to identify the tracked object;
10. The program according to any one of claims 7 to 9, for executing by said computer. 10. The computer according to claim 9, wherein the bounding box is further divided to form a grid of multiple rows and multiple columns, and the function of acquiring the position information specified by the coordinate values of the grid is executed by the computer. program. 11. The program according to claim 10, further causing the computer to execute a function of dividing the bounding box to form a grid of multiple rows and multiple columns and acquiring the part information specified by the coordinate values of the grid. . an image information acquisition unit that acquires at least two-dimensional image information representing a tracked object in time series;
a depth information acquisition unit that acquires depth information of the tracked object in time series;
a feature information acquisition unit that acquires orientation information and feature information of the tracked object from the image information;
a database that classifies and stores the acquired feature information according to the direction information;
Obtaining position information of the tracked target using the image information and the depth information, identifying the tracked target using the position information, and determining the orientation of the tracked target from the image information when the position information cannot be used to identify the tracked target. an identification processing unit that acquires information and the feature information of the face region, compares the acquired feature information with the feature information classified by the direction information of the database, and identifies the tracked target;
an information presentation unit that presents tracking information together with image information representing the tracking target;
object tracking system. two or more devices including at least the image information acquisition unit, the depth information acquisition unit, the feature information acquisition unit, and the identification processing unit, and outputting tracking information of a tracked target;
a server that obtains the tracking information from each device and extends the tracking angle or tracking range of the tracked target to track the tracked target;
14. The object tracking system of claim 13, comprising: A recording medium storing the program according to any one of claims 7 to 12.

Images