US20220254033A1 - Movement history change method, storage medium, and movement history change device - Google Patents

Movement history change method, storage medium, and movement history change device Download PDF

Info

Publication number
US20220254033A1
US20220254033A1 US17/542,422 US202117542422A US2022254033A1 US 20220254033 A1 US20220254033 A1 US 20220254033A1 US 202117542422 A US202117542422 A US 202117542422A US 2022254033 A1 US2022254033 A1 US 2022254033A1
Authority
US
United States
Prior art keywords
log
identification information
person
movement history
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US17/542,422
Other languages
English (en)
Inventor
Takahiro Yoshioka
Katsuhisa NAKAZATO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujitsu Ltd
Original Assignee
Fujitsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujitsu Ltd filed Critical Fujitsu Ltd
Assigned to FUJITSU LIMITED reassignment FUJITSU LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: NAKAZATO, KATSUHISA, YOSHIOKA, TAKAHIRO
Publication of US20220254033A1 publication Critical patent/US20220254033A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the embodiments discussed herein are related to a movement history change method, a storage medium, and a movement history change device.
  • the business process analysis involves analyzing a business flow based on data such as personal computer (PC) operation logs, system logs, and sensor logs to find a bottleneck and optimize the business flow.
  • PC personal computer
  • a video analysis apparatus that tracks a person from consecutive image frames captured by a camera is known.
  • a movement history change method for a computer to execute a process includes acquiring identification information of a first object detected at a first location by a sensor and a detection timing of the first object from object detection history information that indicates time-series of detection; identifying identification information of a second object present at the first location at a first timing corresponding to the detection timing in the object movement history information generated from a video; and changing the identification information of the second object to the identification information of the first object in the object movement history information.
  • FIG. 1 is a functional configuration diagram of a movement history change apparatus
  • FIG. 2 is a flowchart of movement history change processing
  • FIG. 3 is a configuration diagram of a business process analysis system
  • FIG. 4 is a first functional configuration diagram of a video analysis apparatus
  • FIG. 5 is a first functional configuration diagram of a collection apparatus
  • FIG. 6 is a first functional configuration diagram of a correction apparatus
  • FIG. 7 is a diagram illustrating a person moving in a monitoring target region
  • FIG. 8 is a diagram illustrating an event list in the business process analysis system
  • FIG. 9 is a diagram illustrating location information
  • FIG. 10 is a diagram illustrating a first business log containing an operation history of a person M
  • FIG. 11 is a diagram illustrating a movement log containing a movement path of the person M
  • FIG. 12 is a diagram illustrating a post-change movement log
  • FIG. 13 is a flowchart of business log generation processing
  • FIG. 14 is a flowchart of first movement log generation processing
  • FIG. 15 is a flowchart of first movement log change processing
  • FIG. 16 is a flowchart of first location determination processing
  • FIG. 17 is a second functional configuration diagram of the video analysis apparatus.
  • FIG. 18 is a second functional configuration diagram of the correction apparatus
  • FIG. 19 is a diagram illustrating a first movement log containing tracking reliability
  • FIG. 20 is a diagram illustrating two persons moving in a monitoring target region
  • FIG. 21 is a diagram illustrating a video in which two persons overlap each other
  • FIG. 22 is a diagram illustrating overlap determination processing
  • FIG. 23 is a diagram illustrating a second business log containing an operation history of a person M
  • FIG. 24 is a diagram illustrating a second movement log containing the tracking reliability
  • FIG. 25 is a diagram illustrating a post-replacement second movement log
  • FIG. 26 is a flowchart of second movement log generation processing
  • FIG. 27 is a flowchart of second location determination processing
  • FIG. 28 is a third functional configuration diagram of the video analysis apparatus.
  • FIG. 29 is a third functional configuration diagram of the correction apparatus.
  • FIG. 30 is a diagram illustrating two persons moving together in a monitoring target region
  • FIG. 31 is a diagram illustrating a third movement log containing the tracking reliability
  • FIG. 32 is a diagram illustrating a post-replacement third movement log
  • FIG. 33 is a flowchart of third movement log generation processing
  • FIG. 34 is a flowchart of third location determination processing
  • FIG. 35 is a configuration diagram of a vehicle management system
  • FIG. 36 is a functional configuration diagram of a video analysis apparatus in the vehicle management system
  • FIG. 37 is a functional configuration diagram of a collection apparatus in the vehicle management system.
  • FIG. 38 is a functional configuration diagram of a correction apparatus in the vehicle management system
  • FIG. 39 illustrates vehicles moving in a monitoring target region
  • FIG. 40 is a diagram illustrating an event list in the vehicle management system
  • FIG. 41 is a diagram illustrating vehicle information
  • FIG. 42 is a diagram illustrating a detection log containing a detection history of a vehicle
  • FIG. 43 is a diagram illustrating a movement log containing movement paths of two vehicles
  • FIG. 44 is a diagram illustrating a post-change movement log containing the movement paths of the two vehicles.
  • FIG. 45 is a flowchart of detection log generation processing
  • FIG. 46 is a flowchart of fourth movement log generation processing
  • FIG. 47 is a flowchart of second movement log change processing
  • FIG. 48 is a flowchart of fourth location determination processing
  • FIG. 49 is a fourth functional configuration diagram of the correction apparatus.
  • FIG. 50 is a diagram illustrating an integrated movement log
  • FIG. 51 is a flowchart of integrated movement log generation processing
  • FIG. 52 is a hardware configuration diagram of an information processing apparatus.
  • a recognition error may occur due to the influence of an obstacle or the like.
  • a movement history of an object is obtained based on a tracking result including a recognition error, the accuracy of the obtained movement history decreases.
  • Such a problem occurs not only in the case of obtaining a movement history of a paper document or a person but also in the case of obtaining movement histories of various objects.
  • an object of the present disclosure is to improve accuracy of a movement history of an object obtained from a video.
  • a recognition error may possibly occur due to noise.
  • a similarity between two person regions does not change much even if a combination of person regions to be associated among frames is changed. Therefore, the accuracy of recognizing the persons decreases.
  • FIG. 1 illustrates a functional configuration example of a movement history change apparatus according to an embodiment.
  • a movement history change apparatus 101 in FIG. 1 includes a storage unit 111 , an identification unit 112 , and a change unit 113 .
  • the storage unit 111 stores object detection history information 121 and object movement history information 122 that is generated from a video.
  • the identification unit 112 and the change unit 113 perform movement history change processing using the object detection history information 121 and the object movement history information 122 .
  • FIG. 2 is a flowchart illustrating an example of the movement history change processing performed by the movement history change apparatus 101 in FIG. 1 .
  • the identification unit 112 acquires identification information of a first object detected at a predetermined location by a sensor and a detection time of the first object from the object detection history information 121 (step 201 )
  • the identification unit 112 identifies identification information of a second object present at the predetermined location at a predetermined time corresponding to the detection time of the first object in the object movement history information 122 (step 202 ).
  • the change unit 113 changes the identified identification information of the second object to the acquired identification information of the first object in the object movement history information 122 (step 203 ).
  • the movement history change apparatus 101 in FIG. 1 is capable of improving the accuracy of a movement history of an object obtained from a video.
  • FIG. 3 illustrates a configuration example of a business process analysis system including the movement history change apparatus 101 in FIG. 1 .
  • the business process analysis system in FIG. 3 includes a business process analysis apparatus 301 , a correction apparatus 302 , a video analysis apparatus 303 , a collection apparatus 304 , an imaging apparatus 305 , and terminal devices 306 - 1 to 306 - 3 .
  • the correction apparatus 302 corresponds to the movement history change apparatus 101 in FIG. 1 .
  • the business process analysis apparatus 301 , the correction apparatus 302 , the video analysis apparatus 303 , and the collection apparatus 304 are able to communicate with each other via a communication network 307 .
  • the communication network 307 is a local area network (LAN) or a wide area network (WAN).
  • a monitoring target region 311 is a workspace in which office work is performed.
  • the terminal device 306 - 3 In the monitoring target region 311 , the terminal device 306 - 3 , a desk 312 - 1 , a desk 312 - 2 , and a shelf 314 are installed.
  • the terminal devices 306 - 1 and 306 - 2 are PCs, and the terminal device 306 - 3 is a printer.
  • the terminal devices 306 - 1 to 306 - 3 are examples of sensors.
  • Four or more terminal devices 306 - i may be installed in the monitoring target region 311 .
  • the imaging apparatus 305 is, for example, a camera having an image sensor such as a charge-coupled device (CCD) and a complementary metal-oxide-semiconductor (CMOS), captures a video of the monitoring target region 311 , and outputs the video to the video analysis apparatus 303 .
  • CCD charge-coupled device
  • CMOS complementary metal-oxide-semiconductor
  • the video analysis apparatus 303 analyzes the video output from the imaging apparatus 305 and generates a movement log indicating a movement path of an object captured in the video.
  • the video analysis apparatus 303 transmits the movement log to the correction apparatus 302 .
  • the object captured in the video is, for example, a person, and the movement log is an example of the object movement history information 122 .
  • the collection apparatus 304 and the terminal devices 306 - i are able to communicate with each other via a communication network 308 .
  • the communication network 308 is a LAN or a WAN.
  • the terminal device 306 - i uses operation log collection software to detect identification information of a person who is seated on the chair 313 - i and is operating the terminal device 306 - i and what type of operation was performed by the person.
  • the terminal device 306 - i transmits the identification information of the person, the detection time, and the operation type to the collection apparatus 304 .
  • the terminal device 306 - 3 detects the identification information of a person who is operating the terminal device 306 - 3 and what type of operation was performed by the person, and transmits the identification information of the person, the detection time, and the operation type to the collection apparatus 304 .
  • the business log is an example of the object detection history information 121 .
  • the correction apparatus 302 uses the business log received from the collection apparatus 304 to correct the movement log received from the video analysis apparatus 303 and transmits the corrected movement log to the business process analysis apparatus 301 .
  • the business process analysis apparatus 301 performs business process analysis by using the movement log received from the correction apparatus 302 , and outputs an analysis result.
  • FIG. 4 illustrates a first functional configuration example of the video analysis apparatus 303 in FIG. 3 .
  • the video analysis apparatus 303 in FIG. 4 includes a storage unit 411 , a communication unit 412 , and a tracking unit 413 .
  • the communication unit 412 receives a video of the monitoring target region 311 from the imaging apparatus 305 .
  • the tracking unit 413 tracks a person captured in the video to generate a movement log 421 indicating a movement path of the person, and stores the movement log 421 in the storage unit 411 .
  • Each entry in the movement log 421 contains the identification information of a person, a time, and a location.
  • the tracking unit 413 is able to track a person in a video by using a learned model generated by machine learning using boosting or the like.
  • the tracking unit 413 is able to determine a location where a person is present by determining whether or not the person enters a region of interest (ROI) set in advance on an image.
  • the communication unit 412 transmits the generated movement log 421 to the correction apparatus 302 .
  • FIG. 5 illustrates a first functional configuration example of the collection apparatus 304 in FIG. 3 .
  • the collection apparatus 304 in FIG. 5 includes a storage unit 511 , a communication unit 512 , and a recording unit 513 .
  • the communication unit 512 receives the identification information of a person, the detection time, and the operation type from the terminal device 306 - i.
  • the recording unit 513 generates a business log 521 containing the received information and stores the business log 521 in the storage unit 511 .
  • the business log 521 is a searchable database.
  • a relational database, an object-oriented database, or the like may be used as the business log 521 .
  • Each entry in the business log 521 contains the identification information of a person, the detection time, the identification information of the terminal device 306 - i, and the operation type.
  • the communication unit 512 transmits the generated business log 521 to the correction apparatus 302 .
  • FIG. 6 illustrates a first functional configuration example of the correction apparatus 302 in FIG. 3 .
  • the correction apparatus 302 in FIG. 6 includes a storage unit 611 , a communication unit 612 , an identification unit 613 , and a change unit 614 .
  • the storage unit 611 , the identification unit 613 , and the change unit 614 correspond to the storage unit 111 , the identification unit 112 , and the change unit 113 in FIG. 1 , respectively.
  • the communication unit 612 receives the business log 521 from the collection apparatus 304 and receives the movement log 421 from the video analysis apparatus 303 .
  • the identification unit 613 stores the received business log 521 and movement log 421 in the storage unit 611 .
  • the storage unit 611 stores an event list 621 and location information 622 .
  • the event list 621 is a list of events each indicating an operation type usable for determining a location contained in the movement log 421 among operation types that may be contained in the business log 521 .
  • the location information 622 is information based on which the identification information of the terminal device 306 - i contained in the business log 521 is associated with a location contained in the movement log 421 .
  • the identification unit 613 acquires the identification information of a first person detected by any of the terminal devices 306 - i, the identification information of the terminal device 306 - i, and the detection time of the first person from the business log 521 based on the events included in the event list 621 . Using the location information 622 , the identification unit 613 identifies the identification information of a second person who was present at the location of the terminal device 306 - i at a predetermined time corresponding to the detection time of the first person in the movement log 421 .
  • the change unit 614 corrects the movement log 421 by changing the identified identification information of the second person to the acquired identification information of the first person in the movement log 421 .
  • the communication unit 612 transmits the corrected movement log 421 to the business process analysis apparatus 301 .
  • FIG. 7 illustrates an example of a person moving in the monitoring target region 311 in FIG. 3 .
  • a solid line 701 indicates a movement path along which a person M who works by using the terminal device 306 - 1 on the desk 312 - 1 temporarily leaves the desk 312 - 1 and moves to the shelf 314 to take a book or the like.
  • a broken line 702 indicates a movement path in a period when the person M stays in front of the shelf 314 .
  • a dashed-dotted line 703 indicates a movement path along which the person M returns to the desk 312 - 1 while holding the book or the like and then moves from the desk 312 - 1 to the terminal device 306 - 3 .
  • the person M performs a log-out operation on the terminal device 306 - 1 when leaving the desk 312 - 1 , and performs a log-in operation on the terminal device 306 - 1 when returning to the desk 312 - 1 .
  • the business log 521 records data containing identification information “E 01 ” of the person M and the operation type “PC log-out” when the person M leaves the desk 312 - 1 , and records data containing “E 01 ” and the operation type “PC log-in” when the person M returns to the desk 312 - 1 .
  • the person M moving along the movement paths indicated by the solid line 701 and the dashed-dotted line 703 is captured.
  • the person M operating the terminal device 306 - 1 is captured in a predetermined region around the terminal device 306 - 1 .
  • the person M on the movement path indicated by the broken line 702 is hidden by the shelf 314 and is not captured.
  • the movement log 421 records “E 01 ” and “desk A” denoting the name of the desk 312 - 1 as data indicating a state where the person M is seated at the desk 312 - 1 on the movement path indicated by the solid line 701 . Thereafter, the person M leaves the desk 312 - 1 and starts moving.
  • the movement log 421 records temporary identification information “R 0001 ” indicating a new person and “shelf Z” denoting the name of the shelf 314 as data indicating a state where the person M appears from the side of the shelf 314 on the movement path indicated by the dashed-dotted line 703 .
  • the person M returns to the desk 312 - 1 to stop moving, logs into the terminal device 306 - 1 , performs editing work for a while, and then starts moving toward the terminal device 306 - 3 .
  • the identification unit 613 of the correction apparatus 302 determines that the person “E 01 ” who performed “PC log-in” at the location “desk A” is the same as the person “R 0001 ” who was present at the location “desk A” at that time. Therefore, the change unit 614 changes all “R 0001 ” consecutively recorded in a section including the detection time of “PC log-in” to “E 01 ” in the movement log 421 .
  • FIG. 8 illustrates an example of the event list 621 illustrated in FIG. 6 .
  • Events contained in the event list 621 in FIG. 8 are “PC log-in”, “security door unlocking”, and “individual shelf unlocking”.
  • an event such as “device unlocking” using a one-time password, a number individually assigned in advance, or the like may be also registered.
  • FIG. 9 illustrates an example of the location information 622 in FIG. 6 .
  • the location information 622 in FIG. 9 indicates an association relationship between a terminal ID and a location.
  • the terminal ID is the identification information of the terminal device 306 - i
  • the location indicates the name of the location where the terminal device 306 - i indicated by the terminal ID is installed.
  • PC- 01 is the identification information of the terminal device 306 - 1
  • “desk A” associated with “PC- 01 ” is the name of the desk 312 - 1 on which the terminal device 306 - 1 is installed
  • “PC- 46 ” is the identification information of the terminal device 306 - 2
  • “desk B” associated with “PC- 46 ” is the name of the desk 312 - 2 on which the terminal device 306 - 2 is installed.
  • FIG. 10 illustrates an example of a first business log 521 containing an operation history of the person M in the monitoring target region 311 in FIG. 7 .
  • the business log 521 illustrated in FIG. 10 indicates an association relationship among the detection time, the personal ID, the terminal ID, and the operation type.
  • the personal ID is identification information, such as an employee number, based on which an individual is identifiable.
  • the detection time specifies a time at which an operation performed on the terminal device 306 - i identified by the terminal ID by the person identified by the personal ID was detected, and the operation type specifies what type of operation was performed.
  • PC lock As the operation type, “PC lock”, “PC log-in”, “PC log-out”, “system activation”, “terminal operation”, “print”, “printer access”, and the like are recorded. “System activation” indicates an operation of activating an application program, “print” indicates an operation of issuing a printout instruction, and “printer access” indicates a printout operation on a printer.
  • the person M identified by “E 01 ” performs “PC lock” on the terminal device 306 - 1 identified by “PC- 01 ” at time “11:30” and performs “PC log-in” on the terminal device 306 - 1 at time “11:40”.
  • the person M performs “system activation” on the terminal device 306 - 1 at time “11:41”.
  • the person M performs “print” on the terminal device 306 - 1 at time “11:55” and performs “PC lock” on the terminal device 306 - 1 at time “11:56”.
  • the person M performs “printer access” on the terminal device 306 - 3 identified by “P- 01 ” at time “11:58”.
  • FIG. 11 illustrates an example of the movement log 421 containing the movement path of the person M in the monitoring target region 311 in FIG. 7 .
  • the movement log 421 in FIG. 11 specifies an association relationship among a track ID, a time, and a location.
  • the track ID is identification information that is generated by tracking processing and is assigned to a person captured in a video.
  • the time indicates a time at which the person was captured, and the location indicates the name of a location at which the captured person was present.
  • “Desk A” denotes the name of the desk 312 - 1
  • “shelf Z” denotes the name of the shelf 314
  • “printer U” denotes the name of the terminal device 306 - 3 .
  • the person M identified by “E 01 ” is present at the location “desk A” at time “11:29” and starts moving from the location “desk A” at time “11:30”.
  • a person identified by “R 0001 ” is present at the location “shelf Z” at time “11:31”, starts moving from the location “desk A” at time “11:56” and is present at the location “printer U” at time “11:58”.
  • the movement log 421 may include not only the names of the locations but also xy coordinates indicating a position in the captured image.
  • the identification unit 613 searches the business log 521 in FIG. 10 for an entry in which any of the events registered in the event list 621 in FIG. 8 is contained as the operation type, and acquires the entry at a detection time “11:32” containing “PC log-in”.
  • the identification unit 613 extracts a personal ID “E 14 ” and a terminal ID “PC- 12 ” from the entry at the detection time “11:32” acquired from the business log 521 , and obtains a location “desk E” associated with “PC- 12 ” by using the location information 622 in FIG. 9 .
  • “Desk E” is an example of a predetermined location.
  • the identification unit 613 acquires, from the movement log 421 in FIG. 11 , an entry at a predetermined time corresponding to the detection time “11:32” of the above-acquired entry.
  • the predetermined time used herein is, for example, a time within a predetermined period including the detection time or a time immediately before the detection time.
  • an entry at time “11:31” is acquired as the entry at the predetermined time.
  • the identification unit 613 extracts the track ID “R 0001 ” and the location “shelf Z” from the entry.
  • the identification unit 613 searches the business log 521 for a next entry in which any of the events registered in the event list 621 is contained as the operation type, and acquires an entry at a detection time “11:40” containing “PC log-in”.
  • the identification unit 613 extracts the personal ID “E 01 ” and the terminal ID “PC- 01 ” from the entry at the detection time “11:40” acquired from the business log 521 , and obtains the location “desk A” associated with “PC- 01 ” by using the location information 622 .
  • “Desk A” is an example of the predetermined location.
  • the identification unit 613 acquires an entry at time “11:38” immediately before the detection time “11:40” from the movement log 421 , and extracts the track ID “R 0001 ” and the location “desk A” from the entry.
  • “Desk A” obtained from the entry in the business log 521 is the same as “desk A” extracted from the entry in the movement log 421 .
  • “E 01 ” extracted from the entry in the business log 521 is different from “R 0001 ” extracted from the entry in the movement log 421 .
  • the identification unit 613 determines that the person “R 0001 ” who was present at the location “desk A” at time “11:38” is the same as the person “E 01 ” who performed “PC log-in” at the location “desk A” at the detection time “11:40”.
  • the change unit 614 changes all “R 0001 ” recorded in a period from time “11:31” to time “11:58” including the predetermined time “11:38” to “E 01 ”.
  • FIG. 12 illustrates an example of a post-change movement log 421 .
  • the track IDs at time “11:31”, time “11:38”, time “11:56” and time “11:58” are changed from “R 0001 ” to “E 01 ”.
  • the business process analysis apparatus 301 searches the movement log 421 in FIG. 12 for the times and locations associated with the track IDs “E 01 ” of an analysis target, and thereby generates a movement path of the person M including the searched-out times and locations.
  • the business process analysis apparatus 301 detects a bottleneck in the work of the person M by using the generated movement path, and optimizes the business flow of the person M.
  • the business process analysis system in FIG. 3 identifies a location where the actions of the persons match each other between the business log 521 and the movement log 421 , and corrects the movement log 421 such that the track ID at the location is changed to the personal ID in the business log 521 . Therefore, even in the case where the movement log 421 is generated from a low-quality video in which it is difficult to identify a person or where tracking of a person in the video was interrupted because the person was hidden, it is possible to correct an error and improve the accuracy of the movement log 421 .
  • the monitoring target region 311 is not limited to an office, and may be a manufacturing site or the like where a work terminal is associated with a specific person by the time.
  • the business process analysis is performed by using the corrected movement log 421 , which leads to an improvement of the efficiency of the processing of searching for the times and the locations associated with the track IDs of an analysis target.
  • FIG. 13 is a flowchart illustrating an example of business log generation processing performed by the collection apparatus 304 in FIG. 5 .
  • the recording unit 513 checks whether an end instruction is input from a user (step 1301 ).
  • the recording unit 513 acquires the detection time of an operation detected by any of the terminal devices 306 - i via the communication unit 512 (step 1302 ). From the terminal device 306 - i, the recording unit 513 acquires the personal ID of the person who performed the operation (step 1303 ), and acquires the operation type of the operation (step 1304 ).
  • the recording unit 513 generates an entry containing the terminal ID of the terminal device 306 - i and the detection time, the personal ID, and the operation type acquired from the terminal device 306 - i and records the entry in the business log 521 (step 1305 ).
  • the recording unit 513 iterates the processing in step 1301 and the subsequent steps.
  • the end instruction is input (step 1301 , YES)
  • the recording unit 513 ends the processing.
  • FIG. 14 is a flowchart illustrating an example of first movement log generation processing performed by the video analysis apparatus 303 in FIG. 4 .
  • the tracking unit 413 checks whether an end instruction is input from a user (step 1401 ).
  • the tracking unit 413 tracks a person captured in the video (step 1402 ) and checks whether the tracking is successful (step 1403 ).
  • the tracking unit 413 adds the name of a location indicated by a ROI that the person entered in the image (step 1404 ).
  • the tracking unit 413 generates an entry of the person containing the track ID, the time, and the location and records the entry in the movement log 421 (step 1405 ).
  • the tracking unit 413 iterates the processing in step 1401 and the subsequent steps.
  • the tracking unit 413 detects a new person (step 1406 ), assigns a new track ID to the detected person (step 1407 ), and performs the processing in step 1404 and the subsequent step.
  • the tracking unit 413 ends the processing.
  • FIG. 15 is a flowchart illustrating an example of first movement log change processing performed by the correction apparatus 302 in FIG. 6 .
  • the identification unit 613 checks whether an end instruction is input from a user (step 1501 ).
  • the identification unit 613 acquires, via the communication unit 612 , the business log 521 from the collection apparatus 304 (step 1502 ) and the movement log 421 from the video analysis apparatus 303 (step 1503 ).
  • the identification unit 613 searches the acquired business log 521 for the operation type indicating any of the events registered in the event list 621 and thereby checks whether the business log 521 contains an event (step 1504 ).
  • the identification unit 613 makes location determination to determine whether to correct the acquired movement log 421 (step 1505 ).
  • the identification unit 613 outputs a correction instruction indicating a change target track ID and a post-change track ID to the change unit 614 .
  • the change unit 614 checks whether the correction instruction is output from the identification unit 613 (step 1506 ).
  • the change unit 614 corrects the movement log 421 by changing the track ID of the change target specified in the correction instruction to the post-change track ID specified therein in the movement log 421 (step 1507 ).
  • the correction apparatus 302 iterates the processing in step 1501 and the subsequent steps.
  • step 1504 When the business log 521 contains no event (step 1504 , NO), the correction apparatus 302 iterates the processing in step 1501 and the subsequent steps. When the correction instruction is not output (step 1506 , NO), the correction apparatus 302 iterates the processing in step 1501 and the subsequent steps. When the end instruction is input (step 1501 , YES), the correction apparatus 302 ends the processing.
  • FIG. 16 is a flowchart illustrating an example of first location determination processing in step 1505 in FIG. 15 .
  • the identification unit 613 acquires an entry containing an event as the operation type from the business log 521 , and extracts the detection time, the personal ID, and the terminal ID from the entry (step 1601 ).
  • the identification unit 613 obtains a location X 1 associated with the extracted terminal ID by using the location information 622 (step 1602 ).
  • the identification unit 613 acquires, from the movement log 421 , an entry at the predetermined time corresponding to the detection time extracted from the entry in the business log 521 , and extracts a track ID and a location X 2 from the entry (step 1603 ).
  • the identification unit 613 compares the location X 1 with the location X 2 (step 1604 ).
  • the identification unit 613 compares the personal ID extracted from the entry in the business log 521 with the track ID extracted from the entry in the movement log 421 (step 1605 ).
  • the identification unit 613 determines to correct the movement log 421 , and outputs a correction instruction specifying the track ID of the change target and the post-change track ID to the change unit 614 (step 1606 ).
  • the track ID of the change target covers all the same track IDs as the track ID extracted from the entry in the movement log 421
  • the post-change track ID is the personal ID extracted from the entry in the business log 521 .
  • the identification unit 613 determines not to correct the movement log 421 , and ends the processing without outputting the correction instruction.
  • the identification unit 613 determines not to correct the movement log 421 , and ends the processing without outputting the correction instruction.
  • FIG. 17 illustrates a second functional configuration example of the video analysis apparatus 303 in FIG. 3 .
  • the video analysis apparatus 303 in FIG. 17 has a configuration in which a reliability assignment unit 1711 is added to the video analysis apparatus 303 in FIG. 4 .
  • the reliability assignment unit 1711 assigns tracking reliability to a person tracked at each time in the video.
  • the tracking unit 413 generates a movement log 1721 containing the tracking reliability and stores the movement log 1721 in the storage unit 411 .
  • Each entry in the movement log 1721 contains the track ID, the tracking reliability, the time, and the location.
  • FIG. 18 illustrates a second functional configuration example of the correction apparatus 302 in FIG. 3 .
  • the correction apparatus 302 in FIG. 18 has the same configuration as that of the correction apparatus 302 in FIG. 6 .
  • the storage unit 611 stores the movement log 1721 in FIG. 17 instead of the movement log 421 in FIG. 6 .
  • the identification unit 613 and the change unit 614 perform the movement log change processing using the movement log 1721 .
  • FIG. 19 illustrates an example of a first movement log 1721 containing the tracking reliability for the person M moving along the movement path illustrated in FIG. 7 .
  • Entries 1901 to 1905 represent entries in the business log 521 in which the terminal ID is omitted.
  • Entries 1911 to 1914 represent entries in the movement log 1721 in which the time is omitted.
  • the entry 1911 in the movement log 1721 is generated.
  • the tracking reliability of the entry 1911 is set to 100%.
  • the person M performs “PC lock” on the terminal device 306 - 1 on the desk 312 - 1 and accordingly the entry 1901 in the business log 521 is generated.
  • the person M temporarily leaves the desk 312 - 1 to move to the shelf 314 , is hidden behind the shelf 314 , and then appears from the side of the shelf 314 .
  • the entry 1912 in the movement log 1721 is generated.
  • the tracking of the person M fails and new identification information “R 0001 ” is assigned. For this reason, the tracking reliability of the entry 1912 is set to 0%.
  • the entry 1913 in the movement log 1721 is generated.
  • the tracking reliability of the entry 1913 is set to 10%.
  • the person M performs “PC log-in” on the terminal device 306 - 1 and accordingly the entry 1902 in the business log 521 is generated.
  • the person M performs “system activation” on the terminal device 306 - 1 and accordingly the entry 1903 in the business log 521 is generated.
  • the person M performs editing work.
  • the person M performs “print” on the terminal device 306 - 1 to issue a printout instruction and accordingly the entry 1904 in the business log 521 is generated.
  • the person M performs “PC lock” on the terminal device 306 - 1 and accordingly the entry 1905 in the business log 521 is generated.
  • the person M leaves the desk 312 - 1 again and moves to the terminal device 306 - 3 .
  • the identification unit 613 of the correction apparatus 302 acquires the entry 1913 in the movement log 1721 generated immediately before the entry 1902 .
  • the identification unit 613 determines that the person “R 0001 ” who was seated at the desk 312 - 1 indicated by the location “desk A” in the entry 1913 is the person “E 01 ” who performed “PC log-in” on the terminal device 306 - 1 on the desk 312 - 1 .
  • the change unit 614 changes “R 0001 ” recorded in the entries 1912 and 1913 in the movement log 1721 to “E 01 ”. Since it is confirmed that the person “R 0001 ” is the person “E 01 ”, the change unit 614 changes the tracking reliability of the entries 1912 and 1913 to 100%.
  • the person M is tracked as the person “E 01 ”, and the entry 1914 in the movement log 1721 is generated when the person M is present at the location of the terminal device 306 - 3 .
  • the tracking reliability of the entry 1914 is set to 100%.
  • FIG. 20 illustrates an example of two persons moving in the monitoring target region 311 in FIG. 3 .
  • a solid line 2001 indicates a movement path along which a person M who works using the terminal device 306 - 1 on the desk 312 - 1 temporarily leaves the desk 312 - 1 and moves to a point P.
  • a broken line 2002 indicates a movement path along which the person M returns to the desk 312 - 1 after having conversations with a person L at the point P.
  • a dashed-dotted line 2003 indicates a movement path along which the person L moves from a point Q to the point P.
  • a dashed-two dotted line 2004 indicates a movement path along which the person L returns to the point Q after having conversations with the person M at the point P. In this case, the person M and the person L are present at the point P at the same time.
  • the two persons are present at the same location at the same time as described above, the persons overlapping each other are seen in the video. Since the person M and the person L overlap each other at the point P, the tracking reliability for the person M and the person L decreases, and it is erroneously determined that the person L moves from the point P to the desk 312 - 1 and the person M moves from the point P to the point Q.
  • the person M performs a log-out operation on the terminal device 306 - 1 when leaving the desk 312 - 1 , and performs a log-in operation on the terminal device 306 - 1 when returning to the desk 312 - 1 .
  • the business log 521 records data containing identification information “E 01 ” of the person M and the operation type “PC log-out” when the person M leaves the desk 312 - 1 , and records data containing “E 01 ” and the operation type “PC log-in” when the person M returns to the desk 312 - 1 .
  • E 01 ” and “point P” are recorded as data indicating a state where the person M has conversations with the person L at the point P. Since the person M and the person L overlap each other at the point P, the tracking reliability of this data is set to 50%. After the conversations end, the person M returns to the desk 312 - 1 by moving along the movement path indicated by the broken line 2002 . However, if the person M is replaced with the person L, the movement path indicated by the dashed-two dotted line 2004 is erroneously associated with “E 01 ”.
  • “R 0001 ” and “point P” are recorded as data indicating a state where the person L has conversations with the person M at the point P.
  • the tracking reliability of this data is set to 0%.
  • the person L returns to the point Q by moving along the movement path indicated by the dashed-two dotted line 2004 .
  • the movement path indicated by the broken line 2002 is erroneously associated with “R 0001 ”.
  • the movement log 1721 records a situation in which the person identified by “R 0001 ” returns to “desk A” and stops moving on the movement path indicated by the broken line 2002 , and then leaves “desk A” again to start moving.
  • the identification unit 613 of the correction apparatus 302 determines that the person “E 01 ” who performed “PC log-in” at the location “desk A” is the same as the person “R 0001 ” who was present at the location “desk A” at that time. Since the person “E 01 ” and the person “R 0001 ” overlapped each other at the point P, the identification unit 613 determines that the movement paths of the person “E 01 ” and the person “R 0001 ” after that time are replaced with each other.
  • the change unit 614 replaces “R 0001 ” and “E 01 ”, which were recorded after the time when the person “E 01 ” and the person “R 0001 ” overlapped each other at the point P, with each other.
  • the point P is an example of a specific location
  • the time when the person “E 01 ” and the person “R 0001 ” overlapped each other at the point P is an example of a specific time.
  • the overlap determination processing is processing of determining whether or not two persons overlap each other in each frame included in the video.
  • the reliability assignment unit 1711 decreases the tracking reliability for each of two persons to 1 / 2 every time the two persons overlap each other.
  • FIG. 21 illustrates an example of a video in which two persons overlap each other.
  • a person 2101 moves to a point R along a movement path 2111 , and moves from the point R along a movement path 2112 .
  • a person 2102 moves to the point R along a movement path 2121 , and moves from the point R along a movement path 2122 .
  • the person 2101 and the person 2102 overlap each other at the point R.
  • Boundary boxes 2131 to 2133 are person tracking frames of the person 2101 .
  • Boundary boxes 2141 to 2143 are person tracking frames of the person 2102 .
  • the boundary box 2132 and the boundary box 2142 also overlap each other.
  • the tracking reliability assigned to the person 2101 at the position of the boundary box 2131 is 100%, the tracking reliability decreases to 50% at the position of the boundary box 2132 because the person 2101 and the person 2102 overlap each other. Accordingly, the tracking reliability assigned to the person 2101 at the position of the boundary box 2133 is also 50%.
  • the tracking reliability assigned to the person 2102 at the position of the boundary box 2141 is 100%
  • the tracking reliability decreases to 50% at the position of the boundary box 2142 because the person 2101 and the person 2102 overlap each other. Accordingly, the tracking reliability assigned to the person 2102 at the position of the boundary box 2143 is also 50%.
  • the person 2101 and the person 2102 are erroneously recognized at the point R, the person 2101 and the person 2102 are replaced with each other in the following tracking processing.
  • the movement path 2112 is erroneously associated with the person 2102
  • the movement path 2122 is erroneously associated with the person 2101 .
  • FIG. 22 illustrates an example of overlap determination processing using boundary boxes.
  • a boundary box 2201 is a boundary box of one of two persons, and a boundary box 2202 is a boundary box of the other person.
  • An overlap region 2203 represents a portion where the boundary box 2201 and the boundary box 2202 overlap each other.
  • x and y coordinates of a vertex of the boundary box are denoted by (x, y).
  • the coordinates of an upper left vertex 2211 are denoted by (ax 1 , ay 1 ) and the coordinates of a lower right vertex 2212 are denoted by (ax 2 , ay 2 ).
  • the coordinates of an upper left vertex 2221 are denoted by (bx 1 , by 1 ) and the coordinates of a lower right vertex 2222 are denoted by (bx 2 , by 2 ).
  • the tracking unit 413 calculates a width W and a height H of the overlap region 2203 in accordance with the following expressions using the coordinates of the vertices 2211 , 2212 , 2221 , and 2222 .
  • max(a, b) denotes the maximum value of a and b
  • min(a, b) denotes the minimum value of a and b.
  • FIG. 23 illustrates an example of a second business log 521 containing an operation history of the person M in the monitoring target region 311 in FIG. 20 .
  • the person M identified by “E 01 ” performs “PC lock” on the terminal device 306 - 1 identified by “PC- 01 ” at time “11:30” and performs “PC log-in” on the terminal device 306 - 1 at time “11:37”.
  • FIG. 24 illustrates an example of a second movement log 1721 containing the tracking reliability for the persons M and L in the monitoring target region 311 in FIG. 20 .
  • the movement log 1721 in FIG. 24 indicates an association relationship among the track ID, the tracking reliability, the time, and the location.
  • the tracking reliability decreases every time the person identified by the track ID overlaps another person.
  • the person “E 01 ” is present at a location “desk A” at time “11:29” and starts moving from the location “desk A” at time “11:30”.
  • the person “E 01 ” is present at “point P” at time “11:31” and is present at “point Q” at time “11:35”.
  • the tracking reliability for the person “E 01 ” is 100% at time “11:29” and time “11:30”. However, since the person “E 01 ” overlaps the person “R 0001 ” at “point P” at time “11:31”, the tracking reliability for the person “E 01 ” decreases to 50% at time “11:31” and time “11:35”.
  • the person “R 0001 ” is present at “point Q” at time “11:29”, is present at “point P” at time “11:31”, and is present at the location “desk A” at time “11:36”.
  • the tracking reliability for the person “R 0001 ” is 0% at time “11:29”, time “11:31”, and time “11:36”.
  • the identification unit 613 searches the business log 521 in FIG. 23 for an entry containing any of the events registered in the event list 621 in FIG. 8 as the operation type, and acquires an entry at a detection time “11:37” containing “PC log-in”.
  • the identification unit 613 extracts the personal ID “E 01 ” and the terminal ID “PC- 01 ” from the entry at the detection time “11:37” acquired from the business log 521 , and obtains the location “desk A” associated with “PC- 01 ” by using the location information 622 in FIG. 9 .
  • the identification unit 613 acquires, from the movement log 1721 in FIG. 24 , an entry at a predetermined time corresponding to the detection time “11:37” of the acquired entry, the entry containing “desk A” as the location.
  • an entry at time “11:36” is acquired as the entry at the predetermined time containing “desk A”.
  • the identification unit 613 extracts the track ID “R 0001 ” and the tracking reliability “0%” from the entry.
  • the identification unit 613 compares the extracted tracking reliability “0%” with a predetermined value. For example, when the predetermined value is “60%”, the extracted tracking reliability “0%” is lower than the predetermined value. “E 01 ” extracted from the entry in the business log 521 is different from “R 0001 ” extracted from the entry in the movement log 1721 .
  • the identification unit 613 determines that the person “R 0001 ” present at the location “desk A” at time “11:36” is the same as the person “E 01 ” who performed “PC log-in” at the location “desk A” at the detection time “11:37”.
  • the identification unit 613 obtains time “11:31” at which the person “E 01 ” and the person “R 0001 ” overlapped at the point P in the movement log 1721 .
  • the tracking reliability of all the entries in a section from time “11:31” to time “11:36” is lower than “60%”. Therefore, the change unit 614 replaces “R 0001 ” and “E 01 ” included in this section with each other.
  • the track IDs of entries each having the tracking reliability lower than the predetermined value are replaced with each other, which makes it possible to inhibit a correct track ID from being erroneously changed.
  • FIG. 25 illustrates an example of a post-replacement second movement log 1721 .
  • “R 0001 ” at time “11:31” and time “11:36” is changed to “E 01 ”, and the tracking reliability thereof is changed to “100%”.
  • “E 01 ” at time “11:31” and time “11:35” is changed to “R 0001 ”, and the tracking reliability thereof is changed to “0%”.
  • the business process analysis system including the video analysis apparatus 303 in FIG. 17 and the correction apparatus 302 in FIG. 18 is capable of correcting the error and improving the accuracy of the movement log 1721 .
  • the business process analysis is performed by using the corrected movement log 1721 , which leads to an improvement of the efficiency of the processing of searching for the times and the locations associated with the track IDs of an analysis target.
  • FIG. 26 is a flowchart illustrating an example of second movement log generation processing performed by the video analysis apparatus 303 in FIG. 17 .
  • the processing in steps 2601 to 2603 , 2606 , 2608 , and 2609 is the same as the processing in steps 1401 to 1404 , 1406 , and 1407 in FIG. 14 .
  • the tracking unit 413 performs the overlap determination processing on the person identified by the track ID (step 2604 ), and the reliability assignment unit 1711 sets the tracking reliability for the person in accordance with a result of the overlap determination processing (step 2605 ).
  • the reliability assignment unit 1711 sets the value of 1 ⁇ 2 of the immediately preceding tracking reliability when the person identified by the track ID overlaps another person, or sets the immediately preceding tracking reliability when the person identified by the track ID does not overlap another person.
  • the tracking unit 413 adds the name of the location indicated by the ROI that the person enters in the image (step 2606 ), generates an entry of the person containing the track ID, the tracking reliability, the time, and the location, and records the entry in the movement log 1721 (step 2607 ).
  • the tracking unit 413 detects a new person (step 2608 ) and assigns a new track ID to the detected person (step 2609 ).
  • the reliability assignment unit 1711 sets the tracking reliability “0%” for the person (step 2610 ), and the video analysis apparatus 303 performs the processing in step 2606 and the subsequent step.
  • the correction apparatus 302 in FIG. 18 performs the same movement log change processing as in FIG. 15 except for step 1505 in FIG. 15 in which the correction apparatus 302 performs location determination processing different from that in FIG. 16 .
  • FIG. 27 is a flowchart illustrating an example of second location determination processing performed by the correction apparatus 302 in FIG. 18 .
  • the processing in steps 2701 and 2702 is the same as the processing in steps 1601 and 1602 in FIG. 16 .
  • the identification unit 613 acquires, from the movement log 1721 , an entry at a predetermined time corresponding to the detection time of the entry acquired from the business log 521 , the entry containing the location X 1 .
  • the identification unit 613 extracts the track ID and the tracking reliability from the entry (step 2703 ).
  • the track ID extracted from the entry in the movement log 1721 in step 2703 will be referred to as a first track ID.
  • the identification unit 613 compares the extracted tracking reliability with a predetermined value (step 2704 ).
  • the tracking reliability is lower than the predetermined value (step 2704 , YES)
  • the identification unit 613 compares the personal ID extracted from the entry in the business log 521 with the first track ID (step 2705 ).
  • the identification unit 613 determines to correct the movement log 1721 .
  • the identification unit 613 searches the movement log 1721 for another track ID associated with the specific location that is the same as that of the first track ID at a specific time T before the first track ID (step 2706 ).
  • the other track ID searched out from the movement log 1721 in step 2706 will be referred to as a second track ID.
  • the identification unit 613 confirms that the tracking reliability associated with each of the first track ID and the second track ID contained in a section from the specific time T to the predetermined time is lower than the predetermined value.
  • the identification unit 613 outputs a correction instruction indicating a correction target section and replacement target track IDs to the change unit 614 (step 2707 ).
  • the correction target section specifies the section from the specific time T to the predetermined time, and the replacement target track IDs specify the first track ID and the second track ID.
  • the identification unit 613 determines not to correct the movement log 1721 , and ends the processing without outputting the correction instruction.
  • the identification unit 613 determines not to correct the movement log 1721 , and ends the processing without outputting the correction instruction.
  • the change unit 614 corrects the movement log 1721 in step 1507 of FIG. 15 by replacing the first track ID and the second track ID included in the correction target section of the movement log 1721 .
  • FIG. 28 illustrates a third functional configuration example of the video analysis apparatus 303 in FIG. 3 .
  • the video analysis apparatus 303 in FIG. 28 has a configuration in which a motion determination unit 2811 is added to the video analysis apparatus 303 in FIG. 17 .
  • the motion determination unit 2811 determines a motion of the person tracked at each time in a video by using, for example, a learned model generated by machine learning.
  • the learned model may be a support vector machine (SVM) for recognizing skeleton shapes.
  • SVM support vector machine
  • the motion of the person means, for example, walking, standing, being-seated, or the like.
  • the tracking unit 413 generates a movement log 2821 containing the motion of the person and stores the movement log 2821 in the storage unit 411 .
  • Each entry in the movement log 2821 contains the track ID, the tracking reliability, the motion of a person, the time, and the location.
  • FIG. 29 illustrates a third functional configuration example of the correction apparatus 302 in FIG. 3 .
  • the correction apparatus 302 in FIG. 29 has the same configuration as that of the correction apparatus 302 in FIG. 6 .
  • the storage unit 611 stores the movement log 2821 in FIG. 28 instead of the movement log 421 in FIG. 6 .
  • the identification unit 613 and the change unit 614 perform the movement log change processing using the movement log 2821 .
  • FIG. 30 illustrates an example of two persons moving together in the monitoring target region 311 in FIG. 3 .
  • a solid line 3001 indicates a movement path along which a person M who works using the terminal device 306 - 1 on the desk 312 - 1 temporarily leaves the desk 312 - 1 and moves to a point P.
  • a broken line 3002 indicates a movement path along which the person M returns to the desk 312 - 1 together with a person L after having conversations with the person L at the point P.
  • a dashed-dotted line 3003 indicates a movement path along which the person L moves from the point Q to the point P.
  • a dashed-two dotted line 3004 indicates a movement path along which the person L moves to the desk 312 - 1 together with the person M after having conversations with the person M at the point P.
  • the business log 521 containing an operation history of the person M in the monitoring target region 311 in FIG. 30 is the same as the business log 521 in FIG. 23 .
  • FIG. 31 illustrates an example of a third movement log 2821 containing the tracking reliability for the persons M and L in the monitoring target region 311 in FIG. 30 .
  • the movement log 2821 in FIG. 31 specifies an association relationship among the track ID, the tracking reliability, the motion of a person, the time, and the location.
  • the motion of a person is an example of motion information of an object.
  • the person “E 01 ” is seated at a location “desk A” at time “11:29”, and stands up and starts moving from the location “desk A” at time “11:30”.
  • the person “E 01 ” is walking at “point P” at time “11:31” and is standing at the location “desk A” at time “11:35”.
  • the tracking reliability for the person “E 01 ” is 100% at time “11:29” and time “11:30”. However, since the person “E 01 ” overlaps the person “R 0001 ” at “point P” at time “11:31”, the tracking reliability for the person “E 01 ” decreases to 50% at time “11:31” and time “11:35”.
  • the person “R 0001 ” is walking at “point Q” at time “11:29”, is walking at “point P” at time “11:31” and is seated at the location “desk A” at time “11:36”.
  • the tracking reliability for the person “R 0001 ” is 0% at time “11:29”, time “11:31”, and time “11:36”.
  • the identification unit 613 searches the business log 521 in FIG. 23 for an entry containing any of the events registered in the event list 621 in FIG. 8 as the operation type, and acquires an entry at a detection time “11:37” containing “PC log-in”.
  • the identification unit 613 extracts the personal ID “E 01 ”, the terminal ID “PC- 01 ”, and the operation type “PC log-in” from the entry at the detection time “11:37” acquired from the business log 521 .
  • the identification unit 613 obtains the location “desk A” associated with “PC- 01 ” by using the location information 622 in FIG. 9 .
  • the identification unit 613 acquires, from the movement log 2821 in FIG. 31 , an entry at a predetermined time corresponding to the detection time “11:37” of the acquired entry, the entry containing “desk A” as the location.
  • an entry at time “11:36” is acquired as the entry at the predetermined time containing “desk A”.
  • the identification unit 613 extracts the track ID “R 0001 ”, the tracking reliability “0%”, and the motion of the person “seated” from the entry.
  • PC log-in extracted from the entry in the business log 521 indicates that an expected motion at “desk A” is to be “seated”. Therefore, “seated” extracted from the entry in the movement log 2821 agrees with the expected motion at “desk A”.
  • the identification unit 613 compares the extracted tracking reliability “0%” with a predetermined value. For example, when the predetermined value is “60%”, the extracted tracking reliability “0%” is lower than the predetermined value. “E 01 ” extracted from the entry in the business log 521 is different from “R 0001 ” extracted from the entry in the movement log 2821 . Then, “seated” extracted from the entry in the movement log 2821 agrees with the motion of the person who performed “PC log-in” extracted from the entry in the business log 521 .
  • the identification unit 613 determines that the person “R 0001 ” who was seated at the location “desk A” at time “11:36” is the same as the person “E 01 ” who performed “PC log-in” at the location “desk A” at the detection time “11:37”.
  • the identification unit 613 obtains time “11:31” at which the person “E 01 ” and the person “R 0001 ” overlap each other at the point P in the movement log 2821 .
  • the tracking reliability of all the entries in a section from time “11:31” to time “11:36” is lower than “60%”. Therefore, the change unit 614 replaces “R 0001 ” and “E 01 ” included in this section with each other.
  • the track IDs of entries each having the tracking reliability lower than the predetermined value are replaced with each other, which makes it possible to inhibit a correct track ID from being erroneously changed.
  • a motion of a person recorded in the movement log 2821 is compared with a motion of a person presumed from the business log 521 , which makes it possible to accurately determine whether or not the two persons are the same person.
  • FIG. 32 illustrates an example of a post-replacement third movement log 2821 .
  • “R 0001 ” at time “11:31” and time “11:36” is changed to “E 01 ” and the tracking reliability thereof is changed to “100%”.
  • “E 01 ” at time “11:31” and time “11:35” is changed to “R 0001 ”, and the tracking reliability thereof is changed to “0%”.
  • the business process analysis system including the video analysis apparatus 303 in FIG. 28 and the correction apparatus 302 in FIG. 29 is capable of correcting the error and improving the accuracy of the movement log 2821 .
  • the business process analysis is performed by using the corrected movement log 2821 , which leads to an improvement of the efficiency of the processing of searching for the times and the locations associated with the track IDs of an analysis target.
  • FIG. 33 is a flowchart illustrating an example of third movement log generation processing performed by the video analysis apparatus 303 in FIG. 28 .
  • the processing in steps 3301 to 3306 and 3309 to 3311 is the same as the processing in steps 2601 to 2606 and 2608 to 2610 in FIG. 26 .
  • the tracking unit 413 determines what motion the person makes (step 3307 ).
  • the tracking unit 413 generates an entry of the person containing the track ID, the tracking reliability, the motion of the person, the time, and the location and records the entry in the movement log 2821 (step 3308 ).
  • the correction apparatus 302 in FIG. 29 performs the same movement log change processing as in FIG. 15 except for step 1505 in FIG. 15 in which the correction apparatus 302 performs location determination processing different from that in FIG. 16 .
  • FIG. 34 is a flowchart illustrating an example of third location determination processing performed by the correction apparatus 302 in FIG. 29 .
  • the processing in steps 3401 , 3402 , 3404 , 3405 , 3407 , and 3408 is the same as the processing in steps 2701 , 2702 and 2704 to 2707 in FIG. 27 .
  • the identification unit 613 acquires, from the movement log 2821 , an entry at a predetermined time corresponding to the detection time of the entry acquired from the business log 521 , the entry containing the location X 1 .
  • the identification unit 613 extracts the track ID, the tracking reliability, and the motion of the person from the entry (step 3403 ).
  • step 3405 the identification unit 613 performs the processing in step 3406 .
  • step 3406 the identification unit 613 checks whether the motion of the person extracted from the entry in the movement log 2821 agrees with the motion of the person who performed the operation of the operation type extracted from the entry in the business log 521 .
  • the motion of the person who performed the operation of the operation type extracted from the entry in the business log 521 is an example of a motion expected at the predetermined location.
  • the identification unit 613 determines to correct the movement log 2821 .
  • the identification unit 613 performs the processing in step 3407 and the subsequent step.
  • the identification unit 613 determines not to correct the movement log 2821 , and ends the processing without outputting the correction instruction.
  • FIG. 35 illustrates a configuration example of a vehicle management system including the movement history change apparatus 101 in FIG. 1 .
  • the vehicle management system in FIG. 35 includes a vehicle management apparatus 3501 , a correction apparatus 3502 , a video analysis apparatus 3503 , a collection apparatus 3504 , an imaging apparatus 3505 , a vehicle detection apparatus 3506 - 1 , and a vehicle detection apparatus 3506 - 2 .
  • the correction apparatus 3502 corresponds to the movement history change apparatus 101 in FIG. 1 .
  • the vehicle management apparatus 3501 , the correction apparatus 3502 , the video analysis apparatus 3503 , and the collection apparatus 3504 are able to communicate with each other via a communication network 3507 .
  • the communication network 3507 is a LAN or a WAN.
  • a monitoring target region 3511 is a space where vehicles move.
  • the monitoring target region 3511 may be a region within a site such as a factory or an airport.
  • the vehicle detection apparatuses 3506 - 1 and 3506 - 2 are examples of sensors. Three or more vehicle detection apparatuses 3506 - i may be installed in the monitoring target region 3511 .
  • the imaging apparatus 3505 is, for example, a camera having an imaging device such as a CCD or CMOS, captures a video of the monitoring target region 3511 , and outputs the captured video to the video analysis apparatus 3503 .
  • the video analysis apparatus 3503 analyzes the video output from the imaging apparatus 3505 and generates a movement log indicating a movement path of an object captured in the video.
  • the video analysis apparatus 3503 transmits the movement log to the correction apparatus 3502 .
  • the object captured in the video is, for example, a vehicle, and the movement log is an example of the object movement history information 122 .
  • the collection apparatus 3504 and the vehicle detection apparatuses 3506 - i are able to communicate with each other via a communication network 3508 .
  • the communication network 3508 is a LAN or a WAN.
  • the vehicle detection apparatus 3506 - i detects a vehicle passing through the gate 3512 - i and transmits a detection time, a detected incident, and information on the detected vehicle to the collection apparatus 3504 .
  • the vehicle detection apparatus 3506 - i captures the vehicle at a short distance with an imaging apparatus to acquire an image of the number plate, and transmits the acquired image as the information on the vehicle to the collection apparatus 3504 .
  • the collection apparatus 3504 identifies identification information of the vehicle and a vehicle type thereof from the information on the vehicle received from the vehicle detection apparatus 3506 - i.
  • the identification information of the vehicle is, for example, a vehicle registration number displayed on the number plate.
  • the vehicle type is an example of type information specifying the type of an object.
  • the collection apparatus 3504 generates a detection log containing the detection time, the identification information of the vehicle, the vehicle type, and the detected incident and transmits the detection log to the correction apparatus 3502 .
  • the detection log is an example of the object detection history information 121 .
  • the correction apparatus 3502 corrects the movement log received from the video analysis apparatus 3503 by using the detection log received from the collection apparatus 3504 , and transmits the corrected movement log to the vehicle management apparatus 3501 .
  • the vehicle management apparatus 3501 analyzes the operation status of the vehicle by using the movement log received from the correction apparatus 3502 , and outputs an analysis result.
  • FIG. 36 illustrates a functional configuration example of the video analysis apparatus 3503 in FIG. 35 .
  • the video analysis apparatus 3503 in FIG. 36 includes a storage unit 3611 , a communication unit 3612 , a tracking unit 3613 , and a reliability assignment unit 3614 .
  • the communication unit 3612 receives a video of the monitoring target region 3511 from the imaging apparatus 3505 .
  • the tracking unit 3613 tracks the vehicle captured in the video, and the reliability assignment unit 3614 assigns the tracking reliability to the vehicle tracked at each time.
  • the tracking unit 3613 is able to track a vehicle in a video by recognizing the vehicle type of the vehicle by use of a learned model, such as a single shot multibox detector (SSD), generated by machine learning.
  • SSD single shot multibox detector
  • the tracking unit 3613 generates a movement log 3621 indicating a movement path of the vehicle and stores the movement log 3621 in the storage unit 3611 .
  • Each entry in the movement log 3621 contains the track ID, the tracking reliability, the time, the vehicle type, and the location.
  • the communication unit 3612 transmits the generated movement log 3621 to the correction apparatus 3502 .
  • FIG. 37 illustrates a functional configuration example of the collection apparatus 3504 in FIG. 35 .
  • the collection apparatus 3504 in FIG. 37 includes a storage unit 3711 , a communication unit 3712 , and a recording unit 3713 .
  • the communication unit 3712 receives the detection time, the detected incident, and the information on the vehicle from the vehicle detection apparatus 3506 - i.
  • the storage unit 3711 stores vehicle information 3721 .
  • the vehicle information 3721 is information in which the identification information of a vehicle is associated with the vehicle type.
  • the recording unit 3713 When the information on the vehicle contains the image of the number plate, the recording unit 3713 recognizes the vehicle registration number from the image of the number plate and acquires the vehicle registration number as the identification information of the vehicle.
  • the recording unit 3713 acquires the vehicle type associated with the acquired identification information of the vehicle from the vehicle information 3721 , generates a detection log 3722 , and stores the detection log 3722 in the storage unit 3711 .
  • Each entry in the detection log 3722 contains the detection time, the vehicle identification information, the vehicle type, and the detected incident.
  • the communication unit 3712 transmits the generated detection log 3722 to the correction apparatus 3502 .
  • FIG. 38 illustrates a functional configuration example of the correction apparatus 3502 in FIG. 35 .
  • the correction apparatus 3502 in FIG. 38 includes a storage unit 3811 , a communication unit 3812 , an identification unit 3813 , and a change unit 3814 .
  • the storage unit 3811 , the identification unit 3813 , and the change unit 3814 correspond to the storage unit 111 , the identification unit 112 , and the change unit 113 in FIG. 1 , respectively.
  • the communication unit 3812 receives the detection log 3722 from the collection apparatus 3504 and receives the movement log 3621 from the video analysis apparatus 3503 .
  • the identification unit 3813 stores the received detection log 3722 and movement log 3621 in the storage unit 3811 .
  • the storage unit 3811 stores an event list 3821 .
  • the event list 3821 is a list of events each indicating a detected incident usable for determining a location included in the movement log 3621 among detected incidents to be contained in the detection log 3722 .
  • the identification unit 3813 acquires the identification information of a first vehicle, the vehicle type, the detection time, and the detected incident from the detection log 3722 based on the events contained in the event list 3821 . In the movement log 3621 , the identification unit 3813 identifies the identification information of a second vehicle present at a predetermined location involved in the detected incident at a predetermined time corresponding to the detection time of the first vehicle.
  • the change unit 3814 corrects the movement log 3621 by changing the identified identification information of the second vehicle to the acquired identification information of the first vehicle in the movement log 3621 .
  • the communication unit 3812 transmits the corrected movement log 3621 to the vehicle management apparatus 3501 .
  • FIG. 39 illustrates an example of vehicles moving in the monitoring target region 3511 in FIG. 35 .
  • a vehicle 3901 is of a vehicle type “A”
  • a vehicle 3902 is of a vehicle type “B”.
  • the body of the vehicle type “A” is smaller than the body of the vehicle type “B”.
  • the vehicle 3902 is stopped at a point V, and the vehicle 3901 moves along a movement path 3911 from the gate 3512 - 1 to the gate 3512 - 2 .
  • the vehicle 3901 passing through the gate 3512 - 1 is seen in a predetermined region around the vehicle detection apparatus 3506 - 1 .
  • the vehicle 3901 passing through the gate 3512 - 2 is seen in a predetermined region around the vehicle detection apparatus 3506 - 2 .
  • the vehicle 3901 When viewed from the imaging apparatus 3505 , the vehicle 3901 passes behind the vehicle 3902 at the point V and therefore is temporarily hidden in the video of the imaging apparatus 3505 . As a result, the tracking is interrupted. When the vehicle 3901 appears on the right side of the vehicle 3902 , a new track ID is assigned to the vehicle 3901 .
  • FIG. 40 illustrates an example of the event list 3821 illustrated in FIG. 38 .
  • An event contained in the event list 3821 in FIG. 40 is “passing through gate G 2 ”.
  • gate G 2 denotes the name of the gate 3512 - 2 .
  • FIG. 41 illustrates an example of the vehicle information 3721 in FIG. 37 .
  • the vehicle information 3721 in FIG. 41 specifies an association relationship between the vehicle type and a vehicle ID.
  • the vehicle ID is identification information of a vehicle.
  • “C 1234 ” is the identification information of the vehicle 3901
  • “T 0012 ” is the identification information of the vehicle 3902 .
  • FIG. 42 illustrates an example of the detection log 3722 containing the detection history of the vehicle 3901 in the monitoring target region 3511 in FIG. 39 .
  • the detection log 3722 illustrated in FIG. 42 indicates an association relationship among the detection time, the vehicle ID, the vehicle type, and the detected incident.
  • the detection time specifies a time at which the vehicle identified by the vehicle ID was detected by the vehicle detection apparatus 3506 - i, and the detected incident specifies what work was done using the detected vehicle. As the detected incident, “passing through gate”, “loading work to aircraft”, or the like is recorded.
  • the vehicle 3901 identified by “C 1234 ” passes through “gate G 1 ” at time “13:30” and passes through “gate G 2 ” at time “13:50”.
  • gate G 1 denotes the name of the gate 3512 - 1 .
  • FIG. 43 illustrates an example of the movement log 3621 containing the movement paths of the two vehicles in the monitoring target region 3511 in FIG. 39 .
  • the movement log 3621 in FIG. 43 specifies an association relationship among the track ID, the tracking reliability, the time, and the location.
  • the tracking reliability decreases every time the vehicle identified by the track ID overlaps another vehicle.
  • the vehicle “C 1234 ” is present at “gate G 1 ” at time “13:29”.
  • the tracking reliability for the vehicle “C 1234 ” is 100% at time “13:29”.
  • the vehicle “T 0012 ” is present at “point V” at time “13:29” and time “13:50”.
  • the tracking reliability for the vehicle “T 0012 ” is 100% at time “13:29” and time “13:50”.
  • the vehicle “R 0001 ” is present at “point V” at time “13:48” and is present at “gate G 2 ” at time “13:50”.
  • the tracking reliability for the vehicle “R 0001 ” is 0% at time “13:48” and time “13:50”.
  • the identification unit 3813 searches the detection log 3722 in FIG. 42 for an entry containing, as the detected incident, any of the events registered in the event list 3821 in FIG. 40 , and acquires an entry at a detection time “13:50” containing “passing through gate G 2 ”.
  • the identification unit 3813 extracts the vehicle ID “C 1234 ”, the vehicle type “A”, and the detected incident “passing through gate G 2 ” from the entry at the detection time “13:50” acquired from the detection log 3722 .
  • the identification unit 3813 acquires, from the movement log 3621 in FIG. 43 , an entry at a predetermined time corresponding to the detection time “13:50” of the acquired entry, the entry containing, as the location, “gate G 2 ” involved in the detected incident.
  • an entry at time “13:50” having the track ID “R 0001 ” is acquired as the entry at the predetermined time containing “gate G 2 ”.
  • the identification unit 3813 extracts the track ID “R 0001 ”, the tracking reliability “0%”, and the vehicle type “A” from the entry.
  • the identification unit 3813 compares the extracted tracking reliability “0%” with a predetermined value. For example, when the predetermined value is “60%”, the extracted tracking reliability “0%” is lower than the predetermined value. “C 1234 ” extracted from the entry in the detection log 3722 is different from “R 0001 ” extracted from the entry in the movement log 3621 . Meanwhile, the vehicle type “A” extracted from the entry in the detection log 3722 matches the vehicle type “A” extracted from the entry in the movement log 3621 .
  • the identification unit 3813 determines that the vehicle “R 0001 ” which was present at “gate G 2 ” at time “13:50” is the same as the vehicle “C 1234 ” which passed through “gate G 2 ” at the detection time “13:50”.
  • the identification unit 3813 obtains a section in the movement log 3621 in which the tracking reliability for the vehicle “R 0001 ” is lower than the predetermined value in a time range before or at time “13:50”, and the change unit 3814 changes “R 0001 ” in the section to “C 1234 ”.
  • the track IDs of entries having the tracking reliability lower than the predetermined value are selected as the change targets, which makes it possible to inhibit a correct track ID from being erroneously changed.
  • the vehicle type recorded in the movement log 3621 is compared with the vehicle type recorded in the detection log 3722 , which makes it possible to accurately determine whether or not two vehicles are the same vehicle.
  • FIG. 44 illustrates an example of a post-change movement log 3621 .
  • “R 0001 ” at time “13:48” and time “13:50” is changed to “C 1234 ” and the tracking reliability thereof is changed to “100%”.
  • the vehicle management apparatus 3501 searches the movement log 3621 in FIG. 44 for the times and locations associated with the track IDs “C 1234 ” of the analysis target, and generates a movement path of the vehicle 3901 including the searched-out times and locations. Then, based on the generated movement path, the vehicle management apparatus 3501 detects a bottleneck in the work performed using the vehicle 3901 and optimizes an operation schedule of the vehicle 3901 .
  • the vehicle management system in FIG. 35 is capable of correcting an error and improving the accuracy of the movement log 3621 .
  • the operation status is analyzed by using the corrected movement log 3621 , which leads to an improvement of the efficiency of the processing of searching for the times and the locations associated with the track IDs of an analysis target.
  • FIG. 45 is a flowchart illustrating an example of detection log generation processing performed by the collection apparatus 3504 in FIG. 37 .
  • the recording unit 3713 checks whether an end instruction is input from a user (step 4501 ).
  • the recording unit 3713 acquires, via the communication unit 3712 , the detection time and the detected incident of a vehicle detected by any of the vehicle detection apparatuses 3506 - i, and the information on the vehicle (step 4502 ).
  • the recording unit 3713 recognizes the vehicle ID from the information on the vehicle (step 4503 ), and acquires the vehicle type associated with the recognized vehicle ID from the vehicle information 3721 (step 4504 ).
  • the recording unit 3713 generates an entry containing the detection time, the vehicle ID, the vehicle type, and the detected incident and records the entry in the detection log 3722 (step 4505 ).
  • the recording unit 3713 iterates the processing in step 4501 and the subsequent steps.
  • the end instruction is input (step 4501 , YES)
  • the recording unit 3713 ends the processing.
  • FIG. 46 is a flowchart illustrating an example of fourth movement log generation processing performed by the video analysis apparatus 3503 in FIG. 36 .
  • the tracking unit 3613 checks whether an end instruction is input from a user (step 4601 ).
  • the tracking unit 3613 tracks a vehicle captured in a video while recognizing the vehicle type of the vehicle (step 4602 ), and checks whether the tracking is successful (step 4603 ).
  • the tracking unit 3613 performs overlap determination processing on the vehicle identified by the track ID (step 4604 ), and the reliability assignment unit 3614 sets the tracking reliability for the vehicle in accordance with a result of the overlap determination processing (step 4605 ).
  • the reliability assignment unit 3614 sets the value of 1 / 2 of the immediately preceding tracking reliability when the vehicle identified by the track ID overlaps another vehicle or sets the immediately preceding tracking reliability when the vehicle identified by the track ID does not overlap another vehicle.
  • the tracking unit 3613 adds the name of a location indicated by the ROI that the vehicle enters in the image (step 4606 ), generates an entry of the vehicle containing the track ID, the tracking reliability, the time, the vehicle type, and the location, and records the entry in the movement log 3621 (step 4607 ).
  • the video analysis apparatus 3503 iterates the processing in step 4601 and the subsequent steps.
  • the tracking unit 3613 detects a new vehicle (step 4608 ) and recognizes the vehicle type of the detected vehicle.
  • the tracking unit 3613 assigns a new track ID to the detected vehicle (step 4609 ).
  • the reliability assignment unit 3614 sets the tracking reliability “0%” for the vehicle (step 4610 ), and the video analysis apparatus 3503 performs the processing in step 4606 and the subsequent step.
  • the end instruction is input (step 4601 , YES)
  • the video analysis apparatus 3503 ends the processing.
  • FIG. 47 is a flowchart illustrating an example of second movement log change processing performed by the correction apparatus 3502 in FIG. 38 .
  • the identification unit 3813 checks whether an end instruction is input from a user (step 4701 ).
  • the identification unit 3813 acquires, via the communication unit 3812 , the detection log 3722 from the collection apparatus 3504 (step 4702 ) and the movement log 3621 from the video analysis apparatus 3503 (step 4703 ).
  • the identification unit 3813 searches the acquired detection log 3722 for a detected incident indicating any of the events registered in the event list 3821 and thereby checks whether the detection log 3722 contains an event (step 4704 ).
  • the identification unit 3813 makes location determination to determine whether to correct the acquired movement log 3621 (step 4705 ).
  • the identification unit 3813 outputs a correction instruction indicating the track ID of the change target and the post-change track ID to the change unit 3814 .
  • the change unit 3814 checks whether the correction instruction is output from the identification unit 3813 (step 4706 ).
  • the change unit 3814 corrects the movement log 3621 by changing the track ID of the change target specified in the correction instruction to the post-change track ID specified therein in the movement log 3621 (step 4707 ).
  • the correction apparatus 3502 iterates the processing in step 4701 and the subsequent steps.
  • step 4704 When the detection log 3722 contains no event (step 4704 , NO), the correction apparatus 3502 iterates the processing in step 4701 and the subsequent steps. When the correction instruction is not output (step 4706 , NO), the correction apparatus 3502 iterates the processing in step 4701 and the subsequent steps. When the end instruction is input (step 4701 , YES), the correction apparatus 3502 ends the processing.
  • FIG. 48 is a flowchart illustrating an example of fourth location determination processing in step 4705 in FIG. 47 .
  • the identification unit 3813 acquires an entry containing an event as the detected incident from the detection log 3722 , and extracts the detection time, the vehicle ID, the vehicle type, and the detected incident from the entry (step 4801 ).
  • the identification unit 3813 acquires, from the movement log 3621 , an entry at a predetermined time corresponding to the detection time extracted from the entry in the detection log 3722 , the entry containing the location involved in the detected incident.
  • the identification unit 3813 extracts the track ID, the tracking reliability, and the vehicle type from the entry (step 4802 ).
  • the identification unit 3813 compares the extracted tracking reliability with a predetermined value (step 4803 ).
  • the tracking reliability is lower than the predetermined value (step 4803 , YES)
  • the identification unit 3813 compares the vehicle ID extracted from the entry in the detection log 3722 with the track ID extracted from the entry in the movement log 3621 (step 4804 ).
  • the identification unit 3813 checks whether the vehicle type extracted from the entry in the detection log 3722 matches the vehicle type extracted from the entry in the movement log 3621 (step 4805 ).
  • the identification unit 3813 determines to correct the movement log 3621 .
  • the identification unit 3813 outputs a correction instruction indicating the track ID of the change target and the post-change track ID to the change unit 3814 (step 4806 ).
  • the track ID of the change target is the same track ID as the track ID extracted from the entry in the movement log 3621 , and covers the track IDs contained in entries in a section in which the tracking reliability is lower than the predetermined value in a time range before or at the predetermined time or entries after the predetermined time.
  • the post-change track ID is the vehicle ID extracted from the entry in the detection log 3722 .
  • the identification unit 3813 determines not to correct the movement log 3621 , and the correction apparatus 3502 ends the processing without outputting the correction instruction.
  • the identification unit 3813 determines not to correct the movement log 3621 , and the correction apparatus 3502 ends the processing without outputting the correction instruction.
  • the identification unit 3813 determines not to correct the movement log 3621 .
  • the correction apparatus 3502 ends the processing without outputting the correction instruction.
  • the correction apparatuses 302 and 3502 may record the post-change movement log as another movement log instead of overwriting the pre-change movement log.
  • the correction apparatuses 302 and 3502 may generate an integrated movement log by integrating the post-change movement log with the business log 521 or the detection log 3722 .
  • FIG. 49 illustrates a fourth functional configuration example of the correction apparatus 302 that generates an integrated movement log.
  • the correction apparatus 302 in FIG. 49 has the same configuration as that of the correction apparatus 302 in FIG. 18 .
  • the change unit 614 integrates the post-change movement log 1721 and the business log 521 to generate an integrated movement log 4901 , and stores the integrated movement log 4901 in the storage unit 611 .
  • the communication unit 612 transmits the integrated movement log 4901 to the business process analysis apparatus 301 , and the business process analysis apparatus 301 performs the business process analysis by using the integrated movement log 4901 .
  • FIG. 50 illustrates an example of the integrated movement log 4901 generated by integrating the business log 521 in FIG. 23 and the post-replacement movement log 1721 in FIG. 25 .
  • the integrated movement log 4901 in FIG. 50 contains the items of the track ID, the time, the location, the terminal ID, and the operation type.
  • Entries 5001 , 5003 to 5005 , and 5007 to 5009 correspond to the entries in the movement log 1721 in FIG. 25 , and do not contain data on the terminal ID and the operation type.
  • Entries 5002 and 5006 correspond to the entries in the business log 521 in FIG. 23 and contain data on the terminal ID and the operation type. Since the business log 521 in FIG. 23 does not contain the location, the data on the location in each of the entries 5002 and 5006 is complemented by using the data on the location in the immediately preceding entry.
  • the location “desk A” in the entry 5002 is the same as the location in the immediately preceding entry 5001
  • the location “desk A” in the entry 5006 is the same as the location in the immediately preceding entry 5005 .
  • the correction apparatus 302 in FIG. 49 is capable of compressing the data of the business log 521 and the movement log 1721 by integrating the business log 521 and the movement log 1721 . Therefore, the free space in the storage unit 611 increases by deleting the business log 521 and the movement log 1721 after the integrated movement log 4901 is generated.
  • FIG. 51 is a flowchart illustrating an example of integrated movement log generation processing performed by the correction apparatus 302 in FIG. 49 .
  • the change unit 614 checks whether an end instruction is input from a user (step 5101 ).
  • the change unit 614 acquires the business log 521 (step 5102 ) and acquires the corrected movement log 1721 (step 5103 ) from the storage unit 611 .
  • the change unit 614 combines the entries in the business log 521 and the entries in the movement log 1721 according to the items in the integrated movement log 4901 (step 5104 ).
  • the change unit 614 sorts the multiple entries thus combined in ascending or descending order of the track ID (step 5105 ), and sorts two or more entries including the same track ID in ascending order of the time (step 5106 ).
  • the change unit 614 finds an entry that does not contain the data on a location, and complements the data on the location in the found entry by using the data on a location in the immediately preceding entry (step 5107 ).
  • the change unit 614 records the multiple entries thus sorted into the integrated movement log 4901 (step 5108 ), and the correction apparatus 302 iterates the processing in step 5101 and the subsequent steps.
  • the correction apparatus 302 ends the processing.
  • objects recorded in the business log, the detection log, and the movement log are not limited to persons or vehicles, and may be other objects such as animals, self-propelled robots, and aircrafts.
  • the other objects also include products conveyed by a conveying mechanism such as a belt conveyor, paper documents, and so on.
  • the sensor that detects an object may be a radio frequency identification (RFID) reader that reads an RFID tag attached to an object.
  • RFID radio frequency identification
  • the configuration of the movement history change apparatus 101 in FIG. 1 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the movement history change apparatus 101 .
  • the configuration of the business process analysis system illustrated in FIG. 3 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the business process analysis system.
  • the configurations of the video analysis apparatus 303 in FIGS. 4, 17, and 28 are merely examples, and some of the components may be omitted or modified depending on the usage or conditions of the business process analysis system.
  • the configuration of the collection apparatus 304 illustrated in FIG. 5 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the business process analysis system.
  • the configuration of the correction apparatus 302 illustrated in FIGS. 6, 18, 29, and 49 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the business process analysis system.
  • the configuration of the vehicle management system illustrated in FIG. 35 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the vehicle management system.
  • the configuration of the video analysis apparatus 3503 in FIG. 36 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the vehicle management system.
  • the configuration of the collection apparatus 3504 in FIG. 37 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the vehicle management system.
  • the configuration of the correction apparatus 3502 in FIG. 38 is merely an example, and some of the components may be omitted or modified depending on the usage or conditions of the vehicle management system.
  • the flowchart in FIG. 2 is merely an example, and part of the processing may be omitted or modified depending on the configuration or conditions of the movement history change apparatus 101 .
  • the flowcharts in FIGS. 13 to 16, 26, 27, 33, 34, and 51 are merely examples, and part of the processing may be omitted or modified depending on the configuration or conditions of the business process analysis system.
  • the flowcharts in FIGS. 45 to 48 are merely examples, and part of the processing may be omitted or modified depending on the configuration or conditions of the vehicle management system. For example, when the vehicle type is not used in the location determination processing in FIG. 48 , the processing in step 4805 may be omitted.
  • the monitoring target regions 311 and the movement paths of the persons illustrated in FIGS. 7, 20, and 30 are merely examples, and the monitoring target region 311 and the movement paths of persons vary depending on an analysis target of the business process analysis system.
  • the various kinds of data illustrated in FIGS. 8 to 12, 19, 23 to 25, 31, 32, and 50 are merely examples, and these kinds of data vary depending on the monitoring target region 311 and the movement paths of persons.
  • the overlap determination processing illustrated in FIGS. 21 and 22 is merely an example, and whether or not two persons overlap each other may be determined by another determination method.
  • the monitoring target region 3511 and the movement paths of the vehicles illustrated in FIG. 39 are merely examples, and the monitoring target region 3511 and the movement paths of vehicles vary depending on an analysis target of the vehicle management system.
  • the various kinds of data illustrated in FIGS. 40 to 44 are merely examples, and these kinds of data vary depending on the monitoring target region 3511 and the movement paths of vehicles.
  • the expressions (1) to (6) are merely examples, and whether or not two persons overlap each other may be determined by using other calculation expressions.
  • FIG. 52 illustrates a hardware configuration example of an information processing apparatus (computer) used as the movement history change apparatus 101 in FIG. 1 , the video analysis apparatus 303 in FIGS. 4, 17 , and 28 , the collection apparatus 304 in FIG. 5 , and the correction apparatus 302 in FIGS. 6, 18, 29, and 49 .
  • the information processing apparatus in FIG. 52 is also used as the video analysis apparatus 3503 in FIG. 36 , the collection apparatus 3504 in FIG. 37 , and the correction apparatus 3502 in FIG. 38 .
  • the information processing apparatus in FIG. 52 includes a central processing unit (CPU) 5201 , a memory 5202 , an input device 5203 , an output device 5204 , an auxiliary storage device 5205 , a medium driving device 5206 , and a network coupling device 5207 . These components are pieces of hardware, and are coupled to each other via a bus 5208 .
  • CPU central processing unit
  • the memory 5202 is, for example, a semiconductor memory such as a read-only memory (ROM), a random-access memory (RAM), or a flash memory, and stores a program and data to be used in processing.
  • the memory 5202 may operate as the storage unit 111 in FIG. 1 , the storage unit 411 in FIGS. 4, 17, and 28 , the storage unit 511 in FIG. 5 , or the storage unit 611 in FIGS. 6, 18, 29, and 49 .
  • the memory 5202 may operate as the storage unit 3611 in FIG. 36 , the storage unit 3711 in FIG. 37 , or the storage unit 3811 in FIG. 38 .
  • the CPU 5201 (processor) operates as the identification unit 112 and the change unit 113 in FIG. 1 by executing a program using the memory 5202 , for example.
  • the CPU 5201 also operates as the tracking unit 413 in FIGS. 4, 17, and 28 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the recording unit 513 in FIG. 5 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the identification unit 613 and the change unit 614 in FIGS. 6, 18, 29, and 49 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the reliability assignment unit 1711 in FIGS. 17 and 28 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the motion determination unit 2811 in FIG. 28 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the tracking unit 3613 and the reliability assignment unit 3614 in FIG. 36 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the recording unit 3713 in FIG. 37 by executing a program using the memory 5202 .
  • the CPU 5201 also operates as the identification unit 3813 and the change unit 3814 in FIG. 38 by executing a program using the memory 5202 .
  • the input device 5203 is, for example, a keyboard, a pointing device, or the like, and is used by an operator or a user to input instructions or information.
  • the output device 5204 is, for example, a display device, a printer, a speaker, or the like, and is used to output inquiries or instructions to an operator or a user and output processing results.
  • the processing result may be the movement log 421 , the movement log 1721 , the movement log 2821 , the movement log 3621 , or the integrated movement log 4901 .
  • the auxiliary storage device 5205 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like.
  • the auxiliary storage device 5205 may be a hard disk drive or a flash memory.
  • the information processing apparatus stores a program and data in the auxiliary storage device 5205 , and may use the program and data by loading them into the memory 5202 .
  • the auxiliary storage device 5205 may operate as the storage unit 111 in FIG. 1 , the storage unit 411 in FIGS. 4, 17, and 28 , the storage unit 511 in FIG. 5 , or the storage unit 611 in FIGS. 6, 18, 29, and 49 .
  • the auxiliary storage device 5205 may operate as the storage unit 3611 in FIG. 36 , the storage unit 3711 in FIG. 37 , or the storage unit 3811 in FIG. 38 .
  • the medium driving device 5206 drives a portable recording medium 5209 , and accesses contents recorded therein.
  • the portable recording medium 5209 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like.
  • the portable recording medium 5209 may be a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a Universal Serial Bus (USB) memory, or the like.
  • the user or operator may store a program and data in the portable recording medium 5209 , and may use the program and data by loading them into the memory 5202 .
  • a computer-readable recording medium in which the programs and data to be used in the processing are stored as described above is a physical (non-transitory) recording medium such as the memory 5202 , the auxiliary storage device 5205 , or the portable recording medium 5209 .
  • the network coupling device 5207 is a communication interface circuit which is coupled to the communication network 307 or 3507 , and performs data conversion for communication.
  • the information processing apparatus may receive a program and data from an external device via the network coupling device 5207 , and may use the program and data by loading them into the memory 5202 .
  • the network coupling device 5207 may operate as the communication unit 412 in FIGS. 4, 17, and 28 , the communication unit 512 in FIG. 5 , or the communication unit 612 in FIGS. 6, 18, 29, and 49 .
  • the network coupling device 5207 may operate as the communication unit 3612 in FIG. 36 , the communication unit 3712 in FIG. 37 , or the communication unit 3812 in FIG. 38 .
  • the information processing apparatus does not necessarily include all of the components in FIG. 52 , and some of the components may be omitted depending on the use or conditions of the information processing apparatus. For example, if an interface with the user or the operator is not used, the input device 5203 and the output device 5204 may be omitted. When the portable recording medium 5209 is not used, the medium driving device 5206 may be omitted.
  • an information processing device similar to that in FIG. 52 may be used.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Analysis (AREA)
US17/542,422 2021-02-10 2021-12-05 Movement history change method, storage medium, and movement history change device Pending US20220254033A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021019762A JP2022122498A (ja) 2021-02-10 2021-02-10 移動履歴変更方法及び移動履歴変更プログラム
JP2021-019762 2021-02-10

Publications (1)

Publication Number Publication Date
US20220254033A1 true US20220254033A1 (en) 2022-08-11

Family

ID=78617293

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/542,422 Pending US20220254033A1 (en) 2021-02-10 2021-12-05 Movement history change method, storage medium, and movement history change device

Country Status (3)

Country Link
US (1) US20220254033A1 (ja)
EP (1) EP4044129A1 (ja)
JP (1) JP2022122498A (ja)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080159596A1 (en) * 2006-12-29 2008-07-03 Motorola, Inc. Apparatus and Methods for Head Pose Estimation and Head Gesture Detection
US20180114078A1 (en) * 2015-08-20 2018-04-26 JVC Kenwood Corporation Vehicle detection device, vehicle detection system, and vehicle detection method
US20200242780A1 (en) * 2019-01-28 2020-07-30 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and storage medium
US20220329740A1 (en) * 2019-09-06 2022-10-13 Canon Kabushiki Kaisha Electronic apparatus, method for controlling electronic apparatus, and non-transitory computer readable storage medium

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014050432A1 (ja) * 2012-09-27 2014-04-03 日本電気株式会社 情報処理システム、情報処理方法及びプログラム
JPWO2015166612A1 (ja) * 2014-04-28 2017-04-20 日本電気株式会社 映像解析システム、映像解析方法および映像解析プログラム
JP7007177B2 (ja) * 2017-12-20 2022-01-24 マクセル株式会社 端末機器、本人認証システムおよび本人認証方法
JP7065557B2 (ja) 2018-12-05 2022-05-12 Kddi株式会社 人物を追跡する映像解析装置、プログラム及び方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080159596A1 (en) * 2006-12-29 2008-07-03 Motorola, Inc. Apparatus and Methods for Head Pose Estimation and Head Gesture Detection
US20180114078A1 (en) * 2015-08-20 2018-04-26 JVC Kenwood Corporation Vehicle detection device, vehicle detection system, and vehicle detection method
US20200242780A1 (en) * 2019-01-28 2020-07-30 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and storage medium
US20220329740A1 (en) * 2019-09-06 2022-10-13 Canon Kabushiki Kaisha Electronic apparatus, method for controlling electronic apparatus, and non-transitory computer readable storage medium

Also Published As

Publication number Publication date
JP2022122498A (ja) 2022-08-23
EP4044129A1 (en) 2022-08-17

Similar Documents

Publication Publication Date Title
US11450027B2 (en) Method and electronic device for processing videos
US20190287265A1 (en) System and method for visual identification, and system and method for classifying and sorting
Chattopadhyay et al. Frontal gait recognition from incomplete sequences using RGB-D camera
US20160307600A1 (en) Case video processing apparatus, case video processing system, and case video processing method
US11048917B2 (en) Method, electronic device, and computer readable medium for image identification
US20230351617A1 (en) Crowd type classification system, crowd type classification method and storage medium for storing crowd type classification program
Dang et al. Object tracking using improved deep SORT YOLOv3 architecture
US20200134875A1 (en) Person counting method and person counting system
JP6943333B2 (ja) オブジェクト追跡装置、オブジェクト追跡方法、及びオブジェクト追跡プログラム
KR20090132339A (ko) 감시 시스템 네트워크를 이용한 객체의 이동 경로 추적방법 및 시스템
Martínez et al. A motion descriptor based on statistics of optical flow orientations for action classification in video-surveillance
Alam et al. A cost-effective computer vision-based vehicle detection system
US20220254033A1 (en) Movement history change method, storage medium, and movement history change device
Jeon et al. A retail object classification method using multiple cameras for vision-based unmanned kiosks
EP3761228A1 (en) Computer-implemented method
US20220309471A1 (en) Systems and methods for machine learning-informed automated recording of time activities with an automated electronic time recording system or service
US11722763B2 (en) System and method for audio tagging of an object of interest
US20210166041A1 (en) System and method for tracking customer movements in a customer service environment
Nieto et al. Perspective multiscale detection and tracking of persons
CN111860223A (zh) 属性识别系统、学习服务器以及计算机可读记录介质
US11093546B2 (en) Method for categorizing digital video data
US20240119353A1 (en) Training data generation method and training data generation system
Münch et al. Towards a real-time situational awareness system for surveillance applications in unconstrained environments
JP7088108B2 (ja) 追跡方法、追跡装置、及びプログラム
KR20140087062A (ko) 하이브리드 인식 기술을 이용한 전사적 인적자원 관리 시스템 및 방법

Legal Events

Date Code Title Description
AS Assignment

Owner name: FUJITSU LIMITED, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YOSHIOKA, TAKAHIRO;NAKAZATO, KATSUHISA;REEL/FRAME:058912/0650

Effective date: 20211102

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED