JP2024077198A - Ticket gate system, passenger ticket system and passenger image identification method - Google Patents

Abstract

[Problem] To provide a ticket gate system capable of identifying a passenger image of a passenger holding a user terminal where a ticket gate abnormality has occurred from among a large number of passenger images. [Solution] A related group data is identified, which is group data created from an image captured of a ticket gate area in a station other than the station where the passenger image of the first group data where the ticket gate abnormality has occurred was captured, and a passenger image is assigned to a user ID of the first group data or the related group data based on the image similarity between the passenger image of the first group data and the passenger image of the related group data, thereby identifying the passenger image of the passenger holding the user terminal that caused the ticket gate abnormality. [Selected Figure] Figure 4A

Description

The present invention relates to a ticket gate system, a ticket system, and a passenger image identification method.

Patent Document 1 discloses that a travel itinerary is constructed by collecting the travel routes of passengers by measuring the position of the mobile devices carried by the passengers. This enables transportation operators to charge fares to passengers without the need to issue dedicated ticket media or install ticket gates, and eliminates the need for passengers to manage dedicated ticket media.

Patent Document 2 discloses a technology for easily identifying a person in images captured from multiple viewpoints by combining the reception strength of a wireless signal received from a wireless signal transmitter by a sensor that provides ambiguous identification of the person, specifically a mobile terminal carried by the person, and monitoring the person's behavior. Training data is generated by combining the position of the same person appearing in each image acquired in advance with sensor data that detects the person acquired at the same time. An estimate of the sensor data at the position of the person appearing in each image acquired in successive time series is calculated based on the training data, and the person's identification information is identified based on the time series change in the similarity between the estimate and the acquired sensor data.

International Publication No. 2021/130024 JP 2018-61114 A

While passengers can increase convenience by being able to use public transportation services using a ticket application installed on their mobile devices, it becomes necessary to deal with abnormalities at ticket gates, such as malfunctions of the mobile device or an invalid ticket application (hereinafter referred to as possession of an invalid ticket).

The inventors have considered implementing a ticket gate function by identifying passengers carrying mobile devices with ticket gate abnormalities using images (video images) from cameras installed in stations as a means of identifying passengers carrying mobile devices with ticket gate abnormalities. For example, Patent Literature 2 discloses technology for identifying images of people carrying mobile terminals. However, to implement ticket gate functions for public transportation, it is necessary to identify passengers carrying mobile devices with ticket gate abnormalities from among a large number of passengers moving in roughly the same direction in a relatively narrow area (corridor). With the technology in Patent Literature 2, it is difficult to identify people who are so close to each other that they cannot be identified with the identification accuracy of sensor data and who behave similarly.

A ticket gate system according to one embodiment of the present invention includes a terminal tracking unit that accumulates location data indicating the location of a user terminal, an itinerary construction unit that identifies the entrance and exit stations of a passenger carrying a user terminal from the location data accumulated in the terminal tracking unit and constructs itinerary data for that passenger, a person tracking unit that identifies and tracks passengers passing through a specified ticket gate area within a station from an image of the ticket gate area captured by the person tracking unit, and a group tracking unit that creates groups of passengers who passed through the ticket gate area in close proximity to each other from the image, and registers, as group data, passenger images of passengers included in the group and user IDs corresponding to user terminals that passed through the ticket gate area during the period in which the group passed through the ticket gate area, identified based on the location data accumulated in the terminal tracking unit. The system has a creation unit, and an identification unit that, when it is determined that a ticket gate abnormality has occurred in the group data, identifies related group data that is group data created from an image captured of a ticket gate area in a station other than the station where the passenger image of the first group data where the ticket gate abnormality occurred was captured, assigns a passenger image to a user ID of the first group data or the related group data based on the image similarity between the passenger image of the first group data and the passenger image of the related group data, and identifies the passenger image of the passenger who owns the user terminal that caused the ticket gate abnormality, and the terminal tracking unit, itinerary construction unit, person tracking unit, group creation unit, and identification unit each perform a predetermined process by executing a program stored in a storage device with a processor.

To provide a ticket gate system that can identify the passenger image of a passenger who is holding a user terminal where a ticket gate abnormality has occurred from among a large number of passenger images.

Other problems and solutions disclosed in this application will be made clear in the detailed description of the invention and the drawings.

FIG. 1 is a diagram showing a station to which the ticket gate system is applied. This is an example of a ticket gate abnormality for an "invalid ticket type." FIG. 1 is a diagram in which passenger images are plotted in a feature space. FIG. 1 is a diagram in which passenger images are plotted in a feature space. This is an example of a "battery dead" type ticket gate abnormality. FIG. 2 is a functional block diagram of the ticket gate system. FIG. 1 is a block diagram of an information processing device. 13 is a flowchart of the ticket gate function. 13 is a flowchart showing group creation. 13 is a flowchart of a process for dealing with a ticket gate abnormality of the "dead battery type." 13 is a flowchart of a ticket gate abnormality response process for an "invalid ticket type." 13 is a flowchart of a clustering process. 13 is an example of a data structure of group data (image). 13 is an example of a data structure of group data (user ID). 13 is an example of user allocation information (initial values). 13 is an example of image allocation information (initial values). 11 is an example of user allocation information (processing result). 11 is an example of image allocation information (processing result). 13 is an example of a data structure of itinerary data. 13 is an example of a display screen on an operation terminal. 13 is an example of an inquiry screen displayed on a user terminal.

Figure 1 shows a station 1 to which the ticket gate system of this embodiment is applied. A passenger carries a mobile device 2 such as a smartphone or smartwatch. A passenger ticket application for using trains is installed on the mobile device 2. Many wireless base stations 3 are installed in the station or train, and one of the wireless base stations 3 is constantly in communication with the mobile device 2 to monitor whether it is alive or not. This allows the itinerary of the mobile device 2, such as entering station X and exiting station Y, to be determined, and the fare according to the itinerary is paid using the fare payment method (e.g., credit card) linked to the passenger ticket application of the holder of the mobile device 2 (passenger).

The mobile device 2 and wireless base station 3 allow passengers to seamlessly locate themselves while they are using the train and keep track of their itineraries, making it possible to eliminate ticket gates from stations. Ticket gates disrupt the flow of people, so eliminating ticket gates has the advantage of allowing passengers to enter and exit the station more smoothly even when there are many passengers getting on and off.

However, a malfunction of the mobile device 2 may prevent communication between the mobile device 2 and the wireless base station 3, making it impossible to confirm the itinerary, or the ticket application may be invalid, making it impossible to pay the fare. Therefore, some kind of ticket gate function is needed to allow direct access to a passenger carrying a mobile device 2 that is causing a ticket gate malfunction, instead of a ticket gate.

In this embodiment, a camera 4 is installed in the passage between the platform and the station entrance/exit, and the ticket gate function is realized by using image information from the camera 4. In the following, the shooting area of the camera 4 is referred to as the ticket gate area. Since recent wireless positioning can distinguish objects with an identification accuracy of about 50 cm, by comparing the image information captured by the camera 4 with the position information of the mobile device 2 that has passed through the ticket gate area, it is possible to link the image of the mobile device 2 and the passenger to the limit of the positioning accuracy.

However, although the positioning accuracy has improved, if mobile devices 2 pass through the ticket gate area at a distance below the limit of positioning accuracy due to reasons such as congestion in the station passageway or reduced positioning accuracy due to the effects of multipath or obstruction, it becomes impossible to link the passenger image captured by camera 4 with the mobile device 2 on a one-to-one basis. Even in such cases, it is desirable to be able to identify the passenger image holding the mobile device that caused the ticket gate abnormality from the passenger image in the captured image. In the ticket gate system of this embodiment, by analyzing image data captured by camera 4 at another station together with the position data of the mobile device 2 that passed through the ticket gate area, it is possible to identify the holder of a mobile device 2 that passed through a ticket gate area at a station at a distance below the limit of positioning accuracy from another mobile device.

Figure 4A shows a functional block diagram of the ticket gate system 10. The ticket gate system 10 functions as a passenger ticket system by cooperating with terminals and other systems via a network. Figure 4A shows the ticket gate system 10 together with the other elements that make up the passenger ticket system.

Specifically, the ticket gate system 10 is realized by an information processing device including a processor (CPU) 51, memory 52, storage device 53, input/output device 54, communication device 55, and bus 56 as main components as shown in FIG. 4B. The processor 51 functions as a functional unit that provides a predetermined function by executing processing according to a program loaded in the memory 52. The storage device 53 stores data and programs used in the functional unit. The input/output device 54 includes input devices such as a keyboard and a pointing device, and an output device such as a display. The communication device 55 enables communication with other information processing devices via a network. These are connected to each other so that they can communicate with each other via the bus 56. It is not necessary to realize all of the functions of the ticket gate system 10 in one information processing device, and they may be realized in multiple information processing devices.

In FIG. 4A, the user terminal 32 and the station camera 35 correspond to the mobile device 2 and the camera 4 shown in FIG. 1, respectively. The position information of the user terminal 32 is calculated by the positioning unit 33. The positioning method of the positioning unit 33 is not particularly limited, and the positioning may be performed by receiving a signal from a beacon 5 installed in the station 1, or based on the reception strength of a signal transmitted to and received from a plurality of wireless base stations 3. In addition to the reception strength, the positioning may be performed by estimating the arrival angle and arrival time. The positioning unit 33 calculates the position information of the user terminal 32 at a predetermined frequency, and the calculated position data of the user terminal is accumulated in the terminal tracking unit 12 of the ticket gate system 10. The itinerary construction unit 13 constructs an itinerary for a passenger who possesses the user terminal 32 by identifying the entrance station and the exit station of the user terminal 32 based on the position data accumulated in the terminal tracking unit 12. The constructed itinerary data is stored in the itinerary database 14. Note that, although an example of performing positioning by the user terminal 32 is shown here, a configuration in which positioning is performed on the base station side is also possible. In this case, the location information of the user terminal 32 is sent from the base station to the terminal tracking unit 12 together with information identifying the user terminal, and is stored.

Figure 13 shows an example of the data structure of itinerary data 100. Itinerary data 100 includes user ID 101, date of use 102, entry time 103, entry station 104, exit time 105, exit station 106, abnormality details 107, response result 108, settlement date 109, and settlement method 110. User ID 101 is an ID (identification information) uniquely assigned to user terminal 32. Entry station 104 and exit station 106 are identified based on the position data of user terminal 32, and are registered in itinerary data 100 together with the date and time. If no ticket gate abnormality occurs, abnormality details 107 is registered as no abnormality, and the result of the settlement process performed by fare settlement system 42 (see Figure 4A) is registered in settlement date 109 and settlement method 110. On the other hand, if a ticket gate abnormality occurs, its details are registered in abnormality details 107, and the response result is registered in response result 108. The itinerary data 100 is updated as needed based on the location data of the user terminal stored in the terminal tracking unit 12. The itinerary data 100 is also updated based on information identified by the identification unit 11, which will be described later.

Returning to the explanation of FIG. 4A, the image (video) from the station camera 35 is sent from the station camera 35 to the person tracking unit 15 of the ticket gate system 10, which extracts passengers in the image. This process can use known techniques for identifying and tracking people in the image from the features of the image. Details will be described later, but the group creation unit 16 stores image data of multiple passengers who passed through the ticket gate area in close proximity to each other and the user IDs of the user terminals 32 that passed through the ticket gate area during the time period when the images were taken in the group database 17 as group data. The identification unit 11 uses the group data stored in the group database 17 to identify the passenger who owns the user terminal 32 that caused the ticket gate abnormality, and updates the itinerary data 100 according to the response results.

Figure 5 shows a flowchart of the ticket gate function of the ticket gate system 10. The left side of the flowchart shows the functional parts of the ticket gate system 10 that execute each step.

First, the identification unit 11 checks whether new group data has been stored in the group database 17 by the group creation unit 16 (S01). If new group data has been stored, the following process is performed. Details of the group data and the method of creating it will be described later.

The number of user IDs n in the newly registered group data is compared with the number of passenger image data m (S02). If n<m, the passenger image data corresponding to the user ID causing the mismatch is identified (S03). This type of ticket gate abnormality is hereinafter referred to as a "battery dead type" ticket gate abnormality based on its typical cause. On the other hand, if n=m, it is confirmed whether there is a user ID with an invalid ticket (S04). If n=m and there is no user ID with an invalid ticket, the system is normal and there is no need to analyze the group data. On the other hand, if n=m and there is a user ID with an invalid ticket, the passenger image data corresponding to the user ID with the invalid ticket is identified (S05). This type of ticket gate abnormality is hereinafter referred to as an "invalid ticket type" ticket gate abnormality. In response to the ticket gate abnormality, the itinerary data 100 in the itinerary database 14 is updated based on the results of identifying the user ID and passenger image in step S03 or step S05 (S06).

Then, the notification unit 18 takes the following measures depending on the type of ticket gate abnormality identified by the identification unit 11. In the case of a "battery dead type" ticket gate abnormality, after the user terminal is restored, an inquiry is output to the user terminal as to whether the complementation performed by the identification unit 11 is correct, and the entry and exit stations in the itinerary data 100 are finally confirmed based on the response (S07). The settlement request unit 20 requests fares from the fare settlement system 42 based on the confirmed itinerary data, and updates the itinerary data 100 based on the result (S08).

On the other hand, in the case of an abnormality at the ticket gate for the "invalid ticket type," the analysis results are displayed on the operation terminal 31 where station staff can check (S09). The station staff identifies the passenger holding the user terminal with the invalid ticket based on the passenger image information displayed on the operation terminal 31, and handles the fare settlement. The station staff updates the itinerary data 100 via the notification unit 18 to reflect the results of the fare settlement (S10).

Figure 6 shows a group creation flowchart implemented by the group creation unit 16. Figures 10A and 10B show examples of group data created by this flow.

First, a new empty group is created (S11). When a new passenger image is detected in the ticket gate area (S12), it is determined whether the difference in the passing time of the last passenger image included in the group is equal to or greater than a predetermined value (S13). If it is less than the predetermined value, the passenger image is added to the group, and if it is equal to or greater than the predetermined value, the passenger image is not added to the group and the group is defined. Here, the predetermined value for determining whether or not to add to the group is determined based on the positioning accuracy of the user terminal 32. As a result, the image of the passenger who holds the user terminal 32 that cannot be identified from the position information is included in the group. By steps S11 to 14, group data (images) such as those shown in FIG. 10A are created. A group ID that identifies the group created by steps S11 to 14 is registered in group 61, and a passenger image ID that identifies the passenger image is registered in image 62. For example, it is shown that group I includes four passenger images consisting of IMG1 to 4.

Once a group has been defined, the user IDs and position data of user terminals acquired during the range (called the passage period) between the group's start time (the time when the first passenger in the group passes through the ticket gate area) and end time (the time when the last passenger in the group passes through the ticket gate area) are acquired from the terminal tracking unit 12 (S15). Next, the time series feature values of the user terminal's position data are calculated, and user terminals with a similarity greater than or equal to a predetermined value are designated as nearby users (S16). In other words, nearby users are users who cannot be distinguished based on position data. Next, the time series feature values of the user terminal position estimated from the image data are compared with the time series feature values of the user terminal's position data, and passenger images with a similarity greater than or equal to a predetermined value are designated as assignable images (S17).

By steps S15 to 17, group data (user ID) as shown in Figure 10B is created. Group 66 has the same group ID as group 61 in Figure 10A, and the user ID identified in step S15 is registered in user ID 67, the user ID of the nearby user identified in step S16 is registered in nearby user 68, and the passenger image ID identified as the assignable image in step S17 is registered in assignable image 69. The example of group data in Figures 10A and B corresponds to the case shown in Figure 2A, so the processing in steps S15 to 17 will be explained along with the examples of Figures 10B and 2A.

10B and 2A, user IDs B to E are identified as user IDs during the passage period of group I. In FIG. 2A, the time series feature of each user ID obtained in step S16 is displayed as a circle surrounding an alphabet representing the user ID. The time series feature is a feature of the change in the position data of the user ID during the passage period, and the closer the user IDs are to each other through the ticket gate area during the passage period of group I, the more similar the time series feature will be, and the farther the user IDs are from each other through the ticket gate area during the passage period of group I, the more different the time series feature will be. Therefore, nearby users who are holders of user terminals that cannot be distinguished by position data are identified based on the similarity of the time series feature. In the example of group I, for example, the time series feature of user ID B is similar to the time series feature of user ID C (the circles overlap), so user ID C is a nearby user of user ID B. In contrast, the time series feature values of user ID B are not similar to the time series feature values of user IDs D and E (the circles do not overlap), so user IDs D and E are not nearby users of user ID B. In a similar manner, nearby users can be found for each user ID included in the group.

On the other hand, since the user terminal position can also be estimated from image data, in step S17, the position information obtained from the image is used to calculate the estimated time series feature of the user terminal position in the same manner as in step S16. This makes it possible to associate passenger image data with position data. Specifically, in the example of Group I, the time series feature of user ID B is compared with the estimated time series feature of passenger image IDs IMG1 to 4, and passenger image IDs having estimated time series feature similar to the time series feature of user ID B are identified as assignable images. As shown in Figure 10B, in the example of Figure 2A, passenger image IDs IMG1 to 2 are identified as assignable images for user ID B.

The group data 60, 65 obtained in this manner is registered in the group database 17 (S18), and the group registration process by the group creation unit 16 ends. The group creation unit 16 executes the flowchart in FIG. 6 each time a passenger image is detected from the images from the station premises cameras 35, and creates group data and registers it in the group database 17.

Next, we will explain the processing that the identification unit 11 executes based on the group data registered in the group database 17 when a ticket gate abnormality occurs.

Figure 7 shows a flowchart of the ticket gate abnormality response process for the "battery dead type" (step S03 in the flowchart of Figure 5). An example of a ticket gate abnormality for the "battery dead type" is shown in Figure 3.

First, a list of related group candidates that can be reached from this group or that can reach this group is created based on the location and time of the group with the mismatch and the train schedule (S21). The train schedule acquisition unit 19 (see FIG. 4A) acquires the train schedule from the train operation management system 41. In the example of FIG. 3, the group with the mismatch is group i. Since group i is an exit station, the related group candidates are groups that have entered any station so that they can reach this group at the time group i exits station Y. Note that in order to reduce the amount of calculation in the subsequent processing, it is not necessary to extract all reachable groups at once. For example, the railway section used by the user may be estimated based on past usage records, and related group candidates may be preferentially extracted from sections with a usage rate of a certain level or higher, and the sections from which related group candidates are extracted may be gradually expanded.

Next, the abnormality details 107 (see FIG. 13) of the itinerary data 100 of the user IDs included in each related group candidate are obtained, and groups including user IDs with details other than "no abnormality" are extracted as related groups (S22). Since the user terminal 32 is constantly under alive management, if the battery runs out during the trip and the location information cannot be obtained, this is registered in the abnormality details 107 of the itinerary data 100. Therefore, from the abnormality details registered in the itinerary data 100, the group IDs of related groups including user IDs that are candidates for mismatched users can be extracted from the related group candidates extracted in step S21. In the example of FIG. 3, group ii (user ID C is a candidate for a mismatched user) and group iii (user ID G is a candidate for a mismatched user) are extracted as related groups.

If there is only one candidate for the mismatched user, the only candidate is identified as the mismatched user (S23, 24), and the abnormality content 107 of the mismatched user's itinerary data 100 is updated to "trip ended midway" (S28).

On the other hand, when there are multiple mismatched user candidates as shown in FIG. 3, a mismatched user is identified from the mismatched user candidates based on the similarity of the passenger images (S23 to S27). To evaluate the similarity of the passenger images, multiple predetermined image features are calculated for each passenger image, and each passenger image is plotted in a feature space spanned by the multiple image features. The similarity of the passenger images is determined by the distance of the passenger images in the feature space. Therefore, for each mismatched user candidate, the user ID and the user ID of the nearby user are acquired as clustering targets for determining the similarity of the passenger images (S25), clustering is performed on the passenger images to be clustered, and the user ID is assigned to the passenger image (S26). Details of this will be described later. As a result, a user ID that is assigned to a passenger image of a group in which a mismatch occurs (group i in the example of FIG. 3) and is not included in the group is identified as a mismatched user (S27). After that, the abnormality content 107 of the itinerary data 100 of the mismatched user is updated to "Ended midway through the itinerary" (S28).

Figure 8 shows a flowchart of the ticket gate abnormality response process for the "invalid ticket type" (step S05 in the flowchart of Figure 5). Also, an example of a ticket gate abnormality for the "invalid ticket type" is shown in Figure 2A.

First, the group data is monitored, and awaits until there are two or more groups that include a user holding an invalid ticket (S31). In the example of FIG. 2A, the user ID C in group I is the user ID of a user holding an invalid ticket, and when that user exits from station Y, the group creation unit 16 creates group II, which is a second group that includes a user holding an invalid ticket.

After that, all groups that include users who have invalid tickets are identified from the group database 17, and groups that include the user ID of the user who has the invalid ticket or the user ID of a nearby user are set as related groups (S32). In the example of FIG. 2A, group III, which includes user ID A, who is a nearby user of user ID C in group II, is included in the related groups and is set as a clustering target.

After that, clustering is performed on the passenger images to be clustered, and a user ID is assigned to each passenger image (S33). The details are the same as in step S26 in the flowchart of FIG. 7, and will be described later. For users who hold invalid tickets identified as a result of this, the abnormality content 107 in the itinerary data 100 is updated to "invalid ticket" (S34).

Figure 9 shows a flowchart of the clustering process (step S26 in the flowchart of Figure 7, and step S33 in the flowchart of Figure 8). In the clustering process, since it is not possible to directly identify a passenger image corresponding to a specific user ID, the similarity of the passenger images is used to first identify passenger images of nearby users, and then the passenger image corresponding to the specific user ID is identified by a process of elimination.

First, all passenger images to be clustered are obtained from the group data stored in the group database 17 (see Figures 10A and 10B) (S41), and the user allocation information and image allocation information are initialized (S42-S43). The clustering targets are the group data in which the ticket gate abnormality occurred and its associated group data. From this, it is advisable to narrow down the data to the user ID corresponding to the user terminal that caused the ticket gate abnormality and the user IDs of nearby users, and passenger images that can be assigned to them. Figure 11A shows the initialized user allocation information 70, and Figure 11B shows the initialized image allocation information 75. Note that Figures 11A and 11B are based on the example shown in Figure 2A.

The user ID 71 of the user assignment information 70 shown in FIG. 11A is the user ID of the clustering target obtained in step S25 in the flowchart of FIG. 7 or step S32 in the flowchart of FIG. 8, the completion status 72 is the assignment status of the passenger image in clustering, and the assigned image 73 is the passenger image ID assigned to that user ID. Upon initialization, the completion status 72 is registered as "incomplete" and nothing is assigned to the assigned image 73.

In the image allocation information 75 shown in FIG. 11B, the passenger image ID 76 is registered as the passenger image ID, the image feature 77 is registered as the plot position of the passenger image in feature space, the allocation candidate user ID label 78 is registered as a list of user IDs that are candidates for allocation to the passenger image ID, and the allocation count 79 is registered as the number of times a user ID has been assigned to the passenger image ID. Upon initialization, the contents of the passenger image ID 76, image feature 77, and allocation candidate user ID label 78 are registered based on the group data stored in the group database 17. The allocation candidate user ID label 78 is registered in the format of "group-user ID". For example, "I-B" indicates "user ID B of group I". The initial value of the allocation count is 0.

Figure 2B shows the plot of the passenger image corresponding to image feature 77 in feature space. The plot position of image feature 77 in image allocation information 75 is expressed as (image feature 1, image feature 2).

Returning to the explanation of FIG. 9, one user ID whose completion status 72 of the user allocation information 70 is incomplete is selected (S44), and if the selected user ID is included in two or more groups, a cluster with a certain value or more of image similarity is calculated for passenger images to which the user ID has not been assigned and which have been assigned less than once (S48). If the selected user ID is included in only one group (No in S45), or if multiple clusters with two or more elements are obtained in the clusters calculated in step S48 (No in S49), a passenger image ID that includes the selected user ID in the assignment candidate user ID label 78 is set as the assigned image for the selected user ID (S46), and the completion status of the user allocation information 70 is set as provisional assignment.

In the example of FIG. 2A, user IDs that are included in only one group (No in S45) include D, F, and G. Since there are no other passenger images of the same user that correspond to these user IDs, there are no passenger images that would normally be clustered. However, while the image feature values are similar for the same person, they can also be similar for different people. In other words, the false negative rate is low but the false positive rate is high. For this reason, all passenger image IDs that can be assigned based on the location information are provisionally assigned to these user IDs, and the corresponding passenger image ID is identified by removing passenger image IDs that have been confirmed to be assigned to other user IDs. For example, passenger image IDs IMG2, 3, and 4 are mechanically provisionally assigned to assigned image 73 of user ID D.

Figure 2C shows the clustering results superimposed on the plot positions of the passenger images in Figure 2B. In this example, four clusters, clusters 81 to 84, are obtained. If there are multiple clusters containing passenger images that may correspond to the selected user ID, it is not possible to narrow down the passenger images that correspond to the selected user ID. For this reason, in this case as well (No in S49), passenger image IDs that may be assigned are provisionally assigned, and identifiable user IDs are identified in advance.

The assigned image for a user ID can be determined when only one cluster with two or more elements is found for the selected user ID. In this case, the user ID is assigned to the passenger image based on the similarity information of the passenger image.

For this reason, the passenger image ID of the cluster is set to the assigned image 73 of the user assignment information 70 of the selected user ID (S50). If the cluster contains multiple passenger images of the same group, the reciprocal of the passenger images of the same group included in the cluster is added to the assignment count 79 of the image assignment information 75 for the passenger images included in the cluster (S51). After that, the user ID whose assignment has been confirmed is excluded from the assignment candidate user ID label 78 of the image assignment information 75 (S52), and for the user assignment information 70, the completion status 72 of the selected user ID is set to completed and the completion status 72 of the provisionally assigned user ID is set to incomplete (S53), and the processes from step S44 to step S54 are repeated until all users are not incomplete. The user assignment information 70 and image assignment information 75 updated by this flow are shown in Figures 12A and 12B, respectively.

In the case of Figure 2C, it would look like this:

When the selected user ID is E, the assignable passenger image IDs are IMG3 and IMG4 in group I and IMG5 and IMG6 in group II, so passenger image IDs IMG4 and IMG5 that make up cluster 83 are identified as user ID E.

When the selected user ID is A, it is possible to identify that cluster 81 including passenger image ID IMG10 of group III is a cluster including a passenger image of user ID A, but it is not possible to identify which of passenger image IDs IMG7 and 8 belonging to the same group II is the passenger image of user ID A. In this case, in step S51, the number of allocations of passenger image IDs IMG7 and 8 is set to 0.5. Next, when the selected user ID is B, the cluster including the passenger image of user ID B must include passenger image ID IMG7 or IMG8 of group II, and since cluster 81 has already been determined, it is determined that cluster 82 is a cluster including a passenger image of user ID A. In this case, it is not possible to identify which of passenger image IDs IMG7 and 8 belonging to the same group II is the passenger image of user ID B, so in step S51, 0.5 is further added to the number of allocations of passenger image IDs IMG7 and 8, making the number of allocations 1. Ultimately, in this case, it is determined that user ID A is either passenger image ID IMG7 or IMG8.

As a result, although the passenger image of user ID C may have initially been passenger image ID IMG5, 6, or 7, because user ID E was assigned passenger image ID IMG5 and user ID A was assigned passenger image ID IMG7 or IMG8, it is determined that cluster 84 including passenger image ID IMG6 is the passenger image of user ID C. In other words, the passenger image of the user ID with the invalid ticket is identified as the passenger image of passenger image ID IMG2 in group I and passenger image ID IMG6 in group II.

In this way, the user ID of the user terminal that caused the ticket gate abnormality and/or an image of the holder are identified. The action taken based on this identification is ultimately reflected in the itinerary data 100 shown in FIG. 13. Record 91 is an example where the action taken by a station staff member in response to a ticket gate abnormality of "invalid ticket type" is registered (steps S9 and S10 of the flowchart in FIG. 5). Record 92 is an example where the exit station is estimated in response to a ticket gate abnormality of "dead battery type" (in this case, "system restore" is displayed) and the fare settlement is completed after confirmation with the user (steps S7 and S8 of the flowchart in FIG. 5).

Figure 14 shows an example of a display screen on the operation terminal 31 for urging station staff to take action when a ticket gate abnormality occurs. Display screen 120 informs station staff that a ticket gate abnormality has occurred and urges them to take action directly. By allowing station staff to directly handle the issue with passengers when a ticket gate abnormality occurs, it is expected that malicious unauthorized boarding will be curbed.

Display screen 120 displays details of the ticket gate abnormality (here, possession of an invalid ticket) 121, user ID 122, entry/exit stations and their passage times 123, 125, and images 124, 126 at the entry/exit stations. For example, in the example of FIG. 2A, image 124 is an image of group I, image 126 is an image of group II, and furthermore, images 124, 126 each display a highlighted image of a passenger who is holding an invalid ticket.

Figure 15 shows an example of an inquiry screen that the notification unit 18 uses to confirm the itinerary in the result confirmation unit 34 of the user terminal 32 when a "battery dead type" ticket gate abnormality occurs. The inquiry screen 130 displays the user's itinerary 131 restored by the ticket gate system 10. If the restored itinerary is correct, the user presses the confirmation button 132, and if it is incorrect, the user presses the correction button 133 and inputs the correct itinerary. In this way, the itinerary restored by the ticket gate system 10 can be performed on the system, eliminating the need for counter service and reducing the burden on both the user and the operator.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments and includes various modified examples. For example, the above-mentioned embodiments are described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. They may also be realized by software program code that realizes each function shown in the embodiment.

In the above embodiment, the control lines and information lines are those considered necessary for explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected. Also, although various types of information have been exemplified above in table format, this information may be managed in a format other than a table.

1: Station, 2: Mobile device, 3: Wireless base station, 4: Camera, 5: Beacon, 10: Ticket gate system, 11: Identification unit, 12: Terminal tracking unit, 13: Itinerary construction unit, 14: Itinerary database, 15: Person tracking unit, 16: Group creation unit, 17: Group database, 18: Notification unit, 19: Bus schedule acquisition unit, 20: Fare settlement request unit, 31: Operation terminal, 32: User terminal, 33: Positioning unit, 34: Result confirmation unit, 35: Station camera, 41: Operation management system, 42: Fare settlement system, 51: Processor, 52: Memory, 53: Storage device, 54: Input/output device, 55: Communication device, 56: Bus, 60: Group data (image), 61: Group, 62: Image, 65: Group data (user ID), 66: Group, 67: User ID, 68: Nearby user, 69: Assignable Image, 70: User allocation information, 71: User ID, 72: Completion status, 73: Allocated image, 75: Image allocation information, 76: Passenger image ID, 77: Image feature, 78: Allocation candidate user ID label, 79: Number of allocations, 81, 82, 83, 84: Cluster, 91, 92: Record, 100: Itinerary data, 101: User ID, 102: Date of use, 103: Entry time, 104: Entry station, 105: Exit time Time, 106: exit station, 107: details of the error, 108: response result, 109: settlement date, 110: settlement method, 120: display screen, 121: details of the ticket gate error, 122: user ID, 123: entrance station and passing time, 124: image at the entrance station, 125: exit station and passing time, 126: image at the exit station, 130: inquiry screen, 131: itinerary, 132: confirmation button, 133: correction button.

Claims (10)

A terminal tracking unit that accumulates location data indicating the location of a user terminal;
an itinerary construction unit that identifies an entrance station and an exit station of a passenger carrying the user terminal from the location data accumulated in the terminal tracking unit and constructs itinerary data of the passenger;
a person tracking unit that identifies and tracks passengers passing through a predetermined ticket gate area in a station from an image captured of the ticket gate area;
a group creation unit that creates groups of passengers who passed through the ticket gate area in close proximity to each other from the images, and registers, for each group, passenger images of the passengers included in the group and user IDs corresponding to user terminals that passed through the ticket gate area during the passage period in which the group passed through the ticket gate area, which are identified based on the position data accumulated in the terminal tracking unit, as group data;
an identification unit that, when it is determined that a ticket gate abnormality has occurred in the group data, identifies related group data which is group data created from an image captured of the ticket gate area in a station premises other than the station where the passenger image of the first group data where the ticket gate abnormality occurred was captured, assigns a passenger image to a user ID of the first group data or the related group data based on the image similarity between the passenger image of the first group data and the passenger image of the related group data, and identifies a passenger image of a passenger holding a user terminal that caused the ticket gate abnormality;
A ticket gate system characterized in that the terminal tracking unit, the itinerary construction unit, the person tracking unit, the group creation unit and the identification unit each perform predetermined processing by executing a program stored in a storage device using a processor. In claim 1,
The group creation unit calculates a first time series feature for a period during which the group passed through the ticket gate area from position data of a user terminal corresponding to a user ID registered as group data, and a second time series feature for a period during which the group passed through the ticket gate area from position information of a passenger image estimated from the image, and if the similarity between the first time series feature and the second time series feature is equal to or greater than a predetermined value, identifies the passenger image for which the second time series feature was calculated as an image that can be assigned to the user ID for which the first time series feature was calculated. In claim 1,
The ticket gate abnormality of the first group data is possession of an invalid ticket,
A ticket gate system characterized in that the identification unit identifies, in addition to the first group data, group data that includes a user ID of a user holding an invalid ticket or a user ID corresponding to a user terminal that is indistinguishable from the user terminal corresponding to the user ID holding the invalid ticket based on the location data as the related group data. In claim 1,
The ticket gate abnormality of the first group data is a mismatch between the number of passenger images and the number of user IDs in the first group data,
A ticket gate system characterized in that the identification unit identifies as the related group data group data that is created from an image captured of the ticket gate area within a station other than the station at which the passenger image of the first group data was captured, and that includes a user ID corresponding to a user terminal in which an abnormality is occurring in the itinerary data, and that may be a passenger of the first group data. In claim 4,
The ticket gate system is characterized in that the identification unit updates the itinerary data by specifying, as an exit station, the station at which a passenger image of the passenger holding the user terminal that caused the ticket gate abnormality was captured. In claim 1,
The group creation unit groups passengers who pass through the ticket gate area with a time difference of less than a predetermined value from the passenger who passed through the ticket gate area immediately before, and the time difference for grouping passengers in the same group is determined based on the positioning accuracy of the position data of the user terminal. Ticket gate system,
A fare settlement system connected to the ticket gate system and performing fare settlement,
The ticket gate system includes:
A terminal tracking unit that accumulates location data indicating the location of a user terminal;
an itinerary construction unit that identifies an entrance station and an exit station of a passenger carrying the user terminal from the location data accumulated in the terminal tracking unit and constructs itinerary data of the passenger;
a person tracking unit that identifies and tracks passengers passing through a predetermined ticket gate area in a station from an image captured of the ticket gate area;
a group creation unit that creates groups of passengers who passed through the ticket gate area in close proximity to each other from the images, and registers, for each group, passenger images of the passengers included in the group and user IDs corresponding to user terminals that passed through the ticket gate area during the passage period in which the group passed through the ticket gate area, which are identified based on the position data accumulated in the terminal tracking unit, as group data;
an identification unit that, when it is determined that a ticket gate abnormality has occurred in the group data, identifies related group data which is group data created from an image captured of the ticket gate area in a station premises other than the station where the passenger image of the first group data where the ticket gate abnormality occurred was captured, and assigns a passenger image to a user ID of the first group data or the related group data based on the image similarity between the passenger image of the first group data and the passenger image of the related group data, and identifies a passenger image of a passenger holding a user terminal that caused the ticket gate abnormality;
a payment request unit for requesting a fare to the fare payment system based on passenger itinerary data;
having
A ticket system characterized in that the terminal tracking unit, the itinerary construction unit, the person tracking unit, the group creation unit, the identification unit and the settlement request unit each perform predetermined processing by executing a program stored in a storage device using a processor. In claim 7,
Station staff have access to operational terminals that can be checked.
A ticket system characterized in that the operational terminal displays a passenger image of the passenger holding the user terminal that caused the ticket gate abnormality identified by the identification unit in a manner that is distinguishable from images of other passengers. A ticket gate system including a terminal tracking unit, an itinerary construction unit, a person tracking unit, a group creation unit, and an identification unit,
The terminal tracking unit accumulates location data indicating a location of the user terminal;
the itinerary construction unit identifies an entrance station and an exit station of a passenger carrying the user terminal from the location data accumulated in the terminal tracking unit, and constructs itinerary data of the passenger;
The person tracking unit identifies and tracks passengers passing through a predetermined ticket gate area in a station premises from an image captured of the ticket gate area,
the group creation unit creates a group of passengers who passed through the ticket gate area in a state where they were close to each other from the image, and for each group, registers as group data passenger images of the passengers included in the group and user IDs corresponding to user terminals that passed through the ticket gate area during a period in which the group passed through the ticket gate area, the user IDs being specified based on the position data accumulated in the terminal tracking unit;
When it is determined that a ticket gate abnormality has occurred in the group data, the identification unit identifies related group data which is group data created from an image captured of the ticket gate area in a station other than the station where the passenger image of the first group data in which the ticket gate abnormality occurred was captured, and assigns a passenger image to a user ID of the first group data or the related group data based on the image similarity between the passenger image of the first group data and the passenger image of the related group data, and identifies the passenger image of the passenger holding the user terminal that caused the ticket gate abnormality.
A passenger image identification method comprising: In claim 9,
The passenger image identification method is characterized in that the group creation unit calculates a first time series feature for a period of time when the group passed through the ticket gate area from position data of a user terminal corresponding to a user ID registered as group data, and a second time series feature for a period of time when the group passed through the ticket gate area from position information of a passenger image estimated from the image, and if the similarity between the first time series feature and the second time series feature is equal to or greater than a predetermined value, the passenger image for which the second time series feature was calculated is identified as an assignable image for the user ID for which the first time series feature was calculated.

Images