JP2022002030A - Pedestrian behavior analysis device and method - Google Patents

Abstract

To accurately analyze a behavior intention and behavior pattern of a pedestrian.SOLUTION: A pedestrian behavior analysis device includes: a sensor unit 13 for detecting a behavior of a pedestrian; an indoor positioning unit for performing map matching based on a context with the use of a detection value of the sensor unit 13 and map information, so as to perform indoor positioning; and a behavior recognition unit for recognizing a behavior of the pedestrian with the use of a detection value of the sensor unit 13, and recognizing to which one of three types the behavior of the pedestrian belongs among stopping, ambiguity walking, and walking. A behavior intention and/or behavior pattern of the pedestrian inside a facility is output in an analyzable manner based on a combination of the three types recognized by the behavior recognition unit.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a pedestrian behavior analysis device and a method.

Conventionally, portable devices such as portable navigation devices, smartphones, and tablet terminals are equipped with a position detection device including a GPS receiver, receive signals from GPS satellites orbiting the earth, and detect and display the current position. Although it is designed to be output to the screen etc., when a pedestrian carrying a portable device passes through an indoor commercial facility such as a department store or a shopping mall, the radio waves transmitted by GPS satellites are blocked. , The position detector will not be able to detect the exact current position. However, there has been an increasing demand in recent years for detecting an accurate current position and outputting it to a display screen or the like even when passing through an indoor commercial facility. In addition, there is a growing demand for analysis of pedestrian behavior as a customer in indoor commercial facilities.

Various methods have been proposed for indoor positioning. For example, a method of indoor positioning using wireless communication equipment such as Wi-Fi (registered trademark), Bluetooth (registered trademark), ultra-wideband wireless communication (UWB), and RFID (Radio Frequency Identification). Have been proposed in large numbers (see, for example, Non-Patent Documents 1 to 10).

Pedestrian Dead Reckoning (PDR) is a stand-alone method that does not require an external device (see, for example, Non-Patent Document 11). However, since PDR accumulates errors over time, indoor positioning methods combine PDRs with other methods to correct the accumulated errors. Map matching is a typical method for correcting the accumulated error of PDR.

The map matching method has shown the possibility of improving the position accuracy of PDR by acquiring landmarks and topology information from a map (see, for example, Non-Patent Documents 12 and 13). In recent years, floor maps of multi-story buildings such as 2.5D maps can be used in general location services such as Google Maps®. The 2.5D map is a map that does not have depth information that is three-dimensionally expressed when viewed from one direction. For example, the connection of layers in the station premises and facilities for each floor that can be seen on websites and the like. , It is a map that expresses the transfer route etc. in three dimensions. It is practical to develop an integrated indoor positioning method in consideration of the use of such indoor map information. Positioning methods based on map matching can be classified into two types: map matching based on sequences and map matching based on particle filters.

In the former method, the escalator and the corner are detected from the motion sensor and the vision sensor. The detected landmarks and movements are then used as the observed sequence to calculate the most probable trajectory on the floor map using the Hidden Markov Model (HMM). Also, the latter method based on particle filters filters impossible movements by obtaining from the map restrictions on pedestrian movements, such as being unable to pass through walls. Previous studies have shown that context map matching is an effective way to improve the accuracy of indoor positioning. Further, for example, position recognition operations such as escalators and corners are considered in map matching (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2020-085783

J. Xiong and K. Jamieson, “ArrayTrack: A Fine-Grained Indoor Location System,” USENIX Symp. Networked Syst. Des. Implement., No. 279976, pp.71-84, 2013. F. Zafari, I. Papapanagiotou, and K. Christidis, “Microlocation for internet-of-things-equipped smart buildings,” IEEE Internet Things J., vol. 3, no. 1, pp. 96-112, 2016. P. Baronti, P. Pillai, VWC Chook, S. Chessa, A. Gotta, and YF Hu, “Wireless sensor networks: A survey on the state of the art and the 802.15.4 and ZigBee standards,” Comput. Commun. , vol. 30, no. 7, pp.1655-1695, 2007. S. Holm, “Hybrid ultrasound-RFID indoor positioning: Combining the best of both worlds,” 2009 IEEE Int. Conf. RFID, RFID 2009, pp.155-162, 2009. S. Gezici et al., “Localization via ultra-wideband radios: A look at positioning aspects of future sensor networks,” IEEE Signal Process. Mag., Vol. 22, no. 4, pp.70-84, 2005. Y. Kuo, P. Pannuto, K. Hsiao, P. Dutta, and A. Arbor, “Luxapose: Indoor Positioning with Mobile Phones and Visible Light,” Proc. 20th Annu. Int. Conf. Mob. Comput. Netw., pp. 447-458, 2014. W. Huang et al., “Swadloon: Direction Finding and Indoor Localization Using Acoustic Signal by Shaking Smartphones,” IEEE Trans. Mob. Comput., Vol. 14, no. 10, pp. 2145-2157, 2015. F. Ijaz, H. K. Yang, A. W. Ahmad, and C. Lee, “Indoor positioning: A review of indoor coaxial positioning systems,” Int. Conf. Adv. Commun. Technol. ICACT, pp. 1146-1150, 2013. H. Sallouha, A. Chiumento, and S. Pollin, “Localization in long-range ultra narrow band IoT networks using RSSI,” IEEE Int. Conf. Commun., Pp. 1-6, 2017. B. Islam, MT Islam, J. Kaur, and S. Nirjon, “LoRaIn: Making a Case for LoRa in Indoor Localization,” 2019 IEEE Int. Conf. Pervasive Comput. Commun. Work. PerCom Work. 2019, pp. 423 -426, 2019. L. Fang et al., “Design of a Wireless Dead Reckoning Pedestrian Navigation System --The NavMote Experience,” IEEE Trans. Instrum. Meas., Vol. 54, no. 6, pp. 2342-2358, 2005. D. Li, Y. Gu, and S. Kamijo, “Smartphone based lifelog with meaningful place detection,” 2018 IEEE Int. Conf. Consum. Electron. ICCE 2018, vol. 2018-Janua, pp. 1-5, 2018. H. Yang, S. Cheng, H. Jiang, and S. An, “An Enhanced Weight-based Topological Map Matching Algorithm for Intricate Urban Road Network,” Procedia --Soc. Behav. Sci., Vol. 96, no. Cictp , pp. 1670-1678, 2013.

However, the above-mentioned conventional technique has not been sufficient. In particular, not only in the position of pedestrians, but also in indoor commercial facilities such as stores, eating and drinking facilities such as coffee shops and cafeterias, business facilities such as office buildings, and public cultural facilities such as libraries and museums. It was difficult to accurately analyze the behavioral intentions and behavior patterns of pedestrians.

Here, it is an object of the present invention to provide a pedestrian behavior analysis device and a method capable of accurately analyzing a pedestrian's behavioral intention and behavior pattern by solving the problems of the conventional technique.

Therefore, in the pedestrian behavior analysis device, indoor positioning is performed by performing map matching based on the context using the sensor unit capable of detecting the movement of the pedestrian, the detection value of the sensor unit, and the map information. The behavior of the pedestrian is recognized by using the detection value of the indoor positioning unit and the sensor unit, and the behavior of the pedestrian is recognized as one of three types: stop, ambiguous walking, and walking. A recognition unit is provided, and the behavioral intention and / or behavioral pattern of the pedestrian in an indoor facility is output in an analyzable manner based on the combination of the three types recognized by the behavior recognition unit.

In other pedestrian behavior analysis devices, the indoor facility is a commercial facility, a business facility, or a public cultural facility.

In yet another pedestrian behavior analysis device, the stop, ambiguous walking, and walking of the pedestrian in the indoor facility are analyzed in chronological order, and the time of the ambiguous walking in a predetermined time region is the stop time. When it is more dominant than, the pedestrian outputs so that it can be determined that the pedestrian is surveying with a rough interest without being interested in a specific display, and in the predetermined time region. When the stop time is more dominant than the ambiguous walking time, the pedestrian can determine that he / she is browsing with interest in a specific display item.

Further, in another pedestrian behavior analysis device, when walking appears in the time area occupied by the stop and the ambiguous walking, it is output so that it can be determined that the behavior is moving between stores or sections.

Further, in another pedestrian behavior analysis device, when it is determined that the pedestrian is in a predetermined store or section of the indoor facility based on the indoor positioning of the indoor positioning unit, the pedestrian is in the store or section. By analyzing the time occupancy rate of stoppage and ambiguous walking in time, the degree of interest of the pedestrian in the store or section can be output in an indexable manner.

In yet another pedestrian behavior analysis device, the sensor unit further includes an accelerometer and a gyroscope, and the pedestrian behavior analysis device shifts from a hand-held posture to a horizontal posture with rotation. By standing still after receiving an impact, the time-series data of the accelerometer and the gyroscope can be output so that it can be determined that the pedestrian behavior analysis device is placed on the table.

In yet another pedestrian behavior analysis device, it is possible to determine that the pedestrian is seated by further determining that the pedestrian behavior analysis device is placed on the table.

In yet another pedestrian behavior analysis device, as a result of analyzing the pedestrian's stop, ambiguous walking, and walking in the indoor facility in chronological order, the walking leads to a stop, and then a long stop and a short ambiguous. If it is determined that the pedestrian is seated after the walking is repeated, it can be determined that the pedestrian has entered the restaurant.

In yet another pedestrian behavior analysis device, the pedestrian's staying time at the restaurant is estimated from the time until a clear walk after it is determined that the pedestrian has entered the restaurant. It is possible.

In yet another pedestrian behavior analysis device, as a result of analyzing the pedestrian's stop, ambiguous walking, and walking in the indoor facility in time series, when a long stop and a short ambiguous walking are repeated, when the long stop and the short ambiguous walking are repeated, The waiting time in the queue and the number of people in the queue are output in an estimateable manner.

In yet another pedestrian behavior analysis device, the map information includes a floor map capable of identifying a store, and a history of the pedestrian operating and browsing the pedestrian behavior analysis device while staying at the store. In addition, the pedestrian's taste is output in an estimateable manner.

Further, the other pedestrian behavior analysis device further outputs advertisements or discount information in a displayable manner according to the visit history and purchase history of the pedestrian's store within a predetermined period.

Further, in another pedestrian behavior analysis device, the degree of congestion of the store can be estimated and output based on the time of participation in the queue of the store and the time of sitting.

In the pedestrian behavior analysis method, the pedestrian movement is detected, the detected pedestrian movement and the map information are used, map matching is performed based on the context, and indoor positioning is performed, and the detected pedestrian movement is performed. The movement of the pedestrian is used to recognize the behavior of the pedestrian, and it is recognized whether the behavior of the pedestrian is one of the three types of stop, ambiguous walking, and walking, and the three types recognized. Based on the combination of, the behavioral intention and / or behavioral pattern of the pedestrian in the indoor facility is output in an analyzable manner.

According to the present disclosure, it is possible to accurately analyze the behavioral intentions and behavioral patterns of pedestrians.

It is a figure which shows the outline of the positioning apparatus for a pedestrian in this embodiment. It is a figure explaining the method of detecting walking in this embodiment. It is a figure explaining the method of estimating the stride length in this embodiment. It is a figure explaining the algorithm which estimates the direction in this embodiment. It is a figure explaining the algorithm which detects a turn in this embodiment. It is a figure explaining the algorithm which estimates the rank in this embodiment. It is a figure explaining the algorithm of PDR in this embodiment. It is a figure explaining the map matching in this embodiment. It is a figure which shows the classification of the behavior of a pedestrian and the possession posture of a positioning device for a pedestrian in this embodiment. It is a figure explaining the method of discriminating the behavior of a pedestrian and the possession posture of a pedestrian positioning device in this embodiment. It is a figure explaining the method which the pedestrian positioning apparatus in this embodiment analyzes a pedestrian's behavioral intention and behavioral mode. It is a figure which shows the result of the experiment which discriminates the behavior of a pedestrian and the possession posture of the positioning device for a pedestrian in this embodiment. It is a figure which shows the result of the experiment which discriminates the behavior of a pedestrian in this embodiment. It is a figure which shows the experimental environment of the indoor positioning and the behavior analysis of a pedestrian in this embodiment. It is a figure which shows the experimental result of the indoor positioning and the behavior analysis of a pedestrian in this embodiment. It is the first figure which shows the consideration about the experimental result of the indoor positioning and the behavior analysis of a pedestrian in this embodiment. It is the 2nd figure which shows the consideration about the experimental result of the indoor positioning and the behavior analysis of a pedestrian in this embodiment. It is a 3rd figure which shows the consideration about the experimental result of the indoor positioning and the behavior analysis of a pedestrian in this embodiment. It is a 4th figure which shows the consideration about the experimental result of the indoor positioning and the behavior analysis of a pedestrian in this embodiment. It is a 5th figure which shows the consideration about the experimental result of the indoor positioning and the behavior analysis of a pedestrian in this embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an outline of a pedestrian positioning device according to the present embodiment. In the figure, (a) is a diagram showing the appearance of a pedestrian positioning device, (b) is a diagram showing an example of time-series data acquired by the pedestrian positioning device, and (c) is a pedestrian positioning device. It is a figure explaining the outline of the operation of.

In the figure, reference numeral 11 is a pedestrian positioning device as a pedestrian behavior analysis device according to the present embodiment. The pedestrian positioning device 11 is a portable device that can be carried and carried by a person, receives a signal from a positioning satellite (not shown), detects the current position, and displays a display screen of the display device or the like. Any kind of device can be used as long as it can output to, for example, a mobile phone or a smartphone, but a tablet terminal, a PDA, a small personal computer, a wearable computer, a portable game machine, or a portable navigation system. It may be any device or the like. The pedestrian positioning device 11 is a kind of computer, such as a CPU, an arithmetic device such as an MPU, a storage device such as a semiconductor memory, a liquid crystal display, an LED display, a display device such as a CRT, a keyboard, a joystick, and a cross. It includes an input device such as a key, a push button, and a touch panel, a display control device for controlling the display device, various sensors, and a transmission / reception device such as a communication interface. Here, it is assumed that the pedestrian positioning device 11 is a smartphone.

Further, the pedestrian positioning device 11 includes a receiving unit, a positioning unit, and a communication unit in addition to the sensor unit 13.

The sensor unit 13 includes an accelerometer 13a that detects the acceleration of the pedestrian positioning device 11 and a gyroscope 13b that detects the angle (attitude), angular velocity, or angular acceleration of the pedestrian positioning device 11. A magnetic sensor (Magnetometer) 13c for detecting the magnetism of the surrounding environment of the pedestrian positioning device 11 and a pressure sensor (Barometer) 13d for detecting the pressure in the surrounding environment of the pedestrian positioning device 11 are included. The sensor unit 13 can detect the movement of the pedestrian positioning device 11 and the movement of the pedestrian carrying the pedestrian positioning device 11 and the surrounding environment.

The receiving unit is, for example, the same as a commercially available GPS receiver, and has a receiving unit that receives satellite signals transmitted from positioning satellites and satellite signals from all positioning satellites received by the receiving unit. A measurement unit that measures and outputs the current position of the pedestrian positioning device 11 and an antenna for receiving a satellite signal transmitted from the positioning satellite are included. Specifically, the positioning satellite is a GPS satellite that orbits the earth, but may include positioning satellites such as the COMPASS system in China, the GALILEO system in Europe, and the Quasi-Zenith satellite system in Japan. Further, the receiving unit outputs information such as the position of the positioning satellite received by the receiving unit and information measured by the measuring unit at predetermined time intervals (for example, every second).

The positioning unit has an initial position calculation unit that calculates the initial position of the pedestrian positioning device 11 and an operation that recognizes the movement motion of a pedestrian carrying the pedestrian positioning device 11 based on the output of the sensor unit 13. The recognition unit, the map matching unit that performs map matching based on the initial position calculated by the initial position calculation unit, the movement motion of the pedestrian recognized by the motion recognition unit, and the map information, and the map performed by the map matching unit. An output determination unit as a current position determination unit that determines and outputs the current position of the pedestrian positioning device 11 based on the matching result, and a floor map including indoor coordinate information as a floor map of a station or a multi-story building. It includes a map database that stores and stores map information, including information on a 2.5D map.

Specifically, the initial position calculation unit calculates the initial position of the pedestrian positioning device 11 using the output of the receiving unit. Then, the motion recognition unit performs an motion such as moving to the upper floor using an elevator or walking on the same floor based on the motion of the pedestrian detected by the sensor unit 13 or the surrounding environment. recognize. Further, the map matching unit specifies the position of a passage or the like on the map on which the pedestrian has moved, based on the initial position calculated by the initial position calculation unit, the movement of the pedestrian recognized by the motion recognition unit, and the like. .. Further, the output determination unit determines and outputs the current position of the pedestrian positioning device 11 based on the position of the pedestrian specified by the map matching unit. As the information of the 2.5D map stored in the map database, for example, the information provided by the general location information service such as Google Map (registered trademark) can be used.

The communication unit is, for example, the same as a commercially available communication module, and information (not shown) is shown through a communication network such as a wired or wireless public communication network, a dedicated communication network, a mobile phone network, or the Internet. Can communicate with the provider server. The communication unit can also communicate with wireless communication equipment such as Wi-Fi (registered trademark), Bluetooth (registered trademark), ultra-wideband wireless communication, and RFID. Then, the pedestrian positioning device 11 communicates with the information providing server via the communication unit, so that, for example, two-dimensional map information of an area including the position of the pedestrian positioning device 11 or a desired area, 3 It is possible to acquire map information such as three-dimensional map information and 2.5D map information, and various information such as road information and weather information. Therefore, it is not necessary to store a large amount of map information in the map database, and only the map information of the necessary area may be acquired from the information providing server and stored at any time.

The pedestrian positioning device 11 in the present embodiment is an indoor positioning unit (not shown) that performs indoor positioning by performing map matching based on the context using the detection value of the sensor unit 13 and map information from the viewpoint of function. And a behavior recognition unit (not shown) that recognizes the pedestrian's behavior using the detected value of the sensor unit 13 and recognizes whether the pedestrian's behavior is one of the three types of stop, ambiguous walking, and walking. Has. Then, the pedestrian positioning device 11 performs indoor positioning and action recognition as shown in FIG. 1 (c) based on the acquired time-series data as shown in FIG. 1 (b), and walks. The position and state of the pedestrian are determined, and the behavior of the pedestrian in operation can be analyzed and output by the pedestrian positioning device 11 alone.

In the present embodiment, the pedestrian positioning device 11 is provided in a business facility such as an office building, an indoor commercial facility such as a department store or a shopping mall, or a public cultural facility such as a library or a museum. Explains how to analyze the behavioral intentions and behavioral patterns of pedestrians who carry them.

In this case, among the movements of pedestrians, the position recognition movements that contribute to recognizing a specific position include movement movements between floors using escalators and elevators, movements on the same floor such as rotation, and stores. Includes walking outside and shopping in-store. In particular, in the present embodiment, an LSTM (Long Short-Term Memory) network, which is a kind of recurrent neural network, is used, and a shopping operation, a queue waiting operation, a seating operation, and the like are recognized from the output of the sensor unit 13. Further, in the present embodiment, these position recognition operations, the locus obtained by the PDR, and the map information of the 2.5D map are integrated in the hidden Markov model to perform accurate indoor positioning. The location of these building facilities assists in performing context-based map matching, as the map information in the 2.5D map contains the location of escalators, elevators, corners and stores as well as the corresponding motion information. Used as information. In particular, the shopping action is used to constrain the trajectory of shoppers entering the store area within the hidden Markov model.

Next, indoor positioning using PDR and map matching based on the context will be described.

FIG. 2 is a diagram illustrating a method of detecting walking in the present embodiment, FIG. 3 is a diagram illustrating a method of estimating a stride in the present embodiment, and FIG. 4 is a diagram illustrating an algorithm for estimating a direction in the present embodiment. The figure to be described, FIG. 5 is a diagram for explaining an algorithm for detecting a turn in the present embodiment, FIG. 6 is a diagram for explaining an algorithm for estimating the number of floors in the present embodiment, and FIG. 7 is a diagram for explaining a PDR in the present embodiment. FIG. 8 is a diagram illustrating an algorithm, and FIG. 8 is a diagram illustrating map matching in the present embodiment.

PDR is a method of detecting the walking of a pedestrian from the information of the acceleration sensor 13a and the gyroscope 13b of the pedestrian positioning device 11 carried by the pedestrian, and estimating the traveling direction and stride thereof. When the initial absolute position is known, the relative position at the time of walking detection can be calculated by using the movement vector calculated from the traveling direction and the stride each time walking is detected. As shown in FIG. 7, by repeating this, that is, by estimating the position of the n + 1th step from the position of the nth step, the route taken by the pedestrian can be estimated.

Here, gait detection is a pedometer that utilizes the characteristics of acceleration when the foot lands on the ground. As in equation (1) shown in FIG. 2, obtaining triaxial acceleration obtained from the acceleration sensor 13a (x-axis, the acceleration acc _x of y-axis and z-axis directions, acc _y and acc _z) acceleration acc obtained by combining However, as shown in FIG. 2, walking is detected by detecting the peak of acc.

Further, in the estimation of the stride length, the distance traveled from the nth step to the n + 1th step is estimated as the stride length. As shown in the equation (2) shown in FIG. 3, _{the difference between the maximum value acc peak} and the minimum value acc _valley of the acceleration from the nth step walking to the n + 1st step (the part circled in FIG. 3 is the maximum value). And the minimum value) can be used to accurately estimate the stride stride. Here, K = 0.47.

Further, as shown in FIG. 4, the traveling direction of the pedestrian is a rotation matrix derived from the gyroscope 13b that detects the angular velocity by applying a low-pass filter to the rotation matrix derived from the acceleration sensor 13a and the magnetic sensor 13c. A high-pass filter is applied to, and both rotation matrices are synthesized and estimated by a complementary filter. Thereby, the angle θ of the direction direction can be estimated.

Then, as shown in FIG. 7, assuming that the traveling direction is θ, the stride is l, and the position of the pedestrian at the nth step is P (lon _n , lng _n ), the update equation represented by the equation (3) is used. Can be obtained. Here, _Clon and _Clng are point-dependent constants that convert the movement vector calculated from the traveling direction and the stride into latitude and longitude.

As described above, since PDR is a method of estimating a relative position from an initial position, there is a problem that the accumulation of errors cannot be ignored as the number of steps to be detected increases. In addition, since PDR alone does not consider the surrounding environment at all, it may return a route that a pedestrian cannot walk, such as penetrating a wall, as an estimation result.

Therefore, in the present embodiment, as shown in FIG. 8, map matching using a Viterbi algorithm based on a hidden Markov model (for example, see Non-Patent Document 14) is performed, and the map is created by using the geometry of the facility. The most probable route was estimated by collating with the created node map. For example, when a left / right turn is detected, the Viterbi algorithm is executed using it as a trigger to estimate the maximum likelihood link. The node map contains the connection / disconnection information of each node, the coordinates of the nodes, and the distance and angle between the nodes.
GD Forney, “The viterbi algorithm,” Proc. IEEE, vol. 61, no. 3, pp. 268-278, 1973.

The Viterbi algorithm is a dynamic programming algorithm that looks for the most plausible sequence of hidden states that results in an observed sequence of events. It is widely used in speech recognition and decoding of convolutional codes as a maximum likelihood estimation method for time-series states such as hidden Markov models. The basis of the Viterbi algorithm is to calculate the maximum likelihood estimation sequence from the transition probability and emission probability defined by the hidden Markov model. Assuming that the current maximum likelihood hidden state is HN _i , the probability thereof is P (HN _i ), and the length of the observation train is len, the equation (4) shown in FIG. 8 holds.

Every time a new state is observed (len = len + 1), the probability of equation (4) is calculated, and the hidden state showing the probability is saved. From the recently observed state, the estimated column showing the transition to the present is calculated by going back to the stored state sequence.

The pedestrian's turning to the left and right is detected by performing threshold processing on the integrated value of the output of the gyroscope 13b according to an algorithm as shown in FIG.

In addition, since the node map is usually created for each floor of the facility (for each floor), in order to correctly perform map matching, what floor of the facility on multiple floors has pedestrians? That is, it is necessary to estimate the number of floors (number of floors) of the place where pedestrians are. Assuming that the value of the temperature sensor at a certain point is t, the value of the barometric pressure sensor is p, _{and the atmospheric pressure above sea level is p 0} , the altitude alt at that point is expressed by the equation (5) shown in FIG. However, in reality, the altitude accuracy obtained from Eq. (5) is strongly affected by the time variation of the sea level atmospheric pressure. Also, very few smartphones are equipped with temperature sensors.

Therefore, in the present embodiment, t = 26, and the difference between the altitude refalt at the start of measurement and the estimated altitude estAlt and the distance H between the floors of the facility are set according to the algorithm as shown in FIG. The current number of floors (number of floors) of pedestrians is estimated by comparison. Also, the altitude refalt is remeasured every time the floor changes.

Next, behavior recognition using the behavior of a pedestrian and the possessed posture of the pedestrian positioning device 11 will be described.

FIG. 9 is a diagram showing the classification of the pedestrian behavior and the possession posture of the pedestrian positioning device in the present embodiment, and FIG. 10 is a diagram showing the pedestrian behavior and the possession posture of the pedestrian positioning device in the present embodiment. FIG. 11 is a diagram illustrating a method for analyzing a pedestrian's behavioral intention and behavior mode by the pedestrian positioning device according to the present embodiment. In FIG. 10, (a) is a diagram showing a method of discriminating a combination of a pedestrian's behavior and a possession posture of a pedestrian positioning device based on an output waveform of a sensor, and (b) is a diagram showing a method of identifying a combination of a pedestrian positioning device and a pedestrian positioning device. It is a figure which shows the method of loading a deep learning technique into a pedestrian.

As shown in FIG. 9, the behavior of a customer, that is, a pedestrian, in an indoor commercial facility such as a department store or a shopping mall is 3 of stopping, loitering, and walking. It can be roughly divided into two types.

Stopping is a mode of behavior when a pedestrian is interested in a particular product and may be an indication of the intention to purchase that product. In addition, ambiguous walking is an ambiguous walking that accompanies shopping, etc., and you can browse while walking slowly while showing interest in various products displayed in show windows, etc., or by stopping for a short time. It is also considered to be an expression of the intention of being interested in the product, although it is not enough to purchase the product. Furthermore, walking is a clear walking intended for movement, etc., and is a behavioral pattern when traveling toward a predetermined destination, and is not purchased for the products displayed at the passing point. It is also considered to be a sign of the intention of not being interested.

Similarly, the behavior of pedestrians in public cultural facilities such as libraries and museums can be roughly classified into three types: stop, ambiguous walking, and walking.

A stop is a mode of behavior when a pedestrian is interested in a particular book or work of art, and may be an indication of the intention to borrow the book or appreciate the work of art. Ambiguous walking is when you are browsing slowly while showing interest in the various books on the bookshelf and the various works of art displayed in the exhibition room, or while stopping for a short time. It is also considered to be a manifestation of the intention of borrowing a specific book or appreciating a specific work of art, but not being interested in it. In addition, walking is a mode of behavior as you progress toward a given destination, with the intention of not borrowing the books lined up at the transit point or appreciating the works of art on display. It is also possible that there is.

These three types of pedestrian behavior can be identified based on the moving speed, moving direction, turning angle, and the like of the pedestrian positioning device 11 carried by the pedestrian. That is, it can be identified based on the output waveforms of the acceleration sensor 13a and the gyroscope 13b in the x-axis, y-axis, and z-axis directions.

However, these output waveforms are affected by the possession posture of the pedestrian positioning device 11. For example, when a pedestrian positioning device 11 such as a smartphone is housed in a pedestrian's pocket, the output waveform is greatly affected by the movement of the pedestrian's foot, and the peak becomes sharp. On the other hand, when the pedestrian positioning device 11 such as a smartphone is housed in the pedestrian's bag, the bag absorbs the vibration, so that the output waveform has a round peak.

Therefore, as shown in FIG. 9, the holding posture of the pedestrian positioning device 11 is also roughly classified into three types: hand grip (hand), pocket (pocket), and bag (bag). ..

In the present embodiment, the LSTM network developed by the inventor is used to obtain output waveforms in the x-axis, y-axis, and z-axis directions of the acceleration sensor 13a and the gyroscope 13b, as shown in FIG. 10 (a). Based on this, the combination of the three actions of the pedestrian and the three possessed postures of the pedestrian positioning device 11 is identified. The inventor used TensorFlow Lite, a library group for mobile terminals developed by Google LLC and released as an open source, which is one of the general frameworks for deep learning, and Fig. 10 As shown in (b), the model was trained in advance on a personal computer, converted into a format that can be used by a smartphone, and read into the pedestrian positioning device 11. As a result, the pedestrian positioning device 11 alone (stand-alone) has three actions of the pedestrian and the pedestrian based on the output waveforms of the accelerometer 13a and the gyroscope 13b in the x-axis, y-axis, and z-axis directions. The combination with the three possession postures of the positioning device 11 can be identified in real time. As shown in FIG. 10A, the LSTM network is composed of two LSTM layers, one fully connected layer, and one SOFTMAX layer.

In addition, since it is possible to detect vertical movement based on the detection value of the pressure sensor 13d, the TensorFlow Lite is used to train the model on a personal computer in advance and convert it into a format that can be used on a smartphone. By loading it into the pedestrian positioning device 11, the pedestrian positioning device 11 has an escalator up, an escalator down, an elevator up, an elevator down, and an elevator down. And five types of stay on the same floor (same floor) can be identified.

As shown in FIG. 11, in the pedestrian positioning device 11 according to the present embodiment, the pedestrian's behavior is stopped, ambiguous walking, or walking based on the output waveforms of the acceleration sensor 13a and the gyroscope 13b. Since it is possible to recognize whether it is a type or not, walking is detected by PDR only when walking is recognized. Therefore, the probability of erroneous detection of walking is reduced, and the accuracy of PDR detection is improved. In addition, only when walking is recognized, the turning used as a trigger in map matching is detected. Therefore, the probability of false detection of turning is reduced, and the accuracy of map matching detection is improved. Further, since the means of movement in the vertical direction can be recognized, matching correction is performed when a pedestrian moves from one floor to another, that is, at the time of floor transition. Therefore, the accuracy of map matching detection is improved.

In the general method of Wi-Fi fingerprinting, GPS positioning data is sucked up from a smartphone to the cloud for calibration, or estimation is performed from information of an access point whose installation position is known. With such a method, although it is possible to estimate the position of about 10 to 20 [m] in the vicinity of a known access point, it is difficult to specify the position under other indoor conditions. However, if the position of the pedestrian is specified by the pedestrian positioning device 11 in the present embodiment and the Wi-Fi fingerprint is calibrated based on the data, the positioning accuracy can be improved even with the Wi-Fi fingerprint. improves. For example, provide the member with an application of the technology used in the pedestrian positioning device 11 in the present embodiment, and calibrate the Wi-Fi fingerprint by utilizing the positioning data collected from the member. do. As a result, non-member and infrequent users, including pedestrians who visit the facility infrequently, such as Japanese or foreign travelers and business travelers, are positioned using only the Wi-Fi fingerprint. It becomes possible to acquire data and provide services.

Since the pedestrian positioning device 11 in the present embodiment performs indoor positioning and behavior recognition, it is possible to analyze the behavior of the pedestrian based on the result. As a result, it is possible to extract information such as store stay time, queue waiting time, correlation of visited stores, hot area identification, and degree of interest in stores.

Next, the results of an experiment conducted by the inventor using the pedestrian positioning device 11 according to the present embodiment will be described.

FIG. 12 is a diagram showing the results of an experiment for identifying the behavior of a pedestrian and the holding posture of a pedestrian positioning device in the present embodiment, and FIG. 13 is a diagram showing the results of an experiment for identifying the behavior of a pedestrian in the present embodiment. FIG. 14 is a diagram showing an experimental environment of indoor positioning and pedestrian behavior analysis in the present embodiment, and FIG. 15 is a diagram and a diagram showing experimental results of indoor positioning and pedestrian behavior analysis in the present embodiment. 16 is a first figure showing the consideration of the experimental results of indoor positioning and pedestrian behavior analysis in the present embodiment, and FIG. 17 is a consideration of the experimental results of indoor positioning and pedestrian behavior analysis in the present embodiment. FIG. 18 is a third figure showing consideration of the experimental results of indoor positioning and pedestrian behavior analysis in the present embodiment, and FIG. 19 is an indoor positioning and pedestrian in the present embodiment. The fourth figure showing the consideration about the experimental result of the behavior analysis, FIG. 20 is the fifth figure which shows the consideration about the experimental result of the indoor positioning and the behavior analysis of the pedestrian in this embodiment. In FIG. 13, (a) is a diagram showing the correct answer rate of identification of pedestrian behavior when the possession posture of the pedestrian positioning device is not identified, and (b) is the figure showing the possession posture of the pedestrian positioning device. It is a figure which shows the correct answer rate of the identification of the behavior of a pedestrian in the case of doing.

In the experiment, a smartphone was used as the pedestrian positioning device 11. The smartphone is Xperia SO-04J manufactured by Sony Corporation, its OS is Android 8.0.0, its sensor frequency is about 54 [Hz], and the development environment of the LSTM network is Android Studio 3.4.1. Yes, the OS of the personal computer used for model training is Ubuntu 16.04LTS, the GPU is GTX-1080, the total training data time is 2 hours 15 minutes, the total test data time is 30 minutes, and it is used. The TensorFlow Lite is tf-nightly 0.0.

First, an experiment was conducted to identify nine combinations of pedestrian behavior and possession posture of the pedestrian positioning device 11 as shown in FIG. 10 (a). As a result, as shown in FIG. 12, it was possible to identify the combination of the pedestrian's behavior and the possession posture of the pedestrian positioning device 11 with an accuracy of 86.1 [%].

Next, an experiment for identifying the behavior of a pedestrian when the possession posture of the pedestrian positioning device 11 is not taken into consideration, that is, when the possession posture of the pedestrian positioning device 11 is not identified, and a pedestrian positioning device An experiment was conducted to identify the behavior of a pedestrian when the possession posture of 11 is taken into consideration, that is, when the possession posture of the pedestrian positioning device 11 is also identified. As a result, the accuracy of identifying the behavior of the pedestrian when the possession posture of the pedestrian positioning device 11 is not taken into consideration is 47.4 [%] as shown in FIG. 13 (a). As shown in FIG. 13 (b), the accuracy of identifying the behavior of the pedestrian when the possession posture of the pedestrian positioning device 11 is taken into consideration was 88.5 [%]. From this, it can be seen that if the possession posture of the pedestrian positioning device 11 is identified, the accuracy of identifying the behavior of the pedestrian is improved.

Next, in order to evaluate the performance of the pedestrian positioning device 11 in an actual shopping mall environment, an experiment was conducted at GINZA SIX® (registered trademark) (6-10-1, Ginza, Chuo-ku, Tokyo). rice field. GINZA SIX (registered trademark) has a multi-story shopping mall, and its floor map was obtained from the location information service of Google Maps (registered trademark). As shown in Fig. 14, the experimenter visited the first store (shop 1: MACKINTOSH (registered trademark)) on the 3rd floor (3rd floor) in GINZA SIX (registered trademark), starting from the 3rd floor. Then use the escalator to go up to the 4th floor (4th floor), visit the 2nd store on the 4th floor (shop 2: Paul Smith®), and then use the elevator to the 6th floor (6th floor). I went up to and visited the cafe, which is the third store (shop 3: Starbucks Coffee (registered trademark)) on the 6th floor. The movement locus is shown by a line in FIG.

FIG. 15 shows indoor positioning, pedestrian behavior recognition, floor number, and vertical moving means determined and output by the pedestrian positioning device 11 as a result of this experiment. In FIG. 15, the pedestrian positioning device 11 can output to the display device for display, and if necessary, the pedestrian positioning device 11 is connected to a personal computer (not shown) and connected to the personal computer. It can also be output and displayed. In FIG. 15, (estimated) is the result of determination and output by the pedestrian positioning device 11, and (correct answer) is the actual behavior of the experimenter. .. Further, the portion displayed as (correct answer) is for convenience of explanation in the present embodiment, and is usually hidden.

Considering this result, first, as shown in FIG. 16, the staying time of the second store is longer than the staying time of the first store, and further, the staying time of the third store is the longest. I understand. It can also be seen that when staying at the first to third stores, walking is almost stopped or ambiguous, and walking is dominant on the aisles on each floor.

Then, as shown in FIG. 17, since the staying time of the second store is longer than the staying time of the first store, the pedestrian goes to the second store rather than the first store. It can be seen that the degree of interest in is high.

Further, as shown in FIG. 18, since the time spent at the elevator node on the 4th floor is long, it can be seen that the waiting time of the elevator for ascending to the 6th floor is long (1 minute 39 seconds).

Further, as shown in FIG. 19, the repetition of long stops and short ambiguous walks in a coffee shop, which is a third store, allows pedestrians to order items such as drinks and food, receive ordered items, and check out. It shows that he is waiting for a long time (4 minutes 59 seconds) until his turn comes while gradually progressing in the queue to perform processing and the like. From the repetition of such a long stop and a short ambiguous walk, it is possible to estimate the waiting time in the queue and the number of people in the queue.

Further, FIG. 20 shows time-series data of the output waveforms of the acceleration sensor 13a and the gyroscope 13b of the pedestrian positioning device 11 in the coffee shop which is the third store. In FIG. 20, the pedestrian positioning device 11 can output to the display device for display, and if necessary, the pedestrian positioning device 11 is connected to a personal computer (not shown) and connected to the personal computer. It can also be output and displayed. In this time-series data, the posture of the pedestrian positioning device 11 shifts to the horizontal posture with rotation from the state where it is identified that the pedestrian positioning device 11 is held by hand, and then the impact pulse is generated. It indicates that the accelerometer 13a and the gyroscope 13b stopped detecting anything after the observation. From this, it is possible to determine the pedestrian's act of placing the smartphone on the table after walking in the third store while holding the smartphone, which is the pedestrian positioning device 11, by hand.

Further, from the act of placing the smartphone on the table, the act of a pedestrian sitting in the third store can be determined. Further, by combining with the consideration shown in FIG. 19, it can be determined that the pedestrian is seated in the store after queuing when entering the third store. Furthermore, if a clear walk is observed as a pedestrian's behavior after sitting down, it can be determined that the person has left the third store. Therefore, it is possible to estimate the pedestrian's staying time at the third store. can.

Consideration of such experimental results is based on the behavioral intentions and behaviors of pedestrians not only in indoor commercial facilities such as shopping malls where experiments were conducted, but also in public cultural facilities such as libraries and museums, and business facilities such as office buildings. It can also be applied to the analysis of styles. This means, for example, that the first store and the second store correspond to a reading room for books of the first book classification and a reading room for books of the second book classification in a library, or in a museum. It is assumed that it corresponds to an exhibition room for works of art of the first genre and an exhibition room of works of art of the second genre, and for example, the third store corresponds to a rest room or a cafeteria in a library or a museum. If you assume it, you can easily understand it.

It should be noted that the pedestrian's taste may be estimated together with the history of the pedestrian operating and browsing the pedestrian positioning device 11 which is a smartphone while the pedestrian is staying in the store. In addition, advertisements or discount information may be displayed according to the visit history and purchase history of pedestrian stores within a predetermined period. Further, the degree of congestion of the store may be estimated based on the time of participation in the queue of the store and the time of sitting.

As described above, the pedestrian positioning device 11 as the pedestrian behavior analysis device in the present embodiment uses the sensor unit 13 capable of detecting the movement of the pedestrian, the detection value of the sensor unit 13, and the map information. , An indoor positioning unit that performs indoor positioning by performing map matching based on the context, and a pedestrian's behavior recognition using the detection value of the sensor unit 13, and the pedestrian's behavior is stopped, ambiguous walking, and walking. It is equipped with a behavior recognition unit that recognizes which of the three types it is, and it is possible to analyze the behavior intention and / or behavior pattern of a pedestrian in an indoor facility based on the combination of the three types recognized by the behavior recognition unit. Output.

Further, in the pedestrian behavior analysis method in the present embodiment, the movement of the pedestrian is detected, and the detected movement of the pedestrian and the map information are used to perform map matching based on the context to perform indoor positioning. , The detected pedestrian movement is used to recognize the pedestrian's behavior, and it is recognized whether the pedestrian's behavior is stopped, ambiguous walking, or walking. Based on the combination of types, the behavioral intention and / or behavioral pattern of the pedestrian in the indoor facility is output in an analyzable manner.

This makes it possible to accurately analyze the behavioral intentions and behavioral patterns of pedestrians.

In addition, indoor facilities are commercial facilities, business facilities or public cultural facilities. Further, when the pedestrian's stoppage, ambiguous gait, and gait in the indoor facility are analyzed in chronological order and the ambiguous gait time in a predetermined time region is more dominant than the stop time, the pedestrian is said to be. When it is output so that it can be determined that the person is surveying with a rough interest without being interested in a specific display, and the stop time in a predetermined time area is more dominant than the ambiguous walking time. Outputs so that it can be determined that the pedestrian is in a state of browsing with interest in a specific display item. Further, when walking appears in the time area occupied by the stop and ambiguous walking, it is output so that it can be determined that it is an action of moving between stores or sections. Furthermore, when it is determined that a pedestrian is in a predetermined store or section of an indoor facility based on the indoor positioning of the indoor positioning unit, the time occupancy rate of stopping and ambiguous walking within the time spent in the store or section is analyzed. Outputs the degree of interest of pedestrians in a store or section in an indexable manner. Further, the sensor unit 13 includes an acceleration sensor 13a and a gyroscope 13b, and the pedestrian positioning device 11 shifts from the posture held by the hand to the horizontal posture with rotation, and is stopped after receiving an impact. , The time-series data of the accelerometer 13a and the gyroscope 13b are output so that it can be determined that the pedestrian positioning device 11 is placed on the table. Further, it is possible to determine that the pedestrian is seated by determining that the pedestrian positioning device 11 is placed on the table. Furthermore, as a result of chronological analysis of pedestrian stops, ambiguous walking, and walking in indoor facilities, the pedestrians were seated after walking to a stop, followed by repeated long and short ambiguous walking. When is determined, it can be determined that a pedestrian has entered the restaurant. Furthermore, it is possible to estimate the staying time of a pedestrian at a restaurant from the time until a clear walk after it is determined that the pedestrian has entered the restaurant. Furthermore, as a result of chronological analysis of pedestrian stops, ambiguous walking, and walking in indoor facilities, when long stops and short ambiguous walking are repeated, the waiting time in the queue and the number of people in the queue can be estimated. do. Further, the map information includes a floor map that can identify the store, and outputs the pedestrian's taste in an estimateable manner together with the history of the pedestrian operating and browsing the pedestrian positioning device 11 while staying in the store. .. Further, the advertisement or the discount information is output so as to be displayable according to the visit history and the purchase history of the pedestrian's store within a predetermined period. Further, the degree of congestion of the store can be estimated and output based on the time of participation in the queue of the store and the time of sitting.

It should be noted that the disclosure herein describes features relating to suitable and exemplary embodiments. Various other embodiments, modifications and modifications within the scope and purpose of the claims attached herein can be naturally conceived by those skilled in the art by reviewing the disclosure of the present specification. be.

The present disclosure can be applied to pedestrian behavior analysis devices and methods.

11 Pedestrian positioning device 13 Sensor unit 13a Accelerometer 13b Gyroscope

Claims (14)

A sensor unit that can detect the movement of pedestrians and
An indoor positioning unit that performs indoor positioning by performing map matching based on the context using the detection value of the sensor unit and map information, and
It is provided with a behavior recognition unit that recognizes the behavior of the pedestrian using the detection value of the sensor unit and recognizes whether the behavior of the pedestrian is stopped, ambiguous walking, or walking. ,
A pedestrian behavior analysis device characterized in that the behavior intention and / or behavior pattern of the pedestrian in an indoor facility is output in an analyzable manner based on the combination of the three types recognized by the behavior recognition unit. The pedestrian behavior analysis device according to claim 1, wherein the indoor facility is a commercial facility, a business facility, or a public cultural facility. When the pedestrian's stoppage, ambiguous gait, and gait in the indoor facility are analyzed in chronological order and the ambiguous gait time in a predetermined time region is more dominant than the stop time, the pedestrian is said to be. , It outputs that it can be determined that it is surveying with a rough interest without being interested in a specific display, and the stop time in the predetermined time area is more dominant than the ambiguous walking time. The pedestrian behavior analysis device according to claim 1 or 2, wherein in some cases, the pedestrian can determine that he / she is browsing a specific display with interest. The pedestrian behavior analysis device according to claim 3, wherein when a walk appears in a time area occupied by a stop or an ambiguous walk, the behavior can be determined to be a movement between stores or sections. When it is determined that the pedestrian is in a predetermined store or section of the indoor facility based on the indoor positioning of the indoor positioning unit, the time occupancy rate of stopping and ambiguous walking within the time in the store or section is analyzed. The pedestrian behavior analysis device according to any one of claims 1 to 4, wherein the degree of interest of the pedestrian in the store or section is output in an indexable manner. The sensor unit includes an accelerometer and a gyroscope.
It is determined that the pedestrian behavior analysis device is placed on the table by shifting from the posture held by the hand to the horizontal posture with rotation and standing still after receiving an impact. The pedestrian behavior analysis device according to any one of claims 1 to 5, which can output time-series data of the acceleration sensor and the gyroscope. The pedestrian behavior analysis device according to claim 6, wherein it is possible to determine that the pedestrian is seated by determining that the pedestrian behavior analysis device is placed on the table. As a result of analyzing the pedestrian's stop, ambiguous walking, and walking in the indoor facility in chronological order, the pedestrian was seated after walking to a stop, followed by repeated long and short ambiguous walking. The pedestrian behavior analysis device according to claim 7, wherein it can be determined that the pedestrian has entered the restaurant. The pedestrian behavior analysis according to claim 8, wherein the staying time of the pedestrian at the restaurant can be estimated from the time until the pedestrian clearly walks after it is determined that the pedestrian has entered the restaurant. Device. As a result of analyzing the pedestrian's stop, ambiguous walking, and walking in the indoor facility in chronological order, when a long stop and a short ambiguous walking are repeated, the waiting time in the queue and the number of people in the queue can be estimated and output. The pedestrian behavior analysis device according to claim 1 or 2. The map information includes a floor map that can identify a store, and includes a floor map that can identify a store.
The invention according to any one of claims 1 to 10, wherein the pedestrian's taste can be estimated and output together with the history of the pedestrian operating and browsing the pedestrian behavior analysis device while staying at the store. Pedestrian behavior analysis device. The pedestrian behavior analysis device according to claim 11, which outputs advertisements or discount information in a displayable manner according to the visit history and purchase history of the pedestrian's store within a predetermined period. The pedestrian behavior analysis device according to claim 12, wherein the degree of congestion of the store is estimated and output based on the time of participation in the queue of the store and the time of sitting. Detects pedestrian movements and
Using the detected pedestrian movement and map information, map matching is performed based on the context to perform indoor positioning.
The detected pedestrian behavior is used to recognize the pedestrian's behavior, and it is recognized whether the pedestrian's behavior is stopped, ambiguous walking, or walking.
A pedestrian behavior analysis method comprising parsely outputting the behavioral intention and / or behavioral pattern of the pedestrian in an indoor facility based on the recognized combination of the three types.

Images