JP2020129234A - Customer service state analyzer, customer service state analysis system, and customer service state analysis method - Google Patents

Abstract

To determine with good accuracy that a store clerk has serviced a customer and generate highly reliability customer service information even when there exists an area in a store that is inappropriate for the execution of customer service determination.SOLUTION: The processor of a customer service state analyzer is constituted so as to: acquire the position information of a person acquired from image data in which a person inside of a store is imaged, the position information of belongings of a store clerk and the position information of a sound acquired from sound data in which sounds inside of the store are collected; determine, on the basis of the position information of the person, the position information of belongings and the position information of sounds, whether or not the store clerk has serviced a customer; generate information relating to a customer service state on the basis of the result of the determination; when making determination, distinguish whether or not the position of the store clerk is included in an effective area that is set as a portion of the monitoring area inside of the store; and, make determination when the position of the store clerk is included in the effective area.SELECTED DRAWING: Figure 10

Description

The present disclosure relates to a customer service situation analysis device, a customer service situation analysis system, and a customer service situation analysis method, in which a processor performs a process of analyzing a customer service situation of a clerk to a customer at a store and generating information on the customer service situation.

At the store, a salesclerk provides customer service such as calling out and giving guidance to customers who have visited the store, but it is important to improve customer satisfaction by promptly serving customers and not leaving them for a long time. Is important in.

As a technique for grasping the customer service situation by a clerk in such a store, conventionally, based on the position information of the person acquired from the image data of the person in the store, and the position information of the mobile terminal device owned by the clerk, An utterance that indicates the position where the clerk uttered, based on the position information of the clerk who acquired the clerk's position information and the customer's position information, and also the position information of the voice acquired from the voice data collected in the store A technology is known that acquires position information, determines whether or not a salesclerk has served customers based on the utterance position information and the position information of the customer, and generates information regarding the customer service status based on the determination result. (See Patent Document 1).

Japanese Patent No. 6300130

By the way, in the stores as described above, environmental sounds (for example, BGM and in-store broadcasting) generated in the store irrespective of the clerk's utterance, and environmental sounds intruding from outside the store irrespective of the clerk's utterance (eg, , There may be an area in which noise (such as noise due to people and vehicles coming and going) is detected (picked up). Further, there may be an area in which it is difficult to detect the clerk's utterance depending on the performance and arrangement of the microphone that collects the clerk's utterance. Such an area is not suitable for conducting customer service determination because it is difficult to properly detect the utterance of the clerk.

However, in the above-mentioned conventional technology, since no special consideration is given to areas unsuitable for conducting customer service determination, when information on the customer service situation in such an unsuitable area is generated, customer service throughout the store It can be difficult to maintain high credibility of situational information. In other words, by preferentially generating the information on the customer service status in the area suitable for conducting the customer service determination, the reliability of the information on the customer service status in the entire store may be improved.

Therefore, the present disclosure is capable of accurately determining that a clerk has performed customer service and generating highly reliable customer service information even when there is an area unsuitable for performing customer service determination in a store. The main purpose is to provide a situation analysis device, a customer service situation analysis system, and a customer service situation analysis method.

The customer service situation analysis device of the present disclosure is a customer service situation analysis device that analyzes a customer service situation of a clerk with respect to a customer in a store, and executes a process of generating information on the customer service state by a processor, wherein the processor is in a store. The position information of the person acquired from the image data of the captured image of the person, the position information of the mobile terminal device owned by the store clerk, and the position information of the voice acquired from the voice data of the voice in the store are acquired, Position information, the position information of the mobile terminal device, and based on the position information of the voice, it is determined whether or not the clerk has served the customer using a predetermined determination criterion, based on the result of the determination, When generating the information regarding the customer service situation and making the determination, it is determined whether or not the position of the clerk is included in an effective area set as a part of the monitoring area in the store, and the position of the clerk. The determination is made when is included in the effective area.

Further, the customer service situation analysis system of the present disclosure is a customer service situation analysis system that analyzes the customer service situation of a clerk to a customer in a store and generates information on the customer service state, and collects sound in the store. A device, an image pickup device for picking up an image of a person in a store, a mobile terminal device carried by a store clerk, and one or more information processing devices, wherein the one or more information processing devices are from the image pickup device. The position information of the person acquired from the output image data, the position information of the portable terminal device, and the position information of the sound output from the sound collecting device are acquired, and the position information of the person and the position of the portable terminal device are acquired. Based on the information and the position information of the voice, it is determined whether or not the store clerk has served the customer using a predetermined criterion, and based on the result of the determination, the information on the customer service situation is generated, and the determination is made. Based on the result of, to generate the information on the customer service situation, when performing the determination, whether the position of the clerk is included in the effective area set as a part of the monitoring area in the store The determination is made, and the determination is made when the position of the clerk is included in the effective area.

Further, the customer service situation analysis method of the present disclosure is a customer service situation analysis method that analyzes the customer service situation of a clerk with respect to a customer in a store, and executes a process of generating information regarding the customer service state by a processor. The position information of the person obtained from the image data of the captured image, the position information of the mobile terminal device carried by the store clerk, and the position information of the voice obtained from the voice data of the voice in the store are obtained, and the position of the person is acquired. Based on the information, the position information of the mobile terminal device, and the position information of the voice, it is determined whether or not the clerk has served the customer using a predetermined criterion, and based on the result of the determination, the customer service status. The information about the clerk, the position of the clerk determines whether the position of the clerk is included in an effective area set as a part of the monitoring area in the store, and the position of the clerk is the The above-mentioned determination is made when included in the effective area.

According to the present disclosure, even when there is an area unsuitable for performing customer service determination in a store, it is possible to accurately determine that a clerk has performed customer service, and generate highly reliable customer service information.

Overall configuration diagram of a customer service situation analysis system according to the first embodiment Plan view showing the layout of the store, the installation status of the microphone 1, the camera 2, and the beacon transmitter 3 and the divided areas set in the store Block diagram showing a schematic configuration of the mobile terminal 4, the voice processing PC 5, the image processing PC 6, the positioning management server 7, and the analysis PC 8. Explanatory diagram showing an outline of processing performed by the mobile terminal 4, the voice processing PC 5, the image processing PC 6, and the positioning management server 7. Explanatory diagram showing an outline of processing performed by the analysis PC 8 Explanatory diagram showing information generated by the voice processing PC 5, the image processing PC 6, the positioning management server 7, and the analysis PC 8. Explanatory diagram showing the outline of the process of specifying a clerk Explanatory diagram showing the outline of utterance determination and customer service determination Flow chart showing the procedure of speech determination Flow chart showing the procedure of customer service determination processing Flow chart showing the procedure for generating customer service information in FIG. Explanatory diagram showing customer service heat map display screen Explanatory diagram showing how to set the effective area for customer service determination Explanatory diagram showing the outline of setting the judgment criteria for customer service judgment Explanatory diagram showing the outline of setting the judgment criteria for customer service judgment Explanatory diagram showing the outline of setting the judgment criteria for customer service judgment Flow chart showing the procedure of presetting the judgment criteria for customer service judgment Explanatory drawing showing the procedure for presetting the judgment criteria for customer service judgment Flow chart showing the procedure of customer service determination according to the second embodiment.

A first invention made to solve the above-mentioned problems is a customer service situation analyzing apparatus that analyzes a customer service situation of a salesclerk to a customer in a store, and executes a process of generating information on the customer service situation by a processor. The processor acquires position information of a person acquired from image data of a person in a store, position information of a clerk's belongings, and position information of a voice acquired from voice data of a voice in the store. Based on the position information of the person, the position information of the belongings, and the position information of the voice, it is determined whether or not the clerk has served the customer using a predetermined determination criterion, and based on the result of the determination. When generating the information on the customer service situation and making the determination, it is determined whether or not the position of the clerk is included in the effective area set as a part of the monitoring area in the store, and the clerk. The determination is performed when the position of is included in the effective area.

According to this, since the determination is made when the position of the clerk is included in the effective area, even if there is an area in the store that is inappropriate for conducting the customer service determination, it is possible to accurately determine that the clerk has served the customer. It becomes possible to generate highly reliable customer service information.

In a second aspect of the invention, the determination is performed using a criterion relating to the voice and a criterion not relating to the voice as the criterion, and when the position of the clerk is not included in the effective area, the voice The determination is performed using only the reference irrelevant.

According to this, when the position of the store clerk is not included in the effective area, the determination is performed using a criterion other than the voice as a determination criterion, so that the store clerk does not affect the environmental sound other than the utterance of the store clerk. It is possible to more accurately determine that the service has been performed and to generate more reliable customer service information.

In the third aspect of the invention, the criterion relating to the voice includes a threshold of the utterance time of the clerk.

According to this, since the determination is performed using the threshold of the utterance time of the store clerk, it is possible to more accurately determine that the store clerk has served, and to generate more reliable customer service information.

Further, in a fourth aspect, the processor sets the threshold value of the speech time to a maximum value when the position of the store clerk is not included in the effective area, and uses the set threshold value to make the determination. It is configured to be performed.

According to this, even when the utterance of the store clerk is located in the area affected by the environmental sound, the criterion regarding the utterance time can be easily excluded from the determination criterion.

In a fifth aspect of the invention, the processor uses a threshold value of the approach time between the clerk and the customer as a reference not related to the voice, and when the position of the clerk is not included in the effective area, The threshold of the approach time is changed, and the changed threshold is used to make the determination.

According to this, when the position of the clerk is not included in the effective area, the threshold value of the approach time between the clerk and the customer is changed to make the determination, so that it is more accurately determined that the clerk has served the customer. It is possible to generate more reliable customer service information.

In a sixth aspect of the present invention, the processor uses a threshold value of the number of customers existing near the clerk as a reference not related to the voice, and when the position of the clerk is not included in the effective area, The threshold value of the number of customers is changed, and the changed threshold value is used for the determination.

According to this, when the position of the clerk is not included in the effective area, the threshold value of the number of customers existing near the clerk is changed to make the determination, so that it is more accurate that the clerk has served the customer. It is possible to determine and generate more reliable customer service information.

In a seventh aspect, the processor uses a threshold value of an approach distance between the clerk and the customer as a reference not related to the voice, and when the position of the clerk is not included in the effective area, the approach distance is set. The threshold is changed and the changed threshold is used to make the determination.

According to this, when the position of the clerk is not included in the effective area, the threshold value of the approach distance between the clerk and the customer is changed to make the determination, so that it is more accurately determined that the clerk has served the customer. It is possible to generate more reliable customer service information.

Further, an eighth invention further comprises a memory for storing the information of the determination standard, wherein the processor, when the determination is performed, stores the position of the clerk in the effective area when the position of the clerk is stored in the memory. In addition, the information of the determination criteria is read.

According to this, since the dedicated determination criterion generated in advance is used, the user can accurately determine that the clerk has performed customer service by a simple operation, and can generate highly reliable customer service information. ..

Further, the ninth invention is configured such that the processor acquires the position information of the clerk and the position information of the customer based on the position information of the person and the position information of the possessed object.

According to this, it becomes possible to easily estimate the positions of the store clerk and the customer and use those positions for determining whether or not the store clerk has served the customer.

A tenth aspect of the present invention is configured such that the processor acquires utterance position information indicating a position uttered by the store clerk, based on the position information of the voice and the position information of the store clerk.

According to this, it is possible to easily estimate the position spoken by the store clerk and use the position spoken for determining whether the store clerk has served the customer.

An eleventh aspect of the present invention is configured such that the processor determines whether or not the clerk has served a customer based on the utterance position information and the position information of the customer.

According to this, based on the utterance position information and the customer position information, it becomes possible to appropriately determine whether or not the store clerk has served the customer.

A twelfth aspect of the present invention is a customer service situation analysis system that analyzes a customer service situation of a clerk to a customer in a store and generates information regarding the customer service situation, and a sound collection device that collects voice in the store. An image pickup device for picking up an image of a person in a store, a mobile terminal device carried by a store clerk, and one or more information processing devices are provided, and the one or more information processing devices are output from the image pickup device. Position information of a person acquired from image data, position information of the mobile terminal device, and position information of voice output from the sound collecting device are acquired, and position information of the person, position information of the mobile terminal device, and Based on the position information of the voice, it is determined whether or not the clerk has served the customer using a predetermined determination criterion, based on the result of the determination, the information regarding the customer service situation is generated, and the result of the determination is Based on the generated information about the customer service situation, when performing the determination, the position of the clerk determines whether or not included in the effective area set as a part of the monitoring area in the store, The determination is performed when the position of the clerk is included in the effective area.

According to this, since the determination is made when the position of the clerk is included in the effective area, even if there is an area in the store that is inappropriate for conducting the customer service determination, it is possible to accurately determine that the clerk has served the customer. It becomes possible to generate highly reliable customer service information.

A thirteenth aspect of the present invention is a customer service situation analysis method in which a processor performs a process of analyzing a customer service situation of a clerk with respect to a customer in a store and generating information on the customer service situation, and images a person in the store. Position information of the person acquired from the image data, position information of the clerk's belongings, and position information of the voice acquired from the voice data collected in the store, position information of the person, the possession Based on the position information and the position information of the voice, it is determined whether or not the store clerk has served the customer using a predetermined criterion, and based on the result of the determination, the information on the customer service situation is generated, and When making the determination, the position of the clerk determines whether or not the position of the clerk is included in the effective area set as a part of the monitoring area in the store, and when the position of the clerk is included in the effective area. The above-mentioned determination is made.

According to this, since the determination is made when the position of the clerk is included in the effective area, even if there is an area in the store that is inappropriate for conducting the customer service determination, it is possible to accurately determine that the clerk has served the customer. It becomes possible to generate highly reliable customer service information.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram of a customer service situation analysis system according to the first embodiment.

This customer service situation analysis system analyzes the customer service situation of a clerk to a customer in a store and generates information on the customer service situation. The microphone 1 (sound pickup device), the camera 2 (imaging device), and the beacon. Transmitter 3, mobile terminal 4 (mobile terminal device), voice processing PC 5 (information processing device), image processing PC 6 (information processing device), positioning management server 7, analysis PC 8 (customer service situation analysis) Device, information processing device) and a browsing terminal 9.

The microphone 1, the camera 2, the voice processing PC 5, and the image processing PC 6 are connected to the in-store network. The in-store network is connected to an external network via the gateway 11. The mobile terminal 4 is connected to an external network via a public access point 12 such as a wireless LAN. The analysis PC 8, the positioning management server 7, and the browsing terminal 9 are connected to an external network.

The microphone 1 is installed in a store and picks up a voice uttered by a person (clerk or customer) staying in the store. The microphone 1 is a so-called omnidirectional microphone that picks up sound in all directions, and includes a microphone array in which a plurality of (for example, 16) microphone elements (sound collecting elements) are circularly arranged at equal intervals. There is. The microphone 1 collects the voice in the store with a plurality of microphone elements, generates voice data of a plurality of channels, and transmits the voice data to the voice processing PC 5. Note that the sound picked up by the microphone 1 includes environmental sounds (for example, BGM and in-store broadcasting) that are generated in the store irrespective of utterances and environmental sounds that enter from outside the store irrespective of utterances (for example, , Noise caused by traffic of people and vehicles).

The camera 2 is installed in a store and images a person (clerk or customer) staying in the store and also a person visiting the store. The camera 2 is a so-called omnidirectional camera that images in all directions. The camera 2 captures an image of the inside of the store to generate image data, and transmits the image data to the image processing PC 6.

The voice processing PC 5 receives voice data transmitted from the microphone 1 and performs voice processing on the voice data. In the present embodiment, voice detection (source position detection) for specifying the position of a sound source is performed, and a voice heat distribution state of a voice level in a store, that is, a voice heat indicating where and how much voice level (volume) is generated. Map information is generated and the sound heat map information is transmitted to the analysis PC 8.

The image processing PC 6 receives the image data transmitted from the camera 2 and performs image processing on the image data. In the present embodiment, the person staying in the store is detected from the image data, the position information for each person is acquired, and the position information is transmitted to the analysis PC 8.

A plurality of beacon transmitters 3 are installed in the store and transmit a beacon signal for positioning.

The mobile terminal 4 is a smartphone or the like, and is carried by a clerk. The mobile terminal 4 performs positioning using the beacon signal transmitted from the beacon transmitter 3 to acquire the position information of its own device, and transmits the position information to the positioning management server 7.

Although the mobile terminal 4 is positioned by the beacon signal emitted by the beacon transmitter 3 in the present embodiment, the mobile terminal may be positioned by another positioning system such as a satellite positioning system such as GPS (Global Positioning System). The positioning of 4 may be performed. Further, using a locator installed on the ceiling (network connection with the positioning management server 7), the position information of the store clerk can be obtained from the radio wave angle/intensity of the tag attached to the belongings by the store clerk and the measurement algorithm.

The positioning management server 7 collects the position information for each mobile terminal 4 from the mobile terminal 4, manages the position information for each mobile terminal 4, and transmits the position information for each mobile terminal 4 to the analysis PC 8.

The analysis PC 8 acquires voice heat map information from the voice processing PC 5, acquires position information of each person from the image processing PC 6, and acquires position information of each mobile terminal 4 from the positioning management server 7. Then, based on these pieces of information, the customer service situation of the store clerk with respect to the customer at the store is analyzed, and information representing the customer service situation is generated.

The browsing terminal 9 is a PC, a smartphone, a tablet terminal, or the like, and is operated by a user, for example, an administrator of the headquarters such as a supervisor or an administrator of a store such as a store manager. By accessing the analysis PC 8, it is possible to perform various settings related to the processing of the analysis PC 8 and to browse the information regarding the customer service status generated by the analysis PC 8. A server dedicated to information distribution may be provided separately from the analysis PC 8.

Further, when there are a plurality of stores to be managed such as a chain store, the analysis PC 8 collects the information of each store from the sound processing PC 5 and the image processing PC 6 and the positioning management server 7 for each of the plurality of stores. Then, the information on the customer service situation for each store is generated.

Next, an example of a store will be described. FIG. 2 is a plan view showing the layout of the store, the installation status of the microphone 1, the camera 2, and the beacon transmitter 3 and the divided area set in the store.

The store is provided with a doorway, a ticket issuing machine, a sofa, furniture (display shelves, display stands, etc.), a table, a counter, a toilet, a work room, a backyard, and the like. The customer enters the store from the doorway, receives the number tag at the ticket issuing machine, sits on the sofa, looks at the products displayed on the furniture, waits until there is a call, and when the call is received, Perform procedures such as purchasing products and contracts at the counter. When a customer enters the store, the store clerk provides customer service such as guidance of ticketing machines and explanation of products displayed on the furniture, or customer service at the counter.

A microphone 1, a camera 2 and a beacon transmitter 3 are installed in the store. The microphone 1 picks up a voice uttered by a store clerk or a customer staying in the store, the camera 2 images the store clerk or the customer staying in the store, and also images a customer visiting the store. In the example shown in FIG. 2, for convenience, only one device in which the microphone 1 and the camera 2 are integrally configured is shown. It can be installed in any position. In addition, a plurality of beacon transmitters 3 are installed at appropriate positions in the store, and the beacon signal transmitted from the beacon transmitters 3 is received by the mobile terminal 4 carried by the clerk.

Further, in the present embodiment, a plurality of cells (divided areas) of a predetermined size (for example, 50 cm square) equally divided into a grid are set in the monitoring area set by the user on the store map. Various processes are performed by the analyzing PC 8 in units of cells. The cell size can be changed to various sizes depending on the sensitivity of the microphone 1, the camera 2, and the mobile terminal 4.

Next, the processing performed by the present customer service situation analysis system will be described. FIG. 3 is a block diagram showing a schematic configuration of the mobile terminal 4, the voice processing PC 5, the image processing PC 6, the positioning management server 7, and the analysis PC 8. FIG. 4 is an explanatory diagram showing an outline of processing performed by the mobile terminal 4, the voice processing PC 5, the image processing PC 6, and the positioning management server 7. FIG. 5 is an explanatory diagram showing an outline of processing performed by the analysis PC 8. FIG. 6 is an explanatory diagram showing information generated by the voice processing PC 5, the image processing PC 6, the positioning management server 7, and the analysis PC 8. FIG. 7 is an explanatory diagram showing the outline of processing for identifying a clerk. FIG. 8 is an explanatory diagram showing the outline of utterance determination and customer service determination. FIG. 9 is a flowchart showing the procedure of speech determination. FIG. 10 is a flow chart showing the procedure of customer service determination. FIG. 11 is a flow chart showing a procedure for generating customer service information in FIG. FIG. 12 is an explanatory diagram showing a customer service heat map display screen.

As shown in FIG. 3, the mobile terminal 4 includes a beacon reception unit 21, a communication unit 22, a processor 23, and a storage unit 24.

The beacon receiver 21 receives the beacon signal transmitted from the beacon transmitter 3 and measures the radio field intensity of the beacon signal.

The communication unit 22 communicates with the positioning management server 7. In the present embodiment, the position information acquired by the processor 23 is transmitted to the positioning management server 7.

The storage unit 24 stores the terminal ID and the like. The storage unit 24 also stores a program executed by the processor 23.

The processor 23 executes various programs by executing the programs stored in the storage unit 24. In this embodiment, beacon positioning processing is performed.

In beacon positioning, the position information (latitude, longitude, altitude, etc.) of the mobile terminal 4 is acquired based on the radio wave intensities of beacon signals from the plurality of beacon transmitters 3. In this processing, for example, using the radio field intensity distribution information indicating the radio field intensity of each beacon transmitter 3 measured in advance at each position, from the radio field intensity of each beacon transmitter 3 acquired by the beacon receiving unit 21, The position information of the terminal 4 is acquired.

As shown in FIG. 4, the positioning management server 7 receives the position information transmitted from each mobile terminal 4, collects the position information for each mobile terminal 4, and represents the position of each mobile terminal 4 at each time. The position information for each terminal is generated, and the position information for each terminal is transmitted to the analysis PC 8. As shown in FIG. 6A, this terminal-by-terminal position information includes time, terminal ID, and position coordinate information.

As shown in FIG. 3, the voice processing PC 5 includes a communication unit 31, a processor 32, and a storage unit 33.

The communication unit 31 communicates with the microphone 1 and the analysis PC 8. In this embodiment, the voice data transmitted from the microphone 1 is received. Also, the voice heat map information generated by the processor 32 is transmitted to the analysis PC 8.

The storage unit 33 stores voice data and the like acquired from the microphone 1. The storage unit 33 also stores a program executed by the processor 32.

The processor 32 executes various programs by executing the programs stored in the storage unit 33. In this embodiment, a voice detection (source position detection) process is performed.

As shown in FIG. 4, in the voice detection process, the voice level (for example, four stages) at each position is obtained based on the voice data of a plurality of (for example, 16) channels obtained from the microphone 1, and the voice heat map is obtained. Generate information. At this time, the sound in each direction can be acquired by controlling the directivity, and the sound level at each position is acquired based on the phase difference of the sound data of a plurality of channels.

In the present embodiment, the sound level is acquired in units of cells obtained by dividing the monitoring area in a grid pattern, and as shown in FIG. 6E, sound heat map information representing the sound level in each cell at each time is generated. .. When this audio heat map information is visualized, it becomes as shown in FIG.

Note that, in the present embodiment, as shown in FIG. 2, the analysis PC 8 performs the analysis processing regarding the customer service situation on a cell-by-cell basis. The cells used in this analysis processing and the cells used in the voice heat map information are different from each other. , The sizes do not necessarily have to be the same.

As shown in FIG. 3, the image processing PC 6 includes a communication unit 41, a processor 42, and a storage unit 43.

The communication unit 41 communicates with the camera 2 and the analysis PC 8. In this embodiment, the image data transmitted from the camera 2 is received. Further, the position information for each person generated by the processor 42 is transmitted to the analysis PC 8.

The storage unit 43 stores image data and the like acquired from the camera 2. The storage unit 43 also stores a program executed by the processor 42.

The processor 42 executes various programs by executing the programs stored in the storage unit 43. In this embodiment, a person detection process is performed.

As shown in FIG. 4, in the person detection processing, a person is detected from the captured image (frame) included in the image data acquired from the camera 2, position information of the person is acquired, and the person's position information at each time is acquired. Position information for each person representing a position is generated. The position information for each person includes time, person ID, and position coordinate information, as shown in FIG. 6(B).

As shown in FIG. 3, the analysis PC 8 includes a communication unit 51, a processor 52, and a storage unit 53.

The communication unit 51 communicates with the voice processing PC 5, the image processing PC 6, and the positioning management server 7. In the present embodiment, the voice heat map information transmitted from the voice processing PC 5 is received. Further, the position information for each person transmitted from the image processing PC 6 is received. Further, the position information for each terminal transmitted from the positioning management server 7 is received.

The storage unit 53 stores information acquired from the voice processing PC 5, the image processing PC 6, and the positioning management server 7, the information generated by the processor 52, and the like. The storage unit 53 also stores a program executed by the processor 52.

In addition, the storage unit 53 stores setting information of the effective area used in the processing of customer service determination, as described later. The effective area can be set by the user.

Further, the storage unit 53 stores information on the determination standard used in the process of generating customer service information described below. The information on the determination standard includes a threshold value set for one or more preset determination indexes. Such a threshold includes at least a threshold regarding voice (that is, a threshold set for a determination index regarding voice). Further, as will be described later, when one or more determination change areas are set in a part of the monitoring area in the store, the determination reference is a dedicated determination reference set individually for the determination change area. , And normal judgment criteria set for the entire monitoring area excluding the judgment change area (that is, a special area).

The processor 52 executes various programs by executing the program stored in the storage unit 53. In the present embodiment, each process of specifying a clerk, utterance determination, customer service determination, first customer service total, second customer service total, store visit total, customer total, and clerk total is performed.

As shown in FIG. 5, in the clerk-specific process, each person in the position information for each person is based on the position information for each terminal acquired from the positioning management server 7 and the position information for each person acquired from the image processing PC 6. Is determined to be a store clerk and position information for each store clerk and position information for each customer are generated. As shown in FIG. 6B, the customer-specific position information and the clerk-specific position information include time, person ID, and position coordinate information.

In this clerk-specific process, as shown in FIG. 7A, the position where the mobile terminal 4 exists in the terminal position information, and as shown in FIG. 7B, the person exists in the person position information. When the person is present near the mobile terminal 4 by comparing the position, the person is determined to be a clerk as shown in FIG. 7C. Further, as illustrated in FIG. 7D, a person other than the person who is determined to be the salesclerk is determined to be the customer. The position information for each person regarding the person determined to be the clerk is set as the position information for each store employee, and the position information for each person about the person who is determined as the customer is set as the customer position information.

Although it is possible to generate the position information for each clerk from the position information for each terminal acquired from the positioning management server 7, the position information for each person acquired from the image of the camera 2 is the position information for each terminal by beacon positioning. Since the accuracy is higher, it is preferable to generate the salesclerk position information from the person position information.

As shown in FIG. 5, in the process of utterance determination, customer-specific position utterance information and clerk-specific position utterance information are generated. More specifically, in the utterance determination process, as shown in FIG. 9, the salesclerk-specific position information and the customer-specific position information by the salesclerk specifying process and the voice heat map information from the voice processing PC 5 are acquired (ST101. ), it is determined whether or not a voice with a predetermined level or higher is detected in the cell in which the clerk is present (ST102). Therefore, when the voice of the predetermined level or higher is detected (Yes), it is determined that the clerk speaks (ST103). On the other hand, when the voice of the predetermined level or higher is not detected (No), the voice of the store clerk is detected. It is determined that there is no utterance (ST104). After that, the position utterance information for each customer and the position utterance information for each clerk are generated (ST105).

As shown in FIG. 6C, the position clerk-specific position utterance information and the customer-specific position utterance information include the stop time and the presence or absence of utterance in the clerk-specific position information and the customer-specific position information shown in FIG. 6B. It is information added. Note that the stoppage time is the time when a store clerk or a customer stays at a certain position.

In this utterance determination processing, as shown in FIG. 8(A), the position where the voice is detected in the voice heat map information, and as shown in FIG. 8(B), the position where the store clerk exists in the position information for each store clerk. 8C is compared, as shown in FIG. 8C, when a voice having a predetermined level or higher is detected at the position where the store clerk exists, it is determined that the store clerk uttered at that position.

In addition, in the present embodiment, determination is made in units of cells obtained by dividing the monitoring area in a grid pattern (see FIG. 2), and when a voice having a predetermined level or higher is detected in a cell in which a clerk exists, the clerk speaks. Judge as something.

Since the footsteps may be erroneously determined to be utterance when the store clerk is walking, it is said that the store clerk uttered when a voice of a predetermined level or higher is detected while the store clerk is stopped. You should make a decision.

As shown in FIG. 5, in the customer service determination process, it is determined whether the clerk has performed customer service based on the salesclerk-specific position utterance information and the customer-specific position utterance information acquired in the utterance determination process, and the customer service information is obtained. (Position clerk/customer information for each salesclerk and position utterance/customer information for each customer) are acquired. In this customer service determination processing, it is determined whether or not the customer clerk has served the customer based on the utterance time of the customer clerk, the approach time between the customer and the customer, and the number of customers existing near the customer.

As shown in FIG. 8D, the clerk's utterance time is compared with the clerk by comparing the position where the clerk who uttered in the clerk-specific position utterance information exists with the position where the customer exists in the customer-specific position utterance information. When the distance to the customer is within a predetermined range (for example, 50 cm) set in advance, this is the time during which the state in which the clerk speaks continues.

Since the utterance is intermittently performed during customer service, it is preferable to determine whether or not the utterance is detected in a predetermined unit time in a predetermined unit time in the determination of the utterance state of the store clerk. Further, in the determination of the utterance state of the store clerk, it may be determined whether or not the utterance of at least one of the store clerk and the customer (that is, without distinguishing the utterance subject).

Further, regarding whether or not the distance between the clerk and the customer is within a predetermined range, for example, when the customer exists in the same cell as the cell in which the clerk exists (or in an adjacent cell), the clerk and the customer It may be determined that the distance between and is within a predetermined range.

The approach time between the store clerk and the customer is the time during which the distance between the store clerk and the customer remains within a predetermined range set in advance. The predetermined range can be set to a value different from the range for determining the utterance time of the store clerk described above.

The number of customers existing near the store clerk is the number of customers whose distance from the store clerk is within a predetermined range set in advance. The predetermined range can be set to a value different from the range for determining the utterance time of the store clerk and the range for determining the approach time.

As shown in FIG. 6(D), the position clerk position information by customer and the customer position utterance information include presence/absence of a customer and a response to the position information by clerk and position information by customer shown in FIG. 6(C). Each piece of information on the number of people is added. In the customer-specific position utterance reception information, the number of customers is the number of shop assistants serving customers, and in the clerk position utterance reception information, it is the number of customers being served.

More specifically, in the customer service determination process, as shown in FIG. 10, the position utterance information for each store clerk and the position utterance information for each customer are acquired (ST101), and further information of a preset effective area is acquired ( ST102). Next, it is determined whether or not the position of the clerk is included in the effective area (ST103). Therefore, when the position of the clerk is within the effective area (ST103: Yes), the customer service information generation process described later is performed (ST104). On the other hand, when the position of the store clerk is not within the effective area (ST103: No), the process ends.

In the processing of generating customer service information in FIG. 10, as shown in FIG. 11, the position utterance reception information by clerk and the position utterance reception information by customer are acquired (ST201), and the vicinity of the clerk (for example, in the same cell as the clerk or It is determined whether or not the customer stays at a distance within a predetermined range from the store clerk for a predetermined time or more (ST202). Therefore, when the customer stays near the store clerk for a predetermined time or longer (Yes), it is determined whether or not the position of the store clerk is included in the preset determination change area (ST203). On the other hand, when the customer does not stay near the store clerk for a predetermined time or longer (ST202: No), it is determined that the store clerk is not serving the customer (ST204).

In step ST203, when the position of the store clerk is included in the determination change area (Yes), a dedicated determination standard set for the determination change area is acquired and replaced with the normal determination standard ( That is, the criterion is changed and used (ST205). On the other hand, when the position of the clerk is not included in the judgment change area (ST203: No), the normal judgment standard is used as it is. Judgment criteria include a threshold of speech time (criteria for voice), a threshold of approach time between a clerk and a customer (criteria not related to voice), and a threshold of the number of customers near a clerk (criteria not related to voice). ) Is included. As will be described later in detail, the user can set thresholds such as the determination change area and the speech time during or before the processing of customer service determination.

Next, the customer likelihood regarding the utterance time is calculated based on the utterance time of the clerk and the threshold value (ST206). As the customer likelihood related to the speech time, a certain numerical value is set when the speech time of the store clerk is equal to or more than the threshold value. Further, the value of the likelihood of customer service relating to such utterance time may be increased according to the magnitude of the difference between the clerk's utterance time and its threshold value. On the other hand, when the speech time is less than the threshold value, the customer service likelihood is invalidated (for example, set to zero).

Then, based on the approach time between the clerk and the customer and the threshold value, the likelihood of customer service related to the approach time between the clerk and the customer is calculated (ST207). As the customer likelihood regarding the approach time, a constant numerical value is set when the approach time is equal to or more than the threshold value. Further, the value of the likelihood of customer service related to such approach time may be increased according to the magnitude of the difference between the approach time between the clerk and the customer and the threshold value. On the other hand, when the approach time is less than the threshold value, the customer likelihood is invalidated.

Then, based on the number of customers existing near the clerk and the threshold value thereof, the customer likelihood regarding the number of customers existing near the clerk is calculated (ST208). As the customer reception likelihood regarding the number of customers, a certain numerical value is set when the number of customers existing near the clerk is equal to or more than the threshold value. Further, such a value of the likelihood of customer service regarding the number of customers may be increased according to the magnitude of the difference between the number of customers existing near the clerk and the threshold value. On the other hand, if the number of customers is less than the threshold value, the likelihood of customer service is invalidated.

Note that, in addition to the above-mentioned customer likelihoods (or instead of those customer likelihoods), for example, the customer likelihood regarding the approach distance between the clerk and the customer can be calculated in the same manner. As the customer likelihood regarding the approach distance, a certain numerical value is set when the approach distance between the store clerk and the customer is equal to or less than the threshold value (a reference not related to voice). Further, the value of the likelihood of customer service related to such an approach distance may be increased according to the magnitude of the absolute value of the difference (negative value) between the approach distance and its threshold value.

Further, the attitudes and face orientations of the store clerk and the customer may be determined, and the value of each customer service likelihood described above may be adjusted according to the result.

Then, it is determined whether the total value of the likelihood of customer service calculated in steps ST206 to ST208 is greater than or equal to a preset threshold of the total value (ST209). Therefore, when the total value of the likelihood of customer service is equal to or more than the threshold value (ST209: Yes), it is determined that the customer is serviced by the store clerk (ST210), and based on that, the clerk-specific position utterance service information and the customer-specific position Utterance reception information is generated (ST211). On the other hand, when the total value of the likelihood of customer service is less than the threshold value (ST209: No), it is determined that the customer is not serving the customer by the clerk (ST204), and based on that, the location utterance by customer clerk customer information and the customer Another position utterance reception information is generated (ST211).

As the total value of the likelihood of customer service, for example, the sum of the likelihood of customer service related to the utterance time, the likelihood of customer service related to the approach time, and the likelihood of customer service related to the number of customers existing near the clerk can be used. In that case, weighting may be performed on each of the customer likelihoods according to the priority of each customer likelihood. Further, the total value of the likelihood of customer service may be calculated by omitting a part of the calculated likelihood of customer service (that is, not invalidated).

As shown in FIG. 5, in the first customer service aggregation, based on the customer-based position utterance customer service information, the time when the customer entered the store from the doorway (entry time) and the first clerk to service the customer (first call The time from when the customer enters the store to when the clerk starts the first customer service, based on the time when the customer started the customer service (the customer service start time) and the time when the customer entered the store and the customer service start time. Calculate a customer service start time.

In addition, in the first process of totaling customer service, an average value of service start times is calculated. At this time, the sales staff and stores are aggregated, and the customer service start time is totaled for each shop staff to calculate an average value for each shop staff.The customer service start time is calculated for each shop, and the average value for each store is calculated. calculate.

In the second process of counting the number of customers, the number of times of customer service, the number of customer service times, the customer service point (position where customer service was performed), and the number of customer services are calculated based on the position utterance by customer clerk information acquired in the customer service determination process. get.

Specifically, the number of customer visits for each clerk is acquired based on the presence or absence of customer attendance at each time included in the position utterance customer service information for each clerk.

Further, the start time and end time of customer service are acquired based on the presence or absence of customer service at each time included in the position utterance customer service information for each clerk, and the customer service time is acquired. In addition, when the clerk performs customer service a plurality of times during the aggregation period, the total value of the customer service time of each time is acquired.

Further, the customer service point is acquired based on the location of the shop assistant at each time and the presence or absence of customer service, which is included in the position utterance customer service information for each shop employee. In this case, the position of the clerk is the customer service point. It should be noted that the customer service points may be acquired in consideration of the customer position utterance customer service information. That is, when there is only one customer, the customer service point may be an intermediate point between the clerk and the customer based on the customer position included in the customer position utterance customer service information.

In the process of totaling visits to the store, the number of people visiting the store is acquired based on the position information for each person acquired from the image processing PC 6. In the present embodiment, a measurement line is set in the vicinity of the entrance/exit of a store in a captured image, and the flow line of the person based on the position information of the person intersects the measurement line in the entering direction, so that the person comes to the store. The number of people who visit the store is detected and the number of people who visit the store is acquired.

A measurement area may be set instead of the measurement line, and the visit of a person may be detected based on the intersection of the flow line with respect to this measurement area.

In addition, in the process of totaling the number of customers who visit the store, the number of groups for each number of people who visit the store, that is, how many groups of customers visit the store, is acquired. At this time, the entry time for each person is acquired, and if the entry interval between the persons is within a predetermined time (for example, 5 seconds), it is determined that the group has come to the store. Further, the group representative may be specified based on the time when the ticket issuing machine receives the number tag, and the group configuration may be grasped.

In the customer totaling process, the staying time and the number of people staying are acquired based on the customer-specific position information acquired in the process of specifying the clerk. At this time, the time spent by each customer in each cell (divided area) is totaled to obtain the stay time for each cell. In addition, the number of stays in each cell is acquired by totaling the period in which each customer stays in each cell.

In the salesclerk totaling process, the amount of exercise, the number of breaks and the break time of each salesclerk, and the number of actual salesclerks are acquired based on the terminal position information acquired from the positioning management server 7. At this time, the amount of movement of each store clerk is calculated based on the position information of the mobile terminal 4 of each store clerk, and the amount of exercise of each store clerk is acquired. Further, based on the position information of the mobile terminal 4 of each store clerk, the rest period in which each store clerk stays in the backyard or outside the store is acquired, and the number of breaks and the rest time of each store clerk are acquired. In addition, the actual work period excluding the break period from the working period of each store clerk is totaled to obtain the number of actual store clerk per unit time.

In the screen generation process, screen information of the customer service heat map display screen is generated as information on the customer service situation. As shown in FIG. 12, the customer service heat map display screen is provided with a condition setting unit 61, a menu display button 62, a heat map display unit 63, a video display unit 64, and a reproduction operation unit 65. ..

The user can set search conditions in the condition setting unit 61, that is, filtering conditions for narrowing down the display targets of the heat map display unit 63, and only information corresponding to the set search conditions is set in the heat map display unit 63. Is displayed. The search conditions include items such as customer service heat map display, a clerk, a customer, an area, a period, a range of customer service time, and the like. When the user performs an operation of designating various search conditions in the condition setting unit 61 and operates the “search” button 77, the display target is filtered based on the specified search condition and the specified search condition is set. The corresponding information is displayed on the heat map display portion 63. When the user operates the menu display button 62, a menu for selecting a customer service heat map, a graph, settings, and the like is displayed.

The video image captured by the camera 2 is displayed on the video display unit 64. In this embodiment, since the camera 2 is an omnidirectional camera using a fisheye lens, a fisheye image is displayed on the image display unit 64. With the captured video displayed on the video display unit 64, the actual situation in the store at the timing of interest is confirmed while the heat map image 81 and the flow line 84 are being reproduced on the heat map display unit 63. be able to. The video display unit 64 may be hidden or may be enlarged and displayed in another window according to the preference of the user. Alternatively, a panoramic image obtained by correcting the fish-eye image or a divided image divided into four images may be displayed.

In the screen generation process, the heat map of the customer service at each time is based on the statistical information (service time, the number of customer service and the number of customer services) acquired in the second customer service aggregation process for the clerk selected by the user as the evaluation target. An image 81 is generated and superposed on the shop map image 82. Further, an image of the flow line 84 of the clerk and the customer at each time is generated based on the position information of the clerk and the customer included in the position utterance and customer service information for each clerk and the position utterance and customer service information for each customer. The image and the person mark 85 are superimposed on the store map image 82. In addition, the first-touch mark 87 is superimposed on the store map image 82 based on the first-touch time (customer service start time) and position. Here, the size of the heat map image 81 represents the number of times of customer service, and the darkness of the fill color of the heat map image 81 represents the customer service time.

In addition, in the screen generation process, various list tables are provided as information regarding the customer service status based on the statistical information acquired in each of the first customer service total, second customer service total, visit total, store employee total, and customer total process. And various graphs (for example, a list or a graph relating to the statistical information acquired in each process of the first customer count, the second customer count, the store visit count, the salesclerk count, and the customer count) are generated, and this list or It is also possible to generate screen information for the statistical information screen on which the graph is displayed.

On the heat map display unit 63, a flow line 84, which depicts the trajectory of the person, is displayed on the store map image 82 as an image showing the behavior of the person (clerk and customer). A person mark 85 is displayed at the end of the flow line 84. The person mark 85 indicates the position of the person at the reproduction time. The flow line 84 is displayed in a different color for each person. Further, the flow line 84 may be colored for each person so that the store clerk and the customer can be easily identified. For example, the flow line 84 of the clerk may be displayed in a cold color, and the flow line 84 of the customer may be displayed in a warm color.

The reproduction operation unit 65 is provided with a reproduction stop button 91. By operating the reproduction stop button 91, reproduction of a moving image (heat map image 81, flow line 84, etc.) of the heat map display unit 63 and You can make a stop. When the reproduction is started with the reproduction stop button 91, the moving image changes according to the progress of the reproduction time. That is, the person mark 85 appears at the position where the clerk or the customer exists, and the flow line 84 of the clerk or the customer gradually extends as the clerk or the customer moves. Then, when the clerk starts serving customers, the heat map image 81 appears, and if it is the first touch, the mark 87 appears. Further, the form (size, color, etc.) of the heat map image 81 gradually changes as the customer service time and the number of customer services increase.

Further, the reproduction operation section 65 is provided with a slider 93 that moves along the bar 92, and by performing an operation of shifting the slider 93, the reproduction time can be adjusted. Also, the start time and end time of the aggregation period are displayed at both ends of the bar 92, and the reproduction time is displayed near the slider 93.

Alternatively, the heat map image 81 may be used to represent the statistical information in the aggregation period that is traced back to the past from the reproduction time, using a period shorter than the search period designated by the user as the aggregation period of the statistical information. In this case, the aggregation period moves (so that it is updated) as the reproduction time advances.

Next, a method of setting the user regarding the above-mentioned effective area for customer service determination will be described. FIG. 13 is an explanatory diagram showing a procedure for setting an effective area for customer service determination. In the setting screen of the effective area shown in FIG. 13, the same components as those of the above-mentioned customer service heat map display screen are designated by the same reference numerals.

As shown in FIG. 13, the condition setting section 61 of the setting screen includes a display setting section 101, an area adding section 102, an area designating section 103, and a period setting section 75.

In the display setting unit 101, the user can select an item to be displayed on the heat map display unit 63 when setting the effective area. In this embodiment, each item of the voice heat map and the flow line can be selected. When the sound heat map item is selected, a sound heat map image 111 (sound visualization image) is displayed as an image showing the sound generation position and the sound level, and when the flow line item is selected, the flow line 84 and the person mark are displayed. 85 is displayed. The sound heat map image 81 is superimposed on the store map image 82 in a transparent state, so that the store map image 82 can be seen through even at the position where the heat map image 81 is superimposed. Further, when the user operates the reproduction stop button 91 to advance the time, the display form of the heat map image 81 of the voice changes according to the change with time of the statistical information of the voice data.

The area adding unit 102 is for setting an area to be newly added as an effective area. For example, as shown in FIG. 13, the user selects the desired area (see the rectangle 117 indicated by the broken line) on the store map displayed on the heat map display unit 63, and then selects the area addition unit 102. A new effective area can be added by inputting "area B" as the area name to be added and pressing the Add button 112. New effective areas can be added. “Area B” for which the additional operation has been completed is indicated by a solid line like the area A that has already been set. Note that the effective area is not limited to a rectangle, and may be set in another shape such as a circle.

When the user adds a new area with the area adding unit 102, the user operates the display setting unit 101 to convert the sound heat map image 81 (standard image) as an image representing the sound generation in the store into the heat map. It can be displayed on the store map image 82 of the display unit 63.

As a result, the user can easily grasp the generation position of the environmental sound generated in the store regardless of the utterance of the store clerk, and can accurately select the desired area to be set as the effective area. Become. For example, the user may, as the effective area, exclude an area (for example, an area where an environmental sound is generated) unsuitable for performing customer service determination in the store, or conversely, perform the customer service determination in the store. An area suitable for implementation (for example, an area less susceptible to the influence of environmental sound) can be set. In this case, the user can understand not only the actual sound generation position by the heat map image 81 of the sound but also the sound generated at a position different from the actual sound generation position due to the reflection of the wall or the like. .. Further, the user can display the flow line 84 and the person mark 85 on the store map image 82 in addition to the heat map image 81, if necessary. As a result, the user can confirm the movement of the person and more accurately grasp the position where the environmental sound is generated.

Here, the display form (for example, hue) of the heat map image 81 gradually changes according to the sound level. In this case, a plurality of thresholds can be set so that the hue of the heat map image 81 changes stepwise, for example, the heat map image 81 changes stepwise from blue to green, yellow, and red. Further, the sound level may be expressed by the size of the heat map image 81. Further, although the shape of the heat map image 81 is circular here, the shape of the heat map image 81 is not limited to a circle, and may be another shape such as a rectangle.

The area designation unit 103 is for the user to designate an effective area applied to the customer service determination. In FIG. 13, the area A is designated (checked) as the application target. After the area B is added by the operation of the area adding unit 102, the area B is also displayed on the area designating unit 103.

Further, on the video display unit 64, an image displaying the effective area set by the user is displayed by being superimposed on the corresponding position in the captured video (here, fish-eye video) at the same time. In FIG. 13, the image displaying the effective area is shown as a rectangular frame 120 for convenience, but in the actual image display unit 64, the frame 120 has a distorted shape corresponding to the fisheye image. Have With this captured image, it is possible to confirm the actual situation in the store and the position of the effective area at the timing of interest while the heat map image 81 and the flow line 84 are being reproduced on the heat map display portion 63. .. When the reproduction is started with the reproduction stop button 91, the captured image (moving image) changes according to the progress of the reproduction time.

The period setting unit 75 is for the user to specify the period (date, start time, end time) to be displayed on the heat map display unit 63. In the period setting unit 75, when the user designates a period and presses the View button 115, information (voice heat map, flow line information, etc.) corresponding to the period designated by the user is displayed on the heat map display unit 63. It

Next, a method of setting the above-mentioned determination criteria (dedicated determination criteria) for customer service determination by the user will be described. 14, FIG. 15, and FIG. 16 are explanatory diagrams showing the procedure for setting the determination standard for customer service determination. 14 to 16, the same reference numerals are attached to the same components as those of the customer service heat map display screen or the effective area setting screen described above. Further, regarding the determination change area described below, matters not particularly mentioned can be the same as those in the case of the effective area described above.

As shown in FIGS. 14 to 16, the condition setting unit 61 of the determination criterion setting screen includes a display setting unit 101, an area adding unit 102, an area designating unit 103, a determination setting unit 104, and a period setting unit. 75 are provided.

In the display setting unit 101, the user can select an item to be displayed on the heat map display unit 63 when setting the determination standard, similarly to the setting of the effective area (see FIG. 13) described above.

The area adding unit 102 allows the user to set an area to be newly added as the determination change area. For example, as shown in FIG. 14, the user selects the desired area (see the rectangle 107 indicated by the broken line) on the store map displayed on the heat map display unit 63, and then selects the area addition unit 102. By inputting “Area C” as the area name to be added and pressing the Add button 112, a new judgment change area can be added. “Area C” where the additional operation is completed is indicated by a solid line (see FIGS. 15 and 16).

When the user adds a new area with the area adding unit 102, the user operates the display setting unit 101 to convert the sound heat map image 81 (standard image) as an image representing the sound generation in the store into the heat map. It can be displayed on the store map image 82 of the display unit 63.

The area designation unit 103 is for the user to designate the determination change area to be the target for setting the determination standard. In FIG. 14, area A and area B are designated (checked) as targets for setting the determination standard. Note that after the area C is added by the operation of the area adding unit 102, the area C is also displayed on the area designating unit 103 (see FIGS. 15 and 16).

Further, on the video display unit 64, similarly to the setting of the effective area described above, the image (see the rectangular frame 120) displaying the determination change area set by the user is displayed at the corresponding position in the captured video at the same time. They are displayed in a superimposed manner.

The determination setting unit 104 is for the user to set a dedicated determination criterion for each determination change area designated by the area designation unit 103. In the present embodiment, the user can set, as the determination criteria, a threshold value regarding the utterance time of the store clerk, a threshold value regarding the approach time between the store clerk and the customer, and a threshold value regarding the number of customers existing near the store clerk. In addition, the determination standard set in the determination change area does not need to reset all the thresholds, and may change only a part of the thresholds set by the normal determination standard.

FIG. 14 shows a state in which the threshold value relating to the speech time is selected as the setting target in the determination setting unit 104. Further, the determination setting unit 104 is provided with a slider 106 that moves along the bar 105, and the value of the threshold value can be adjusted by performing an operation of shifting the slider 106. Similarly, FIG. 15 illustrates a state in which the threshold value relating to the approach time between the store clerk and the customer is selected as the setting target in the determination setting unit 104, and FIG. 16 relates to the number of customers existing near the store clerk. The state where the threshold is selected is shown.

As shown in FIG. 14, for example, by setting the threshold of the utterance time to the maximum (MAX), the customer likelihood regarding the utterance time of the target judgment change area (here, areas A and B) is invalidated. You can That is, in the processing for customer service determination shown in FIG. 11 described above, when the clerk is located in the area A or the area B as the determination change area (ST203: Yes), the maximum is set as the determination reference (threshold value) of the utterance time. (ST205), and as a result, the customer likelihood regarding the speech time is invalidated (ST206).

Note that, for example, when setting the threshold of the utterance time, the user can consider the arrangement of the speakers actually arranged in the store, the position of the entrance and exit, the heat map image 81 described above, and the like. Further, the user can return the threshold value to be set to the preset initial value by pressing the Default button 114 in the determination setting unit 104.

It should be noted that the user is preset according to the characteristics of the area, for example, together with the determination change area and the determination criteria dedicated to the determination change area set as described above, or instead of the determination change area and the determination criteria. It is also possible to use a dedicated criterion.

FIG. 17 is a flow chart showing a procedure of presetting a determination standard for customer service determination. FIG. 18 is an explanatory diagram showing a procedure for presetting a determination standard for customer service determination. In addition, in the preset screen of the determination standard of the customer service determination illustrated in FIG. 18, the same reference numerals are given to the same components as those of the determination standard setting screen described above.

As shown in FIG. 17, in the preset process of determination criteria, the user sets a preset name (ST301), and with respect to the preset name, a threshold (ST302) relating to the clerk's utterance time and the approaching time between the clerk and the customer. The threshold value (ST303) and the threshold value (ST304) of the number of customers existing near the clerk are sequentially set. Thereafter, with respect to those settings, preset registration is performed (ST305), and the processing ends.

In FIG. 18, the area adding unit 102 is for setting an area to be newly added as a preset determination change area. For example, the user selects the desired area (see the rectangle 207 indicated by the broken line) on the store map displayed on the heat map display unit 63, and selects “entrance” as the area name to be added to the area addition unit 102. By inputting “A” and pressing the Add button 112, a newly preset judgment change area can be added.

The existing preset display unit 123 displays the preset name that has already been set. Here, an example is shown in which preset judgment criteria are set for the counters A-C in the store. Further, the user can select (check) an existing preset name to set a desired threshold value in the determination setting unit 104 in the area corresponding to the preset name in the same manner as described above. is there.

In the determination setting unit 104, when the user selects preset settings and further selects (checks) existing preset names, the dedicated determination criteria set for them are valid. Here, preset settings (determination criteria) for the counters A and B are valid.

The determination criterion set in association with the preset name is stored in the storage unit 53 (memory) of the analysis PC 8 and, when the preset is valid, is applied to the customer service determination process without the need for the user to set each time. If a judgment change area is set in the judgment reference setting screen for customer service judgment shown in FIGS. 14 to 16 for an area that overlaps with the area to which the preset judgment reference is applied, the setting screen The judgment standard set in step 1 is given priority.

(Second embodiment)
FIG. 19 is a flowchart showing a procedure of customer service determination processing of the customer service situation analysis system according to the second embodiment. Regarding the customer service situation analysis system according to the second embodiment, matters not particularly mentioned below are the same as in the case of the above-described first embodiment, and description thereof will be omitted.

In the customer service determination processing in FIG. 19, steps ST301 to 305, which are the same as steps ST201 to ST205 in the customer service information generation processing (see FIG. 11) in the customer service determination processing of the first embodiment described above, are executed. However, the customer service determination according to the second embodiment differs from the customer service determination processing of the first embodiment in that it determines whether or not the position of the clerk is included in a preset effective area (ST306). ..

When it is determined in step ST306 that the clerk is located in the effective area (Yes), steps ST307-ST312 similar to steps ST206-ST2011 of the above-described customer service information generation processing (see FIG. 11) are executed. It On the other hand, when it is determined that the store clerk is not located in the effective area (ST306: No), the calculation of the customer likelihood regarding the utterance time (ST307) is omitted, and the customer likelihood regarding the approach time between the clerk and the customer (ST307) is omitted. ST308) and the customer likelihood (ST309) regarding the number of customers existing near the store clerk are calculated.

It should be noted that instead of omitting the calculation of the likelihood of customer service regarding speech time (ST307), as described above (see FIG. 14), the threshold of speech time is set to the maximum (MAX), and the target determination is changed. It is also possible to invalidate the likelihood of customer service regarding the speech time of the area.

As a result, in the customer service determination according to the second embodiment, when the position of the clerk is not included in the effective area, the determination is performed using only the reference irrelevant to the voice.

The embodiments have been described above as examples of the technology disclosed in the present application. However, the technology of the present disclosure is not limited to this, and can be applied to embodiments in which changes, replacements, additions, omissions, etc. have been made. Further, it is also possible to combine the constituent elements described in the above-described embodiment to form a new embodiment.

For example, in the above-described embodiment, the voice processing PC 5, the image processing PC 6, the positioning management server 7, and the analysis PC 8 are provided, but the functions of these devices are not limited to a single function. It is not necessary to realize it with the above information processing device, and it can be appropriately integrated into one or a plurality of information processing devices. It is also possible that the functions of the analysis PC are shared by a plurality of information processing devices. Further, the monitoring camera in which the microphone 1 and the camera 2 are integrated has been made multi-functional, and the processor of the monitoring camera generates the position information for each person and the sound heat map without using the sound processing PC 5 or the image processing PC 6, It is also possible to transmit it to the analysis PC 8.

In recent years, in the IoT (Internet of Things) world, CPS (Cyber Physical Systems), which is a new concept of creating new added value by information collaboration between physical space and cyber space, has been attracting attention. Under these circumstances, the CPS concept can also be adopted in this embodiment. That is, as a basic configuration of CPS, for example, an edge server arranged in a physical space and a cloud server arranged in a cyber space are connected via a network, and the edge server and the processor mounted in the cloud server serve customers. It is possible to disperse the analytical processing. In this customer service analysis process, a server including the voice processing PC 5, the image processing PC 6, and the positioning management server 7 is constructed as an edge server. Next, the cloud server executes processing for generating customer service data based on the processing data (voice heat map information, person position information, terminal position information) received from the edge server via the network, and receives the customer service data. It is taken over by the analysis application software and displayed on the display, etc. in the output format defined by the application. Here, it is preferable that each data generated in the edge server or the cloud server is generated by a Web application or the like installed on a standardized platform. By using such a standardized platform, various sensor groups can be collected. It is possible to improve efficiency when constructing a system including IoT application software.

Further, the configuration of the store to which the customer service situation analysis device, the customer service situation analysis system, and the customer service situation analysis method of the present disclosure are not limited to the above-described example, and the commercial activities performed in the store are also sold by the store. It is possible to adopt various forms by financial institutions, restaurants, and the like.

The customer service situation analysis device, the customer service situation analysis system, and the customer service situation analysis method according to the present disclosure accurately determine that a clerk has performed customer service even when there is an area unsuitable for conducting customer service determination in the store. However, it has the effect of being able to generate highly reliable customer service information, analyzes the customer service situation of the salesclerk to the customer at the store, and executes the processing of generating information on the customer service situation by the processor. It is useful as a customer service situation analysis system and a customer service situation analysis method.

1 microphone (sound pickup device)
2 cameras (imaging device)
3 Beacon transmitter 4 Mobile terminal (mobile terminal device)
5 Voice processing PC (information processing device)
6 Image processing PC (information processing device)
7 Positioning management server 8 Analysis PC (customer service status analysis device, information processing device)
9 Viewing Terminal 11 Gateway 12 Public Access Point 52 Processor 63 Heat Map Display 81 Heat Map Image (Standard Image)
82 store map image 84 flow line 85 person mark 91 playback stop button 101 display setting unit 102 area adding unit 103 area designating unit 104 judgment setting unit 111 heat map image 123 existing preset display unit

Claims (13)

A customer service situation analysis device that analyzes a customer service situation of a clerk to a customer in a store, and executes a process of generating information on a customer service situation by a processor,
The processor is
Position information of a person obtained from image data of a person in a store, position information of a clerk's belongings, and position information of a voice acquired from voice data of a voice in the store are acquired,
Based on the position information of the person, the position information of the belongings, and the position information of the voice, it is determined whether or not the clerk has served the customer using a predetermined determination criterion,
Based on the result of the determination, to generate information about the customer service situation,
When making the determination, it is determined whether the position of the clerk is included in the effective area set as a part of the monitoring area in the store, and the position of the clerk is included in the effective area. A customer service situation analysis device, characterized in that the above determination is performed. The processor is
The determination is performed using a criterion relating to the voice and a criterion not relating to the voice as the criterion,
The customer service situation analyzing apparatus according to claim 1, wherein when the position of the clerk is not included in the effective area, the determination is performed using only a reference that is not related to the voice. The customer service situation analyzing apparatus according to claim 2, wherein the criterion regarding the voice includes a threshold value of the utterance time of the clerk. The processor is
The threshold of the utterance time is set to a maximum value when the position of the clerk is not included in the effective area, and the determination is performed using the set threshold. Customer service situation analyzer. The processor is
Using a threshold of the approach time between the clerk and the customer as a reference not related to the voice,
5. When the position of the clerk is not included in the effective area, the threshold value of the approach time is changed and the changed threshold value is used to make the determination. The customer service situation analyzer described in Crab. The processor is
Using a threshold of the number of customers existing near the clerk as a reference not related to the voice,
If the position of the clerk is not included in the effective area, the threshold value of the number of the customers is changed, and the changed threshold value is used to make the determination. Customer service situation analyzer described in. The processor is
Using a threshold of the approach distance between the clerk and the customer as a reference not related to the voice,
The threshold of the approach distance is changed when the position of the clerk is not included in the effective area, and the determination is performed using the changed threshold. Customer service situation analyzer described. Further comprising a memory for storing the information of the determination criteria,
The customer service situation according to claim 1, wherein the processor reads information on the determination criteria stored in the memory when the position of the clerk is included in the effective area when performing the determination. Analysis equipment. The processor is
9. The customer service status according to claim 1, wherein the position information of the clerk and the position information of the customer are acquired based on the position information of the person and the position information of the belongings. Analysis equipment. The processor is
10. The customer service situation analyzing apparatus according to claim 9, wherein utterance position information indicating a position uttered by the clerk is acquired based on the position information of the voice and the position information of the clerk. The processor is
11. The customer service situation analyzing apparatus according to claim 10, wherein it is determined whether or not the clerk has performed customer service based on the utterance location information and the location information of the customer. A customer service situation analysis system that analyzes the customer service situation of a clerk to a customer in a store and generates information on the customer service situation,
A sound pickup device that picks up the sound in the store,
An imaging device for imaging a person in a store,
A mobile terminal device carried by a clerk,
One or more information processing devices,
Equipped with
The one or more information processing devices,
Position information of the person acquired from the image data output from the imaging device, position information of the mobile terminal device, and position information of the sound output from the sound collecting device,
Based on the position information of the person, the position information of the mobile terminal device, and the position information of the voice, it is determined whether or not the store clerk has served by using a predetermined determination criterion,
Based on the result of the determination, to generate information about the customer service situation,
Based on the result of the determination, to generate information about the customer service situation,
When making the determination, it is determined whether the position of the clerk is included in the effective area set as a part of the monitoring area in the store, and the position of the clerk is included in the effective area. A customer service situation analysis system characterized by making the above determination. A customer service situation analysis method for analyzing a customer service situation of a clerk to a customer in a store, and executing a process of generating information on a customer service situation by a processor,
Position information of a person obtained from image data of a person in a store, position information of a clerk's belongings, and position information of a voice acquired from voice data of a voice in the store are acquired,
Based on the position information of the person, the position information of the belongings, and the position information of the voice, it is determined whether or not the clerk has served the customer using a predetermined determination criterion,
Based on the result of the determination, to generate information about the customer service situation,
When making the determination, it is determined whether the position of the clerk is included in the effective area set as a part of the monitoring area in the store, and the position of the clerk is included in the effective area. A customer service situation analysis method, characterized in that the above determination is performed.

Images