JP6876379B2 - Behavioral analyzers and programs - Google Patents

Description

Embodiments of the present invention relate to behavioral analytics devices and programs.

In order to prevent or solve human relationship troubles in facilities, it is considered effective to detect specific events or signs in human behavior at an early stage.
In the prior art, there is an attempt to provide a system that can easily make a report by a communication device in order to prevent domestic violence, abuse, and bullying. The system determines the degree of risk and abnormality by analyzing the situation based on voice. In addition, when it is difficult for infants and the elderly to write characters, a technique has been proposed in which voice recognition is performed and the characters are converted into characters. However, there was a possibility that the situation could not be grasped only by voice analysis.
Further, in the prior art, there is a technique of installing a friendship status program on a personal terminal device such as a smartphone and detecting a change in the friendship status based on an increase or decrease in the number of friends on an SNS site or the like. However, the information on the increase / decrease in the number of friends on the SNS site may not be sufficient to represent even the detailed daily behavior of a person.

Japanese Unexamined Patent Publication No. 2003-346256 JP-A-2015-225540

An object to be solved by the present invention is to provide a behavioral analyzer and a program capable of comprehensively grasping information on events and signs related to problems such as bullying.

The behavior analysis device of the embodiment includes a video acquisition unit, a video analysis unit, a voice acquisition unit, a voice analysis unit, a comprehensive analysis unit, and an information output unit. The video acquisition unit acquires the video. The image analysis unit analyzes a person appearing in the image acquired by the image acquisition unit and detects an image pattern. The voice acquisition unit acquires voice. The voice analysis unit analyzes a human voice included in the voice acquired by the voice acquisition unit and detects a voice pattern. The comprehensive analysis unit detects a specific event related to human behavior based on the video pattern detected by the video analysis unit and the audio pattern detected by the audio analysis unit. The information output unit outputs information about the event detected by the comprehensive analysis unit.

The figure which shows the system configuration example which applied the behavior analysis apparatus of 1st Embodiment. The block diagram which shows the schematic functional structure of the behavior analysis apparatus of 1st Embodiment. The schematic diagram which shows the example of the result which the image analysis part of 1st Embodiment analyzed. The schematic diagram which shows the example of the result which the image analysis part of 1st Embodiment analyzed. The schematic diagram which shows the example of the result which the image analysis part of 1st Embodiment analyzed. The block diagram which shows the internal functional structure of the comprehensive analysis part of 1st Embodiment. The schematic diagram which shows the example of the rule which the symptom detection rule storage part of 1st Embodiment stores. The schematic diagram which shows the structure of the authority setting table for setting the authority of the user in the behavior analysis apparatus of 1st Embodiment. The schematic diagram which shows an example of the information output by the information output part of 1st Embodiment.

Hereinafter, the behavioral analyzer and the program of the embodiment will be described with reference to the drawings.

FIG. 1 is a diagram showing a system configuration example to which the behavior analysis device of the present embodiment is applied, and shows an example of using the behavior analysis device of the present embodiment in a school facility. In the illustrated example, the behavioral analyzer is installed in a school facility. The behavior analysis device can be installed not only in schools but also in kindergartens or nursery schools, nursing care facilities, hospitals, public institutions such as government offices, stores, companies, and other facilities used by groups. .. The behavior analysis device analyzes using the video acquired by the camera 82 provided in the classroom in the school and the voice acquired by the microphone 83. As a means of communication within the facility (inside the school), for example, the school intranet is used. Then, the behavior analysis device estimates the behavior evaluation as a result of the behavior analysis. In the illustrated example, the behavioral analyzer estimates the degree of bullying occurrence. Specifically, for example, a behavioral analyzer estimates that the degree of bullying in one classroom is "low" and that the degree of bullying in another classroom is "high". In addition, the behavior analysis device outputs information on the analysis result. Specifically, the behavior analysis device sends an e-mail containing the analysis result to the teacher, the guardian (parent) of the student, or the like via the Internet, for example. In addition, the behavior analysis device exchanges information with the external device 81 via the Internet. Here, the external device 81 is, for example, a behavior analysis device installed in another school, a server device for a website called an "unofficial school site", or the like. The behavior analysis device 1 exchanges information with the external device 81 by communication via the Internet.

FIG. 2 is a block diagram showing a schematic functional configuration of the behavior analysis device of the present embodiment. As shown in the figure, the behavior analysis device 1 includes a video acquisition unit 11, a video storage unit 12, a video analysis unit 13, a voice acquisition unit 21, a voice storage unit 22, a voice analysis unit 23, and personal feature storage. It is composed of a unit 31, a comprehensive analysis unit 32, an information exchange unit 41, and an information output unit 42. In addition, the behavior analysis device 1 acquires images from a plurality of external cameras 82. In addition, the behavior analysis device 1 acquires voice from a plurality of external microphones 83. Further, the behavior analysis device 1 exchanges information with the external device 81 via a communication network or the like.
Although only two cameras 82 are shown in the figure, the number of cameras to be installed is arbitrary and may be one or three or more. Further, although only two microphones 83 are shown in the figure, the number of microphones to be installed is arbitrary and may be one or three or more. The camera 82 and the microphone 83 are installed in pairs at the same location. Further, although the figure shows only one external device 81, there may be a plurality of external devices with which the behavior analysis device 1 exchanges information.

The behavior analysis device 1 is realized as an information processing device using an electronic circuit. Also, the behavior analysis device 1. It may be realized by using a computer and a program. Further, when each functional unit included in the behavior analysis device 1 has a function of storing information, a storage means using a semiconductor memory, a magnetic disk device, or the like is used.

As the present embodiment, a case where the behavior analysis device 1 is installed in a facility such as a school as shown in FIG. 1 will be described as an example. The camera 82 and the microphone 83 are provided as a pair at a predetermined location in the facility. For example, a pair of cameras 82 and a microphone 83 are provided in classrooms, gymnasiums, dining rooms, and other spaces within the facility. The camera 82 acquires an image of the installed space. The microphone 83 acquires voice in the installed space. Then, the behavior analysis device 1 attempts to prevent the problem of bullying by detecting problematic behavior such as bullying and its signs and reporting the detection result.

Hereinafter, the functions of each part of the behavior analysis device 1 will be described.

The image acquisition unit 11 captures signals of images taken by a plurality of cameras 82 provided externally, and writes them in the image storage unit 12 together with information on a shooting location and a shooting time.
The video storage unit 12 stores video information acquired by the video acquisition unit 11. In order for the image storage unit 12 to hold the image, for example, a recording medium such as a magnetic disk device is used. The video held by the video storage unit 12 is stored with information on the shooting location and shooting time added. In addition, information on the analysis result, which is the result of the image analysis process performed by the image analysis unit 13, is added to the image and stored.

The image analysis unit 13 performs a process of analyzing the image stored in the image storage unit 12. The image analysis unit 13 passes the information of the analysis result to the comprehensive analysis unit 32, and also records it in the image storage unit 12 in the form of adding it to the image to be analyzed. In particular, the image analysis unit 13 analyzes a person in the image. That is, the image analysis unit 13 analyzes the person appearing in the image acquired by the image acquisition unit 11 and detects the image pattern. There are various analysis processes performed by the image analysis unit 13, and for example, they are as follows.
The image analysis unit 13 recognizes and extracts what is reflected in the image, a person, and the like.
In addition, the image analysis unit 13 performs individual identification processing based on the characteristics on the image of each individual stored in the personal feature storage unit 31. For personal identification, information on facial features and clothing characteristics is used. The personal identification process itself can be performed with sufficient accuracy using existing technology. That is, the image analysis unit 13 performs the personal identification process of the person appearing in the image based on the individual characteristics related to the image read from the personal feature storage unit 31.
In addition, the image analysis unit 13 extracts the relative positional relationship of a plurality of people shown in the image, and extracts a specific pattern regarding the arrangement of the people. The pattern of arrangement of people is, for example, a pattern in which a person is close to or far from another person, a pattern in which multiple people surround another person, or a group of people (a group of people). It is a pattern such as whether it belongs to a cluster) or not.
In addition, the video analysis unit 13 analyzes the behavior of the person shown in the video. A person's behavior is, for example, whether the person is stationary or moving, whether the person is moving straight or curvilinearly, and whether the person is moving fast or slow. .. In addition, the image analysis unit 13 analyzes the trajectory of the movement of a person's hand, foot, or head. In addition, the video analysis unit 13 analyzes the action type from the combination of those movements and the movements of the hands, feet, head, and the like. Here, the action type analyzed by the combination is, for example, an action such as a person hitting another person, kicking, or making another attack.

The voice acquisition unit 21 takes in voice signals acquired by a plurality of external microphones 83 and writes them in the voice storage unit 22 together with information on the sound collection location and time. The sound collection location information includes information on the location where the microphone 83 is installed, such as a classroom number. The microphone 83 may acquire multi-channel (two-channel or more) audio.
The voice storage unit 22 stores the voice acquired by the voice acquisition unit 21. In order for the sound storage unit 22 to record sound, for example, a recording medium such as a magnetic disk device is used. The voice held by the voice storage unit 22 is stored with information on the sound collection location and time added. In addition, information on the analysis result, which is the result of the voice analysis process performed by the voice analysis unit 23, is added to the voice and stored.

The voice analysis unit 23 performs a process of analyzing the voice stored in the voice storage unit 22. The audio analysis unit 23 passes the analysis result information to the comprehensive analysis unit 32, and also records the analysis result information in the audio storage unit 22 in the form of adding audio to the video to be analyzed. In particular, the voice analysis unit 23 analyzes a human voice. That is, the voice analysis unit 23 analyzes the human voice included in the voice acquired by the voice acquisition unit 21 and detects the voice pattern. There are various analysis processes performed by the voice analysis unit 23, and for example, they are as follows.
The voice analysis unit 23 recognizes and extracts a portion of a human utterance. The process itself of extracting a person's utterance from the recorded voice can be performed using existing technology.
Further, when the microphone 83 used for sound collection acquires multi-channel (2 channels or more) voice, the voice analysis unit 23 determines the place or direction in which the extracted person's utterance is made. To identify. The location and direction are specified by analyzing the distribution of the volume of the utterance in each channel and the delay of the utterance between the channels.
In addition, the voice analysis unit 23 analyzes the volume of a person's utterance. Then, when the volume of a certain utterance exceeds a predetermined threshold value, a mark indicating that the utterance is particularly loud is added to the utterance portion.
In addition, the voice analysis unit 23 performs a process of identifying the speaker based on characteristics such as frequency distribution of the extracted utterance. At this time, the voice analysis unit 23 reads out the information representing the individual characteristics of the voice from the personal feature storage unit 31 and uses it. The speaker identification process itself by voice should be performed using existing technology. Even when it is difficult for the voice identification process to completely identify a large number of people, it is possible to output likelihood information indicating who is the speaker of a certain utterance. That is, the voice analysis unit 23 performs personal identification processing of the human voice included in the voice based on the personal characteristics related to the voice read from the personal feature storage unit 31.
In addition, the voice analysis unit 23 performs voice recognition processing on the extracted utterance portion using the data of the recognition voice model for an unspecified speaker. The voice recognition process itself can be performed by existing technology. The voice analysis unit 23 outputs word string data for each utterance as a result of the voice recognition process. By performing such a voice recognition process, for example, it becomes possible to detect a situation in which a specific keyword is spoken.
In addition, the voice analysis unit 23 estimates emotions based on voice. In this analysis, a model of the acoustic characteristics of each emotion expressed, such as a happy laughing voice, a mocking voice, a crying voice, and an angry voice, is stored in advance, and the actually acquired voice is obtained for each emotion. It can be done by analyzing based on the model. Emotion estimation itself can be done using existing techniques. As a result of estimating the emotion, the voice analysis unit 23 outputs information on the likelihood of the emotion for each utterance (for example, "the likelihood of crying is 90%").

The personal feature storage unit 31 stores personal features that can be used in video analysis processing and audio analysis processing. Information on personal characteristics can be broadly divided into visual information and acoustic information. The personal feature information is used for personal identification processing by the video analysis unit 13 and personal identification processing by the audio analysis unit 23. Individual visual features include facial features, physical features, and clothing features. An individual's acoustic characteristics include characteristics such as frequency distribution in speech.

The comprehensive analysis unit 32 performs comprehensive analysis on human behavior based on the image analysis result by the image analysis unit 13 and the audio analysis result by the audio analysis unit 23, and outputs the result. That is, the comprehensive analysis unit 32 detects a specific event related to human behavior based on the image pattern detected by the image analysis unit 13 and the audio pattern detected by the audio analysis unit 23. The details of the function of the comprehensive analysis unit 32 will be described later with reference to another figure.

The information exchange unit 41 exchanges information with the external device 81 via a communication line such as the Internet. The external device 81 is, for example, a behavior analysis device installed in another facility. Alternatively, the external device 81 is a computer server device that provides parameters and the like used in the analysis processing performed by the behavior analysis device 1. Alternatively, the external device 81 is a computer server device that collects information on the results analyzed by the behavior analysis device 1 (information on events and signs related to human behavior, etc.).
The information exchange unit 41 acquires data such as parameters from the outside, and the data is used by the comprehensive analysis unit 32, the video analysis unit 13, and the voice analysis unit 23 to improve the analysis accuracy. Further, the information exchange unit 41 provides the same data to the external device 81, so that the analysis accuracy in the other behavior analysis device is improved.

The information output unit 42 has a function of outputting an event or a sign detected by the comprehensive analysis unit 32 for reporting. The information output unit 42 outputs, for example, a paper report, an e-mail, or an SNS message intended to be read by the user of the behavior analysis device 1. Note that SNS is an abbreviation for social networking service. That is, the information output unit 42 outputs information about the event detected by the comprehensive analysis unit 32.
In addition, "event" and "symptom" may be collectively called "event".

The external device 81 is a device of the other party with which the behavior analysis device 1 exchanges information. Specifically, as described above, the external device 81 stores or provides information input / output by another behavior analysis device installed in another facility or the like, or the behavior analysis device 1. It is a computer server device. Further, the external device 81 may be a web server device (for example, a server of a so-called unofficial school site) that provides information in the form of a web.

The camera 82 is a video camera for capturing an image. The camera 82 may be a still camera for capturing a still image.
The microphone 83 is an audio signal acquisition means that collects sound at a place where it is installed and outputs it as an electric signal.
As already described, the camera 82 and the microphone 83 are basically paired and set in a predetermined place.

Next, an example of image analysis by the image analysis unit 13 will be described.
FIG. 3 is a schematic view showing an example of the result of analysis by the image analysis unit 13 of the image taken by one camera 82. The analysis process shown in the figure is human recognition and individual identification. This example is an analysis result of a classroom image taken by a camera 82 installed in the classroom. The image analysis unit 13 recognizes a person included in the image. As a result, the image analysis unit 13 assigns labels such as A, B, and C to the people included in the image. Then, put those labels on the places where people are shown in the video. Further, the image analysis unit 13 performs a process of identifying an individual based on the personal feature information read from the personal feature storage unit 31. As a result, the image analysis unit 13 associates the label with the information that identifies the person in the recognition result (for example, name, personal ID, etc.). In the example shown in the figure, a label is displayed at a place where a person is recognized in the classroom. Further, outside the video frame, labels (A, B, C, etc.) and personal identification information (name in this example) are displayed in association with each other. The image analysis unit 13 can save the personal identification information in a form associated with the position in the image.
The location of the camera is not limited to the classroom. It is also possible to install a camera so as to show the necessary space other than the classroom so that the person can be recognized and the individual can be identified in the image.

FIG. 4 is a schematic view showing an example of the result of analysis by the image analysis unit 13 of the image taken by one camera 82. The analysis process shown in the figure is to grasp the distance between people in the image. This example is an analysis result of a classroom image taken by a camera 82 installed in the classroom. The image analysis unit 13 recognizes the person included in the image and identifies each individual. As a result, the image analysis unit 13 assigns labels such as A and B to the people included in the image, and associates the labels with personal identification information (for example, name). Then, the image analysis unit 13 measures the distance between the persons labeled A and B. The image analysis unit 13 can add the calculated distance information to the image and save it.
The calculation of the distance between individuals in the video is performed using a surveying method based on parameters given in advance. Here, the parameters are the plan view of the classroom to be photographed, the installation location of the camera 82 (the position when viewed in a plan view and the height from the floor surface), and the orientation of the camera (the orientation in the plane and the depression angle (direction in the plane and the depression angle). Or elevation angle)) or the shooting angle of view of the camera. The distance calculation itself by the surveying method can be performed by using the existing technique. Further, when calculating the distance, the position of the subject may be appropriately corrected by using a specific subject (object fixedly installed in the classroom, an image marker, etc.) included in the image.
In the illustrated example, only two people are shown on the screen, but there is no restriction on the number of people in the video. The distance between any two people in the video can be measured.
Also, the location of the camera is not limited to the classroom. A camera may be installed so as to show a necessary space other than the classroom, and the distance between people shown in the image may be calculated.

FIG. 5 is a schematic view showing an example of the result of analysis by the image analysis unit 13 of the image taken by one camera 82. The analysis process shown in the figure is the detection of behavior patterns. This example is an analysis result of a classroom image taken by a camera 82 installed in the classroom. The image analysis unit 13 recognizes the person included in the image and identifies each individual. As a result, the image analysis unit 13 assigns labels such as A, B, C, and D to the person in the image, and associates the label with personal identification information (for example, name). Then, the video analysis unit 13 refers to the behavior pattern stored in advance and determines whether or not the corresponding behavior pattern is included in the video to be analyzed. Pre-stored behavior patterns include patterns of movement of a specific person, patterns of relative positional relationships of a plurality of people, and patterns of combinations thereof. The pattern of human movement represents the locus of human movement and the locus of movement of human limbs and the like. In the illustrated example, a pattern in which A, B, and C surround D is detected. The video analysis unit 13 can add the detected behavior pattern information to the video and save it.

FIG. 6 is a block diagram showing an internal functional configuration of the comprehensive analysis unit 32. As shown in the figure, the comprehensive analysis unit 32 includes a sign detection rule storage unit 321, a temporary pattern detection unit 322, a long-term pattern detection unit 323, and a detection result storage unit 324.

The symptom detection rule storage unit 321 stores a rule for detecting an event or a symptom based on a video analysis result, an audio analysis result, or both. That is, the sign detection rule storage unit 321 stores a rule for detecting an event based on the condition of the video pattern detected by the video analysis unit 13 and the condition of the audio pattern detected by the audio analysis unit 23.
An example of the data configuration of the sign detection rule storage unit 321 will be described later.

The temporary pattern detection unit 322 has a function of detecting a temporary pattern based on the analysis result of the video or audio passed from the video analysis unit 13 or the audio analysis unit 23. The temporary pattern detection unit 322 writes the detection result in the detection result storage unit 324. The temporary pattern detection unit 322 detects a temporary pattern based on the rules stored in the sign detection rule storage unit 321. That is, when the pattern acquired from the outside matches the condition stored in the sign detection rule storage unit 321, the temporary pattern detection unit 322 detects the event as a temporary pattern. That is, the temporary pattern detection unit 322 detects an event based on the video pattern and the audio pattern detected at the temporary point.
The temporary pattern detection unit 322 also provides the detected result to the outside of the comprehensive analysis unit 32.

The long-term pattern detection unit 323 has a function of detecting a long-term pattern based on a time series of predetermined patterns stored in the detection result storage unit 324. The long-term pattern detection unit 323 writes the detection result in the detection result storage unit 324. The long-term pattern detection unit 323 may detect a further long-term pattern based on the detection result written in the detection result storage unit 324 by itself. Further, when the long-term pattern detection unit 323 detects a long-term pattern, the image analysis result at that time may be received from the image analysis unit 13 and used. Further, when the long-term pattern detection unit 323 detects a long-term pattern, the voice analysis result at that time may be received from the voice analysis unit 23 and used.
The long-term pattern detection unit 323 detects a long-term pattern based on the rules stored in the sign detection rule storage unit 321. That is, when the pattern acquired from the outside and the past detection results stored in the detection result storage unit 324 match the conditions stored in the sign detection rule storage unit 321, the long-term pattern detection unit 323 sets the pattern. Detect the event as a temporary pattern. That is, the long-term pattern detection unit 323 detects an event over a plurality of time points based on the event detected by the temporary pattern detection unit 322 at least at a plurality of time points.
The long-term pattern detection unit 323 also provides the detected result to the outside of the comprehensive analysis unit 32.

The detection result storage unit 324 stores and stores patterns (events, signs, etc.) detected by the temporary pattern detection unit 322 and the long-term pattern detection unit 323.

FIG. 7 is a schematic view showing an example of a rule stored in the sign detection rule storage unit 321. As shown in the figure, the sign detection rule storage unit 321 stores data having a tabular structure. This table has items of numbers, events, video analysis methods, and audio analysis methods. The number is a serial number assigned to each rule. The event column stores the name of the symptom to be detected. The image analysis method column stores the patterns detected by the image analysis method. The voice analysis method column stores the patterns detected by the voice analysis method. When the patterns described in the video analysis method or audio analysis method column match, it is detected that the event (or sign) described in the event column has occurred. As a rule, when both the pattern by the audio analysis method and the pattern by the video analysis method are described, it is detected that the event (or sign) has occurred when both patterns match.

The temporary pattern detection unit 322 and the long-term pattern detection unit 323 determine binary whether or not a predetermined event (or sign) has occurred based on the rules stored in the sign detection rule storage unit 321. Not only that, the likelihood may be output as a judgment result.
That is, the likelihood may be determined according to the degree to which the precondition is satisfied, instead of the judgment based only on the binary value of whether or not the precondition of a certain event (or sign) is satisfied.
Also, when multiple preconditions are specified, it is not determined that the event (or sign) has occurred only when all the preconditions are satisfied, but how many of the multiple preconditions are satisfied. Depending on the situation, the likelihood of the event (or sign) may be calculated and output. At this time, weights may be given to each of the plurality of preconditions in advance and used for the likelihood calculation.

The first rule in the illustrated example is a rule regarding "disappearing, ignoring". The condition by the image analysis method in this rule is the pattern that "a specific student is frequently isolated from the group during breaks or lunch". In addition, this rule does not have a condition based on the voice analysis method.
In addition, the comprehensive analysis unit 32 refers to the time information associated with the video and audio and the schedule information at the facility (for example, the information of the timetable at the school), so that the video and audio to be analyzed can be, for example, ". Determine if it is "at break or lunch".
In addition, the "high frequency" specified in this rule is determined based on a threshold value for frequency specified separately.

The second rule in the illustrated example is a rule regarding "attack on the body". The condition of the image analysis method in this rule is a pattern of "violent movement, attack on the opponent by hands / arms / legs / legs". Further, the condition by the voice analysis method in this rule is a pattern of "detection of a loud voice exceeding a predetermined amount and a specific keyword".
The comprehensive analysis unit 32 detects "violent movement" depending on whether or not the amount of movement of the person shown in the image per unit time exceeds a predetermined threshold value even temporarily.

The third rule in the illustrated example is a rule regarding "harassment". The condition by the image analysis method in this rule is the pattern that "the facial expression is crying or in trouble". In addition, the condition by the voice analysis method in this rule is a pattern of "ridicule, crying, detection of a specific keyword".
The image analysis unit 13 holds in advance the facial expression features for each emotion type (crying, laughing, anger, confusion, etc.), and the degree to which the facial features of the person appearing in the image match the emotion type. The likelihood for each emotion type is output according to.
Then, the comprehensive analysis unit 32 calculates the degree of matching with this rule according to the likelihood of the emotion type.

Here, the significance of the long-term pattern detection unit 323 possessed by the comprehensive analysis unit 32 will be supplementarily described. When analyzing a person's behavior with video or audio, the superficial features of the temporary point do not always correspond exactly to the person's emotions. For example, the first rule above also includes the condition that "at high frequency during breaks or lunch, specific students are isolated from the group" and "highly frequently". This means that it is important not only to have a meaning that can be grasped only from temporary behavior, but also to have a meaning that can be read from the tendency of long-term and continuous behavior patterns.
Therefore, the sign detection rule storage unit 321 according to the present embodiment can store not only the condition of hitting the pattern at one time point but the condition of hitting the pattern over a long period of time as a rule. Further, in the present embodiment, by providing the long-term pattern detection unit 323, it is possible to detect a symptom while referring to the past detection results.
As described above, in the present embodiment, it is possible to analyze the behavior by a long-term tendency based on video and audio.

FIG. 8 is a schematic view showing the configuration of the authority setting table for setting the user's authority in the behavior analysis device 1. The authority setting table is provided in a predetermined area in the storage means in the behavior analysis device 1. As shown in the figure, the authority setting table includes each item of user ID, name, job title, class in charge, and access authority. The access authority includes information on the access authority to the school-wide data and the access authority to the class data in charge. In consideration of the privacy of the parties, the behavior analysis device 1 restricts access to the data according to the user. The responsible class data is the data of the class that the user is in charge of. School-wide data is data for the entire school, including data for each class. Then, in the figure, "○" in the access authority means that the user has the access authority to the data. Further, "-" in the access authority means that the user does not have the access authority to the data.

In the illustrated example, if the job titles are "Principal", "Vice Principal", and "Vice Principal", the user has access to school-wide data. In addition, when the job title is "teacher (homeroom teacher)", the user does not have the access authority to the data of the whole school, but has the access authority to only the data of the class to which he / she is in charge.

The user can refer to the detection result data only within the range of the granted access authority. In addition, the user can play a video file or an audio file only within the range of the granted access authority. The behavior analysis device 1 transmits only the permitted information to the terminal device of the user according to the authority of the user who is signed in.

FIG. 9 is a schematic view showing an example of the information output by the information output unit 42. The illustrated output information is one of the analysis results by the comprehensive analysis unit 32. The illustrated output information includes destination, type, occurrence date and time, urgency, event summary, and information on each item of the video link. This output information relates to an event detected as a result of analyzing the video and audio of a classroom in a certain class in the school.

The destination is the destination of this output information. For example, when this information is output by e-mail, the e-mail address of the appropriate user is described as the destination. In the illustrated example, "E-tree teacher" is described as the destination. This user is the homeroom teacher of the class in which the event was detected.

The type represents the type of event (or symptom) detected. The name of the type is stored in the sign detection rule storage unit 321. In the illustrated example, the type is "attack on the body".
The date and time of occurrence is the time associated with the video or audio of the detected event (or sign). If a long-term event (or symptom) is detected, the date and time of occurrence column will indicate a wide range of periods.

The urgency is the distinction information indicating the urgency of this output information. For example, the degree of urgency is classified into three distinctions: "urgent", "estimated", and "attention". In these three distinctions, the degree of urgency is in the order of "urgent"> "estimation"> "caution". For example, "urgent" means that an action requiring an urgent response (such as bullying) is presumed. In addition, "estimation" indicates that behavior (such as bullying) that does not require urgent response is presumed. In addition, "caution" indicates that a matter that requires caution but is not estimated, or an event (or sign) that is estimated when the same type of caution is continued for two days or more has been detected.

The event summary is summary information including a specific description of the detected event and identification information such as the names of the parties (students) involved in the event. The identification information of the parties is obtained by the personal identification processing of the video by the image analysis unit 13 and the personal identification processing of the audio by the voice analysis unit 23. These personal identification information are stored in association with the detected event (or sign). That is, the information output unit 42 outputs the event summary information including the personal identification information identified by at least one of the video analysis unit 13 and the audio analysis unit 23 regarding the event.
The video link is link information for reproducing the corresponding part of the video or audio that is the basis for determining the detected event.

According to this embodiment, not only audio but also video and audio-based events can be widely grasped, which leads to prevention or early detection of problem behavior (bullying, etc.).

Further, according to the present embodiment, a temporary pattern detection unit that detects an event based on a video pattern and an audio pattern detected at a temporary point, and at least an event detected by the temporary pattern detection unit at a plurality of time points are used. Since it has a long-term pattern detection unit that detects an event over a plurality of time points, it is possible to detect an event over a plurality of time points.

In the above embodiment, the personal feature storage unit 31 memorizes features including facial features, body features, and clothing features as individual visual features. In this variant, the visual characteristics of this individual are updated at a given time (eg, at the beginning of the first daily lesson in school). Specifically, according to the schedule information in the facility (for example, school timetable information), the image acquisition unit 11 acquires and stores the image storage unit 12 at a predetermined time on a daily basis (excluding holidays of the facility). The image analysis unit 13 calculates the image features based on the information obtained. At this time, the image analysis unit 13 estimates the relationship between the position in the image and the individual by referring to the seating chart information stored in advance. Then, the image analysis unit 13 writes the calculated feature amount in the personal feature storage unit 31. For example, an individual's facial features change little daily, but an individual's clothing (color and shape) and hairstyle can change significantly from day to day. In addition, although the degree of daily change in individual physical characteristics (height, body shape, etc.) is small, it may change significantly when viewed on a semi-annual basis, for example. According to this modification, since the data of individual characteristics is updated at a predetermined timing, the accuracy of individual identification can be further improved.
The seating chart information is information that represents the range of places where individuals can be identified. Alternatively, the seating chart information is information that represents the relative positional relationship between a plurality of individuals that can be identified.

That is, the image analysis unit 13 reads out the schedule information in the facility and the seating chart information of the individual in the predetermined place stored in the storage medium. Then, the image analysis unit 13 calculates the visual characteristics of each individual corresponding to the individual identification information determined or estimated by the seating chart information based on the image at the predetermined timing determined by the schedule information. Then, the image analysis unit 13 updates the personal characteristics related to the image stored in the personal feature storage unit 31 by the calculated visual features. In addition, the use of seat information is not limited to improving the accuracy of personal identification, but can also be used to grasp the tendency of changes in human relationships due to seat changes.

According to at least one embodiment described above, a comprehensive analysis unit that detects a specific event related to human behavior based on a video pattern detected by the video analysis unit and an audio pattern detected by the audio analysis unit. By having it, it is possible to prevent behavioral problems and detect signs early.
In the above embodiment, the detection of specific events related to human behavior based on images and sounds acquired from a camera or microphone in a school facility has been described mainly for one space (classroom), but each space has been described. By comparing the analysis results in (classroom), it is possible to grasp the tendency in each class and the tendency of the whole school, and for example, the analysis result can be used for class change and homeroom assignment.
Further, in the above embodiment, the behavior analysis device 1 has been described as an example of being applied to a school facility, but it can also be applied to a kindergarten, a nursery school, a nursing facility, etc. where group behavior is performed in one facility, and similarly, in the facility. By analyzing human behavior in Japan, it is possible to detect and prevent signs of problems in advance.

In addition, at least a part of the functions of the behavioral analysis device of the above-described embodiment may be realized by a computer. In that case, the program for realizing the function of the behavioral analyzer may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. .. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, as well as in the scope of the invention described in the claims and the equivalent scope thereof.

1 ... Behavioral analyzer, 11 ... Video acquisition unit, 12 ... Video storage unit, 13 ... Video analysis unit, 21 ... Voice acquisition unit, 22 ... Voice storage unit, 23 ... Voice analysis unit, 31 ... Personal feature storage unit, 32 ... Comprehensive analysis unit, 41 ... Information exchange unit, 42 ... Information output unit, 81 ... External device, 82 ... Camera, 83 ... Microphone, 321 ... Sign detection rule storage unit (Detection rule storage unit), 322 ... Temporary pattern detection unit 323 ... Long-term pattern detection unit, 324 ... Detection result storage unit

Claims (4)

The video acquisition unit that acquires video and
An image analysis unit that analyzes a person appearing in the image acquired by the image acquisition unit and detects an image pattern, and an image analysis unit.
A voice acquisition unit that acquires voice, and
A voice analysis unit that analyzes a human voice included in the voice acquired by the voice acquisition unit and detects a voice pattern, and a voice analysis unit.
A comprehensive analysis unit that detects a specific event related to human behavior based on the video pattern detected by the video analysis unit and the audio pattern detected by the audio analysis unit.
An information output unit that outputs information about the event detected by the comprehensive analysis unit, and an information output unit.
With
The comprehensive analysis unit includes a detection rule storage unit that stores rules for detecting an event based on the conditions of the video pattern detected by the video analysis unit and the conditions of the audio pattern detected by the audio analysis unit. By referring to the rule read from the detection rule storage unit, the event is detected based on the video pattern and the audio pattern.
The comprehensive analysis unit
A temporary pattern detection unit that detects the event based on the video pattern and the audio pattern detected at one time point,
At least, a long-term pattern detection unit that detects an event over a plurality of time points based on the event detected by the temporary pattern detection unit at a plurality of time points.
Behavioral analyzer equipped with. The video acquisition unit that acquires video and
An image analysis unit that analyzes a person appearing in the image acquired by the image acquisition unit and detects an image pattern, and an image analysis unit.
A voice acquisition unit that acquires voice, and
A voice analysis unit that analyzes a human voice included in the voice acquired by the voice acquisition unit and detects a voice pattern, and a voice analysis unit.
A comprehensive analysis unit that detects a specific event related to human behavior based on the video pattern detected by the video analysis unit and the audio pattern detected by the audio analysis unit.
An information output unit that outputs information about the event detected by the comprehensive analysis unit, and an information output unit.
A personal feature storage unit that stores personal features related to video and audio,
With
The image analysis unit performs personal identification processing of a person appearing in the image based on the personal characteristics of the image read from the personal feature storage unit.
The voice analysis unit performs personal identification processing of the human voice included in the voice based on the personal characteristics related to the voice read from the personal feature storage unit.
The information output unit outputs information about the event, including personal identification information identified by at least one of the video analysis unit and the audio analysis unit regarding the event.
The image analysis unit reads out the schedule information in the facility and the seating chart information of the individual in the predetermined place stored in the storage medium, and based on the video at the predetermined timing determined by the schedule information, the seating chart The visual features of each individual corresponding to the personal identification information determined or estimated by the information are calculated, and the calculated visual features are used to obtain the personal features related to the image stored in the personal feature storage unit. Update,
Behavioral analyzer. The behavior analysis device according to any one of claims 1 and 2 , wherein the behavior analysis device is installed in a facility used in a group. Computer,
A program for functioning as the behavioral analyzer according to any one of claims 1 to 3.

Images