JP2010005326A - Event identification system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an event identification system capable of recording who did what and when in a series of working actions such as a surgery accurately and along a time series in such a way as can be easily recognized on a list by a simple device structure. <P>SOLUTION: The event identification system for recording the time when a prescribed event occurs in a series of working actions in such a way as can be retrieved is configured as follows. The event identification system includes: an acoustic data collecting means 300 for collecting acoustic data including heartbeat sounds of a person in charge of the series of working actions; a filtering means 500 for extracting data related to the heartbeat sounds from the acoustic data; and a heartbeat sound data processing means 700A for generating respiration information on the respiration of the person in charge of the series of working actions based on the heartbeat sound data and determining and recording the time when the prescribed event occurs in the series of working actions based on the respiration information generated by a respiration information generating means. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention stores, in a time series, information related to operations performed in a series of work activities such as surgery and events occurring therein, thereby performing post-analysis, verification, and further verification of the series of work actions. Relates to an event identification system that can be used effectively for optimizing the action procedure.

  Taking a medical site as an example, when a medical practice such as surgery is performed, the details are recorded as a document by a doctor involved in the medical practice and attached to a medical record or the like and stored. The record book created in this way is often attached with charts and photographs showing data output from various devices during surgery.

  For this reason, after performing the said medical practice, when it is going to verify or reproduce the procedure and content of those medical practice, it is centered on browsing the record book mentioned above. However, in the above-described record book, various kinds of data are often described (non-chronologically) regardless of the order of time. Moreover, not all information is recorded in the record book. In other words, there are many cases where information related to medical practice is not centralized and is stored individually in each department concerned.

  For example, even an operation that is considered to be relatively simple, in addition to monitor information for observing the patient's biological conditions such as electrocardiogram, respiration, blood pressure, etc., an anesthesia monitor (anesthetic amount, input rate) and electric knife There are many devices related to medical practices such as (voltage, time). Further, as medical practices performed by the surgeon, information related to various medical practices such as use of instruments and medication is often involved.

  For this reason, even doctors who are familiar with medical knowledge must collect and analyze information over a considerable amount of time in order to confirm and verify all aspects of medical practice after the fact. The effort and time required are enormous. Also, it is not easy to collect all the information that seems necessary, and unrecorded information such as the behavior of staff other than the surgeon who is the center of the operation and changes in the operating room environment is important. There are also many. In other words, even if an attempt is made to construct a more optimal procedure by examining medical practices such as surgery performed, a great deal of effort is required to realize such optimization because of limited information that can be studied. is there.

  As described above, even a specialist such as a doctor is very difficult to confirm and verify the entire medical practice after the fact. Moreover, it is almost impossible for ordinary people who are not familiar with medical practice to read the medical practice accurately from the record book after the fact.

  Therefore, in order to confirm and verify objectively the medical practice such as surgery and treatment, in addition to creating the record book as described above, the conversation exchanged at the site of the medical practice is recorded. A technique has been proposed in which a scene of the medical practice is photographed with a video camera or the like, and audio obtained by recording and video obtained by photography are used in combination.

  However, even when audio and video are used in this way, the audio and video acquired by recording and the medical actions recorded in the above-described record book are linked on the time axis. The amount of work required to do this is enormous, requiring a lot of time and labor. Such a problem is caused by a lack of the concept of organizing and capturing each medical action by an operator or medical staff during a medical action such as surgery.

  In view of such circumstances, for example, Patent Document 1 proposes the following technique. That is, in Patent Literature 1, a sensor that detects information related to surgery, a server that stores information detected by the sensor in connection with time information, and information stored by the server is displayed in time series. There is disclosed a medical information system including a display device, the server having a function capable of analyzing stored information based on time information.

According to the medical information system disclosed in Patent Document 1, almost all information related to medical practices such as surgery and treatment can be recorded over time. Furthermore, the recorded information is presented in a time-series display, and the medical practice based on the time information can be easily analyzed.
JP 2004-280455 A

  By the way, for example, when analyzing a medical practice such as surgery, a system including a device that does not hinder the movement of the operator as much as possible is preferable. In other words, the device used for collecting various information is preferably as simple as possible.

  Furthermore, if data relating to the breathing of a person engaged in the medical practice can be collected, the analysis of the medical practice can be made more precise. However, as is well known to those skilled in the art, in order to collect data related to human respiration, a very large device is required. Therefore, such a system has not been realized yet. The technique disclosed in Patent Document 1 is not a technique for solving these problems.

  The present invention has been made in view of the above circumstances. For example, in a series of work activities such as surgery, it is possible to grasp who performed what and when in a list with a simple device configuration and in time series. An object of the present invention is to provide an event identification system for recording.

In order to achieve the above object, an event identification system according to the present invention according to claim 1 comprises:
An event identification system for recordably recording a time when a predetermined event occurs in a series of work actions,
Acoustic data collection means for collecting acoustic data including heartbeat sounds of persons involved in the series of work actions;
Heartbeat data extraction means for extracting data relating to the heartbeat from the acoustic data collected by the acoustic data collection means;
Respiration information generating means for generating respiration information related to respiration of a person involved in the series of work actions based on the heartbeat data extracted by the heartbeat data extracting means;
Event identifying means for determining and recording the time at which the predetermined event occurred in the series of work actions based on the respiratory information generated by the respiratory information generating means;
It is characterized by comprising.

In order to achieve the above object, an event identification system according to the present invention according to claim 8 comprises:
On-site information storage means for storing, in chronological order, a video image of a site where a series of work actions are performed, information for identifying a staff member present at the site, and information for specifying the position of the staff member in the video image ,
Acoustic data collection means for collecting acoustic data including the staff's heartbeat;
Heartbeat data extraction means for extracting data relating to the heartbeat from the acoustic data collected by the acoustic data collection means;
Respiration information generating means for generating respiration information related to breathing of the staff based on the heartbeat data extracted by the heartbeat data extracting means;
Event identifying means for determining a time when a predetermined event occurs in the series of work actions based on the respiratory information generated by the respiratory information generating means;
A queue index storage means for storing a time when the event identification means determines that the predetermined event has occurred;
A cue display control means for displaying a cue mark corresponding to the time stored by the cue index storage means, and displaying the video from the time corresponding to the cue mark designated by an operator;
It is characterized by comprising.

In order to achieve the above object, an event identification system according to the present invention according to claim 9 comprises:
An event identification system for recordably recording a time when a predetermined event occurs in a series of work actions,
Photographing means for photographing the site where the series of work actions are performed and obtaining video information;
A transmitter that is attached to a person involved in the series of work actions and transmits an individual identification signal that is an individual identification information for identifying an individual;
Receiving means for receiving an individual identification signal transmitted from the transmission device, having a spatial resolution for specifying a transmission location of the individual identification signal;
Generalization means for associating the video information acquired by the imaging means with the transmission location of the individual identification signal received by the reception means;
A registration device for associating the individual identification information given to each of the transmission devices with personal identification information about a person wearing the transmission device;
Acoustic data collection means for collecting acoustic data including heartbeat sounds of persons involved in the series of work actions;
Heartbeat data extraction means for extracting data relating to the heartbeat from the acoustic data collected by the acoustic data collection means;
Respiration information generating means for generating respiration information related to respiration of a person involved in the series of work actions based on the heartbeat data extracted by the heartbeat data extracting means;
Event identifying means for determining the time at which the predetermined event occurred in the series of work actions based on the respiratory information generated by the respiratory information generating means;
It is characterized by comprising.

  According to the present invention, there is provided an event identification system that records when and what has been performed in a series of work activities such as surgery, for example, so that the list can be grasped in a time-series manner with a simple device configuration. be able to.

[First Embodiment]
Hereinafter, an event identification system according to a first embodiment of the present invention will be described with reference to the drawings. Here, for convenience of explanation, an operation will be described as an example of a work action to be analyzed by the event identification system according to the first embodiment.

  FIG. 1 is a block diagram showing a configuration example of the event identification system according to the first embodiment. As shown in the figure, the event identification system includes an acoustic data collection unit 300, a filter unit 500, a heartbeat data processing unit 700A, and a heartbeat removal data processing unit 700B.

  The acoustic data collecting unit 300 is a unit for collecting acoustic data related to a work action to be analyzed by the event identification system according to the first embodiment.

  Specifically, as shown in FIG. 2, for example, the acoustic data collection unit 300 includes a contact-type microphone 300A worn by a person involved in the analysis target work action, and the contact-type microphone 300A. The transmitter 300B for transmitting the acoustic data collected by the transmitter 300C to be described later, and the receiver 300C for receiving the acoustic data transmitted by the transmitter 300B.

  More specifically, the contact-type microphone 300A is a microphone that uses a technique used in a so-called electronic stethoscope, and is different from the electronic stethoscope in that it is used by being attached to a subject.

  Of course, the configuration of the acoustic data collection means 300 shown in FIG. 2 is merely an example. Therefore, for example, the transmitter 300B may be configured integrally with the filter unit 500, the heartbeat data processing unit 700A, and the heartbeat removal data processing unit 700B. In such a configuration, the data transmitted by the transmitter 300B is data after analysis processing by the filter unit 500, the heartbeat data processing unit 700A, and the heartbeat removal data processing unit 700B, which will be described later. The amount of data related to transmission can be reduced.

  Further, data transmitted and received between the transmitter 300B and the receiver 300B may be analog data or digital data. Of course, the data may be subjected to pre-processing such as predetermined compression processing before being transmitted by the transmitter 300B.

  The filter unit 500 performs a predetermined filtering process on the acoustic data collected by the acoustic data collection unit 300, and extracts heartbeat data that is data related to the heartbeat of the wearer of the contact microphone 300A. The sound is output to the sound data processing means 700A. The filter unit 500 generates data obtained by removing the heartbeat data from the acoustic data collected by the acoustic data collection unit 300, and outputs the data to the heartbeat removal data processing unit 700B as heartbeat removal data.

  In detail, the acoustic data relating to the heartbeat sound has a very low frequency (specifically, includes many frequencies around 50 Hz). For example, among the acoustic data collected by the acoustic data collecting means 300, for example, By cutting a frequency of 200 Hz or higher, it is possible to extract acoustic data relating to heartbeat sounds. Therefore, the filter unit 500 extracts the heartbeat data by applying such a filtering process to the acoustic data.

  When the filter means 500 extracts the heartbeat data from the acoustic data by the above-described process, the filter means 500 performs a confirmation process that the data related to the extraction is data having a specific cycle as the heartbeat data. Do.

  That is, in the heartbeat sound data extraction process, the filter unit 500 performs a confirmation process by using a normal pattern recognition technique to determine whether or not the data related to the extraction process is valid as heartbeat data.

  The heartbeat data processing means 700A is means for performing analysis processing on the heartbeat data. The specific processing content by the heartbeat data processing means 700A will be described later.

  The heartbeat removal data processing means 700B is a means for performing analysis processing on the heartbeat removal data. The specific processing contents by the heartbeat sound removal data processing means 700B will be described later.

  Hereinafter, the heartbeat data analysis processing by the heartbeat data processing unit 700A will be described in detail with reference to FIGS.

  First, the heartbeat sound data processing unit 700A performs a process described later on the basis of the heartbeat sound data collected by the acoustic data collection unit 300 and extracted by the filter unit 500. "Is calculated.

  Specifically, as shown in FIG. 3, one period in the waveform of the heartbeat data is substantially the same as the RR interval calculated from the waveform acquired by the electrocardiogram. That is, the generation timing interval of the wave with the largest amplitude in the waveform acquired by the electrocardiogram is substantially the same as the heartbeat generation timing interval of the wave with the largest amplitude in the heartbeat sound data.

  By utilizing this fact, in the first embodiment, “substantially the same interval as the RR interval” is calculated based on the heartbeat data.

Specifically, the above-described “substantially the same interval as the RR interval” uses, for example, time t1 and time t2 (both are predetermined heartbeat generation timings) shown in FIG.
t2-t1
It can be expressed as Hereinafter, this time interval is referred to as “instantaneous time interval”.

And the heart rate derived from this instantaneous time interval (instantaneous heart rate = [beats / minute]) is
60 / (t2-t1)
It can be expressed as

  In other words, the instantaneous time interval and the instantaneous heart rate described above are in a reciprocal relationship with each other. It is known that the instantaneous heart rate and respiration have a correlation. In the first embodiment, using this fact, the heartbeat processing data 700A estimates the respiratory state of the wearer of the contact microphone 300A based on the temporal change of the instantaneous heart rate. Hereinafter, an example of a method for estimating a respiratory state from a graph showing temporal changes in instantaneous heart rate will be described.

  First, as shown in FIG. 4, the graph showing the temporal change in the instantaneous heart rate is always a graph showing physiological fluctuations (only with normal breathing). Note that there is a slight delay time until the change in the respiratory state actually appears as a time change in the instantaneous heart rate (see the wavy line shown in FIG. 4).

  Specifically, it is possible to estimate the respiration depth from the amplitude of the waveform indicating the temporal change in the instantaneous heart rate. That is, as shown in FIG. 5, it can be estimated that the period in which the waveform amplitude is large is a period in which deep breathing such as deep breathing is performed. On the other hand, during the period when breathing is stopped, the amplitude of the waveform is extremely small as shown in FIG.

  Incidentally, FIG. 7 is a graph obtained by reading the respiration frequency and the respiration depth from the graph showing the time change of the heart rate as described above, and making these data correspond to each other on the same time axis. The heartbeat sound data 700A performs, for example, the following breathing state estimation processing based on such breathing frequency and breathing depth data.

(Estimation Example 1) It can be estimated that the period in which the respiration frequency is high and the respiration depth is shallow is a period in which precise work is not performed.

(Estimation example 2) A period in which the frequency of breathing is low and the breathing is shallow can be estimated as a period in which precise work is performed (for example, approximately 10 seconds to 30 seconds with an interval substantially the same as the RR interval). If there is almost no fluctuation, it can be estimated that the person has stopped breathing, and the total time during which the breath is stopped is measured, for example, in units of 1 to 5 minutes (in other words, the frequency is measured). (It can be presumed that you are working precisely and carefully)
(Estimation example 3) It can be estimated that the period in which the frequency of breathing is high and the depth of breathing is deep is not a work period but a rest period.

  FIG. 8 is a graph showing a temporal change in instantaneous heart rate when the time axis is taken in units of time. Here, as the value of the instantaneous heart rate at a certain time, as shown in FIG. 8, the average value of the instantaneous heart rate in the corresponding time zone may be used.

  From the temporal change of the instantaneous heart rate when the time axis is taken in units of hours, it is possible to estimate the change in the respiratory state with a larger trend. For example, a sudden change in the instantaneous heart rate occurs near the time T1 shown in FIG. From this, it can be estimated that an event having a great influence on the subject mentally / physically occurred in the time zone corresponding to the time T1.

  FIG. 9 is a diagram showing a graph in which the frequency estimated that the breathing is stopped by the above-described estimation processing is taken on the vertical axis and time is taken on the horizontal axis, that is, a graph of the temporal change in the respiratory stop frequency. . By referring to such a temporal change in the frequency of respiratory arrest, it becomes possible to perform more accurate event identification.

  As described above, in the first embodiment, reading the breathing state, which was impossible before without using an unpractical large-scale device, is based on the very simple device configuration as described above. Made possible by a system consisting of:

  Specifically, in order to directly measure the breathing state, a large-scale device that cannot be worn by a person who operates must be worn by the subject. However, in order to simply determine only the breathing state, quantitatively detailed information such as “how many hours and how many milliliters of breathing” is not necessary using such a large-scale device. In the first embodiment, paying attention to this point, the respiratory state can be estimated by a system having a very simple device configuration by analyzing the heartbeat data described above.

  The event identification accuracy is improved by using the respiratory state estimation process as a means of event identification.

  Hereinafter, the heartbeat data analysis processing by the heartbeat removal data processing means 700B will be described in detail with reference to FIG.

FIG. 10 is a diagram illustrating an example of a graph in which the vertical axis represents the frequency ω and the horizontal axis represents the time t, that is, a graph obtained by frequency mapping the heartbeat removal data along the time axis. Based on such frequency mapping of the heartbeat removal data, the heartbeat removal data processing means 700B can perform, for example, the following estimation processing (Estimation Example 1) frequency band corresponding to normal conversation It can be estimated that the time zone in which many signals appear is the time zone in which normal conversation is made.

(Estimation Example 2) When a signal appears in a frequency band corresponding to an utterance in an emergency, it can be estimated that some emergency has occurred in that time slot. That is, by performing the frequency analysis in this way, it is possible to identify whether it is an utterance or noise. Specifically, the utterance has characteristics such as containing almost no high-frequency component and a fundamental frequency (formant). More specifically, it is possible to detect the state of the conversation by measuring the total time during which the utterance is performed, for example, in units of 10 to 30 seconds (in other words, measuring the frequency).

(Estimation Example 3) Since noise has characteristics such as a large high-frequency component and almost no autocorrelation, the noise can be estimated from such characteristics. More specifically, noise often occurs when clothes or the like rub the contact-type microphone 300A, and the total amount of time in which noise is present is measured, for example, in units of 10 to 30 seconds (in other words, the frequency is measured). The degree of body movement can be measured.

(Estimation example 4)
As shown in the signal 51 shown in FIG. 10, a signal that appears in a very wide range from a low frequency to a high frequency is generally a rubbing sound. In such a time zone, the subject moves or the like. It can be estimated that there are many cases.

(Estimation Example 5) For example, the frequency band near 300 to 1000 Hz is a frequency band corresponding to a human voice, and the time zone in which many signals appear in such a frequency band is the time zone in which conversation is performed. It can be estimated that there is.

(Estimation Example 6) In general, there is little change in heart rate and little change in breathing during the time period in which conversation is performed. Therefore, it can be estimated that such a time zone is a time zone in which precision work is not performed.

(Estimation Example 7) A time zone in which the concentration of the signal appearing in the frequency mapping graph as shown in FIG. 10 is high can be simply estimated as a time zone in which some sort of situation has occurred.

  FIG. 11 is a graph showing a time change graph of the frequency estimated that the conversation is performed by the above-described processing. In other words, FIG. 11 is a diagram showing a graph of the time change of the percentage of the time when the conversation was performed. By referring to such a change in conversation frequency over time, it becomes possible to perform more accurate event identification. Specifically, the change time of the work state can be easily detected by referring to this graph. However, it is preferable to estimate with a certain time width.

  Hereinafter, with reference to FIG. 12A and FIG. 12B, event identification by comprehensive determination processing using multivariate pattern recognition technology will be described. FIG. 12A and FIG. 12B are diagrams showing the concept of event identification by comprehensive determination processing using multivariate pattern recognition technology. In other words, after acquiring a plurality of pieces of information that can be used for event identification by the above-described processing, the following processing may be performed using the plurality of pieces of information to perform more precise event identification processing. .

  Here, for convenience of explanation, event identification by comprehensive determination processing using multivariate pattern recognition technology will be described by taking a case where the number of variables is three as an example. However, in practice, the number of variables is not limited to three.

  First, for example, “breathing stop frequency” is set as the first variable, for example, “time change in breathing stop frequency” is set as the second variable, and “frequency at which clothing is rubbed” is set as the third variable, for example. To do. Then, these variables in a predetermined time zone are plotted on a three-dimensional space as shown in FIG. 12A, and pattern classification is performed on the most plotted area.

  Here, for convenience of explanation, as shown in FIG. 12B, the case where each variable is classified into two types of “large” or “small” with a predetermined threshold as a boundary will be described as an example.

  That is, in the example shown in FIGS. 12A and 12B, the first variable “frequency of breathing stop” is “small”, and the second variable “time change in breathing stop frequency” is “large”. In addition, the third variable “frequency of occurrence of rubbing” is also “large”. In this case, the time zone is classified as “pattern A” in the example shown in FIG. The “pattern A” indicates that the time zone is “a time zone in which a change in work state is detected”.

  Then, if a predetermined marking is added to the time zone (time) determined to be the “time zone in which a change in the working state” is detected in this way, only audio and video in the time zone before and after the marking can be obtained. By simply confirming, it is possible to quickly and easily detect a scene where a problem has occurred.

  As described above, by performing event identification processing based on a plurality of variables, more accurate estimation processing can be performed.

  In addition, the estimation result acquired by the estimation process demonstrated above is memorize | stored in a memory | storage means not shown. The heartbeat sound data processing means 700A and the heartbeat sound removal data processing means 700B may be provided with storage means for storing the estimation processing result.

  As described above, according to the first embodiment, for example, in a series of work activities such as surgery, it is easy to grasp who performed what and when in a list easily and accurately in time series. It is possible to provide an event identification system realized by a simple device configuration. Therefore, for example, it is possible to detect and confirm only the time zone in which a desired event has occurred, without confirming all the records of a long work activity for 10 hours or more.

  Specific examples of work actions to which the event identification system according to the first embodiment can be applied include surgery, and operation of airplanes, cars, trains, and the like. For example, if the event identification system according to the first embodiment is applied in the operation of an airplane, a car, a train, etc., when an accident occurs, how many seconds before the accident the driver felt dangerous. Know exactly. Moreover, even if an actual accident has not occurred, all events occurring in a series of work activities are recorded and analyzed by applying the event identification system according to the first embodiment. In other words, it is possible to solve the problem that a so-called near-miss is overlooked without being reported, and subsequently leads to an actual accident or the like.

  According to the event identification system according to the first embodiment, since various types of information are recorded in chronological order, the time zone in which some event such as an abnormal state has occurred in the work act illustrated above can be seen at a glance. Can be grasped.

  Further, as described above, according to the event identification system according to the first embodiment, it is possible to obtain information over time about the respiratory stop frequency, the conversation frequency, and the operation frequency as shown in FIG. That is, the working state of the wearer of the contact microphone 300A is presented in a list display with time.

(Modification)
Furthermore, by combining the event recognition system according to the first embodiment with a voice recognition technology, a specific important keyword issued during a certain work action is determined as to which of the entire work action is performed. It is possible to display a list of whether or not it was issued at the time.

  Analysis of surgery using voice recognition technology has been attempted in the past, but has not been successful. The reason for the lack of success is that, with conventional speech recognition technology, if environmental noise or the speech of other speakers is mixed into the acoustic data to be analyzed, the success rate of speech recognition will be extremely low. is there. Further, for example, in an operating room or a work place, the echo sound is loud or the noise generated by the instrument is large, so that it is difficult to accurately perform voice recognition itself.

  By the way, in the conventional speech recognition technology, speech recognition is performed by matching with a template existing in a dictionary of speech recognition target vocabularies. However, during work activities such as surgery, in most cases, it is possible to communicate with only utterances and shouts (“it”, “yes”, “do”, etc.) as utterances. Such a shout or the like naturally has little redundancy and cannot be matched with the template described above with high accuracy. Such circumstances can also be cited as a cause of the low success rate of speech recognition.

  Furthermore, the conventional speech recognition technology aims to create text (character string) data based on speech data. However, meaningful text data cannot be obtained with the above-mentioned call or the like even if the speech recognition is successful.

  On the other hand, in the first embodiment, by using the contact-type microphone 300A that is directly worn by the subject as described above, adverse effects due to echoes and noise caused by other speakers, devices, etc. are greatly reduced. be able to. For this reason, high-quality audio data can be obtained. That is, by combining the event identification system according to the first embodiment and the existing speech recognition technology, it is possible to pick up (extract) important keywords with a certain degree of accuracy.

  Furthermore, in operations such as surgery, utterances having important meanings such as “wait”, “dangerous”, and “stop” are often uttered particularly clearly. Therefore, it is relatively highly probable that they can be found by speech (keyword) speech recognition having such an important meaning.

  Therefore, in this modified example, the existing speech recognition technology is used not for the purpose of recognizing the content of an utterance but for the purpose of detecting a time or a time zone when an important utterance is likely to be performed. Is used. In other words, keywords related to important utterances are detected by applying existing speech recognition technology to the first embodiment. Of course, a mark (so-called cue) may be attached to the location detected in this way to improve the later searchability.

[Second Embodiment]
The event identification system according to the second embodiment of the present invention will be described below. Although details will be described later, in the second embodiment, the event identification system according to the first embodiment is applied to a medical site display system to be described later. As a result, information indicating the working state is extracted based on information of various sensors collected at the exact time, and based on the extracted information, an important part of the recording of the video or the like is estimated and marked. (So-called queue) can be attached. Details will be described below.

  FIG. 14 is a diagram illustrating a functional configuration of the medical site display system 101 that records and displays a medical site.

  The medical site display system 101 records a medical site such as an operating room by video, identifies who the medical site staff is along with its location, and identifies and displays the staff in the medical site on the recorded video System. It is used to reproduce medical scenes with images and perform analysis related to improving medical quality and safety.

  The medical site display system 101 includes an imaging device 102, an identification unit 103, a position measurement unit 104, an image processing device 105, a display device 106, and an input device 107. The imaging device 102, the identification unit 103, the position measurement unit 104, and the input device 107 are connected to the video processing device 105 so that signals can be input. The video processing device 105 is connected to the display device 106 so that signals can be input.

  The imaging device 102 is installed in a medical site and captures an image of the medical site. There may be a case where a plurality of imaging devices 102 are installed. The photographing apparatus 102 is, for example, a video camera. The identification unit 103 identifies who the staff is in the medical field. The position measurement unit 104 measures the position coordinates of the staff whose existence is confirmed by the identification unit 103 using a measurement system with the installation position of the identification unit 103 as the origin. The image processing device 105 calculates which pixel on the image corresponds to the position measured by the position measurement unit 104, and associates the staff identification information of the staff identified by the identification unit 103 with the calculated pixel. The superimposed video is generated and displayed on the display device 106. The display device 106 is a monitor such as a liquid crystal display or a CRT display. The input device 107 is a keyboard or mouse that inputs operation signals to the video processing device 105.

  FIG. 15 is a block diagram illustrating a specific example of the configuration of the identification unit 103 and the position measurement unit 104.

  The identification unit 103 includes an RF tag 121 and a wireless transmission / reception unit 122. The position measurement unit 104 includes an ultrasonic oscillator 131, an ultrasonic sensor 132, and a measurement system position calculation unit 133.

  The RF tag 121 and the ultrasonic transmitter 131 are built in the signal transmitter 181. The signal transmitter 181 is attached to the staff. The wireless transmission / reception unit 122, the ultrasonic sensor 132, and the measurement system position calculation unit 133 are built in the transmitter measurement apparatus 182. The transmitter measurement device 182 further includes a control signal transmitter 183. The transmitter measuring device 182 is installed in a medical site.

  The RF tag 121 includes an IC chip and an antenna. The identification code of the staff wearing the signal transmitter 81 is stored in advance in the IC chip. The staff identification code is an ID uniquely assigned to each staff, and is a kind of staff identification information. The RF tag 121 receives radio waves emitted from the outside with an antenna. When an electric wave is received from the outside, an electromotive force supplied to the IC chip is generated by a resonance action, and the RF tag 121 transmits the staff identification code stored in the IC chip via the antenna.

  Further, the RF tag 121 receives a control signal including a staff identification code from the outside, and outputs a trigger signal to the ultrasonic transmitter 131 when the received staff identification code is stored in the IC chip. To do.

  The wireless transmission / reception unit 122 includes an antenna. The wireless transmission / reception unit 122 transmits a radio wave that generates an electromotive force in the RF tag 121, and receives a staff identification code transmitted from the RF tag 121.

  In the identification unit 103, the staff identification code is transmitted from the RF tag 121 to the wireless transmission / reception unit 122, whereby the staff represented by the identification code is specified.

  The control signal transmitter 183 transmits a control signal to the signal transmitter 181. The control signal includes the identification code of the staff whose position is to be measured. When the wireless transmission / reception unit 122 receives the staff identification code, the control signal transmission unit 183 includes the received staff identification code in the control signal and transmits it. When a plurality of staff identification codes are received, they are sequentially included in the control signal and transmitted at a predetermined interval in order to avoid interference.

  The ultrasonic oscillator 131 oscillates an ultrasonic signal to the outside when the trigger signal is input from the RF tag 121.

  The ultrasonic sensor 132 receives the ultrasonic signal generated by the ultrasonic oscillator 131. A plurality of ultrasonic sensors 132 are installed at a medical site at a predetermined distance. Each ultrasonic sensor 132 receives ultrasonic waves having different phase differences depending on the distance relationship between the installed position and the signal transmitter 181.

  The measurement system position calculation unit 133 determines the signal transmitter 181 represented by measurement system coordinates (Xs, Ys, Zs) with the ultrasonic sensor 132 as the origin from the phase difference of the ultrasonic waves received by each ultrasonic sensor 132. Measurement system position information is calculated.

  The position measuring unit 104 transmits the identification code of the staff stored in the signal transmitter 181 worn by the staff whose existence has been confirmed, so that only the signal transmitter 181 is caused to respond to the ultrasonic wave. And the position of the signal transmitter 181 is calculated from the received ultrasonic wave, so that the position of the staff wearing the signal transmitter 181 whose position has been calculated is acquired in the measurement system coordinates.

  The transmitter measurement device 182 outputs the identification code for identifying the staff acquired by the identification unit 3 and the measurement system position information of the staff acquired by the measurement system position calculation unit 133 to the video processing device 105 in pairs.

  FIG. 16 is a block diagram illustrating a specific example of the configuration of the video processing device 5.

  The video processing apparatus 105 includes a video recording unit 151, a position storage unit 152, a video system position calculation unit 153, a video system position storage unit 154, a display control unit 155, and a staff database 156.

  The video recording unit 151 includes, for example, a magnetic storage circuit such as an HDD, a flash memory, and the like. The video recording unit 151 stores the video signal output from the photographing apparatus 102 for each video frame. When recording a video signal, time information indicating the time when the video frame was shot is attached to each video frame and stored.

  The position storage unit 152 includes a semiconductor storage circuit such as a RAM, for example. As illustrated in FIG. 17, the position storage unit 152 stores a pair of a staff identification code output from the transmitter measurement apparatus 82 and the staff measurement system position information. A pair of the staff identification code and the staff measurement system position information is stored with time information indicating the measured time.

  Based on the measurement system position information output from the transmitter measurement device 182, the video system position calculation unit 153 determines which pixel on the image the signal transmitter 181, that is, the staff wearing the signal transmitter 181 is reflected on. The video system position information to be represented is calculated. The video system position calculation unit 153 first converts staff measurement system position information output from the transmitter measurement apparatus 182 into imaging system coordinates (Xv, Yv, Zv) with the imaging apparatus 102 as the origin, and further the imaging system. Image system position information is acquired by calculating a pixel (Xn, Yn) onto which a space represented by coordinates is projected.

  The video system position storage unit 154 includes a magnetic storage circuit such as an HDD, a flash memory, and the like. As shown in FIG. 18, the video system position storage unit 154 stores the video system position information calculated by the video system position calculation unit 153. Further, when storing the video system position information, the video system position storage unit 154 associates the staff identification code and time information associated with the measurement system position information that is the calculation source with the video system position information of the calculation destination. Remember. Note that measurement system position information that is a calculation source may be stored as raw data in association with each other.

  The display control unit 155 superimposes the staff identification code stored in the position storage unit 152 or other staff identification information in addition to this in correspondence with the video system position information calculated by the video system position calculation unit 153. The generated video is generated and displayed on the display device 6.

  FIG. 19 is a flowchart showing the operation of the medical site display system 101.

  The imaging device 102 images a medical site (S01) and outputs an image of the medical site to the video processing device 105 (S02). The video processing device 5 attaches time information for each video frame to the video output from the photographing device 102 (S03), and stores the video in the video recording unit 151 in time series for each video frame (S04). ).

  In parallel with the imaging of the medical site by the imaging device 102, in the transmitter measuring device 182, the wireless transmission / reception unit 122 transmits a radio wave that generates an electromotive force to the signal transmitter 181 existing in the medical site (S05). This radio wave transmission is performed at regular intervals.

  When a radio wave is emitted from the transmitter measuring device 182, in the signal transmitter 181, the RF tag 121 receives the radio wave, supplies an electromotive force to the IC chip by a resonance action, and the staff of the staff stored in advance in the IC chip. The identification code is read and transmitted as a signal (S06).

  When the transmitter / receiver 122 receives the signal including the staff identification code, the transmitter measuring apparatus 182 temporarily stores the received staff identification code (S07). In the transmitter measuring device 182, when the staff identification code is acquired, the control signal transmission unit 183 transmits a control signal including the received staff identification code to the medical site (S08).

  In the signal transmitter 181, it is determined by the RF tag 121 whether or not the staff identification code included in the control signal is the same as the staff identification code stored in advance (S <b> 09), and is included in the received control signal. If the stuff identification code is not stored in the IC chip (S09, No), the control signal is discarded and the process is terminated. Transmission of radio waves by the transmitter measuring device 82 is repeated at predetermined intervals, and the processing from S05 to S09 is repeated each time.

  On the other hand, if the stuff identification code included in the received control signal is stored in the IC chip (S09, Yes), the RF tag 121 outputs a trigger signal to the ultrasonic oscillator 131 (S10). The ultrasonic oscillator 131 oscillates an ultrasonic signal triggered by the input of the trigger signal (S11). That is, the presence of the staff is identified by the transmission of the identification code of the staff by the signal transmitter 181. By using the identification code of the staff as a control signal, only the staff signal transmitter 181 whose existence is identified responds. An ultrasonic signal for position measurement is oscillated.

  In the transmitter measuring device 182, when the ultrasonic sensor 132 receives the ultrasonic signal emitted from the signal transmitter 181, the measurement system position calculator 133 attaches the signal transmitter 181 from the phase difference of the received ultrasonic wave. The measurement system position information of the staff is calculated (S12).

  Then, the transmitter measurement apparatus 182 outputs a pair of the staff identification code and the measurement system position information that are temporarily stored to the video processing apparatus 105 (S13).

  In the video processing apparatus 105, time information is attached to the pair of the input staff identification code and the measurement system position information (S14), and is stored in the position storage unit 52 in time series (S15).

  In the video processing apparatus 105, the video system position calculation unit 153 reads out the measurement system position information from the position storage unit 152, and the staff wearing the signal transmitter 181 uses the read measurement system position information as to which pixel on the video. (S16). Then, the video system position calculation unit 153 associates the identification code and time information of the staff associated with the measurement system position information of the conversion source with the video system position information, and stores the video system position information with the video system position storage unit. It memorize | stores in 154 (S17).

  When the video system position information is acquired, in the video processing device 105, the display control unit 155 sequentially reads video frames from the video recording unit 151, and the video system position information and the staff at the same time as the video frames are read from the video system position storage unit 154. The identification information is read out, and an image in which the staff identification code or other staff identification information is superimposed in correspondence with the pixel represented by the video system position information is generated (S18). The display control unit 155 outputs the generated video to the display device 106 and displays it on the display device 106 (S19).

  The above operations are sequentially repeated while the image capturing apparatus 102 captures an image and the signal transmitting body 181 and the signal receiving body 182 identify the staff and measure the position. Both data stored in the video recording unit 151 and the video system position storage unit 154 are stored in pairs in a nonvolatile memory such as an HDD so that the data can be reproduced at any time by the medical site display system 101. Alternatively, it is stored in a portable storage medium such as a DVD or HD DVD as a pair.

  The conversion from the measurement system position information expressed by the measurement system coordinates (Xs, Ys, Zs) to the video system position information expressed by the video system coordinates (Xn, Yn) by the video system position calculation unit 153 will be described in more detail. To do.

  FIG. 20 is a schematic diagram illustrating an installation state of the ultrasonic sensor 132 on the receiving side of the imaging apparatus 102 and the position measurement unit 104.

  Since the imaging device 102 and the ultrasonic sensor 132 are installed at different locations for physical reasons, the imaging system coordinates (Xv, Yv, Zv) of the space imaged by the imaging device 102 and the position measurement unit 104 measured. The coordinate system does not coincide with the staff measurement system coordinates (Xs, Ys, Zs).

  In order to calculate which pixel on the image captured by the imaging apparatus 102 corresponds to the position of the staff measured by the position measurement unit 104, the measurement system coordinates (Xs, Ys, Zs) and the imaging system coordinates (Xv) , Yv, Zv) and the relationship between the imaging system coordinates (Xv, Yv, Zv) and the video system coordinates (Xn, Yn) of the imaging device surface of the imaging device 102 must be known.

First, the measurement system coordinates s = (Xs, Ys, Zs) and the imaging system coordinates (Xv, Yv, Zv) are related by translation and rotation around the X, Y, and Z axes. Each can be represented by the following matrix.

Here, x0, y0, and z0 are parallel movement amounts.

Here, θx is a rotation angle around the X axis.

Here, θy is a rotation angle around the Y axis.

  Here, θz is the rotation angle around the Z axis.

From the above (Equation 1) to (Equation 4), the relationship between the measurement system coordinates (Xs, Ys, Zs) and the imaging system coordinates (Xv, Yv, Zv) is expressed as the following (Equation 5). Can do.

  In Equation (5), x0, y0, z0, θx, θy, and θz are unknown numbers, and if these values are known, the measurement system and the imaging system are completely associated with each other.

Further, the relationship between the imaging system coordinates (Xv, Yv, Zv) and the video system coordinates (Xn, Yn) is shown as in FIG. 21, where the pinhole of the imaging apparatus 102 is the origin of the imaging system, and the pinhole. When the distance between the image pickup device 102 and the image pickup device surface is Z0, the shooting system coordinates (Xv, Yv, Zv) and the video system coordinates (Xn, Yn) are expressed by the following (Equation 6) and (Equation 7). expressed.

  Here, for the sake of simplicity, an example of a pinhole camera has been described. However, even in the case of a more complicated video camera in which a plurality of lenses are combined, calculation is performed in consideration of the focal length and the like according to the lens system.

  The video system position calculation unit 153 calculates the video system coordinates (Xn, Yn) obtained from (Equation 5), (Equation 6), (Equation 7), and the distance Z0 in advance and the unknowns x0, y0, z0, θx, θy. , Θz is stored.

  When the medical site display system 101 is installed, a marker is placed in the medical site, and the position measurement and imaging are performed to acquire the measurement system coordinates and video system coordinates of the marker. From the relationship between the measurement system coordinates of the marker and the image system coordinates, the image system position calculation unit 153 obtains a relational expression E1 in which specific values of the unknowns x0, y0, z0, θx, θy, and θz are stored in advance. To obtain a relational expression E2 between the video system coordinates (Xn, Yn) and the measurement system coordinates (Xs, Ys, Zs). The relational expression E2 between the projection system coordinates (Xn, Yn) and the measurement system coordinates (Xs, Ys, Zs) and the measurement system position information of the staff obtained by the measurement by the position measurement unit 104 represent the measurement system. Using the specific coordinates (Xs, Ys, Zs), the staff position information on the video screen is calculated. In addition, since two expressions are obtained by measuring one marker (Xn =..., Yn =...), The markers are placed at three points and measured.

  An image displayed on the medical site display system 101 is shown in FIG.

  When the display control unit 155 reads a video frame to be displayed on the display device 106 from the video recording unit 151, the display control unit 155 also reads time information paired with the video frame. Then, the same time information as the read time information is searched from the video system position storage unit 154, and the video system position information and the staff identification code associated with the corresponding time information are read out. When the video system position information and the staff identification code are read, the display control unit 155 fills the pixels on the video frame corresponding to the read video system position information with an identification color such as black, and the read staff identification code. The character string to be represented is superimposed and displayed by pointing to the filled pixels.

  The display control unit 155 performs this staff identification code display process each time the next video frame is displayed on the display device 106. The display of the staff identification code is updated each time a new video frame is displayed. When there is no new video system position information associated with the time information paired with the video frame to be displayed in a certain signal transmitter 181, the pixel to be filled is maintained as it is. If there is new video system position information, when a new video frame is displayed, a new pixel corresponding to the video system position information is filled and a staff identification code indicating the pixel is superimposed and displayed. Unpaint the pixels that had been filled up to the video frame.

  FIG. 23 shows a modified example of the image displayed on the medical site display system 101. FIG. 24 shows the data structure of the staff database 156.

  As shown in FIG. 23, the display control unit 155 may display other identification information of the staff represented by the staff identification code in addition to the staff identification code.

  As shown in FIG. 24, the staff database 56 stores other identification information of the staff such as the staff's name, age, affiliation, duties, and years of in-service experience in association with the staff identification codes. The display control unit 155 searches the staff database 156 with the staff identification code read from the video system position storage unit 154, and acquires other identification information of the staff associated with the read staff identification code. The display control unit 155 causes the display device 6 to superimpose and display the acquired other identification information of the staff together with the identification code of the staff.

  Other identification information of the staff acquired from the staff database 156 depends on the operation signal input from the input device 107. When an operation for selecting the staff identification information to be displayed on the video using the input device 107 is performed, the display control unit 155 reads other identification information of the selected staff from the staff database 156 and displays it in a superimposed manner. If no identification information is selected, only the identification code of the staff is superimposed and displayed. When an operation that does not display the staff identification code using the input device 107 is performed, the display control unit 155 performs only the process of sequentially reading the video frames from the video recording unit 151 and displaying them on the display device 106.

  As described above, in the medical site display system 101, the staff present at the medical site is identified together with the existing position, and the existing position is plotted on the image obtained by photographing the medical site, and is associated with the plotted pixel. Identification information such as ID and name for identifying the staff was displayed. Thereby, the staff can be individually identified on the video, and it becomes easy to perform video analysis of who and what the staff performed for each staff.

  Further, since the staff is identified by the staff identification code and the position of the staff is measured, it is not necessary to determine who is wearing the signal transmitter 181 that oscillates the ultrasonic signal, and management is simplified. .

  Next, a medical site analysis system 109 that supports video analysis using video obtained by identifying and displaying the staff acquired by the medical site display system 101 will be described. FIG. 25 is a diagram illustrating a configuration of a medical site analysis system 109 that supports analysis of video recorded by the medical site display system 101.

  The medical site analysis system 109 acquires, from the video processing device 105, a pair of video and time information obtained by photographing the medical site, a set of a staff identification code, video system position information, and time information, Is calculated. This medical site analysis system 109 includes a medical site information storage unit 191, an ROI setting unit 192, a motion amount measurement unit 193, a motion amount storage unit 194, a queue setting unit 195, a queue index storage unit 196, and a cue control unit 197. Composed.

  The medical site information storage unit 191 includes a pair of video and time information obtained by photographing the medical site recorded by the medical site display system 101, a combination of a staff identification code, video system position information, and time information. Remember. The medical site display system 101 and the medical site analysis system 109 may be configured as an integrated system connected by a network, or may be configured as independent systems.

  The ROI setting unit 192 mainly includes a CPU. The ROI setting unit 192 stores a region of interest when the region of interest is designated on the screen displayed on the display device 106 using the input device 107.

  The motion amount measurement unit 193 is configured mainly including a CPU. The movement amount measuring unit 193 individually calculates the movement amount of each staff member shown in the video from the shape change of the staff image on the video. When the region of interest is set by the ROI setting unit 192, the motion amount measuring unit 193 determines whether or not the staff exists in the region of interest on the video, and when the staff exists in the region of interest. The amount of motion is calculated only.

  The motion amount storage unit 194 mainly includes a semiconductor storage circuit such as a RAM. The movement amount storage unit 194 stores the movement amount of each staff measured by the movement amount measurement unit 193 in time series.

  The queue setting unit 195 mainly includes a CPU. The queue setting unit 195 causes the queue index storage unit 196 to store the time at which the amount of movement indicating that the movement is particularly intense among the time series of the amount of movement of the staff is detected.

  The queue index storage unit 196 indexes a set of time information representing the time detected by the queue setting unit 95 in order of time series and stores it as a queue index for each staff member. Further, when the region of interest is set, the period in which the staff exists in the region of interest, which is the determination result of the motion amount measuring unit 193, is also included in the queue index and stored as the region-of-interest information.

  The cueing control unit 197 displays the cue index as a graph on the display device 106. When a cue mark in the cue index is designated using the input device 107, time information corresponding to the designated cue mark is output to the display control unit 155, and a video frame paired with this time information is output. The medical site information storage unit 191 is searched, and the cue is reproduced from the corresponding video frame. The cue mark is a mark indicating a time when the amount of motion detected by the cue setting unit 195 is intense.

  FIG. 26 is a flowchart showing the movement amount measurement and cue setting operations of the medical site analysis system 109.

  First, the ROI setting unit 192 stores the designated region of interest when an operation for designating the region of interest in the video is performed using the input device 107 (S31, Yes) (S32).

  The motion amount measuring unit 193 sequentially reads out the video frames (S33), and if the region of interest is stored (S34, Yes), whether the image region of the staff reflected in the read out video frame is included in the region of interest. It is determined whether or not (S35).

  If the image area of the staff is not included in the region of interest (S35, No), the movement amount measurement for the read video frame and the staff not included in the region of interest is terminated (S36).

  On the other hand, if the stuff image area is included in the region of interest (S35, Yes), the stuff image area included in the read video frame and the previous or subsequent video frame is subtracted. (S37). Then, as the amount of motion, the number of pixels in the stuffed image area that does not overlap the previous or subsequent video frame in the read video frame is counted (S38). When the movement amount is measured, the movement amount is stored in the movement amount storage unit 194 in time series for each staff member (S39).

  When the movement amount of all the staff members reflected in the video is measured for the read video frame (S40, Yes) and the movement amount is measured for all the video frames (S41, Yes), the motion amount measurement process is completed (S42). . If the movement amount of all staff is not calculated (S40, No), S34 to S39 are repeated for all staff. If the movement amount measurement has not been completed for the avant-garde frame (No in S41), S33 to S39 are repeated for the next video frame.

  When the movement amount for each staff is measured in time series, the queue setting unit 195 searches the movement amount stored in the movement amount storage unit 194 for a movement amount that represents the intensity of the predetermined or higher movement ( S43), the staff corresponding to the corresponding motion amount and the time when the motion occurred are read from the motion amount storage unit 194 and stored in the queue index storage unit 196 (S44).

  27 and 28 are schematic diagrams illustrating setting of a region of interest by the ROI setting unit 92 in S31 to S32. FIG. 27 is a schematic diagram showing a mode in which the region of interest is displayed on the display screen. FIG. 28 is a schematic diagram showing a mode of acquiring information indicating the range of the region of interest corresponding to the displayed region of interest.

  As shown in FIG. 27, the display screen of the display device 106 displays an image obtained by photographing a medical site. When the operator operates the cursor so as to draw one diagonal line on the display screen using the input device 107, the ROI setting unit 192 displays a rectangular region of interest having the diagonal line on the display screen. By drawing one diagonal line at a plurality of locations, a plurality of regions of interest having respective diagonal lines are displayed. Then, as shown in FIG. 28, the ROI setting unit 192 stores an address on the drawing memory corresponding to each pixel included in the displayed region of interest. The address on the drawing memory corresponding to each pixel constituting the region of interest indicates the range of the region of interest on the video.

  29 and 30 are schematic diagrams illustrating movement amount measurement by the movement amount measurement unit 193 in S33 to S39. FIG. 29 is a schematic diagram illustrating a process of extracting on the video a staff who is a target for measuring the amount of movement. FIG. 30 is a schematic diagram illustrating a process of measuring the amount of stuff movement from the stuff image area of each extracted video frame.

  As shown in FIG. 29 (a), each video frame includes an image area of a staff whose motion amount is to be analyzed. In this image area, a video as the location of the signal transmitter 181 is displayed. A pixel represented by the system position information is included. The motion amount measuring unit 193 reads from the medical site information storage unit 191 video system position information that is closest to the time attached to the read video frame and is paired with time information that represents the time before this time. If there is video system position information paired with the same time information as the time attached to the read video frame, the video system position information is read out. The video system position information is read out for each staff identification code stored in the medical site information storage unit 191.

  Then, as shown in FIG. 29B, the motion amount measuring unit 193 extracts the outline of the region including the pixel represented by the read video system position information. In the contour extraction, an edge region including a pixel represented by the video system position information is extracted from the edges extracted by the edge extraction method. Alternatively, it is extracted from the pixel represented by the position information on the image by the region expansion method. This extracted area corresponds to the staff image area.

  When the outline of the staff image area is extracted, the motion amount measurement unit 193 determines whether or not the extracted staff image area is included in the region of interest stored in the ROI setting unit 192. It may be determined that all of the staff image area is included in the region of interest, or it may be determined that a part of the staff image region is included in the region of interest good.

  In the case where the extracted staff image area is included in the range of the region of interest stored in the ROI setting unit 192, the time information attached to the extracted video frame and the region of interest The video system position information included in the image area of the staff included in the image and the identification information of the staff in a pair are associated with each other and held as the in-region information of interest. That is, the region-of-interest information indicates who was in the region of interest and when.

  When the extracted staff image area is included in the range of the region of interest stored in the ROI setting unit 192, the contour extracted staff image area is formed as shown in FIG. Addresses on the drawing memory corresponding to all pixels are temporarily stored.

  As shown in FIG. 30A, the motion amount measuring unit 193 also performs the extraction of the staff image area on the video frame before or after the video frame on which the motion amount is measured. Also for the video frames, addresses on the drawing memory corresponding to all the pixels constituting the staff image area are temporarily stored.

  When the staff's image area is specified for the video frame for which the motion amount is to be measured and the previous or subsequent video frame, the motion amount measuring unit 193, as shown in FIG. Subtraction of the stuff image area in the frame. By superimposing these two video frames, the number of pixels in the area of the stuff image area of the video frame whose movement amount is to be measured does not overlap with the stuff image area extracted in the previous or subsequent video frame. Count. That is, the addresses indicating the stuff image areas stored in the two video frames are compared, and the number of addresses that are not in the previous or subsequent video frame is counted. When the stuff image areas do not overlap at all in the two video frames due to the frame rate, the number of all pixels in the stuff image areas of the video frame that is the subject of motion measurement is counted. This counted number of pixels corresponds to the amount of motion.

  The motion amount measurement unit 193 stores the counted number of pixels in the motion amount storage unit 194 as a motion amount. At this time, the staff identification code corresponding to the image area of the staff whose movement amount is to be measured and the time information attached to the video frame whose movement amount is to be measured are associated with this movement amount. . In addition, the motion amount measuring unit 193 also stores the region-of-interest information in the motion amount storage unit 194.

  The motion amount measuring unit 193 calculates the motion amount for each video frame and for each staff identified as existing in the medical field, and stores the motion amount in the motion amount storage unit 194.

  FIG. 31 is a schematic diagram of a data configuration representing the amount of movement stored in the movement amount storage unit 194 in S39.

  As shown in FIG. 31A, the movement amount storage unit 194 stores a staff identification code, time information, and a movement amount as a set. That is, the movement amount of the staff at each time represented by the identification code is stored. When a plurality of imaging devices 102 are installed in the medical field, there are a plurality of types of images, and the motion amount measurement unit 193 captures each of the imaging devices 102 obtained by imaging. The amount of movement of the staff is measured for the video frame and stored in the movement amount storage unit 194.

  Further, as shown in FIG. 31 (b), the staff identification code and the time zone in the region of interest are stored in combination as the region-of-interest information.

  FIG. 32 is a schematic diagram illustrating a modified example of the motion amount measurement by the motion amount measurement unit 193.

  As shown in FIG. 32A, the stuff image area is strictly extracted by an area expansion method, an edge extraction method, or the like, and the location where the signal transmitter 181 is known to be surely present, that is, the measurement system. An area within a predetermined range from the pixel represented by the position information may be extracted as a staff image area. The motion amount measuring unit 193 stores information representing the predetermined range in advance, applies the information representing the predetermined range around the pixel represented by the measurement system position information, and regards it as an image region of the staff and extracts it.

  Then, as shown in FIG. 32 (b), the video frame that is the subject of measurement of the amount of motion is overlapped with the previous video frame, and the number of pixels in the portion where the staff image areas do not overlap is counted. And the amount of movement.

  FIG. 33 is a graph showing a mode in which the queue setting unit 195 stores the time when the movement is intense in S43 to S44.

  As shown in FIG. 33, the movement amount measuring unit 193 measures the movement amount of each staff existing in the medical field in time series. The queue setting unit 195 stores a motion amount threshold value in advance, and compares this threshold value with the motion amount at each time. When the queue setting unit 195 detects a motion amount that exceeds the threshold, the queue setting unit 195 reads the time information and the staff identification code corresponding to the motion amount from the motion amount storage unit 194, and reads the time information and the staff identification code. Is stored in the queue index storage unit 196.

  FIG. 34 is a data structure diagram showing a queue index stored in the queue index storage unit 196.

  As shown in FIG. 34, the queue setting unit 195 stores time information in a queue index stored in a storage area secured as the queue index storage unit 196. At this time, when the queue setting unit 195 reads the time information from the motion amount storage unit 194, the queue setting unit 195 also reads the identification code of the staff paired with the time information, and stores the identification code in association with the queue index.

  In addition to the time information and the staff identification code, the cue index includes properties including the amount of motion and the origin of the video frame that is the source of calculation, that is, the identification information of the photographing device 102 that has been photographed, the type of threshold, and the like. You may make it memorize | store in one flag.

  FIG. 35 is a graph showing a modification of the aspect in which the cue setting unit 195 stores the time of intense movement in S43 to S44.

  As shown in FIG. 35, the threshold value may be generated in accordance with an operation of the operator using the input device 107. From the input device 107, an operation signal for designating a range on the time axis for setting a threshold and a magnification is input. The queue setting unit 195 calculates an average value of the motion amount in the specified range on the time axis, and generates a threshold value by multiplying the average value by the specified magnification. The queue setting unit 195 uses the generated threshold value in a specified range on the time axis. When a plurality of ranges on the time axis are designated, a threshold value is generated for each range using a magnification designated for each range.

  In this modification, the queue setting unit 195 also reads time information when sequentially reading the motion amount, and uses the threshold value corresponding to the range on the time axis including the read time information, Compare

  FIG. 36 is a flowchart showing an operation of reproducing a video based on the movement amount measurement and the cue setting performed in the medical site analysis system 109.

  When an operation for reproducing a video is performed using the input device 107 (S51), the cueing control unit 197 reads the queue index from the queue index storage unit 196 (S52), and the time information included in the queue index And a graph corresponding to the identification information of the staff are created (S53), and this graph is displayed on the display control unit 155 (S54).

  When the cue mark on the graph is designated using the input device 107 (S55), the cue control unit 197 reads time information corresponding to the designated cue mark from the cue index storage unit 196 (S56), and displays it. The data is output to the control unit 155. The display control unit 155 reads a video frame associated with the input time information (S57), and starts a video that identifies and displays the staff from the video frame (S58).

  FIG. 37 is a schematic diagram showing a display mode of the cue index displayed on the display control unit 155 by the cueing control unit 197.

  As shown in FIG. 37, the cue control unit 197 generates a graph with the horizontal axis as the time axis and the vertical axis as the identification code of each staff from the queue index Qi stored in the queue index storage unit 196. It is displayed on the display device 106 in the form. The queue index Q is displayed side by side below the displayed video, for example.

  The cue control unit 197 reads the pair of the staff identification code and the time information stored in the queue index storage unit 196, and displays the cue mark Qm at a location corresponding to the read staff identification code and the time information. Let Also, the cue control unit 197 reads out the information in the region of interest, draws a thick arrow Ar at the location corresponding to the staff identification code and the distribution range of the time information included in the information in the region of interest, and exists in the region of interest The fact that it was done is displayed.

  In addition, when the movement amount is also stored in the queue index Qi, the cue control unit 197 adds up the movement amounts of all the staff members and the movement amounts of the staff members in the region of interest at every predetermined time interval. The total amount graph G in which the amount of movement is plotted on the time axis is also displayed.

  On the display screen displayed on the display device 106, for example, the thick arrow Ar of the camera assistant is interrupted at the time A, so that the camera assistant is out of the region of interest at the time A, that is, has left the position. Recognize. In addition, since the entire quantity graph G draws a severe waveform at time B, a cue mark Qm is displayed for each staff member, and a surgeon is newly added, some event that may lead to a medical accident occurs at this time B, It can be estimated that the surgeon rushed to support. In addition, the surgeon's thick arrow Ar is interrupted at time C, and the thick arrow Ar does not exist. Therefore, it is understood that the surgeon has left the medical field at time C, and the operation is largely completed. Is done.

  Therefore, by specifying the cue mark Qm on this time B and starting and reproducing the video from this time B, it is possible to verify some event. At this time, when observing the movement of the surgeon or surgeon on the video, it immediately recognizes on the video that the staff including the pixel pointed to by the identification information representing the surgeon or surgeon is the surgeon or surgeon. can do.

  As described above, when an event that may lead to an accident occurs in the medical field, the movement of the staff becomes intense in order to take an urgent action. For example, if a large bleeding occurs during surgery, the surgeon quickly sucks (in some cases, cleans), identifies the bleeding site, and stops bleeding. At that time, the necessary extra gauze is received by hand from the instrument table nurse. In addition, the anesthesiologist confirms whether blood pressure is decreasing, touches the patient directly, makes a diagnosis, removes the vasopressor from the drug cart, dispenses it as needed, and administers the tube. . Since these operations are often urgent, the staff moves more than during normal operations. Therefore, according to the medical field analysis system 109 according to this embodiment, in order to measure the amount of movement of the staff from the video recording, which time zone on the video should be verified based on the amount of movement This makes it easier to grasp and enables efficient video analysis.

  For example, the event identification system according to the first embodiment can be applied to the medical site analysis system described above as follows.

  That is, each of the staff involved in the work to be analyzed is attached with the contact microphone 3A and the transmitter 3B. And the estimation process of the event identification demonstrated in 1st Embodiment is performed about the acoustic data collected for every staff.

  Here, the queue setting unit 195 stores, in the queue index storage unit 196, the time when the movement amount indicating that the movement is particularly intense in the time series of the movement amount of the staff, as described above, The cue index storage unit 196 stores the time at which it is estimated that a predetermined event has occurred based on the result of analyzing the acoustic data collected by the contact microphone 3A.

  By adopting such a system configuration, according to the second embodiment, the same effect as that of the first embodiment can be obtained, and the working state can be based on information of various sensors collected by matching the time accurately. It is possible to provide an event identification system capable of extracting information indicating the above and estimating a significant part of a recording such as a video and adding a mark (so-called cue) based on the extracted information. In addition, it is possible to provide an event identification system that can identify who is the staff reflected in the image obtained by photographing the medical site and visually understand who is doing what. it can.

(Modification)
Furthermore, by extracting only the time zone in which the conversation was performed, a tape or the like in which only the time in which the conversation was performed during a series of work actions can be created. When a plurality of persons are wearing the contact type microphone 300A, the conversation by the plurality of persons is performed by extracting only the time zone in which all the contact type microphones 300A detect human voices. It is possible to create a tape or the like on which only the time is recorded. Further, the speaker in each utterance can be specified by the device number of the contact microphone 300A. Therefore, it is possible to immediately search and refer to only the time zone in which a specific speaker speaks among recorded video / audio.

  By the way, in a working conversation, the time from the completion of the first speaker's utterance to the start of the second speaker's utterance is often about 0.3 to 0.8 seconds. Thereafter, the first speaker responds more frequently to the utterance of the second speaker.

  Therefore, it is possible to estimate whether or not it is a conversation (acceptance answer) from the timing of utterances. Here, as a matter of course, a series of utterances (by a plurality of speakers) may be grouped and recognized as “conversations”, and marks (so-called queues) may be attached to the conversations.

  By using the mark attached to the conversation by such processing, it is possible to quickly search the conversation part from the recorded video / audio. Then, by reproducing only the conversation related to the search, it is possible to quickly and easily know problems that occurred during a series of work actions.

  When the event identification system according to the second embodiment is actually operated, it is necessary to accurately set the time in each contact microphone 300A.

[Third Embodiment]
The event identification system according to the third embodiment of the present invention will be described below. Although details will be described later, in the third embodiment, the event identification system according to the first embodiment is applied to an individual information identification system described later. Thereby, according to the third embodiment, staff positions, images, utterances, and heartbeat data can be easily grasped in a list along time series. For example, in a series of work activities such as surgery, To provide an event identification system that realizes when and who did what by identifying each person recorded in a video etc. and accurately and easily grasping the list in time series with a simple device configuration Can do. Details will be described below.

  FIG. 38 is a diagram illustrating a configuration example of the individual information identification system. As shown in FIG. 38, the individual information identification system includes an ID identification camera 1 and an overall device 6 shown in FIG.

  Note that the optical system shown in FIG. 38 is a simplified optical system for convenience in order to explain the principle of the ID identification camera 1. That is, in actual implementation, it is naturally preferable to add an optical system, a diaphragm, and the like that correct various aberrations as appropriate.

  As shown in the figure, the ID identification camera 1 includes a photographing means 2 for photographing a predetermined area and acquiring video information, and a spatial resolution for specifying a transmission place of an ID signal (details will be described later). And a receiving device 4 that receives the ID signal.

  The photographing unit 2 and the receiving device 4 are connected to the central device 6 that associates the video information acquired by the photographing unit 2 with the transmission location of the ID signal received by the receiving device 4. .

  An optical system 8 including a photographing lens 11 and a beam splitter 13 is provided in the photographing unit 2 and the receiving device 4 as an optical system for guiding a light beam from a subject to be photographed.

  The photographing means 2 includes, for example, a camera light receiving surface 2A that is an image sensor such as a CCD (Charge Coupled Device), and a camera control device (Camera) that converts an electric signal output from the camera light receiving surface 2A into a video signal and outputs the video signal. Control Unit; hereinafter referred to as CCU) 2B. The digital moving image (hereinafter simply referred to as video information) acquired by the photographing means 2 is a video as shown in FIG. 39, for example.

  The receiving device 4 includes a plurality of ID signal receiving devices 4A (described in detail with reference to FIG. 41). The ID signal light receiving surface 4A11 is an assembly of photodetecting devices for receiving ID signals as shown in FIG. 40, and a plurality of ID signal detecting surfaces 4A1 that are light receiving surfaces of each ID signal receiving device 4A are provided. It is configured by arranging.

  FIG. 40A shows a configuration in which the receiving device 4 is configured by a 9-channel ID signal receiving device 4A. In this way, the ID signal detecting surface 4A1 is arranged and the ID signal receiving surface 4A11 is arranged. Constitute. Here, the ID signal detection surfaces 4A1 for nine channels are arranged in a grid pattern, and channel numbers are assigned to the respective ID signal receiving devices 4A corresponding to the ID signal detection surfaces 4A1 as shown in FIG. 40 (b). An example is shown.

  The beam splitter 13 is for causing the light incident on the ID identification camera 1 through the photographing lens 11 to enter the camera light receiving surface 2A and the ID signal light receiving surface 4A11 so as to have the same optical axis. It is a member.

  In order to make the light incident on the ID identification camera 1 through the photographing lens 11 incident on the camera light receiving surface 2A and the ID signal light receiving surface 4A11 so as to have the same optical axis, a half mirror or the like is used. Of course, it may be used. In actual implementation, the light incident on the ID identification camera 1 through the photographing lens 11 is incident on the camera light receiving surface 2A and the ID signal light receiving surface 4A11 so that they have exactly the same optical axis. There is no need to make it incident, and it is sufficient to make it incident so as to have substantially the same optical axis.

  In this way, by making the light incident through the photographing lens 11 enter the camera light receiving surface 2A and the ID signal light receiving surface 4A11 so as to have the same or substantially the same optical axis, the overall device 6 Can accurately associate the position of each image on the video acquired by the photographing means 2 with the position of the transmission source of the ID signal received by the receiving device 4.

  Hereinafter, a configuration example of the ID signal receiving device 4A will be described. FIG. 41 is a diagram illustrating a configuration example of the ID signal receiving device 4A.

  As shown in the figure, the ID signal receiving device 4A includes an ID signal detection surface 4A1, a photosensitive element 4A2, a high-pass filter 4A3, a demodulator 4A4, a buffer memory 4A5, and an MPU 4A6.

  As the ID signal detection surface 4A1, for example, when an infrared light emitting diode is used as the light emitting element 21D in the ID signal transmission device 21 described later with reference to FIG. 42, an optical filter that blocks visible light is used. Thereby, the ID signal light and the visible light in the environment are separated. That is, the ID signal detection surface 4A1 is a member for separating the ID signal light and other light and extracting only the ID signal light.

  The photosensitive element 4A2 converts the optical signal that has passed through the ID signal detection surface 4A1 into a voltage signal.

  The high-pass filter 4A3 performs a process of blocking low-frequency components on the voltage signal output from the photosensitive element 4A2, and leaves only high-frequency components. This process eliminates the influence of the background and its slow fluctuation.

  The demodulator 4A4 reproduces and digitizes the waveform of the signal transmitted from the high-pass filter 4A3.

  The buffer memory 4A5 temporarily stores the signal digitized by the demodulator 4A4.

  The MPU 4A6 reads a signal from the buffer memory 4A5 and performs error detection by decoding processing. Here, if no error is detected, the ID information is extracted and the ID information is transmitted to the overall device 6.

  Hereinafter, with reference to FIG. 42, a configuration example of an ID signal transmission device that transmits an ID signal received by the reception device 4 will be described.

  As shown in the figure, the ID signal transmission device 21 includes a power source 21A, a ROM 21B, a driver circuit 21C, a light emitting element 2ID, and a scattering plate 21E.

  The power source 21 </ b> A supplies power necessary for the operation of the ID signal transmission device 21.

  The ROM 21B holds ID information converted into a predetermined light emission pattern by a driver circuit 21C described later. The ID information held by the ROM 21B is different from each other for each ID signal transmission device 21.

  The driver circuit 21C reads the ID information held in the ROM 21B, converts it into a light emission pattern of the light emitting element 2ID, and blinks the light emitting element 2ID. Therefore, the light emitting elements 2ID in the respective ID signal transmission devices 21 emit light with different temporal patterns. And each ID signal transmitter 21 transmits ID information as ID signal light by this light emission pattern.

  The scattering plate 21E is a member for transmitting the ID signal light to a wider space. That is, the scattering plate 21E substantially enlarges the light emitting area of the light emitting element 2ID, and allows the ID signal light to fly over a wider range of space. In other words, this scattering plate 21E reduces the probability that the ID signal light from the light emitting element 2ID is blocked by some obstacle. Therefore, for example, even when the light emitting element 2ID in the ID signal transmission device 21 is attached to the shoulder of the monitoring subject, communication is disabled due to the ID signal light being blocked by the monitoring subject's head or the like. Can be prevented.

  As described above, when an infrared light emitting diode is used as the light emitting element 2ID, for example, a filter that blocks visible light is used for the ID signal light receiving surface 4A11, so that the ID signal light and the visible light coming from the environment can be obtained. It can be easily separated.

  In such a configuration, the ID signal light can be transmitted even with a small output, and power for transmission can be reduced. Therefore, for example, a solar cell can be used as the power source 21A. In this case, battery replacement is not necessary.

  Furthermore, since the infrared rays can be transmitted if the cloth is thin and light in color, the ID signal transmission device 21 can be attached under the clothes of the person to be monitored. Therefore, for example, when it is necessary to wear clean clothes (for example, an operating room or a clean room), the ID signal transmission device 21 can be attached under the clothes without performing sterilization or cleaning. .

  Hereinafter, an example of the transmission timing of the ID signal light by the ID signal transmission device 21 will be described with reference to the timing shown in FIG.

  As shown in FIG. 43, when the ID signal transmission device 21 transmits the ID signal light, a transmission pause time is randomly provided. Specifically, as in the transmission timing shown in FIG. 43, transmission is performed by inserting a pause period of, for example, several tens of milliseconds to hundreds of tens of milliseconds without continuously transmitting.

  This prevents a situation in which the receiving device 4 cannot receive the ID information transmitted by the ID signal transmitting devices 21 due to the fact that the plurality of ID signal transmitting devices 21 always transmit simultaneously. To.

  In other words, by performing the ID signal light transmission timing control as shown in FIG. 43, while another ID signal transmission device 21 repeatedly executes the transmission of the ID signal light, another ID signal transmission device. 21 is suspended, and an event occurs in which the certain ID signal transmission device 21 transmits the ID signal light alone. At this time, the ID signal is correctly received by the receiving device 4.

  The ID signal light transmitted by the ID signal transmission device 21 is signal light obtained by encoding (encoding processing) ID information provided for each ID signal transmission device 21 with an error detection code. Since this ID signal light has a small amount of information, for example, several milliseconds are sufficient for transmission. Therefore, the transmission timing as shown in FIG. 43 is possible.

  Further, the power consumption of the ID signal transmission device 21 is greatly reduced by inserting a pause period of, for example, several tens of milliseconds to hundreds of tens of milliseconds without continuously transmitting like the transmission timing shown in FIG. This also has the effect of reducing it.

  As a result, when, for example, a battery is used as the power source 21A, it is possible to extend the time that can be continuously used without battery replacement. On the other hand, when, for example, a solar cell is used as the power source 21A, the area of the solar cell can be reduced to downsize the ID signal transmission device 21 and can be configured to be easily worn by the monitoring subject.

  44 is a diagram in which the video acquired by the photographing unit 2 and the emission position of the ID signal light are overlapped. In other words, FIG. 44 is a diagram showing the correspondence between the video acquired by the photographing means 2 and the ID signal light. Here, for example, when the ID signal light is an infrared ray, the ID signal light cannot be recognized on the video obtained by the photographing unit 2 as a matter of course.

  FIG. 45 is a diagram schematically showing ID signal light projected on the ID signal light receiving surface 4A11 in the example shown in FIG. In FIG. 45, the ID signal light receiving surface 4A11 corresponds to an ID signal receiving device 4A of 3 × 3 = 9 channels.

  Specifically, FIG. 45 shows positions where the ID signal light from the four ID signal transmission devices 21 is projected in total, and two ID signal lights are received in the upper left channel in FIG. . One of them is partially hidden by the head of the person wearing the ID signal transmission device 21 and is partially darkened.

  The ID signal light projected on the ID signal detection surface 4A1 of each channel is detected by the channel. However, in one channel, it is impossible to detect at which position in the channel the ID signal light is projected. However, it is of course possible to improve the spatial resolution of the receiving device by increasing the number of channels (increasing the number of ID signal receiving devices 4A).

  Although the details will be described later, the diagram shown in FIG. 45 is shown for convenience of explanation, and actually, a plurality of ID signal lights are not simultaneously received.

  Hereinafter, processing by the MPU 4A6 in the ID signal receiving apparatus 4A will be described with reference to a flowchart shown in FIG.

  First, it waits until a fluctuation component having a specific frequency is input (step S1). Subsequently, error detection is performed assuming that the input signal is an ID signal (step S2). Then, based on the result of error detection executed in step S2, it is determined whether or not there is an error (step S3). When step S3 is branched to YES, it is considered that the signal has not been received (step S5) and the process returns to step S1. On the other hand, when step S3 is branched to NO, ID identification information is extracted from the signal and transmitted to the overall device 6 (step S4), and the process returns to step S1.

  As described above, when the ID signal receiving device 4A fails to correctly receive the ID signal light or receives the ID signal light from the plurality of ID signal transmitting devices 21 at the same time, the ID signal light is transmitted. Is detected as an error and the ID signal light is not received. This eliminates the possibility of errors and collisions (confusion caused by simultaneous transmission of a plurality of ID signal lights).

  Hereinafter, processing by the overall device 6 will be described.

  The overall device 6 is constituted by a computer, and receives video information transmitted from the photographing means 2 in real time. Furthermore, the overall device 6 receives the ID identification information transmitted from all the ID signal receiving devices 4A and determines which ID signal receiving device (channel) 4A is the ID identification information from the ID signal receiving device 4A. The ID identification information is temporarily recorded in a memory provided in the overall device 6 together with when the ID identification information is obtained (time).

  Then, the latest position of the ID signal transmission device 21 is determined using these pieces of information (detailed with reference to FIG. 47). The determination result may be recorded, the determination result may be displayed superimposed on the video information, and a video created by this superposition may of course be recorded.

  Hereinafter, an example of a method for determining the latest position of the ID signal transmission device 21 by the overall device 6 will be described.

  FIG. 47A is a diagram showing channel numbers assigned to each ID signal detection surface 4A1, and FIG. 47B is a method for determining the latest position of the ID signal transmission device 21 by the overall device 6. It is a figure which shows an example.

  That is, the overall device 6 holds data indicating when the ID signal light from each ID signal transmission device 21 is detected by which ID signal detection surface 4A1 (channel) as shown in FIG. 47B, for example. Keep it. In FIG. 47B, the latest data to the data for the past 3 seconds are shown.

  And when the overall device 6 determines the position of each ID signal transmission device 21, for example, a detection record of ID signal light from each ID signal transmission device 21 as shown in FIG. The position of each ID signal transmission device 21 is determined by using the latest data in which the ID signal light from each ID signal transmission device 21 is detected in order from the latest data in time series.

That is, in the example shown in FIG.
The ID signal transmission device 21 with ID 129 was detected by the ID signal reception device 4A of channel 8.

ID ID transmitter 21 of ID026 was detected by ID signal receiver 4A of channel 1.

The ID signal transmission device 21 of ID315 was detected by the ID signal reception device 4A of channel 1.

The ID signal transmission device 21 of ID 108 was detected by the ID signal reception device 4A of channel 6.

  Is determined. When this determination result is expressed in a rectangle corresponding to the position of each channel shown in FIG. 47A, it is as shown in FIG. Further, when the image information acquired by the photographing means 2 is superimposed and expressed, for example, as shown in FIG.

  As shown in FIG. 49, by superimposing and displaying the video and the position of the ID identification information, it is possible to easily list “who is an individual person in the video”.

  Specifically, in the example shown in FIG. 49, the person wearing the ID 026 ID signal transmission apparatus 21 and the person wearing the ID 315 ID signal transmission apparatus 21 both receive the ID signal of channel 1. It is located within the detection range by the device 4A. For this reason, it is impossible to identify which person in the video is the person wearing the ID signal transmission device 21 of which ID number, but if these persons move over time, Identification becomes possible.

  FIG. 49 shows an example in which the receiving device 4 has nine ID signal receiving devices 4A, that is, an example in which the number of channels is nine. However, more ID signal receiving devices 4A are provided and channels are provided. By increasing the number and improving the spatial resolution, it is possible to increase the possibility that each person can be individually identified even when many people are densely located.

  FIG. 50 shows an example in which the name of the person wearing the ID signal transmission device 21 is superimposed on the image of the person on the video acquired by the photographing unit 2 instead of the ID number. .

  In order to display the name instead of displaying the ID number in this way, it is necessary to create a correspondence table (database) between the ID number of the ID signal transmission device 21 and the name of the person wearing the ID signal transmission device in advance. . For example, if each individual always wears a dedicated ID signal transmission device 21, the database may be created once.

  By the way, in the actual operation of the individual information identification system, it is more convenient for the person to be monitored to attach the ID signal transmission device 21 only when necessary, and the ID signal transmission device 21 may need to be broken or replaced. In view of this, the same person does not always wear the ID signal transmission device 21 to which the same ID number is assigned.

  In view of such circumstances, in order to create the database in actual operation, a correspondence table between the ID number assigned to each ID signal transmission device 21 and the name of the person wearing the ID signal is created at any time. There is a need to.

  Hereinafter, a registration device for easily creating a correspondence table between the ID number assigned to each ID signal transmission device 21 and the name of the person wearing the ID number will be described with reference to FIG.

  As shown in the figure, the registration device 31 includes an ID signal receiving device 4A and a magnetic card reading slot 31A. And the registration apparatus 31 performs the electromagnetic lock in the doorway to the area where the individual information identification system was applied, and its release.

  In actual operation, the person 41 to be monitored who wears the ID signal transmission device 21 on the shoulder or the like enters the area to which the individual information identification system is applied in order to release the electromagnetic lock of the entrance door, for example, When the magnetic card 33 such as an identification card is passed through the magnetic card reading slot 31A of the registration device 31 and the ID signal light from the ID signal transmission device 21 is received by the ID signal reception device 4A, the registration device 31 is electromagnetically The ID correspondence information created by associating the ID information from the ID signal transmitting device 21 detected by the ID signal receiving device 4A and the personal identification information (name, etc.) read from the magnetic card 33 while releasing the lock. , To the overall device 6.

  Based on this ID correspondence information, the overall device 6 wears each ID signal transmission device 21 based on only the ID information obtained from the ID signal transmission device 21 during the above-described monitoring target person identification process. Human names can be specified.

  As described above, according to the above-described individual information identification system, individual information that can identify a person or the like shown in an image captured and acquired by a monitoring camera simply by visually observing the image. An identification system can be provided.

  Specifically, according to the above-described individual information identification system, for example, when monitoring a scene where a plurality of people or devices work together while moving, such as an operating room or a factory, for example, only from the image of the monitoring camera. Even if it is not easy to identify what is a person or an object shown in the video, it is possible to clearly identify what a person or an object is in the video.

  The above-described event identification system according to the first embodiment can be applied to the individual information identification system described above, for example, as follows.

  That is, each of the staff involved in the work to be analyzed is attached with the contact microphone 3A and the transmitter 3B. Here, the ID signal transmission device 21 and the transmitter 3B are integrally configured. And the analysis process demonstrated in 1st Embodiment is performed about the acoustic data collected for every staff.

  By adopting such a system configuration, according to the third embodiment, staff positions, images, utterances, and heartbeat data can be easily grasped in a time-sequential manner, such as surgery. Thus, it is possible to provide an event identification system that realizes, with a simple device configuration, accurately and easily comprehending who has done what and when in a series of work actions.

  As mentioned above, although this invention was demonstrated based on 1st Embodiment thru | or 3rd Embodiment, this invention is not limited to embodiment mentioned above, A various deformation | transformation and application are within the range of the summary of this invention. Of course it is possible.

  Further, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and the effect described in the column of the effect of the invention Can be extracted as an invention.

1 is a block diagram showing a configuration example of an event identification system according to a first embodiment of the present invention. The figure which shows the example of 1 structure of an acoustic data collection means. The figure explaining the analysis process of the heartbeat data by the heartbeat data processing means. The figure which shows an example of the graph of the time change of an instantaneous heart rate. The figure which shows an example of the graph of the time change of an instantaneous heart rate. The figure which shows an example of the graph of the time change of an instantaneous heart rate. The figure which shows an example of the graph of the time change of the respiration frequency, and an example of the graph of the time change of the respiration depth. The figure which shows an example of the graph of the time change of an instantaneous heart rate. The figure which shows an example of the graph of the time change of the respiration stop frequency. The figure which shows an example of the graph of the time change of a frequency. The figure which shows an example of the graph of the time change of conversation frequency. The figure explaining the comprehensive determination process using the multivariate pattern recognition technique. The figure explaining the comprehensive determination process using the multivariate pattern recognition technique. The figure which shows an example of the data which can be acquired by the event identification system which concerns on 1st Embodiment of this invention. It is a figure which shows the function structure of the medical field display system which records and displays a medical field. It is a block diagram which shows the specific example of a structure of an identification part and a position measurement part. It is a block diagram which shows the specific example of a structure of an image processing apparatus. It is a figure which shows the identification of the staff obtained by measurement, and the information of a presence position. It is a figure which shows the information showing the position on the image | video of the staff obtained by coordinate transformation. It is a flowchart which shows operation | movement of a medical field display system. It is a schematic diagram which shows the installation state of the ultrasonic sensor which is a receiving side of an imaging device and a position measurement part. It is a figure which shows the relationship between an imaging | photography system coordinate and its video system coordinate. It is a figure which shows the image | video displayed with a medical field display system. It is a figure which shows the modification of the image | video displayed with a medical field display system. It is a figure showing the data structure of a staff database. It is a figure which shows the structure of a medical field analysis system. It is a flowchart which shows the operation | movement of movement amount measurement and queue setting of a medical field analysis system. It is a schematic diagram which shows the aspect which displayed the region of interest on the display screen. It is the schematic diagram which showed the aspect which acquires the information which shows the range of this region of interest corresponding to the displayed region of interest. It is the schematic diagram which showed the process which extracts the staff used as the object which measures movement amount on an image | video. It is the schematic diagram which showed the process of measuring the amount of movement of a staff from the image area | region of the staff of each extracted video frame. It is a schematic diagram of the data structure showing the motion amount memorize | stored in the motion amount memory | storage part. It is a schematic diagram which shows the modification of the motion amount measurement by a motion amount measurement part. It is a graph which shows the aspect which memorize | stores time with intense motion. It is a data structure figure which shows a queue index. It is a graph which shows the modification of the aspect which memorize | stores time with intense motion. It is a flowchart which shows the operation | movement which reproduces | regenerates an image | video based on a motion amount measurement and cue setting. It is a schematic diagram which shows the display mode of a queue index. The figure which shows the example of 1 structure of the individual information identification system which concerns on 3rd Embodiment of this invention. The figure which shows the example of 1 display of the image | video acquired by the imaging | photography means. (A) is a figure which shows one structural example of an ID signal light-receiving surface in case a receiver is comprised by 9-channel ID signal receiver. FIG. 7B is a diagram illustrating a configuration example in which ID signal detection surfaces of ID signal receiving apparatuses for nine channels are arranged in a lattice pattern and channel numbers are assigned. The figure which shows the example of 1 structure of ID signal receiver. The figure which shows the example of 1 structure of ID signal transmitter which transmits ID signal received by the receiver. The figure which shows the transmission timing of ID signal light by ID signal transmitter. The figure which showed the image | video acquired by the imaging | photography means, and the light emission position of ID signal light superimposed. The figure which shows typically the ID signal light projected on an ID signal light-receiving surface. The figure which shows the flowchart of the process by MPU in an ID signal receiver. (A) is a figure which shows the channel number provided to each ID signal detection surface. (B) is a figure which shows an example of the method of determining the newest position of the ID signal transmitter by a supervising apparatus. FIG. 11 is a diagram illustrating a determination result by the overall device expressed in a rectangle corresponding to the position of each channel illustrated in FIG. The figure which superimposed and showed the image | video acquired by the imaging | photography means on the figure shown in FIG. The figure which showed the example which superimposes and displays on the image of the said person on the image | video acquired by the imaging | photography means the name of the person who is mounting | wearing with the ID signal transmission apparatus instead of ID number. The figure which shows the structure of the registration apparatus for producing easily the correspondence table of the ID number provided to each ID signal transmission apparatus, and the name of the person who wears it.

Explanation of symbols

    DESCRIPTION OF SYMBOLS 1 ... ID identification camera, 2 ... Imaging | photography means, 2A ... Camera light-receiving surface, 2B ... Camera control apparatus, 2ID ... Light-emitting element, 3 ... Identification part, 3A ... Contact-type microphone, 3B ... Transmitter, 4 ... Receiver, 4A ... ID signal receiving device, 4A11 ... ID signal light receiving surface, 4A1 ... ID signal detecting surface, 4A2 ... Photosensitive element, 4A3 ... High pass filter, 4A4 ... Demodulator, 4A5 ... Buffer memory, 4A6 ... MPU, 5 ... Video processing device, DESCRIPTION OF SYMBOLS 11 ... Shooting lens, 13 ... Beam splitter, 21 ... ID signal transmission device, 21D ... Light emitting element, 21A ... Power source, 21B ... ROM, 21C ... Driver circuit, 21E ... Scattering plate, 31 ... Registration device, 31A ... Taking slot, 33 ... Magnetic card, 41 ... Person to be monitored, 52 ... Location memory, 56 ... Staff day Base: 81: Signal transmitter, 82: Transmitter measuring device, 92: ROI setting unit, 95: Queue setting unit, 96: Cue index storage unit, 101: Medical field display system, 102: Imaging device, 103: Identification unit 104 ... Position measurement unit 105 ... Video processing device 106 ... Display device 107 ... Input device 109 ... Medical field analysis system 121 ... RF tag 122 122 Wireless transmission / reception unit 131 ... Ultrasonic oscillator 132 ... Ultra Sonic sensor 133 ... measurement system position calculation unit 151 ... video recording unit 152 ... position storage unit 153 ... video system position calculation unit 154 ... video system position storage unit 155 ... display control unit 156 ... staff database, 181 ... Signal transmitter, 182 ... Signal receiver, 183 ... Control signal transmitter, 191 ... Medical site information Memory unit, 192 ... ROI setting unit, 193 ... Movement amount measurement unit, 194 ... Motion amount storage unit, 195 ... Cue setting unit, 196 ... Cue index storage unit, 197 ... Control unit, 300 ... Acoustic data collection means, 300A ... Contact type microphone, 300C ... receiver, 300B ... transmitter, 500 ... filter means, 700A ... heartbeat data processing means, 700B ... heartbeat removal data processing means.

Claims (9)

An event identification system for recordably recording a time when a predetermined event occurs in a series of work actions,
Acoustic data collection means for collecting acoustic data including heartbeat sounds of persons involved in the series of work actions;
Heartbeat data extraction means for extracting data relating to the heartbeat from the acoustic data collected by the acoustic data collection means;
Respiration information generating means for generating respiration information related to respiration of a person involved in the series of work actions based on the heartbeat data extracted by the heartbeat data extracting means;
Event identifying means for determining and recording the time at which the predetermined event occurred in the series of work actions based on the respiratory information generated by the respiratory information generating means;
An event identification system comprising: Heartbeat sound removal data extraction means for extracting data other than data relating to the heartbeat sound from the acoustic data collected by the acoustic data collection means;
Determination means for determining a time at which the predetermined event occurs in the series of work actions based on the data extracted by the heartbeat removal data extraction means;
The event identification system according to claim 1, comprising:   The event identification system according to claim 1, wherein the acoustic data collection unit includes a contact microphone.   The event identification system according to claim 1, wherein the heartbeat data extraction unit extracts the heartbeat data from the acoustic data by a predetermined filter process.   The event identification means determines whether or not a breathing stop, an utterance, and an action of a person involved in the series of work actions have occurred based on the breathing information generated by the breathing information generation means. The event identification system according to claim 1.   The event identification system according to claim 1, wherein the respiration information generated by the respiration information generation unit includes information indicating a temporal change in instantaneous heart rate.   The determination means determines whether or not a person involved in the series of work actions has stopped breathing, uttered, or moved based on the data extracted by the heartbeat sound removal data extraction means. The event identification system according to claim 2. On-site information storage means for storing time-series video images of a scene where a series of work activities are performed,
Acoustic data collection means for collecting acoustic data including the staff's heartbeat;
Heartbeat data extraction means for extracting data relating to the heartbeat from the acoustic data collected by the acoustic data collection means;
Respiration information generating means for generating respiration information related to breathing of the staff based on the heartbeat data extracted by the heartbeat data extracting means;
Event identifying means for determining a time when a predetermined event occurs in the series of work actions based on the respiratory information generated by the respiratory information generating means;
A queue index storage means for storing a time when the event identification means determines that the predetermined event has occurred;
A cue display control means for displaying a cue mark corresponding to the time stored by the cue index storage means and displaying the video from the time corresponding to the cue mark designated by the operator;
An event identification system comprising: An event identification system for recordably recording a time when a predetermined event occurs in a series of work actions,
Photographing means for photographing the site where the series of work actions are performed and obtaining video information;
A transmitter that is attached to a person involved in the series of work actions and transmits an individual identification signal that is an individual identification information for identifying an individual;
Receiving means for receiving an individual identification signal transmitted from the transmission device, having a spatial resolution for specifying a transmission location of the individual identification signal;
Generalization means for associating the video information acquired by the imaging means with the transmission location of the individual identification signal received by the reception means;
A registration device for associating the individual identification information given to each of the transmission devices with personal identification information about a person wearing the transmission device;
Acoustic data collection means for collecting acoustic data including heartbeat sounds of persons involved in the series of work actions;
Heartbeat data extraction means for extracting data relating to the heartbeat from the acoustic data collected by the acoustic data collection means;
Respiration information generating means for generating respiration information related to respiration of a person involved in the series of work actions based on the heartbeat data extracted by the heartbeat data extracting means;
Event identifying means for determining the time at which the predetermined event occurred in the series of work actions based on the respiratory information generated by the respiratory information generating means;
An event identification system comprising:

Images