JP6726067B2 - Action recording system, terminal device and action recording method - Google Patents

Description

The present invention relates to an action recording system, a terminal device, and an action recording method.

As a technique for monitoring the state of a person in a predetermined room, a care system described in Patent Document 1 and a monitoring device described in Patent Document 2 are known. These Patent Documents 1 and 2 propose a monitoring technique in consideration of the privacy of the monitoring target person.
The care system described in Patent Document 1 has a camera device installed in a care room and a monitor capable of displaying the state of the care receiver. The characteristic part of the cared person's body is analyzed from the image captured by the camera device, and the comparison result between the analyzed characteristic part and the pre-registered characteristic part is displayed on the monitor instead of the captured image.
The monitoring device described in Patent Document 2 is a person image in which a portion having a predetermined temperature or higher is extracted from an image captured by a thermography camera installed in a patient room, and a background image captured by a normal camera in an unattended patient room. And are combined to generate a monitoring image.

JP, 2014-007685, A JP, 2007-089727, A

In recent years, the assault of a care giver by a staff member has become a social problem in nursing homes and the like. Such violent acts may be confirmed from the images of surveillance cameras installed in the facility.
However, for example, when the privacy-conscious monitoring is performed as described above, or when the assault is performed at the blind spot of the surveillance camera, the assault may not be confirmed.
It is an object of the present invention to prevent an assault of a facility staff member against a user of the facility and to enable the proof of the innocence of the staff member in the unlikely event of an accident.

According to an aspect of the present invention, there is provided an action recording system including a communication device installed in a predetermined area and a terminal device wearable by a person.
The terminal device includes a photographing unit that photographs the surroundings of the terminal device, a sound detecting unit that detects a sound signal, a communication unit that communicates with the communication device, and a photographing unit based on communication between the communication unit and the communication device. Before and after the human body detection unit detects the human body during the shooting by the shooting unit, the shooting control unit that controls the shooting by the sound detection unit and the sound detection by the sound detection unit, the human body detection unit that detects the human body in the image shot by the shooting unit. And an abnormality determination unit that determines that the acoustic detection unit is abnormal when the acoustic detection unit does not detect an acoustic signal at a predetermined level or higher.

The abnormality determination unit is configured to detect an acoustic signal when the acoustic detection unit detects an acoustic signal of a second predetermined level or higher, which is higher than a predetermined level, during a predetermined period including before and after the human body detection unit detects the human body during image capturing by the imaging unit. It may be determined that the detector is abnormal.

The abnormality determination unit does not include a vowel component in the acoustic signal even when the acoustic detection unit detects an acoustic signal of a predetermined level or higher during a predetermined period including before and after the human body detection unit detects the human body during imaging by the imaging unit. In this case, it may be determined that the sound detector is abnormal.

According to another aspect of the present invention, a terminal device that can be worn by a person is provided. The terminal device includes a photographing unit that photographs the surroundings of the terminal device, an acoustic detection unit that detects acoustic signals, a communication unit that communicates with a communication device installed in a predetermined area, and communication between the communication unit and the communication device. Based on the above, a shooting control unit that controls shooting by the shooting unit and sound detection by the sound detection unit, a human body detection unit that detects a human body shown in an image shot by the shooting unit, and a human body detection unit during shooting by the shooting unit. An abnormality determination unit that determines that the acoustic detection unit is abnormal when the acoustic detection unit does not detect an acoustic signal of a predetermined level or higher during a predetermined period including before and after detecting a human body.

According to another aspect of the present invention, a method for recording a worker's behavior is provided. In the action recording method, communication is performed between a terminal device that can be worn by a person and a communication device installed in a predetermined area, and based on communication between the terminal device and the communication device, a photographing unit included in the terminal device is used. The human body detection unit detects the human body in the image captured by the imaging unit by controlling the imaging of the surroundings of the terminal device and the detection of the acoustic signal by the acoustic detection unit included in the terminal device, and the human body detection unit detects the human When the sound detection unit does not detect a sound signal of a predetermined level or higher within a predetermined period including before and after the detection of, the sound detection unit is determined to be abnormal.

According to the present invention, it is possible to prevent an assault of a facility staff member against a user of the facility and prove the innocence of the staff member in the event of an accident.

It is a schematic block diagram which shows the whole action recording system of embodiment. (A)-(c) is explanatory drawing of the example of the mounting method of a portable terminal device, respectively. It is a figure which shows the structural example of the portable terminal device of embodiment. (A) is a figure which shows the example of the picked-up image image|photographed with the camera of a portable terminal device, (b) is a figure which shows the example of the mosaic process given to a picked-up image. It is a figure which shows the structural example of the server apparatus of embodiment. It is a flowchart which shows an example of the action recording method of embodiment. It is a flow chart which shows an example of wearing state judging processing. It is a flow chart which shows an example of sound detection state judging processing.

Embodiments of the present invention will be described below with reference to the drawings. The embodiments of the present invention described below exemplify devices and methods for embodying the technical idea of the present invention, and the technical idea of the present invention is the material, shape, and The structure, arrangement, etc. are not limited to the following. Various changes can be added to the technical idea of the present invention within the technical scope defined by the claims described in the claims.

(Overall structure of behavior recording system)
Please refer to FIG. In this specification, a case will be described in which the behavior of a staff member who is engaged in a nursing care service is recorded as a behavior record. However, the behavior recording system of the present invention is not limited to the system that records the behavior of the staff member of the care facility, but can be widely applied to the system that records the behavior of the worker who is engaged in the work in the predetermined area.

The action recording system is attached to a communication device 2 installed in a living room 1 of a nursing care facility, which is a predetermined area, and a staff 4 of a nursing care facility who cares for a resident 3 who is a care receiver receiving nursing care service in the living room 1. The mobile terminal device 5 is provided. The action recording system also includes a server device 7 installed in the monitoring center and connected to the mobile terminal device 5 through the communication line 6.
The mobile terminal device 5 includes a camera that captures an image of the periphery of the mobile terminal device 5 (for example, the front of the staff member 4) and a sound detection unit that detects a sound signal. The behavior recording system records the behavior of the staff member 4 by recording and recording the image and conversation of the resident 3 while the staff member 4 is in the living room 1.

In this way, the privacy of the resident 3 is protected by monitoring the staff 4 only during the period when the staff 4 is in the living room 1, and the behavior of the staff 4 is recorded, and the assault of the staff 4 on the resident 3 is performed. The act can be prevented. Furthermore, the innocence of the staff 4 can be proved in the unlikely event of an accident.
Moreover, since the image of the resident 3 is taken by the mobile terminal device 5 attached to the staff 4, unlike the fixed surveillance camera, the resident 3 supported by the staff 4 enters the blind spot of the camera. Hateful. Therefore, it is possible to prevent violent acts from being performed by the blind spot of the camera.

Next, each of the components of the action recording system will be described.
(Communication device)
The communication device 2 is installed on a wall surface, a ceiling, or the like in each predetermined area such as the living room 1 of the resident 3 or other common rooms. The communication device 2 transmits an identification signal (ID signal) unique to the communication device 2, for example, a beacon signal to the surroundings of the communication device 2 in a predetermined cycle according to a short-range wireless communication standard such as BLE (Bluetooth Low Energy). The communication device 2 may be an active RFID (Radio Frequency IDentification) tag that has a power source and periodically transmits a beacon signal. Such an active RFID tag includes, but is not limited to, iBeacon (trademark) compliant with Bluetooth (registered trademark) LE. Further, it is desirable that the living room or the like is electromagnetically isolated so that the beacon signal cannot be received outside the area where the communication device 2 is installed. Hereinafter, in this specification, a case where the communication device 2 transmits a beacon signal will be described.

(Mobile terminal device)
The mobile terminal device 5 is attached to the body of the staff member 4 at the time of use, and the camera of the mobile terminal device 5 captures an image of the surroundings of the mobile terminal device 5 so that the resident 3 is viewed from the position of the staff member 4. Used for. For example, the mobile terminal device 5 is attached to the staff member 4 such that the camera captures the front of the staff member 4 (that is, the field of view of the camera includes the front portion of the staff member 4). Thereby, the resident 3 who the staff 4 corresponds to can be photographed by the camera.
As shown in (a) of FIG. 2, the staff member 4 may wear the mobile terminal device 5 on a string that is hung from the neck. The mobile terminal device 5 may be provided with a wearing tool to be worn by the staff member 4, or may be housed in a dedicated case having such a wearing tool.

Alternatively, the staff member 4 may attach the mobile terminal device 5 to another part and attach the camera unit for transmitting an image to the mobile terminal device 5 to a string which is lowered from the neck. As shown in FIG. 2B, an external camera 12 such as a headset camera may be attached to the head.
Alternatively, as shown in (c) of FIG. 2, the uniform of the staff member 4 may be provided with a mounting space for housing the mobile terminal device 5 or an external camera.

Please refer to FIG. The mobile terminal device 5 includes an operation unit 10, a display unit 11, a camera 12, a first communication unit 13, a second communication unit 14, an alarm output unit 15, a sound detection unit 16, and a control unit 17. , A storage device 18.
The operation unit 10 functions as an information input unit, and may include an input button, a touch panel provided on the display screen of the display unit 11, and the like. The operation unit 10 is used to receive information input and operation input to the mobile terminal device 5 by the staff member 4. For example, the operation unit 10 receives an input of the operation of the staff member 4 that stops the shooting by the camera 12.

The display unit 11 functions as a display unit and displays various operation screens and information under the control of the control unit 17. The operation screen of the camera 12, for example, is displayed on the display unit 11. On the operation screen of the camera 12, for example, a shooting start button for starting shooting by the camera 12, a shooting stop button for stopping shooting by the camera 12, and the like may be displayed.
The camera 12 may be a known built-in camera or an external camera formed by including an image pickup device such as a CCD device or a C-MOS device, an optical system component, an A/D converter, and an encoder. The camera 12 outputs the captured image to the control unit 17.

The first communication unit 13 includes an interface circuit for performing wireless communication according to a short-range wireless communication standard such as BLE (Bluetooth Low Energy). The short-range wireless communication range may be several meters to several tens of meters. As a result, the first communication unit 13 receives the beacon signal transmitted from the communication device 2.
The second communication unit 14 is a circuit that includes a wired or wireless communication interface and that performs wired or wireless communication with the server device 7 via the communication line 6.

The alarm output unit 15 outputs an alarm by a visual signal or an acoustic signal under the control of the control unit 17. The alarm output unit 15 may be, for example, a buzzer or a speaker that outputs an acoustic signal, or a display device or a lamp that outputs a visual signal.
The sound detection unit 16 detects a sound signal around the mobile terminal device 5. The sound detector 16 may be a built-in microphone or an external microphone. The sound detection unit 16 outputs the detected sound signal to the control unit 17.

The control unit 17 may include a processor such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit) for controlling the operation of the mobile terminal device 5. The control unit 17 may be realized by a functional logic circuit set in a general-purpose semiconductor integrated circuit. For example, the control unit 17 may include a programmable logic device (PLD: Programmable Logic Device) such as a field programmable gate array (FPGA).

The control unit 17 includes a communication processing unit 20, an entry determination unit 21, an imaging control unit 22, an alarm control unit 23, a human body detection unit 24, and an abnormality determination unit 25. The control unit 17 executes the computer program stored in the storage device 18 to execute the communication processing unit 20, the entry determination unit 21, the photographing control unit 22, the alarm control unit 23, the human body detection unit 24, and the abnormality described below. The processing of the determination unit 25 is performed. That is, the computer program includes an instruction that causes the control unit 17 to perform the processes of the communication processing unit 20, the entry determination unit 21, the imaging control unit 22, the alarm control unit 23, the human body detection unit 24, and the abnormality determination unit 25.

The control unit 17 does not have to have all the functions of the communication processing unit 20, the entry determination unit 21, the imaging control unit 22, the alarm control unit 23, the human body detection unit 24, and the abnormality determination unit 25, and the server device 7 is not necessary. May have some of these functions.
The storage device 18 may include any one of a semiconductor storage device, a magnetic storage device, and an optical storage device. The storage device 18 may include a memory such as a register, a cache memory, a ROM (Read Only Memory) and a RAM (Random Access Memory) used as a main storage device.

The communication processing unit 20 performs communication processing with the server device 7 via the second communication unit 14. When the captured image by the camera 12 and the acoustic signal detected by the acoustic detection unit 16 are input to the control unit 17, the captured image and acoustic signal are not stored in the mobile terminal device 5 and are not stored by the communication processing unit 20. It is transmitted to the server device 7.
As shown in FIG. 4A, the image 30 captured by the camera 12 may include a video 31 of the resident 3. Similarly, the acoustic signal detected by the acoustic detector 16 may include the utterance of the resident 3.

By not storing the captured image and the audio signal in the mobile terminal device 5, it is possible to prevent the information from being illegally taken out and leaked from the mobile terminal device 5. As a result, the privacy of the resident 3 is protected.
Note that the communication processing unit 20 may perform mosaic processing on the portion of the image 31 of the resident 3 in the captured image 30 and then transmit it to the server device 7 as shown in FIG. 4B for privacy protection. The acoustic signal detected by the acoustic detector 16 may be subjected to processing such that the speaker cannot be identified.

Please refer to FIG. When the first communication unit 13 receives the beacon signal periodically transmitted from the communication device 2, the communication processing unit 20 sends the identification information (ID information) specific to the communication device 2 included in the beacon signal to the server device 7. Send. At this time, the entry determination unit 21 determines whether the staff member 4 wearing the mobile terminal device 5 is in the living room 1 depending on whether or not the first communication unit 13 receives the beacon signal periodically transmitted from the communication device 2. You may decide whether you have entered the room. That is, the entry determination unit 21 may determine that the staff member 4 has entered the living room 1 when the first communication unit 13 can communicate with the communication device 2. The communication processing unit 20 transmits the ID information of the communication device 2 to the server device 7 when the entry determination unit 21 determines that the staff member 4 has entered the living room 1.
The communication processing unit 20 receives the shooting permission signal and the shooting stop prohibition signal transmitted from the server device 7 in response to the ID information.

The shooting control unit 22 controls the start and stop of shooting by the camera 12. When it is determined that the staff member 4 has entered the living room 1 based on the reception of the beacon signal, and the communication processing unit 20 receives the shooting permission signal from the server device 7, the shooting control unit 22 causes the shooting by the camera 12 and the sound detection unit 16. Start sound detection. That is, when the first communication unit 13 can communicate with the communication device 2, the shooting control unit 22 starts shooting by the camera 12 and sound detection by the sound detection unit 16. In other words, the shooting control unit 22 controls the shooting by the camera 12 and the detection of the acoustic signal by the sound detection unit 16 based on the communication between the first communication unit 13 and the communication device 2.

Further, when it is determined that the staff member 4 has entered the living room 1 based on the reception of the beacon signal, and the communication processing unit 20 receives the photographing stop prohibition signal from the server device 7, the photographing control unit 22 prohibits the camera 12 from stopping the photographing. To do. That is, when the first communication unit 13 can communicate with the communication device 2, the shooting controller 22 prohibits the camera 12 from stopping shooting. In other words, the shooting control unit 22 prohibits the camera 12 from stopping shooting based on the communication between the first communication unit 13 and the communication device 2.

For example, the shooting control unit 22 may invalidate the operation of the shooting stop button on the operation screen of the camera 12 displayed on the display unit 11. Further, for example, the shooting control unit 22 may prohibit the display itself of the shooting stop button.
By prohibiting the staff member 4 from stopping the shooting in the living room 1, it is possible to prevent the staff member's 4 behavior record from being illegally interrupted, and during that time, the staff member 4 being assaulted on the resident person 3. ..

When the staff member 4 leaves the living room 1 and the first communication unit 13 no longer receives the beacon signal of the communication device 2, the entry determination unit 21 judges that the staff member 4 has left the living room 1. That is, the entry determination unit 21 determines that the staff member 4 has left the living room 1 when the first communication unit 13 cannot communicate with the communication device 2.
When it is determined that the staff member 4 has left the living room 1 based on the reception of the beacon signal, the shooting control unit 22 stops the shooting by the camera 12. That is, the shooting control unit 22 stops the shooting by the camera 12 when the first communication unit 13 cannot communicate with the communication device 2. The photographing control unit 22 may stop the photographing by the camera 12 when the first communication unit 13 cannot communicate with the communication device 2. For example, the image capturing control unit 22 may stop the image capturing by the camera 12 when a predetermined time elapses after the first communication unit 13 cannot communicate with the communication device 2.

Further, when it is determined that the staff member 4 has exited the living room 1 based on the reception of the beacon signal, the shooting control unit 22 cancels the prohibition of stopping the shooting by the camera 12.
Further, while the staff member 4 is outside the living room 1 and does not receive the beacon signal from the communication device 2, the imaging control unit 22 follows the operation performed on the operation screen of the camera 12 displayed on the display unit 11 according to the operation of the camera 12 To start and stop shooting. As a result, the staff member 4 can manually operate the camera 12 while the staff member 4 is outside the living room 1.

Due to a failure of the image capturing control unit 22 or the camera 12, the image capturing by the camera 12 may not be automatically stopped even when the employee 4 exits the living room 1 after the working period of the employee 4 ends. If the shooting by the camera 12 does not stop even after the work period of the staff member 4 ends, the privacy of the staff member 4 may be infringed.
Therefore, the alarm control unit 23 generates a predetermined alarm when the first communication unit 13 cannot communicate with the communication device 2 and the image capturing by the camera 12 continues even after a predetermined time elapses. For example, the alarm controller 23 may output an alarm sound or an alarm message from the alarm output unit 15, which is a buzzer or a speaker. The alarm control unit 23 may output a visual alarm display by the alarm output unit 15 which is a display device or a lamp.

If the mobile terminal device 5 is not properly attached, the staff member 4 cannot capture the image of the resident 3 who the staff member 4 corresponds to, so that the behavior of the staff member 4 cannot be recorded. Therefore, the abnormality determination unit 25 determines an abnormality in the mounting state of the mobile terminal device 5. The abnormality of the mounting state of the mobile terminal device 5 includes the abnormality of the mounting state of the mobile terminal device 5 itself and the abnormality of the mounting state of the camera 12 when an external camera is used as the camera 12.

Therefore, the human body detection unit 24 detects the human body shown in the image captured by the camera 12. The abnormality determining unit 25 determines whether the staff 4 has entered the living room 1 by the entrance determining unit 21 and the camera 12 is capturing an image of the human body while the human body detecting unit 24 detects the human body. An abnormality in the mounting state of the device 5 is determined.
This does not mean that the staff 4 does not look at the resident 3 at all even though the staff 4 is in the living room 1 (that is, the staff 4 does not look at the resident 3 at all). Based on the finding that it is natural.

The abnormality determination unit 25 determines, according to the length of a period in which the human body detection unit 24 does not detect a human body in a state where the room entry determination unit 21 determines that the staff member 4 has entered the living room 1 and the camera 12 is capturing an image. The abnormality of the mounting state of the mobile terminal device 5 may be determined. For example, when the abnormality determination unit 25 determines that the staff 4 has entered the living room 1 by the entry determination unit 21 and the camera 12 is capturing an image, the human body detection unit 24 does not detect the human body continuously for a predetermined period or longer. First, it may be determined that the wearing state is abnormal.
The abnormality determination unit 25 notifies the server device 7 of an abnormality in the mounting state of the mobile terminal device 5 via the communication processing unit 20.

Further, the abnormality determination unit 25 determines that the staff 4 has entered the living room 1 by the entry determination unit 21 and the camera 12 is capturing an image of the human body detection unit 24 in accordance with the frequency at which the human body detection unit 24 detects a human body. The abnormality of the wearing state of No. 5 may be determined. For example, in the state where the entry determination unit 21 determines that the staff member 4 has entered the living room 1 and the camera 12 is capturing an image, the abnormality determination unit 25 determines that the number of times the human body detection unit 24 has detected a human body within a predetermined period is equal to or less than a threshold value. In this case, it may be determined that the mounting state is abnormal.

The human body detection unit 24 may detect the human body based on the detection result of the face of the person shown in the image captured by the camera 12.
For example, the human body detection unit 24 may detect the human head as a face by scanning the edge image of the captured image using an elliptical reference pattern similar to the human head shape. Further, the human body detection unit 24 inputs an image into a discriminator pre-learned using an image in which a human face exists and an image in which a human face does not exist, and the human body detection unit 24 detects a person imaged in a captured image by the discriminator. Faces may be detected.

In addition, the human body detection unit 24 may detect a human face by detecting a skin color region of a captured image. As shown in FIG. 4B, when the mosaic processing is performed on the captured image, the mosaic processing is performed so that the color information remains so that the face of the person shown in the captured image is detected based on the skin color information. can do.
A marker having a predetermined shape or a reflective material may be attached to clothes of the resident 3. The human body detection unit 24 detects the human body based on the detection result of the predetermined marker shown in the image captured by the camera 12. Good.

It should be noted that the staff member 4 who knows the mounting determination logic of the mobile terminal device 5 based on the detection of the human body places the mobile terminal device 5 at a position where the face of the resident 3 appears, and the staff member 4 is not mounted with the mobile terminal device 5. However, it may be possible to take measures so that the abnormality cannot be detected.
Therefore, even when the human body detection unit 24 detects a human body, the abnormality determination unit 25 attaches the mobile terminal device 5 when the state in which there is no change between temporally consecutive images continues for a predetermined period or more. It is determined that the status is abnormal.

For example, the abnormality determination unit 25 determines that the mounting state of the mobile terminal device 5 is abnormal when the change between the captured images (that is, “frames”) captured by the camera 12 at different times is less than or equal to a predetermined amount.
In this case, a change occurs between frames even when the subject of the camera 12 moves a little, but when the mobile terminal device 5 is normally attached, a change occurs in the entire image in the process of the staff member 4 working. The amount of change is larger because it occurs. Therefore, whether or not the mounting state of the mobile terminal device 5 is abnormal can be identified based on the amount of change between frames.

Further, the abnormality determination unit 25 determines an abnormality such as a failure of the sound detection unit 16 or a policy action of the staff member 4 with respect to the sound detection unit 16.
The abnormality determination unit 25 detects an abnormality in the sound detection unit 16, that is, sound detection, when the sound detection unit 16 does not detect a sound signal even though the human body detection unit 24 detects a human body during image capturing by the camera 12. The failure of the unit 16 and any misconduct to the sound detection unit 16 are determined. For example, the abnormality determination unit 25 detects the acoustic detection unit 16 when the acoustic detection unit 16 does not detect an acoustic signal equal to or higher than the predetermined level Th1 even though the human body detection unit 24 detects the human body during shooting by the camera 12. 16 abnormalities may be determined.
The abnormality determination unit 25 notifies the server device 7 of the abnormality of the sound detection unit 16 via the communication processing unit 20.

Here, in the “misconduct”, for example, a normal care activity is recorded in the image captured by the camera 12, but the staff member 4 makes a remark such as abusing the resident 3, and the sound signal at that time is recorded by the staff member. 4 includes acts of obstruction.
For example, as a case where the sound detecting unit 16 does not detect a sound signal, it is assumed that the staff member 4 covers the sound detecting unit 16 with some material and no sound is input to the sound detecting unit 16. In this case, the abnormality determination unit 25 may determine that the acoustic detection unit 16 is abnormal when the acoustic detection unit 16 does not detect an acoustic signal of a predetermined level Th1 or more before and after the human body detection unit 24 detects the human body.

The timing of detecting the human body by the human body detection unit 24 and the timing of detecting the acoustic signal may not be the same. That is, if the acoustic signal of the predetermined level Th1 or higher is not detected within the predetermined period before and after the human body detection unit 24 includes the human body detection time, it may be determined that the acoustic detection unit 16 is abnormal.

Further, the fraudulent activity may include an activity of making the voice of the staff member 4 indistinguishable by inputting noise to the sound detection unit 16. For example, it is conceivable to continuously hit the acoustic detector 16 to input noise.
In this case, the abnormality determination unit 25 causes the sound detection unit 16 to output an acoustic signal of a second predetermined level Th2 or higher, which is higher than the predetermined level, while the human body detection unit 24 is detecting the human body during the image capturing by the camera 12. If detected, it is determined that the sound detection unit 16 is abnormal.
The timing of detecting the human body by the human body detection unit 24 and the timing of detecting the acoustic signal do not have to match. That is, when the acoustic signal of the second predetermined level Th2 or higher is detected within a predetermined period before and after the human body detection unit 24 includes the human body detection time, it may be determined that the acoustic detection unit 16 is abnormal.

Further, even if the sound detection unit 16 detects a sound signal of a predetermined level Th1 or higher during the period in which the human body detection unit 24 detects a human body during the image pickup by the camera 12, the abnormality determination unit 25 detects the detected vowel sound. If the component is not included, it may be determined that the sound detection unit 16 is abnormal. That is, it may be determined whether or not the human voice can be detected by determining whether or not the detected acoustic signal includes a vowel.
For example, by calculating the frequency feature amount of the acoustic signal detected for each predetermined period and comparing the calculated vowel feature amount with the vowel feature amount learned based on voice data collected in advance from a large number of speakers. It may be determined whether the acoustic signal includes a vowel.
It should be noted that the timing of the human body detection by the human body detection unit 24 and the timing of the detection of the acoustic signal containing the vowel component may not be the same. That is, if the acoustic signal does not include a vowel component even if the acoustic detection unit 16 detects an acoustic signal of a predetermined level Th1 or more within a predetermined period before and after including the human body detection time by the human body detection unit 24, the acoustic signal It may be determined that the detection unit 16 is abnormal.

(Server device)
Please refer to FIG. When receiving the ID information of the communication device 2 from the mobile terminal device 5, the server device 7 determines whether or not the mobile terminal device 5 can be photographed in the living room 1 in which the communication device 2 is installed.
When permitting the photographing by the mobile terminal device 5 in the living room 1, the server device 7 transmits a photographing permission signal to the mobile terminal device 5.
The server device 7 securely stores the captured image of the camera 12 and the sound signal from the sound detection unit 16 received from the mobile terminal device 5 in the predetermined storage device 42 as the action record information of the staff member 4.

Further, the server device 7 determines whether or not the current time is within the scheduled work period of the staff member 4 in the living room 1 (for example, the nursing care schedule period). When the current time is within the scheduled work period of the staff member 4 in the living room 1, the server device 7 transmits a photographing stop prohibition signal to the mobile terminal device 5. When the current time is outside the scheduled work period of the staff member 4 in the living room 1, the photographing stop prohibition signal is not transmitted to the mobile terminal device 5. As a result, the shooting control unit 22 of the mobile terminal device 5 prohibits the camera 12 from stopping the shooting when the current time is the scheduled work period, and stops the shooting by the camera 12 when the current time is outside the scheduled work period. Allow stop.

In this way, in the work outside the scheduled work period such as the emergency response, the staff member 4 may be allowed to stop the shooting by himself even if the shooting is started. That is, in such an emergency, it becomes possible for the staff member 4 to judge whether or not to record his/her actions. With this, for example, it is possible to prevent hesitating from taking drastic action due to being photographed in an emergency situation, and conversely, if it is desired to record one's own actions, it is possible to make a decision not to stop photographing. Become.

The server device 7 includes a communication unit 40, a control unit 41, and a storage device 42.
The communication unit 40 is a circuit that includes a wired or wireless communication interface and that performs wired communication or wireless communication with the mobile terminal device 5 via the communication line 6.
The control unit 41 may include a processor such as a CPU or MPU for controlling the operation of the server device 7.

The control unit 41 includes a communication processing unit 50, a photographing availability determination unit 51, a work period comparison unit 52, a control signal generation unit 53, and an action recording unit 54. The control unit 41 executes the computer program stored in the storage device 42 to execute the communication processing unit 50, the photographing availability determination unit 51, the work period comparison unit 52, the control signal generation unit 53, and the action record described below. The processing of the unit 54 is performed. That is, the computer program includes an instruction that causes the control unit 41 to perform the processing of the communication processing unit 50, the photographing availability determination unit 51, the work period comparison unit 52, the control signal generation unit 53, and the action recording unit 54.

The storage device 42 may include any one of a semiconductor storage device, a magnetic storage device, and an optical storage device. The storage device 42 may include a memory such as a register, a cache memory, a ROM and a RAM used as a main storage device. The storage device 42 stores resident information 60, schedule information 61, recordability information 62, and action record information 63.
The resident information 60 is information about the resident 3 in the living room 1, and is stored in association with the ID information of the communication device 2 installed in the living room 1.
The schedule information 61 is information related to the scheduled work period of the staff member 4 in the living room 1, that is, the nursing care schedule period, and is stored in association with the ID information of the communication device 2 installed in the living room 1.

The record enable/disable information 62 is information on whether the portable terminal device 5 in the living room 1 can shoot and record, and is stored in association with the ID information of the communication device 2 installed in the living room 1. When the resident 3 in the living room 1 agrees to the shooting and the sound recording, information indicating that the shooting and the sound recording are permitted is stored as the record permission/inhibition information 62. When the resident 3 in the living room 1 does not agree with the shooting and the sound recording, information indicating that the shooting and the sound recording are prohibited is stored as the record permission/inhibition information 62.
The action record information 63 is a captured image of the camera 12 received from the mobile terminal device 5 and an acoustic signal by the acoustic detection unit 16, and includes ID information of the communication device 2 installed in the living room 1 and resident information of the resident 3. Securely stored in association with 60.

The communication processing unit 50 performs communication processing with the mobile terminal device 5 via the communication unit 40. The communication processing unit 50 receives the ID information of the communication device 2 transmitted from the mobile terminal device 5.
When the ID information of the communication device 2 is received, the photographing possibility determination unit 51 determines whether to permit photographing and recording by the mobile terminal device 5 in the living room 1 in which the communication device 2 is installed. The photographing propriety determination unit 51 determines whether to permit photographing and recording based on the ID information of the communication device 2 and the recording propriety information 62 associated with this ID information.
When permitting shooting and recording in the living room 1, the control signal generation unit 53 generates a shooting permission signal. The communication processing unit 50 transmits a shooting permission signal to the mobile terminal device 5 via the communication unit 40.

When the ID information of the communication device 2 is received, the work period collation unit 52 determines whether the current time is within the work scheduled period of the staff member 4 in the living room 1.
When the current time is within the scheduled work period of the staff member 4 in the living room 1, the control signal generation unit 53 generates a photography stop prohibition signal. The communication processing unit 50 transmits a shooting stop prohibition signal to the mobile terminal device 5 via the communication unit 40.
When the current time is outside the scheduled work period of the staff member 4 in the living room 1, the control signal generation unit 53 does not generate the photographing stop prohibition signal.
The action recording unit 54 stores the captured image of the camera 12 and the acoustic signal from the acoustic detection unit 16 received from the mobile terminal device 5 in the storage device 42 as the action record information 63 of the staff member 4.

(motion)
Next, the operation of the action recording system will be described. Please refer to FIG. The operation shown in FIG. 6 is repeatedly executed at a predetermined cycle (for example, 0.1 seconds).
In step S1, the entry determination unit 21 of the mobile terminal device 5 determines whether or not a beacon signal has been received from the communication device 2. When the beacon signal is received (step S1: Y), the communication processing unit 20 transmits the ID information of the communication device 2 included in the beacon signal to the server device 7. After that, the process proceeds to step S2. When the beacon signal is not received (step S1:N), the process proceeds to step S11.

When the shooting permission signal transmitted from the server device 7 is received in response to the ID information, the shooting control unit 22 determines in step S2 whether or not the camera 12 is shooting. If shooting is in progress (step S2: Y), the process proceeds to step S6. If the shooting is not being performed (step S2: N), the process proceeds to step S3.
In step S3, the shooting control unit 22 starts shooting with the camera 12. Thereafter, the process proceeds to step S4.

In step S4, the work period matching unit 52 of the server device 7 determines whether the current time is within the care schedule period of the staff member 4 in the living room 1. When the current time is within the care schedule period (step S4: Y), the control signal generation unit 53 generates a photography stop prohibition signal. The mobile terminal device 5 receives the photography stop prohibition signal. After that, the process proceeds to step S5. If the current time is outside the care schedule period (step S4: N), the process proceeds to step S6.
When the shooting stop prohibition signal is received, the shooting control unit 22 sets the prohibition of the shooting stop by the camera 12 in step S5. Then, the process proceeds to step S6.

In step S6, the abnormality determination unit 25 executes a mounting state determination process. In the wearing state determination process, the abnormality determining unit 25 determines the wearing state of the mobile terminal device 5 on the staff member 4.
If the mounted state is normal (step S7: Y), the process proceeds to step S8. If the mounting state is not abnormal (step S7: N), the process proceeds to step S10.

In step S8, the abnormality determination unit 25 executes sound detection state determination processing. In the sound detection state determination process, the abnormality determination unit 25 determines whether the sound detection unit 16 is abnormal.
When the sound detection unit 16 is abnormal (step S9:Y), the process proceeds to step S10. If the sound detecting unit 16 is not abnormal (step S9: N), the process ends.
In step S10, the abnormality determination unit 25 transmits the wearing state of the mobile terminal device 5 or the abnormality of the sound detection unit 16 to the server device 7. After that, the process ends.

In step S11, the shooting control unit 22 determines whether the camera 12 is shooting. If shooting is in progress (step S11: Y), the process proceeds to step S12. If the shooting is not being performed (step S11: N), the process ends.
In step S12, the shooting control unit 22 stops shooting by the camera 12.
In step S13, the shooting control unit 22 releases the prohibition of stopping the shooting by the camera 12. After that, the process ends.

The mounting state determination process will be described with reference to FIG. 7.
In step S20, the abnormality determination unit 25 determines whether or not the human body detection unit 24 has detected the human body shown in the captured image. When the human body is detected (step S20: Y), the process proceeds to step S23. When the human body is not detected (step S20: N), the process proceeds to step S21.

In step S21, the abnormality determination unit 25 determines whether or not the value of the counter that counts the period in which the human body is not detected exceeds a predetermined value, that is, whether or not the predetermined period has elapsed without detecting the human body. When the predetermined period has elapsed (step S21: Y), the process proceeds to step S22. When the predetermined period has not elapsed (step S21: N), the process proceeds to step S24.

In step S22, the abnormality determination unit 25 determines that the wearing state of the mobile terminal device 5 is abnormal. Then, the mounting state determination process ends.
In step S23, the abnormality determination unit 25 resets the counter that counts the period during which no human body is detected. After that, the process proceeds to step S24.
In step S24, the abnormality determination unit 25 determines that the wearing state of the mobile terminal device 5 is normal. Then, the mounting state determination process ends.

The sound detection state determination processing will be described with reference to FIG.
In step S30, the abnormality determination unit 25 determines whether or not the human body detection unit 24 has detected the human body shown in the captured image within a predetermined period including before and after the current time. When the human body is detected (step S30: Y), the process proceeds to step S31. When the human body is not detected (step S30: N), the sound detection state determination process ends.

In step S31, the abnormality determination unit 25 determines whether or not the acoustic level of the acoustic signal detected during the period including before and after the human body detection time by the human body detection unit 24 is less than the predetermined level Th1. That is, it is determined whether or not the acoustic signal of the predetermined level Th1 or more is detected. When the sound level is less than Th1 (step S31: Y), the process proceeds to step S35. When the sound level is not less than Th1 (step S31: N), the process proceeds to step S32.

In step S32, the abnormality determination unit 25 determines whether or not the sound level of the sound signal detected during the period including before and after the human body detection time by the human body detection unit 24 is the second predetermined level Th2 or more. When the sound level is equal to or higher than the second predetermined level Th2 (step S32:Y), the process proceeds to step S35. When the sound level is not equal to or higher than the second predetermined level Th2 (step S32:N), the process proceeds to step S33.

In step S33, the abnormality determination unit 25 determines whether or not the detected acoustic signal includes a vowel component. When the acoustic signal includes the vowel component (step S33:Y), the process proceeds to step S34. When the acoustic signal does not include a vowel component (step S33:N), the process proceeds to step S35.
In step S34, the abnormality determination unit 25 determines that the sound detection unit 16 is not abnormal. Then, the sound detection state determination process ends.
In step S35, the abnormality determination unit 25 determines that the sound detection unit 16 is abnormal. Then, the sound detection state determination process ends.

(Effects of the embodiment)
(1) The action recording system includes a communication device 2 installed in a living room 1 which is a predetermined area, and a mobile terminal device 5 that can be worn by a staff member 4.
The mobile terminal device 5 is based on a camera 12 that captures an image of the surroundings of the mobile terminal device 5, a first communication unit 13 that communicates with the communication device 2, and communication between the first communication unit 13 and the communication device 2. An image capturing control unit 22 that prohibits stopping of image capturing by the camera 12.

As a result, the privacy of the resident 3 is protected by monitoring the staff 4 only while the occupant 1 is in the living room 1, and the behavior of the staff 4 is recorded and the assault of the staff 4 against the resident 3 is performed. Can be prevented. Furthermore, the innocence of the staff 4 can be proved in the unlikely event of an accident.
Moreover, since the image of the resident 3 is taken by the mobile terminal device 5 attached to the staff 4, unlike the fixed surveillance camera, the resident 3 supported by the staff 4 enters the blind spot of the camera. Hateful. Therefore, it is possible to prevent violent acts from being performed by the blind spot of the camera.
Further, by prohibiting the staff member 4 from stopping the shooting in the living room 1, the behavior record of the staff member 4 is illegally interrupted and the staff member 4 is prevented from assaulting the resident member 3 during that time. You can

(2) The image capturing control unit 22 starts the image capturing by the camera 12 when the first communication unit 13 can communicate with the communication device 2, and the image capturing by the camera 12 when the first communication unit 13 cannot communicate with the communication device 2. You can stop.
With this, when the staff member 4 enters the living room 1, the shooting can be automatically started, so that it is possible to prevent the behavior record of the staff member 4 from being omitted due to forgetting the shooting. Further, since the photographing is automatically stopped when the staff member 4 leaves the living room 1, it is possible to avoid the invasion of the privacy of the staff member 4 by recording the behavior of the staff member 4 outside the working hours.

(3) The behavior recording system generates an alarm when the first communication unit 13 cannot communicate with the communication device 2 and a predetermined time elapses, and when the camera 12 continues to capture an image, an alarm control unit. 23 may be provided.
As a result, it is possible to avoid the invasion of the privacy of the staff member 4 due to the recording of the behavior of the staff member 4 outside the work period due to the failure of the photographing control unit 22 or the camera 12.

(4) The action recording system may include the storage device 42 that stores the scheduled work period in the living room 1. The shooting control unit 22 prohibits the camera 12 from stopping the shooting when the current time is the scheduled work period in the living room 1, and stops the shooting by the camera 12 when the current time is outside the scheduled work period in the living room 1. You may allow it.
This allows the staff member 4 to stop the photographing by himself/herself even if the photographing is started in the work outside the scheduled work period such as the emergency measure. For this reason, for example, it is possible to prevent hesitation from taking drastic action due to being filmed in an emergency situation, and conversely, if it is desired to record one's own actions, it is possible to make a decision not to stop shooting. Become.

(5) The action recording system includes the communication device 2 installed in the living room 1 which is a predetermined area, and the mobile terminal device 5 that can be worn by the staff 4.
The mobile terminal device 5 is based on a camera 12 that captures an image of the surroundings of the mobile terminal device 5, a first communication unit 13 that communicates with the communication device 2, and communication between the first communication unit 13 and the communication device 2. A mobile terminal device based on a shooting control unit 22 that controls shooting by the camera 12, a human body detection unit 24 that detects a human body shown in an image shot by the camera 12, and a detection result by the human body detection unit 24 during shooting by the camera 12. The abnormality determination unit 25 for determining abnormality of the wearing state of No. 5 is provided.

As a result, even if the staff member 4 is in the living room 1, the staff member 4 does not look at the resident 3 at all (that is, the staff member 4 does not look at the resident 3 at all). The natural state can be detected to determine the mounting state of the mobile terminal device 5. Therefore, it is possible to avoid that the staff member 4 does not normally wear the mobile terminal device 5 and disturbs the activity record of the staff member 4.

(6) The abnormality determination unit 25 may determine the abnormality of the wearing state of the mobile terminal device 5 according to the length of the period during which the human body detection unit 24 does not detect the human body continuously while the camera 12 is capturing images. By making a determination according to the detection period in this way, it is possible to prevent erroneous detection of an abnormality in the wearing state due to the temporary absence of the human body in the captured image.
(7) The abnormality determination unit 25 may determine the abnormality of the wearing state of the mobile terminal device 5 according to the frequency of detection of the human body by the human body detection unit 24 during the image capturing by the camera 12. By making the determination according to the detection frequency in this manner, it is possible to prevent erroneous detection of an abnormality in the wearing state due to the temporary absence of the human body in the captured image.

(8) The human body detection unit 24 may detect the human body based on the detection result of the human face in the image captured by the camera 12. The human body can be accurately detected by detecting the face, which is a characteristic part of the human body.
(9) The human body detection unit 24 may detect the human body based on the detection result of the predetermined marker included in the image captured by the camera 12. By detecting an object other than the human face, the human body can be accurately detected even if mosaic processing is performed on the image of the human face portion for privacy protection, for example.

(10) Even if the human body detection unit 24 detects a human body, the abnormality determination unit 25 has an abnormal wearing state of the mobile terminal device 5 when the change between the captured images taken by the camera 12 at different times is less than or equal to a predetermined amount. May be determined.
As a result, the staff member 4 who knows the mounting determination logic of the mobile terminal device 5 based on the detection of the human body places the mobile terminal device 5 at the position where the face of the resident 3 appears, and the staff member 4 is wearing the mobile terminal device 5. Even if it does not exist, it is possible to prevent the measures from being taken so that the abnormality cannot be detected.

(11) The action recording system includes the communication device 2 installed in the living room 1 which is a predetermined area, and the mobile terminal device 5 that can be worn by the staff 4.
The mobile terminal device 5 communicates with the camera 12 that captures an image of the surroundings of the mobile terminal device 5, a sound detection unit 16 that detects a sound signal, a first communication unit 13 that communicates with the communication device 2, and a first communication unit 13. A shooting control unit 22 that controls shooting by the camera 12 and sound detection by the sound detection unit 16 based on communication with the device 2, and a human body detection unit 24 that detects a human body in an image shot by the camera 12. If the sound detecting unit 16 does not detect a sound signal of a predetermined level Th1 or more during a predetermined period including before and after the human body detecting unit 24 detects a human body during image capturing by the camera 12, it is determined that the sound detecting unit 16 is abnormal. The abnormality determination unit 25 is provided.

As a result, it is possible to prevent an act of the staff 4 making an abusive comment on the resident 3 while the normal care activity is recorded in the image captured by the camera 12, and the staff 4 interferes with the recording of the acoustic signal at that time. be able to.

(12) The abnormality determination unit 25 detects the acoustic signal of the second predetermined level Th2 or more, which is higher than the predetermined level Th1, during the predetermined period including before and after the human body detection unit 24 detects the human body during the image capturing by the camera 12. When 16 detects, it may determine with the sound detection part 16 being abnormal.
Accordingly, it is possible to prevent the action of making the voice of the staff member 4 indistinguishable by inputting noise to the sound detecting unit 16.

(13) Even if the sound detection unit 16 detects a sound signal of a predetermined level Th1 or more during a predetermined period including before and after the human body detection unit 24 detects a human body during shooting by the camera 12, even if the sound detection unit 16 detects an acoustic signal If the signal does not include a vowel component, it may be determined that the sound detector 16 is abnormal.
As a result, it is possible to accurately detect the abnormality of the sound detection unit 16.

(Modification)
(1) The communication device 2 may be an IC tag. An IC tag may be provided on the mobile terminal device 5. For example, the action recording system may be interlocked with an entry/exit management system that performs authentication and entry/exit management when entering the room by holding the mobile terminal device 5 over an IC tag or a reader installed on the wall surface inside or outside the living room 1. For example, it is possible to permit entry into the room by succeeding in authentication, permit photography by the camera 12 and prohibit photography stop. Further, when leaving the room, the mobile terminal device 5 may be held over the IC tag or the reader to stop the image capturing by the camera 12.

(2) The ID information of the communication device 2, the resident information 60, the schedule information 61, and the recording availability information 62 are downloaded to the storage device 18 of the mobile terminal device 5, and the photographing control unit 22 of the mobile terminal device 5 causes the server device to operate. Regardless of the permission or prohibition signal from 7, the entry determination unit 21 refers to the recording permission/prohibition information 62 stored in the storage device 18 to permit or prohibit the photographing of the camera 12 in the living room 1, and stop the photographing when leaving the room. It may be canceled.
Further, the imaging control unit 22 may refer to the schedule information 61 stored in the storage device 18 to determine whether or not the current time is within the scheduled work period of the staff member 4 in the living room 1. The shooting control unit 22 prohibits the camera 12 from stopping the shooting when the current time is the scheduled work period in the living room 1, and stops the shooting by the camera 12 when the current time is outside the scheduled work period in the living room 1. You may allow it.

1... Living room, 2... Communication device, 3... Resident, 4... Staff, 5... Mobile terminal device, 6... Communication line, 7... Server device, 10... Operation part, 11... Display part, 12... Camera, 13... 1st communication part, 14... 2nd communication part, 15... Alarm output part, 16... Sound detection part, 17... Control part, 18... Storage device, 20... Communication processing part, 21... Entrance determination part, 22... Shooting control Part, 23... Alarm control part, 24... Human body detection part, 25... Abnormality determination part, 40... Communication part, 41... Control part, 42... Storage device, 50... Communication processing part, 51... Shooting permission judgment part, 52... Work period collating unit, 53... Control signal generating unit, 54... Action recording unit, 60... Resident information, 61... Schedule information, 62... Recordable/unavailable information, 63... Action record information

Claims (5)

A behavior recording system comprising: a communication device installed in a predetermined area; and a terminal device wearable by a person,
The terminal device,
A photographing unit for photographing the surroundings of the terminal device;
An acoustic detector for detecting acoustic signals,
A communication unit that communicates with the communication device;
An imaging control unit that controls imaging by the imaging unit and sound detection by the sound detection unit based on communication between the communication unit and the communication device,
A human body detection unit that detects a human body reflected in a captured image by the capturing unit;
Abnormality that is determined to be abnormal in the sound detecting unit when the sound detecting unit does not detect an acoustic signal of a predetermined level or higher during a predetermined period including before and after the human body detecting unit detects a human body during image capturing by the image capturing unit A judgment unit,
An action recording system comprising: The abnormality determination unit detects, by the sound detection unit, an acoustic signal of a second predetermined level or higher that is higher than the predetermined level during a predetermined period including before and after the human body detection unit detects a human body during image capturing by the image capturing unit. The action recording system according to claim 1, wherein when the sound is detected, it is determined that the sound detection unit is abnormal. The abnormality determination unit, even if the acoustic detection unit detects an acoustic signal of a predetermined level or higher during a predetermined period including before and after the human body detection unit detects a human body during image capturing by the imaging unit, the acoustic signal is The action recording system according to claim 1 or 2, wherein it is determined that the sound detecting unit is abnormal when a vowel component is not included. A terminal device that can be worn by a person,
A photographing unit for photographing the surroundings of the terminal device;
An acoustic detector for detecting acoustic signals,
A communication unit that communicates with a communication device installed in a predetermined area,
An imaging control unit that controls imaging by the imaging unit and sound detection by the sound detection unit based on communication between the communication unit and the communication device,
A human body detection unit that detects a human body reflected in a captured image by the capturing unit;
Abnormality that is determined to be an abnormality of the acoustic detection unit when the acoustic detection unit does not detect an acoustic signal of a predetermined level or higher during a predetermined period including before and after the human body detection unit detects a human body during imaging by the imaging unit A judgment unit,
A terminal device comprising: Communicates between a terminal device that can be worn by a person and a communication device installed in a predetermined area,
Based on the communication between the terminal device and the communication device, controlling the imaging of the surroundings of the terminal device by the imaging unit included in the terminal device and the detection of the acoustic signal by the acoustic detection unit included in the terminal device. ,
Detects the human body in the image taken by the imaging unit,
Wherein a predetermined period including before and after detecting a human body during photographing by the photographing unit, if the sound detector does not detect a predetermined level or more acoustic signals, determines an abnormality of the sound detector,
A behavior recording method characterized by the above.

Images