JP2024021234A - Information processing system and information processing method - Google Patents

Abstract

To provide an information processing system and an information processing method, etc., that appropriately support the provision of assistance to persons being assisted by caregivers.SOLUTION: The information processing system includes a first terminal device that operates in accordance with a positioning application that performs processing based on teaching data that represents correct answers to positions regarding persons and/or things in the provision of assistance, and a second terminal device that is used by a second user who supports the provision of assistance regarding a first user using the first terminal device. The first terminal device transmits a sensing result to the second terminal device via a network, and the second terminal device presents the sensing result to the second user and accepts input for setting up teaching data by the second user, with the first terminal device acting in accordance with a positioning application that corresponds to the teaching data created by setup input.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing system, an information processing method, and the like.

2. Description of the Related Art Conventionally, systems have been known that are used in situations where a caregiver provides assistance to a person being assisted. Patent Document 1 discloses a method of arranging a sensor in a living space and generating provided information regarding the state of a resident living in the living space based on a temporal change in detection information acquired by the sensor. .

Additionally, Patent Document 2 discloses a method for providing actions such as calling an ambulance in telemedicine based on biological information and a user response obtained from an interaction device.

Japanese Patent Application Publication No. 2021-18760 US Patent Application Publication No. 2018/0114591

Provided is an information processing system, an information processing method, etc. that appropriately supports assistance of a person being cared for by a carer.

One aspect of the present disclosure provides a first terminal device that stores a positioning application that performs processing based on teacher data representing a correct position of at least one of a person and an object in assistance, and operates according to the positioning application; a second terminal device used by a second user who supports the assistance for the first user using the device, the first terminal device receiving sensing results regarding at least one of the person and the object in the assistance. , to the second terminal device via a network, the second terminal device presents the sensing result to the second user, receives setting input of the teacher data by the second user, and transmits the sensing result to the first terminal device. The apparatus relates to an information processing system that operates according to the positioning application corresponding to the teacher data created by the setting input.

Another aspect of the present disclosure is a first terminal device that stores a positioning application that performs processing based on teacher data representing a correct position of at least one of a person and an object in assistance, and operates according to the positioning application; An information processing method in an information processing system, comprising: a second terminal device used by a second user who supports the assistance of the first user using the terminal device, the information processing method comprising: at least one of the person and the object in the assistance; transmitting a sensing result from the first terminal device to the second terminal device via a network, presenting the sensing result to a second user in the second terminal device, and transmitting the sensing result by the second user to the teacher data. The present invention relates to an information processing method of accepting a setting input and operating the first terminal device according to the positioning application corresponding to the teacher data created by the setting input.

FIG. 1 is a diagram illustrating a configuration example of an information processing system. 1 is a diagram illustrating a configuration example of a server system. It is a figure showing an example of composition of a 1st terminal device. This is an example of training data for a positioning application. This is an example of training data superimposed and displayed in a positioning application. It is a figure which shows the example of a structure of a 2nd terminal device. FIG. 2 is a sequence diagram illustrating the flow of processing in a setting mode. FIG. 3 is a diagram illustrating devices placed around a bed. It is a figure which illustrates the relationship between an application and a device in bed position adjustment. This is an example of a screen used to obtain teacher data in the setting mode. This is an example of a screen used to obtain teacher data in the setting mode. This is an example of a screen used to obtain teacher data in the setting mode. This is an example of a screen used to obtain additional information in the setting mode. FIG. 2 is a diagram illustrating a device placed on a bed. FIG. 2 is a diagram illustrating devices placed around a wheelchair. It is a figure which illustrates the relationship between an application and a device in wheelchair position adjustment. This is an example of a screen used to obtain teacher data in the setting mode. FIG. 3 is a diagram illustrating a pressure sensor placed in a wheelchair. This is an example of a screen used to obtain teacher data in the setting mode. This is an example of a screen that presents the results of algorithm creation. FIG. 3 is a diagram illustrating a device placed during a meal. This is an example of a screen used to obtain teacher data in the setting mode. This is an example of teacher data that is set. This is an example of teacher data that is set. This is an example of a screen used to obtain teacher data in the setting mode. FIG. 3 is a sequence diagram illustrating a process of selecting teacher data. FIG. 3 is a sequence diagram illustrating a process of changing an operation mode in consideration of load. FIG. 3 is a sequence diagram illustrating a process of changing an operation mode in consideration of load.

This embodiment will be described below with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description will be omitted. Note that this embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in this embodiment are essential configuration requirements of the present disclosure.

1. System Configuration Example FIG. 1 is a configuration example of an information processing system 10 according to this embodiment. The information processing system 10 according to the present embodiment is capable of digitizing the "intuition" and "tacit knowledge" of a caregiver regarding work performed based on the "intuition" and "tacit knowledge" in, for example, medical and nursing care situations. This system provides instructions to caregivers so that they can provide appropriate assistance regardless of skill level.

The assistant here may be a care worker at a nursing care facility, or a nurse or an assistant nurse at a medical facility such as a hospital. That is, the assistance in this embodiment includes various actions to support the person being assisted, and may include nursing care or medical acts such as injections. Further, the assisted person here is a person who receives assistance from a caregiver, and may be a resident of a nursing care facility or a patient admitted to or visiting a hospital. Further, the person being assisted may be a person being assisted who may be suffering from dementia, for example.

Further, the assistance in this embodiment may be provided at home. For example, the person being assisted in this embodiment may be a care recipient who receives home care, or a patient who receives home medical care. Further, the caregiver may be a family member of the care recipient or the patient, or may be a visiting helper. For example, the method of the present embodiment can be used to assist family members, etc. in home care by utilizing the tacit knowledge of skilled caregivers, care managers, etc. when the caregiver is a family member of the person being cared for who is not an expert. It may also support.

As shown in FIG. 1, the information processing system 10 includes a server system 100, a first terminal device 200, a second terminal device 300, and a sensing device 400. However, the configuration of the information processing system 10 is not limited to that shown in FIG. 1, and modifications such as omitting some configurations or adding other configurations are possible. For example, there may be a plurality of first terminal devices 200. Further, there may be a plurality of second terminal devices 300. Further, in FIG. 1, a bedside sensor 420 and a detection device 430, which will be described later with reference to FIG. 11, are illustrated as the sensing device 400. However, the sensing device 400 is not limited to this, and may be a seat sensor 440, which will be described later using FIG. 15, a swallowing discomfort detection device 460, which will be described later using FIG. It may be a device. The sensing device 400 may also include a bed 610, a mattress 620, a wheelchair 630, etc. that can detect pressure. Note that hereinafter, if there is no need to distinguish the plurality of sensing devices 400 from each other, they will simply be referred to as sensing devices 400.

The server system 100 is connected to a first terminal device 200 and a second terminal device 300 via a network NW, for example. The network NW here is, for example, a public communication network such as the Internet. However, the network NW is not limited to a public communication network, and may be a LAN (Local Area Network) or the like. For example, the server system 100 may perform communication according to the IEEE802.11 standard. However, various modifications can be made to the communication method between each device.

Server system 100 may be one server or may include multiple servers. For example, server system 100 may include a database server and an application server. For example, the application server may perform processing to create a positioning application AP1. The positioning application AP1 here is software that performs processing regarding the position of at least one of a person and an object during assistance. Details of the positioning application AP1 will be described later. The database server may also store the created positioning application AP1. Note that the plurality of servers here may be physical servers or virtual servers. Further, when a virtual server is used, the virtual server may be provided in one physical server, or may be distributed and arranged in a plurality of physical servers. As described above, the specific configuration of the server system 100 in this embodiment can be modified in various ways.

The first terminal device 200 is a device used, for example, by a person being cared for at home, a family member caring for the person being cared for, or the like. However, the person being assisted here may be a person being assisted who resides in a facility such as a nursing care facility. Hereinafter, the user who uses the first terminal device 200 will be referred to as a first user.

The first terminal device 200 is a device on which a positioning application AP1 operates. The first terminal device 200 is, for example, a mobile terminal device such as a smartphone or a tablet terminal. However, the first terminal device 200 may be other devices such as a PC (Personal Computer), a headset, or a wearable device such as AR (Augmented Reality) glasses or MR (Mixed Reality) glasses. Further, the first user may use multiple first terminal devices 200. For example, the first user may use both a smartphone and a headset. Further, the person being assisted and the caregiver (family member) may each use different smartphones.

The second terminal device 300 is a device used by a user who has specialized knowledge regarding assistance, such as a caregiver, nurse, doctor, or care manager. Hereinafter, the user who uses the second terminal device 300 will be referred to as a second user. The second terminal device 300 is also similar to the first terminal device 200, and can be realized by various devices such as a smartphone, a tablet terminal, a PC, a headset, and a wearable device.

The sensing device 400 is a device used to assist a person being assisted. For example, the sensing device 400 has various sensors and acquires sensing data based on the sensors. The sensing data here may be the sensor output itself, or may be information obtained by calculation processing based on the sensor output. The sensing device 400 may be a device that outputs input data used when setting or using the positioning application AP1. Details will be described later.

FIG. 2 is a block diagram showing a detailed configuration example of the server system 100. The server system 100 includes, for example, a processing section 110, a storage section 120, and a communication section 130.

The processing unit 110 of this embodiment is configured by the following hardware. The hardware can include at least one of a circuit that processes digital signals and a circuit that processes analog signals. For example, the hardware can be configured by one or more circuit devices mounted on a circuit board or one or more circuit elements. The one or more circuit devices are, for example, an IC (Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like. The one or more circuit elements are, for example, resistors, capacitors, etc.

Further, the processing unit 110 may be realized by the following processor. The server system 100 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information includes, for example, programs and various data. The memory may be the storage unit 120 or another memory. A processor includes hardware. As the processor, various types of processors can be used, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor). The memory may be a semiconductor memory such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, or may be a register, or may be a hard disk drive (HDD), etc. The storage device may be a magnetic storage device, or an optical storage device such as an optical disk device. For example, the memory stores computer-readable instructions, and when the processor executes the instructions, the functions of the processing unit 110 are realized as processing. The instructions here may be instructions of an instruction set that constitutes a program, or instructions that instruct a hardware circuit of a processor to operate.

The storage unit 120 is a work area of the processing unit 110 and stores various information. The storage unit 120 can be realized by various types of memory, and the memory may be a semiconductor memory such as SRAM, DRAM, ROM (Read Only Memory), or flash memory, or may be a register. It may be a magnetic storage device or an optical storage device.

The communication unit 130 is an interface for communicating via a network, and includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit when the server system 100 performs wireless communication. However, the server system 100 may perform wired communication, and in that case, the communication unit 130 may include a communication interface such as an Ethernet connector, a control circuit for the communication interface, and the like. The communication unit 130 may operate under the control of the processing unit 110, and may include a communication control processor different from the processing unit 110. The communication unit 130 may perform communication according to a method defined by IEEE802.11 or IEEE802.3, for example. However, the specific communication method can be modified in various ways.

FIG. 3 is a block diagram showing a detailed configuration example of the first terminal device 200. The first terminal device 200 includes, for example, a processing section 210, a storage section 220, a communication section 230, a display section 240, an operation section 250, and an imaging section 260. However, the configuration of the first terminal device 200 is not limited to that shown in FIG. 3, and modifications such as omitting some configurations or adding other configurations are possible. For example, in addition to the image sensor included in the imaging unit 260, the first terminal device 200 may include various sensors such as a motion sensor such as an acceleration sensor and a gyro sensor, a pressure sensor, and a GPS (Global Positioning System) sensor. good.

The processing unit 210 is configured by hardware including at least one of a circuit that processes digital signals and a circuit that processes analog signals. Further, the processing unit 210 may be realized by a processor. As the processor, various types of processors such as a CPU, GPU, and DSP can be used. The functions of the processing unit 210 are realized as processing by the processor executing instructions stored in the memory of the first terminal device 200.

The storage unit 220 is a work area of the processing unit 210, and is realized by various types of memory such as SRAM, DRAM, and ROM. The storage unit 220 stores, for example, positioning application AP1. The processing unit 210 may operate according to the positioning application AP1.

The positioning application AP1 in this embodiment is application software that presents desirable positions of people and objects on the bed 610, wheelchair 630, and the like. The person here is, for example, a person being assisted or a caregiver. In addition, things include cushions, diapers, etc.

For example, the positioning application AP1 may operate in a setting mode for making settings and a usage mode for supporting actual position adjustment according to the settings. In the setting mode, for example, the positioning application AP1 acquires training data obtained by capturing an image of a person or object at a desired position. In the usage mode, the positioning application AP1 superimposes and displays training data subjected to transparency processing on a captured image of a person or object to be adjusted.

FIG. 4A is an example of teacher data acquired in the setting mode. In the example of FIG. 4A, image information representing a desirable posture of a person being assisted with the name "AAA" lying on the bed 610 is acquired as teacher data. FIG. 4B is an example of teacher data that is displayed superimposed on the captured image in the usage mode and that has been subjected to transparency processing. For example, the terminal device 200 displays the image of FIG. 4B superimposed on the captured image of the person to be assisted whose position is to be adjusted. The assistant assists the person being helped so that the person being helped on the captured image matches the teacher data. In this way, it becomes possible to appropriately support the position adjustment by the caregiver. Although an example in which superimposed display of teacher data, which is image information, is explained here, the positioning application AP1 may output a determination result (OK/NG) as to whether the posture of the person being assisted is appropriate. good. For example, the positioning application AP1 may determine OK/NG based on the difference between the teacher data and the captured image. For example, the positioning application AP1 calculates a norm (for example, the sum of absolute differences) based on each RGB pixel value, and if the ratio of pixels for which the norm is equal to or greater than a threshold is greater than or equal to a predetermined value, the difference is large and the result is NG. You may judge. Details of the processing executed by the positioning application AP1 will be described later.

The communication unit 230 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 communicates with the server system 100, for example, via the network NW. The communication unit 230 may perform wireless communication with the server system 100 based on, for example, the IEEE802.11 standard.

The display unit 240 is an interface that displays various information, and may be a liquid crystal display, an organic EL display, or another type of display. The operation unit 250 is an interface that accepts user operations. The operation unit 250 may be a button or the like provided on the first terminal device 200. Further, the display section 240 and the operation section 250 may be a touch panel configured as one unit.

The imaging unit 260 includes an image sensor that outputs image information by imaging a predetermined imaging range. The image information here may be a still image or a moving image. Further, the image information may be in color or monochrome. Further, the imaging unit 260 may include a depth sensor that detects the distance to the subject, or a sensor (for example, an infrared sensor) that detects the heat of the subject.

Further, the first terminal device 200 may include components not shown in FIG. 3, such as a light emitting section, a vibration section, a sound input section, and a sound output section. The light emitting unit is, for example, an LED (light emitting diode), and provides notification by emitting light. The vibration unit is, for example, a motor, and provides notification by vibration. The sound input section is, for example, a microphone. The sound output section is, for example, a speaker, and provides sound notification.

FIG. 5 is a block diagram showing a detailed configuration example of the second terminal device 300. The second terminal device 300 includes, for example, a processing section 310, a storage section 320, a communication section 330, a display section 340, an operation section 350, and an imaging section 360. However, the configuration of the second terminal device 300 is not limited to that shown in FIG. 5, and modifications such as omitting some configurations or adding other configurations are possible. Since the configuration of each part is the same as that of the first terminal device 200, detailed explanation will be omitted.

2. Description of the Related Art In assistance, it is known that, for example, it is important to adjust the position of the person to be cared for, the position of the caregiver, and the position of objects placed around the person to be cared for. For example, adjusting the position of the person being assisted is useful for reducing the risk of bedsores, falls, and the like. Adjusting the position of the assistant is useful for teaching movements for appropriately performing assistance. Adjusting the position of objects is useful, for example, when arranging a cushion or the like to help the person being assisted take an appropriate posture. By using the positioning application AP1 described above, it becomes possible to support position adjustment in assistance.

However, how to adjust the position changes depending on the situation and is tacit knowledge of experienced caregivers. For example, when adjusting the position of a person being assisted, the desired posture may vary depending on the attributes of the person being assisted. The attributes here include age, gender, height, weight, medical history, medication history, etc. Therefore, different persons being assisted have different desirable postures, and even for the same person being assisted, the desirable posture may change if there is a factor that changes the attributes. The same applies to the position of the assistant who provides assistance.

Therefore, when using the positioning application AP1, the teacher data representing the desired position can only be set by a skilled assistant with tacit knowledge. However, in recent years, home care and telemedicine have become widespread. In this case, since it is difficult for a skilled caregiver to directly adjust the posture of the person being helped, it is desirable to set the tacit knowledge of the caregiver remotely. Conventional methods such as Patent Document 1 and Patent Document 2 do not disclose a method for efficiently remotely setting tacit knowledge.

The information processing system 10 of this embodiment includes a first terminal device 200 and a second terminal device 300, as shown in FIG. The first terminal device 200 stores a positioning application AP1 that performs processing based on teacher data representing the correct position of at least one of a person and an object in assistance, and operates according to the positioning application AP1. The second terminal device 300 is used by a second user who supports the first user using the first terminal device 200. Note that the first user may include a person being cared for (for example, a person being cared for at home) and a caregiver (for example, a family member of the person being cared for), and the assistance support for the first user refers to the person being cared for by the person being cared for. It may be to support behavior, or it may be to support assistance behavior by family members.

The first terminal device 200 then transmits the sensing results regarding at least one of a person and an object during assistance to the second terminal device 300 via the network NW. The sensing result here may be a captured image that is sensing data of the image sensor (imaging unit 260) of the first terminal device 200, or various information that is sensing data of the sensing device 400. . A specific example of the sensing data of the sensing device 400 will be described later. The second terminal device 300 presents the sensing results to the second user and accepts input of teacher data settings from the second user. The first terminal device 200 operates according to the positioning application AP1 corresponding to the teacher data created based on the setting input.

According to the method of this embodiment, it becomes possible to allow the second user to view sensing results related to position adjustment provided from the first terminal device 200. Further, the second user's setting input is reflected in the creation of teacher data used in the positioning application AP1. Therefore, even in a situation where the first terminal device 200 and the second terminal device 300 are connected via the network NW, it is possible to digitize the tacit knowledge of an expert as teacher data, and to digitize it in the positioning application AP1. It becomes possible to use the training data that has been prepared.

Furthermore, part or all of the processing performed by the information processing system 10 of this embodiment may be realized by a program. The processing performed by the information processing system 10 may be the processing performed by the processing unit 210 of the first terminal device 200, the processing performed by the processing unit 310 of the second terminal device 300, The processing may be performed by the processing unit 110 of the server system 100. Further, the processing performed by the information processing system 10 may be the processing performed by two or more devices among the server system 100, the first terminal device 200, and the second terminal device 300.

The program according to this embodiment can be stored, for example, in a non-transitory information storage medium (information storage device) that is a computer-readable medium. The information storage medium can be realized by, for example, an optical disk, a memory card, an HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 110 and the like perform various processes of this embodiment based on programs stored in the information storage medium. That is, the information storage medium stores a program for causing the computer to function as the processing unit 110 and the like. A computer is a device that includes an input device, a processing section, a storage section, and an output section. Specifically, the program according to this embodiment is a program for causing a computer to execute each step described later using FIGS. 6, 22, 23A, 23B, etc.

Further, the method of this embodiment can be applied to an information processing method including the following steps. The information processing method includes a first terminal device 200 that stores a positioning application AP1 that performs processing based on teacher data representing correct positions of at least one of a person and an object in assistance, and operates according to the positioning application AP1; This is an information processing method in an information processing system 10 including a second terminal device 300 used by a second user who supports assistance of a first user using the device 200. The information processing method includes the steps of transmitting sensing results regarding at least one of a person and an object in assistance from a first terminal device 200 to a second terminal device 300 via a network NW, and transmitting the sensing results in the second terminal device 300. The steps include the steps of presenting the teacher data to the second user and receiving the teacher data setting input from the second user, and operating the first terminal device 200 in accordance with the positioning application AP1 corresponding to the teacher data created by the setting input. .

2. Setting of Positioning Application Next, a specific example of the positioning application AP1 will be explained. The positioning application AP1 may operate in a setting mode in which teacher data used for position adjustment is set and a usage mode in which processing is performed based on the set teacher data. Note that the positioning application AP1 may include a plurality of application software. For example, the positioning application AP1 may include a setting application that performs a process of setting teacher data, and a usage application that executes a process based on the teacher data. In this case, the setting mode is realized by a setting application, and the usage mode, which will be described later, is realized by a usage application. In addition, various modifications can be made to the specific configuration of the positioning application AP1.

Below, an overview of the processing in the setting mode will first be explained. Thereafter, examples of setting modes corresponding to each of the bed position, wheelchair position, and eating position will be explained in detail. Note that the bed position refers to the position of a person or object on the bed 610. The wheelchair position represents the position of a person or object in the wheelchair 630. The eating position refers to the position of a person or thing when eating, and may include, for example, a wheelchair position, a bed position, or a position different from these.

2.1 Flow of Processing in Setting Mode FIG. 6 is a diagram illustrating the flow of processing when setting the positioning application AP1 in the information processing system 10 of this embodiment. As described above using FIG. 1, the information processing system 10 includes a server system 100, a first terminal device 200, a second terminal device 300, and a sensing device 400. Among these, the first terminal device 200 and the sensing device 400 are placed, for example, in a home where home care is provided. For example, the sensing device 400 may be an assistance device purchased by a person being assisted or their family for use in home care. The second terminal device 300 is placed in a place different from the home, such as a workplace where a professional such as a caregiver works. The server system 100 can be placed at any location.

First, in step S101, the first terminal device 200 and the second terminal device 300 establish a connection. For example, a positioning application AP1 (setting application) is installed in the first terminal device 200 and the second terminal device 300, and the processing unit 210 and the processing unit 310 operate according to the positioning application AP1, so that the Communication is realized. For example, as will be described later using FIGS. 9A to 9C, the positioning application AP1 (setting application) here may have the function of a web conference application that communicates images and sounds with each other. Note that the communication here may be via the server system 100. For example, by establishing communication between the first terminal device 200 and the server system 100 and communication between the second terminal device 300 and the server system 100, the first terminal device 200 and the second terminal device 300 may be connected.

In step S102, the first terminal device 200 acquires information regarding the installed sensing device 400, and transmits the information to the second terminal device 300. For example, the family of the care recipient purchases a mattress 620 that detects pressure for the purpose of assistance, a bedside sensor 420 and detection device 430, which will be described later using FIG. 11, a seat sensor 440, which will be described later using FIG. 15, etc. If so, information identifying these devices is transmitted to the second terminal device 300. The process in step S102 may be realized by the first user of the first terminal device 200 inputting the type, model number, etc. of the device using the operation unit 250. Alternatively, the process of step S102 may be a process of transmitting information about the sensing device 400 stored in the storage unit 220. For example, when the first terminal device 200 and the sensing device 400 are connected when the sensing device 400 is first introduced, the storage unit 220 of the first terminal device 200 may store information about the sensing device 400. The initial installation can be performed by a person with knowledge, such as a manufacturer's service person, so even if the first user does not have knowledge about the sensing device 400, the process in step S102 can be performed. Appropriately practicable.

In step S103, the second terminal device 300 receives a selection input of a connection target device from the second user. As described above, the second user is an expert (also referred to as a caregiver) who can provide assistance. The connection target device here is the sensing device 400 to be connected to the first terminal device 200. For example, the second terminal device 300 may display the list of sensing devices 400 transmitted from the first terminal device 200, and may receive input from the second user who selects a connection target device from the list.

As described above, various sensing devices 400 such as the bedside sensor 420, the detection device 430, and the seat sensor 440 may have been installed around the first terminal device 200. Since these devices are capable of sensing the posture of the bed 610 or the wheelchair 630, they are useful for setting teacher data and using the set teacher data.

However, the sensing device 400 that is useful for setting teacher data differs depending on the target location. For example, the bedside sensor 420 and the detection device 430 are useful for setting teacher data for a bed position, and the seat sensor 440 is useful for setting teacher data for a wheelchair position.

Furthermore, even when targeting the same location (for example, bed 610), it is useful to set training data depending on the manufacturer and model number of bed 610 and mattress 620, attributes of the target care recipient, etc. The sensing device 400 may be different. In setting the teacher data (and using the teacher data to be described later), which sensing device 400 is useful to use may be tacit knowledge of a skilled assistant.

In the method of this embodiment, since it is possible to receive the selection input of the device to be connected from the second user, an expert can select not only the teacher data itself but also the sensing device 400 related to the teacher data. It becomes possible to use tacit knowledge appropriately.

In step S104, the second terminal device 300 outputs connection information used for connecting the sensing device 400 that outputs the sensing result and the first terminal device 200 to the first terminal device 200 via the network NW. For example, a QR code (registered trademark) for connection is provided to the sensing device 400 in advance, and a connection between the first terminal device 200 and the sensing device 400 can be established by reading the QR code using the first terminal device 200. It may be. In this case, the connection information transmitted by the second terminal device 300 may include information specifying the device to be connected and control information instructing the activation of the QR code reading application. For example, the first terminal device 200 that has received the connection information executes a process of displaying the text "Please read the QR code of the seat sensor 440" and a process of starting a QR code reading application. Note that the positioning application AP1 may have a QR code reading function.

However, the connection information is not limited to this. For example, if the IP address of the sensing device 400 is known, the connection information may be information that instructs connection to a device having the IP address. In this case, for example, the first terminal device 200 displays an authentication screen including text such as "Can I connect to the seat sensor? Yes/No," and the first user allows the connection on the authentication screen. When an input operation (for example, an operation of selecting Yes) is performed, a connection may be established with the sensing device 400 having the specified IP address.

In addition, the connection information here may be any information used to establish a connection between the sensing device 400 specified by the second user and the first terminal device 200, and various modifications can be made to the specific aspect. be.

In step S105, the first terminal device 200 establishes a connection with the sensing device 400 specified by the second user based on the connection information. In step S106, the sensing device 400 acquires sensing data and transmits the sensing data to the first terminal device 200. Details of the sensing data will be described later.

Note that FIG. 6 describes an example in which the first terminal device 200 and the sensing device 400 are successfully connected based on the connection information. However, there are cases where the connection between the first terminal device 200 and the sensing device 400 fails due to the occurrence of some kind of error. For example, the sensing device 400 has been installed (included in the information transmitted in step S102), but the power is not turned on, or there is some abnormality in the communication function of the sensing device 400. In some cases the connection fails. In this case, the first terminal device 200 may transmit information indicating that the connection with the sensing device 400 has failed to the second terminal device 300. The second terminal device 300 may then send advice for improving the connection state to the first terminal device 200. For example, the second user may give advice such as turning on the sensing device 400 or restarting the sensing device 400. In this way, the second user not only supports the selection of the sensing device 400 to be used, but also receives support from the second user for improving the connection state when the connection with the sensing device 400 fails. becomes possible to receive. The second user is likely to be more familiar with how to use and configure the sensing device 400 than the first user, and the second user's support makes it possible to use the sensing device 400 appropriately.

Moreover, although a connection error during a connection attempt has been described above, the connection state between the first terminal device 200 and the sensing device 400 may be periodically monitored even after the connection is successful. In this way, even if some error occurs during communication, the error can be appropriately detected. For example, although not shown in FIG. 6, after the process of step S105 is completed, the first terminal device 200 periodically executes a process of checking whether it is connected to the sensing device 400. For example, the first terminal device 200 may periodically transmit a confirmation packet to the sensing device 400 and determine whether an ACK has been received. Alternatively, the first terminal device 200 may detect a connection error when sensing data is not transmitted from the sensing device 400 for a predetermined period or more. If a connection error is detected, the first terminal device 200 may transmit information indicating that the connection with the sensing device 400 has failed to the second terminal device 300 in the same manner as described above.

Further, in step S107, the first terminal device 200 uses the imaging unit 260 to acquire a captured image including the person being assisted. For example, when considering a situation in which the bed position is adjusted, the imaging unit 260 acquires a moving image of a family member who is a caregiver adjusting the posture of a person being cared for who is lying on the bed 610. Good too.

Further, the information processing system 10 of the present embodiment may perform emotional analysis of the first user of the first terminal device 200 based on a captured image of the first user (also referred to as a first captured image). For example, in step S108 of FIG. 6, the first terminal device 200 performs a process of estimating the emotion of the care recipient based on the captured image acquired in step S107. As a method for estimating a person's emotion based on an image, for example, a machine based on training data in which information specifying the person's emotion is added as correct data to input data that is an image of a person's face. Learning may also be used. For example, the first terminal device 200 performs sentiment analysis by inputting the captured image acquired in step S107 to the learned model acquired by the machine learning. The model here is, for example, a CNN (Convolutional Neural Network). In addition, various methods are known to analyze emotions based on image processing, such as the Face API disclosed at “https://azure.microsoft.com/ja-jp/services/cognitive-services/face/”. In this embodiment, these methods can be widely applied. Furthermore, if the family member of the care recipient is captured in the captured image, emotional analysis of the family member of the care recipient may be performed in step S108. Note that emotion estimation is not limited to that performed by the first terminal device 200. For example, a captured image may be transmitted to the server system 100, and emotion analysis may be performed in the processing unit 110 of the server system 100.

In step S109, the first terminal device 200 transmits information including the sensing data, the captured image, and the result of emotional analysis to the second terminal device 300. In step S110, the display unit 340 of the second terminal device 300 displays a screen containing the transmitted information. Specific screens will be described later using FIGS. 9A to 9C, FIG. 14, FIG. 16, FIG. 19, and the like.

Note that the process of step S107 in FIG. 6 is not limited to being performed once, but may be performed repeatedly. For example, the first terminal device 200 transmits a first captured image of the first user of the first terminal device 200 to the second terminal device 300 during the setting period of the positioning application AP1 (period in which teacher data is set). You can. In this way, it is possible to clearly show the second user of the second terminal device 300 the state of the person being assisted and the person's family while setting the teacher data remotely. It becomes possible.

At this time, the second terminal device 300 may display the first captured image transmitted from the first terminal device 200 in association with the result of the emotion analysis. For example, as will be described later with reference to FIG. 9C and the like, an object EM representing an emotion may be displayed superimposed on a portion of the captured image. This makes it possible to associate the position and motion of the person to be assisted, etc. with the emotion of the person to be assisted, etc., and to present this to the second user in an easy-to-understand manner. For example, the second user can easily understand how the person being assisted feels about their current posture, and how the family providing the assistance feels about the content of the assistance and the reaction of the person being assisted. As a result, tacit knowledge that the person being cared for or their family does not like is prevented from being set as teacher data, so it becomes possible to set teacher data suitable for the person being cared for.

Further, similar to the process in step S107, the processes in steps S106, S108, S109, and S110 in FIG. 6 are not limited to being performed once each, but may be performed repeatedly. For example, sensing data and emotional analysis results may be acquired at predetermined intervals and transmitted from the first terminal device 200 to the second terminal device 300 at any timing. In step S110, the second terminal device 300 may display a screen in which display areas for sensing data, captured images, and emotion analysis results are updated at predetermined intervals. That is, the display processing in step S110 may be display processing of an image that is updated over time.

Next, in step S111, the second user inputs settings for teacher data while viewing the displayed screen, and the second terminal device 300 accepts the settings input. For example, the second user determines whether the current posture is appropriate for the target person being assisted by viewing the captured image and sensing data transmitted from the first terminal device 200. If it is determined that it is inappropriate, for example, the second user instructs the care recipient's family to change their posture, add a cushion, etc. using audio, video, etc. If it is determined that the posture is appropriate, for example, the second user performs a setting input indicating that. For example, when a selection operation is performed on the algorithm creation button (object OB4), which will be described later using FIG. 9C, the captured image etc. that were displayed at that time are set as the teacher data. Furthermore, as will be described later with reference to FIG. 16, the second user may perform an operation to select part of the sensing data, and various modifications can be made to the details of the setting input.

Note that, as can be seen from the above description, in the setting mode, the second user's voice, etc. may be transmitted to the first user. For example, the second terminal device 300 may transmit a second captured image of the second user of the second terminal device 300 to the first terminal device during a setting period in which remote setting of teacher data is performed. In this way, the second user can give instructions regarding posture using voice and movements, and can provide the person being assisted and their family with the necessary movements and auxiliary goods (such as cushions) to take an appropriate posture. ) etc. can be presented in an easy-to-understand manner.

Further, the first terminal device 200 may store a captured image (second captured image) of the second user of the second terminal device 300 as log data. For example, the second captured image may be used as part of the teacher data, and the first terminal device 200 may store the positioning application AP1 corresponding to the teacher data. Alternatively, the second captured image may be stored as information different from the positioning application AP1.

For example, the second captured image here may be an image in which the second user, who is an assistance expert, is showing a sample motion for making the person being assisted take an appropriate posture. Note that, since the second user is assumed to be in a location apart from the first user, it may be difficult to actually assist the person being assisted included in the first user. Therefore, the person being assisted by the second user may be another person or may be a dummy such as a doll. Furthermore, showing samples in a virtual space such as the metaverse is not prohibited. In this way, the first user can review the movements of the skilled caregiver, making it possible for even the family of the person being assisted who is not an expert to make appropriate position adjustments. . For example, the first terminal device 200 displays the second captured image stored as log data during the usage period of the positioning application AP1. In this way, it becomes possible to check the movements of the skilled assistant when the position is actually adjusted.

Further, the log data here may include a captured image of the first user of the first terminal device 200 (first captured image). For example, if the second captured image is an image showing a sample, the first captured image is an image of the first user acting according to the sample. In this way, the first user can check his or her own actions in the past (at the time of setting) when performing position adjustment, making it easier to perform appropriate actions. For example, when the positioning application AP1 (setting application) has a web conference function, information (video) including the first user's voice and image and the second user's voice and image are recorded in chronological order, and the recorded data may be saved as log data. In this way, the first user can check what kind of interaction has taken place with the second user, making it easier for the first user to perform more appropriate actions.

If the teacher data settings have been input, the second terminal device 300 transmits information based on the settings input to the server system 100 in step S112. When the teacher data is a captured image or sensing data, the information transmitted in step S112 may be the teacher data itself. Further, the information transmitted in step S112 may be information for generating teacher data. For example, the setting input by the second user is data representing a correct answer when performing machine learning such as classification or regression, and teacher data may be generated by performing machine learning based on the data. Note that when a process such as machine learning is performed, the process may be executed by the second terminal device 300 or by the server system 100. Details will be described later using FIGS. 16 and 17.

In step S113, the server system 100 creates an algorithm corresponding to the teacher data created based on the second user's setting input. The algorithm here may be update data for adding training data to the existing positioning application AP1. Alternatively, the algorithm may be a new positioning application AP1.

In step S114, the server system 100 transmits the created algorithm to the first terminal device 200. This allows the first terminal device 200 to use the positioning application AP1 corresponding to the teacher data created by the second user's setting input.

2.2 Bed Position FIG. 7 is a diagram showing an example of arrangement of devices around the bed 610. As shown in FIG. 7, the first terminal device 200 in this embodiment may be a terminal device 200-1 fixed to the footboard side of the bed 610, or a terminal device 200-1 fixed to the side rail of the bed 610. It may be the device 200-2 or both. However, the location where the first terminal device 200 is fixed is not limited to this, and may be fixed at another location on the bed 610, or at another location where the person being assisted on the bed 610 can be imaged (for example, in the living room). It may also be fixed to walls, furniture, etc.). Alternatively, the first terminal device 200 is carried by a caregiver, and the process described below can be performed by using the first terminal device 200 while the caregiver is positioned on the footboard side or side rail side of the bed 610. May be executed.

Furthermore, a display DP may be placed on the opposite side of the terminal device 200-2 with respect to the bed 610. Note that the display DP may be fixed to the bed 610, or may be placed at another position where it can be viewed naturally by an assistant who adjusts the bed position using the positioning application AP1. For example, the display DP may be fixed to a wall, or may be fixed to a stand that stands on the floor. Further, the display DP can be omitted.

As described above using FIG. 3, the first terminal device 200 is, for example, a device such as a smartphone having an imaging unit 260 (camera). The first terminal device 200 transmits the captured image to the server system 100. The display DP receives an image transmitted from the server system 100 directly or via another device such as the first terminal device 200, and displays the received image.

The positioning application AP1 may be an application that supports adjusting the bed position. Adjustment of the bed position may be, for example, controlling the posture of the person being assisted to prevent bedsores or controlling the arrangement of cushions. Further, the adjustment of the bed position may be the posture control of the person being assisted during diaper changing or the position control of the diaper.

FIG. 8 is a diagram illustrating the relationship of devices used in the setting mode. For example, the positioning application AP1 may acquire image information, which is an image captured by the imaging unit 260 included in the first terminal device 200, and information from at least one of the bed 610 and the mattress 620. The bed 610 and the mattress 620 here are devices that can visualize the pressure distribution while being used by the person being assisted. For example, as a method for visualizing pressure distribution, a known method disclosed in Japanese Patent Application Publication No. 2019-039788 and the like can be widely applied.

For example, the first terminal device 200, in step S109 of FIG. The information including the pressure distribution is transmitted to the second terminal device 300.

9A to 9C are examples of screens displayed on the display unit 240 of the first terminal device 200 and the display unit 340 of the second terminal device 300 in the teacher data setting process. The positioning application AP1 (setting application) according to this embodiment may have a web conference function. The screen shown in FIG. 9A includes various buttons such as a participant button, a chat button, a camera button, a microphone button, and the like. When the participant button is selected, the participants of the web conference are displayed. When the chat button is selected, an area for chatting (text communication) is displayed. When the camera button is selected, the camera of the device (first terminal device 200 or second terminal device 300) that displays the screen of FIG. 9A is turned on and off. When the microphone button is selected, the microphone of the device displaying the screen of FIG. 9A is turned on and off. Further, the screen shown in FIG. 9A includes a display area for each participant. In the example of FIG. 9A, the participants are user A and user B, and the area corresponding to user A is relatively large. For example, when one of the first user and the second user sets up a web conference and each user participates in the conference, the process of step S101 in FIG. 6 is executed, and the screen shown in FIG. 9A is displayed.

FIG. 9B is a diagram displayed when the selection operation of the cooperation button displayed at the top of the screen shown in FIG. 9A is performed, and is a diagram of the sensing device 400 to be connected shown in step S103 of FIG. 6. It is a figure corresponding to selection processing.

When the selection operation of the cooperation button is performed, an object OB2 for selecting the sensing device 400 to be connected may be displayed on a part of the screen (the left side of the screen in the example of FIG. 9B). Note that, as described above using FIG. 6, since the process of step S103 is executed by the second user, the cooperation button and object OB2 are displayed on the second terminal device 300, even if they are not displayed on the first terminal device 200. good. However, since the first user may select the sensing device 400 to be connected, the cooperation button and object OB2 may be displayed on the first terminal device 200. A display example on the second terminal device 300 will be described below.

As shown in step S102 in FIG. 6, the second terminal device 300 acquires information regarding the sensing devices 400 that have been introduced around the first terminal device 200, and determines whether each of the sensing devices 400 is connected or disconnected. The object for selecting may be displayed as object OB2. In the example of FIG. 9B, a mattress 630, a bed 610, a sleep scan (detection device 430 described later with reference to FIG. 11), a bedside sensor 420, a seat sensor 440 (described later with reference to FIG. 15), a swallowing motion app ( A swallowing choking detection device 460 (to be described later with reference to FIG. 18) can be connected to the first terminal device 200, and check boxes corresponding to the objects OB2 are displayed. The second terminal device 300 may also display text prompting the user to select a device to be connected, such as "Please select a device to connect." The second user determines the sensing device 400 to be connected by checking the checkbox using object OB2. The first terminal device 200 establishes a connection with the sensing device 400 based on the result of the selection operation on the object OB2 (corresponding to steps S104-S105 in FIG. 6).

Further, as shown in FIG. 9B, when a selection operation of the cooperation button is performed, an object OB1 corresponding to a check box for switching on/off of skeleton tracking may be displayed.

Based on the user's operation on the object OB1, the positioning application AP1 switches whether or not to perform the skeleton tracking process of the person being cared for. For example, in a checked state, the positioning application AP1 performs skeleton tracking processing based on the captured image, and in an unchecked state, the positioning application AP1 omits the skeleton tracking processing. Image-based skeleton tracking methods include “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields” (https://arxiv.org/pdf/1611.08050.pdf), OpenPose disclosed by Zhe Cao et al. Various techniques are known, and they can be widely applied in this embodiment.

FIG. 9C is an example of a screen displayed on the display unit 340 of the second terminal device 300 in step S110 of FIG. For example, a first terminal device 200 that takes an image of the person being cared for, a first terminal device 200 that takes an image of the person's family, and a second terminal device 300 that takes an image of the second user participate in a web conference. You can. In the example of FIG. 9C, the screen corresponding to the person being assisted (for example, user A) is displayed relatively large, and the screens corresponding to the family (for example, user C) and the second user (for example, user B) are displayed relatively small. Is displayed. The screen corresponding to the person being assisted is a real-time captured image of the person being assisted. Note that it may be possible to switch which user's screen is displayed in a larger size. Further, the imaging of the person to be assisted is not limited to being performed in real time, and may also be in a format in which the second user gives advice on a recorded video or the like. Further, the person being cared for and the family member may participate in the web conference from one first terminal device 200. Further, the screen shown in FIG. 9C may be displayed on the display section of another device connected via the network. In addition, various modifications can be made to the specific aspect of the display screen.

Among the screens shown in FIG. 9C, for example, the screen corresponding to the person being assisted includes an area RE1 that displays image information and an area RE2 that displays pressure distribution. In the example of FIG. 9C, an image of the person being assisted lying on the bed 610 captured from the side of the assisted person's feet is displayed in the area RE1. The image is captured using, for example, the imaging unit 260 of the terminal device 200-1 in FIG. 7. For example, in the setting mode, a moving image captured using the imaging unit 260 of the terminal device 200-1 may be displayed in the area RE1 in real time. Naturally, an image captured by the terminal device 200-2 may be displayed in the area RE1, or an image captured by the first terminal device 200 placed at another position may be displayed in the area RE1.

As described above, the information processing system 10 of this embodiment may perform emotional analysis of the first user. In the example of FIG. 9C, an object EM representing the user's emotion analysis result is displayed as part of the image information corresponding to the captured image. The object EM may be, for example, an illustration expressing an emotion. In the example of FIG. 9C, an object EM is displayed, which is an illustration showing that the person being cared for has favorable feelings. However, the results of the emotional analysis may be displayed using text or the like, and the specific display format can be modified in various ways.

Further, the pressure distribution is information outputted by the bed 610 and the mattress 620 described above, and is, for example, information expressing the pressure value at each position on the surface on which the person to be assisted lies using color shading or the like. For example, in the setting mode, the pressure distribution acquired in time series may be displayed in real time.

The screen shown in FIG. 9C may also include an object OB3 corresponding to the rotation button, an object OB4 corresponding to the algorithm creation button, and pointers PO1 and PO2 corresponding to the first user and the second user, respectively.

Object OB3 is an operation interface for rotating at least one of the image information displayed in region RE1 and the pressure distribution displayed in region RE2. In the example of FIG. 9C, the left side of the screen is toward the head of the person being assisted in both the image information and pressure distribution, so the user viewing the screen of FIG. 9C can easily associate the image information with the pressure distribution. It is. However, in an environment where the positioning application AP1 is used, the direction in which the first terminal device 200 images the person being assisted is not necessarily fixed. Therefore, there is a possibility that the direction of the person being assisted differs depending on the image information and the pressure distribution. In this regard, when the positioning application AP1 receives a selection operation on the rotation button, it is possible to align the direction of the person being assisted by rotating the image information or the pressure distribution.

However, the orientation of the image information and the orientation of the pressure distribution are not limited to being manually changed by the second user, and may be adjusted automatically by the positioning application AP1. For example, a marker such as a two-dimensional barcode (QR code) may be provided in advance at a predetermined position in the range to be imaged by the imaging unit 260. For example, a sticker or the like with a printed marker is pasted on the bedside position. The positioning application AP1 performs a process of detecting the position and direction of a marker from an image captured using the imaging unit 260 of the first terminal device 200. In this way, the direction of the person's head in the image information, etc. can be appropriately detected, so that the direction of the image information and the direction of the pressure distribution can be automatically adjusted.

Further, the object OB3 corresponding to the rotation button may be used for processing to rotate both the image information and the pressure distribution by the same angle. For example, region RE1 and region RE2 are arranged side by side vertically, but by rotating the image information and pressure distribution 90 degrees clockwise or counterclockwise, region RE1 and region RE2 are arranged side by side horizontally. You may migrate. In this way, it becomes possible to arrange various pieces of information in a positional relationship desired by the user using the rotation button. For example, the second user may arrange the regions RE1 and RE2 in a positional relationship that is easy for the second user to see, or may arrange them in a positional relationship that is suitable for explanation to the first user.

Further, pointers PO1 and PO2 are objects that move on the screen in FIG. 9C based on operations by the first user and the second user, respectively. For example, when the first terminal device 200 and the second terminal device 300 are PCs, the pointers PO1 and PO2 may be objects that move based on an operation on a device such as a mouse or a trackball. Further, when the first terminal device 200 and the second terminal device 300 are smartphones, the pointers PO1 and PO2 may be objects that move based on a touch operation on a touch panel.

For example, when the second user wants to explain to the first user about a specific part of the screen displayed in FIG. 9C, the second user may point to the relevant part using the pointer PO2. For example, when explaining that a particular part should be placed at a certain position in the skeleton tracking results, the first user's understanding becomes easier by pointing to the relevant part or target position with the pointer PO2. Pointer PO2 is displayed on both the first terminal device 200 and the second terminal device 300. Similarly, when the first user asks a question to the second user, by pointing the pointer PO1 to a part related to the question, the first user can make it easier for the second user to understand the content of the question. Pointer PO1 is displayed on both the first terminal device 200 and the second terminal device 300. Note that although both the pointer PO1 for the first user and the pointer PO2 for the second user are illustrated in FIG. 9C, one or both may be omitted.

As explained above, by using the screens shown in FIGS. 9A to 9C, the second user can view images of the person being assisted, pressure distribution, and emotions, even if the second user is located far away from the first user. You can refer to the analysis results, and if necessary, you can also view the results of skeleton tracking processing. The results of the skeleton tracking process may be displayed superimposed on the image information in the region RE1, for example, as shown in FIG. 9C.

The second user here is a skilled caregiver, and the second user can determine whether or not the person being assisted is in an appropriate posture by referring to various information displayed on the screen in FIG. 9C. judge. The appropriate posture here may be a posture suitable for suppressing bedsores, a posture suitable for changing a diaper, or another posture. For example, a user who is a skilled assistant determines whether or not the posture is suitable for the patient based on his or her own tacit knowledge after considering the patient's attributes and the like.

For example, when the second user determines that the person being assisted has taken an appropriate posture, the second user selects the algorithm creation button shown in the object OB4. The first terminal device 200 transmits the image displayed when the algorithm creation button was operated to the server system 100 as teacher data. In this way, it becomes possible to register positions that an expert considers preferable as teacher data. Note that the teacher data is not limited to image information, and may include pressure distribution or skeleton tracking results. Furthermore, although an example using the terminal device 200-1 has been described above, the present invention is not limited to this, and the terminal device 200-2 may be used to acquire image information. That is, the image information may be an image captured from the side of the person being assisted. Note that, as can be seen from the above description, the algorithm creation button is assumed to be operated by the second user who is an expert, and is not assumed to be operated by the first user. Therefore, the second terminal device 300 may display the object OB4, and the first terminal device 200 may hide the object OB4. For example, the positioning application AP1 may determine whether the logged-in user is the first user or the second user, and may switch the object to be displayed according to the user attribute.

Further, some additional information may be added to the teacher data including image information. FIG. 10 is an example of a screen used when adding additional information to image information. For example, when a user inputs to select image information, the image information is displayed on the screen of FIG. 10 together with the file name. For example, the image information when the algorithm creation button is selected is stored in the storage unit 220 of the first terminal device 200 or the storage unit 120 of the server system 100, and the positioning application AP1 stores the image information. Accepts user input to select one. In the example of FIG. 10, the selected image information is displayed in the area RE3, and an object OB5 representing the file name of the image information is displayed.

Further, as shown in FIG. 10, an object OB6, which is a button for adding a specific mark to image information, may be displayed on the additional information input screen. For example, the specific mark may be a mark provided on a cushion used to encourage the user to take an appropriate posture. Here, a triangle corresponds to a cushion of a predetermined size or less, and a rectangle corresponds to a cushion larger than a predetermined size. For example, if the above-mentioned correspondence relationship has been presented to the second user in advance using the manual of the positioning application AP1, and the second user has placed a cushion to make the user take an appropriate posture, the second user may use the object OB6 to create an image. Perform an operation to add a triangular or square mark to the cushion inside. Note that since this embodiment assumes remote setting, the actual arrangement of the cushions may be performed by the first user (such as a family member of the person being assisted). In this way, it becomes possible to include information specifying the position and size of the cushion in the teacher data.

Further, the additional information may be a comment added by a skilled assistant. For example, on the additional information input screen, object OB7, which is a text box for inputting a comment, may be displayed. The comment here may be any text input using object OB7, which is a text box. For example, a skilled person may need to check the angle of a specific body part with other body parts, the positional relationship with pillows and cushions, the details of the cushion used (size, shape, manufacturer, product number, etc.), etc. in order to help the person take the appropriate bed position. Enter the important points in text.

Further, on the additional information input screen shown in FIG. 10, input of information specifying a part of the person to be cared for that a skilled caregiver considers important may be accepted. For example, if the position of the person being cared for's shoulders is important for ensuring proper posture, a skilled helper may input an input specifying the shoulders of the person being cared for. For example, the positioning application AP1 may accept an input for arranging the triangular or quadrangular mark shown in FIG. 10 at a specific location. Furthermore, a skilled caregiver may consider a plurality of parts to be important. In this case, the positioning application AP1 may accept input of the priority of each position. Further, the additional information may be audio information. For example, on the additional information input screen, object OB8, which is a check box for setting whether to input audio information, may be displayed. For example, when the checkbox is checked, a process of recording the second user's voice may be performed.

Further, the screen shown in FIG. 10 may be able to preview what will happen when it is used. For example, on the additional information input screen, object OB9, which is a button for previewing, may be displayed. Preview here means displaying a screen corresponding to the display screen in the usage mode. Note that the preview display may be performed on the additional information input screen shown in FIG. Alternatively, a pop-up screen different from the input screen of FIG. 10 may be displayed based on the selection operation of the object OB9, and a preview display may be performed on the pop-up screen.

For example, as described above using FIGS. 4A and 4B, when using the positioning application AP1, it is conceivable to superimpose and display transparently processed teacher data and a real-time image of the person being assisted. . Therefore, when a selection operation is performed on the object OB9 in FIG. 10, the positioning application AP1 uses the imaging unit 260 to acquire a captured image, and also performs transparency processing on the teacher data being set for the captured image. superimpose the image. In this way, it becomes possible to check in advance what the display screen will look like when the teacher data being set is actually used. At this time, the second user may be able to set the transmittance of the teacher data using the screen in FIG. It may be possible to select which camera to use. Further, since it is the first user who actually uses the positioning application AP1, the transmittance may be settable by the first user. For example, on the additional information input screen, as shown in FIG. 10, object OB10, which is a text box for inputting transmittance, and object OB11, which is a radio button for selecting a camera, may be displayed.

Furthermore, the teacher data used in the positioning application AP1 is not limited to this. For example, the teacher data may include sensing data corresponding to the timing when the algorithm creation button is selected. Sensing data, which is training data, may be data at one timing when the selection operation of the algorithm creation button is performed, or may be time-series data over a predetermined period including the timing at which the selection operation is performed. good. Further, as described above, a moving image captured by the second user to put the person being assisted in an appropriate posture may be used as the teacher data. Further, the teacher data may include a skeleton tracking result of the second user detected based on the video image.

Through the processing using the above screens, the server system 100 acquires teacher data. In step S113 of FIG. 6, the processing unit 110 of the server system 100 generates an algorithm based on the teacher data. Specifically, the algorithm here is a positioning application AP1 that performs processing based on teacher data. For example, the processing unit 110 may create an algorithm that superimposes and displays image information, which is training data, on an image captured by the imaging unit 260 of the first terminal device 200 using a set transmittance. Alternatively, the processing unit 110 compares the teacher data with the data acquired during actual position adjustment, determines that the posture is appropriate (OK) when the degree of similarity is greater than or equal to the threshold, and determines that the posture is appropriate (OK) when the degree of similarity is less than the threshold. An algorithm may be created that determines that the posture is inappropriate (NG) if this is the case. As mentioned above, the algorithm here may be a new application or update data for an existing application.

Moreover, although the bed 610 and the mattress 620 capable of detecting pressure have been exemplified as the sensing device 400 used for setting and using the positioning application AP1, the present invention is not limited thereto. For example, the output of the sensing device 400 described below may be displayed in area RE2 of FIG. 9C.

FIG. 11 is another example of the sensing device 400 that can be used in the positioning application AP1, and is a diagram illustrating an example of a bedside sensor 420 and a detection device 430 arranged at the bottom of the bed 610. Bedside sensor 420 and detection device 430 are sheet-like or plate-like devices provided between the bottom of bed 610 and mattress 620, as shown in FIG. 11, for example.

Bedside sensor 420 includes a pressure sensor that outputs a pressure value as sensing data, and is disposed on the bottom side of the bed that is used by the caregiver to go up and down the bed 610. In the example of FIG. 11, the caregiver climbs up and down using the front side of the bed 610. At this time, as shown in FIG. 11, a fall prevention fence may be placed on the front side of the bed 610, and the bedside sensor 420 may be placed at a position where the fence is not provided. In this way, the user who goes up and down the bed 610 temporarily sits on the bedside sensor 420. Bedside sensor 420 may output time-series pressure data as sensing data. Alternatively, the bedside sensor 420 may determine the presence or absence of movement by executing the process described below, and may output the determination result as sensing data.

The bedside sensor 420 acquires a pressure value as input data by operating according to an application installed on the bedside sensor 420, for example, and performs a process of determining the movement of the person being assisted on the bed 610 from the pressure value. Execute.

For example, when the person to be assisted gets up from the bed 610, the person to be assisted shifts from a lying position on the bed to a sitting position at the bedside (hereinafter referred to as edge sitting position), Furthermore, it is assumed that the user will perform a standing motion by applying force to the knees or the bottom surface. The pressure value detected by the bedside sensor 420 increases in the order of lying position, sitting position, and standing up position. For example, the bedside sensor 420 may determine that a start of movement has been detected when a change from an edge sitting position to a standing motion is detected based on a process of comparing the pressure value and a given threshold value. Alternatively, from the viewpoint of detecting a rising motion at a faster stage, the bedside sensor 420 may detect a change from a lying position to an edge sitting position based on a comparison process between a pressure value and a given threshold value. It may be determined that it has been detected.

Alternatively, if the standing up motion continues, the buttocks of the person being assisted rise above the bottom surface, and thus the pressure value output from the pressure sensor decreases significantly. Therefore, the bedside sensor 420 performs a rising operation when the pressure value increases to a first threshold value or more and then decreases to a second threshold value or less, which is smaller than the first threshold value, based on a time-series change in the pressure value. It may be determined that In addition, various modifications can be made to the specific processing contents of the motion start determination.

Further, a detection device 430 shown in FIG. 11 is a device that senses information regarding the sleep of the person being assisted. Detection device 430 includes a pressure sensor that outputs a pressure value.

The detection device 430 detects the user's body vibrations (body movements, vibrations) via the mattress 620 when the user goes to bed. Based on the body vibration detected by the detection device 430, information regarding breathing rate, heart rate, amount of activity, posture, wakefulness/sleep, and leaving/staying in bed is obtained. The detection device 430 may also determine non-REM sleep and REM sleep, and determine the depth of sleep. For example, the periodicity of body movements may be analyzed and the respiratory rate and heart rate may be calculated from the peak frequency. The analysis of periodicity is, for example, Fourier transform. Respiration rate is the number of breaths per unit time. Heart rate is the number of heartbeats per unit time. The unit time is, for example, one minute. Alternatively, body vibrations may be detected per sampling unit time, and the number of detected body vibrations may be calculated as the amount of activity. Furthermore, when the user leaves the bed, the detected pressure value is reduced compared to when the user is in bed, so it is possible to determine whether the user is out of bed or in bed based on the pressure value and its chronological changes.

For example, the detection device 430 may output the output of a pressure sensor as sensing data. Alternatively, the detection device 430 may output information regarding the above-mentioned breathing rate, heart rate, amount of activity, posture, wakefulness/sleep, and leaving/staying in bed as sensing data.

The detection device 430 may also make a determination regarding the start of movement of the person being assisted. For example, the detection device 430 determines that the person being assisted has started moving when the person being assisted moves from a state in bed to a state out of bed. Furthermore, from the viewpoint of detecting signs of starting to move at an earlier stage, the detection device 430 may determine that the person being assisted has started moving when the person being assisted transitions from a sleeping state to an awake state. The detection device 430 may output the detection result of the start of movement as sensing data.

2.3 Wheelchair Position FIG. 12 is a diagram illustrating a system configuration when adjusting the wheelchair position. As shown in FIG. 12, in adjusting the wheelchair position, the first terminal device 200 is fixed at a height at which it can image at least the upper body of the person being assisted sitting in the wheelchair 630. The first terminal device 200 is placed, for example, at a predetermined position in a home where home care is provided, and a family member who is a caregiver transfers the person being assisted to a wheelchair 630 and then moves the person to the front of the first terminal device 200. After that, adjust the wheelchair position. However, the process described below may be executed by the family member standing in front of the wheelchair 630 and holding the first terminal device 200 in their hand.

As described above with reference to FIG. 3, the first terminal device 200 is, for example, a device such as a smartphone having an image capturing unit 260 camera. The positioning application AP1 may be an application that supports adjusting the wheelchair position. Adjustment of the wheelchair position may include controlling the posture of the person being assisted to reduce the risk of bedsores or falling, or controlling the placement of cushions.

FIG. 13 is a diagram illustrating the relationship between an application and a device used in a wheelchair position. For example, the positioning application AP1 may acquire image information that is the output of the imaging unit 260 included in the first terminal device 200 and information from at least one of the wheelchair 630 and the seat sensor 440. The wheelchair 630 here is a device that can visualize pressure distribution while being used by a person being assisted. The seat sensor 440 will be described later using FIG. 15.

FIG. 14 is an example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. Further, the screen shown in FIG. 14 may be displayed on the display unit 240 of the first terminal device 200 for confirmation by the first user. For example, the screen shown in FIG. 14 may be displayed in the area corresponding to the person being assisted on the web conference screen, similar to the example shown in FIG. 9C.

The configuration of the screen shown in FIG. 14 is such that the image information displayed in area RE1 is an image of the person being assisted sitting on wheelchair 630, and the pressure distribution displayed in area RE2 is the pressure on the seat surface of wheelchair 630. This is similar to FIG. 9C, except that the information represents the information. Therefore, further explanation of the details of the information displayed on the screen in FIG. 14 will be omitted.

By using the screen shown in FIG. 14, the second user can refer to the captured image of the person being assisted, the pressure distribution, and the results of the emotional analysis, and can also view the results of the skeleton tracking process if necessary. Therefore, for example, the second user, who is a skilled caregiver, can determine whether or not the person being assisted at a remote location is in an appropriate posture by referring to the various information displayed on the screen shown in FIG. do. The appropriate posture here may be a posture suitable for suppressing pressure ulcers, or may be another posture.

Similar to the bed position example, a position considered preferable by an expert can be registered as training data based on the selection operation of the algorithm creation button shown in object OB4. The teacher data is not limited to image information, and may include pressure distribution or skeleton tracking results.

Furthermore, similar to the bed position example, some additional information may be added to the teacher data including image information. For example, the screen described above using FIG. 10 is used to add additional information.

Moreover, although the wheelchair 630 capable of pressure detection has been exemplified above as a device used for setting and using the positioning application AP1, the present invention is not limited to this.

FIG. 15 is another example of the sensing device 400 that can be used in the positioning application AP1, and is a diagram showing a seat sensor 440 disposed on the seat of a wheelchair 630, for example. Seat sensor 440 includes a pressure sensor that outputs a pressure value, and outputs the pressure value.

Further, the seat sensor 440 may be capable of processing other than outputting the pressure value to the positioning application AP1. For example, the seat sensor 440 determines whether the posture of the person being assisted while sitting in the wheelchair 630 (hereinafter also referred to as sitting posture) is normal, forward-shifted, lateral-shifted, etc., based on the pressure value of the pressure sensor. It may also be determined which of the postures it is in. Forward shift refers to a state in which the user's center of gravity has shifted further forward than usual, and lateral shift refers to a state in which the user's center of gravity has shifted further to the left or right than usual. Both forward slippage and lateral slippage correspond to conditions where the risk of falling from the seat is relatively high. Further, the seat sensor 440 may perform a fall possibility determination to determine whether there is a possibility that the person being assisted will fall from the seat.

In the example of FIG. 15, four pressure sensors Se1 to Se4 are arranged on the back side of a cushion 441 arranged on the seat of the wheelchair 630. Pressure sensor Se1 is a sensor placed in the front, pressure sensor Se2 is a sensor placed in the rear, pressure sensor Se3 is a sensor placed on the right, and pressure sensor Se4 is placed on the left. It is a sensor. Note that the front, rear, left, and right directions here refer to directions seen from the person being assisted while the person being assisted is sitting on the wheelchair 630.

As shown in FIG. 15, pressure sensors Se1 to Se4 are connected to a control box 442. The control box 442 includes a processor that controls the pressure sensors Se1 to Se4 and a memory that serves as a work area for the processor. The processor detects pressure values by operating pressure sensors Se1 to Se4.

The person receiving assistance sitting in the wheelchair 630 may feel pain in the buttocks and may shift the position of the buttocks. For example, a state in which the buttocks shift further forward than usual is called anterior shift, and a state in which the buttocks shift from side to side is called lateral shift. In addition, forward and lateral shifts may occur at the same time, causing the center of gravity to shift diagonally. By using the pressure sensor disposed on the cushion 441 as shown in FIG. 15, changes in the position of the buttocks can be appropriately detected, making it possible to accurately detect forward displacement and lateral displacement. For example, the seat sensor 440 determines a forward shift when the value of the pressure sensor Se1 increases by a predetermined value or more compared to the initial state, and determines a lateral shift if the pressure sensor Se3 or Se4 increases by a predetermined value or more compared to the initial state. judge.

The seat sensor 440 may output the pressure values that are the outputs of the pressure sensors Se1 to Se4 as sensing data, or may output the determination results of forward slippage and lateral slippage, the determination results of the possibility of falling, etc. as sensing data. Good too. Further, the control box 442 may include a light emitting section or the like, and the light emitting section may be used to notify the caregiver. In this way, changes in the sitting posture of the wheelchair 630 or the like can be clearly notified to the caregiver, thereby making it possible to prevent the caregiver from falling.

FIG. 16 is another example of the screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The screen shown in FIG. 16 may be displayed on the display unit 240 of the first terminal device 200 for confirmation by the first user.

The screen shown in FIG. 16 is different from the screen shown in FIG. 14 in that it includes a region RE4 and a region RE5 in which the output of the seat sensor 440 is displayed instead of the region RE2 in which the pressure distribution is displayed. For example, region RE4 is a two-dimensional plane in which the front-rear direction of the seat of the wheelchair 630 is the first axis (for example, the horizontal axis), and the left-right direction is the second axis (for example, the vertical axis perpendicular to the horizontal axis) that intersects the first axis. , information obtained by plotting changes in the position of the center of gravity of the person being assisted over time obtained using the pressure sensors Se1 to Se4 is displayed. Further, in the region RE5, the horizontal axis represents time and the vertical axis represents pressure values, and displays information representing time-series changes in the pressure values of the pressure sensors Se1 to Se4.

In the example shown in FIG. 16, similarly to the example shown in FIG. 14, the second user determines whether the first user is in an appropriate posture based on image information and sensing data, and determines that the posture is appropriate. The algorithm creation button may be selected at the appropriate timing. However, the second user's setting input is not limited to this.

For example, the second user may perform an input to select a range of sensing data corresponding to an appropriate posture on the screen of FIG. 16 . In the example of FIG. 16, for example, the second terminal device 300 receives an input of a figure surrounding a part of the two-dimensional plane in the region RE4. In FIG. 16, a figure S is illustrated as an input result by the second user. The figure S here may be drawn freehand, or a preset shape such as a circle or a rectangle may be used. For example, if the second terminal device 300 is a smartphone with a touch panel, the second user inputs a figure S that selects a part of the two-dimensional plane by drawing a line in a part of the area RE4 using a finger or a touch pen. You may. Similarly, the second terminal device 300 may receive an input to select a range that the second user considers to be an appropriate posture from the region RE5. Furthermore, the second terminal device 300 accepts an input for selecting a range that the second user considers to be an appropriate posture from the area RE2 in which sensing data is displayed on the screen described above using FIG. 9C or FIG. You can.

In step S112 in FIG. 6, the second terminal device 300 transmits information specifying the range selected by the second user to the server system 100. The server system 100 sets boundaries for classifying sensing data into a class corresponding to OK and a class corresponding to NG by classifying the information as correct. Note that the processing in the server system 100 may be regression analysis. For example, in the example of FIG. 16, the second user draws the figure S to distinguish between the inside of the figure S (correct answer) and the outside of the figure S (misalignment). Therefore, in this case, the boundary to be set is a boundary that divides the two-dimensional plane into two. However, the second user's input and the number of classifications are not limited to this.

FIG. 17 is an example of a screen that presents information generated by the server system 100 to the second user, and is, for example, a screen displayed on the display unit 340 of the second terminal device 300. The screen in FIG. 17 includes an area RE6 that displays processing results by the server system 100, and an object OB12 that represents a registration button. Here, an example is shown in which a boundary is set to divide a two-dimensional plane into four, with the front-rear direction of the seat of the wheelchair 630 being the first axis and the left-right direction being the second axis. For example, if the second user draws a figure in area RE4 etc. that divides the two-dimensional plane into four areas: a region representing normality, a region representing anterior deviation, a region representing leftward deviation, and a region representing rightward deviation. In this case, the results of four classifications as illustrated in FIG. 17 are often obtained. For example, one of the four regions is a region representing an appropriate posture (OK), and the other three regions among the four regions correspond to an inappropriate posture (NG). This is an area that represents leftward and rightward shifts. When the second user determines that there is no problem with the processing result based on the screen of FIG. 17, the registration button is selected. When the registration button is selected, the server system 100 creates an algorithm that uses information representing the set boundaries as training data. For example, when sensing data from the seat sensor 440 is acquired, the processing unit 110 creates an algorithm that identifies the center of gravity position and determines to which of four regions classified by boundaries the center of gravity position belongs. do. Furthermore, the processing unit 110 may create an algorithm for determining OK/NG based on the determination result of the area to which the center of gravity position belongs.

As described above, also by accepting the second user's operation on the sensing data as the setting input of the teacher data, it becomes possible to appropriately digitize and utilize the second user's tacit knowledge.

2.4 Meal Position Next, adjustment of the meal position will be explained. In addition, below, the case where a person eats while riding in a wheelchair 630 will be illustrated. Therefore, regarding the eating position, settings for the wheelchair position may be performed using, for example, the wheelchair 630 or the seat sensor 440 that can detect pressure. However, since it is conceivable that a meal may be eaten while lying down on the bed 610, the bed position using the bed 610, mattress 620, bedside sensor 420, detection device 430, etc. that can detect pressure, for example, is considered. settings may also be made.

FIG. 18 is a diagram illustrating a system configuration when adjusting the wheelchair position. As shown in FIG. 18, during meals, a first terminal device 200 and a swallowing discomfort detection device 460, which is a sensing device 400 that detects various information related to meals, such as swallowing and discomfort of the person being cared for, are used. It's okay to be hit. As shown in FIG. 18, the swallowing awkwardness detection device 460 includes a throat microphone 461 that is worn around the neck of the person being assisted, and a terminal device 462 that has a camera.

The throat microphone 461 outputs audio data caused by swallowing, coughing, etc. of the care recipient. The camera of the terminal device 462 outputs a captured image of the person being assisted while eating. The terminal device 462 is, for example, a smartphone or a tablet PC placed on the table where the person being assisted eats. The throat microphone 461 is connected to a terminal device 462 using Bluetooth (registered trademark) or the like, and the terminal device 462 is connected to the server system 100 via a network. However, both the throat microphone 461 and the terminal device 462 may be directly connectable to the server system 100, and the specific connection mode can be modified in various ways.

As shown in FIG. 18, in adjusting the eating position, the first terminal device 200 may be placed on the side of the person being assisted. The imaging unit 260 of the first terminal device 200 outputs a first captured image of the side of the person being assisted.

FIG. 19 is an example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The configuration of the screen shown in FIG. 19 is such that, instead of the area RE1 that displays an image taken from the front of the person being cared for sitting in the wheelchair 630, an area RE7 that displays an image taken from the side of the person being cared for is used. It is the same as FIG. 16 except for. Therefore, further explanation of the details of the information displayed on the screen in FIG. 19 will be omitted. Note that in area RE7 of FIG. 19, tables, dishes, etc. are omitted.

By using the screen shown in FIG. 19, the second user can refer to the captured image of the person being assisted, the results of emotional analysis, and the sensing data, and can also view the results of the skeleton tracking process if necessary. Therefore, for example, the second user, who is a skilled caregiver, can determine whether or not the person being assisted at a remote location is in an appropriate posture by referring to the various information displayed on the screen shown in FIG. do. The appropriate posture here may be a posture suitable for eating, or may be another posture. A posture suitable for eating is, for example, a posture that can suppress stuffiness and aspiration.

Similar to the example of the bed position and wheelchair position, a position considered preferable by an expert can be registered as teacher data based on the selection operation of the algorithm creation button shown in object OB4. Furthermore, similar to the example of the bed position and wheelchair position, some additional information may be added to the teacher data including image information. For example, the screen described above using FIG. 10 is used to add additional information.

Further, the teaching data in the eating position is not limited to the above example. For example, to prevent stiffness and aspiration when eating, consider making sure that your back is as straight as possible, that your jaw is tucked in, and that you are not leaning too far forward. Therefore, for example, the second user may input a setting indicating whether or not the position of the assisted person's head is appropriate.

FIGS. 20A and 20B are examples of teacher data acquired by the second user's setting input. For example, the teacher data for the eating position may be a boundary dividing an image of the person being assisted from the side. The boundary here is, for example, a straight line. In the example of FIG. 20A, the image is divided into four regions by setting two straight lines. In an example in which this teacher data is used, if the head of the person being assisted is located in the upper right area, it is determined to be OK, and if it is located in any other area, it is determined to be NG. However, the number of straight lines is not limited to two, and may be one as shown in FIG. 20B. In the example of FIG. 20B, the image is divided into two regions by setting one straight line. In an example in which this teacher data is used, if the head of the person being assisted is located in the right region, it is determined to be OK, and if it is located in the left region, it is determined to be NG. Further, three or more straight lines may be set, or a curved line may be used as a boundary for dividing an image.

Furthermore, when using the teacher data in FIG. 20A, skeleton tracking may also be used. The positioning application AP1 may determine OK/NG depending on which area a certain part (for example, neck, nose, head, etc.) of the skeleton tracking result is located. Alternatively, the positioning application AP1 compares a partial image belonging to a certain region (correct region in a narrow sense) of the teacher data with a partial image belonging to the same region of the captured image at the time of position adjustment, and compares In some cases, it may be determined as NG. For example, the norm calculated from the RGB pixel values as described above may be used for image difference determination. Moreover, when using the teacher data shown in FIG. 20A, determination of OK/NG is not essential. For example, the positioning application AP1 may display data indicating boundaries included in the teacher data (two straight lines in FIG. 20A) superimposed on the captured image. In this case, the first user who viewed the superimposed image can adjust his or her posture appropriately based on the positional relationship between the specific region and the boundary. In this way, various modifications can be made to the process using the teacher data in FIG. 20A. The same applies when using the teacher data in FIG. 20B.

For example, the second user may set the straight line shown in FIGS. 20A and 20B by drawing a line using a finger or a touch pen in the area RE7 of the screen in FIG. 19. Alternatively, the second user may determine OK/NG for each of the plurality of captured images, and input the determination result to the second terminal device 300 as a setting input. The server system 100 may perform a process of automatically setting the straight line shown in FIG. 20A etc. by performing classification or regression analysis in which the setting input is considered as the correct answer. Alternatively, as in the example of FIG. 16, the second user inputs a setting to select a part of the sensing data that he or she considers preferable, and by performing processing based on the setting input, the image shown in FIG. 20A etc. The straight line shown may be found.

FIG. 21 is another example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The first terminal device 200 may include a device that images the person being assisted from the front and a device that images the person being assisted from the side. For example, the terminal device 462 of the dysphagia detection device 460 in FIG. 18 may operate as the first terminal device 200. Alternatively, one of a device that images the front side of the person to be assisted and a device that images the side surface of the person to be assisted operates as the first terminal device 200, and the other device transmits the captured image to the first terminal device 200. You may also perform processing to do so.

The screen shown in FIG. 21 may include an area RE1 that displays an image of the person being assisted from the front, and an area RE7 that displays an image of the person being assisted from the side. In this way, the amount of information that can be presented to the second user increases, making it possible to improve the accuracy of the teacher data. Note that other points are the same as those in FIGS. 16 and 19, so detailed explanation will be omitted.

Next, details of the swallowing discomfort detection device 460 will be explained. For example, the swallowing awkwardness detection device 460 determines the assistance recipient's awkwardness and swallowing based on audio data from the throat microphone 461. A device that detects swallowing using a microphone worn around the neck is described, for example, in U.S. Patent Application No. 16/276,768 entitled “Swallowing action measurement device and swallowing action support system” filed on February 15, 2019. ing. This patent application is incorporated herein by reference in its entirety. Based on the audio data, the processor can detect the number of times the patient has a choking, the time of the choking (time of occurrence, duration, etc.), and whether or not the patient has swallowed. For example, the output of the swallowing choking detection device 460 may be output as sensing data and displayed on a screen such as FIG. 19 or FIG. 21. Furthermore, the swallowing discomfort detection device 460 is capable of obtaining various information regarding meals as described below. In this embodiment, this information may be output as sensing data. For example, any of the information described below may be displayed in the area RE4 or RE5 of FIG. 19. For example, a waveform signal representing the sound detected by the throat microphone 461, an average value of the swallowing time described below, etc. may be displayed on the screen shown in FIG. 19.

Furthermore, the camera of the terminal device 462 can detect the mouth, eyes, and chopsticks, spoons, etc. of the person being assisted by capturing an image of the person being assisted from the front, as shown in FIG. 18, for example. Note that there are various known methods for detecting these facial parts and objects based on image processing, and in this embodiment, a wide variety of known methods can be applied.

For example, the swallowing awkwardness detection device 460 can determine whether the mouth of the person to be assisted is open, whether food is coming out of the mouth, and whether or not the person is chewing the food, based on an image captured by a camera. Furthermore, the swallowing awkwardness detection device 460 can determine whether or not the eyes of the person being assisted are open based on the image captured by the camera. Furthermore, the swallowing awkwardness detection device 460 can determine whether chopsticks, spoons, etc. are near tableware, whether the person being assisted is holding them, and whether food has been spilled. .

In the method of this embodiment, the situation regarding swallowing and stiffness of the care recipient is estimated based on this information. For example, the swallowing awkwardness detection device 460 may obtain information regarding meals based on the detection results of the awkwardness and swallowing, and the open/closed mouth opening/closing determination results of the person being assisted.

For example, the swallowing discomfort detection device 460 may determine whether or not the discomfort occurs frequently based on the number of times the discomfort occurs or the time, and may output the determination result. For example, the swallowing discomfort detection device 460 may determine that the swallowing discomfort occurs frequently when the number of times the swallowing discomfort occurs exceeds a threshold value. In this way, it is possible to automatically determine the situation regarding the mouse.

Furthermore, the swallowing awkwardness detection device 460 may determine the swallowing time from when the person being assisted opens their mouth to when they swallow, based on the detection result of swallowing and the result of determining whether the assisted person's mouth is open/closed. If you do this, for example, if the number of times you swallow is decreasing, you can find out if the action of putting food in your mouth is not being performed, or if you are not swallowing even after putting food in your mouth. Can judge the situation. For example, the swallowing awkwardness detection device 460 starts counting up a timer when the mouth changes from a closed state to an open state based on the captured image of the terminal device 462, and when swallowing is detected by the throat microphone 461. The timer measurement may be stopped. The time at stop represents the swallowing time. In this way, it is possible to accurately determine whether the situation is such that the risk of aspiration during meals is high and the caregiver should take some action.

Furthermore, the swallowing choking detection device 460 may determine the pace of the meal based on the swallowing time. Furthermore, the swallowing choking detection device 460 determines whether or not the swallowing time is long based on changes in the swallowing time during one meal (for example, the amount of increase or ratio to the initial swallowing time, etc.). Good too. Alternatively, the processor may obtain the average swallowing time for each of a plurality of meals for the same person being assisted, and determine whether the swallowing time has become longer based on a change in the average swallowing time.

Furthermore, by using the result of mouth opening/closing determination based on the captured image of the terminal device 462, it is possible to determine whether the caregiver is unable to open the mouth even when a spoon or the like is brought close to the mouth. In this way, if the person being assisted is reluctant to open their mouth and the swallowing time becomes long, it can be assumed that food is left in the mouth and congestion has occurred. Furthermore, by using the captured image to determine whether food is coming out of the mouth and whether the person is chewing the food, it can be determined whether the person being assisted is unable to chew the food. For example, if the number of times you chew is normal, but it takes a long time to swallow, it is assumed that you are unable to chew the food enough. Furthermore, if it is determined using the captured image that the eyes are closed, it can be determined whether the person being assisted is in a sleepy state.

Furthermore, by performing recognition processing for chopsticks, spoons, etc. using captured images, it may be determined whether the person is playing with food, cannot hold a utensil, or is not doing anything. For example, if an object such as a spoon is placed on the person's hand, but the time it takes to bring the object to the person's mouth is longer than a predetermined threshold, it is determined that the recipient is unable to hold the tableware or is playing with the food. . In addition, if an object such as a spoon does not overlap the care recipient's hand, and the time the care recipient's line of sight is directed towards the food (cooking) is longer than a predetermined threshold, the care recipient looks at the food without doing anything. It is determined that

Further, it is known that the feature amount of the audio data detected by the throat microphone 461 changes depending on various situations. The feature amount here may be the amplitude in the waveform of the audio data, the frequency, or the number of waves included in one swallow. For example, the sound of swallowing changes depending on factors such as the hardness and consistency of the food and the pace of eating. It has also been found that the sound of swallowing changes depending on the physical condition of the person being assisted, even when the hardness is the same.

Therefore, the swallowing discomfort detection device 460 may obtain a feature representing the sound of swallowing based on the output of the throat microphone 461, and may estimate the physical condition of the person being assisted based on the feature. For example, the swallowing discomfort detection device 460 stores data that associates the feature amount of the swallowing sound with the physical condition of the person being assisted based on the past history. Then, the swallowing discomfort detection device 460 estimates the physical condition of the person being assisted by comparing the data with the actually acquired feature amount. Note that the data that associates the feature amount of the swallowing sound with the physical condition of the person being assisted may be a trained model created by machine learning.

3. Use of Positioning Application As described above, in the setting mode, the algorithm (update data or new positioning application AP1) based on the second user's setting input is transmitted to the first terminal device 200. Therefore, the first terminal device 200 can operate the positioning application AP1 in the usage mode. As described above, the positioning application AP1 in the usage mode may perform a process of transparently superimposing image information, which is teaching data, or adjust the posture of the person being assisted based on a comparison process of image information and sensing data. OK/NG may be determined. The positioning application AP1 may also determine whether the posture of the caregiver (family member of the person being assisted) is OK or NG. When simply using the positioning application AP1 stored in the first terminal device 200, the second user does not need to intervene using the second terminal device 300 in the usage mode.

However, even when the same care recipient is targeted, suitable teacher data may differ depending on the situation. Therefore, if the first terminal device 200 is capable of executing processing corresponding to a plurality of teacher data, it is necessary to select the teacher data to be used. Further, the information processing system 10 of the present embodiment has accumulated a large amount of teacher data (algorithms), and the teacher data set for the first person being assisted is a second person being different from the first person being assisted. There may also be cases where the person is suitable as a caregiver. For example, when the attributes of the first person being assisted and the second person being assisted are similar, there is a possibility that the teacher data that is suitable for one person is also suitable for the other person.

Therefore, the second terminal device 300 may accept a selection operation by the second user, which is an operation for selecting teacher data to be used from a plurality of pieces of teacher data. As mentioned above, the teacher data is the tacit knowledge of a skilled caregiver, and the selection of which tacit knowledge to use in which situations can also be tacit knowledge. According to the method of this embodiment, since judgment by the second user, who is an expert, is also used to select tacit knowledge, it is possible to realize position adjustment suitable for the first user. For example, in the use mode, the positioning application AP1 may display a plurality of tabs on a transparent screen shown in FIG. 4B, and perform a process of switching the teacher data based on the selection operation of the tab. For example, one tab may correspond to one teacher data, and the teacher data may be selected based on manual switching of tabs by the second user.

FIG. 22 is a sequence diagram illustrating a process when teacher data (tacit knowledge, algorithm) is selected using the second terminal device 300. The process in FIG. 22 is executed, for example, before starting specific position adjustment.

First, in step S201, the first terminal device 200 and the second terminal device 300 are connected. In step S202, the first terminal device 200 transmits information regarding the surrounding sensing devices 400 to the second terminal device 300. The processes in steps S201 and S202 are similar to steps S101 and S102 in FIG.

In step S203, the second terminal device 300 receives a selection input of a device to be connected by the second user. This process is similar to step S103 in FIG. Further, in step S203, the second terminal device 300 receives an input for selecting an algorithm (teacher data) to be used.

For example, the second terminal device 300 may obtain a list of algorithms registered in the server system 100 by a process not shown in FIG. 22, and may perform a process of displaying the list. For example, the server system 100 performs a process of transmitting to the second terminal device 300 a list of algorithms that are executable using the sensing device 400 installed around the first terminal device 200 among the algorithms registered in the storage unit 120. You may do so. The second user performs an operation to select an algorithm suitable for the person being assisted from the list according to his or her tacit knowledge.

In step S204, the second terminal device 300 transmits the connection information and information specifying the algorithm to the first terminal device 200. The connection information is the same as the example described above in step S104 of FIG. The information specifying the algorithm may be, for example, an ID uniquely assigned to the algorithm. Further, the information specifying the algorithm may include information such as the name of the algorithm, the attributes of the person being assisted at the time of creation, the ID of the second user who created it, the assistance history, the qualifications held, the affiliated facility, and the like.

In step S205, the first terminal device 200 establishes a connection with the sensing device 400 based on the connection information. The process in step S205 is similar to the process in step S105 in FIG.

In step S206, the first terminal device 200 performs a process of identifying the algorithm to be used based on the information received in step S204. In step S207, it is determined whether the identified algorithm has been installed on the first terminal device 200. For example, the first terminal device 200 may perform a process of comparing the positioning application AP1 stored in the storage unit 220 with the identified algorithm. This process may be a determination as to whether the same positioning application AP1 is installed. Furthermore, if the algorithm is implemented as update data (for example, an add-in) for the positioning application AP1, the process in step S206 may be a determination as to whether an add-in corresponding to the identified algorithm has been installed. .

If it is determined in step S207 that the algorithm selected by the second user has been installed, the processes after step S208 are omitted, and the process according to the algorithm is executed.

If it is determined in step S207 that the algorithm selected by the second user has not been installed, the first terminal device 200 requests the algorithm from the server system 100 in step S208. In step S209, the server system 100 transmits the requested algorithm to the first terminal device 200. In step S210, the first terminal device 200 installs the transmitted algorithm. This enables the first terminal device 200 to execute processing according to the algorithm.

Note that the above has described an example in which sensing data from the sensing device 400 is used also in the usage mode of the positioning application AP1. However, the positioning application AP1 may be an algorithm that allows image information representing training data to be transmitted through and displays it in a superimposed manner, or determines OK/NG by comparing the image information and a captured image. , the use of sensing device 400 is not essential. In this case, the processes of steps S202, S204, and S205 related to the sensing device 400 can be omitted. Further, in step S203, the process of accepting input of the connection target device can also be omitted.

4. Change of Operation Mode Based on Load In the above, an example has been described in which the first terminal device 200 establishes a connection with the sensing device 400. That is, sensing data from sensing devices 400 placed at home or the like is aggregated into the first terminal device 200. If the first terminal device 200 is in the setting mode, the first terminal device 200 remotely sets the teacher data by transmitting data including the sensing data to the second terminal device 300. In addition, in the usage mode, the first terminal device 200 acquires sensing data from the sensing device 400 and performs processing (for example, comparison processing with teacher data, etc.) according to the sensing data. However, if the processing load on the first terminal device 200 or the communication load related to transmission and reception of sensing data is large, the processing on the first terminal device 200 may be delayed.

Therefore, the first terminal device 200 compares the processing load of the positioning application AP1 and the communication load of the sensing result with the processing capacity of the first terminal device 200, and determines the operation mode of the first terminal device 200 based on the comparison result. You may also perform processing to switch between the two. Note that the process described below may be considered as a process of switching the operation mode of the sensing device 400 (a process of switching the destination of sensing data). An example in the setting mode and an example in the usage mode will be explained below.

FIG. 23A is a sequence diagram illustrating switching of operation modes in setting mode. First, in step S301, the first terminal device 200 calculates the processing load of the positioning application AP1. The processing load of the positioning application AP1 may be set in advance based on the algorithm of the positioning application AP1, or may be determined based on the CPU usage amount and memory usage amount at the time of execution.

Further, in step S301, the first terminal device 200 acquires the communication load of sensing data. If the algorithm is known, the type of sensing data to be communicated and the expected amount of data to be communicated per unit time are known. Therefore, the communication load may be set in advance based on an algorithm. Alternatively, the first terminal device 200 may determine the communication load based on the actual communication log.

Further, in step S301, the first terminal device 200 acquires the processing capabilities of the processing section 210 and the communication section 230. The processing capacity may be set in advance based on the configuration of the first terminal device 200 or communication equipment (wireless LAN access point, etc.). Alternatively, the first terminal device 200 may dynamically determine the processing capacity by considering the processing load of applications running other than the positioning application AP1.

If it is determined in step S301 that the processing load and communication load do not exceed the processing capacity of the first terminal device 200, the processes of steps S302 to S307 are executed.

In step S302, a connection between the first terminal device 200 and the sensing device 400 is established. For example, in step S302, processing similar to steps S102 to S105 in FIG. 6 is executed. Further, the subsequent processing is similar to S107 to S110 in FIG. Specifically, in step S303, the sensing device 400 transmits sensing data to the first terminal device 200. In step S304, the first terminal device 200 acquires a captured image of the person being assisted. In step S305, the first terminal device 200 performs emotional analysis of the first user. In step S306, the first terminal device 200 transmits the first captured image, sensing data, and emotion analysis results to the second terminal device 300. The second terminal device 300 displays a screen such as that shown in FIG. 9C based on the received information. As described above, if there is enough processing capacity for the load, the same process as in FIG. 6 may be executed.

On the other hand, if it is determined in step S301 that the processing load and communication load exceed the processing capacity of the first terminal device 200, the processes of steps S308 to S314 are executed.

First, in step S308, a connection between the sensing device 400 and the server system 100 is established. For example, if it is determined in step S301 that the processing load and communication load exceed the processing capacity, the first terminal device 200 directly or via the server system 100 or the like instructs the sensing device 400 about the destination of the sensing data. An instruction to switch may be given to the server system 100. For example, the sensing device 400 connects to the server system 100 by connecting to a wireless LAN access point located at home. The connection here may be supported by the second user as in the example of FIG.

In step S309, the sensing device 400 acquires sensing data and transmits the sensing data to the server system 100. In step S310, the server system 100 transmits sensing data to the second terminal device 300.

The processing in steps S311 and S312 is similar to steps S304 and S305. In step S306, the first terminal device 200 transmits the captured image and the emotion analysis result to the second terminal device 300.

Through the above processing, the second terminal device 300 can acquire sensing data from the server system 100, and can acquire captured images and emotional analysis results from the first terminal device 200. Therefore, in step S314, the second terminal device 300 displays a screen such as that shown in FIG. 9C based on the received information.

When the processes in steps S308 to S314 are executed, unlike the processes in steps S302 to S307, the server system 100 can receive sensing data from the sensing device 400 and transmit it to the second terminal device 300. Furthermore, when sensing data is stored in the server system 100 as log data, transmission of the log data from the first terminal device 200 to the server system 100 can also be omitted. Therefore, the load on the first terminal device 200 can be reduced.

Furthermore, in the method of this embodiment, the sensing device 400 that is not used for setting tacit knowledge may operate. For example, in the method of this embodiment, if there is enough processing capacity for the load, both the sensing device 400 used for tacit knowledge setting and the sensing device 400 not used are connected to the first terminal device 200. mode may be used. In addition, when there is not enough processing capacity to handle the load, a second mode is used in which the sensing device used for tacit knowledge setting is connected to the first terminal device 200 and the other sensing devices 400 are connected to the server system 100. It's okay to be hit. Furthermore, even if the sensing device 400 not used for tacit knowledge setting is directly connected to the server system 100, if the load exceeds the processing capacity, the sensing device 400 used for tacit knowledge setting and the sensing device not used for tacit knowledge setting 400 may be connected to the server system 100 without going through the first terminal device 200.

Furthermore, the mode switching is not limited to the two steps shown in FIG. 23A or the three steps using the first to third modes described above. For example, a priority is set for each sensing device, and if the processing capacity is insufficient for the load, the sensing device may be switched to direct connection to the server system 100 in order of priority. For example, the sensing device 400 that is not used for tacit knowledge setting is set to have a lower priority than the sensing device 400 that is used for tacit knowledge setting. By using the priority, it becomes possible to flexibly switch between the sensing device 400 connected to the first terminal device 200 and the sensing device 400 directly connected to the server system 100. Note that this priority itself is tacit knowledge, and the priority of each sensing device 400 may be changeable by the second user. As described above, various modifications can be made regarding which sensing device 400 is connected to which equipment.

FIG. 23B is a sequence diagram illustrating an example of operation mode switching processing in the usage mode. The process in step S401 is similar to step S301 in FIG. 23A.

If it is determined in step S401 that the processing load and communication load do not exceed the processing capacity of the first terminal device 200, the processes of steps S402 to S406 are executed.

In step S402, the sensing device 400 acquires sensing data and transmits the sensing data to the first terminal device 200. In step S403, the first terminal device 200 makes a determination based on the sensing data. For example, the first terminal device 200 performs a comparison process between teacher data and sensing data, and performs an OK/NG determination of the posture based on the processing result. In step S404, the first terminal device 200 uses the imaging unit 260 to obtain a first captured image of the first user.

In step S405, the first terminal device 200 performs display processing and the like according to the positioning application AP1. For example, the first terminal device 200 displays superimposed teaching data on the captured image, and displays OK/NG determination results based on sensing data and the like. Although not shown in FIG. 23B, the first terminal device 200 may perform skeletal tracking and emotional analysis, and display determination results based on these.

In step S406, the first terminal device 200 transmits the processing result to the second terminal device 300. This makes it possible to notify the second user of the position adjustment results. Although not shown in FIG. 23B, the first terminal device 200 may transmit the processing result to the server system 100. In this case, it becomes possible to accumulate the position adjustment results in the server system 100 as log data.

On the other hand, if it is determined in step S401 that the processing load and communication load exceed the processing capacity of the first terminal device 200, the processes of steps S407 to S412 are executed.

In step S407, the sensing device 400 acquires sensing data and transmits the sensing data to the server system 100. In step S408, the server system 100 makes a determination based on the sensing data. For example, the server system 100 performs OK/NG determination based on teacher data, sensing data, and the like. As described above, since the storage unit 120 of the server system 100 stores the positioning application AP1 (algorithm), it is possible to execute the process of step S408. In step S409, the server system 100 transmits the processing result to the first terminal device 200. The information transmitted in step S409 is, for example, flag data indicating either OK or NG.

In step S410, the first terminal device 200 uses the imaging unit 260 to acquire a first captured image of the first user. In step S411, the first terminal device 200 performs display processing etc. according to the positioning application AP1. For example, the first terminal device 200 displays superimposed teaching data on the captured image, and displays OK/NG determination results based on sensing data and the like. In step S412, the first terminal device 200 transmits the processing result to the second terminal device 300.

When the processing in steps S407 to S412 is executed, unlike the processing in steps S402 to S406, the server system 100 can receive sensing data and perform determination processing based on the sensing data. Therefore, the processing load and communication load on the first terminal device 200 can be reduced.

Although FIGS. 23A and 23B are not shown, when the processing load of emotional analysis or skeleton tracking is large, the first terminal device 200 transmits the captured image to the server system 100, and the server system 100 processes the emotional analysis etc. You may do so. In this case, although the load associated with communication of captured images increases, it becomes possible to reduce the processing load associated with emotion analysis and the like.

Note that FIGS. 23A and 23B are examples of switching the operation mode of the first terminal device 200, and the specific processing in cases where the processing load and communication load do not exceed the processing capacity and when they exceed the processing capacity is the same as in the example described above. Various modifications are possible without limitation.

5. Reliability The information processing system 10 of the present embodiment may determine the reliability of the second user from the first user based on the emotional analysis result of the first user using the first captured image. The process of determining reliability may be performed by any of the first terminal device 200, the second terminal device 300, and the server system 100, or may be realized by distributed processing of two or more devices. An example in which the first terminal device 200 performs sentiment analysis will be described below.

For example, in emotion analysis, each emotion may be associated with information indicating whether the emotion is favorable or negative. For example, emotions such as joy and enjoyment are positive, while anger and sadness are negative. If the first terminal device 200 determines that the first user has favorable feelings through emotion analysis, the first terminal device 200 determines that the first user has a high degree of trust in the second user. Further, if the first terminal device 200 determines that the first user has negative emotions through emotional analysis, it determines that the first user has low trust in the second user. Further, a reliability score indicating the level of reliability may be associated with each emotion, and the reliability may be calculated based on the reliability score.

For example, assume that in the setting mode, a second user using the second terminal device 300 is instructing a first user, who is a person being assisted and his or her family, about posture. If the first user's feelings at this time are favorable, it is considered that the first user views the second user's instruction content favorably, and the degree of trust in the second user is high. On the other hand, if the first user's feelings in the setting mode are negative, the first user may not be happy with the second user's guidance, and the degree of trust in the second user is considered to be low.

Furthermore, emotion analysis is not limited to that executed in the setting mode. For example, the first user and the second user may have a conversation, video conference, etc. using the first terminal device 200 and the second terminal device 300 in order to provide feedback in the usage mode. At this time, also when the first user's emotions are analyzed, it is possible to estimate the degree of trust that the first user has in the second user.

Based on the result of the emotion analysis, the manner in which the second user is involved in assistance regarding the first user may be changed. Examples of each of the setting mode and usage mode will be explained below.

For example, when there are a plurality of second users, the reliability from the first user of the first terminal device 200 may be determined for each second user based on the result of sentiment analysis. For example, it is conceivable that a plurality of second users having different roles, such as a caregiver, a nurse, a doctor, and a care manager, may be involved as experts in charge of one person being assisted. It is also conceivable that multiple caregivers may be in charge of one person being cared for. In this case, the reliability from the first user is determined for each second user.

In the second user's setting input for setting the teacher data, a second user with a high degree of reliability may be given priority over a second user with a low degree of reliability. For example, consider a situation in which a first user requests a second user to set teacher data for making some kind of position adjustment. At this time, the first terminal device 200 performs a process of preferentially selecting a user with high reliability from among the plurality of second users associated with the first user. For example, in step S101 of FIG. 6, the first terminal device 200 may attempt connection in order from the second terminal device 300 of the second user with the highest reliability. In this way, there is a high probability that the second user with high reliability sets the teacher data, so that tacit knowledge that is more suitable for the first user can be digitized.

Further, in the usage mode, teacher data set by a second user with high reliability may be used preferentially compared to teacher data set by a second user with low reliability. As described above using FIG. 22, in the method of this embodiment, it is assumed that there are a plurality of second users, and each second user registers his or her own tacit knowledge as training data (positioning application, algorithm). Ru. Therefore, the selection of tacit knowledge to use is also important.

In this regard, by using reliability as a criterion for selecting tacit knowledge, it becomes possible to preferentially use the tacit knowledge of the second user who is considered to be compatible with the first user. For example, the selection input of the algorithm to be used in step S203 in FIG. 22 may be omitted, and the second terminal device 300 or the server system 100 may automatically select the teacher data to be used based on the reliability. Alternatively, the selection of the algorithm itself is performed by the second user, but when displaying a list of candidate algorithms, the algorithm created by the second user that has a high degree of trust from the first user is displayed preferentially. good. In addition, various modifications can be made to the specific processing.

Note that in the field of assistance, there may be a situation in which a second user with a low degree of trust must be involved with the first user. For example, even if a doctor's reliability is low, if a doctor's diagnosis is required for some kind of response, it is difficult for a caregiver or nurse to handle the situation on his or her behalf.

Therefore, in the method of the present embodiment, if the reliability of the second user from the first user of the first terminal device 200 is determined to be less than or equal to a predetermined threshold based on the result of emotion analysis, the first terminal device 200 and the second user At least one of the two terminal devices 300 transmits information that encourages the participation of a family member of the care recipient or a second user with a relatively high degree of trust during communication between the first terminal device 200 and the second terminal device 300. May be presented.

For example, if the person being cared for has low trust in the doctor, the information processing system 10 may require the intervention of a caregiver, nurse, care manager, etc., instead of having a conversation or video conference between the person being cared for and the doctor alone. Communication may be facilitated by making suggestions. Alternatively, the information processing system 10 may facilitate communication between the person being cared for and the doctor by suggesting the intervention of the person's family. For example, a proposal to involve a caregiver or the like may be made on the second terminal device 300, and a proposal to involve the care recipient's family may be made on the first terminal device 200. However, the presentation method can be modified in various ways, such as suggesting intervention by a caregiver or the like using the first terminal device 200.

Furthermore, if there is a second user with low reliability, a proposal may be made to change the second user to another user. For example, the second user is assumed to be a user with some kind of qualification, such as a physical therapist, occupational therapist, speech therapist, or the like. Therefore, if there is a second user who has a low degree of trust from the first user, the information processing system 10 proposes another user who has the same qualifications as the second user to the first user as a new second user. Processing may be performed. For example, the server system 100 maintains a database of experts who are candidates for the second user for each qualification, and proposes a new second user based on the database. In this way, even if the second user is changed, the qualifications held by the second user can be maintained. In the case of assistance, which specialist should be supported depending on the situation, such as cases where the knowledge of a physical therapist is effective, cases where the knowledge of an occupational therapist is effective, cases where the knowledge of a speech-language hearing therapist is effective, etc. Although the details are different, if the above method is used, it will be possible to receive appropriate support after the change.

Furthermore, if there is a second user with high reliability and other second users have low reliability, there is a proposal to change the second user with low reliability to a person related to the second user with high reliability. May be done. For example, if a doctor has a high degree of trust but a care manager has a low degree of trust, the care manager may be changed to another care manager related to the doctor. If they are related parties, there is a high probability that their assistance policies will be similar to some extent, so by changing the second user to a highly reliable related person, the relationship between the second user and the first user can be maintained in a good manner. becomes possible. Note that when changing a second user with low reliability, a determination may be made based on qualifications as described above. For example, if the care manager is to be changed, a new second user is selected from among the persons who have the qualifications of care manager. The same applies when the user with low reliability is a physical therapist, occupational therapist, speech therapist, etc.

6. Remote setting of tacit knowledge in applications other than positioning applications In the above, a method for remotely setting tacit knowledge used in the positioning application AP1 has been described. For example, the training data in the positioning application AP1 is the tacit knowledge of the second user, and according to the above-mentioned method, even when the second user is in a location far from the first user, the tacit knowledge can be appropriately digitized. It is possible to do so.

However, the object on which the remote setting of tacit knowledge is performed is not limited to the positioning application AP1, but may be other devices or applications. For example, the method of the present embodiment includes an electronic device that acquires sensing results regarding at least one of a person and an object during assistance, and a second terminal used by a second user who supports the assistance regarding the first user using the electronic device. The present invention may be applied to the information processing system 10 including the device 300. The electronic device transmits the sensing result to the second terminal device 300 via the network. The second terminal device 300 presents the sensing result to the second user, and receives the second user's input of settings for the electronic device. The setting input here is information representing the second user's tacit knowledge regarding the operation of the electronic device. The electronic device then obtains the setting input and operates according to the setting input.

The electronic device here may be, for example, any of the sensing devices 400 described above. For example, the seat sensor 440 may determine a plurality of states including forward shift, right shift, left shift, and normal as described above. The determination process by the seat sensor 440 may be executed as a part of the operation in the positioning application AP1 as described above, but it may also be executed independently from the positioning application AP1. For example, the seat sensor 440 may perform a process of determining front shift or the like based on the pressure value, and notifying the determination result with a notification unit (for example, a light emitting unit such as an LED) provided in the control box 442 or the like. Note that the determination based on the pressure value may be performed by a processor included in the control box 442 or may be executed by the processing unit 110 of the server system 100. In this way, it becomes possible to clearly present the condition of the person being assisted using the wheelchair 630 to a caregiver located near the wheelchair 630 without using an application such as the positioning application AP1. .

However, the criteria for determining forward displacement, etc., performed by the seat sensor 440 are based on tacit knowledge by a skilled assistant. Therefore, the seat sensor 440 may generate the determination criteria by receiving a setting input from the second user. For example, the seat sensor 440 may be connected to the first terminal device 200 used by a first user who is the user of the seat sensor 440. The first terminal device 200 and the second terminal device 300 may display a screen corresponding to the area RE4 in FIG. 16 or the area RE6 in FIG. 17. At this time, the first terminal device 200 and the second terminal device 300 may communicate using the web conference function, as described above using FIG. 9. In this way, even if the second user, who is an expert, is located at a location different from the location where the seat sensor 440 is placed, the settings of the seat sensor 440 can be changed using the tacit knowledge of the second user. It becomes possible to execute using .

Further, the electronic device here may be, for example, the swallowing discomfort detection device 460. As described above, the swallowing difficulty detection device 460 can perform various determinations regarding the meal of the person being assisted, and these can be performed independently of the positioning application AP1.

For example, which of these judgments is useful changes depending on the condition of the person being cared for and the details of the intervention that can be performed by the carer, and thus becomes tacit knowledge of the expert. Therefore, the swallowing awkwardness detection device 460 may transmit the output of the throat microphone 461 (audio data or waveform signal), the image captured by the camera of the terminal device 462, etc. to the second terminal device 300. The second user may input settings to determine whether each determination is on or off based on the transmitted information. In this way, the second user's tacit knowledge regarding which determination should be performed can be appropriately utilized in the swallowing awkwardness detection device 460.

Further, the tacit knowledge used in the swallowing choking detection device 460 is not limited to this. For example, the swallowing awkwardness detection device 460 may determine whether or not intervention by an assistant is necessary based on the swallowing time, the frequency of the awkwardness, and the result of image recognition by a camera. In this case, the standard for how much abnormality is detected to require intervention may be the tacit knowledge of the expert. Therefore, the swallowing awkwardness detection device 460 may receive a setting input regarding the intervention criteria from the second user, and the swallowing awkwardness detection device 460 may determine the intervention criteria based on the setting input.

Further, in the first terminal device 200 of this embodiment, an application related to assistance other than the positioning application AP1 may be operated. For example, the first terminal device 200 may operate a meal application to support meal assistance, may operate a restlessness application to determine the presence or absence of restless behavior due to dementia, or may operate other than this. An assistance application may also operate. In the determination in these applications, settings may be input based on the second user's tacit knowledge.

For example, the method of the present embodiment is used by a first terminal device 200 that stores an application that performs processing in assistance and operates according to the application, and a second user who uses the first terminal device 200 to support assistance for the first user. The present invention may also be applied to an information processing system 10 including a second terminal device 300 that is configured to operate. The first terminal device 200 transmits sensing results regarding at least one of a person and an object during assistance to the second terminal device 300 via the network. The sensing result here may be the output of a sensor included in the first terminal device 200 or the output of the sensing device 400. The second terminal device 300 presents the sensing results to the second user and receives application setting input from the second user. The setting input here is a parameter that determines the operation of the application, and for example, the second user's appropriate judgment result (for a meal application, appropriate assistance behavior in meal assistance, etc., for an anxiety application, regarding restlessness behavior, etc.) This information represents the second user's determination result, etc.). The first terminal device 200 operates according to the application corresponding to the setting input. In this way, even when an application other than the positioning application AP1 is targeted, the tacit knowledge of the second user can be appropriately utilized by performing remote settings by the second user.

Although this embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications can be made without substantively departing from the novelty and effects of this embodiment. . Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term that appears at least once in the specification or drawings together with a different term with a broader or synonymous meaning may be replaced by that different term anywhere in the specification or drawings. Furthermore, all combinations of this embodiment and modifications are also included within the scope of the present disclosure. Further, the configurations and operations of the information processing system, server system, first terminal device, second terminal device, sensing device, etc. are not limited to those described in this embodiment, and various modifications are possible.

10... Information processing system, 100... Server system, 110... Processing unit, 120... Storage unit, 130... Communication unit, 200... First terminal device, 200-1, 200-2... Terminal device (first terminal device), 210... Processing section, 220... Storage section, 230... Communication section, 240... Display section, 250... Operation section, 260... Imaging section, 300... Second terminal device, 310... Processing section, 320... Storage section, 330... Communication Part, 340...Display unit, 350...Operation unit, 360...Imaging unit, 400...Sensing device, 420...Bedside sensor, 430...Detection device, 440...Seat surface sensor, 441...Cushion, 442...Control box, 460... Swallowing discomfort detection device, 461...Throat microphone, 462...Terminal device, 610...Bed, 620...Mattress, 630...Wheelchair, AP1...Positioning application, DP...Display, NW...Network, OB1 to OB12, EM...Object, RE1 to RE7...Region, Se1-Se4...Pressure sensor, S...Graphic, PO1-PO2...Pointer

Claims (11)

A first terminal device that stores a positioning application that performs processing based on teacher data representing a correct position of at least one of a person and an object in assistance, and operates according to the positioning application;
a second terminal device used by a second user supporting the assistance for the first user using the first terminal device;
including;
The first terminal device is
transmitting a sensing result regarding at least one of the person and the object in the assistance to the second terminal device via a network;
The second terminal device is
Presenting the sensing results to a second user and accepting input of setting of the teacher data by the second user;
The first terminal device is
An information processing system that operates according to the positioning application corresponding to the teacher data created by the setting input. In claim 1,
The second terminal device is
An information processing system that outputs connection information used for connection between a sensing device that outputs the sensing result and the first terminal device to the first terminal device via the network. In claim 1,
The first terminal device is
During the setting period of the positioning application, transmitting a first captured image of the first user of the first terminal device to the second terminal device;
The second terminal device is
An information processing system that transmits a second captured image of the second user of the second terminal device to the first terminal device during the set period. In claim 3,
The first terminal device is
An information processing system that stores the second captured image as log data and displays the second captured image during a usage period of the positioning application. In claim 3 or 4,
An emotional analysis of the first user of the first terminal device is performed based on the first captured image,
The second terminal device is
An information processing system that associates and displays the first captured image transmitted from the first terminal device and the result of the emotion analysis. In claim 5,
When there is a plurality of second users, the reliability from the first user is determined for each second user based on the result of the sentiment analysis,
In the information processing system, in the setting input of the teacher data, the second user with the higher reliability is given priority over the second user with the lower reliability. In claim 5,
When there is a plurality of second users, the reliability from the first user is determined for each second user based on the result of the sentiment analysis,
The positioning application corresponding to the teacher data set by the second user with a high reliability is compared to the positioning application corresponding to the teacher data set by the second user with a low reliability. An information processing system that is used preferentially. In claim 5,
Based on the result of the emotion analysis, when it is determined that the reliability of the second user from the person being assisted who is the first user is less than or equal to a predetermined threshold;
At least one of the first terminal device and the second terminal device,
An information processing system that presents information that encourages the participation of a family member of the person being assisted or the second user who has a relatively high degree of reliability during communication between the first terminal device and the second terminal device. In any one of claims 1 to 4,
The first terminal device is
Execute processing corresponding to a plurality of the teacher data,
The second terminal device is
An information processing system that accepts a selection operation by the second user to select the teacher data to be used from a plurality of the teacher data. In any one of claims 1 to 4,
The first terminal device is
An information processing system that compares a processing load of the positioning application, a communication load of the sensing result, and a processing capacity of the first terminal device, and switches an operation mode of the first terminal device based on the comparison result. A first terminal device that stores a positioning application that performs processing based on teacher data representing a correct position of at least one of a person and an object in assistance, and operates according to the positioning application, and a first user who uses the first terminal device. An information processing method in an information processing system including a second terminal device used by a second user who supports assistance,
transmitting a sensing result regarding at least one of the person and the object in the assistance from the first terminal device to the second terminal device via a network;
In the second terminal device, presenting the sensing result to a second user and accepting a setting input of the teacher data by the second user;
operating the first terminal device according to the positioning application corresponding to the teacher data created by the setting input;
Information processing method.