WO2014104357A1

WO2014104357A1 - Motion information processing system, motion information processing device and medical image diagnosis device

Info

Publication number: WO2014104357A1
Application number: PCT/JP2013/085244
Authority: WO
Inventors: 八木　稔; 池田　智; 真哉杉山; 弘祐坂上
Original assignee: 株式会社東芝; 東芝メディカルシステムズ株式会社
Priority date: 2012-12-28
Filing date: 2013-12-27
Publication date: 2014-07-03
Also published as: JP2014130519A

Abstract

A motion information processing system (1) of an embodiment is provided with an acquisition unit (141), an extraction unit (142), a selection unit (143), and a display control unit (145). The acquisition unit (141) acquires skeletal information including positional information of joints of a patient the motion of which is to be acquired. The extraction unit (142) extracts an affected part on the basis of the positional information of the joints in the patient's skeletal information acquired by the acquisition unit (141). The selection unit (143) then selects a schema related to the affected part extracted by the extraction unit (142). The display control unit (145) then performs control such that the schema selected by the selection unit (143) is displayed in an output unit.

Description

Motion information processing system, motion information processing apparatus, and medical image diagnostic apparatus

Embodiments of the present invention relate to a motion information processing system, a motion information processing device, and a medical image diagnostic device.

Conventionally, in medical institutions, an electronic medical record system has been used to manage medical information of visiting patients. In such an electronic medical record system, each patient's medical record (electronic medical record) is called at a doctor terminal provided in each clinic, and various information is input to the called patient's electronic medical record. For example, for information input in electronic medical records, doctors use a doctor's terminal to input information such as chief complaints, findings, etc., schema diagrams, order contents, etc. according to the medical condition of the patient's electronic medical records. This is done by individually searching / selecting the definition contents and inputting them. If the patient is a revisited patient, the patient's past medical chart information may be called, copied as it is, and input to the current electronic medical chart.

On the other hand, in recent years, development of motion capture technology that digitally records the movement of a person or an object has been advanced. As a method of motion capture technology, for example, an optical type, a mechanical type, a magnetic type, a camera type and the like are known. For example, a camera system is known in which a marker is attached to a person, the marker is detected by a tracker such as a camera, and the movement of the person is digitally recorded by processing the detected marker. As a method that does not use markers and trackers, an infrared sensor is used to measure the distance from the sensor to the person and detect the movement of the person digitally by detecting the size of the person and various movements of the skeleton. The recording method is known. As a sensor using such a method, for example, Kinect (registered trademark) is known.

JP 2010-176213 A JP 2010-176209 A JP 2010-079567 A JP 2007-151686 A

The problem to be solved by the present invention is to provide a motion information processing system, a motion information processing device, and a medical image diagnostic device that make it easy to input information in an electronic medical record.

The motion information processing system of the embodiment includes an acquisition unit, an extraction unit, a selection unit, and a display control unit. The acquisition unit acquires motion information including position information of a joint of a subject who is a target of motion acquisition. The extraction unit extracts the affected part based on the joint position information in the motion information of the subject acquired by the acquisition unit. The selection unit selects related information related to the affected part extracted by the extraction unit. The display control unit controls the display unit to display related information selected by the selection unit.

FIG. 1 is a diagram illustrating an example of a configuration of a motion information processing system according to the first embodiment. FIG. 2A is a diagram for explaining an example of a display screen of the electronic medical record according to the first embodiment. FIG. 2B is a diagram for explaining an example of inputting chart information according to the first embodiment. FIG. 3 is a diagram illustrating an example of the configuration of the doctor terminal according to the first embodiment. FIG. 4A is a diagram for explaining processing of the motion information generation unit according to the first embodiment. FIG. 4B is a diagram for explaining processing of the motion information generation unit according to the first embodiment. FIG. 4C is a diagram for explaining processing of the motion information generation unit according to the first embodiment. FIG. 5 is a diagram illustrating an example of skeleton information generated by the motion information generation unit according to the first embodiment. FIG. 6 is a diagram illustrating an example of a detailed configuration of the doctor terminal according to the first embodiment. FIG. 7 is a diagram for explaining an example of processing by the extraction unit according to the first embodiment. FIG. 8A is a diagram for explaining an example of processing by the selection unit according to the first embodiment. FIG. 8B is a diagram for explaining an example of processing performed by the selection unit according to the first embodiment. FIG. 8C is a diagram for explaining an example of processing performed by the selection unit according to the first embodiment. FIG. 9A is a diagram illustrating an example of processing performed by the mark assigning unit according to the first embodiment. FIG. 9B is a diagram illustrating an example of processing performed by the mark assigning unit according to the first embodiment. FIG. 9C is a diagram illustrating an example of processing performed by the mark assigning unit according to the first embodiment. FIG. 10 is a diagram illustrating an example of schema selection and mark assignment according to the first embodiment. FIG. 11 is a diagram for explaining an example of display control by the display control unit according to the first embodiment. FIG. 12 is a diagram for explaining an example of processing by the medical record information storage unit according to the first embodiment. FIG. 13 is a flowchart illustrating a processing procedure performed by the doctor terminal according to the first embodiment. FIG. 14 is a diagram for explaining an example of processing by the selection unit according to the second embodiment. FIG. 15 is a diagram for explaining operation information collection condition changing processing by the doctor terminal according to the third embodiment.

Hereinafter, a motion information processing system, a motion information processing apparatus, and a medical image diagnostic apparatus according to embodiments will be described with reference to the drawings. Hereinafter, a motion information processing system including a doctor terminal as a motion information processing apparatus will be described as an example.

(First embodiment)
FIG. 1 is a diagram illustrating an example of a configuration of a motion information processing system 1 according to the first embodiment. As illustrated in FIG. 1, the motion information processing system 1 according to the first embodiment includes a doctor terminal 100, a reception terminal 200, and a server device 300. The doctor terminal 100, the reception terminal 200, and the server device 300 are in a state where they can communicate with each other directly or indirectly, for example, via a hospital LAN (Local Area Network) installed in the hospital. . For example, PACS (Picture Archiving and Communication System), HIS (Hospital Information System), RIS (Radiology Information System), or the like is applied to the motion information processing system 1. The motion information processing system 1 also includes functions as an electronic medical record system, a receipt computer processing system, an ordering system, a reception (individual / qualification authentication) system, a medical assistance system, and the like.

The reception terminal 200 executes reception registration when a patient visits, creation of a patient's electronic medical record, or calling of the patient's electronic medical record from the electronic medical record managed by the server device 300. Then, the reception terminal 200 creates a queue of reception information and electronic medical record information for each patient according to the reception time or the reservation time, and transmits the queue to the server device 300.

The server apparatus 300 manages the electronic medical records of registered patients. The server apparatus 300 manages a queue of reception information and electronic medical record information received from the reception terminal 200. For example, the server device 300 manages a queue of received reception information and electronic medical record information for each department.

The doctor terminal 100 is a terminal installed in each examination room, for example, and the medical chart information of the electronic medical record is input by the doctor. Here, the chart information includes, for example, symptoms and doctor's findings. The doctor operates the doctor terminal 100 to read reception information and electronic medical record information from the server device 300 in the queue order. Then, the doctor examines the corresponding patient and inputs medical chart information to the read electronic medical chart.

FIG. 2A is a diagram for explaining an example of a display screen of the electronic medical record according to the first embodiment. For example, as illustrated in FIG. 2A, the doctor terminal 100 includes an operation area in which a chart area R1 that is an area in which patient chart information is input and operation buttons for inputting chart information in the chart area R1 are arranged. An electronic medical chart display screen having R2 is displayed. Here, as shown in FIG. 2A, the chart area R1 includes, for example, an area R3 in which patient data such as name, date of birth, and gender is displayed, an area R4 in which current chart information is input, and previous chart information. It is comprised from the area | region R5 etc. which are displayed. The operation area R2 includes an area R6 that is an area for selecting a schema to be used when using a schema for medical chart information, an area R7 in which operation buttons to which various functions are assigned, and the like. The

For example, the doctor operates the buttons arranged in the region R6, the region R7, etc., and inputs medical chart information to the electronic medical record. FIG. 2B is a diagram for explaining an example of inputting chart information according to the first embodiment. FIG. 2B shows a schema selection window displayed by operating a button arranged in the region R6 shown in FIG. 2A when using a schema as chart information.

For example, when a doctor inputs a head schema as chart information, the doctor operates a button arranged in the region R6 shown in FIG. 2A to display a plurality of heads as shown in the right region of FIG. 2B. A selection window with the schema is displayed. Then, the doctor selects a desired schema from the displayed plurality of head schemas, thereby displaying the selected schema in the left region of the window as shown in FIG. 2B. Furthermore, the doctor gives a mark to a position corresponding to the affected part of the schema as shown in FIG. 2B by operating a button arranged in the left region of FIG. 2B.

Then, the doctor inputs the schema to which the mark is added into the electronic medical record as medical record information. For example, the doctor inputs a schema having a mark added to the region R4 in FIG. 2A. When the examination ends, the doctor terminal 100 transmits the electronic medical record in which the medical chart information is input to the server device 300 based on the operation of the doctor. The server device 300 manages the received electronic medical record for each patient.

As described above, in the electronic medical record, various types of medical record information are input at the doctor terminal 100. Here, in the prior art, the input of information in the electronic medical record becomes complicated and may take time. For example, as shown in FIG. 2B, when inputting a schema, first, information on the affected part is obtained from a patient by pointing orally or by hand, which part of the body is used, and for each part To select which schema to use. Furthermore, the doctor gives a mark on the schema according to the range of the affected part. Therefore, when inputting medical chart information into the electronic medical chart, the selection operation may be repeated, and the information input may be complicated and troublesome.

Therefore, the motion information processing system 1 according to the first embodiment makes it easy to input information in the electronic medical chart by the processing of the doctor terminal 100 described in detail below. FIG. 3 is a diagram illustrating an example of the configuration of the doctor terminal 100 according to the first embodiment. As shown in FIG. 1, in the first embodiment, the doctor terminal 100 is connected to the motion information collection unit 10.

The motion information collection unit 10 detects the motion of a person or an object in the space where the examination is performed, and collects motion information representing the motion of the person or the object. Note that the operation information will be described in detail when the processing of the operation information generation unit 14 described later is described. For example, Kinect (registered trademark) is used as the operation information collection unit 10.

As shown in FIG. 3, the motion information collection unit 10 includes, for example, a color image collection unit 11, a distance image collection unit 12, a voice recognition unit 13, and a motion information generation unit 14. Note that the configuration of the operation information collection unit 10 illustrated in FIG. 3 is merely an example, and the embodiment is not limited thereto.

The color image collection unit 11 shoots a subject such as a person or an object in a space where a medical examination is performed, and collects color image information. For example, the color image collection unit 11 detects light reflected from the subject surface with a light receiving element, and converts visible light into an electrical signal. Then, the color image collection unit 11 converts the electrical signal into digital data, thereby generating one frame of color image information corresponding to the shooting range. The color image information for one frame includes, for example, shooting time information and information in which each pixel included in the one frame is associated with an RGB (Red Green Blue) value. The color image collection unit 11 shoots a moving image of the shooting range by generating color image information of a plurality of continuous frames from visible light detected one after another. The color image information generated by the color image collection unit 11 may be output as a color image in which the RGB values of each pixel are arranged in a bitmap. The color image collection unit 11 includes, for example, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) as a light receiving element.

The distance image collection unit 12 photographs a subject such as a person or an object in a space where rehabilitation is performed, and collects distance image information. For example, the distance image collection unit 12 irradiates the surrounding area with infrared rays, and detects a reflected wave obtained by reflecting the irradiation wave on the surface of the subject with the light receiving element. Then, the distance image collection unit 12 obtains the distance between the subject and the distance image collection unit 12 based on the phase difference between the irradiation wave and the reflected wave and the time from irradiation to detection, and corresponds to the shooting range. Generate frame distance image information. The distance image information for one frame includes, for example, shooting time information and information in which each pixel included in the shooting range is associated with the distance between the subject corresponding to the pixel and the distance image collection unit 12. included. The distance image collection unit 12 captures a moving image of the shooting range by generating distance image information of a plurality of continuous frames from reflected waves detected one after another. The distance image information generated by the distance image collection unit 12 may be output as a distance image in which color shades corresponding to the distance of each pixel are arranged in a bitmap. The distance image collection unit 12 includes, for example, a CMOS or a CCD as a light receiving element. This light receiving element may be shared with the light receiving element used in the color image collection unit 11. The unit of the distance calculated by the distance image collection unit 12 is, for example, meters [m].

The voice recognition unit 13 collects surrounding voices, identifies the direction of the sound source, and performs voice recognition. The voice recognition unit 13 has a microphone array including a plurality of microphones, and performs beam forming. Beam forming is a technique for selectively collecting sound from a specific direction. For example, the voice recognition unit 13 specifies the direction of the sound source by beam forming using a microphone array. The voice recognition unit 13 recognizes a word from the collected voice using a known voice recognition technique. That is, the speech recognition unit 13 generates, as a speech recognition result, for example, information associated with a word recognized by the speech recognition technology, a direction in which the word is emitted, and a time at which the word is recognized.

The motion information generation unit 14 generates motion information representing the motion of a person or an object. This motion information is generated by, for example, capturing a human motion (gesture) as a series of a plurality of postures (poses). In brief, the motion information generation unit 14 first obtains the coordinates of each joint forming the skeleton of the human body from the distance image information generated by the distance image collection unit 12 by pattern matching using a human body pattern. The coordinates of each joint obtained from the distance image information are values represented by a distance image coordinate system (hereinafter referred to as “distance image coordinate system”). For this reason, the motion information generation unit 14 next represents the values of the coordinates of each joint in the distance image coordinate system in a coordinate system of a three-dimensional space in which medical examination is performed (hereinafter referred to as “world coordinate system”). Convert to The coordinates of each joint represented in this world coordinate system become the skeleton information for one frame. Further, the skeleton information for a plurality of frames is the operation information. Hereinafter, processing of the motion information generation unit 14 according to the first embodiment will be specifically described.

4A to 4C are diagrams for explaining processing of the motion information generation unit 14 according to the first embodiment. FIG. 4A shows an example of a distance image generated by the distance image collection unit 12. In FIG. 4A, for convenience of explanation, an image expressed by a line drawing is shown. However, an actual distance image is an image expressed by shading of colors according to the distance. In this distance image, each pixel has a “pixel position X” in the left-right direction of the distance image, a “pixel position Y” in the up-down direction of the distance image, and a subject corresponding to the pixel and the distance image collection unit 12. It has a three-dimensional value associated with “distance Z”. Hereinafter, the coordinate value of the distance image coordinate system is expressed by the three-dimensional value (X, Y, Z).

In the first embodiment, the motion information generation unit 14 stores in advance human body patterns corresponding to various postures, for example, by learning. Each time the distance image collection unit 12 generates distance image information, the motion information generation unit 14 acquires the generated distance image information of each frame. Then, the motion information generation unit 14 performs pattern matching using a human body pattern on the acquired distance image information of each frame.

Here, human body patterns will be described. FIG. 4B shows an example of a human body pattern. In the first embodiment, since the human body pattern is a pattern used for pattern matching with distance image information, it is expressed in the distance image coordinate system, and is similar to the person depicted in the distance image, on the surface of the human body. Information (hereinafter referred to as “human body surface”). For example, the human body surface corresponds to the skin or clothing surface of the person. Furthermore, as shown in FIG. 4B, the human body pattern includes information on each joint forming the skeleton of the human body. That is, in the human body pattern, the relative positional relationship between the human body surface and each joint is known.

In the example shown in FIG. 4B, the human body pattern includes information on 20 joints from joint 2a to joint 2t. Of these, the joint 2a corresponds to the head, the joint 2b corresponds to the center of both shoulders, the joint 2c corresponds to the waist, and the joint 2d corresponds to the center of the buttocks. The joint 2e corresponds to the right shoulder, the joint 2f corresponds to the right elbow, the joint 2g corresponds to the right wrist, and the joint 2h corresponds to the right hand. The joint 2i corresponds to the left shoulder, the joint 2j corresponds to the left elbow, the joint 2k corresponds to the left wrist, and the joint 2l corresponds to the left hand. Also, the joint 2m corresponds to the right hip, the joint 2n corresponds to the right knee, the joint 2o corresponds to the right ankle, and the joint 2p corresponds to the right foot. Further, the joint 2q corresponds to the left hip, the joint 2r corresponds to the left knee, the joint 2s corresponds to the left ankle, and the joint 2t corresponds to the left foot.

In FIG. 4B, the case where the human body pattern has information on 20 joints has been described. However, the embodiment is not limited to this, and the operator may arbitrarily set the position and number of joints. . For example, when only the change in the movement of the limbs is captured, information on the joint 2b and the joint 2c among the joints 2a to 2d may not be acquired. In addition, when capturing changes in the right hand movement in detail, not only the joint 2i but also the finger joint of the right hand may be newly set. The joint 2a, the joint 2h, the joint 2l, the joint 2p, and the joint 2t in FIG. 4B are different from so-called joints because they are the end portions of the bone, but are important points that represent the position and orientation of the bone. For the sake of convenience, it is described here as a joint.

The motion information generation unit 14 performs pattern matching with the distance image information of each frame using the human body pattern. For example, the motion information generation unit 14 extracts a person with a certain posture from the distance image information by pattern matching the human body surface of the human body pattern shown in FIG. 4B and the distance image shown in FIG. 4A. In this way, the motion information generation unit 14 obtains the coordinates of the human body surface depicted in the distance image. Further, as described above, in the human body pattern, the relative positional relationship between the human body surface and each joint is known. Therefore, the motion information generation unit 14 calculates the coordinates of each joint in the person from the coordinates of the human body surface depicted in the distance image. Thus, as illustrated in FIG. 4C, the motion information generation unit 14 acquires the coordinates of each joint forming the skeleton of the human body from the distance image information. Note that the coordinates of each joint obtained here are the coordinates of the distance coordinate system.

Note that the motion information generation unit 14 may use information representing the positional relationship of each joint as an auxiliary when performing pattern matching. The information representing the positional relationship between the joints includes, for example, a joint relationship between the joints (for example, “joint 2a and joint 2b are coupled”) and a movable range of each joint. A joint is a site that connects two or more bones. The angle between the bones changes according to the change in posture, and the range of motion differs depending on the joint. For example, the range of motion is represented by the maximum and minimum values of the angles formed by the bones connected by each joint. For example, when learning the human body pattern, the motion information generation unit 14 also learns the range of motion of each joint and stores it in association with each joint.

Subsequently, the motion information generation unit 14 converts the coordinates of each joint in the distance image coordinate system into values represented in the world coordinate system. The world coordinate system is a coordinate system in a three-dimensional space where rehabilitation is performed. For example, the position of the motion information collection unit 10 is the origin, the horizontal direction is the x axis, the vertical direction is the y axis, and the direction is orthogonal to the xy plane. Is a coordinate system with z as the z-axis. The coordinate value in the z-axis direction may be referred to as “depth”.

Here, the process of converting from the distance image coordinate system to the world coordinate system will be described. In the first embodiment, it is assumed that the motion information generation unit 14 stores in advance a conversion formula for converting from the distance image coordinate system to the world coordinate system. For example, this conversion formula receives the coordinates of the distance image coordinate system and the incident angle of the reflected light corresponding to the coordinates, and outputs the coordinates of the world coordinate system. For example, the motion information generation unit 14 inputs the coordinates (X1, Y1, Z1) of a certain joint and the incident angle of reflected light corresponding to the coordinates to the conversion formula, and coordinates (X1, Y1) of the certain joint , Z1) are converted into coordinates (x1, y1, z1) in the world coordinate system. Since the correspondence relationship between the coordinates of the distance image coordinate system and the incident angle of the reflected light is known, the motion information generation unit 14 inputs the incident angle corresponding to the coordinates (X1, Y1, Z1) into the conversion equation. can do. Although the case has been described here in which the motion information generation unit 14 converts the coordinates of the distance image coordinate system to the coordinates of the world coordinate system, it is also possible to convert the coordinates of the world coordinate system to the coordinates of the distance coordinate system. is there.

Then, the motion information generation unit 14 generates skeleton information from the coordinates of each joint represented in the world coordinate system. FIG. 5 is a diagram illustrating an example of the skeleton information generated by the motion information generation unit 14. The skeleton information of each frame includes shooting time information of the frame and coordinates of each joint. For example, as illustrated in FIG. 5, the motion information generation unit 14 generates skeleton information in which joint identification information and coordinate information are associated with each other. In FIG. 5, the shooting time information is not shown. The joint identification information is identification information for identifying a joint and is set in advance. For example, joint identification information “2a” corresponds to the head, and joint identification information “2b” corresponds to the center of both shoulders. Similarly for the other joint identification information, each joint identification information indicates a corresponding joint. The coordinate information indicates the coordinates of each joint in each frame in the world coordinate system.

5, joint identification information “2a” and coordinate information “(x1, y1, z1)” are associated with each other. That is, the skeleton information in FIG. 5 indicates that the head is present at the coordinates (x1, y1, z1) in a certain frame. Also, in the second row of FIG. 5, joint identification information “2b” and coordinate information “(x2, y2, z2)” are associated. That is, the skeleton information in FIG. 5 represents that the center of both shoulders exists at the coordinates (x2, y2, z2) in a certain frame. Similarly, other joints indicate that each joint exists at a position represented by each coordinate in a certain frame.

In this way, every time the distance image information of each frame is acquired from the distance image collection unit 12, the motion information generation unit 14 performs pattern matching on the distance image information of each frame, and the world coordinate from the distance image coordinate system. By converting into a system, skeleton information of each frame is generated. Then, the motion information generation unit 14 outputs the generated skeleton information of each frame to the doctor terminal 100 and stores it in the later-described motion information storage unit 131.

In addition, the process of the operation information generation part 14 is not restricted to the method mentioned above. For example, in the above description, the method in which the motion information generation unit 14 performs pattern matching using a human body pattern has been described, but the embodiment is not limited thereto. For example, instead of the human body pattern or together with the human body pattern, a pattern matching method using a pattern for each part may be used.

For example, in the above description, the method in which the motion information generation unit 14 obtains the coordinates of each joint from the distance image information has been described, but the embodiment is not limited thereto. For example, the motion information generation unit 14 may obtain a coordinate of each joint using color image information together with distance image information. In this case, for example, the motion information generation unit 14 performs pattern matching between the human body pattern expressed in the color image coordinate system and the color image information, and obtains the coordinates of the human body surface from the color image information. The coordinate system of this color image does not include the “distance Z” information referred to in the distance image coordinate system. Therefore, for example, the motion information generation unit 14 obtains the information of “distance Z” from the distance image information, and obtains the coordinates of the world coordinate system of each joint by calculation processing using these two pieces of information.

Further, the motion information generation unit 14 needs the color image information generated by the color image collection unit 11, the distance image information generated by the distance image collection unit 12, and the voice recognition result output by the voice recognition unit 13. Accordingly, the information is appropriately output to the doctor terminal 100 and stored in the operation information storage unit 131 described later. The pixel position of the color image information and the pixel position of the distance image information can be associated in advance according to the positions of the color image collection unit 11 and the distance image collection unit 12 and the shooting direction. For this reason, the pixel position of the color image information and the pixel position of the distance image information can be associated with the world coordinate system calculated by the motion information generation unit 14. Further, by using this association and the distance [m] calculated by the distance image collection unit 12, the height and the length of each part of the body (for example, the length of the arm and the length of the abdomen), and the color image It is also possible to calculate the distance between the two points specified above (between two pixels).

Similarly, the shooting time information of the color image information and the shooting time information of the distance image information can be associated in advance. In addition, the motion information generation unit 14 refers to the speech recognition result and the distance image information, and if there is a joint 2a in the vicinity of the direction in which the speech-recognized word is issued at a certain time, the person including the joint 2a is displayed. It can be output as an emitted word. Furthermore, the motion information generation unit 14 also appropriately outputs information representing the positional relationship between the joints to the doctor terminal 100 as necessary, and stores the information in the motion information storage unit 131 described later.

In addition, although the case where the operation | movement of one subject was detected by the operation | movement information collection part 10 was demonstrated here, embodiment is not limited to this. If included in the detection range of the motion information collection unit 10, the motion information collection unit 10 may detect the motions of a plurality of subjects. In such a case, the motion information generation unit 14 generates skeleton information of a plurality of persons from the distance image information of the same frame, and outputs information associated with the generated skeleton information to the doctor terminal 100 as motion information. To do.

Further, the configuration of the operation information collection unit 10 is not limited to the above configuration. For example, when motion information is generated by detecting the motion of a person by other motion capture, such as optical, mechanical, magnetic, etc., the motion information collection unit 10 does not necessarily include the distance image collection unit 12. It does not have to be. In such a case, the motion information collection unit 10 includes, as motion sensors, a marker that is worn on the human body to detect the motion of the person, and a sensor that detects the marker. Then, the motion information collection unit 10 detects motion of a person using a motion sensor and generates motion information. Further, the motion information collecting unit 10 associates the pixel position of the color image information with the coordinates of the motion information using the position of the marker included in the image photographed by the color image collecting unit 11, and if necessary, Output to the doctor terminal 100 as appropriate. In addition, for example, the motion information collection unit 10 may not include the speech recognition unit 13 when the speech recognition result is not output to the doctor terminal 100.

Furthermore, in the embodiment described above, the motion information collection unit 10 outputs the coordinates of the world coordinate system as the skeleton information, but the embodiment is not limited to this. For example, the motion information collection unit 10 may output the coordinates of the distance image coordinate system before conversion, and the conversion from the distance image coordinate system to the world coordinate system may be performed on the doctor terminal 100 side as necessary.

Returning to the explanation of FIG. The doctor terminal 100 uses the operation information output from the operation information collection unit 10 to perform processing for assisting input of electronic medical record information (medical record information). The doctor terminal 100 is an information processing device such as a computer or a workstation, for example, and includes an output unit 110, an input unit 120, a storage unit 130, and a control unit 140, as shown in FIG. The doctor terminal 100 includes a communication unit (not shown) and performs communication with the reception terminal 200, the server device 300, and the like.

The output unit 110 outputs various information for creating an electronic medical record. For example, the output unit 110 displays a GUI (Graphical User Interface) for an operator (doctor) operating the doctor terminal 100 to input various requests using the input unit 120 to create an electronic medical record, An output image generated in the terminal 100 is displayed. For example, the output unit 110 is a monitor, a speaker, or the like.

The input unit 120 receives input of various information for creating an electronic medical record. For example, the input unit 120 accepts input of various requests (for example, a request for reading an electronic medical record from the server device 300 or a request for inputting medical record information to the electronic medical record) from the operator (doctor) of the doctor terminal 100, The received various requests are transferred to the doctor terminal 100. Here, the input unit 120 accepts information such as a patient ID (name number) of a patient as a reading request of the electronic medical record. The input unit 120 is, for example, a mouse, a keyboard, a touch command screen, a trackball, or the like.

The storage unit 130 is, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, a storage device such as a hard disk device or an optical disk device. The control unit 140 can be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or a CPU (Central Processing Unit) executing a predetermined program.

The configuration of the doctor terminal 100 according to the first embodiment has been described above. Under such a configuration, the doctor terminal 100 according to the first embodiment extracts and extracts the affected part by analyzing the motion information of the patient (target of motion acquisition) collected by the motion information collecting unit 10. By selecting related information related to the affected area, it is possible to easily input information in the electronic medical record. Here, in 1st Embodiment, the case where a schema is selected as an example of the relevant information relevant to an affected part is demonstrated. The schema is a schematic diagram showing the positional relationship between the human body structure and the affected part.

FIG. 6 is a diagram illustrating an example of a detailed configuration of the doctor terminal 100 according to the first embodiment. As shown in FIG. 6, in the doctor terminal 100, for example, the storage unit 130 includes an operation information storage unit 131, an extraction condition storage unit 132, and a medical record information storage unit 133.

The operation information storage unit 131 stores various types of information collected by the operation information collection unit 10. Specifically, the motion information storage unit 131 stores the motion information generated by the motion information generation unit 14. More specifically, the motion information storage unit 131 stores skeleton information for each frame generated by the motion information generation unit 14. Here, the motion information storage unit 131 can further store the color image information, the distance image information, and the speech recognition result output by the motion information generation unit 14 in association with each other.

The extraction condition storage unit 132 stores extraction conditions for extracting an affected part of a patient from operation information. Specifically, the extraction condition storage unit 132 stores an extraction condition for extracting an affected part based on information on a hand joint in the operation condition. For example, the extraction condition storage unit 132 determines the distance from the joint position of the hand at a predetermined timing to the line segment connecting two joints different from the joint of the hand in the joint position information in the motion information of the subject. Is stored below the extraction condition in which the part corresponding to the position on the shortest line segment is equal to or less than the predetermined threshold value. Here, the extraction condition storage unit 132 includes, as the predetermined timing, a timing at which predetermined audio information is acquired, or a case where the movement of the hand joint is stopped for a certain period of time. A detailed example of the extraction condition described above will be described later.

Further, for example, the extraction condition storage unit 132 stores the extraction condition in which the part corresponding to the predetermined movement range of the joint position of the hand is the affected part in the joint position information in the motion information of the subject. In other words, when the hand joint moves within a predetermined area, the extraction condition storage unit 132 stores an extraction condition in which a part corresponding to the area is an affected area. For example, when the joint of the hand moves back and forth in a certain area, or when the extraction condition storage unit 132 stops for a certain period of time in a certain joint area, the extraction condition storage unit 132 The extraction condition with the part corresponding to is affected is stored.

The chart information storage unit 133 stores the chart information of the electronic chart acquired from the server device 300 via a communication unit (not shown). Specifically, the medical chart information storage unit 133 stores the patient's medical chart information acquired from the server apparatus 300 by the communication unit in response to the electronic medical chart read request received from the operator (doctor) via the input unit 120. To do. For example, the chart information storage unit 133 stores the patient's past chart information corresponding to the read request and the chart information stored by the control of the control unit 140 described later. In other words, the chart information storage unit 133 is a storage unit that stores the electronic chart read by the doctor terminal 100 from the server device 300 during diagnosis. That is, the medical record information stored by the medical record information storage unit 133 is stored in the server apparatus 300 via the communication unit after the diagnosis is completed. A detailed example of the chart information stored by the chart information storage unit 133 will be described later.

Returning to the description of FIG. 6, in the doctor terminal 100, for example, the control unit 140 has an acquisition unit 141, an extraction unit 142, a selection unit 143, a mark addition unit 144, a display control unit 145, and medical record information storage. Unit 146, and using various information stored in the storage unit 130, it is possible to facilitate the input of electronic medical record information by assisting in inputting medical record information.

The acquisition unit 141 acquires motion information of a target person (patient) that is a target of motion acquisition. Specifically, the acquisition unit 141 acquires patient motion information collected by the motion information collection unit 10 and stored in the motion information storage unit 131. More specifically, the acquisition unit 141 acquires patient skeleton information stored for each frame by the motion information storage unit 131.

For example, the acquisition unit 141 acquires the skeleton information of the operation corresponding to the time point when the patient who visited the hospital made a chief complaint such as a symptom or an affected part during the examination. For example, the acquisition unit 141 acquires the skeleton information of the patient from the frame at the time when the medical inquiry by the doctor is started to the frame at the time when the medical examination is completed. The acquisition of the skeletal information by the acquisition unit 141 is performed each time a patient's examination is started.

The extraction unit 142 extracts the affected part based on the joint position information in the motion information of the subject (patient) acquired by the acquisition unit 141. Specifically, the extraction unit 142 refers to the extraction condition stored by the extraction condition storage unit 132 and extracts the affected part from the skeleton information of the patient acquired by the acquisition unit 141. FIG. 7 is a diagram for explaining an example of processing by the extraction unit 142 according to the first embodiment. In FIG. 7, (A) of FIG. 7 shows the color image information and joint position information of the patient collected by the motion information collection unit 10. FIG. 7B shows a first example in which the affected part is extracted from the information shown in FIG. FIG. 7C shows a second example in which the affected part is extracted from the information shown in FIG.

First, the first example will be described. The first example is a distance from a joint position of a hand at a predetermined timing to a line segment connecting two joints different from the joint of the hand in the position information of the joint in the motion information (skeleton information) of the patient. Is a case where a region corresponding to a position on the shortest line segment that is equal to or less than a predetermined threshold value is an affected part.

For example, the extraction unit 142 monitors patient skeleton information acquired in real time in a time-series order from the frame at the time when the diagnosis is started by the acquisition unit 141, and extracts the affected part from the position of the hand joint at a predetermined timing. To do. Here, the extraction unit 142, for example, indicates a point in time when a word indicating a human body part such as “the right arm hurts” or a pronoun word such as “here hurts” is emitted from the patient. It is possible to use. That is, the extraction unit 142 uses, as a predetermined timing, the time when the corresponding word is recognized in the speech recognition result generated by the speech recognition unit 13.

Also, the extraction unit 142 can use, for example, a point in time when the joint of the hand stops for a certain period of time at a position below a predetermined distance from the bone. That is, the extraction unit 142 uses the time as a predetermined timing when a certain place on the surface of the body is touched or pressed for a certain time. Note that whether or not the joint of the hand has stopped for a certain time at a position equal to or less than a predetermined distance from the bone is determined by analyzing changes in coordinates of the position of the hand joint in the skeleton information acquired by the acquisition unit 141. Can be done. Here, the allowable amount of coordinate change when determining that the joint of the hand is stopped and the stop time used for the determination can be arbitrarily set. For example, it may be set such that the allowable amount of coordinate change and the stop time are changed according to the age of the patient.

Note that the predetermined timing described above is merely an example, and can be arbitrarily applied in addition to these timings. For example, in the case of voice recognition, it may be the time when a doctor issues a word that prompts the patient to touch the affected part in addition to the word generated from the patient.

The extraction unit 142 extracts the affected part using the position information of the hand joint at the timing described above. For example, as illustrated in (B) of FIG. 7, the extraction unit 142 extracts the affected part using the position information of the joint identification information “2l” corresponding to the left hand and the information on the surrounding bone. That is, the extraction unit 142 first acquires coordinate information corresponding to the joint identification information “2l” of the left hand.

Then, the extraction unit 142 calculates the coordinate information of the portion corresponding to the bone from the coordinate information of the joint identification information corresponding to other joints in the same frame. For example, the extraction unit 142 performs the coordinate information of the bone between “2h” and “2g” shown in FIG. 7B, the coordinate information of the bone between “2g” and “2f”, and “2f” to “2e”. Bone coordinate information between “2e” and “2b”, Bone coordinate information between “2b” and “2i”, Bone coordinate information between “2i” and “2j”, “ The coordinate information of the bone between 2j ”and“ 2k ”is calculated.

Here, the extraction unit 142 calculates the coordinate information of the bone not connected to the joint identification information “2l” corresponding to the left hand. That is, the extraction unit 142 calculates the coordinate information of the bone that can be easily touched by the hand. Then, the extraction unit 142 calculates the distance from the left hand to each bone using the coordinate information of the left hand and the calculated coordinate information of each bone. For example, from the left hand “2l”, the extraction unit 142 performs the bones “2h” to “2g”, the bones “2g” to “2f”, the bones “2f” to “2e”, “2e” to “2e” The distances to the bone between “2b”, the bone between “2b” and “2i”, the bone between “2i” and “2j”, and the bone between “2j” and “2k” are calculated.

Then, the extraction unit 142 extracts a bone whose calculated distance is equal to or less than a predetermined threshold and whose distance is the shortest. For example, the extraction unit 142 extracts a bone between “2g” and “2f” that has a minimum distance from “2l” and is a predetermined threshold value or less. Then, the extraction unit 142 projects the tip of the hand onto the extracted bone and calculates a position on the bone. For example, as shown in FIG. 7B, the extraction unit 142 projects (dotted line) from “2l” to the bone between “2g” and “2f”, and projects the projected position (“2g” to “2g”). The position of the intersection between the bone and the dotted line between “2f” is calculated. After that, the extraction unit 142 calculates the ratio of the distance on the bone by calculating the distance from each joint at the intersection between the bone and the dotted line between “2g” and “2f”, for example.

That is, the extraction unit 142 calculates the distance between the intersection between the bone and the dotted line between “2g” to “2f” and the distance between “2g” and “2f” shown in FIG. The ratio between the distance from “2g” to the intersection and the distance from the intersection to “2f” with respect to the distance from “2f” is calculated. Then, the extraction unit 142 extracts the position of the calculated ratio in the bones “2g” to “2f” as the affected part.

Next, a second example will be described. In the second example, a case where a site corresponding to a predetermined movement range of the joint position of the hand in the position information of the joint in the motion information of the subject (patient) is used as the affected part will be described. For example, the extraction unit 142 extracts the affected part from the movement range of the joint of the hand. For example, as shown in FIG. 7C, the extraction unit 142 starts when the left hand “2l” moves in a reciprocating manner starting from the joint identification information “2j” corresponding to the left elbow. The coordinate information of the stop position of the left hand “2l” is calculated at both ends of the reciprocating movement. Then, the extracting unit 142 extracts the calculated coordinate information of both ends as both ends of the affected area, and extracts the area between them as the affected area.

That is, the extraction unit 142 extracts the range indicated by the double-ended arrows in FIG. Note that, similarly to the first example, the extraction unit 142 can be set to start calculating the distance between the hand joint and the bone at a predetermined timing. In such a case, the extraction unit 142 obtains coordinate information of joints other than the joints of the hand at the predetermined timing (speech recognition or monitoring the movement of the hand joint), and calculates the distance between the hand and the bone. To do. Then, the extraction unit 142 extracts a bone whose distance from the joint of the hand is equal to or less than a predetermined threshold and is the shortest, and analyzes the movement state of the joint of the hand with respect to the extracted bone. Here, the extraction unit 142 extracts only the range where the distance between the hand and the bone is moving below the predetermined threshold as the affected area.

Returning to FIG. 6, the selection unit 143 selects related information related to the affected part extracted by the extraction unit 142. For example, the selection unit 143 selects a schema for indicating the positional relationship between the human body structure and the affected part extracted by the extraction unit 142 as the related information. 8A to 8C are diagrams for explaining an example of processing performed by the selection unit 143 according to the first embodiment. FIG. 8A to FIG. 8C show a case where a schema is selected for three patterns extracted as a certain position by the extraction unit 142. The extraction of the affected area by the extraction unit 142 is the same as the method described above.

For example, as shown in the diagram of FIG. 8A, the selection unit 143 has a position indicated by a circle between joint identification information “2f” corresponding to the right elbow and joint identification information “2g” corresponding to the right wrist. , The schema on the front surface of the whole body including the position extracted as the affected part (the position marked with a circle) is selected. Here, when selecting a schema, the selection unit 143 selects a schema by combining information on a medical department in which a patient is undergoing medical care and information specialized by a doctor. In other words, the selection unit 143 selects a schema that takes into account the difference in the schema used in each department and the proper use by doctors. Note that information relating to the schema selection is set in advance and stored in the storage unit 130.

The selection unit 143 is a schema as described in the medical chart based on the information on the affected part extracted by the extraction unit 142, information on the schema selection, information on the medical department and doctor's specialty that the patient is visiting, and the like. Choose the best schema. Here, the selection unit 143 first selects a schema of a part including the affected part, and selects a schema that can more express the positional relationship of the affected part from a plurality of schemas corresponding to the selected part.

For example, as illustrated in FIG. 8B, when the extraction unit 142 extracts an affected part on the head, the selection unit 143 selects a plurality of schemas indicating the head (for example, see FIG. 2B). Then, the selection unit 143 determines that the affected part is the front surface of the head from the positional relationship of the coordinates of the left hand joint “2l” with respect to the coordinates of the head “2a”, and selects from among a plurality of schemas indicating the head Select the schema in front of the head.

Similarly, as illustrated in FIG. 8C, the selection unit 143 includes information on the schema corresponding to the affected part extracted by the extraction unit 142 and information on the specialty of the department and doctor who the patient is receiving. Select based on.

Returning to FIG. 6, the mark assigning unit 144 assigns information indicating the affected part position to the position corresponding to the affected part in the schema selected by the selecting unit 143. 9A to 9C are diagrams illustrating an example of processing performed by the mark assigning unit 144 according to the first embodiment. 9A to 9C show a case where information (marks) indicating the position of the affected area is given to each of the schemas selected in FIGS. 8A to 8C.

For example, as shown in FIG. 9A, the mark assigning unit 144 assigns the mark M1 to the right elbow portion of the schema on the front of the whole body selected by the selection unit 143. Here, the mark imparting unit 144 imparts a mark on the schema using the ratio information on the bone calculated by the extracting unit 142. Similarly, as shown in FIG. 9B and FIG. 9C, the mark assigning unit 144 assigns a mark M2 and a mark M3 on each schema using the ratio information on the bone calculated by the extracting unit 142.

The selection of the schema and the application of the mark when the affected part is extracted as the predetermined position by the extraction unit 142 have been described above. Next, a case where an affected part is extracted as a region will be described with reference to FIG. FIG. 10 is a diagram illustrating an example of schema selection and mark assignment according to the first embodiment. FIG. 10 shows the selection of the schema and the application of marks to the affected area extracted in FIG.

In such a case, as shown in FIG. 10, the selection unit 143 includes information on the affected area (area indicated by a double-ended arrow), information on selection of a schema, information on clinical departments, and information on specialist doctors. Select the front schema of the whole body. Here, the selection of the schema by the selection unit 143 is the same as the processing described above.

Then, as shown in FIG. 10, the mark assigning unit 144 assigns a mark M4 indicating the affected part to the left arm region on the schema selected by the selecting unit 143. Here, the mark assigning unit 144 determines and determines the region to which the mark on the schema is to be applied based on the information on the affected region extracted by the extracting unit 142 (coordinate information on the region indicated by the double-ended arrows) and the like. Mark the marked area.

Returning to FIG. 6, the display control unit 145 causes the output unit 110 to display the related information selected by the selection unit 143. Specifically, the display control unit 145 causes the output unit 110 to display the schema added with the mark by the mark adding unit 144 on the schema selected by the selection unit 143.

Here, the display control unit 145 extracts and extracts a schema in which the position where the mark is added by the mark applying unit 144 substantially matches the affected part from the past schema included in the patient's electronic medical record from which the affected part has been extracted. The schema is displayed on the output unit 110. FIG. 11 is a diagram for explaining an example of display control by the display control unit 145 according to the first embodiment.

For example, as illustrated in FIG. 11, the display control unit 145 compares a schema provided with a mark closer to the position of the mark provided this time from the chart information of the patient's electronic medical record as compared to the schema selected this time. The data is read out and displayed on the output unit 110. For example, the display control unit 145 acquires the chart information of the schema stored in the electronic chart of the same patient, and in the acquired schema, a mark is provided at a position close to the position of the mark applied this time. The previously selected schema is displayed on the output unit 110.

Also, the display control unit 145 can read out the chart information of the same schema as the currently selected schema and display it on the output unit 110. In addition, the display control unit 145 displays the latest schema when a plurality of schema information with the same mark as the schema selected this time or the schema with the same mark as the current selected schema is stored. It is also possible to control to display. In addition, the display control unit 145 displays all schemas in the case where a plurality of schema information of the same schema as the schema selected this time or the schema assigned the current mark is stored. It is also possible to control to display.

Returning to FIG. 6, the chart information storage unit 146 associates at least one of the voice of the patient's chief complaint from which the affected part is extracted and the image showing the position of the affected part of the patient with the related information in the storage unit. Store. Specifically, the medical record information storage unit 146 stores voice information of the main complaint issued by the patient in the medical record information storage unit 133 in association with the schema information.

FIG. 12 is a diagram for explaining an example of processing by the medical record information storage unit 146 according to the first embodiment. As shown in FIG. 12, when the patient complains to the doctor that “the elbow is uncomfortable and it hurts when bent” and the right elbow is pointed with the left hand, first, the acquisition unit 141, the extraction unit 142, the selection The unit 143 and the mark providing unit 144 perform the above-described processing, thereby inputting a mark to the schema. Here, the chart information storage unit 146 stores the voice data of the chief complaint “the elbow is uncomfortable and hurts when bent” included in the voice recognition result generated by the voice identification unit 13 in the chart information, A schema in which a voice data link is added to the mark given by the unit 144 is stored in the chart information storage unit 133.

For example, as shown in FIG. 12, the chart information storage unit 146 stores the chart information in which the voice file, schema, and mark ID are associated with the patient ID in the chart information storage unit 133. That is, the chart information storage unit 146 stores the chart information “patient ID: 100033, audio file: 2012-07-02-0005.mp3, schema: whole body, mark ID: 00000021” in the chart information storage unit 133, At the time, a voice link is given to the mark ID.

As described above, the doctor terminal 100 according to the first embodiment extracts the affected part from the operation information when the patient is examined, selects the optimum schema for the extracted affected part, and gives the mark. Can do. Therefore, it is possible to omit the work that the doctor has selected and provided with the mark conventionally, and the doctor terminal 100 according to the first embodiment can facilitate the information input of the electronic medical record. To do.

Next, processing of the doctor terminal 100 according to the first embodiment will be described with reference to FIG. FIG. 13 is a flowchart illustrating a processing procedure performed by the doctor terminal 100 according to the first embodiment. Although FIG. 13 shows the process when displaying a past schema, the past schema may not be displayed.

As illustrated in FIG. 13, in the doctor terminal 100 according to the first embodiment, when the voice storage mode is ON (Yes in step S101), the motion information collection unit 10 acquires the voice of the patient's main complaint. (Step S102). If the voice storage mode is OFF (No at Step S101), the process proceeds to Step S103 without executing Step S102.

In step S103, information about the subject (patient) is acquired (step S103), and the extraction unit 142 extracts the affected part from the patient motion information collected by the motion information collection unit 10 (step S104). Then, the selection unit 143 selects a schema corresponding to the affected part extracted by the extraction unit 142 (step S105), and the mark imparting unit 144 imparts a normal mark or a mark linked with the voice of the chief complaint to the schema. (Step S106).

Thereafter, the display control unit 145 displays the schema (step S107), further displays the past schema (step S108), and determines whether or not the save operation is accepted (step S109). Here, when the input unit 120 receives a storage operation from the operator (doctor) (Yes at Step S109), the medical record information storage unit 146 determines whether or not the voice storage mode is set (Step S110).

Here, if it is not the voice saving mode (No at Step S110), the medical record information storage unit 146 stores the medical record information in the medical record information storage unit 133 (Step S111), and the process is terminated. On the other hand, when it is the voice storage mode (Yes at Step S110), the medical record information storage unit 146 stores the medical record information associated with the voice data in the medical record information storage unit 133 (Step S112), and ends the process. . The doctor terminal 100 continues displaying the schema until accepting the save operation (No at Step S109).

As described above, according to the first embodiment, the acquisition unit 141 acquires skeleton information including position information of a patient's joint that is a target of motion acquisition. Then, the extraction unit 142 extracts the affected part based on the joint position information in the patient skeleton information acquired by the acquisition unit 141. Then, the selection unit 143 selects a schema related to the affected part extracted by the extraction unit 142. Then, the display control unit 145 controls the output unit 110 to display the schema selected by the selection unit 143. Therefore, the doctor terminal 100 according to the first embodiment can omit the selection work related to the medical record information of the electronic medical record, and can easily input the information of the electronic medical record.

Further, according to the first embodiment, the extraction unit 142 connects two joints different from the joints of the hand from the position of the joints of the hand at a predetermined timing in the joint position information in the skeleton information of the patient. The part corresponding to the position on the bone that has the shortest distance to the bone and is equal to or less than a predetermined threshold is extracted as an affected part. Therefore, the doctor terminal 100 according to the first embodiment can extract the affected area based on the action taken at the time of medical treatment of the patient, and enables accurate chart information selection processing.

Further, according to the first embodiment, the extraction unit 142 extracts a part corresponding to a predetermined movement range of the joint position of the hand as the affected part in the joint position information in the skeleton information of the patient. Therefore, the doctor terminal 100 according to the first embodiment makes it possible to cope with a case where the affected area covers a wide range.

Also, according to the first embodiment, the selection unit 143 selects the schema of the part extracted by the extraction unit 142 as the related information. Therefore, the doctor terminal 100 according to the first embodiment can omit a selection operation that tends to be complicated in inputting information in the electronic medical record, and makes it easier to input information.

Further, according to the first embodiment, the mark assigning unit 144 assigns information indicating the affected part position to the position corresponding to the affected part in the schema selected by the selecting unit 143. Then, the display control unit 145 controls the output unit 110 to display the schema to which the information indicating the affected area position is added by the mark adding unit 144. Therefore, the doctor terminal 100 according to the first embodiment can omit the work of adding a mark on the schema, and can further facilitate the input of information in the electronic medical record.

Further, according to the first embodiment, the display control unit 145 extracts the position to which the information on the affected part position is added by the mark applying part 144 from the past schema included in the electronic medical record of the patient from which the affected part is extracted. A schema that substantially matches the affected part is extracted, and the extracted schema is controlled to be displayed on the output unit 110. Therefore, the doctor terminal 100 according to the first embodiment can automatically read past medical data that is required to be compared with the current medical care.

Further, according to the first embodiment, the medical record information storage unit 146 corresponds to at least one of the voice of the patient's chief complaint from which the affected part is extracted and the image showing the position of the affected part of the patient as related information. At the same time, it is stored in the chart information storage unit 133. Therefore, the doctor terminal 100 according to the first embodiment makes it possible to save the information issued by the patient at the time of medical care as audio or video.

(Second Embodiment)
In the first embodiment described above, the case where a schema is selected as the related information has been described. In the second embodiment, a case will be described in which an input item corresponding to an affected part is selected and displayed in an electronic medical record. In the second embodiment, the selection processing by the selection unit 143 and the content of display control by the display control unit 145 are different from those in the first embodiment. Hereinafter, these will be mainly described.

The selection unit 143 according to the second embodiment selects items of the electronic medical record related to the affected part extracted by the extraction unit 142 as the related information. FIG. 14 is a diagram for explaining an example of processing by the selection unit 143 according to the second embodiment. For example, as shown in FIG. 14, the selection unit 143 selects from the contents of the doctor's examination for the patient (the act of placing a stethoscope on the chest) to display the heading “Auscultation Result” in the column of the findings of the electronic medical record. To do.

Further, as shown in FIG. 14, the selection unit 143 selects the auscultation result input screen for inputting the auscultation result from the contents of the doctor's examination for the patient (an act of applying a stethoscope to the chest). The display control unit 145 controls the output unit 110 to display the electronic medical record item selected by the selection unit 143.

Here, as shown in FIG. 14, when extracting a doctor's action on a patient, the doctor terminal 100 acquires the patient's motion information and the doctor's motion information in each frame. That is, the acquisition unit 141 acquires patient skeleton information and doctor skeleton information for each frame collected by the motion information collection unit 10. Then, the extraction unit 142 extracts a doctor's action (examination contents) on the patient from the patient's skeleton information (coordinate information of each joint) and the doctor's skeleton information (coordinate information of each joint) acquired by the acquisition unit 141. To do.

For example, the extraction unit 142 extracts a doctor's action (examination contents) on the patient from the positional relationship of the coordinates of the joints of the doctor's hand with respect to the coordinates of each joint of the patient. Information for extracting a doctor's action (examination contents) is set in advance and stored in the storage unit 130. For example, the storage unit 130 stores information indicating that a stethoscope is applied to the chest when the joint of the doctor's hand moves around the chest of the patient with a temporary stop. To do.

Further, it may be a case where a doctor is performing auscultation by detecting a stethoscope by pattern matching. In addition, what kind of method may be used for identification with a patient and a doctor in operation | movement information. For example, even when the position where the patient sits and the position where the doctor sits are set in advance in coordinates and the skeleton information acquired on the coordinates is identified as the skeleton information of the patient and the skeleton information of the doctor, respectively. Good.

As described above, according to the second embodiment, the selection unit 143 selects the item of the electronic medical record related to the affected part extracted by the extraction unit 142 as the related information. Therefore, the doctor terminal 100 according to the second embodiment can omit selection by the doctor of various items related to the creation of the electronic medical record, and can facilitate information input of the electronic medical record.

(Third embodiment)
Although the first and second embodiments have been described so far, the present invention may be implemented in various different forms other than the first and second embodiments described above.

In the first embodiment described above, the case where voice information including a patient's chief complaint is stored as medical record information has been described. However, the embodiment is not limited to this. For example, the image may be stored in association with the chart information. For example, the chart information storage unit 146 stores, as a still image or a moving image, an image of the patient touching the affected area in the color image collected by the color image collection unit 11 of the motion information collection unit 10.

In the first to third embodiments described above, the case has been described in which the operation information collection unit 10 collects operation information under a certain collection condition. However, the embodiment is not limited to this. For example, the collection condition may be changed according to the information on the affected part extracted by the extraction unit 142.

FIG. 15 is a diagram for explaining operation information collection condition changing processing by the doctor terminal 100 according to the third embodiment. For example, as illustrated in FIG. 15, when the extraction unit 142 extracts the patient's right arm as the affected part, the doctor terminal 100 controls the motion information collecting unit 10 to change the camera direction and zoom. . For example, as shown in FIG. 15, the doctor terminal 100 changes the direction of the camera so that the affected part of the right arm is located at the center of the screen, and further enlarges and captures the affected part in an appropriate size. To control.

Further, as shown in FIG. 15, the doctor terminal 100 can also control to cut out and save an image area extracted as an affected part in a captured color image. In addition, the doctor terminal 100 can measure, for example, the area and color of the affected part included in the color image based on the extracted information on the affected part, and can compare it with the results measured in the past. In addition, when the past image of the affected area is stored, the doctor terminal 100 controls to shoot the current affected area at the same enlargement rate as when the image was taken in the past and to display the affected area in parallel. Is also possible.

In the first embodiment described above, the case where the doctor terminal 100 extracts an affected area and selects and displays related information related to the affected area (for example, items such as a schema and an electronic medical record) has been described. However, the embodiment is not limited to this. For example, each process may be executed by the server device 300 on the network.

That is, the server apparatus 300 provides the doctor terminal 100 with the same processing as the doctor terminal 100. In such a case, the server apparatus 300 includes an acquisition unit 141, an extraction unit 142, a selection unit 143, and a display control unit 145. The acquisition unit 141 acquires skeleton information including position information of a joint of a patient that is a target of motion acquisition. Then, the extraction unit 142 extracts the affected part based on the joint position information in the patient skeleton information acquired by the acquisition unit 141. Then, the selection unit 143 selects a schema related to the affected part extracted by the extraction unit 142. Then, the display control unit 145 performs control so that the schema selected by the selection unit 143 is displayed on the output unit 110 of the doctor terminal 100.

In the first embodiment described above, the case where the doctor terminal 100 extracts an affected area and selects and displays related information related to the affected area (for example, items such as a schema and an electronic medical record) has been described. However, the embodiment is not limited to this. For example, a medical image diagnostic apparatus such as an ultrasonic diagnostic apparatus or an X-ray diagnostic apparatus may execute each process.

For example, when processing is executed in the ultrasonic diagnostic apparatus, the ultrasonic diagnostic apparatus first acquires coordinate information of a joint of a doctor's hand that operates an ultrasonic probe with respect to patient coordinate information. Then, the ultrasonic diagnostic apparatus extracts the affected part to which the ultrasonic probe is applied from the acquired coordinate information, and controls to display a body mark corresponding to the extracted affected part on an output unit such as a monitor. At this time, the ultrasonic diagnostic apparatus can also be controlled to display the body mark together with the ultrasonic image.

Also, the functions of the acquisition unit 141, the extraction unit 142, the selection unit 143, and the display control unit 145 described in the first to second embodiments can be realized by software. For example, the functions of the acquisition unit 141, the extraction unit 142, the selection unit 143, and the display control unit 145 have been described as being performed by the acquisition unit 141, the extraction unit 142, the selection unit 143, and the display control unit 145 in the above embodiment. This is realized by causing a computer to execute an operation information processing program that defines the above procedure. The operation information processing program is stored in, for example, a hard disk or a semiconductor memory device, and is read and executed by a processor such as a CPU or MPU. The operation information processing program can be recorded and distributed on a computer-readable recording medium such as a CD-ROM (Compact Disc-Read Only Memory), MO (Magnetic Optical disk), or DVD (Digital Versatile Disc). .

As described above, according to the first to third embodiments, the motion information processing system, the motion information processing device, and the medical image diagnostic device of the present embodiment can facilitate information input of electronic medical records. To do.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope of the present invention and the gist thereof, and are also included in the invention described in the claims and the equivalent scope thereof.

Claims

An acquisition unit that acquires motion information including position information of a joint of a subject who is a target of motion acquisition;
An extraction unit that extracts an affected area based on joint position information in the motion information of the subject acquired by the acquisition unit;
A selection unit for selecting related information related to the affected area extracted by the extraction unit;
A display control unit for controlling the related information selected by the selection unit to be displayed on the display unit;
An operation information processing system comprising:
In the position information of the joints in the motion information of the subject, the extraction unit has a predetermined distance from a joint position of the hand at a predetermined timing to a line segment connecting two joints different from the joint of the hand. The motion information processing system according to claim 1, wherein a part corresponding to a position on a line segment that is equal to or less than a threshold value and is shortest is extracted as the affected part.
The motion information processing according to claim 1, wherein the extraction unit extracts, as the affected part, a part corresponding to a predetermined movement range of the joint position of the hand in the joint position information in the motion information of the subject. system.
The motion information processing system according to claim 1, wherein the selection unit selects a schematic diagram of a part extracted by the extraction unit as the related information.
In the schematic diagram selected by the selection unit, further includes a granting unit that gives information indicating the affected part position to a position corresponding to the affected part
The motion information processing system according to claim 4, wherein the display control unit controls the display unit to display a schematic diagram to which information indicating the affected part position is provided by the providing unit.
The display control unit is a model in which the position where the information on the affected part position is given by the assigning unit substantially matches the extracted affected part from the past schematic diagram included in the electronic medical record of the patient from which the affected part is extracted. The motion information processing system according to claim 5, wherein a diagram is extracted and the extracted schematic diagram is displayed on the display unit.
The storage unit further stores at least one of the voice of the patient's main complaint from which the affected part is extracted and an image showing the position of the affected part of the patient in the storage unit in association with the related information. The operation information processing system described in 1.
The acquisition unit controls to change at least one of a camera orientation and an enlargement ratio for acquiring the motion information of the subject according to the position of the affected part extracted by the extraction unit. Item 2. The information processing system according to Item 1.
The motion information processing system according to claim 1, wherein the selection unit selects, as the related information, an item of an electronic medical record related to an affected part extracted by the extraction unit.
An acquisition unit that acquires motion information including position information of a joint of a subject who is a target of motion acquisition;
An extraction unit that extracts an affected area based on joint position information in the motion information of the subject acquired by the acquisition unit;
A selection unit for selecting related information related to the affected area extracted by the extraction unit;
A display control unit for controlling the related information selected by the selection unit to be displayed on the display unit;
A motion information processing apparatus comprising:
A medical image generation unit for generating a medical image;
An acquisition unit that acquires motion information including position information of a joint of a subject who is a target of motion acquisition;
An extraction unit that extracts an affected area based on joint position information in the motion information of the subject acquired by the acquisition unit;
A selection unit for selecting related information related to the affected area extracted by the extraction unit;
A display control unit that controls the display unit to display the medical image generated by the medical image generation unit and the related information selected by the selection unit;
A medical image diagnostic apparatus comprising: