JP6804293B2 - Terminal equipment - Google Patents

Description

The present disclosure relates to a terminal device, and more particularly to a terminal device for capturing a measurement result of the health device by imaging the health device.

Various measuring devices have been developed to measure various biological information regarding the health condition of the human body such as blood pressure and pulse. The health condition of the user is confirmed by incorporating the measurement data of such a measuring device into a terminal device for collecting the measurement data.

For example, Japanese Patent Application Laid-Open No. 2001-224557 (Patent Document 1) discloses a data input device. The data input device includes an image acquisition means for obtaining an image of the data displayed on the data display unit of the measuring instrument, a number reading means for reading the numbers in the obtained image, and a display means for displaying the read numbers. ..

Japanese Unexamined Patent Publication No. 2001-224557

The display data analysis device according to Patent Document 1 is considering using various markers attached to the measurement device in order to facilitate detection of measurement data from the captured image. However, Patent Document 1 has a problem that the user has to put various markers on the measuring device.

The present disclosure has been made in view of the above, and an object in a certain aspect is to provide a terminal device capable of more easily acquiring a measurement result displayed on a display unit of a health device. ..

A terminal device according to an embodiment is a character indicating a measurement result displayed on a display of a health device from an image pickup means for imaging a health device as a subject and an image of the health device included in the image captured by the image pickup means. The recognition means for recognizing the columns, the captured image generated by the imaging means, and the guide for assisting the user's operation when the recognition means recognizes the character string indicating the measurement result are displayed on the display of the terminal device. A display control means for causing the display is provided. The guide includes a first object that indicates the recognition range of the recognition means.

Preferably, the measurement result comprises the measurement value of the health device. The character string indicating the measured value is composed of characters in the 7-segment display system. The guide further includes a second object that is present within the first object and indicates each segment in the seven segments.

Preferably, the guide further includes a third object that represents the shape of the health device.
Preferably, the display control means displays the abnormality information on the display of the terminal device when the numerical value represented by the recognition character string, which is the recognition result of the character string indicating the measured value, is out of the predetermined range.

Preferably, the display control means makes each of the plurality of recognition characters included in the recognition character string, which is the recognition result of the character string indicating the measured value, a predetermined color according to the accuracy indicating the recognition accuracy of the recognition character. Is displayed on the display of the terminal device.

Preferably, the display control means provides information indicating the accuracy of recognition of the recognition character for each of the plurality of recognition characters included in the recognition character string which is the recognition result of the character string indicating the measured value. Display on the display of.

According to the present disclosure, it is possible to more easily obtain the measurement result displayed on the display unit of the health device.

It is a figure which shows the schematic structure of the system including a terminal device and a health device. It is a block diagram which shows an example of the hardware composition of a terminal device. It is a figure which shows an example of the appearance of the main body of a sphygmomanometer. It is a figure for demonstrating the display method (the 1) of a guide. It is a figure for demonstrating the display method (2) of a guide. It is a figure for demonstrating the display method (3) of a guide. It is a figure which shows the screen example which displayed the recognition result in real time. It is a figure for demonstrating the method of synthesizing a plurality of images. It is a figure for demonstrating the notification function of an imaging direction. It is a figure for demonstrating the method of generating the captured image using the HDR function. It is a block diagram which shows the functional structure of a terminal device. It is a flowchart which shows an example of the processing procedure of a terminal apparatus. It is a flowchart which shows the modification of the processing procedure of a terminal apparatus.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated.

<Overall configuration>
FIG. 1 is a diagram showing a schematic configuration of a system 1 including a terminal device and a health device.

With reference to FIG. 1, the system 1 includes a terminal device 10 which is a user terminal, blood pressure monitors 20 and 21 which are examples of health devices for measuring biometric information of the user, and a weight / body composition meter 23. .. The health device is not limited to the sphygmomanometers 20 and 21 and the body weight / body composition meter 23, and may be a sleep meter, a pedometer, a thermometer or the like capable of displaying the measurement results.

The sphygmomanometer 20 is a stationary upper arm type sphygmomanometer in which the main body and the cuff (arm band) are separate bodies. The sphygmomanometer 21 is a wristwatch-type sphygmomanometer in which the main body and the cuff are integrated. The sphygmomanometer 21 has a function of being worn on the wrist for a long time like a wristwatch and measuring pulsation continuously for 24 hours for each beat, and a function of being constantly worn and measuring by simply pressing the measurement start button. .. As a result, the sphygmomanometer 21 can constantly measure the user's blood pressure. The body weight / body composition analyzer 23 can measure body weight, body fat percentage, visceral fat level, subcutaneous fat percentage, etc., and displays the measurement results on a display. In the following, for convenience of explanation, the sphygmomanometer 20 will be described as a representative example of the “health device”.

The terminal device 10 is, for example, a smartphone having a camera and a touch panel. The terminal device 10 has a function of detecting (recognizing) the measurement result displayed on the display of each health device from the captured images generated by capturing the image of each health device using a camera. In the following, a smartphone, which is a mobile device, will be described as a typical example of a “terminal device”. However, the terminal device may be another terminal device having an imaging function such as a foldable mobile phone, a tablet terminal device, or a PDA (Personal Data Assistance).

Although the details will be described later, the terminal device 10 uses a camera to image (photograph) a health device in a state where the measurement result is displayed. The terminal device 10 recognizes the measurement result from the generated captured image by using various recognition methods, and outputs the recognized measurement result. For example, the terminal device 10 displays the recognized measured value (maximum blood pressure value, etc.) on the display of its own device.

<Hardware configuration>
(Terminal device)
FIG. 2 is a block diagram showing an example of the hardware configuration of the terminal device 10. With reference to FIG. 2, the terminal device 10 has, as main components, a processor 152, a memory 154, an input device 156, a display 158, a wireless communication unit 160, a communication interface 162, a camera 163, and the like. It includes a memory interface (I / F) 164, a communication interface (I / F) 166, a speaker 168, a microphone 170, and a direction sensor 172. Further, the terminal device 10 further includes a time measuring unit for measuring the time.

The processor 152 is typically an arithmetic processing unit such as a CPU (Central Processing Unit) or an MPU (Multi Processing Unit). The processor 152 functions as a control unit that controls the operation of each unit of the terminal device 10 by reading and executing the program stored in the memory 154. The processor 152 realizes each of the processes (steps) of the terminal device 10 described later by executing the program.

The memory 154 is realized by a RAM (Random Access Memory), a ROM (Read-Only Memory), a flash memory, or the like. The memory 154 stores a program executed by the processor 152, data used by the processor 152, and the like.

The input device 156 receives an operation input to the terminal device 10. Typically, the input device 156 is realized by a touch panel. The touch panel is provided on the display 158 having a function as a display unit, and is, for example, a capacitance type. The touch panel detects a touch operation on the touch panel by an external object at predetermined time intervals, and inputs the touch coordinates to the processor 152. However, the input device 156 may include a button or the like.

The wireless communication unit 160 connects to the mobile communication network via the communication antenna 162 and transmits / receives signals for wireless communication. As a result, the terminal device 10 can communicate with other communication devices via a mobile communication network such as LTE (Long Term Evolution).

The camera 163 is realized by, for example, a CCD (Charge Coupled Device) system, a CMOS (Complementary Mental Oxide Semiconductor) system, or another system.

The memory interface 164 reads data from the external storage medium 165. The processor 152 reads the data stored in the storage medium 165 via the memory interface 164 and stores the data in the memory 154. The processor 152 reads data from the memory 154 and stores the data in the external storage medium 165 via the memory interface 164.

The storage medium 165 is non-volatile such as a CD (Compact Disc), a DVD (Digital Versatile Disk), a BD (Blu-ray (registered trademark) Disc), a USB (Universal Serial Bus) memory, and an SD (Secure Digital) memory card. Includes media for storing programs.

The communication interface (I / F) 166 is a communication interface for exchanging various data with other devices, and is realized by an adapter, a connector, or the like. In this embodiment, BLE (Bluetooth (registered trademark) low energy) is adopted as the communication method. However, the communication method may be a wireless communication method using a wireless LAN or the like, or a wired communication method using USB (Universal Serial Bus) or the like.

The speaker 168 converts the voice signal given from the processor 152 into voice and outputs it to the outside of the terminal device 10. The microphone 170 receives a voice input to the terminal device 10 and gives a voice signal corresponding to the voice input to the processor 152.

The direction sensor 172 detects the direction in which the terminal device 10 is facing. Specifically, the direction sensor 172 detects in which direction the display surface of the display 158 is facing. For example, the directional sensor 172 has at least one of a 3-axis accelerometer, a 3-axis gyro sensor, and a 3-axis geomagnetic sensor. As a result, the terminal device 10 can acquire the posture information of the terminal device 10 when the user is holding the terminal device 10 by detecting the inclination of the own device on three axes using the direction sensor 172. it can. The terminal device 10 may acquire the orientation information by detecting the geomagnetic vector on the three axes by the three-axis geomagnetic sensor.

(Health equipment)
As the health device, it suffices if it can provide information processing as described later as a whole, and a known hardware configuration can be adopted. For example, a health device has a processor for executing various processes, a memory for storing programs and data, a display for displaying measurement results, and an input interface for receiving instructions from a user. Includes a time measuring unit for measuring time and various hardware used for measuring biological information. For example, the sphygmomanometer 20 includes a cuff that can be wrapped around the user's upper arm (or wrist), a cuff pressure adjusting mechanism such as a piezoelectric pump and an exhaust valve, and a pressure for detecting the cuff pressure as the various hardware. Includes a sensor and a temperature sensor that detects the outside air temperature.

FIG. 3 is a diagram showing an example of the appearance of the main body of the sphygmomanometer 20. With reference to FIG. 3, the main body 22 of the sphygmomanometer 20 includes a display 24, a forward / back button 26, a lamp 27, a user number switch 28, and a blood pressure measurement start / stop button 29. The outer surface of the main body 22 is provided with brand information 41 (for example, brand "AAA") and model information 43 of the sphygmomanometer 20 (for example, model "XYZ-ABC"). The user number switch 28 is used when the measurement results for two people are stored separately. In the example of FIG. 3, the user with the number "1" is selected. The lamp 27 is a lamp indicating the suitability of the cuff winding method, and is shown to be appropriate in the example of FIG.

The measurement result is displayed on the display 24. Specifically, the measurement result includes the measured value including the systolic blood pressure value (for example, 118 mmHg), the diastolic blood pressure value (for example, 78 mmHg) and the pulse rate (for example, 70 nights / minute) displayed in the region 50, and the region. The measurement date and time displayed in 52 (for example, 7:30 on July 1), the user number displayed in area 54 (for example, "1"), and the current temperature displayed in area 56 (for example, for example). 24 degrees) and various messages displayed in the area 58 (for example, a message indicating the completion of measurement). The measurement result may include information indicating the pulse wave state, information indicating the suitability of the cuff winding method, and the like.

<Recognition of measurement results>
The terminal device 10 according to the present embodiment has various functions used for recognizing the measurement result displayed on the health device. Various functions will be described below.

(Display of guide)
A guide display method used to assist the user's operation when the terminal device 10 recognizes a character string indicating a measured value of the blood pressure monitor 20 will be described. The "character" in the present embodiment means a symbol in which lines, dots, etc. are combined to transmit and record a word (language), and is not only an alphanumeric symbol, a kanji, a kana character, etc., but also a pictogram. , Stamps and other arbitrary figures with an identification code attached.

FIG. 4 is a diagram for explaining a guide display method (No. 1). When the terminal device 10 receives an instruction to shift to the character recognition mode from the user via the input device 156, the terminal device 10 shifts the operation mode of the terminal device 10 to the recognition mode and recognizes the character string included in the captured image. To start. The terminal device 10 automatically repeats the generation of the captured image, the display of the generated captured image, and the character recognition process at a predetermined cycle in the character recognition mode. With reference to FIG. 4, in the display method (No. 1), the screen 500 including a frame 510 for guiding the range in which the measured values (maximum blood pressure, diastolic blood pressure, pulse) of the sphygmomanometer 20 are stored in the character recognition mode. Is displayed on the display 158. Here, the screen 500 in FIG. 4 is a screen in which the captured image of the sphygmomanometer 20 and the frame 510 are superimposed.

The frame 510 is displayed as a frame indicating the outer four corners of the recognition range (analysis range) in which the character string can be recognized. Therefore, the inside of the frame 510 is the recognition range. The terminal device 10 executes a process of recognizing the measurement result imaged so as to fall within the recognition range. Specifically, the terminal device 10 recognizes various character strings included in the measurement result.

Here, the terminal device 10 stores the screen information indicating what kind of information is displayed in which area of the display 24 of the sphygmomanometer 20 in the memory 154. Therefore, for example, when the display 24 is imaged so as to fit within the frame 510 which is the recognition range, the terminal device 10 uses the screen information to determine what kind of information each recognized character string is. Can be identified. For example, in the terminal device 10, the plurality of character strings "118", "78", and "70" recognized in the area 50 of the display 24 housed in the frame 510 are the systolic blood pressure value, the diastolic blood pressure value, and the pulse rate, respectively. It can be specified that it is a character string indicating.

Further, the terminal device 10 receives an instruction from the user and identifies that the health device to be the subject is the sphygmomanometer 20. Alternatively, the terminal device 10 extracts the feature information (shape, model, brand) of the health device from the captured image, and compares the extracted feature information with the feature information of each health device stored in advance. Thereby, it may be specified that the health device to be the subject is the sphygmomanometer 20.

According to the display method (No. 1), the user can grasp that the inside of the frame 510 is the recognition range, so that the terminal device 10 can easily recognize the measured value.

FIG. 5 is a diagram for explaining a guide display method (No. 2). With reference to FIG. 5, in the display method (No. 2), the screen 530 including the object 520 representing the outline of the shape of the sphygmomanometer 20 is displayed on the display 158 during the character recognition mode. The terminal device 10 may further display an object indicating characters (brand information 41, etc.) provided on the outer surface of the sphygmomanometer 20. In the example of FIG. 5, the captured image of the sphygmomanometer 20 is not shown because the object 520 is easy to see.

By moving the terminal device 10, the user takes an image so that the sphygmomanometer 20 overlaps the object 520. When the sphygmomanometer 20 is imaged so as to overlap the object 520, the terminal device 10 recognizes various character strings included in the measurement result.

According to the display method (No. 2), the user can be guided to image the sphygmomanometer 20 at a position where the terminal device 10 can easily recognize the measurement result.

FIG. 6 is a diagram for explaining a guide display method (No. 3). With reference to FIG. 6, in the display method (No. 3), the screen 550 including the square semi-transparent object 540 and the semi-transparent objects 542, 544, 546 indicating each segment in the 7 segments in the character recognition mode. Is displayed on the display 158.

As shown in FIG. 3, the character string indicating the measured value (maximum blood pressure value, minimum blood pressure value, pulse rate) of the sphygmomanometer 20 is composed of characters in the 7-segment display system. Therefore, as in the example of FIG. 6, a translucent object 542 indicating each segment with respect to the systolic blood pressure value, a translucent object 544 indicating each segment with respect to the diastolic blood pressure value, and a translucent object 546 indicating each segment with respect to the pulse rate. Is displayed. Further, the translucent object 540 indicates a recognition range, and corresponds to an inner portion of the frame 510 in FIG. In the example of FIG. 6, the captured image of the sphygmomanometer 20 is not shown for the sake of easy viewing of each translucent object.

By moving the terminal device 10, the user images each object 542, 544, 546 so that the measured value displayed on the display 24 of the sphygmomanometer 20 overlaps. When each object 542, 544, 546 is imaged so that each measured value overlaps, the terminal device 10 recognizes various character strings included in the measurement result.

By using the screen information of the sphygmomanometer 20, the terminal device 10 can grasp in which area of the display 24 each measured value is displayed. Therefore, when the terminal device 10 is imaged so that the measured values overlap each of the translucent objects 542, 544, 546, the recognition range indicated by the translucent object 540 includes the entire display 24 included in the captured image. As described above, the position, size, etc. of each translucent object can be set.

According to the display method (No. 3), the user can be guided to image the sphygmomanometer 20 at a position where the terminal device 10 can easily recognize the measurement result. In the case of the 7-segment display method, since each segment is easy to read, there is an advantage that the user can easily adjust each measured value to each translucent object 542,544,546. However, when the character string indicating the measured value of the sphygmomanometer 20 is displayed by a display method other than the 7-segment display method (for example, a dot matrix display method), each translucent object 542,544,546 is also a dot. It is displayed in a matrix display method.

The terminal device 10 may use a display method that combines at least two of the above display methods (No. 1) to (No. 3).

(Use of recognition accuracy)
The terminal device 10 recognizes a character string indicating a measured value (hereinafter, also referred to as “displayed measured value”) displayed on the display 24 of the sphygmomanometer 20 from the captured image, and uses the recognized character string as the blood pressure. It is regarded as a total of 20 measured values. At this time, in order to improve the accuracy of the measured value displayed on the display 158 (that is, the degree of coincidence between the numerical value indicated by the displayed measured value and the numerical value indicated by the recognized character string) as much as possible, the terminal device 10 has recognized the characters. Use the accuracy that indicates the recognition accuracy of (recognition character).

First, the terminal device 10 executes a process of recognizing the display measurement value from the captured image. As a result, the terminal device 10 acquires the recognition character string recognized from the captured image and the accuracy of each recognition character included in the recognition character string. The accuracy may be calculated using a known technique. For example, the accuracy is calculated by comparing the feature amount of the character image with the dictionary pattern registered in advance in the memory 154.

Next, the terminal device 10 has an accuracy of less than the threshold value Th1 (for example, 85 %) for each of the plurality of recognized characters included in the recognition character string which is the recognition result of the character string indicating the display measurement value. Judge whether or not. For example, as shown in FIG. 4, it is assumed that the captured image contains the character string “118” indicating the systolic blood pressure value, and the recognition character string “118” is obtained by character recognition. In this case, the accuracy of each recognition character "1", "1", "8" is acquired as, for example, "85", "90", "95", respectively. The terminal device 10 determines whether or not each of the probabilities "85", "90", and "95" is less than the threshold Th1. The terminal device 10 makes the same determination for the character string indicating the diastolic blood pressure value and the character string indicating the pulse rate.

When the terminal device 10 determines that the accuracy of all the recognized characters corresponding to the displayed measured values is the threshold value Th1 or more, the terminal device 10 uses the recognized character string composed of these recognized characters as the measured value of the sphygmomanometer 20 and displays 158. To display. For example, since the accuracy "85", "90", and "95" for each of the plurality of recognized characters corresponding to the systolic blood pressure values are all thresholds Th1 or more, the terminal device 10 is the most measured by the sphygmomanometer 20. It is determined (deemed) that the hypertension value is "118", and the determination result (for example, the systolic blood pressure value "118") is displayed on the display 158.

On the other hand, when the accuracy of at least one recognized character among the plurality of recognized characters is less than the threshold value Th1, the terminal device 10 obtains a display measurement value from each of the captured images for a predetermined number of times (for example, twice). Executes the process of recognizing the indicated character string. For example, a character string "118" indicating a systolic blood pressure value is included in the captured image, a recognition character string "110" is obtained by recognition processing, and each recognition character "1", "1", "0" It is assumed that the accuracy is "85", "90", and "60", respectively. In this case, since the accuracy "60" for the recognition character "0" is less than the threshold value Th1, the terminal device 10 executes a process of recognizing the character string indicating the display measurement value from each of the captured images for a predetermined number of times. To do.

Next, the terminal device 10 is based on the recognition result of the character string indicating the display measurement value included in the first captured image and the recognition result of the character string included in each of the two additional captured images. Then, the recognition character string to be adopted as the measured value of the sphygmomanometer 20 is determined.

For example, a plurality of characters in the character string included in the first captured image are recognized as "1", "1", and "0", respectively, and the plurality of characters in the second captured image are "1", respectively. It is assumed that "1" and "8" are recognized and the plurality of characters in the third captured image are recognized as "1", "1" and "8", respectively. In this case, the recognition character obtained by recognizing the hundreds digit is only "1", and the number of times of recognition as the recognition character "1" is three times. Similarly, the recognition character obtained by recognizing the tens digit is only "1", and the number of times of recognition as the recognition character "1" is three times. On the other hand, there are "0" and "8" in the recognition character obtained by recognizing the ones digit character, the number of times of recognition as the recognition character "0" is one, and the recognition character "8". The number of times recognized as "" is two times.

Here, the terminal device 10 identifies that the most recognized recognition character is "8" among the plurality of recognition characters "0" and "8" which are the recognition results of the ones digit character. To do. The recognition character "1" is specified for each of the hundreds place character and the tens place character. The terminal device 10 determines the recognition character string "118" composed of the recognition characters "1", "1", and "8" specified in this way as the systolic blood pressure value, and displays the determination result on the display 158. indicate.

As described above, when the terminal device 10 determines that the accuracy of the recognition character is low, the terminal device 10 executes the recognition process a predetermined number of times again, and determines each recognition character adopted by the majority vote as the measured value of the blood pressure monitor 20. .. Therefore, the recognition character string composed of the recognition characters having low accuracy is not adopted as the measured value of the sphygmomanometer 20. Therefore, the measured value displayed on the display 24 of the sphygmomanometer 20 can be acquired more accurately.

(Real-time display of recognition results)
Here, it is assumed that the terminal device 10 erroneously recognizes a character string indicating a display measurement value included in the captured image. In this case, the terminal device 10 determines the erroneous recognition character string as the measured value of the sphygmomanometer 20, and the determination result is displayed. The user erroneously recognizes the measured value by comparing the measured value displayed on the display 24 of the sphygmomanometer 20 with the measured value displayed on the display 158 of the terminal device 10. To figure out. Therefore, the user needs to perform the work of causing the terminal device 10 to recognize the measured value displayed on the display 24 of the sphygmomanometer 20.

In order to avoid the above work as much as possible, a function of displaying the recognition result of the display measurement value included in the captured image on the display 158 in real time will be described.

FIG. 7 is a diagram showing a screen example in which the recognition result is displayed in real time. Here, the terminal device 10 repeats the generation of the captured image, the display of the captured image, and the recognition of the character string included in the captured image at a predetermined cycle in the character recognition mode. It is assumed that the maximum blood pressure value "133", the minimum blood pressure value "106", and the pulse rate "82" are displayed on the display 24 of the sphygmomanometer 20. Area 570 in FIG. 7 indicates an area where the screen brightness is high due to the reflection of external light or the like.

With reference to FIG. 7, the terminal device 10 recognizes the character strings “133”, “106”, and “82” of the systolic blood pressure value, the diastolic blood pressure value, and the pulse rate, respectively, from the captured image. The terminal device 10 displays the captured image, the frame 510, and the screen 600 including the objects 562, 564, 566 which are the recognition results thereof. For ease of explanation, the screen 600 shows only the captured image of the display 24 portion of the sphygmomanometer 20. Further, character strings other than the measured values displayed on the display 24 are also omitted.

The object 562 shows the recognition result of the character string indicating the maximum blood pressure value (that is, the recognition character string "133"), and the object 564 shows the recognition result of the character string indicating the minimum blood pressure value (that is, the recognition character string "133"). 106 "), and the object 566 shows the recognition result (that is, the recognition character string" 82 ") of the character string indicating the pulse rate. From this, the user can grasp that the measured value displayed on the display 24 of the sphygmomanometer 20 is correctly recognized by the terminal device 10. After that, the user can determine the recognized character string as the measured value of the sphygmomanometer 20 by, for example, pressing the enter button or the like.

Further, as shown in the example of FIG. 7, the terminal device 10 may display each of the plurality of recognized characters in a predetermined color according to the accuracy of the recognized characters. Specifically, among the objects 562 that are the recognition results of the character string indicating the systolic blood pressure value, the probability of the hundreds digit "1" is 70% or more and less than 90%, and the tens digit and the ones. The accuracy of the digit "3" is 90% or more. Therefore, the terminal device 10 displays the hundreds digit "1" in the object 562 and the tens digit and the ones digit "3" in different colors. As a result, the user can perform operations such as changing the direction of the camera 163 of the terminal device 10 while checking the recognition character string and the color of the recognition character string.

(Image composition)
Here, the function of synthesizing a plurality of captured images will be described. FIG. 8 is a diagram for explaining a method of synthesizing a plurality of images. Here, a method of synthesizing the captured image generated in a certain cycle p and the captured image generated in the next cycle p + 1 of the cycle p will be described. For ease of explanation, the captured images 722 and 726 show captured images of the display 24 portion of the sphygmomanometer 20. Further, the character strings other than the measured values displayed on the display 24 are also omitted.

With reference to FIG. 8, in the captured image 722 generated in the cycle p, each region showing the first place of the systolic blood pressure value, the first place of the diastolic blood pressure value, and the pulse rate is reflected by external light or the like. It overlaps with the area 752 where the screen brightness is high. Therefore, the accuracy of each recognized character, which is the recognition result of each character indicating the systolic blood pressure value, the diastolic blood pressure value, and the pulse rate, is low, for example, the threshold Th2 (for example, 40%). Is less than.

In this case, from the captured image 722, the terminal device 10 indicates each character (in this case, the hundreds and tens of digits of the systolic blood pressure value) corresponding to each remaining recognition character other than each recognition character having a certainty of less than the threshold Th2. A partial image 724 is generated in which a region containing a character and a character in the tens digit of the diastolic blood pressure value is cut out. The terminal device 10 stores the partial image 724 in the memory 154.

Next, in the captured image 726 generated in the cycle p + 1, the region showing the hundreds place of the systolic blood pressure value overlaps with the region 754 where the screen brightness is high. In this case, the terminal device 10 indicates from the captured image 726 each character corresponding to each remaining recognition character other than the recognition character having a certainty of less than the threshold Th2 (in this case, each character other than the hundreds digit character of the systolic blood pressure value). A partial image 728 is generated by cutting out an area including characters). The terminal device 10 stores the partial image 728 in the memory 154.

Then, the terminal device 10 synthesizes the partial image 724 and the partial image 728 stored in the memory 154 to generate the composite image 730. The terminal device 10 re-executes the process of recognizing the character string indicating the display measurement value from the composite image. Since the composite image contains only the characters corresponding to the recognition characters having a relatively high accuracy, the recognition accuracy of the character string indicating the display measurement value can be improved.

(Notification of imaging direction)
FIG. 9 is a diagram for explaining a notification function in the imaging direction. For ease of explanation, FIG. 9 shows only the captured image of the display 24 portion of the sphygmomanometer 20 among the captured images displayed on the screen of the display 158. It should be noted that the character strings other than the measured values displayed on the display 24 are also omitted.

With reference to FIG. 9, the terminal device 10 compares the captured image 702 included in the screen 810 displayed on the display 158 with the reference image 700, and detects the inclination of the captured image 702 with respect to the reference image 700. For example, the inclination is detected by comparing the shape, size, and the like of the display 24 included in the reference image 700 and the captured image 702. Further, in the example of FIG. 9, the depth direction, the short side direction, and the long side direction of the display 24 are defined as the X-axis direction, the Y-axis direction, and the Z-axis direction, respectively.

The terminal device 10 detects that the captured image 702 is tilted about the X-axis with respect to the reference image 700. The terminal device 10 displays an object 772 on the screen 810 that prompts the terminal device 10 to rotate about the X axis in order to reduce the inclination with respect to the reference image 700. When the terminal device 10 is rotated, a screen 840 including the captured image 708 is obtained. The terminal device 10 may use the attitude information of the three axes of its own device detected via the direction sensor 172 when calculating the rotation direction for reducing the inclination with respect to the reference image 700.

Further, the terminal device 10 compares the captured image 704 included in the screen 820 with the reference image 700, and detects the inclination of the captured image 704 with respect to the reference image 700. Subsequently, the terminal device 10 detects that the captured image 704 is tilted about the Y axis with respect to the reference image 700. The terminal device 10 displays an object 774 on the screen 820 that prompts the terminal device 10 to rotate about the Y axis in order to reduce the inclination with respect to the reference image 700. When the terminal device 10 is rotated, a screen 840 including the captured image 708 is obtained.

Further, the terminal device 10 compares the captured image 706 included in the screen 830 with the reference image 700, and detects the inclination of the captured image 706 with respect to the reference image 700. Subsequently, the terminal device 10 detects that the captured image 706 is tilted about the Z axis with respect to the reference image 700. The terminal device 10 displays an object 776 on the screen 830 that prompts the terminal device 10 to rotate about the Z axis in order to reduce the inclination with respect to the reference image 700. When the terminal device 10 is rotated, a screen 840 including the captured image 708 is obtained. As a result, the recognition accuracy of the character string indicating the display measurement value can be improved.

(High dynamic range composition)
In the case of a captured image with many overexposure or a captured image with many blackouts, it is highly possible that only recognition with low accuracy as a whole can be performed from the captured image. Therefore, the terminal device 10 may generate an image by using the HDR (high dynamic range imaging) function. FIG. 10 shows only the captured image of the display 24 portion of the sphygmomanometer 20. It should be noted that the character strings other than the measured values displayed on the display 24 are also omitted.

FIG. 10 is a diagram for explaining a method of generating a captured image using the HDR function. With reference to FIG. 10, when the display 24 of the sphygmomanometer 20 is imaged, the terminal device 10 adjusts the exposure amount to generate an image 742 with less overexposure and an image 744 with less blackout. To do. Then, the terminal device 10 generates a high dynamic range composite image 746 by synthesizing the captured image 742 and the captured image 744.

<Others>
The terminal device 10 according to the present embodiment further has the following functions.

(Specification of model, etc.)
The terminal device 10 includes identification information (device ID) of the health device, shape of the housing of the health device, character information provided in the housing, display arrangement position, and feature information such as a button or switch arrangement position. Is associated and stored in the memory 154.

The terminal device 10 compares the feature information extracted from the captured image of the health device with the feature information of various health devices stored in the memory 154, and thereby includes the type and model of the health device included in the captured image. , Brand, etc. can be specified.

(Judgment of abnormal value)
If the systolic blood pressure value, the diastolic blood pressure value, and the pulse rate obtained by recognizing the character string of the display measurement value included in the captured image are abnormal values, the terminal device 10 may notify the abnormality. Good. Specifically, the terminal device 10 notifies that the systolic blood pressure value is abnormal when the acquired systolic blood pressure value is not included in the predetermined range Ra (for example, 40 to 260 mmHg). When the acquired minimum blood pressure value is not included in the predetermined range Rb (for example, 40 to 260 mmHg), the terminal device 10 notifies that the minimum blood pressure value is abnormal. When the acquired pulse rate is not included in the predetermined range Rc (for example, 40 to 180 beats / minute), the terminal device 10 notifies that the pulse rate is abnormal. Further, when the diastolic blood pressure value is higher than the systolic blood pressure value, the terminal device 10 notifies that these values are abnormal.

(Elimination of duplicate data)
The terminal device 10 is configured so as not to store the same measurement result in duplicate in order to ensure the reliability of the measurement result and to effectively use the memory 154. Specifically, the terminal device 10 compares the acquired measurement result with the measurement result already stored in the memory 154, and determines whether or not they match. For example, the terminal device 10 compares the user number, the measurement date and time, and the measured value included in each measurement result with each other with the model specified by image recognition. Then, when each information of the acquired measurement result and each information of the stored measurement result completely match, the terminal device 10 does not store the acquired measurement result in the memory 154. As a result, the terminal device 10 can prevent the measurement results from being stored in duplicate.

<Functional configuration>
FIG. 11 is a block diagram showing a functional configuration of the terminal device 10. With reference to FIG. 10, the terminal device 10 includes an image pickup unit 202, an output control unit 204, a recognition unit 206, a determination unit 208, a determination unit 210, an abnormality determination unit 212, and an image generation unit 214. It includes a synthesis unit 216 and a tilt detection unit 218.

The imaging unit 202 captures a subject at predetermined intervals to generate an captured image. In the present embodiment, the imaging unit 202 generates an captured image including an image of the sphygmomanometer 20 as a subject. The image pickup unit 202 is a function realized mainly by linking the processor 152 and the camera 163. The imaging unit 202 sequentially outputs the generated captured images to the output control unit 204, the recognition unit 206, and the image generation unit 214.

The output control unit 204 causes the display 158 to display the captured images generated by sequentially capturing the images by the imaging unit 202. In one aspect, the output control unit 204 causes the display 158 to display the captured image generated by the imaging unit 202 and a guide for assisting the user's operation.

The guide is displayed to assist the user in the imaging operation so that the recognition process becomes easy when the character string indicating the measurement result displayed on the display 24 of the sphygmomanometer 20 is recognized by the recognition unit 206. Will be done. The guide includes an object (for example, frame 510, translucent object 540) indicating the recognition range of the recognition unit 206. The guide further includes an object (for example, a translucent object 542, 544, 546) indicating each segment in the seven segments, which exists in the object indicating the recognition range. In addition, the guide may include an object (eg, object 520) that represents the shape of the sphygmomanometer 20.

The recognition unit 206 recognizes a character string indicating the measurement result displayed on the display 24 of the sphygmomanometer 20 from the images of the sphygmomanometer 20 included in the image captured by the image pickup unit 202. The measurement results include, for example, the measured values (maximum blood pressure value, diastolic blood pressure value, pulse rate), measurement date and time, temperature, user number, various messages, information (symbol) indicating the suitability of cuff winding, and pulse wave state. Contains information (symbols) to indicate. Further, the recognition unit 206 has a plurality of recognition characters included in the recognition character string which is the recognition result of the character string (for example, the character string indicating the systolic blood pressure value) indicating the measurement value (display measurement value) displayed on the display 24. For each of the above, the probability of representing the recognition accuracy of the recognition character is calculated. The recognition unit 206 outputs the recognition result (each recognition character and accuracy) to the determination unit 208 and the determination unit 210.

The determination unit 208 determines whether or not the accuracy of the recognition character is less than the threshold Th1 for each of the plurality of recognition characters.

When the determination unit 208 determines that the accuracy of at least one recognition character among the plurality of recognition characters is less than the threshold Th1, the recognition unit 206 further generates N (N is) by the imaging unit 202. A process of recognizing a character string indicating a display measurement value is further executed from each of the captured images (an integer of 1 or more).

The determination unit 210 determines the recognition character string, which is the recognition result of the character string indicating the display measurement value, as the measurement value of the sphygmomanometer 20 based on a predetermined condition.

Specifically, when the determination unit 208 determines that the accuracy of at least one recognition character among the plurality of recognition characters is less than the threshold Th1, the determination unit 210 indicates a character string indicating a display measurement value ( For example, each character (characters "1", "1") included in the character string is based on the recognition result (that is, N + 1 recognition character string) of N + 1 times of the character string "118" representing the systolic blood pressure value. , "8"), of at least one recognition character obtained as a recognition result of the character (for example, two recognition characters "8", "0" obtained as a recognition result of the character "8"). Identify the most recognized characters.

The determination unit 210 determines each of the identified recognition characters (for example, the recognition character "1" specified for the hundreds digit "1" and the recognition character "1" specified for the tens digit "1". , A recognition character string (for example, the recognition character string "118") composed of the recognition character string "8" specified for the ones digit "8") is determined as a measurement value of the sphygmomanometer 20.

On the other hand, when the determination unit 208 determines that the accuracy for each of the plurality of recognition characters is the threshold value Th1 or more, the determination unit 210 sets the recognition character string composed of the plurality of recognition characters on the sphygmomanometer 20. Determined as a measured value. In other words, when the certainty of each recognition character included in the recognition character string first recognized by the recognition unit 206 is the threshold Th1 or more, the determination unit 210 determines the first recognized recognition character string. It is determined as a measured value of the sphygmomanometer 20.

The determination unit 210 stores the determination result in the memory 154 and outputs the determination result to the output control unit 204 and the abnormality determination unit 212. The output control unit 204 outputs the determination result of the determination unit 210. For example, the output control unit 204 causes the determination result to be displayed on the display 158. The output control unit 204 may output the determination result to the external device via the communication interface 166 (or the wireless communication unit 160), or output the determination result by voice via the speaker 168. May be good.

The abnormality determination unit 212 determines the presence or absence of an abnormality based on the numerical value represented by the recognition character string determined by the determination unit 210 as the measured value of the sphygmomanometer 20. For example, the abnormality determination unit 212 determines that there is no abnormality when the value is within the predetermined range, and determines that there is an abnormality when the value is outside the predetermined range. The abnormality determination unit 212 outputs the determination result to the output control unit 204. The output control unit 204 outputs the abnormality information when the abnormality determination unit 212 determines that there is an abnormality. For example, the output control unit 204 causes the display 158 to display an error notification indicating that the numerical value is out of the predetermined range.

Here, in another aspect, the determination unit 208 has a threshold value Th2 (<] for each of the plurality of recognition characters included in the recognition character string, which is the recognition result of the character string indicating the display measurement value, with respect to the recognition character. It may be further determined whether or not it is less than the threshold Th1). The determination unit 208 outputs the determination result to the image generation unit 214 or the tilt detection unit 218.

The determination unit 208 has an accuracy of less than the threshold Th2 for at least one recognition character (for example, the recognition character "0") among the plurality of recognition characters (for example, the recognition characters "1", "2", "0"). When the image generation unit 214 determines that, from the captured image generated by the imaging unit 202, the remaining recognition characters other than the at least one recognition character (for example, the recognition characters "1" and "2") A partial image (for example, partial image 724,728) is generated by cutting out an area including each character (for example, the characters "1" and "2") corresponding to. The image generation unit 214 outputs a partial image to the composition unit 216.

The compositing unit 216 synthesizes a plurality of partial images generated by the image generating unit 214. The compositing unit 216 outputs the composite image to the recognition unit 206. The recognition unit 206 executes a process of recognizing a character string indicating a display measurement value from the composite image (for example, the composite image 730) synthesized by the synthesis unit 216.

Further, in still another aspect, when the determination unit 208 determines that the accuracy of at least one recognition character among the plurality of recognition characters is less than the threshold Th2, the inclination detection unit 218 takes an image of the reference image. The inclination of the captured image generated by the unit 202 is detected. The output control unit 204 causes the display 158 to display the rotation direction for reducing the inclination of the captured image with respect to the reference image.

The output control unit 204 displays each of the plurality of recognition characters included in the recognition character string, which is the recognition result of the character string indicating the display measurement value, on the display 158 in a predetermined color according to the accuracy of the recognition character. It may be displayed. Further, the output control unit 204 may display information indicating the accuracy of the recognition character on the display 158. For example, the information indicating the certainty is displayed at a predetermined position in the frame 510. Further, the output control unit 204 may display information indicating the accuracy of the recognized character in the vicinity of the recognized character.

<Processing procedure>
FIG. 12 is a flowchart showing an example of the processing procedure of the terminal device 10.

With reference to FIG. 12, the processor 152 of the terminal device 10 generates an image captured by the sphygmomanometer 20 captured by the camera 163, and displays the captured image on the display 158 (step S10). The processor 152 displays a guide on the display 158 for assisting the user's operation (step S12). The processor 152 recognizes a character string indicating a measured value (displayed measured value) displayed on the display 24 of the sphygmomanometer 20 from the captured image (step S14).

The processor 152 has a threshold value Th1 or more for each of the plurality of recognition characters included in the recognition character string, which is a recognition result of the character string indicating the display measurement value (that is, the accuracy for each recognition character is all threshold values Th1 or more). Whether or not it is determined (step S16). When the accuracy for each of the plurality of recognized characters is equal to or higher than the threshold value Th1 (YES in step S16), the processor 152 determines the recognition character string composed of the plurality of recognized characters as the measured value of the sphygmomanometer 20. (Step S24), the determination result (numerical value indicated by the recognition character string) is displayed on the display 158 (step S26), and the process ends.

When the accuracy of at least one recognized character among the plurality of recognized characters is less than the threshold Th1 (NO in step S16), the processor 152 receives the captured image different from the captured image targeted for the recognition process. The recognition process of the character string indicating the display measurement value is re-executed (step S18). The processor 152 determines whether or not the recognition process of the character string indicating the display measurement value has been executed a predetermined number of times (step S20).

If the recognition process has not been executed a predetermined number of times (NO in step S20), the processor 152 repeats the process of step S18. When the recognition process is executed a predetermined number of times (YES in step S20), the processor 152 displays a recognition character string composed of each recognition character specified based on the recognition results for the predetermined number of times in the sphygmomanometer 20. It is determined as the measured value of (step S22). Specifically, the processor 152 selects, for each character included in the character string indicating the display measurement value, the recognized character with the highest number of recognitions among at least one recognized character obtained by recognizing the character. Identify. Then, the processor 152 displays the determination result on the display 158 (step S26), and ends the process.

(Modification example)
FIG. 13 is a flowchart showing a modified example of the processing procedure of the terminal device 10.

With reference to FIG. 13, the processes of steps S50 to S56 are the same as the processes of steps S10 to S16 in FIG. 12, and therefore the detailed description thereof will not be repeated. When the accuracy of each of the plurality of recognized characters included in the recognition character string, which is the recognition result of the character string indicating the display measurement value, is the threshold Th1 (for example, 85 %) or more ( YES in step S56), step S66, The process of S68 is executed. Since the processes of steps S66 and S68 are the same as the processes of steps S24 and S26 in FIG. 12, the detailed description thereof will not be repeated.

When the accuracy for at least one of the plurality of recognized characters is less than the threshold Th1 ( NO in step S56), the processor 152 has the accuracy for each of the plurality of recognized characters the threshold Th2 (for example, 40). %) Or not (step S58). When the accuracy for each of the plurality of recognized characters is the threshold value Th2 or more (YES in step S58), the processes of steps S60 to S64 and S68 are executed. Since the processes of steps S60 to S64 and S68 are the same as the processes of steps S18 to S22 and S26 in FIG. 12, the detailed description thereof will not be repeated.

When the accuracy of at least one recognized character among the plurality of recognized characters is less than the threshold Th2 (NO in step S58), the processor 152 determines each of the remainders other than at least one recognized character from the generated captured image. A partial image is generated in which a region including each character corresponding to the recognition character (that is, the recognition character having a relatively high accuracy of the threshold Th2 or more) is cut out (step S70). The processor 152 stores the generated partial image in the memory 154.

The processor 152 determines whether or not a predetermined number of partial images are stored in the memory 154 (step S72). If a predetermined number of partial images are not stored (NO in step S72), the processor 152 repeats the process from step S50. When a predetermined number of partial images are stored (YES in step S72), the processor 152 synthesizes the predetermined number of partial images (step S74). The processor 152 executes a process of recognizing a character string indicating a display measurement value from the composite image (step S76).

The processor 152 determines whether or not the accuracy of each of the plurality of recognition characters included in the recognition character string recognized from the composite image is the threshold Th1 or more (step S78). When the accuracy for each of the plurality of recognized characters is equal to or greater than the threshold value Th1 (YES in step S78), the processor 152 executes the process of step S66. When the accuracy of at least one recognized character among the plurality of recognized characters is less than the threshold Th1 (NO in step S78), the processor 152 gives an error notification (step S80) and ends the process.

<Advantage>
According to the present embodiment, when the terminal device 10 determines that the accuracy of the recognition character is low, it executes the recognition processing a predetermined number of times again, and the recognition character string composed of each recognition character adopted by the majority decision. Is determined as a measured value, and the determined recognition character string is displayed on the display 158. Therefore, the low-accuracy recognition characters are not displayed on the display 158 as they are, and the accuracy of the measured value displayed on the display 158 can be improved.

Further, according to the present embodiment, when recognizing the character string indicating the display measurement value, a guide for assisting the user's operation is displayed. Further, since an object showing a frame, a recognition range, each segment of the 7-segment display method, and the like is displayed as a guide, the user can be guided to image the blood pressure monitor 20 at an optimum position.

<Other embodiments>
(1) In the above-described embodiment, the terminal device 10 may evaluate the reliability of the measured value by using various information included in the measurement result. For example, the processor 152 (evaluation unit) of the terminal device 10 calculates the period from the measurement date and time to the imaging date and time, and determines whether or not the calculated period is within a predetermined period. If the calculated period is not within the predetermined period, the processor 152 evaluates that the reliability of the measurement result is low. In this case, the processor 152 may output an error notification that the calculated period is not within the predetermined period.

(2) In the above-described embodiment, the configuration for notifying the user of an appropriate imaging direction based on the inclination of the captured image 702 with respect to the reference image 700 has been described in FIG. 9, but the configuration is not limited to this. Specifically, the following steps A to F may be executed.

The terminal device 10 detects the inclination of the captured image 702 with respect to the reference image 700 (step A). Based on the tilt, the terminal device 10 corrects the captured image 702 so as to reduce the tilt with respect to the reference image 700 (step B). The terminal device 10 executes a process of recognizing a character string indicating a display measurement value from the corrected captured image 702 (step C). The terminal device 10 determines whether or not the accuracy of all the recognized characters corresponding to the displayed measured values is equal to or higher than a predetermined threshold value (for example, threshold value Th1) (step D).

When it is determined that the accuracy of all the recognized characters corresponding to the displayed measured values is equal to or higher than a predetermined threshold value, the terminal device 10 uses the recognized character string composed of these recognized characters as the measured value of the sphygmomanometer 20 and displays 158. Is displayed in (step E).

On the other hand, when the accuracy of at least one recognized character among the plurality of recognized characters is less than a predetermined threshold value, the terminal device 10 rotates the terminal device 10 in an appropriate direction in order to reduce the inclination with respect to the reference image 700. An object (for example, objects 772, 774, 776) prompting the user to be displayed is displayed on the display 158 (step F), and the process from step A is repeated again.

(3) In the above-described embodiment, it is also possible to provide a program that causes a computer to function and execute control as described in the above-mentioned flowchart. Such programs are recorded on non-temporary computer-readable recording media such as flexible disks, CD- ROMs (Compact Disk Read Only Memory), secondary storage devices, main storage devices, and memory cards that come with the computer. It can also be provided as a program product. Alternatively, the program can be provided by recording on a recording medium such as a hard disk built in the computer. The program can also be provided by downloading via the network.

The program may be a program module provided as a part of a computer operating system (OS), in which necessary modules are called in a predetermined array at a predetermined timing to execute processing. In that case, the program itself does not include the above module and the process is executed in cooperation with the OS. A program that does not include such a module may also be included in the program according to the present embodiment.

Further, the program according to the present embodiment may be provided by being incorporated into a part of another program. Even in that case, the program itself does not include the modules included in the other programs, and the processing is executed in cooperation with the other programs. A program incorporated in such another program may also be included in the program according to the present embodiment.

(4) The configuration exemplified as the above-described embodiment is an example of the configuration of the present invention, can be combined with another known technique, and a part thereof is not deviated from the gist of the present invention. It is also possible to change the configuration by omitting it. Further, in the above-described embodiment, the process or configuration described in the other embodiments may be appropriately adopted and implemented.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 system, 10 terminal devices, 20, 21 blood pressure monitor, 22 main unit, 23 body composition meter, 24,158 display, 26 forward / back button, 27 lamp, 28 user number switch, 29 start / stop button, 41 brand information , 43 model information, 152 processor, 154 memory, 156 input device, 160 wireless communication unit, 162 communication antenna, 163 camera, 164 memory interface, 165 storage medium, 166 communication interface, 168 speaker, 170 microphone, 172 direction sensor, 202 Imaging unit, 204 output control unit, 206 recognition unit, 208 judgment unit, 210 determination unit, 212 abnormality determination unit, 214 image generation unit, 216 synthesis unit, 218 tilt detection unit.

Claims (5)

It ’s a terminal device,
An imaging unit that captures images of health equipment as a subject,
Wherein the healthcare equipment image included in the captured image generated by the imaging unit, and the health recognition unit for recognizing a character string that indicates the measurement results displayed on the display of the device,
A captured image generated by the imaging unit, the measurement result display control unit that a guide for assisting a user operation, the display on the display of the terminal device when the recognition unit string recognizes showing With and
The guide is seen containing a first object representing a recognition range of the recognition unit,
The measurement result includes the measurement value of the health device.
The character string indicating the measured value is composed of characters of the 7-segment display method.
The guide further comprises a second object existing within the first object, indicating each segment of the seven segments . The terminal device according to claim 1 , wherein the guide further includes a third object representing the shape of the health device. Wherein the display control unit, when numbers recognized character string is a recognition result of the character string indicating the measurement value represents is out of the predetermined range, and displays the abnormality information on the display of the terminal device, to claim 1 or 2 The terminal device described. The display control unit sets each of the plurality of recognition characters included in the recognition character string, which is the recognition result of the character string indicating the measured value, in a predetermined color according to the accuracy indicating the recognition accuracy of the recognition character. The terminal device according to any one of claims 1 to 3 , which is displayed on the display of the terminal device. The display control unit provides information indicating the accuracy of recognition of the recognized character for each of the plurality of recognized characters included in the recognition character string which is the recognition result of the character string indicating the measured value. The terminal device according to any one of claims 1 to 3 , which is displayed on the display of the above.