JP2012183269A - Processing method for image data, program, and terminal unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a processing method which distinguishes numerical information obtained from images to correspond to the kind of measuring instruments or physical data.SOLUTION: The image data processing method includes: (500) storing a caught image data in a memory (114); (518) extracting numerical information from the image data; (520) extracting a shape feature from the image data; selecting a shape feature similar to the extracted shape feature from among a plurality of shape features which are stored in the memory and relate to the displays of a plurality of physical data measuring instruments; deciding the kind of a physical data measuring instrument to correspond to the selected shape feature; deciding a measurement data represented by the numerical information based on the decided kind; and (522, 524) storing the measurement data in the memory so as to correspond to the decided kind.

Description

  The present invention relates to processing for extracting numerical information from image data.

  For example, it is known that personal records or logs on a daily life such as personal meal images are stored in a server via a network and managed by the server. Also, for example, for health management, it is known that personal body data (for example, body weight, body temperature, blood pressure, etc.) measured using a measuring device is stored in a server via a network and managed by the server. It has been.

  A known data measurement and recording system includes a digital measuring instrument, a data reader for converting the measured digital information into information of a personal computer / keyboard interface, and a personal computer. The personal computer records the information of the personal computer / keyboard / interface output by the conversion with a text editor. The data reader includes a video camera that captures the displayed measurement value, a video signal-RGB conversion unit, a horizontal position setting transmission circuit, and a vertical / horizontal position setting synchronization unit. The data reading device further includes a light / dark judgment value setting switch, a setting position light / dark signal extraction unit, a setting confirmation monitor, a light / dark judgment unit, a light / dark signal buffer, a light / dark signal-value conversion unit, a personal computer keyboard interface And a conversion unit. The data reader analyzes the image of the measurement value displayed on the measuring instrument, and acquires the measurement value in the image.

  Built-in wireless function in the measuring instrument, for example, body data such as weight, blood pressure, etc. is measured by the measuring instrument, and the measured value is wirelessly transferred to a personal computer and stored, and further, the measured value is transmitted via a network. There are known services that are transmitted to a server, stored, and managed by the server.

  In a known network system, a device transmits a first notification including first information for specifying the device to the management server, and the management server performs a second notification including the type of image pattern. Is transmitted to the device, and the device displays the image pattern in the second notification on its own display screen. The management server transmits a third notification for instructing the search of the image pattern displayed by the device to the camera server, and the camera server transmits a fourth notification for instructing the photographing to the camera. When the camera captures the image pattern displayed by the device in response to the fourth notification, the camera transmits the captured image to the camera server. The camera server evaluates the captured image and transmits second information for specifying the camera and a fifth notification including a result of the evaluation to the management server. When the management server receives the fifth notification, the management server stores the first information and the second information in association with each other based on the evaluation result.

JP 2001-12976 A JP 2008-211412 A

Keiichi Masato, “Intel, reveals a new service for managing weight and blood pressure on a PC-“ Continue ”can be communicated by different manufacturers” [online], April 15, 2009, Impress Watch [Search March 4, 2011], Internet <http://kaden.watch.impress.co.jp/docs/news/20090415_125414.html>

  In a service in which measurement data related to a normal body is managed by a server, a user inputs measurement data using a personal computer or its keyboard. Furthermore, in the known data measurement and recording system described above, the measurement value displayed on the measuring instrument can be analyzed to obtain the measurement value in the image. However, in the known data measurement and recording system described above, the measurement data extracted from the image cannot be distinguished from any measurement data of body data such as body temperature, blood pressure, and the like, and thus different types I cannot manage the measurement data.

  Further, in the above-described known service, data measured by a specific measuring instrument can be wirelessly transferred to a server and managed by the server without a user input operation. However, the server cannot centrally manage a plurality of different types of body data measured by different measuring devices without distinguishing them from the user's input operation.

  It is advantageous that the inventor can distinguish measurement values acquired from images for each type of measuring device or body data in order to distinguish and manage various body data without a user input operation of data. I recognized.

An object of the embodiment is to make it possible to distinguish numerical information acquired from an image for each type of measuring instrument or body data.
Another object of the embodiment is to be able to distinguish and manage different body data without performing data input operation by the user.

  According to one embodiment of the present invention, an information processing device stores captured image data in a storage device, extracts numerical information from the image data, extracts geometric features from the image data, and the storage device Select a shape feature similar to the extracted shape feature from the plurality of shape features related to the display unit of each of the plurality of body data measuring devices stored in the corresponding to the selected shape feature Determining the type of the physical data measuring device to be determined, determining the measurement data represented by the numerical information based on the determined type, and storing the measurement data in the storage device in association with the determined type An image data processing method is provided for execution.

  According to one aspect of the embodiment, numerical information acquired from an image can be distinguished for each type of measuring instrument or body data, and different body data is distinguished and managed without performing data input operation by the user. be able to.

FIG. 1 shows an example of a system including a server device, an information processing terminal, and a mobile terminal connected via a network according to the embodiment. FIG. 2 shows an example of a schematic configuration of the information processing terminal of FIG. FIG. 3 shows an example of a schematic configuration of the mobile terminal of FIG. FIG. 4 shows an example of a schematic configuration of the processor of the information processing terminal or the portable terminal. FIG. 5 shows an example of a list of measuring instruments stored in the memory or hard disk drive of the information processing terminal or portable terminal. FIG. 6 shows an example of a list of measurement items in FIG. 5 stored in the memory or hard disk drive of the information processing terminal or portable terminal. FIG. 7A shows an example of a thermometer including a display area. FIG. 7B shows an example of an image of the display area of the thermometer of FIG. 7A taken by a camera or an imaging unit. FIG. 7C shows an example of a sequence of numbers and a circumscribed rectangle (broken line) of each number in the image of FIG. 7B extracted by the processor. FIG. 7D shows an example of a circumscribed rectangle (broken line) of each pixel group corresponding to the segment. FIG. 8A shows an example of a weight scale including a display area. FIG. 8B shows an example of an image of the display area of the weight scale of FIG. 8A taken by a camera or an imaging unit. FIG. 8C shows an example of a sequence of numbers and a circumscribed rectangle of each number in the image of FIG. 8B extracted by the processor. FIG. 9A shows an example of a sphygmomanometer including a display area. FIG. 9B shows an example of an image of the display area of the sphygmomanometer of FIG. 9A taken by a camera or an imaging unit. FIG. 10A shows an example of another sphygmomanometer including a display area. FIG. 10B shows an example of an image of the display area of the sphygmomanometer of FIG. 10A taken by a camera or an imaging unit. FIG. 11 shows an example of a flowchart for processing from photographing of a display image of measurement values to transmission of measurement data to the server device, which is executed by the processor of the information processing terminal or portable terminal. 12A and 12B show an example of a flowchart for processing for removing noise in an image including an image on the display unit of the measuring instrument in step 502 of FIG. (Explained in Fig. 12A) FIG. 13 shows an example of a flowchart for the process of grouping the pixels of the frame of the image in step 504 of FIG. FIG. 14 shows an example of grouping pixels in the display area in an image frame. FIG. 15 shows an example of a label table including label numbers, statistical information, and recognized numbers regarding the pixels in the display area of FIG. 16A to 16D show examples of flowcharts for the labeling process in step 670 of FIG. (Explained in Figure 16A) (Explained in Figure 16A) (Explained in Figure 16A) FIG. 17 shows an example of a flowchart for the label table registration process in step 780 of FIGS. FIG. 18 shows an example of a flowchart for the process of determining a pixel group representing an annular shape in step 508 of FIG. FIG. 19 shows an example of a flowchart for the process of extracting the pixel group representing the display frame of the display area in step 510 of FIG. FIG. 20 shows an example of a flowchart for the process of extracting the pixel group inside the pixel group representing the display frame in step 512 of FIG. 21A and 21B show an example of a flowchart for the process of combining pixel groups that are adjacent to each other in Step 514 of FIG. (Explained in Figure 21A) FIG. 22A shows an example in which circumscribed rectangles of two pixel groups partially overlap each other. FIG. 22B shows an example in which each of the two pixel groups is divided into blocks of three regions. FIG. 23 shows an example of a flowchart for the process of correcting the inclination of the image in step 516 of FIG. 24A and 24B show examples of correction of image tilt. FIG. 25 shows an example of a flowchart for the process of extracting a recognized number string in step 518 of FIG. FIG. 26 shows an example of a flowchart for the process of extracting features or parameters specific to the display unit of the measuring instrument in use in step 520 of FIG. FIG. 27 shows an example of a flowchart for the process of determining the measurement item corresponding to the extracted feature of the display unit and the measurement value represented by the extracted number string in step 522 of FIG. Yes. FIG. 28A shows an example of display of measurement items and measurement values on the display unit. FIG. 28B shows another example of display of measurement items and measurement values on the display unit. FIG. 29 shows an example of a display screen of history data of measurement data displayed on the display unit of the information processing terminal or portable terminal.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It is understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the invention.

  Non-limiting embodiments of the present invention will be described with reference to the drawings. In the drawings, similar components and elements have the same reference numerals.

  A measurement value extracted from an image by a known data measurement and recording system may be transmitted to a server via a network and managed by the server. However, with known data measurement and recording systems, it is not possible to distinguish between measurement data extracted from an image, for example body data such as body temperature or blood pressure, and therefore different types of measurements. The data cannot be managed. In addition, in the known service, various types of body data measured by different measuring devices cannot be distinguished and managed in a centralized manner without a user input operation.

  The inventor has recognized that it is advantageous to be able to distinguish measurement values acquired from images for each type of physical data in order to distinguish and manage various physical data without a user input operation. . Further, the inventor has limited types of measuring instruments used by general users, and acquires in advance information relating to the dimensional characteristics of display images of measured values for each measuring instrument and the data format of measurement items. I realized that it was easy. In addition, if the inventor can correctly recognize a string of numbers (numerical information) in the display image of the measured value, the inventor uses the string of numbers based on the known information regarding the dimension and shape characteristics of the measured value display of each measuring instrument. Recognized that the measurement items and measurements represented can be determined. The inventor has recognized that the function and accuracy for character recognition of the measurement item, measurement value, and unit image in the display image may be low, thereby reducing the processing load for character recognition.

FIG. 1 shows an example of a system including a server device 30, an information processing terminal 100, and a mobile terminal 102 connected via a network 5 according to the embodiment.
In FIG. 1, network 5 may include the Internet, a public switched telephone network (PSTN), a packet switched network (PSN), ISDN, and / or a mobile communication network. The information processing terminal 100 is an information processing apparatus, is connected to the digital camera 202, and captures or captures and stores data of an image on the display unit of the measuring instrument taken by the user from the digital camera 202. The information processing terminal 100 and the portable terminal 102 extract the measurement data from the stored image data by using the character recognition technique with reference to the known information regarding the measurement value display and measurement item of the measuring instrument, and are included in the image data. The shooting date / time is acquired and determined as the measurement date / time. The information processing terminal 100 and the portable terminal 102 transmit the measurement data to the server device 30 via the network 5 together with the measurement date and time information. The server device 30 receives and stores the measurement data together with the measurement date information, and manages such a large number of data as recording data over a long period.

  The information processing terminal 100 is a computer including, for example, a CPU (Central Processing Unit), a main storage device, a hard disk drive (HDD), a semiconductor memory, a bus, an input device, an output device, a communication interface, a drive for reading a recording medium, and the like. A personal computer, a PDA (Personal Digital Assistant), or a device. The mobile terminal 102 is a mobile phone including a CPU (Central Processing Unit), a semiconductor memory, a bus, an input device, an output device, a communication interface, a recording medium interface, an imaging unit, a speaker, a receiver, a microphone, and the like. Also good.

  The server device 30 is an information processing device, and includes, for example, a processor 312, a network interface (NW I / F) 344, a storage device 16 including a database, and a drive 328 for reading a recording medium. The drive 328 is for reading a recording medium 129 such as an optical disc on which a program is recorded.

FIG. 2 shows an example of a schematic configuration of the information processing terminal 100 of FIG.
In FIG. 2, the information processing terminal 100 may be a computer or a device such as a personal computer including a processor 112, a memory 114, an input / output interface (I / O) 116, an internal bus, and the like. The processor 112 may be a CPU (Central Processing Unit) for a computer. The processor 112 and memory 114 are coupled to an input / output interface 116. The memory 114 includes, for example, a main storage device and a semiconductor memory. The information processing terminal 100 further includes a display device (DSP) 122, an input unit or input device 126, a recording medium reading drive 128, and a hard disk drive (HDD) coupled to an input / output interface (I / O) 116. 130). The input unit 126 may include, for example, a keyboard, a pointing device such as a mouse or a touch pad, and a touch panel. The drive 128 is provided for reading a recording medium 129 such as an optical disc on which a program is recorded. The information processing terminal 100 further includes a USB interface (USB I / F) 142 and a network interface (NW I / F) 144 coupled to an input / output interface (I / O) 116. In the information processing terminal 100, the USB interface 142 may also function as an input unit that inputs at least an image signal from the digital camera 202.

  The processor 112 may be a dedicated processor implemented as, for example, an integrated circuit including image processing and image recognition functions. Further, the processor 112 may operate in accordance with an application program stored in the memory 114 and / or the hard disk drive 130. The application program may be stored in the recording medium 129, read from the recording medium 129 by the drive 128, and installed in the information processing terminal 100.

  The camera 202 is coupled to the information processing terminal 100 via the USB interface 142 by a USB cable. The information processing terminal 100 may be connected to the server device 30 via the network interface 144 and further via the external network 5.

FIG. 3 shows an example of a schematic configuration of the mobile terminal 102 of FIG.
In FIG. 3, the mobile terminal 102 may be, for example, a mobile phone or a mobile information processing terminal including a processor 112, a memory 114, an input / output interface (I / O) 116, an internal bus, and the like. The portable terminal 102 further includes a display device (DSP) 122, an input unit or input device 126, a removable memory 129, and an imaging unit 203 coupled to an input / output interface (I / O) 116. The input unit 126 may include, for example, a key and a touch panel. The removable memory 129 may include, for example, an SD memory and a writing / reading unit thereof. Mobile terminal 102 further includes a communication unit 146 coupled to an input / output interface (I / O) 116. Mobile terminal 102 may further include a microphone 140, a speaker 142, and a receiver 144 coupled to an input / output interface (I / O) 116.

  The processor 112 may be a dedicated processor implemented as, for example, an integrated circuit including image processing and image recognition functions. Further, the processor 112 may operate according to an application program stored in the memory 114. The application program may be stored in the removable memory 129, read from the removable memory 129, and installed in the mobile terminal 102.

FIG. 4 shows an example of a schematic configuration of the processor 112 of the information processing terminal 100 or the portable terminal 102.
The processor 112 includes, for example, a control unit 1120, a noise removal unit 1122, a binarization unit 1124, a grouping unit 1126, an annular determination unit 1128, a group extraction unit 1130, a group combination unit 1132, and an image correction unit 1134. Also good. The processor 112 may further include, for example, a number extraction unit 1136, a feature extraction unit 1138, a measurement data determination unit 1140, and another processing unit 1144. For example, the processing unit 1144 may include a communication processing unit. The control unit 1120 supplies control signals to the noise removal unit 1122, the binarization unit 1124, the grouping unit 1126, the annular determination unit 1128, the group extraction unit 1130, the group combination unit 1132, and the image correction unit 1134. The operation of these elements may be controlled. The control unit 1120 may further supply control signals to the number extraction unit 1136, the feature extraction unit 1138, the measurement data determination unit 1140, and the other processing unit 1144 to control the operation of these elements. As an alternative, the processor 112 of the server device 30 may include the elements 1122 to 1140.

  FIG. 5 shows an example of a list of measuring instruments stored in the memory 114 or the hard disk drive 130 of the information processing terminal 100 or the portable terminal 102. The list of measuring instruments includes, for example, measuring instrument sequence numbers, measuring instrument types (types), vertical length (height) (Wd) and horizontal length (width) (Hd) of the display area of the display unit. , The vertical height (Hc) of the character (number) representing the measurement value, the interval (Pc) between adjacent characters, and the thickness (Tc) of the character may be included. The list of measuring instruments further includes the number of items of measurement items (physical data type) of the measuring instrument, the number of measurement display lines, the item name of the measuring item (physical data type), the usage status (used / not used), Line spacing (Pl) (not shown), allowable range of values (not shown), etc. may be included. The data in the table of FIG. 5 may be created in advance by a health data management service provider or a user. The type (type) of the measuring device is, for example, a weight scale, body composition meter (body fat rate meter, subcutaneous fat rate meter, skeletal muscle rate meter, etc.), thermometer, blood pressure meter, pulse rate meter, etc. A combination (for example, a body weight / body composition meter) may be used. The type of measuring device may not be the name of the measuring device assigned by the manufacturer. Each measurement item may be, for example, body weight, body composition (body fat percentage, subcutaneous fat percentage, skeletal muscle percentage, etc.), body temperature, systolic blood pressure, diastolic blood pressure, pulse rate, and the like. The usage state (used / not used) is information indicating whether or not each measuring instrument is used, and is set by the user operating the information processing terminal 100 or the input unit 126 of the portable terminal. The list of measuring devices may be further stored in the storage device 16 of the server device 30.

  FIG. 6 shows an example of a list of measurement items in FIG. 5 stored in the memory 114 or the hard disk drive 130 of the information processing terminal 100 or the portable terminal 102. The list of measurement items may include the measurement item name, the data format of the measurement item (digits, numbers / characters, decimal point position), the unit of the measurement item, the allowable range of the measurement value of the measurement item, and the like. Memory 114 or hard disk drive 130 stores the latest and past user measurement data. The list of measurement items may be further stored in the storage device 16 of the server device 30.

  FIG. 7A shows an example of a thermometer 402 including a display area 404. FIG. 7B shows an example of an image 408 of the display area 404 of the display unit of the thermometer 402 of FIG. 7A taken by the camera 202 or the imaging unit 203. FIG. 7C shows an example of a string of numbers in the image 408 of FIG. 7B extracted by the processor 112 and a circumscribed rectangle of each number with a broken line. In this case, the display area 404 includes one measurement item per line. FIG. 7D shows an example of a circumscribed rectangle of each pixel group corresponding to each segment by a broken line. The circumscribed rectangle is not displayed but is a virtual one in processing.

  FIG. 8A shows an example of a scale 412 including a display area 414. FIG. 8B shows an example of an image 418 in the display area 414 of the scale 412 in FIG. 8A taken by the camera 202 or the imaging unit 203. FIG. 8C shows an example of a sequence of numbers in the image 418 of FIG. 8B extracted by the processor 112 and a circumscribed rectangle of each number. In this case, the display area 414 includes one measurement item per line.

  FIG. 9A shows an example of a sphygmomanometer including a display area 424. FIG. 9B shows an example of an image 428 of the display area 424 of the sphygmomanometer of FIG. 9A taken by the camera 202 or the imaging unit 203. In this case, the display area 424 includes a plurality of measurement items in a plurality of rows.

  FIG. 10A shows an example of another sphygmomanometer including a display area 434. FIG. 10B shows an example of an image 438 of the display area 434 of the sphygmomanometer of FIG. 10A taken by the camera 202 or the imaging unit 203. In this case, the display area 434 includes two measurement items in one row. As another form, the measuring device may switch and display a plurality of measured values in order on the display unit in accordance with a user operation. For example, the measuring device may sequentially switch and display the maximum blood pressure, minimum blood pressure, and pulse rate measurement values.

  The user measures the physical state or physical quantity such as body temperature, weight, blood pressure, etc. with a measuring instrument as shown in FIGS. 7A, 8A, 9A, 10A, and then displays the display area 404, 414, 424 or 434 on the camera 202 or Imaging is performed using the imaging unit 203 of the mobile terminal 102. The user stores the data of the captured images 408, 418, 128, and 438 in the memory 114 of the information processing terminal 100 or the portable terminal 102 or the storage area of the hard disk drive 130. The processor 112 of the information processing terminal 100 or the portable terminal 102 processes the stored image data, and extracts only one or more numeric columns representing measurement values as shown in FIG. 5C or 6C, for example. The decimal point may not be extracted.

  FIG. 11 shows an example of a flowchart for processing from the capture of a display image of measurement values to the transmission of measurement data to the server device 30, which is executed by the processor 112 of the information processing terminal 100 or the portable terminal 102. .

  In step 500, the processor 112 or its processing unit 1144 converts the image data including the image of the display unit of the measuring instrument taken by the camera 202 or the imaging unit 203 into the image of the memory 114 or the hard disk drive 130 according to the user's operation. Store in the storage area. The image data may be color multi-tone image data. The processor 112 or its processing unit 1144 processes the image data as necessary to generate image data of the luminance component Y.

  In step 502, the processor 112 or its noise removing unit 1122 adjusts the luminance of the image including the image on the display unit of the measuring instrument in the memory 114 or the hard disk drive 130, and removes the noise of the image using a filter. . In step 504, the processor 112 or the binarization unit 1124 binarizes the image to generate binarized luminance image data (monochrome, light and dark).

  In step 506, the processor 112 or its grouping unit 1126 groups the binarized image pixels (eg, black or dark pixels) into, for example, one or more pixel groups. For example, when each digit of the measurement value of the measuring instrument is a 7-segment display, a plurality of pixels representing one number are grouped into respective pixel groups representing two or more segments in the 7 segments. . For example, a plurality of pixels representing two upper and lower segments representing the number “1” are grouped into two pixel groups based on the adjacent relationship of the pixels. For example, a plurality of pixels representing seven segments representing the number “8” are grouped into seven corresponding pixel groups based on the adjacency relationship of the pixels.

  In step 508, the processor 112 or its ring determining unit 1128 determines and extracts a pixel group representing a ring from a plurality of pixel groups in the image. In step 510, the processor 112 or the circular determination unit 1128 selects one of the circular pixel groups in the image including the other pixel group on the inner side, and selects the innermost circular pixel group as the measuring instrument's The pixel group representing the display area or display frame is determined.

  In step 512, the processor 112 or its group extraction unit 1130 extracts a pixel group that may represent a number or a number segment inside the pixel group that represents the display frame in the image. In step 514, the processor 112 or its group combining unit 1132 combines adjacent pixel groups among the plurality of extracted pixel groups in the image, and sets the combined group of pixel groups representing each number, respectively. Form. In step 516, the processor 112 or its correction unit 1134 detects the inclination of each pixel group representing the number segment in the image, corrects the inclination of the image representing the number, and forms a number without inclination.

  In step 518, processor 112 or its number extractor 1138 recognizes each set of combined pixel groups in the image using character recognition techniques. The processor 112 or its number extraction unit 1138 may extract only numbers by recognizing only numbers from a plurality of pixel groups. The processor 112 or its numeral extraction unit 1138 extracts one or more numeric strings (numeric strings) in one or more regions separated from other pixel groups as numerical information. In step 520, the processor 112 or the feature extraction unit 1138 obtains the display area or display frame size shape and the size shape of each number in the display image of the measurement device used, and the display unit of the measurement device used. To extract features (information) or parameters specific to.

  In step 522, the processor 112 or its measurement data determination unit 1140 compares the extracted display feature or shape feature of the measuring device in the image with that of the list of measuring devices stored in the memory 114 or the like. The processor 112 or the measurement data determination unit 1140 selects, from the list, a type of measuring device having a similar or approximate feature as a type of measuring device to be used. The processor 112 or the measurement data determination unit 1140 further selects one or more measurement items (physical data types) corresponding to the type of the selected measuring instrument.

  The processor 112 or the measurement data determination unit 1140 further determines, as measurement data, a measurement value represented by a string of numbers (numerical information) extracted from the image based on the data format of the determined measurement item. In this case, if there are two or more measurement items corresponding to the type of the selected measuring instrument, the number of digits in the numeric string (numeric information) and the numeric value (size) are the data format (number of digits) of each measurement item. ), With reference to or compared with numerical tolerances, order of appearance, the measurement item and the measurement value represented by the numeric string are determined. In addition, when a plurality of numeric strings are extracted, each numeric string is compared with each other, referring to or compared with the data format (number of digits, etc.) of each measurement item, the allowable range of numerical values, the order of appearance, A measurement item and a measurement value represented by a numeric string (numerical information) may be determined.

  The processor 112 or the measurement data determination unit 1140 may further display the determined measurement item and measurement value on the display unit 122 to allow the user to confirm and determine the confirmed measurement item and measurement value as measurement data. Good. The processor 112 or the measurement data determination unit 1140 further stores the measurement value in the memory 114 or the hard disk 130 together with the measurement date / time information in association with the measurement device type and the measurement item (physical data type).

  In step 524, the processor 112 or the processing unit 1144 transmits the determined measuring instrument type, measurement item, and measurement value (measurement data) to the server device 30 via the network 5 together with the user's personal information and date / time information. To do. The server device 30 or its processor 312 receives the measurement device type, measurement item, and measurement value (measurement data), and associates the measurement value with the measurement device type and measurement item according to the measurement date and time information in the database (DB) of the storage device 16. ) And managed as history data or recorded data. Thereafter, the server device 30 or the processor 312 transmits the history data or the recording data stored in the storage device 16 in response to a request from the information processing terminal 100 or the portable terminal 102. The (processor 112) of the information processing terminal 100 or the portable terminal 102 receives the history data or the recorded data, and displays each measurement value in association with the measurement item on the display unit 122 in the order of measurement date and time information.

  12A and 12B show an example of a flowchart for the process of removing noise in an image including an image on the display unit of the measuring instrument in step 502 of FIG.

  In FIGS. 7B and 8B, the position of the central reference frame 420 for luminance correction in frames 408 and 418 of the captured image (picture) is indicated by a broken line. The horizontal and vertical sizes of the image frames 408 and 418 have a horizontal and vertical size xw × yh in pixels. The reference frame 420 has, for example, a size (xw / 10) × (yh / 10). The reference frame 420 is used to determine the maximum luminance Ymax that serves as a reference in the central region of the frames 408 and 418 of the image. The user captures the display area of the measuring instrument so that the center of the display areas 404 and 414 of the measuring instrument is located at the center of the monitor screen of the digital camera 202 or the display unit 122 of the portable terminal 102. Thereby, the center of the display area 404, 414 of the measuring instrument is positioned approximately at the center of the display area 404, 414 of the photographed measuring instrument.

  Since the pixels in the periphery of the display areas 404 and 414 or in the area outside the display areas 404 and 414 may be undesirably bright due to reflection of light to the measuring instrument or the like, the maximum brightness of the entire image is limited to the maximum brightness in the reference frame 420. It is desirable to process.

  Referring to FIG. 12A, processor 112 begins its processing at step 600. In step 602, the processor 112 sets the maximum luminance Ymax to an initial value 0 (zero) (Ymax = 0) and the center coordinates (xc, yc) of the reference frame 420 to the center coordinates (xw / 2, yh / 2) of the image. ((Xc, yc) = (xw / 2, yh / 2)). Further, the processor 112 sets the upper left coordinate (xt, yt) of the reference frame 420 to the coordinate (xc−xw / 20, yc−yh / 20) ((xt, yt) = (xc−xw / 20, yc-yh / 20)). Further, the processor 112 sets the lower right coordinate (xb, yb) of the reference frame 420 to the coordinate (xt + xw / 10, yt + yh / 10) ((xb, yb) = (xt + xw / 10, yt + yh / 10)). . Further, the processor 112 sets the coordinates (x, y) of the pixel of interest to the upper left initial value (xt, yt) of the reference frame 420 in order to execute processing for determining the maximum luminance Ymax in the reference frame 420. ((X, y) = (xt, yt)).

  In step 604, the processor 112 determines whether or not the luminance Y (x, y) of the pixel of interest at the coordinates (x, y) is larger than the current maximum luminance Ymax (Y (x, y)> Ymax?). If it is determined that the luminance Y (x, y) is greater than the maximum luminance Ymax, in step 606, the maximum luminance Ymax = Y (x, y) is set. If it is determined that the luminance Y (x, y) is not greater than the maximum luminance Ymax, the procedure proceeds to step 608.

  In step 608, the processor 112 increments the coordinate x of the target pixel by one pixel (x = x + 1). In step 610, the processor 112 determines whether or not the coordinate x is larger than the lower right coordinate xb of the reference frame 420 (x> xb?). If it is determined that the coordinate x is not larger than the coordinate xb of the reference frame 420, the procedure returns to step 604. If it is determined that the coordinate x is larger than the coordinate xb of the reference frame 420, in step 612, the processor 112 sets the coordinate x of the target pixel to an initial value x = xt, and sets the coordinate y of the target pixel to one pixel. Increment by (x = xt, y = y + 1).

  In step 614, the processor 112 determines whether or not the coordinate y of the target pixel is larger than the coordinate yb of the reference frame 420 (y> yb?). If it is determined that the coordinate y is not greater than the lower right coordinate yb of the reference frame 420, the procedure returns to step 604. If it is determined that the coordinate y is greater than the lower right coordinate yb of the reference frame 420, the procedure proceeds to step 616 in FIG. 12B. By steps 602 to 614, the maximum luminance of the pixels in the reference frame 420 is determined as the standard maximum luminance Ymax.

  Referring to FIG. 12B, in step 616, the processor 112 sets the coordinate (x, y) of the pixel of interest to the initial value (1, 1) of the upper left coordinate of the image frame 408 or 418 ((x, y). y) = (1, 1)).

  In step 618, the processor 112 determines whether or not the luminance Y (x, y) of the pixel of interest at the coordinates (x, y) is larger than the maximum luminance Ymax (Y (x, y)> Ymax?). If it is determined that the luminance Y (x, y) is greater than the maximum luminance Ymax, in step 620, the maximum luminance Y (x, y) = Ymax is set. If it is determined that the luminance Y (x, y) is not greater than the maximum luminance Ymax, the procedure proceeds to step 622.

  In step 622, the processor 112 increments the coordinate x of the target pixel by one pixel (x = x + 1). In step 624, the processor 112 determines whether the coordinate x is greater than the lower right coordinate xw of the frame 408, 418 of the image (x> xw?). If it is determined that the coordinate x is not greater than the coordinate xw, the procedure returns to step 618. If it is determined that the coordinate x is larger than the coordinate xw, in step 626, the processor 112 sets the coordinate x of the target pixel to an initial value x = 1, and increments the coordinate y of the target pixel by one pixel ( x = 1, y = y + 1).

  In step 628, the processor 112 determines whether or not the coordinate y of the pixel of interest is greater than the lower right coordinate yh of the frames 408 and 418 of the image (y> yh?). If it is determined that the coordinate y is not greater than the coordinate yh, the procedure returns to step 618. If it is determined that the coordinate y is greater than the coordinate yh, the procedure proceeds to step 630. Steps 616 to 628 correct the luminance Y (x, y) exceeding the maximum luminance Y (x, y) in the image frames 408 and 418 so as to be limited to the maximum luminance Ymax.

  In step 630, the processor 112 spatially filters the image frames 408, 418 with a filter having a known weighting factor, eg, a 3 × 3 size, to remove or reduce noise. In the spatial filtering, for example, the value of the pixel of interest and neighboring pixels of the 3 × 3 matrix is multiplied by the coefficient of the corresponding matrix position in the filter, and the value of the pixel of interest is corrected by the sum of the products. .

  Referring back to FIG. 11, in step 504, the processor 112 processes the luminance Y (x, y) of all the coordinates (x, y) in the frame of the image with the threshold Yth to obtain the luminance Y (x, y y) is binarized to the value “0” or “1”, and the value (0, 1) is set to the pixel Px (x, y) at the coordinate (x, y). In this case, the value “0” represents white or bright luminance, and the value “1” represents black or dark luminance. The threshold value Yth may be set to, for example, 1/2 of the maximum luminance Ymax, that is, Yth = Ymax / 2. For example, when the luminance Y (x, y) is the threshold value Yth or darker than that, the processor 112 sets the pixel Px (x, y) = 1 (dark), and the luminance Y (x, y) is the threshold value Yth. If it is brighter, the pixel Px (x, y) = 0 (bright) may be set. As an alternative, the processor 112 sets the pixel Px (x, y) = 1 (bright), for example, when the luminance Y (x, y) is the threshold Yth or brighter, and the luminance Y (x, y). May be set as pixel Px (x, y) = 0 (dark). The binarization of the image facilitates the recognition of numbers in the image.

  FIG. 13 shows an example of a flowchart for the process of grouping the pixels of the image frames 408, 418, 428, and 438 in step 506 of FIG.

FIG. 14 shows an example of grouping pixels in the display area (404, 414, 424, 434) in the image frame (408, 418, 428, 438).
In FIG. 14, the same label number or group number 11 to 45 is assigned to a plurality of pixels representing luminance 1 (dark) in an adjacent relationship or a plurality of pixels in one segment. In this case, as an example, in the binary image on the display unit of the measuring instrument, a label number is assigned to each pixel of an image including numerical values “51.8” and “32.4” in the upper and lower two rows. Yes.

  FIG. 15 shows an example of a label table including label numbers, statistical information, and recognized numbers regarding pixels in the display area (404, 414, 424, 434) of the measuring instrument in the image of FIG.

  In FIG. 15, the left item numbers or row numbers 1 to 35 correspond to different label numbers (column 2), and the upper column numbers 1 to 13 are statistical information, dimension shapes, Corresponds to items such as shape features, recognition target, recognition results (numbers), and the like. Columns 2 to 8 show the number of pixels, the minimum value (xmin) of the coordinate x of the pixel, the minimum value (ymin) of the coordinate y, the maximum value (xmax) of the coordinate x for the pixel group of each label number in column 1. The maximum value (ymax) of the coordinate y, the coordinate x (xc) of the central pixel, and the coordinate y (yc) of the central pixel are included. Columns 9 to 12 include, for the pixel group of each label number in column 1, whether it is circular, whether it is an enclosure (envelopment line or outline), whether it is a recognition target, a label combination number, and a number of recognition results. Yes. One label combination number is an identification number for combining a plurality of adjacent pixel groups as a set that may represent one number.

  Referring to FIG. 13, processor 112 starts its processing at step 660. In step 662, the processor 112 sets the label number i_LBL as an initial value i_LBL = 11, and the coordinates (x, y) of the target pixel as initial values x = 2 and y = 2. Here, the label number i_LBL is set to an integer of 11 or more (i_LBL ≧ 11) in order to distinguish it from the luminance values “0” (bright) and “1” (dark).

  In step 664, the processor 112 determines whether or not the label number i_LBL is greater than an upper threshold value i_LBL_max for the label number. If it is determined that the label number i_LBL is greater than the threshold value i_LBL_max, the procedure proceeds to step 680. If it is determined that the label number i_LBL is not greater than the threshold value i_LBL_max, in step 666, the processor 112 binarizes the value of the pixel of interest Px (x, y) at the coordinates (x, y) to “1”. Determine whether it is (dark). In this case, the luminance “1” represents a black or dark level, and the luminance “0” represents a white or bright level. When the target pixel Px (x, y) has already been labeled, the target pixel has a value of Px (x, y) ≧ 11. If it is determined that the pixel of interest Px (x, y) has a luminance of 1 (dark), the processor 112 labels the pixel of interest and its neighboring pixels in step 670. Thereafter, the procedure proceeds to Step 671. If it is determined in step 666 that the pixel Px (x, y) is not luminance 1 (dark), that is, luminance 0 (zero) (bright) or label number (≧ 11), the procedure proceeds to step 672.

  In step 670, the same label number i_LBL (≧ 11) is assigned to the group of pixels Px (x, y) having luminance 1 (dark) adjacent to each other in the image. Step 670 is executed as many times as the number of pixel groups of luminance 1 (dark), that is, the number of label numbers i_LBL.

  In step 671, the processor 112 increments the label number i_LBL by 1 (i_LBL = i_LBL + 1).

  In step 672, the processor 112 increments the coordinate x of the target pixel by one pixel (x = x + 1). In step 674, the processor 112 determines whether the coordinate x is greater than or equal to the lower right coordinate xw of the image frames 408, 418 (x ≧ xw?). If it is determined that the coordinate x is not greater than or equal to the coordinate xw, the procedure returns to step 664. If it is determined that the coordinate x is greater than or equal to the coordinate xw, in step 676, the processor 112 sets the coordinate x of the target pixel to the initial value x = 2, and increments the y coordinate of the target pixel by one pixel. (X = 2, y = y + 1).

  In step 678, the processor 112 determines whether or not the coordinate y of the pixel of interest is greater than or equal to the lower right coordinate yh of the image frames 408 and 418 (y ≧ yh?). If it is determined that the coordinate y is not greater than or equal to the coordinate yh, the procedure returns to step 664. If it is determined that the coordinate y is greater than or equal to the coordinate yh, the procedure proceeds to step 680.

  Steps 662 to 678 replace all pixels Px (x, y) = 1 with luminance 1 (dark) in the frames 408 and 418 of the image with the label number (Px (x, y) ≧ 11). In step 680, the processor 112 sets the total number of label numbers i_LBL or the maximum label order number It to be the last maximum value label number It = i_LBL-10. This is because the initial value of the label number i_LBL is set to i_LBL = 11.

  16A to 16D show examples of flowcharts for the labeling process in step 670 of FIG.

  Referring to FIG. 16A, processor 112 begins its processing at step 700. In step 702, the processor 112 sets the label number i_LBL to the target pixel Px (x, y) at the coordinates (x, y) (Px (x, y) = i_LBL). The first pixel Px (x, y) is, for example, Px (x, y) = i_LBL = 11 (label number). The processor 112 uses the label number L_TBL (I, 1) = L_TBL (i_LBL-10, as item 1 (column 1) of the label table of the label order number I (= i_LBL-10) in the table of labels in FIG. 1) is set as the label number i_LBL of the current pixel of interest. The processor 112 sets the total number of pixels having the label number i_LBL (number of pixels) L_TBL (I, 2) as the initial value L_TBL (I) as the item 2 (column 2) of the label table with the label order number I = i_LBL-10. , 2) = 1. The processor 112 uses the minimum x-coordinate L_TBL (I, 3) on the left side of the pixel having the label number i_LBL as the label table item 3 and 4 (columns 3 and 4) of the label order number I = i_LBL-10 and the label The upper minimum y-coordinate L_TBL (I, 4) is set as coordinates x and y, respectively. The processor 112 uses the maximum x coordinate L_TBL (I, 5) on the right side of the pixel having the label number i_LBL and the lower side as items 5 and 6 (columns 5 and 6) of the label table with the label order number I = i_LBL-10. Are set to coordinates x and y, respectively.

  In step 704, the processor 112 sets the processing count (count value) i_CNT as an initial value i_CNT = 0, and sets the coordinate parameter (xs, ys) as (xs, ys) = (2, 2). In step 705, the processor 112 determines whether or not the pixel of interest P (xs, ys) at the coordinates (xs, ys) is the current label number i_LBL (P (xs, ys) = i_LBL?). When it is determined that the pixel P (xs, ys) = i_LBL is not satisfied, the procedure proceeds to Step 762 in FIG. 16D. If it is determined that the pixel P (xs, ys) = i_LBL, the procedure proceeds to step 706.

  In step 706, the processor 112 sets coordinate parameters xm = xs−1, xp = xs + 1, ym = ys−1, and yp = ys + 1 for selecting a pixel adjacent to the pixel of interest at coordinates (xs, ys). . A pixel position xm = xs−1 represents a position on the left side of the coordinate xs of the target pixel. The pixel position xp = xs + 1 represents the position on the right side of the coordinate xs of the target pixel. A pixel position ym = ys−1 represents a position above the coordinate ys of the target pixel. A pixel position yp = ys + 1 represents a position below the coordinate ys of the target pixel. Based on these four coordinate parameters and the coordinates (xs, ys) of the pixel of interest, the positions of eight adjacent pixels out of the 3 × 3 coordinate positions are selected.

  In step 708, the processor 112 determines whether or not the pixel Px (xm, ym) at the upper left coordinate (xm, ym) of the target pixel Px (x, y) is Px (xm, ym) = 1 (dark). . If it is determined that the adjacent pixel Px (xm, ym) is not luminance 1 (dark), that is, luminance 0 (zero) (bright) or label number (≧ 11), the procedure proceeds to step 722 in FIG. 16B. If it is determined that the pixel Px (xm, ym) has luminance 1 (dark), in step 710, the processor 112 sets the current value 1 of the pixel Px (xm, ym) to the current label number i_LBL (≧ 11). And the pixel Px (xm, ym) is set to the current label number i_LBL (≧ 11). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to Step 722 of FIG. 16B.

  In step 722 of FIG. 16B, the processor 112 determines whether or not the pixel Px (x, ym) of the coordinate (x, ym) on the target pixel P (x, y) is Px (x, ym) = 1 (dark). Determine. If it is determined that the adjacent pixel Px (x, ym) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 732. If it is determined that the pixel Px (x, ym) has a luminance of 1 (dark), in step 724, the processor 112 sets the current label number i_LBL to the pixel Px (x, ym). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to Step 732.

  In step 732, the processor 112 determines whether or not the pixel Px (xp, ym) at the upper right coordinate (xp, ym) of the pixel of interest P (x, y) has a luminance of 1 (dark). If it is determined that the adjacent pixel Px (xp, ym) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 738. If it is determined that the pixel Px (xp, ym) has luminance 1 (dark), in step 734, the processor 112 sets the current label number i_LBL to the pixel Px (xp, ym). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to Step 738.

  In step 738, the processor 112 determines whether or not the pixel Px (xm, y) at the left coordinate (xm, y) of the target pixel P (x, y) has a luminance of 1 (dark). If it is determined that the adjacent pixel Px (xm, y) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 738. If it is determined that the pixel Px (xm, y) has a luminance of 1 (dark), in step 740, the processor 112 sets the current label number i_LBL to the pixel Px (xm, y). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to Step 742 in FIG. 16C.

  In Step 742 of FIG. 16C, the processor 112 determines whether or not the pixel Px (x, yp) at the right coordinate (x, yp) of the pixel of interest P (x, y) has luminance 1 (dark). If it is determined that the pixel Px (x, yp) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 748. If it is determined that the pixel Px (x, yp) is luminance 1 (dark), in step 744, the processor 112 sets the current label number i_LBL to the pixel Px (x, yp). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to step 748.

  In step 748, the processor 112 determines whether or not the pixel Px (xm, yp) at the lower left coordinate (xm, yp) of the target pixel P (x, y) has a luminance of 1 (dark). If it is determined that the adjacent pixel Px (xm, yp) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 752. If it is determined that the pixel Px (xm, yp) has luminance 1 (dark), in step 750, the processor 112 sets the current label number i_LBL to the pixel Px (xm, yp). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to step 752.

  In step 752, the processor 112 determines whether or not the pixel Px (x, yp) at the coordinates (x, yp) under the target pixel P (x, y) has a luminance of 1 (dark). If it is determined that the pixel Px (x, yp) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 758 in FIG. 16D. If it is determined that the pixel Px (x, yp) is luminance 1 (dark), in step 754, the processor 112 sets the current label number i_LBL to the pixel Px (x, yp). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to Step 758 of FIG. 16D.

  In Step 758 of FIG. 16D, the processor 112 determines whether or not the pixel Px (xp, yp) at the lower right coordinate (xp, yp) of the pixel of interest P (x, y) has luminance 1 (dark). If it is determined that the pixel Px (xp, yp) is not luminance 1 (dark), that is, luminance 0 (bright) or label number (≧ 11), the procedure proceeds to step 762. If it is determined that the pixel Px (xp, yp) has luminance 1 (dark), in step 760, the processor 112 sets the current label number i_LBL to the pixel Px (xp, yp). In step 780, the processor 112 performs label table registration as described below. Thereafter, the procedure proceeds to Step 762.

  In step 762, the processor 112 increments the coordinate xs of the target pixel by one pixel (xs = xs + 1). In step 764, the processor 112 determines whether the coordinate xs is less than the lower right coordinate xw of the frame 408, 418 of the image (xs <xw?). If it is determined that the coordinate xs is smaller than the coordinate xw, the procedure returns to step 705 in FIG. 16A. If it is determined that the coordinate xs is not smaller than the target xw, in step 766, the processor 112 sets the coordinate xs of the target pixel to an initial value xs = 2 and increments the coordinate y of the target pixel by one pixel. (Xs = 2, ys = ys + 1).

  In step 768, the processor 112 determines whether or not the coordinate y of the pixel of interest is smaller than the lower right y coordinate yh of the frames 408 and 418 of the image (y <yh?). If it is determined that the coordinate y is smaller than the coordinate yh, the procedure returns to step 705 in FIG. 16A. If it is determined that the coordinate y is not smaller than the coordinate yh, in step 770, the processor 112 determines whether the count i_CNT is larger than zero. If it is determined that the count i_CNT is greater than 0, the procedure returns to step 704 in FIG. 16A. If it is determined that the count i_CNT is not greater than 0, the procedure exits the routines of FIGS. 16A-16D because there are no other newly labeled pixels and no newly labeled pixels.

  16A to 16D, a label number of any pixel group is assigned to all pixels having luminance 1 (dark) in the image. The label number assignment process is not limited to this.

  FIG. 17 shows an example of a flowchart for the label table registration process in step 780 of FIGS.

  Referring to FIG. 17, processor 112 starts its processing at step 780. In step 782, the processor 112 increments the processing count i_CNT by 1 (i_CNT = i_CNT + 1), and increments the pixel number L_TBL (I, 2) of the pixel of the current label number i_LBL by 1 (L_TBL (I, 2). ) = L_TBL (I, 2) +1).

  In step 784, the processor 112 determines that the coordinate x (ie, any of xm, xs, xp) of the adjacent pixel Px (x, y) given the label number i_LBL has the label number i_LBL at the current left minimum x. It is determined whether or not the coordinate L_TBL (i_LBL-1, 3) is smaller. The adjacent pixel Px (x, y) is assigned with the label number i_LBL in the previous steps 710, 724, 734, 740744, 750, 754, 760 of FIGS. If it is determined that the coordinate x is not smaller than the minimum x coordinate L_TBL (i_LBL-1, 3), the procedure proceeds to step 788. If it is determined that the coordinate x is smaller than the minimum x coordinate L_TBL (i_LBL-1,3), in step 786, the processor 112 changes the minimum x coordinate L_TBL (i_LBL-1,3) to the coordinate x of the current adjacent pixel. (L_TBL (i_LBL-1, 3) = x). The procedure then proceeds to step 788.

  In step 788, the processor 112 determines that the coordinate y (ie, any of ym, ys, yp) of the adjacent pixel Px (x, y) given the label number i_LBL is the current upper minimum y having the label number i_LBL. It is determined whether or not the coordinate L_TBL (i_LBL-1, 4) is smaller. If it is determined that the coordinate y is not smaller than the minimum y coordinate L_TBL (i_LBL-1, 4), the procedure proceeds to step 792. If it is determined that the coordinate y is smaller than the minimum y coordinate L_TBL (i_LBL-1, 4), in step 790, the processor 112 sets the minimum y coordinate L_TBL (i_LBL-1, 4) as the coordinate y (L_TBL). (I_LBL-1, 4) = y). Thereafter, the procedure proceeds to Step 792.

  In step 792, the processor 112 determines that the coordinate x (ie, xm, xs, or xp) of the adjacent pixel Px (x, y) given the label number i_LBL has the current right maximum x-coordinate L_TBL () with the label number i_LBL. It is determined whether i_LBL-1, 5) is greater. If it is determined that the coordinate x is not greater than the maximum x coordinate L_TBL (i_LBL-1, 5), the procedure proceeds to step 796. If it is determined that the coordinate x is larger than the maximum x coordinate L_TBL (i_LBL-1, 5), in step 794, the processor 112 sets the maximum x coordinate L_TBL (i_LBL-1, 5) as the coordinate x (L_TBL). (I_LBL-1,5) = x). Thereafter, the procedure proceeds to Step 796.

  In step 796, the processor 112 determines that the coordinate y (ie, ym, ys or yp) of the adjacent pixel Px (x, y) given the label number i_LBL is the current lower maximum y coordinate L_TBL having the label number i_LBL. It is determined whether it is larger than (i_LBL-1, 6). If it is determined that the coordinate y is not greater than the maximum y coordinate L_TBL (i_LBL-1, 6), the procedure exits the routine of FIG. If it is determined that the coordinate y is greater than the maximum y coordinate L_TBL (i_LBL-1, 6), in step 798, the processor 112 sets the maximum y coordinate L_TBL (i_LBL-1, 6) as the coordinate y (L_TBL). (I_LBL-1, 6) = y). Thereafter, the procedure exits the routine of FIG.

  Thus, in the pixel group having a certain label number i_LBL, the coordinate x of the leftmost pixel, the coordinate x of the rightmost pixel, the coordinate y of the uppermost pixel, and the lowermost pixel The coordinate y is determined.

  FIG. 18 shows an example of a flowchart for the process of determining a pixel group representing a ring in step 508 of FIG.

  Referring to FIG. 18, processor 112 starts its processing at step 800. In step 802, the processor 112 sets the label order number I = i_LBL-10 of the pixel group to an initial value I = 1. In step 804, the processor 112 sets the label number i_LBL as label number i_TBL (I, 1) = i_TBL (i_LBL-10,1) in the label table (FIG. 15). Further, the processor 112 sets the center x coordinate of the pixel group with the label number i_LBL as the center of the maximum and minimum x coordinates L_TBL (I, 3) and L_TBL (I, 5), and xc = (L_TBL (I, 3) Set as + L_TBL (I, 5)) / 2. Further, the processor 112 sets the center y coordinate of the pixel group with the label number i_LBL as the center of the maximum and minimum y coordinates L_TBL (I, 4) and L_TBL (I, 6), and yc = (L_TBL (I, 4) Set as + L_TBL (I, 6)) / 2.

  In step 806, the processor 112 determines whether or not the label number of the pixel at the center coordinate (xc, yc) of the pixel group with the label number i_LBL is the same as the label number i_LBL (Px (xc, yc) = i_LBL?). If the label number Px (xc, yc) of the pixel at the center coordinate is not the same as the label number i_LBL of the pixel group, the pixel group may have an annular shape and the procedure proceeds to step 808. If it is determined that the label number Px (xc, yc) of the pixel at the center coordinate is the same as the label number i_LBL of the pixel group, the pixel group is not circular and the procedure proceeds to step 816.

  In step 808, the processor 112 checks to see if the pixel group is circular until the contour pixel having the pixel group label number i_LBL in the upward direction of the image, starting from the center coordinate (xc, yc). Move the trace position. In step 810, the processor 112 starts from the contour line pixel, starts at the center coordinate (xc, yc), and rotates in the positive or negative direction (clockwise) or the circumferential direction, for example, the pixel with the label number i_LBL. Trace the pixel group along the outline of the group. When the trace position returns to the start point, the processor 112 determines that the pixel group is annular. On the other hand, when the position reaches the circumscribed rectangle of the pixel group or moves in the direction opposite to the trace direction, the processor 112 determines that the pixel group is not circular.

  In step 812, the processor 112 determines whether or not the pixel group with label order number I is annular. If it is determined that the pixel group is not annular, the procedure proceeds to step 816. If it is determined that the pixel group is annular, in step 814, the annular parameter L_TBL (I, 9) of the pixel group having the label number i_LBL is set to L_TBL (I, 9) = in the label table (FIG. 15). Set to “annular”.

  In step 816, the processor 112 increments the label order number I of the pixel group by 1 (I = I + 1). In step 818, the processor 112 determines whether or not the label sequence number I is greater than the label number It (maximum label number i_LBL-10). If it is determined that the label order number I is not greater than the label number It (maximum label number i_LBL-10), the procedure returns to step 804. If it is determined that the label sequence number I is greater than the label number It (maximum label number i_LBL-10), the procedure exits the routine of FIG.

  In this way, it is determined whether or not all pixel groups having the label number i_LBL are circular, and a pixel group that is circular is specified.

  FIG. 19 shows an example of a flowchart for the process of extracting the pixel group representing the display frame of the display area (404, 414, 424, 434) in step 510 of FIG.

  Referring to FIG. 19, processor 112 starts its processing at step 850. In step 852, the processor 112 sets the label order number I of the pixel group to an initial value I = 1. In step 854, the processor 112 sets the label number i_LBL as the label number L_TBL (I, 1) on the label table (FIG. 15) (i_LBL = L_TBL (I, 1)).

  In step 856, the processor 112 determines whether or not the circular parameter L_TBL (I, 9) of the label sequence number I is “circular” in the label table (FIG. 15). If it is determined that it is not “annular”, the procedure proceeds to step 866. If it is determined that it is “annular”, then at step 858, processor 112 searches the image for the presence of another pixel group inside the pixel group having that label number L_TBL (I, 1). In step 862, the processor 112 determines whether another pixel group exists inside the pixel group of the label number L_TBL (I, 1). If it is determined that there is no other pixel group inside the pixel group, the procedure proceeds to step 866. If it is determined that another pixel group exists inside the pixel group, the processor 112, in step 864, causes the label sequence number I, that is, the label number L_TBL, because the pixel group surrounds the other pixel group. The surrounding parameter L_TBL (I, 10) = “envelopment” (or “outline”) of the pixel group of (I, 1) is set.

  In step 866, the processor 112 increments the label sequence number I by 1 (I = I + 1). In step 868, the processor 112 determines whether or not the label sequence number I is greater than the label number It (maximum label number i_LBL-10). If it is determined that the label order number I is not greater than the label number It (maximum label number i_LBL-10), the procedure returns to step 854. If it is determined that the label sequence number I is greater than the label number It (maximum label number i_LBL-10), that is, if it is determined to be “circular” for all label numbers i_LBL, then at step 870 the processor 112 A pixel group representing the inner “enclosure” is determined as a “display frame”. The processor 112 further obtains the horizontal width Wd and the vertical height Hd of the pixel group determined as the “display frame”. Thereafter, the procedure exits the routine of FIG.

  In this manner, the pixel group of the innermost annular display frame (404, 414, 424, 434) including the pixel group representing the number is extracted from the image of the display unit or display region of the measuring instrument. .

  FIG. 20 shows an example of a flowchart for the process of extracting a pixel group inside the pixel group representing the “display frame” in step 512 of FIG.

  Referring to FIG. 20, processor 112 begins its processing at step 900. In step 902, the processor 112 sets the label order number I of the pixel group to an initial value I = 1. In step 904, the processor 112 sets the label number i_LBL as label number i_LBL = L_TBL (I, 1) in the label table (FIG. 15).

  In step 906, the processor 112 determines whether or not the pixel group having the label number L_TBL (I, 1) is inside the display frame (its pixel group) in the image on the display unit of the measuring instrument. If it is determined that the pixel group is not inside the display frame, the procedure proceeds to step 910. If it is determined that the pixel group is inside the display frame, in step 908, the processor 112 sets the target parameter L_TBL (I, 11) of the label number as L_TBL (I, 11) = “target”. . Thereby, a pixel group that may represent a number to be recognized or extracted within the display frame is extracted.

  In step 910, the processor 112 increments the label order number I of the pixel group by 1 (I = I + 1). In step 912, the processor 112 determines whether or not the label sequence number I is greater than the label number It (maximum label number i_LBL-10). If it is determined that the label order number I is not greater than the label number It (maximum label number i_LBL-10), the procedure returns to step 904. If it is determined that the label sequence number I is greater than the label number It (maximum i_LBL-10), the procedure exits the routine of FIG. Thereby, it is determined whether all the pixel groups are pixel groups that may represent numbers to be recognized or extracted.

  21A and 21B show an example of a flowchart for the process of combining adjacent pixel groups in step 514 of FIG.

  Referring to FIG. 21A, processor 112 begins its processing at step 950. In step 952, the processor 112 sets the label order number I of the pixel group to an initial value I = 1. In step 954, the processor 112 determines whether or not the target parameter L_TBL (I, 11) of the pixel group of the label number i_LBL of the label sequence number I is “target” in the label table (FIG. 15). If it is determined that the pixel group is not “target”, the procedure proceeds to step 974. If it is determined that the pixel group is “target”, in step 958, the processor 112 sets another label order number J = I + 1.

  In step 958, the processor 112 determines whether or not the target parameter L_TBL (J, 11) of the pixel group of the label number i_LBL of the label order number I is “target”. If it is determined that the pixel group is not “target”, the procedure proceeds to step 970. If it is determined that the pixel group is “target”, in step 960, the processor 112 determines the target parameters L_TBL (I, 11) and L_TBL (J, J) of the label numbers i_LBL of the two label order numbers I and J. It is determined whether or not a circumscribed rectangle (for example, a broken line in FIG. 7D) of the pixel group 11) partially overlaps.

  FIG. 22A shows an example in which circumscribed rectangles of two pixel groups partially overlap each other.

  If it is determined that they overlap, the procedure proceeds to step 966. If it is determined that they do not overlap, in step 962, the processor 112 divides each of the two pixel groups into blocks of a plurality of regions, and selects the combination of the blocks closest to each other in each of the blocks of the two groups. select.

  FIG. 22B shows an example in which each of two adjacent pixel groups is divided into blocks of three regions.

  In step 964, the processor 112 determines whether the distance between the two pixel blocks of the combination is less than a threshold. If it is determined that the distance is smaller than the threshold, the procedure proceeds to step 970. If it is determined that the distance is not less than the threshold, the procedure proceeds to step 966. By dividing each of the two pixel groups into blocks of a plurality of regions and comparing the respective distances, each distance between all the pixels of one pixel group and all the pixels of the other pixel group is a threshold value. And the processing load is reduced, and the processing can be speeded up.

  At step 966, processor 112 combines the two pixel groups of the combination as pixel groups in the same combined relationship that may represent a single number. For this purpose, the label combination parameter i_LBL (I, 12) of the label number i_LBL of two pixel groups is assigned the label number of one pixel group or the smallest label number i_LBL of another pixel group having the same connection relationship. Thereafter, the procedure proceeds to Step 970 in FIG. 21B.

  In step 970 of FIG. 21B, the processor 112 increments the label order number J of the other pixel group by 1 (J = J + 1). In step 972, the processor 112 determines whether or not the label sequence number J is larger than the label number It (maximum label number i_LBL-10). If it is determined that the label sequence number J is not greater than the label number It (maximum label number i_LBL-10), the procedure returns to step 958. If it is determined that the label sequence number J is larger than the label number It (maximum label number i_LBL-10), in step 974, the processor 112 increments the label sequence number I of one pixel group by 1 (I = I + 1). In step 976, the processor 112 determines whether or not the label sequence number I is larger than the label number It (maximum label number i_LBL-10). If it is determined that the label order number I is not greater than the label number It (maximum label number i_LBL-10), the procedure returns to step 954. If it is determined that the label sequence number I is greater than the label number It (maximum label number i_LBL-10), the procedure exits the routine of FIGS. 22A and 22B.

FIG. 23 shows an example of a flowchart for the process of correcting the tilt of the image in step 516 of FIG.
24A and 24B show examples of correction of image tilt.

  Referring to FIG. 23, processor 112 starts its processing at step 1100. In step 1102, the processor 112 calculates an image inclination θ (tan θ or cot θ) from the distance from the right side (FIG. 24) or the left side of the circumscribed rectangle of the pixel group to the pixel of the pixel group closest in the horizontal direction.

In step 1104, the processor 112 determines whether or not there is an inclination θ (= 0 °). If it is determined that there is no inclination (θ = 0 °), the procedure exits the routine of FIG. 23A. If it is determined that there is an inclination (θ ≠ 0 °), in step 1106, the processor 112 corrects the inclination θ by moving the pixels of each pixel group in the image in the horizontal direction by the inclination (−θ). To do. Thereafter, the procedure exits the routine of FIG.
Accordingly, as shown in FIGS. 24A and 24B, the image of FIG. 24A having the inclination θ is corrected to the image having no inclination of FIG. 24B.

  FIG. 25 shows an example of a flowchart for the process of extracting the recognized number sequence or numerical information in step 518 of FIG.

  Referring to FIG. 25, processor 112 starts its processing at step 1150. In step 1152, processor 112 determines the length and width of the bounding rectangle (dashed lines in FIGS. 7C and 8C) of the combined pixel group set that may represent a number in the label table (FIG. 15). A ratio is determined and it is determined whether the vertical to horizontal ratio is greater than a threshold (eg, 7: 1 or 7/1). If it is determined that the ratio of length to width is greater than the threshold value, in step 1154, the processor 112 determines that the set of pixel groups is the number “1” in the label table. Thereafter, the procedure exits the routine of FIG.

  If it is determined in step 1152 that the aspect ratio is not greater than the threshold value, in step 1156, the processor 112 determines the combined pixel group set based on its circumscribed rectangle (dashed lines in FIGS. 7C and 8C). The pixel block is divided into a plurality of pixel blocks (for example, 7 × 4 blocks) in the vertical and horizontal directions, and the number of pixels having a label number in each pixel block is obtained.

  In step 1162, the processor 112 calculates the ratio (%) of the number of pixels (dark) in the pixel block at each position in the image or the occupancy ratio (%) at each position of the pattern representing the numbers “0” and “2” to “9”. Compare with the percentage or occupancy (%) of the value 1 (dark) in the block. The ratio or occupancy (%) of the value 1 (dark) in the block at each position of the pattern representing the numbers “0”, “2” to “9” is stored in the memory 114 or the hard disk drive 130 in advance. The processor 112 determines whether or not the pattern of the pixel block of the image matches the pattern of each numerical block stored in advance within the allowable error range. If it is determined that the pixel block pattern of the image matches the pattern of any of the numbers “0” and “2” to “9” within the allowable error range, the processor 112 determines in step 1166 that The group of pixel groups is determined as one of the numbers “0” and “2” to “9”. If it is determined in step 1162 that the pattern of the pixel block of the image does not match the pattern of each numerical block, in step 1164, the processor 112 determines that the recognition result is not a number representing the measurement value. The processor 112 stores the recognized number of results for the set of pixel groups in the label table (FIG. 15). For example, a pixel group representing a decimal point “.”, Units “° C.”, “mmHg”, characters and symbols other than numbers, small numbers, etc. may be determined not to be a number representing a measurement value. Thereafter, the procedure exits the routine of FIG.

  In this way, individual numbers representing the measured values are extracted from the image on the display unit of the measuring instrument in a processing form for simple number recognition.

  The flowchart of FIG. 25 is executed for a set of pixel groups (each number) to which the same label combination number is assigned, whereby characters representing each number are recognized and extracted.

  FIG. 26 shows an example of a flowchart for the process of extracting features or parameters specific to the display unit of the used measuring instrument in step 520 of FIG.

  Referring to FIG. 26, processor 112 starts its processing at step 1200. In step 1202, the processor 112 determines the maximum vertical and horizontal lengths of the circumscribed rectangles (broken lines in FIGS. 7C and 8C) of the pixel group set (each number) in the connected relationship recognized as representing each number. And the average value is obtained as the horizontal width Wc and the vertical height Hc. The processor 112 stores the numeric width Wc and the vertical height Hc in the memory 114 or the hard disk drive 130.

  In step 1204, the processor 112 obtains the thickness Tc of the number (character) of the number pixel group. For this purpose, the processor 112 scans each pixel group in the x and / or y direction, obtains the longest distance in which the number of consecutive pixels of the black image is equal to or less than the threshold value, and the thickness of the number (character). You may determine an average value as thickness Tc of a number. The threshold value may be, for example, a value Wc / 3 obtained by multiplying a horizontal width Wc of a number by 1/3. The processor 112 stores the thickness Tc of the number in the memory 114 or the hard disk drive 130.

  In step 1206, the processor 112 obtains the character interval Pc and the character line interval Pl from the pixel group of the set recognized as a number (ie, a number). For this purpose, the processor 112 obtains the interval (pitch) in the x direction between the circumscribed rectangles (for example, the right side) of a set of pixel groups of adjacent numbers as the character interval, and calculates the average value as the character interval between adjacent numbers. It may be determined as Pc. When the interval between two adjacent numbers is more than a threshold (for example, 2 times, 2Pc) more than the average of the intervals between other adjacent numbers, the processor 112 determines that the numbers are different from each other. It may be determined so that it is not used for obtaining the average value Pc of the character spacing. In addition, the processor 112 obtains the average value of the center coordinates y of the heights of adjacent numbers as the row position in the y direction, and calculates the average value of the distances between the two adjacent row positions in the vertical direction of the two rows. You may determine as the space | interval P1 between. The processor 112 further uses the average character spacing Pc of adjacent numbers as a reference (= 1.0) to set other parameters such as the horizontal and vertical lengths of the numerical characters, the line spacing, and the thickness of the numbers. , Etc. may be normalized. The processor 112 stores the character spacing Pc and the line spacing Pl in the memory 114 or the hard disk drive 130.

  In step 1208, the processor 112 divides a plurality of numbers into items and groups them from the character spacing Pc of the numbers and the line spacing Pl in the vertical direction of the numbers. When the character spacing Pc of the numbers in one line is less than or equal to a threshold (for example, double character spacing 2Pc), the processor 112 groups the numbers into one measurement item. On the other hand, when the character interval Pc between two adjacent numbers in one line is larger than the threshold value, the processor 112 separates and groups the two numbers into different measurement items identified by the interval. The processor 112 further groups the numbers in different rows into different measurement items for each row. In this way, the number of digits representing the numerical value of each measurement item and the numerical string (numerical value information) having the numerical value are determined. In this case, characters, symbols, decimal points, etc. other than numbers may be ignored and excluded from processing. The processor 112 stores a string of numerical values (numerical information) for each determined item in the memory 114 or the hard disk drive 130.

  In step 1210, the processor 112 determines the vertical and horizontal lengths (Hd, Wd) of the display area or display frame (404, 414, 424, 434) based on the average character spacing Pc (= 1.0). Is normalized (Hd / Pc, Wd / Pc), and the dimension and shape (Hd, Wd) of the normalized display area are determined. As an alternative, the vertical and horizontal dimension shapes (Hd, Wd) of the display area or display frame may be normalized based on the average character height Hc instead of the character spacing Pc. The processor 112 stores the determined size (Hd, Wd) of the display area in the memory 114 or the hard disk drive 130.

  FIG. 27 shows an example of a flowchart for the process of determining the measurement item corresponding to the extracted feature of the display unit and the measurement value represented by the extracted number string in step 522 of FIG. Yes.

  Referring to FIG. 27, processor 112 starts its processing at step 1250. In step 1252, the processor 112 refers to the list of instruments (FIG. 5) and selects a display feature in the list that is similar to the extracted display feature. For this purpose, the processor 112 sets the display area (display frame) size and shape (Hd, Wd), character size (height Hc), character spacing (Pc), and character thickness as characteristics of the image display unit. Compared with the order number (No. 1, 4, 7) of each measuring instrument used in the measuring instrument list (FIG. 5). The processor 112 selects a display feature or a corresponding instrument sequence number in the list that is similar to or close to the dimensional feature of the extracted display.

  At step 1254, the processor 112 determines or selects a meter type from the list that corresponds to the display feature or sequence number of the meter selected in the meter list. The processor 112 stores the determined instrument type in the memory 114 or the hard disk drive 130.

  In step 1256, the processor 112 determines a measurement item corresponding to the selected measurement device type in the measurement device list. When only one measurement item corresponding to the selected measuring instrument type exists, the measuring item is uniquely determined by selecting the measuring instrument type. If there are two or more measurement items corresponding to the measuring device type, two or more possible measurement items are temporarily determined.

  In step 1258, the processor 112 refers to the list of measurement items (FIG. 6) and determines the measurement value represented by the extracted number string (numerical information) based on the determined measurement item. For this purpose, the processor 112 determines the measurement value (data) represented by the numeric string based on the data format of the determined measurement item. For example, when the numeric string (numerical information) is “370” and the measurement item is body temperature, both are three-digit numbers, and there is a decimal point between the first and second digits. The measured value is determined as “37.0” or “37.0 ° C.”.

  When there are two or more measurement items corresponding to the measuring instrument type, the following processing may be performed. The processor 112 determines the number of digits of the numeric string (numerical information), the size of the numeric value, and the appearance order of the items in the image, the data format (number of digits) of each measurement item in the measurement item list (FIG. 6), the appearance order, And optionally refer to or compare numerical tolerances (not shown). Thereby, the processor 112 determines the measurement item represented by the numeric string and its measurement value (data). In the comparison, the decimal point may be ignored. Further, the processor 112 refers to the list of measurement items (FIG. 6) based on the determined measurement item, and determines a numerical value and a unit including a decimal point represented by a numeric string as a measurement value. In addition, the processor 112 compares two or more numeric strings extracted from the image with each other, and the data format (number of digits) of each measurement item in the measurement item list, the appearance order, and optionally a numerical tolerance Or may be compared with these. Thereby, the processor 112 may determine the measurement item represented by each numeric string (numerical information) and the measurement value (data).

  For example, the processor 112 compares two sets of numeric sequences (numerical information) related to blood pressure extracted from the image, determines that a sequence of numeric values (numerical information) with a large value represents the systolic blood pressure, A string of numbers (numerical information) may be determined to represent the minimum blood pressure. For example, the processor 112 compares three sets of numbers (numerical information) related to blood pressure and pulse rate extracted from an image with a list of measurement devices (FIG. 5) and a list of measurement items (FIG. 6). It may be determined that the highest blood pressure, the lowest blood pressure, and the pulse are expressed in the order in the image and in the order of appearance in the list.

  For example, for the numeric string “65” extracted from the image, the measurement items “maximum blood pressure” and “minimum blood pressure” are selected, the appearance order in the image is second, and smaller than another numeric string “120”. To do. On the other hand, in the list of measurement items, “minimum blood pressure” is a three-digit number, the allowable range is “40 to 80”, and it appears after the measurement item “maximum blood pressure”. In this case, for the numeric string “65”, “diastolic blood pressure” in the list of measurement items is selected as the measurement item, and “65” or “65 mmHg” is determined as the measurement value.

  After step 11258, the procedure may proceed to step 1260 or proceed to step 1262.

  In step 1260, the processor 112 further compares the numerical value of the current measured value (data) with a reference numerical value as auxiliary data to determine whether the numerical value of the current measured value is within the allowable range of the reference numerical value. Thus, it may be determined whether the determined measuring instrument type and the measurement item are appropriate. The reference numerical value as auxiliary data may be a numerical value of a general measurement item (physical data type) such as age and gender close to the user's personal data, or a numerical value set by the user's individual Alternatively, it may be a past measurement value within the previous time or several times before the individual user. The allowable range of numerical values for measurement items is, for example, within ± 15% of general reference values or within ± specific values, but within ± 10% of user's past measured values or within ± specific values. There may be. The user's past measurement data is stored in the memory 114 or the hard disk drive 130 and the server device 30. For example, when the measurement item is systolic blood pressure, the permissible range of the numerical value may be the maximum blood pressure of the user 120 ± 12 (ratio 10%). For example, when the measurement item is body temperature, the allowable range of the numerical value may be ± 2.5 (specific numerical value) of the user's individual body temperature 36.8.

  When it is determined that the numerical value of the measurement value is not within the allowable range, the processor 112 may determine that the determined measurement device type and item name are not valid. In that case, the procedure returns to step 1256 or 1254. If it is determined that the numerical value is not within the allowable range, the procedure returns to step 1256, and the processor 112 determines another measurement that may correspond to the display characteristic selected from the plurality of measurement items. An item may be selected. Nevertheless, if it is determined again in step 1260 that the measured value is not within the allowable range of the reference value, the procedure may return to step 1254. In this case, the processor 112 determines another display feature similar to the extracted display feature from the display features in the list of instruments in step 1254 and corresponds to it in step 1256. The measurement value to be measured may be determined to determine the measurement value. Such comparison of the characteristics of the display unit described above may be repeated to select another measurement device type and measurement item that are more appropriate for the measurement device type and measurement item. In step 1260, processor 112 determines a more reasonable measurement based on the difference between the measured value of each measurement item and each reference value determined by performing step 1256 multiple times for a single digit string. Items and measurements may be determined.

  In this way, a measurement item represented by a string of numbers from a string of numbers (numerical information) extracted from an image without a user input operation and without using high-performance and high-precision character recognition technology And the measured value is determined.

  In step 1262, the processor 112 acquires the shooting date / time included in the image data and determines it as the measurement date / time, and associates the measured value with the information of the measurement date / time in association with the determined measuring instrument type and measurement item in the memory 114. Alternatively, it is stored in the hard disk drive 130.

  In step 1264, the processor 112 displays the determined measuring instrument type, measurement date / time, measurement item (body data type) and measurement value on the display device 122 together with the measurement date / time information and the captured image (FIGS. 28 and 29). ), Let the user check the measurement items and measurement values.

  FIG. 28A shows an example of a measurement item (body temperature) and measurement value display 1310 on the display unit 122. FIG. 28B shows another example of the measurement item (weight) on the display unit 122 and the measurement value display 1320.

  If the displayed content is correct, the user operates the confirmation key in the input unit 126 to determine the measurement item and the measurement value as measurement data. If there is an error in the measurement item and the measurement value, the user can correct the measurement item and measurement value by operating the input unit 126, and confirm and determine the corrected measurement item and measurement value. The processor 112 determines or corrects the measuring instrument name, the measurement date and time, and the measurement data according to the operation by the user input unit 126. The processor 112 stores the determined measurement data in the memory 114 or the hard disk drive 130 as current measurement data.

  A user may use different types of measuring instruments to measure the same physical data. Even in such a case, the processor 112 only needs to display the common measurement item and the measurement value on the display device 122, and does not need to distinguish and display each measurement instrument name or measurement instrument type. For example, the processor 112 may display both the measurement value of the thermometer A and the measurement value of the thermometer B as the measurement item “body temperature”, and does not need to display the thermometer A and the thermometer B separately.

  In step 524 of FIG. 11, the processor 112 follows the operation of the input unit 126 of the user, along with the user's personal information (identification information, name, gender, age, etc.), the name of the measuring instrument, the measurement date and time, the measurement item name, The measurement value (measurement data) is transmitted to the server device 30.

  The processor 312 of the server device 30 manages the received user measurement device type, measurement date / time, measurement item, and measurement value (measurement data) as a database (DB) of the storage device 16 as history data or recorded data. The database stores and manages the user's personal information, measurement date and time, measurement items (physical data types), and measurement values. The measuring instrument type used for measurement may or may not be accumulated.

  Thereafter, the server device 30 or the processor 312 transmits the history data or the recording data stored in the storage device 16 in response to a request from the information processing terminal 100 or the portable terminal 102. Information related to the analysis result and / or advice of the measurement data may be added to the historical data. The (processor 112) of the information processing terminal 100 or the portable terminal 102 receives history data or recorded data.

FIG. 29 shows an example of a display screen of history data of measurement data displayed on the display unit 122 of the information processing terminal 100 or the portable terminal 102.
The processor 112 of the information processing terminal 100 or the portable terminal 102 displays the received history data file on the display unit 122 as shown in FIG. As shown in FIG. 29, the measurement values are displayed in the order of the measurement date and time in association with the measurement items on the display unit 122. Further, the additional information as described above may be displayed on the display unit 122.

  At least a part of the flowchart of FIG. 11, for example, all or part of steps 502 to 522 may be executed by the processor 312 of the server device 30. In this case, the storage device 16 is used instead of the memory 114.

  In this case, in step 500, the processor 312 or its processing unit 1144 includes an image of the display unit of the measuring instrument taken by the camera 202 or the imaging unit 203 from the information processing terminal 100 or the portable terminal 102 according to the user's operation. Is received and stored in the database (DB) of the storage device 16.

  Similarly, step 502 is executed by the processor 312 or its noise removal unit 1122. Step 504 is executed by the processor 312 or its binarization unit 1124. Step 506 is executed by the processor 312 or its grouping unit 1126. Step 508 is executed by the processor 312 or its circular determination unit 1128. Step 510 is executed by the processor 312 or the annular determination unit 1128. Step 512 is executed by the processor 312 or its group extraction unit 1130. Step 514 is executed by the processor 312 or its group combiner 1132. Step 518 is executed by the processor 312 or its number extraction unit 1138. Step 520 is executed by the processor 312 or the feature extraction unit 1138.

  Step 522 is executed by the processor 312 or its measurement data determination unit 1140. In step 1264 of FIG. 522 (FIG. 27), the processor 312 or its measurement data determination unit 1140 transmits the measurement item and the measurement value to the information processing terminal 100 or the portable terminal 102 and causes the display unit 122 to display them. Let them check it and correct it if it is wrong. The user determines measurement items and measurement values confirmed on the information processing terminal 100 or the portable terminal 102 as measurement data, stores them in the memory 114 or the hard disk 130, and transmits the determined measurement data to the server device 30. To do.

  In step 524, the processor 312 or the processing unit 1144 sends the received measurement item (physical data type) and measurement value as the determined measurement data together with the user's personal information and measurement date and time to the measurement data in the storage device 16. It is stored in a database (DB) and managed as history data or recorded data.

  According to the above-described embodiment, the measurement data of more types of measuring instruments having no communication function are captured and transmitted to the server device 30, and the various types of measurement data over a long period are managed by the server device 30 as history data. Can be made. Thereby, based on the measurement data of more measurement items related to the user, the server device 30 can perform a more appropriate analysis on the user's health and present more appropriate analysis results and advice to the user as additional information. .

  All examples and conditional expressions given here are intended to help the reader understand the inventions and concepts that have contributed to the promotion of technology, such examples and It is understood that the present invention is not limited to the conditions, and that the organization of such examples in the specification is not related to the superiority or inferiority of the present invention. Although embodiments of the present invention have been described in detail, it will be understood that various changes, substitutions and variations can be made thereto without departing from the spirit and scope of the invention.

Regarding the embodiment including the above examples, the following additional notes are further disclosed.
(Appendix 1) Information processing device
Captured image data is stored in a storage device, numerical information is extracted from the image data, geometric features are extracted from the image data, and a plurality of display units of each of the plurality of physical data measuring instruments stored in the storage device A shape feature similar to the extracted shape feature is selected from among the shape features, and a type of body data measuring device corresponding to the selected shape feature is determined, and the determined type is A method of processing image data, comprising: determining measurement data represented by the numerical information based on the result, and storing the measurement data in the storage device in association with the determined type.
(Supplementary Note 2) The information processing apparatus selects and selects a geometric feature similar to the geometric feature extracted from the image data from the plurality of geometric features stored in the storage device. Body data type corresponding to the determined shape feature is determined, the measurement data represented by the numerical information is determined based on the determined body data type, and the measurement data is associated with the determined body data type The method according to appendix 1, wherein a process of storing data in the storage device is executed.
(Additional remark 3) When storing the said measurement data, the said measurement data are further stored with date information, The processing method of the image data of Additional remark 1 or 2 characterized by the above-mentioned.
(Additional remark 4) The said information processing apparatus further performs the process which transmits this measurement data matched with this classification stored in the said memory | storage device to another apparatus, Additional remark 1 thru | or 3 characterized by the above-mentioned The method according to any one.
(Additional remark 5) In response to the request | requirement of the historical information regarding the some measurement data in which the said information processing apparatus contains the said measurement data, the said measurement data matched with the said body data classification transmitted to another apparatus The method according to any one of appendices 2 to 4, further comprising: executing a process of receiving the plurality of measurement data on the display device in order of date and time information.
(Supplementary Note 6) When determining the measurement data represented by the numerical information based on the determined physical data type, whether the numerical information is within an allowable range as a value represented by the physical data type Determine
If it is determined that the numerical information is not within the allowable range, another physical data type that may correspond to the selected geometric feature is selected as the physical data type, or stored in the storage device Determining another geometric feature similar to the extracted geometric feature from the plurality of stored geometric features, and determining the measurement data based on the other geometric feature The method according to Supplementary Note 2 or 3.
(Supplementary note 7) The captured image data is stored in a storage device.
Numeric information is extracted from the image data.
Extract shape features from the image data.
A shape feature similar to the extracted shape feature is selected from the plurality of shape features stored in the storage device and related to the display unit of each of the plurality of physical data measuring instruments, and the selected shape feature is selected. The type of the physical data measuring device corresponding to the feature is determined, the measurement data represented by the numerical information is determined based on the determined type, and the measurement data is stored in association with the determined type Store in device
A program for causing an information processing apparatus to execute processing.
(Supplementary Note 8) A shape feature similar to the shape feature extracted from the image data is selected from the plurality of shape features stored in the storage device, and corresponds to the selected shape feature. Determining the physical data type to be determined, determining the measurement data represented by the numerical information based on the determined physical data type, and associating the measured data with the determined physical data type in the storage device The program according to appendix 7 for causing the information processing apparatus to execute a storing process.
(Additional remark 9) Furthermore, in response to the request | requirement of the historical information regarding the some measurement data containing the said measurement data, the said measurement data matched with the said body data classification transmitted to another apparatus are sent from said another apparatus. The program according to appendix 8, for causing the information processing apparatus to execute a process of receiving and displaying the plurality of measurement data on a display device in order of date and time information.
(Supplementary Note 10) Stores and captures a plurality of types of physical data measuring instruments and a plurality of geometric features related to the display units of the plurality of physical data measuring instruments respectively associated with the types of the plurality of measuring instruments A storage unit for storing the processed image data;
A number extraction unit for extracting numerical information from the image data;
A feature extraction unit for extracting shape features from the image data;
A shape feature similar to the extracted shape feature is selected from the plurality of shape features, a type of body data measuring device corresponding to the selected shape feature is determined, and the determined A processing unit that determines measurement data represented by the numerical information based on a type, and stores the measurement data in the storage device in association with the determined type;
The terminal device characterized by including.
(Supplementary Note 11) The storage unit stores a plurality of body data types respectively associated with the plurality of geometric features,
The processing unit selects a shape feature similar to the shape feature extracted from the image data from the plurality of shape features, and determines a body data type corresponding to the selected shape feature. And determining the measurement data represented by the numerical information based on the determined body data type, and storing the measurement data in the storage device in association with the determined body data type. The terminal device according to appendix 10.

5 Network 30 Server Device 312 Database 312 Processor 100 Information Processing Terminal 102 Portable Terminal 112 Processor 114 Memory 130 Hard Disk Drive 202 Digital Camera 203 Imaging Unit

Claims (7)

Information processing device
The captured image data is stored in a storage device,
Extracting numerical information from the image data;
Extracting shape features from the image data;
A shape feature similar to the extracted shape feature is selected from the plurality of shape features stored in the storage device and related to the display unit of each of the plurality of physical data measuring instruments, and the selected shape feature is selected. The type of the physical data measuring device corresponding to the feature is determined, the measurement data represented by the numerical information is determined based on the determined type, and the measurement data is stored in association with the determined type A method of processing image data, characterized by executing processing stored in an apparatus.   The information processing apparatus selects a geometric feature similar to the geometric feature extracted from the image data from the plurality of geometric features stored in the storage device, and the selected geometric feature Body data type corresponding to the physical data type is determined, the measurement data represented by the numerical information is determined based on the determined body data type, and the measurement data is stored in association with the determined body data type The method according to claim 1, wherein a process of storing in an apparatus is executed.   In response to a request for history information regarding a plurality of measurement data including the measurement data, the information processing apparatus further transmits the measurement data associated with the physical data type transmitted to another device to the other measurement data. The method according to claim 2, wherein a process of receiving the plurality of measurement data on the display device in order of date and time information is received from the device. The captured image data is stored in a storage device,
Extracting numerical information from the image data;
Extracting shape features from the image data;
A shape feature similar to the extracted shape feature is selected from the plurality of shape features stored in the storage device and related to the display unit of each of the plurality of physical data measuring instruments, and the selected shape feature is selected. The type of the physical data measuring device corresponding to the feature is determined, the measurement data represented by the numerical information is determined based on the determined type, and the measurement data is stored in association with the determined type A program for causing an information processing apparatus to execute processing stored in the apparatus.   A body data type corresponding to the selected shape feature is selected from among the plurality of shape features stored in the storage device, and a shape feature similar to the shape feature extracted from the image data is selected. Determining the measurement data represented by the numerical information based on the determined body data type, and storing the measurement data in the storage device in association with the determined body data type The program according to claim 5 for causing said information processing apparatus to execute. A plurality of physical data measuring device types and a plurality of geometric features related to the display units of the plurality of physical data measuring devices respectively associated with the plurality of measuring device types, and captured image data A storage unit for storing
A number extraction unit for extracting numerical information from the image data;
A feature extraction unit for extracting shape features from the image data;
A shape feature similar to the extracted shape feature is selected from the plurality of shape features, a type of body data measuring device corresponding to the selected shape feature is determined, and the determined A processing unit that determines measurement data represented by the numerical information based on a type, and stores the measurement data in the storage device in association with the determined type;
The terminal device characterized by including. The storage unit stores a plurality of body data types respectively associated with the plurality of geometric features,
The processing unit selects a shape feature similar to the shape feature extracted from the image data from the plurality of shape features, and determines a body data type corresponding to the selected shape feature. And determining the measurement data represented by the numerical information based on the determined body data type, and storing the measurement data in the storage device in association with the determined body data type. The terminal device according to claim 6.

Images