JP7659155B2 - Image Analysis System - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies October 14, 2020 Presented at IECON 2019 - 45th Annual Conference of the IEEE Industrial Electronics Society

The present invention relates to an image analysis system .

2. Description of the Related Art Conventionally, there are techniques for performing image analysis such as character recognition on paper media on which characters and tables are written.
An example of such a technique for image analysis is disclosed in Patent Literature 1. In the technique disclosed in Patent Literature 1, what information is written at what position is set in advance according to the format (i.e., appearance) of the document to be read. Then, based on this setting, image analysis such as detecting character strings from a position in the image that matches the format can be performed.

JP 2019-57311 A

However, the format of the document to be read is not necessarily the same, and various formats may be mixed for each document. In such cases, it is practically difficult to set the format in advance or modify the format each time it is read, as in the technology disclosed in Patent Document 1.

In addition, in the general technology disclosed in Patent Document 1 and the like, it is assumed that an original is placed on a mechanism such as a document tray of a multifunction printer or an ADF (Auto Document Feeder), and the original is read in an appropriate state with a predetermined distance between the scanner and the original, and an image is generated. However, the original is not necessarily read in an appropriate state and an image is generated. For example, when a user takes a picture of an original with a portable camera and generates an image, the original may be curved when photographed, or the positional relationship between the camera and the original when photographed may not be appropriate, resulting in distortion or the like in the image.
Thus, even when various formats are mixed or when distortion or the like occurs in the image, it is desirable to perform image analysis with high accuracy.

The present invention has been made in light of these circumstances. The objective of the present invention is to perform image analysis with greater accuracy.

In order to solve the above problem, an image analysis system according to an embodiment of the present invention comprises:
a table detection means for detecting an area corresponding to a table from an image including a table as a subject;
a frame detection means for detecting a plurality of frames constituting the table from an area corresponding to the table and for assigning position information in the image to each of the detected frames;
an information acquisition means for acquiring acquisition target information based on a character recognition result for each of the plurality of frames and position information assigned to each of the plurality of frames;
The present invention is characterized by comprising:

The present invention makes it possible to perform image analysis with greater accuracy.

1 is a block diagram showing an example of the overall configuration of an image analysis system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a configuration of a terminal according to an embodiment of the present invention. 1 is a block diagram showing an example of a configuration of an image analysis device according to an embodiment of the present invention. 11 is a flowchart illustrating a flow of a photographing process. 11 is a flowchart illustrating a flow of an image analysis process. 11 is a schematic diagram showing how a frame line in a first direction and a frame line in a second direction are separately detected; FIG. 1A and 1B are schematic diagrams showing a table structure included in a target image and a method for acquiring acquisition target information. 11 is a flowchart illustrating the flow of a character detection process. 11 is a schematic diagram showing the process of removing an outer frame area from a binarized image and then performing projective transformation. FIG.

An example of an embodiment of the present invention will now be described with reference to the accompanying drawings.

[System configuration]
Fig. 1 is a block diagram showing the overall configuration of an image analysis system S according to this embodiment. As shown in Fig. 1, the image analysis system S includes a terminal 1 and an image analysis device 2. Fig. 1 also shows a document 3 that is a target for image analysis processing in this embodiment. Here, it is assumed that the document 3 is a document that includes some kind of table.

The terminal 1 and the image analysis device 2 are connected to each other so that they can communicate with each other directly or via a network (not shown). In this case, the network is realized by, for example, the Internet, a LAN (Local Area Network), a mobile phone network, or a combination of these. Furthermore, this communication may be performed in accordance with any communication method, and the communication method is not particularly limited. Furthermore, the communication connection format may be a wired connection or a wireless connection.

The terminal 1 is a terminal equipped with a photographing function. The terminal 1 is realized, for example, by a smartphone, a tablet-type terminal, or a digital camera. The terminal 1 performs a photographing process. Here, the photographing process is a series of processes for photographing the original 3 to generate an image including the original 3 as a subject (hereinafter referred to as a "target image"). In the photographing process, the terminal 1 performs photographing based on the user's operation to generate a target image including the original 3 as a subject. Then, the terminal 1 transmits the generated target image to the image analysis device 2.

The image analysis device 2 is a device that performs image analysis processing. The image analysis device 2 is realized, for example, by a personal computer or a server device. The image analysis device 2 acquires the target image by receiving it sent from the terminal 1. The image analysis device 2 then performs image analysis processing on this target image. Here, the image analysis processing is a series of processes that perform image analysis such as character recognition to detect characters written in the target image.

In this image analysis process, the image analysis device 2 detects an area corresponding to a table (e.g., a table) from an image that contains the table as a subject. The image analysis device 2 also detects multiple frames (e.g., cells) that make up the table from the area corresponding to the table, and assigns positional information in the image to each of the detected multiple frames. Furthermore, the image analysis device 2 acquires the target information (e.g., pairs of test items and test values in a health check sheet) based on the character recognition results for each of the multiple frames and the positional information assigned to each of the multiple frames.

In this way, the image analysis system S including the terminal 1 and the image analysis device 2 detects tables and frames in stages, and then uses the position information to obtain the desired target information. Therefore, it is possible to perform image analysis with higher accuracy than image analysis methods that simply perform image analysis processing without detecting tables and frames or using position information. Therefore, even if, for example, various formats are mixed and the arrangement of tables and frames varies depending on the manuscript, or there is distortion in the image that would normally make image analysis difficult, it is possible to perform appropriate image analysis.

Next, we will provide a more detailed explanation of each device included in the image analysis system S to realize this type of image analysis processing.

In the following, as an example for the purpose of explanation, it is assumed that the document 3 is a "health checkup chart" for a user who uses the image analysis system S. The reason for this will be explained.
First, health checkups are extremely beneficial for early detection and treatment of illness. By undergoing regular health checkups, users can grasp changes in their own health conditions.

However, the format (i.e., appearance) of health checkup charts varies depending on the business entity that creates them or the institution that performs the examination. For example, the layout of tables and frames varies from one health checkup chart to another. In addition, the positions of test items and corresponding test values, as well as the positions of reference values and previous test values, vary from one health checkup chart to another. Therefore, it is difficult to perform image analysis of health checkup charts using general technology.
In addition, the medical examination chart is not shared as electronic data between each testing institution. Moreover, the user who is the patient only receives the medical examination chart printed on paper from each testing institution, and does not receive the electronic data in the first place. In this situation, it is difficult for the user to grasp the changes in his/her own health condition. Moreover, it is difficult for each testing institution, hospital, etc. to grasp the changes in the user's health condition.

Therefore, in this embodiment, the above-mentioned image analysis process is performed on a target image captured with such a medical checkup chart as a subject. As a result, in this embodiment, image analysis can be performed more accurately from the medical checkup chart. This also allows a user to convert their own medical checkup chart into electronic data by a simple operation such as taking a picture of the chart with the terminal 1. Furthermore, by using such electronic data, for example, it becomes easier for a user to understand changes in their own health condition over time. In addition, such electronic data can be used by each testing institution, hospital, etc.
In this way, a medical examination chart is a suitable target for the image analysis process in this embodiment, and therefore, the following description will be given assuming that the document 3 is a medical examination chart.

However, this is merely an example for the purpose of explanation and does not limit the use of this embodiment. In other words, this embodiment can be applied to various cases where the document 3 that is the subject of the target image is not only a medical examination chart, but also includes some kind of table.

[Device configuration]
The configuration of the terminal 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the terminal 1. As described above, the terminal 1 is realized by, for example, a smartphone or tablet terminal, or a digital camera.

As shown in FIG. 2, the terminal 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication unit 14, a drive 15, a storage unit 16, an input unit 17, an output unit 18, and an imaging unit 19. These units are connected by signal lines and send and receive signals between each other.

The CPU 11 executes various processes (for example, a process of acquiring a target image by photographing a document 3 as a subject) according to a program recorded in the ROM 12 or a program loaded from the storage unit 16 to the RAM 13.
The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes.

The communication unit 14 controls communication so that the CPU 11 can communicate with other devices (e.g., the image analysis device 2).

Removable media (not shown) such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached to the drive 15. Programs read from the removable media by the drive 15 are installed in the storage unit 16 as required. In addition, various data read from the removable media by the drive 15 is used for calculation processing by the CPU 11 as required.

The storage unit 16 is configured with a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various types of data.
The input unit 17 is composed of various buttons and a touch panel, or an external input device such as a mouse and a keyboard, and inputs various information in response to user instructions.
The output unit 18 is composed of a display, a speaker, etc., and outputs images and sounds.

The imaging unit 19 is composed of a lens, an imaging element, etc., and captures an image of a subject (e.g., a document 3). Through this capture, the imaging unit 19 generates an image (e.g., a target image).

In the terminal 1, these units cooperate with each other to execute various processes in this embodiment. When the various processes in this embodiment are executed, an imaging control unit 111 and a terminal side notifying unit 112 function in the CPU 11, as shown in FIG.
In addition, an image storage section 161 and an analysis result storage section 162 are set in one area of the storage section 16 .
Including cases not specifically mentioned below, data required to realize processing is transmitted and received between these functional blocks at appropriate times as appropriate.

The shooting control unit 111 controls shooting in the terminal 1. To this end, the shooting control unit 111 displays a user interface for the user to shoot with the document 3 as the subject, a live view image acquired by the imaging unit 19, and the like, on a display included in the output unit 18. Furthermore, the shooting control unit 111 generates and acquires an image by controlling the imaging unit 19 based on a shooting instruction operation from the user. As described above, the image to be acquired is assumed to be a target image that includes the document 3 (here, a medical examination chart) as the subject.

The imaging control unit 111 stores the acquired target image in the image storage unit 161. That is, the image storage unit 161 functions as a storage unit that stores the target image.
In addition, the imaging control unit 111 transmits the acquired target image to the image analysis device 2 .

The terminal-side notification unit 112 acquires the image analysis results by receiving the image analysis results in the image analysis process for the target image transmitted from the image analysis device 2. The terminal-side notification unit 112 also stores the acquired image analysis results in the analysis result storage unit 162. In other words, the analysis result storage unit 162 functions as a storage unit that stores the image analysis results.

The terminal-side notification unit 112 also notifies the user of the acquired image analysis results. The user can refer to the notified image analysis results to understand the contents of the image analysis results (e.g., the test values in the user's health checkup chart). This notification to the user can be realized, for example, by displaying the results on a display included in the output unit 18, or by printing the results on paper using a printer (not shown).

The specific content that is notified as a result of image analysis will be described later together with an explanation of the operation with reference to the flowchart in Figure 5.

[Configuration of image analysis device]
Next, the configuration of the image analyzing device 2 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the image analyzing device 2. As described above, the image analyzing device 2 is realized by, for example, a personal computer or a server device.

As shown in FIG. 3, the image analysis device 2 includes a CPU 21, a ROM 22, a RAM 23, a communication unit 24, a drive 25, a memory unit 26, an input unit 27, and an output unit 28. These units are connected by signal lines and send and receive signals between each other. The hardware functions of these units are the same as the units of the same name that are included in the terminal 1 described above with reference to FIG. 2. Therefore, redundant explanations of the hardware functions will be omitted.

In the image analysis device 2, these units work together to execute various processes in this embodiment. When the various processes in this embodiment are executed, an image acquisition unit 211, a table detection unit 212, a frame detection unit 213, a character recognition unit 214, an information acquisition unit 215, and a device-side notification unit 216 function in the CPU 21, as shown in FIG.
In addition, an image storage section 261 and an analysis result storage section 263 are set in one area of the storage section 26 .
Including cases not specifically mentioned below, data required to realize processing is transmitted and received between these functional blocks at appropriate times as appropriate.

The image acquisition unit 211 acquires the target image by receiving it, the target image being transmitted from the image capture control unit 111 included in the terminal 1. The image acquisition unit 211 also stores the acquired target image in the image storage unit 261. In other words, the image storage unit 261 functions as a storage unit that stores the target image.

The shooting control unit 111 of the terminal 1 may store the target image in removable media via the drive 15. The image acquisition unit 211 may then acquire the target image by reading the removable media on which the target image is stored via the drive 25.

The table detection unit 212 detects areas (hereinafter referred to as "table areas") corresponding to each of the multiple tables (or tables in some cases, there may be only one) contained in the target image stored in the image storage unit 261. For example, the frame detection unit 213 detects table areas corresponding to each of the multiple tables constituting the medical examination chart that is the subject of the target image.

The frame detection unit 213 detects multiple frames constituting a table from each table region detected by the table detection unit 212. For example, the frame detection unit 213 detects multiple frames constituting a medical examination table that is the subject of the target image. In this case, for example, the frame detection unit 213 detects the frame lines of the first direction and the frame lines of the second direction, which are the frame lines of the multiple frames constituting the table, from each table region, and detects multiple frames based on the intersections of the detected frame lines of the first direction and the frame lines of the second direction. In this way, by detecting the frame lines of the first direction and the frame lines of the second direction separately, the frame detection unit 213 can detect the frame lines and frames more accurately than when detecting without separating them.

Furthermore, the frame detection unit 213 divides and cuts out each of the detected frames, and assigns position information in the target image to each of the cut-out frames. For example, the frame detection unit 213 sets a combination of table identification information (e.g., an identifier assigned to the table) assigned to each detected table region and frame identification information (e.g., an identifier assigned to the frame) as position information, and assigns this position information to each of the frames. Alternatively, for example, the frame detection unit 213 sets coordinate values in the image coordinate system used in the target image (e.g., values on each coordinate axis corresponding to the center of the frame, etc.) as position information, and assigns this position information to each of the frames.
As a result, the frame detection unit 213 generates an image (hereinafter referred to as a "frame image") in which each of a plurality of frames included in the target image is treated as a unit and in which position information is assigned to each of the frames. For example, in the case of a medical examination chart, a frame image is a unit in which a specific character string (e.g., test item, test value, reference value, previous test value, attribute information such as the patient's name, etc.) is recorded, and can be treated as structured data having position information.

The character recognition unit 214 performs character recognition processing on each of the multiple box images generated by the box detection unit 213. This character recognition processing can be realized, for example, by using existing optical character recognition (OCR) technology. In this way, the character recognition unit 214 can perform image analysis processing on each box containing a specific character string based on the box images generated by the box detection unit 213, and can perform character recognition with higher accuracy than when image analysis processing is performed on the entire image.
The character recognition unit 214 may not only use existing optical character recognition technology, but may also perform machine learning using character strings that may be included in the target image in advance (here, character strings such as words used in a medical examination chart).Then, the character recognition process may be performed using a learning model generated by this machine learning.

The information acquisition unit 215 acquires acquisition target information based on the character recognition results for each of the multiple frame images by the character recognition unit 214 and the position information assigned to each of the multiple frame images. Here, the acquisition target information is information that the user wishes to acquire, such as a set of "test item" and "test value" in a medical examination chart.

In order for the information acquisition unit 215 to acquire the target information, in this embodiment, dictionary data is prepared for the first information, which is a part of the target information. For example, when the first information is a "search item," dictionary data is prepared for the text of words (e.g., height, weight, blood pressure, etc.) that are expected to be used as test item names. Here, there are variations in the notation of test item names in health checkups, and although they refer to the same test item in meaning, the test item name may differ depending on the format of the health checkup chart. For example, even if the same test item is high density lipoprotein (HDL: High Density Lipoprotein cholesterol), the test item name in the health checkup chart may be "HDL cholesterol," "HDL-C," or "HDL." Therefore, in the dictionary data, these variations in notation are taken into consideration, and text of multiple test item names is associated with the same test item.

Such dictionary data is, for example, created in advance by an administrator of the image analysis system S and stored in the dictionary data storage unit 262. In other words, the dictionary data storage unit 262 functions as a storage unit that stores dictionary data.

The information acquisition unit 215 then scores the similarity between the character recognition results for each of the multiple frame images by the character recognition unit 214 (i.e., the text of the character strings included in each frame image) and each of the texts of words used as test item names included in the dictionary data stored in the dictionary data storage unit 262. Furthermore, the information acquisition unit 215 marks the frame image that contains the character string with the highest similarity for a certain test item name as a frame image in which the certain test item (i.e., the first information) is described. For example, if a certain frame image contains the character string "HDL-C", the certain frame image is marked as a frame image in which the test item "high density lipoprotein" is described.

Next, the information acquisition unit 215 acquires second information, which is part of the information to be acquired, based on the position information assigned to this marked frame image (i.e., the frame image from which the first information was acquired). For example, assume that this second information is a "search value" for a certain test item. In this case, the information acquisition unit 215 acquires position information of the frame image marked in correspondence with this certain test item. Furthermore, the information acquisition unit 215 scans through each frame image existing in any predetermined direction (e.g., to the right) from this position information to search whether the test value is included.

Here, the realistic upper and lower limits of the test value for each test item can be predicted from a medical perspective. Therefore, for example, a realistic upper limit and lower limit are associated with each test item in the dictionary data. Then, in the process of performing a scanning search for a certain test item, when a frame image containing a value that is equal to or lower than the upper limit value associated with the certain test item and equal to or higher than the lower limit value is detected, the information acquisition unit 215 identifies the frame image as a frame image in which the test value for the certain test item is written. Then, this test value is acquired as the second information. As a result, even in a table such as a health checkup chart in which the relative positional relationship between the frame in which the first information is written and the frame in which the second information is written differs depending on the format, a scanning search is performed, making it possible to perform image analysis with high accuracy. In addition, since the second information is acquired based on the realistic upper and lower limits, it is possible to prevent the acquisition of erroneous second information, and from this perspective, it is also possible to perform image analysis with high accuracy.

In this way, the information acquisition unit 215 can acquire a set of first information (e.g., a certain test item) and corresponding second information (test value of this certain test item), which are the information to be acquired. Note that such marking and the scanning search based on the position information of the marked frame image may be performed each time a piece of first information (e.g., a certain test item) is marked. Alternatively, when there are multiple pieces of first information, all of the first information may be marked and then the scanning search may be performed.

The information acquisition unit 215 stores the target information thus acquired (for example, a pair of the test item name corresponding to the test item and its test value) in the analysis result storage unit 263 as the image analysis result. That is, the analysis result storage unit 263 functions as a storage unit that stores the image analysis result. The information acquisition unit 215 also transmits the image analysis result to the terminal 1 so that the user can be notified of the image analysis result on the terminal 1 as well.

The device-side notification unit 216 notifies the user of the image analysis results stored in the analysis result storage unit 263 by the information acquisition unit 215. The user can understand the contents of the image analysis results by referring to the notified image analysis results. This notification to the user is performed in the same manner as the terminal-side notification unit 112 of the terminal 1. For example, this can be realized by displaying on a display included in the output unit 28, or printing on paper media from a printer (not shown).

The specific content that is notified as a result of image analysis will be described later together with an explanation of the operation with reference to the flowchart in Figure 5.

Above, we have explained in detail the configuration of each device included in the image analysis system S. Next, we will explain in more detail the processing content of each process performed by each device included in the image analysis system S.

[Image analysis processing]
4 is a flowchart for explaining the flow of the photographing process performed by the terminal 1. The photographing process is executed when the user turns on the power to the terminal 1, for example.

In step S11, the image capture control unit 111 starts controlling image capture in the terminal 1. For example, the image capture control unit 111 displays a user interface that allows the user to capture an image of the document 3 as a subject, a live view image acquired by the imaging unit 19, and the like, on a display included in the output unit 18.

In step S12, the shooting control unit 111 determines whether or not a shooting instruction operation has been received from the user. If a shooting instruction operation has been received, the result of step S11 is determined as Yes, and the process proceeds to step S12. On the other hand, if a shooting instruction operation has not been received, the result of step S11 is determined as No, and the process repeats the determination of step S12.

In step S13, the shooting control unit 111 controls the imaging unit 19 to generate and acquire a target image.

In step S14, the imaging control unit 111 transmits the target image acquired in step S13 to the image analysis device 2. This ends the present process.
By the photographing process described above, the image analyzing device 2 can obtain a target image.

[Image analysis processing]
5 is a flowchart illustrating the flow of image analysis processing performed by the image analysis device 2. The image analysis processing is executed when a target image is transmitted from the terminal 1 or when an instruction operation to start the image analysis processing is received from the user to the image analysis device 2.

In step S21, the image acquisition unit 211 acquires the target image.

In step S22, the table detection unit 212 detects table regions corresponding to each of a plurality of tables (there may be only one table) included in the target image acquired in step S21.
Specifically, the table detection unit 212 first performs preprocessing. For example, the table detection unit 212 converts the target image to grayscale and adjusts (i.e., resizes) the size to 1280×960. Next, the table detection unit 212 separates the background region from the frame and character region in the target image and performs binarization to express them as 0 or 1. For example, the table detection unit 212 performs binarization using retinex filtering. In binarization, for example, white in the image is expressed as 0 and black is expressed as 1. In addition, the table detection unit 212 performs filtering to remove the character region. As a premise, the frame of the table spreads over a wide area in the target image, and characters are included within the frame. Therefore, it is considered that the pixels are sparsely connected. For this reason, it is considered that the variance in both the x direction and the y direction of the frame is large.

Next, the line segments where the borders of a table have been split by noise are considered to be extremely long in either the x or y direction, and extremely short in the other direction. For this reason, the variance of the line segments is considered to be large in one of the x and y directions, and small in the other. Based on this property, for example, the table detection unit 212 identifies the line segments and borders by threshold processing using the eigenvalues of the covariance matrices in the x and y directions of the image, and any areas that do not fit into either of these line segments or borders are deemed to be character areas and removed. The table detection unit 212 also performs thinning to thin the line segments and borders so that only one pixel at the center remains.

Then, the table detection unit 212 detects table regions corresponding to each of the multiple (or single) tables contained in the target image. For example, when multiple tables exist in the target image, the table detection unit 212 divides the image so that only one table exists in one image. For this purpose, the table detection unit 212 creates cumulative energy maps in the x and y directions used in seam carving. In addition, the table detection unit 212 performs backtracking in valleys in the energy map consisting of pixels whose energy difference with adjacent pixels is equal to or greater than a threshold value. Then, the table detection unit 212 detects each table region by using the pixels traced by this backtracking as the boundaries of table or document regions, and divides each table region into an image corresponding to one table region.

In step S23, frame detection unit 213 detects a plurality of frames constituting a table from each of the table regions detected in step S22.
Specifically, the frame detection unit 213 first performs frame line separation. In this case, as described above, the frame detection unit 213 detects the frame lines of the first direction and the frame lines of the second direction, which are the frame lines of the multiple frames constituting the table, separately. This point will be described with reference to FIG. 6. FIG. 6 is a schematic diagram showing the frame lines of the first direction and the frame lines of the second direction being detected separately. FIG. 6(a) shows the frame lines before separation in the image. In this way, the frame lines of the multiple frames constituting the table include frame lines of the first direction (here, the vertical direction) and the second direction (here, the horizontal direction). The frame detection unit 213 detects these frame lines by separating them into the frame lines of the first direction (here, the vertical direction) as shown in FIG. 6(b-1) and the frame lines of the second direction (here, the horizontal direction) as shown in FIG. 6(b-2).

The method assumes that the width of the border is adjusted to 1 by the thinning performed in step S22. Therefore, by deleting pixels with a width of 1 in the first direction (here, the vertical direction) of the image, the border in the second direction (here, the horizontal direction) is deleted. This results in an image with only the border in the first direction (here, the vertical direction) as shown in FIG. 6(b-1). Similarly, by deleting pixels with a width of 1 in the second direction (here, the horizontal direction), the border in the first direction (here, the vertical direction) is deleted. This results in an image with only the border in the second direction (here, the horizontal direction) as shown in FIG. 6(b-2). This makes it possible to separate the borders in the first direction (here, the vertical direction) and the second direction (here, the horizontal direction), and to accurately detect the borders.

We will now explain the case where such separation is not performed. As a premise, when photographing a paper medium such as a medical examination chart, distortion occurs in both the vertical and horizontal frame lines due to curling of the paper, etc. For this reason, if separation is not performed, the accuracy of frame line detection will decrease. In contrast, in this embodiment, the frame lines are separated in multiple directions by separating them into the vertical and horizontal directions and then detecting them. Then, each of the separated frame lines in multiple directions is detected as a curve with a limit on the curvature of the line. This simplifies the detection of frame lines in multiple directions and allows frame lines to be accurately detected in response to each distortion of the frame lines in both the vertical and horizontal directions.

Next, the frame detection unit 213 performs frame line detection. As described above, because the frame lines in multiple directions have been separated, the separated images each contain only frame lines in the first direction (here, the vertical direction) or only frame lines in the second direction (here, the horizontal direction). Therefore, the set of pixels obtained by looking at the connectivity of the eight neighbors for these images becomes the vertical or horizontal frame lines, respectively. In this embodiment, curve approximation is performed on this set of pixels by using the least squares method.

Specifically, pixel groups are detected by looking at the connection relationships of the eight neighbors of the separated image. The pixel groups obtained in this way include not only the table frame, but also characters that were not deleted during filtering because they were adjacent to the frame. Therefore, characters are deleted by threshold processing using the number of pixels in the pixel group used during filtering and the eigenvalues of the covariance matrix.
In addition, curve approximation is performed starting from a pixel set with a large number of pixels, and the degree is gradually increased from 1 to 4 using a polynomial. A threshold judgment is performed as shown in the following equation (1), and an approximation curve is created only if the value is below the threshold.

ただし、式（１）において、ｘ及びｙは座標である。

In formula (1), x and y are coordinates.

Here, the threshold value (for example, 1.1 as described above in equation (1)) can be set appropriately. Then, when the approximation curve is created, the separated frame lines are synthesized. To synthesize the frame lines, a pixel set is searched for within three pixels surrounding the created approximation curve. If a pixel set is found, an iterative method is used to perform curve approximation again using the least squares method, using both that pixel set and the pixel set used for the curve approximation.

As soon as the curve approximation is completed for all pixel sets, the frame detection unit 213 deletes the intersecting lines. The frame lines of a table are composed of parallel lines, and vertical lines and horizontal lines do not intersect with each other. Therefore, if lines intersect within a frame line image separated vertically and horizontally, it is assumed that there is a high possibility that a line outside the frame of the table has been mixed in. In this embodiment, if vertical and horizontal frame lines intersect, the outer frame lines are deleted by deleting the intersecting lines that have the fewest number of pixels from the pixel sets used for the approximation.

Then, the frame detection unit 213 performs frame detection. In frame detection, a frame structure is detected using lines. When the above-mentioned frame line detection is completed, a group of vertical and horizontal approximation curves is generated. In frame detection in this step, first, the position of the intersection of the vertical and horizontal approximation curves is identified. The coordinates of the intersection obtained in this way become candidate positions for the corners of the frame, and one vertical frame line can obtain intersections for the number of horizontal frame lines, and one horizontal frame line can obtain intersections for the number of vertical frame lines.

After that, one vertical or horizontal frame line is selected, and one pair of adjacent intersection points on the approximation curve is selected. The intersection points are then searched for in the frame line image, and if a certain number of pixels are present, it is determined to be a frame line. By applying this process to adjacent pairs of intersection points on all approximation curves, frame lines in the image can be detected. The area enclosed by the detected frame line is then detected as a frame.

The frame detection unit 213 also divides and cuts out each of the detected multiple frames. As described above, the frame detection unit 213 detects the frame based on the frame line surrounding the frame. Therefore, the frame detection unit 213 searches for the outermost coordinates in each of the x and y coordinates included in the frame line surrounding the frame, and divides and cuts out each of the multiple frames using the coordinates obtained by this search. Furthermore, the frame detection unit 213 assigns position information in the target image to each of the cut-out multiple frames. For example, the frame detection unit 213 sets the combination of table identification information (e.g., an identifier assigned to the table) assigned to each detected table area and frame identification information (e.g., an identifier assigned to the frame) as position information, and assigns this position information to each of the multiple frames. Alternatively, for example, the frame detection unit 213 sets the coordinate values in the image coordinate system used in the target image (e.g., values on each coordinate axis corresponding to the center of the frame, etc.) as position information, and assigns this position information to each of the multiple frames. As a result, the frame detection unit 213 generates a frame image that is an image made up of each of the multiple frames contained in the target image, each of which is assigned position information.

Returning to FIG. 5, in step S24, the character recognition unit 214 performs character recognition processing on each of the multiple frame images generated in step S23, with the frame image serving as a unit.

In step S25, the information acquisition unit 215 acquires target information based on the character recognition results for each of the multiple frame images that were character recognized in step S24 and the position information assigned to each of the multiple frame images. This will be described with reference to FIG. 7. FIG. 7 is a schematic diagram showing a table structure included in the target image and a method for the information acquisition unit 215 to acquire target information.

Figure 7(a) shows the table structure included in the target image. In Figure 7(a), the target image 5 includes a table 51 and multiple frames 52 (for convenience of illustration, only one frame is marked in the figure). Note that, for simplicity of explanation, only one table 51 is shown in the figure, but the target image 5 may include multiple tables 51. By performing the above-mentioned image analysis process on such a target image 5, the table detection unit 212 detects a table area corresponding to the table 51. In addition, the frame detection unit 213 generates a frame image for each of the multiple frames 52 included in the table 51.

7B shows a method of acquiring information to be acquired by the information acquisition unit 215. As described above, the information acquisition unit 215 scores the similarity between the character recognition result for each of the multiple box images by the character recognition unit 214 (i.e., the text of the character string included in each box image) and each text of the words used as the names of the test items included in the dictionary data stored in the dictionary data storage unit 262. In addition, the information acquisition unit 215 marks the box image that contains the character string with the highest similarity for a certain test item name as a box image containing the test item name as a result of the scoring, as a box image containing the test item name. For example, assume that the box 52a is marked this time. Then, the information acquisition unit 215 acquires the position information of the box image 52a marked in correspondence with this certain test item. In addition, the information acquisition unit 215 scans and searches for the test value for each box image that exists in any predetermined direction (for example, the right direction) from the position information of the box image 52a, as shown by the arrow in the figure.

In this way, the information acquisition unit 215 can acquire a set of first information (e.g., a certain test item) and corresponding second information (test value of this certain test item), which are the acquisition target information. The acquisition target information acquired in this way (e.g., a set of the test item name corresponding to the test item and its test value) is regarded as the image analysis result.

In step S26, the terminal-side notification unit 112 or the device-side notification unit 216 notifies the user of the image analysis result in step S25. The user can understand the contents of the image analysis result by referring to the notified image analysis result. Note that the notification may be performed by only either the terminal-side notification unit 112 or the device-side notification unit 216, or by both. This ends the process.

Here, the notification in step S26 may be, but is not limited to, simply displaying the image analysis results. For example, when image analysis processing is performed on multiple health checkup charts corresponding to one user (e.g., each of the health checkup charts for each year in the case where a health checkup is performed once a year), changes in the test values corresponding to a specified test item along a time series may be detected based on the image analysis results, and the detected changes may be notified.

In this case, for example, the order in which each health check was performed is identified by detecting the implementation date written on each health check sheet using character recognition or based on user operation, etc. Then, the results of the health check (e.g., test values) obtained by character recognition from each health check sheet are notified in correspondence with the order in which the health check was performed. For example, notification is made by displaying test values for the same test item in table format in correspondence with the order in which the health check was performed. Alternatively, notification is made by displaying in graph format, for example, with time (e.g., the date the health check was performed) on the horizontal axis and test values for the same test item on the vertical axis.

In this case, further, for example, a threshold value (e.g., an upper limit value or a lower limit value) indicating the appropriate range of the test value may be set, and if the test value falls outside this appropriate range (e.g., if it is equal to or greater than the upper limit value or equal to or less than the lower limit value), the test value may be displayed in a different color or together with a warning message to notify the user. In addition, for example, a threshold value for the amount of change in the test value from the previous health check may be set, and if the amount of change in the test value exceeds the threshold (e.g., if the test value has changed significantly), the test value may be displayed in a different color or together with a warning message to notify the user. This allows the user to not only grasp the changes in the test values of the same test item over time, but also to grasp changes such as the test value being outside the appropriate range or having changed significantly.
Therefore, the user can more appropriately understand the results of the health check.

According to the photographing process and image analysis process described above, the tables and frames are detected in stages, and then the desired target information is obtained using the position information. This makes it possible to perform image analysis with higher accuracy than image analysis methods that simply perform image analysis without detecting tables and frames or using position information. As a result, even in cases where, for example, various formats are mixed and the arrangement of tables and frames differs depending on the document, or where distortion has occurred in the image that would normally make image analysis difficult, it is possible to perform appropriate image analysis.

[Modification]
Although the embodiment of the present invention has been described above, this embodiment is merely an example and does not limit the technical scope of the present invention. The present invention can take various other embodiments and can be modified in various ways, such as omissions and substitutions, without departing from the gist of the present invention. In this case, these embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents.
As an example, the embodiment of the present invention described above may be modified as follows.

In the above-described embodiment, the image analysis system S is realized as a client-server type system, such as a combination of a terminal 1 and an image analysis device 2. However, the image analysis system S may be configured in a different way from the example shown in the above-described embodiment.

For example, image analysis device 2 may be realized by multiple computers rather than one computer. In this case, for example, some of the functional blocks of image analysis device 2 (e.g., character recognition unit 214 and information acquisition unit 215) may be realized by one computer, and other functional blocks may be realized by another computer. Also, for example, some or all of the functional blocks of image analysis device 2 may be realized by distributed processing by multiple computers (e.g., a cloud system).

In addition, for example, the functions of the terminal 1 and the image analysis device 2 may be realized by a single computer by providing an imaging unit in the image analysis device 2. In this case, the communication function may be omitted from this computer to make it a stand-alone device.

Furthermore, the person using terminal 1 may be the user who has undergone the health check, or another entity such as a business that employs the user who has undergone the health check.

Furthermore, for example, in order to further improve the accuracy of the character recognition process in step S24, character detection processing may be performed on each of the multiple frame images generated in step S23. This processing is performed, for example, by the frame detection unit 213 that generated the multiple frame images following the generation of the frame images. In this case, the frame detection unit 213 detects characters from the target image for each region of the multiple frame images that it has detected. This allows the frame detection unit 213 to accurately detect a string of characters (for example, one word) from a specified region (for example, within a frame containing characters) based on its own accurate frame detection results.

FIG. 8 is a flowchart illustrating the flow of the character detection process.
In step S31, the frame detection unit 213 acquires a frame image to be processed this time from the multiple frame images it has generated.

In step S32, the frame detection unit 213 performs binarization. In this modified example, characters are detected using a projection method in step S35 described later. When using a projection method, binarization is necessary to count black pixels in the character region and separate the character region from the background region. Any method can be used for binarization, but for example, Otsu's binarization is used. Otsu's binarization is also called discriminant analysis, and creates a histogram of brightness values for an image, and performs binarization using a threshold value that maximizes the inter-class variance when the histogram is divided into two. Many medical checkup charts use a variety of colors, but if you look inside the frame, you will see a binary image of the characters and background, so by using Otsu's binarization, it is possible to automatically calculate the optimal threshold value and extract the characters.

In step S33, the frame detection unit 213 deletes the outside-frame area. This will be described with reference to FIG. 9. FIG. 9 is a schematic diagram showing the deletion of the outside-frame area from a binarized image and then the projection transformation. In step S32, the frame image is cut out using the outermost of the four frame line coordinates used to detect the frame. Therefore, if the frame line is tilted or distorted, characters and frame lines outside the frame are detected as shown in the binarized image in FIG. 9(a). These characters and frame lines will be an obstacle to subsequent character detection. Therefore, by deleting the pixels outside the frame lines used in frame detection, they can be deleted as shown in FIG. 9(b).

In step S34, the frame detection unit 213 performs a projective transformation. The image of the target captured by the camera is distorted due to the curvature of the paper and the positional relationship between the camera and the target. In order to deal with this distortion, a projective transformation is used. The transformation formula for the projective transformation is expressed by the following formula (2), and can be executed when H is regular.

ただし、式（２）において、ｘ’及びｙ’は変換後の座標であり、ｘ及びｙは変換前の座標であり、Ｈ_１１～Ｈ_３３は変換係数である。また、式（２）において、ｆ_０はスケールを表す定数であり、ｘ／ｆ_０及びｙ／ｆ_０が０になるように調整される。

In formula (2), x' and y' are coordinates after conversion, x and y are coordinates before conversion, and H ₁₁ to H ₃₃ are conversion coefficients. Also, in formula (2), f ₀ is a constant representing a scale, and is adjusted so that x/f ₀ and y/f ₀ become 0.

The transformation matrix of this projective transformation contains eight transformation coefficients, and two equations can be obtained from one set of coordinates before and after transformation, so the transformation coefficients can be found by preparing four pairs of coordinates before and after transformation. In this modified example, the intersections of the frame lines are used for these four coordinates. For the transformed coordinates, the coordinates of the top left of the frame are fixed, and the distances in the x-axis direction between the intersections of the top left and top right, and the intersections of the bottom left and bottom right are calculated, and the longest distance is set as the horizontal frame length after transformation. Similarly, the distances in the y-axis direction between the intersections of the top left and bottom left, and the intersections of the top right and bottom right are calculated, and the longest distance is set as the vertical frame length after transformation, so that the transformed frame is defined as a rectangle. The coordinates of the intersections are used as they are for the coordinates before transformation, and transformation coefficients are determined using these four coordinates, and projective transformation is performed as shown in Figure 9 (c).

In step S35, the frame detection unit 213 performs character detection. A projection method is used for character detection. In the projection method, the number of pixels that correspond to black after binarization at each coordinate in the y and x directions of the image is counted, and ranges where the number of pixels is not 0 are detected as lines and characters. However, this method has problems such as the components of katakana characters such as "リ" and "ル" and the radicals and components of kanji characters such as "研" and "痛" being detected separately to the left and right. Therefore, in this modified example, the characters are synthesized after being detected separately.

In this case, first, for the characters detected using the projection method, the horizontal length of each character is divided by the vertical length of each character to calculate the aspect ratio for each detected character, as shown in the following formula (3).

ただし、式（３）において、ｒ_ｉは各文字の縦横比であり、ｗ_ｉは各文字の幅であり、ｈは行の高さである。

In equation (3), r _i is the aspect ratio of each character, w _i is the width of each character, and h is the line height.

This aspect ratio _ri is relatively large for full-width characters and relatively small for half-width characters. The medical examination sheet that is the subject of image analysis processing has character areas such as test items and numerical areas of test results, and the numerical areas are written in half-width characters. Therefore, if the characters present within the frame are only half-width characters, it is considered that character synthesis is not necessary. Therefore, this aspect ratio _ri is used to distinguish between the character area and the numerical area of the table. For this purpose, a threshold value is set for discrimination, and if there is anything that is equal to or exceeds the threshold, it is regarded as a character area, and the characters within the frame of that character area are candidates for character synthesis.

Whether or not to actually combine these characters is determined by considering adjacent characters that satisfy the condition expressed by the following formula (4).

ただし、式（４）において、ｘ_{ｉ＋１，１}及びｘ_ｉ，２は文字の横方向の始点及び終点であり、ｓ_ｉは検出文字間の距離である。

In equation (4), x _i+1,1 and x _i,2 are the horizontal start and end points of a character, and s _i is the distance between detected characters.

Then, it is considered whether to combine characters in ascending order of distance, and the aspect ratio of the characters after the character combination is calculated using the above formula (3). Then, for characters whose calculated aspect ratio is equal to or greater than the threshold, they are actually combined as a single character detected separately in the character region. However, for characters whose calculated aspect ratio is less than the threshold, they are actually not combined as a single character detected without separation in the character region. This prevents, for example, erroneous combination of numbers written in half-width characters, and on the other hand, when the radical and the part of a kanji character are detected separately, they can be combined as a single character. This ends the process. The character recognition unit 214 can then perform the character recognition process in step S24 with high accuracy on a group of characters (for example, one word) accurately detected by this process.

[Configuration example]
As described above, the image analysis system S according to this embodiment includes the table detection unit 212 , the frame detection unit 213 , and the information acquisition unit 215 .
The table detection unit 212 detects an area corresponding to a table from an image that includes a table as a subject.
The frame detection unit 213 detects a plurality of frames constituting a table from an area corresponding to the table, and assigns position information in the image to each of the detected frames.
The information acquisition unit 215 acquires the acquisition target information based on the character recognition results for each of the multiple boxes and the position information assigned to each of the multiple boxes.

In this way, the image analysis system S detects tables and frames in stages, and then uses the position information to obtain the desired target information. This makes it possible to perform image analysis with greater accuracy than image analysis methods that simply perform image analysis processing without detecting tables and frames or using position information. As a result, even in cases where, for example, various formats are mixed and the arrangement of tables and frames differs from manuscript to manuscript, or where distortions have occurred in the image that would normally make image analysis difficult, appropriate image analysis can be performed.

The information acquisition unit 215 acquires the first information, which is a part of the acquisition target information, based on dictionary data for the first information and the character recognition results for each of the multiple boxes.
The information acquisition unit 215 acquires second information, which is a part of the acquisition target information, based on position information corresponding to the frame from which the first information was acquired.
This allows image analysis to be performed with higher accuracy by using dictionary data corresponding to the target information to be obtained.

The information acquisition unit 215 acquires the second information by using the position information corresponding to the frame from which the first information was acquired as a reference, and by scanningly searching for other frames in a specified direction from the reference based on the position information of the other frames.
This allows for more accurate image analysis to be performed, for example, in a table in which related information is arranged vertically or horizontally.

The frame detection unit 213 detects the frame lines of multiple frames, that is, the frame lines in a first direction and the frame lines in a second direction, separately from the area corresponding to the table, and detects multiple frames based on the intersections of the detected frame lines in the first direction and the frame lines in the second direction.
In this way, by detecting the frame line in the first direction and the frame line in the second direction separately, the frame line and the frame can be detected more accurately than if they were detected without being separated.

The frame detection unit 213 cuts out each of the detected frames,
The character recognition results are generated by performing image analysis processing on each of the extracted frames.
This allows image analysis processing to be performed for each box containing a character string, thereby enabling character recognition with higher accuracy.

The image is an image generated by photographing any one of a plurality of tables having different formats on which the medical examination results are recorded,
The information to be acquired includes at least the medical examination results.
This makes it possible to obtain medical examination results recorded in various formats with high accuracy.

[Realization of functions through hardware and software]
The function of executing the series of processes according to the above-mentioned embodiment can be realized by hardware, software, or a combination of these. In other words, it is sufficient that the function of executing the series of processes described above is realized in any one of the image analysis systems S, and there is no particular limitation on how this function is realized.

For example, when the function of executing the above-mentioned series of processes is realized by a processor that executes arithmetic processing, the processor that executes this arithmetic processing includes not only those that are composed of various processing devices alone, such as single processors, multiprocessors, and multicore processors, but also those that combine these various processing devices with processing circuits such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays).

For example, when the function of executing the above-mentioned series of processes is realized by software, the program constituting the software is installed on a computer via a network or a recording medium. In this case, the computer may be a computer with dedicated hardware built in, or a general-purpose computer (e.g., a general-purpose electronic device such as a general-purpose personal computer) that can execute a specified function by installing a program. The steps of writing the program may include only processes that are performed chronologically according to the order, but may also include processes that are executed in parallel or individually. The steps of writing the program may be executed in any order within the scope of the gist of the present invention.

A recording medium on which such a program is recorded may be provided to a user by being distributed separately from the computer main body, or may be provided to a user in a state in which it is already installed in the computer main body. In this case, the storage medium distributed separately from the computer main body may be, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. The optical disk may be, for example, a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc), or a Blu-ray (registered trademark) Disc. The magneto-optical disk may be, for example, an MD (Mini Disc). These storage media are, for example, loaded into drive 15 in FIG. 2 and drive 25 in FIG. 3 and installed in the computer main body. Furthermore, the recording medium provided to the user in a state where it is pre-installed in the computer main body is composed of, for example, ROM 12 in FIG. 2 and ROM 22 in FIG. 3 in which the program is recorded, and an HDD or SSD included in storage unit 16 in FIG. 2 and storage unit 26 in FIG. 3, etc.

1 Terminal, 2 Image analysis device, 3 Document (health checkup chart), 11, 21 CPU, 12, 22 ROM, 13, 23 RAM, 14, 24 Communication unit, 15, 25 Drive, 16, 26 Storage unit, 17, 27 Input unit, 18, 28 Output unit, 111 Shooting control unit, 112 Terminal side notification unit, 161, 261 Image storage unit, 162, 263 Analysis result storage unit, 211 Image acquisition unit, 212 Table detection unit, 213 Frame detection unit, 214 Character recognition unit, 215 Information acquisition unit, 216 Device side notification unit, 262 Dictionary data storage unit, S Image analysis system

Claims (5)

a table detection means for detecting a plurality of table regions corresponding to a plurality of tables from a single target image including the plurality of tables as subjects;
a frame detection means for detecting a plurality of frames constituting the table for each of the plurality of table regions and assigning position information in the target image to each of the detected plurality of frames;
an information acquisition means for acquiring acquisition target information for each of the plurality of tables based on a character recognition result for each of the plurality of frames and position information assigned to each of the plurality of frames;
Equipped with
the target image in which the table detection means detects a plurality of table regions is an image in which distortion occurs with respect to the plurality of tables that are objects;
the frame detection means detects intersections of frame lines, and performs projective transformation for each frame using coordinates of the intersections, thereby correcting the distortion.
1. An image analysis system comprising: a table detection means for detecting a plurality of table regions corresponding to a plurality of tables from a single target image including the plurality of tables as subjects;
a frame detection means for detecting a plurality of frames constituting the table for each of the plurality of table regions and assigning position information in the target image to each of the detected plurality of frames;
an information acquisition means for acquiring acquisition target information for each of the plurality of tables based on a character recognition result for each of the plurality of frames and position information assigned to each of the plurality of frames;
Equipped with
The target image includes a frame containing only half-width numbers and a frame containing full-width characters,
the frame detection means detects characters from frames included in the target image, and regards a frame containing a character whose aspect ratio, which is a value obtained by dividing the width of one character by the height of the one character, is equal to or greater than a threshold value as a frame containing a full-width character, and sets the character within the frame as a candidate for processing to perform character synthesis.
1. An image analysis system comprising: a table detection means for detecting a plurality of table regions corresponding to a plurality of tables from a single target image including the plurality of tables as subjects;
a frame detection means for detecting a plurality of frames constituting the table for each of the plurality of table regions and assigning position information in the target image to each of the detected plurality of frames;
an information acquisition means for acquiring acquisition target information for each of the plurality of tables based on a character recognition result for each of the plurality of frames and position information assigned to each of the plurality of frames;
Equipped with
The frame detection means
generating an image by deleting pixels having a width in a first direction less than a predetermined value from the table region, and detecting a frame line in the first direction ;
generating an image by deleting pixels having a width less than a predetermined value in a second direction intersecting with the first direction from the table region , and detecting a frame line in the second direction;
detecting the plurality of frames based on intersections of the detected frame lines in the first direction and the detected frame lines in the second direction;
1. An image analysis system comprising: the table detection means divides the target image so that one image contains one table;
the frame detection means further divides the target image divided by the table detection means so that one frame is included in one image;
the information acquisition means acquires the acquisition target information for each of the target images in units of frames further divided by the frame detection means.
4. The image analysis system according to claim 1 , wherein the image analysis system further comprises: a first section for detecting a first image ; the target image is an image in which distortion occurs in a plurality of tables having different formats on which medical examination results are recorded, the plurality of tables being generated by photographing the plurality of tables;
The information to be acquired includes at least a medical examination result.
5. The image analysis system according to claim 1 , wherein the image analysis system comprises: a first section;

Images