JP7536463B2 - Imaging device, control method thereof, and program - Google Patents

Description

The present invention relates to an imaging apparatus, a control method thereof , and a program therefor .

When a person or animal is lying down, their body weight puts pressure on the contact area between the body and the surface, which can lead to the development of bedsores. Patients who develop bedsores should receive bedsore care such as pressure redistribution care and skin care, and regularly evaluate and manage the bedsores.

Non-Patent Document 1 proposes the DESIGN-R (registered trademark), a pressure ulcer status assessment scale developed by the Academic Education Committee of the Japanese Society of Pressure Ulcers, as a tool for assessing pressure ulcers. DESIGN-R is a tool for evaluating the healing process of wounds, including pressure ulcers. The name of this scale is derived from the initials of the observation items: depth, exudate, size, inflammation/infection, granulation, and necrotic tissue.

DESIGN-R is available in two versions: one for severity classification, which is used for simple daily evaluation, and one for progress evaluation, which shows the flow of the healing process in detail. DESIGN-R for severity classification divides six evaluation items into two categories, mild and severe, with mild being represented by lowercase letters and severe being represented by uppercase letters.
By using the severity classification system to evaluate the pressure ulcer at the time of initial treatment, it is possible to grasp the general condition of the ulcer. By knowing which items are problematic, it is easy to decide on a treatment plan.

Meanwhile, DESIGN-R has also been established for progress evaluation, which allows for comparison of severity between patients in addition to progress evaluation. R stands for rating. Each item is weighted differently, and the total score (0-66 points) of the six items other than depth represents the severity of the pressure ulcer. After treatment has begun, the treatment progress can be evaluated in detail and objectively, allowing not only an individual progress evaluation but also comparison of severity between patients.
Here, the size evaluation of DESIGN-R involves measuring the major axis and minor axis (the maximum axis perpendicular to the major axis) of the skin damage area (cm), multiplying them together to obtain a size, which is then classified into seven levels. The seven levels are: s0: no skin damage, s3: less than 4, s6: 4 to less than 16, s8: 16 to less than 36, s9: 36 to less than 64, s12: 64 to less than 100, and S15: 100 or more.

As described in Non-Patent Document 1, it is recommended that DESIGN-R be scored once every one to two weeks to evaluate the healing process of pressure ulcers and select appropriate care, and pressure ulcers need to be regularly evaluated and managed. In addition, accuracy is required in the evaluation to confirm changes in the condition of pressure ulcers.
Currently, the size of a bedsore is often determined based on manual measurements made by placing a tape measure on the affected area. Specifically, the longest two points in the damaged area of the skin are measured, and this is taken as the long axis. The length perpendicular to the long axis is taken as the short axis, and the value obtained by multiplying the long axis and the short axis is taken as the size of the bedsore.

Shorinsha Bed Sore Guidebook, 2nd Edition, Based on the Bed Sore Prevention and Management Guidelines (4th Edition), Edited by the Japanese Bed Sore Society, ISBN13 978-4796523608, 23 pages

However, when photographing a bedsore, the photographer must assume an unnatural position, such as half-raising a bedridden patient to photograph his/her back, which may result in inappropriate photographing. Since the shape and area of the bedsore change depending on the patient's posture, the appearance of the bedsore may change each time it is photographed. Therefore, even when comparing images of bedsores, it may be difficult to accurately compare their progress. This is not limited to bedsores, but is also true when photographing burns or lacerations.
The present invention has been made in consideration of the above-mentioned problems, and has an object to make it possible to appropriately photograph an affected area.

The present invention is an imaging device having an imaging means for photographing an affected area, and a control means for acquiring information on the size of the affected area in an image photographed by the imaging means, and controlling the timing of photographing the affected area by the imaging means based on the information on the size of the affected area, or controlling the timing of prompting a user to perform a photographing operation, wherein the control means controls the imaging means to photograph the affected area, or controls the user to prompt the user to perform a photographing operation, when the size of the affected area is equal to or larger than a threshold, and the threshold is a value based on the size of the affected area previously photographed by the same patient .

The present invention allows the affected area to be photographed properly.

FIG. 1 is a diagram illustrating an overview of an image processing system. FIG. 2 is a diagram illustrating a hardware configuration of an imaging apparatus. FIG. 2 is a diagram illustrating a hardware configuration of an image processing device. 4 is a flowchart showing a process of the image processing system. FIG. 13 is a diagram for explaining a method for calculating the area of an affected region. 11A and 11B are diagrams for explaining a method of superimposing information on image data of an affected area. 11A and 11B are diagrams for explaining a method of superimposing information on image data of an affected area. 4 is a flowchart showing a process of the image processing system.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First Embodiment
FIG. 1 is a diagram showing an example of an outline of an image processing system 1. As shown in FIG.
The image processing system 1 includes an imaging device 200, which is a handheld portable device, and an image processing device 300. In this embodiment, an example of the pathology of the affected part 102 of the subject 101 will be described as a bedsore occurring on the buttocks.

In the image processing system 1, the imaging device 200 captures an image of the affected area 102 of the subject 101 using a live view, and transmits the captured image data of the live view to the image processing device 300. The image processing device 300 extracts the affected area of the affected area 102 from the received image data, calculates the area of the affected area, and transmits information on the calculated area to the imaging device 200. The imaging device 200 automatically captures the affected area 102 when the area of the received affected area is equal to or greater than a threshold value. Note that in this embodiment, the description will be given using an example in which the affected area 102 is a bedsore, but this is not limited to a bedsore, and may be a burn or laceration.

FIG. 2 is a diagram showing an example of a hardware configuration of the imaging device 200. As shown in FIG.
The imaging device 200 may be a general single-lens camera, a compact digital camera, or a smartphone or tablet terminal equipped with a camera with an autofocus function.
The imaging unit 211 has a lens group 212, a shutter 213, and an image sensor 214. The focus position and zoom magnification can be changed by changing the positions of the multiple lenses included in the lens group 212. The lens group 212 also has an aperture for adjusting the amount of exposure.

The image sensor 214 is composed of a charge-storage type solid-state image sensor such as a CCD or CMOS sensor that converts an optical image into image data. Reflected light from the subject that passes through the lens group 212 and the shutter 213 is focused on the image sensor 214. The image sensor 214 generates an electrical signal corresponding to the subject image and outputs image data based on the generated electrical signal.

The shutter 213 exposes or blocks light to the image sensor 214 by opening and closing the blade members, thereby controlling the exposure time of the image sensor 214. The shutter 213 may be an electronic shutter that controls the exposure time by driving the image sensor 214. When the electronic shutter is performed with a CMOS sensor, a reset scan is performed for each pixel or for each area consisting of multiple pixels (for example, for each line) to reduce the amount of accumulated charge in the pixel to zero. After that, for each pixel or area where the reset scan was performed, a scan is performed to read out a signal corresponding to the amount of accumulated charge after a predetermined time has elapsed.
The zoom control circuit 215 controls a motor for driving a zoom lens included in the lens group 212 , and controls the optical magnification of the lens group 212 .

The distance measurement system 216 calculates distance information to the subject. The distance measurement system 216 may use a general phase difference type distance measurement sensor mounted on a single-lens reflex camera, or may use a TOF (Time Of Flight) sensor. The TOF sensor is a sensor that measures the distance to the object based on the time difference (or phase difference) between the timing of transmitting an irradiated wave and the timing of receiving a reflected wave of the irradiated wave reflected by the object. Furthermore, the distance measurement system 216 may use a PSD type using a PSD (Position Sensitive Device) as a light receiving element.

The image sensor 214 may have multiple photoelectric conversion regions for each pixel, and may be configured to be able to obtain distance information for each pixel or region position from the phase difference between images obtained from each photoelectric conversion region.
In addition, the ranging system 216 may be configured to obtain distance information in one or more predetermined ranging areas within the image, or may be configured to obtain a distance map showing the distribution of distance information for a large number of pixels or areas within the image.
In addition, the distance measurement system 216 may perform TV-AF or contrast AF to extract and integrate high-frequency components of the image data, determine the position of the focus lens where the integrated value is maximized, and obtain distance information from this position of the focus lens.

The image processing circuit 217 performs predetermined image processing on the image data output from the image sensor 214. The image processing circuit 217 performs various image processing such as white balance adjustment, gamma correction, color interpolation or demosaicing, and filtering on the image data output from the imaging unit 211 or the image data stored in the internal memory 221. In addition, the image processing circuit 217 performs compression processing on the image data that has been subjected to image processing in accordance with a standard such as JPEG.
The AF control circuit 218 determines the position of the focus lens included in the lens group 212 based on distance information obtained by the distance measurement system 216, and controls a motor that drives the focus lens.

The communication device 219 is a communication interface for communicating with external devices such as the image processing device 300 via a wireless network. A specific example of the network is a network based on the Wi-Fi (registered trademark) standard. Note that communication using Wi-Fi may be realized via a router. The communication device 219 may also be realized by a wired communication interface such as a USB or LAN.

The system control circuit 220 has a CPU (Central Processing Unit) and performs overall control of the imaging device 200 by executing a program stored in the internal memory 221. The system control circuit 220 also controls the imaging unit 211, the zoom control circuit 215, the distance measurement system 216, the image processing circuit 217, the AF control circuit 218, and the like. Note that the system control circuit 220 is not limited to having a CPU, and may use an FPGA, ASIC, or the like. The system control circuit 220 also generates an internal signal based on predetermined shooting conditions, similar to that generated when the user issues a shooting instruction via the operation unit 224, if the shooting conditions are met.

The internal memory 221 may be, for example, a rewritable memory such as a flash memory or SDRAM. The internal memory 221 temporarily stores various setting information such as information on the focus position when capturing an image required for the operation of the imaging device 200, image data captured by the imaging unit 211, image data processed by the image processing circuit 217, etc. The internal memory 221 may also temporarily store analysis data such as image data and information on the size of the subject received by the communication device 219 through communication with the image processing device 300.

The external memory 222 is a non-volatile recording medium that can be attached to the imaging device 200 or is built into the imaging device 200. For example, an SD card or a CF card can be used as the external memory 222. The external memory 222 records image data that has been image-processed by the image processing circuit 217, image data and analysis data that the communication device 219 receives by communicating with the image processing device 300, and the like. In addition, the image data recorded in the external memory 222 can be read out during playback and output to the outside of the imaging device 200.

The display device 223 may be, for example, a TFT (Thin Film Transistor) liquid crystal display, an organic EL display, an EVF (Electronic Viewfinder), etc. The display device 223 displays image data temporarily stored in the internal memory 221 and image data recorded in the external memory 222, and displays a setting screen of the imaging device 200, etc.
The operation unit 224 is composed of buttons, switches, keys, a mode dial, or a touch panel that also serves as the display device 223, which are provided on the imaging device 200. Instructions such as mode settings and shooting instructions by the user are notified to the system control circuit 220 via the operation unit 224.

The common bus 225 is a signal line for transmitting and receiving signals between each component of the imaging device 200.

FIG. 3 is a diagram illustrating an example of a hardware configuration of the image processing device 300.
The image processing device 300 includes a CPU 310, a storage device 312, a communication device 313, an output device 314, an auxiliary calculation device 317, and the like.
The CPU 310 includes an arithmetic unit 311. The CPU 310 executes a program stored in a storage unit 312 to perform overall control of the image processing device 300.

The storage device 312 includes a main storage device 315 (such as a ROM or a RAM) and an auxiliary storage device 316 (such as a magnetic disk device or an SSD (Solid State Drive)).
The communication device 313 is a wireless communication module for communicating with external devices such as the imaging device 200 via a wireless network.
The output device 314 outputs data processed by the arithmetic device 311, data stored in the storage device 312, etc. to a display, a printer, or an external network connected to the image processing device 300.

The auxiliary calculation device 317 is an auxiliary calculation IC that operates under the control of the CPU 310. A GPU (Graphic Processing Unit) or the like can be used as the auxiliary calculation device 317. The GPU is originally a processor for image processing, but since it has multiple multiply-accumulators and excels at matrix calculations, it can also be used as a processor that performs processing for signal learning. Therefore, GPUs are generally used in processing that performs deep learning. For example, the JetsonTX2 module from NVIDIA can be used as the auxiliary calculation device 317. In addition, an FPGA, an ASIC, or the like can also be used as the auxiliary calculation device 317. The auxiliary calculation device 317 performs processing to extract the affected area from the image data.

The image processing device 300 may have one or more CPUs 310 and storage devices 312. That is, at least one or more CPUs are connected to at least one storage device, and when at least one or more CPUs execute a program stored in at least one or more storage devices, the image processing device 300 executes each function described below. Furthermore, the image processing device 300 is not limited to having a CPU 310, and may use an FPGA, ASIC, or the like.

FIG. 4 is a flowchart showing an example of processing by the image processing system 1.
4, steps S401 to S420 are processes performed by the imaging device 200, and steps S431 to S456 are processes performed by the image processing device 300. The flowchart in Fig. 4 starts when the imaging device 200 and the image processing device 300 are each connected to a network conforming to the Wi-Fi standard, which is a wireless LAN standard.

In S431, the image processing device 300 performs a search process for the imaging device 200 to be connected.
In S401, the imaging device 200 performs a response process to the search process by the image processing device 300. Note that UPnP (Universal Plug and Play) is used as a technology for searching for devices via a network. Here, in UPnP, individual devices are identified by UUID (Universally Unique IDentifier).

In S402, the imaging device 200 starts live view processing by being connected to the image processing device 300 so as to be able to communicate with each other. Specifically, the imaging unit 211 generates image data, and the image processing circuit 217 performs development processing on the generated image data to generate image data for live view display. The imaging unit 211 and the image processing circuit 217 repeatedly perform processing, and a live view image is displayed on the display device 223 at a predetermined frame rate.

In S403, the distance measurement system 216 obtains distance information to the subject, and the AF control circuit 218 starts AF processing to control the driving of the lens group 212 so that the subject is in focus. If the distance measurement system 216 adjusts the focus position using TV-AF or contrast AF, distance information is obtained from the position of the focus lens in the focused state to the subject 101 of the focused image. The position to be focused may be the subject located in the center of the image data, or the subject located in the closest position to the imaging device 200. If the distance measurement system 216 has obtained a distance map of the subject, it may estimate a region to be noted from the distance map and focus on the estimated region. If the position of the affected area in the live view image has already been specified by the image processing device 300, the distance measurement system 216 may focus on the specified position. The imaging device 200 repeatedly displays the live view image and performs AF processing until the release button is pressed in S413 described later.

In S404, the image processing circuit 217 performs development and compression processing on any image data of the live view image to generate image data in, for example, JPEG standard. The image processing circuit 217 performs resizing processing on the compressed image data to reduce the size of the image data.
In S405, the communication device 219 acquires the resized image data and distance information calculated by the distance measuring system 216. Furthermore, the communication device 219 acquires information on the zoom magnification and information on the size (number of pixels) of the resized image data, as necessary.

In S406, the communication device 219 transmits the acquired image data and one or more pieces of information including distance information to the image processing device 300 by wireless communication. Here, the larger the size of the image data to be transmitted, the longer the wireless communication takes. Therefore, the system control circuit 220 determines the size of the image data to be resized by the image processing circuit 217 in S404 based on the allowable communication time. However, if the size of the image data is too small, it will affect the accuracy of the affected area extraction process performed by the image processing device 300 in S442, which will be described later. Therefore, the system control circuit 220 determines the size of the image data based on the communication time as well as the accuracy of the affected area extraction process. Note that the processes from S404 to S406 may be performed for each frame, or may be performed once every few frames.

Next, the process proceeds to the image processing device 300.
In S<b>441 , the communication device 313 of the image processing device 300 receives one or more pieces of information including image data and distance information transmitted from the communication device 219 of the imaging device 200 .
In S442, the CPU 310 and the auxiliary processing unit 317 of the image processing device 300 extract the affected area from the received image data.

As a method for extracting the affected area, semantic area segmentation by deep learning is performed. That is, a neural network model is trained in advance on a high-performance computer for learning using images of the affected areas of multiple actual bedsores as teacher data to generate a trained model. The auxiliary calculation device 317 acquires the trained model from the high-performance computer and estimates the area of the bedsore from the image data based on the trained model. As an example of a neural network model, a fully convolutional network (FCN (Fully Convolutional Network)), which is a segmentation model using deep learning, can be applied. Inference by deep learning is processed by the auxiliary calculation device 317, which is good at parallel execution of product-sum operations. However, inference by deep learning may be performed by an FPGA, an ASIC, or the like. Note that area segmentation may be realized using other deep learning models. In addition, the segmentation method is not limited to deep learning, and for example, graph cut, area growing, edge detection, and rule-and-divide method may be used. Furthermore, inside the auxiliary computing device 317, a neural network model may be trained using images of the affected area of the bedsore as training data.

In S443, the arithmetic unit 311 of the CPU 310 calculates the area of the affected area as information regarding the size of the extracted affected area.
FIG. 5 is a diagram for explaining a method for calculating the area of the affected region.
When the imaging device 200 is a general camera, it can be treated as a pinhole model as shown in FIG. 5. Incident light 501 passes through the lens principal point of the lens 212a and is received on the imaging surface of the image sensor 214. Here, when the lens group 212 is approximated to a single lens 212a with no thickness, the two principal points, the front principal point and the rear principal point, can be considered to be coincident. By adjusting the focal position of the lens 212a so that an image is formed on the plane of the image sensor 214, the imaging device 200 can focus on the subject 504. By changing the focal length 502, which is the distance F from the imaging surface to the lens principal point, the angle of view θ503 is changed, and the zoom magnification changes. At this time, the width 506 of the subject on the focal plane is geometrically determined from the relationship between the angle of view θ503 of the imaging device 200 and the subject distance 505. The width 506 of the subject is calculated using a trigonometric function. That is, the width 506 of the subject is determined by the relationship between the angle of view θ 503, which changes according to the focal length 502, and the subject distance 505. The width of the subject 504 is divided by the number of pixels on the line of the image data to obtain the length on the in-focus plane corresponding to one pixel on the image data.

The arithmetic unit 311 calculates the area of the affected area as the product of the number of pixels in the affected area obtained from the area extracted in S442 and the area of one pixel obtained from the length on the focal plane corresponding to one pixel on the image. The length on the focal plane corresponding to one pixel on the image corresponding to the combination of the focal length 502 and the subject distance 505 may be obtained in advance and prepared as table data. The image processing device 300 may store table data corresponding to the imaging device 200 in advance.
For the calculation device 311 to correctly calculate the area of the affected region, it is a prerequisite that the subject 504 is a plane, and that this plane is perpendicular to the optical axis.

In S444, the calculation device 311 generates image data in which information indicating the extraction result of the affected area and information regarding the size of the affected area are superimposed on the image data from which the affected area is extracted.
FIG. 6 is a diagram for explaining a method of superimposing information indicating the extraction result of the affected area and information regarding the size of the affected area on image data.
An image 601 shown in Fig. 6(a) is an example of displaying image data before superimposition processing, and includes a subject 101 and an affected area 102. An image 602 shown in Fig. 6(b) is an example of displaying image data after superimposition processing.

Label 611 is superimposed in the upper left corner of image 602 shown in FIG. 6(b), displaying text 612 indicating the area of the affected area in white text on a black background. Here, the information regarding the size of the affected area is text 612, which is the area of the affected area calculated by the calculation device 311. Note that the background color and text color of label 611 are not limited to black and white, as long as they are easy to see. In addition, the amount of transparency may be set and alpha blending may be performed so that the user can confirm the image of the part where label 611 overlaps.

In addition, an indicator 613 indicating the estimated area of the affected area extracted in S442 is superimposed on the image 602. By alpha-blending the indicator 613 indicating the estimated area with the image data on which the image 601 is based, the user can check whether the estimated area on which the area of the affected area is calculated is valid or not. It is preferable that the color of the indicator 613 indicating the estimated area is a color different from the color of the subject. It is also preferable that the transmittance range of the alpha-blending is a range in which the estimated area and the original affected area 102 can be distinguished. Note that if the indicator 613 indicating the estimated area of the affected area is superimposed and displayed, the user can check whether the estimated area is valid or not without displaying the label 611, so S443 may be omitted.

In S445, the communication device 313 of the image processing device 300 transmits information indicating the extraction result of the affected area and information regarding the size of the affected area to the imaging device 200 via wireless communication. In this embodiment, the communication device 313 transmits the image data generated in S444 and the area of the affected area, which is information regarding the size of the affected area, to the imaging device 200.

Next, the process returns to the process performed by the imaging device 200 .
In S407, the communication device 219 of the imaging device 200 receives the data transmitted from the image processing device 300, if any.
In S408, the system control circuit 220 determines whether image data and the area of the affected area, which is information about the size of the affected area, have been received. If so, the process proceeds to S409, and if not, the process proceeds to S410.

In S409, the display device 223 displays the received image data for a predetermined time. Here, the display device 223 displays the image 602 shown in FIG. 6(b). In this way, by superimposing information indicating the extraction result of the affected area on the live view image, the user can confirm whether the area of the affected area and the estimated area are appropriate before proceeding to capture for recording. Note that, although the present embodiment has been described as displaying an indicator 613 indicating the estimated area of the affected area and a label 611 displaying the area of the affected area, only one of them may be displayed, or neither may be displayed (S409 may be omitted).

S410 is a process in which the system control circuit 220 controls the timing of photographing depending on whether the photographing conditions are satisfied, and includes S411 and S412. If the affected area is tilted relative to the imaging device 200, the area of the affected area cannot be calculated correctly. Furthermore, if the degree of tilt of the affected area relative to the imaging device 200 varies each time photographing is performed, it is not possible to correctly determine how the size has changed when comparing images photographed later. Moreover, it is difficult to keep a bedridden patient in the same position all the time, and it is not easy for the photographer to position the imaging device 200 directly against the affected area. Therefore, in S411 and S412, a process is performed to estimate whether the imaging device 200 is directly facing the affected area.

In S411, the system control circuit 220 compares the area of the received affected area with a predetermined threshold value, and determines whether the area is equal to or greater than the threshold value. Here, the threshold value is recorded in advance in the external memory 222 as an initial value. The system control circuit 220 of the imaging device 200 reads the threshold value from the external memory 222 at startup. The threshold value is set to, for example, an area equivalent to the size recognized as a bedsore. However, the user can freely change and set the threshold value later. Therefore, the threshold value may be changed and set for each subject to be photographed. Note that if the area is not equal to or greater than the threshold value in S411, the process returns to S404, and the above-described processing from S404 onwards is performed.

If it is determined in S411 that the area is equal to or greater than the threshold, the process proceeds to S412, where the ratio or difference between the last area received and the area received immediately before is calculated. While the photographer is moving the imaging device 200 in an attempt to orient the imaging device 200 directly to the affected area, the last area received and the area received immediately before will change significantly. Specifically, as the imaging device 200 approaches a state in which it is tilted relative to the affected area and a state in which it is directly facing the affected area, the area of the affected area will increase. Conversely, as the imaging device 200 approaches a state in which it is tilted relative to the affected area and a state in which it is directly facing the affected area, the area of the affected area will decrease. If the imaging device 200 approaches a state in which it is directly facing the affected area, the fluctuation in the area of the affected area will be smaller. Therefore, in S412, if the ratio or difference between the last area received and the area received immediately before falls within a predetermined range, it is assumed that the imaging device 200 is facing the affected area, and the process proceeds to S413. If the area is not equal to or greater than the threshold in S412, the process returns to S404 and performs the above-described processes from S404 onward.

In S413, the system control circuit 220 issues a release signal. Here, the release signal is equivalent to the signal issued when a user presses the release button included in the operation unit 224 of the imaging device 200 to instruct shooting, and is issued in a manner that simulates the user pressing the release button. In other words, while the photographer is moving the imaging device 200 to orient it directly against the affected area, the system control circuit 220 can issue a release signal at the point in time when the imaging device 200 is close to being directly opposed to the affected area.

In S414, the distance measurement system 216 obtains information on the distance to the subject, and the AF control circuit 218 performs AF processing to control the driving of the lens group 212 so as to focus on the subject. This processing is the same as that in S403. Since the AF processing is performed in S403, the processing in S414 may be omitted.
In S415, the imaging unit 211 captures a still image for recording at the timing when the release signal is issued, and generates image data.

In S416, the image processing circuit 217 develops and compresses the generated image data to generate image data in, for example, JPEG format. The image processing circuit 217 resizes the compressed image data to reduce the size of the image data. In this case, in order to prioritize accuracy when measuring the affected area, it is preferable that the size of the resized image data is larger than or the same as the size of the image data resized in S404. For example, the size of the resized image data is approximately 4.45 megabytes in the case of 1440 pixels x 1080 pixels and 4-bit RGB color. However, the size of the resized image data is not limited to this case.

In S417, the communication device 219 acquires the resized image data and the distance information calculated in S414 by the distance measuring system 216. In addition, the communication device 219 acquires information on the zoom magnification and information on the size (number of pixels) of the resized image data, as necessary.
In S418, the communication device 219 transmits one or more pieces of information including the acquired image data and distance information to the image processing device 300 by wireless communication.

Next, the process proceeds to the image processing device 300.
In S<b>451 , the communication device 313 of the image processing device 300 receives one or more pieces of information including image data and distance information transmitted from the communication device 219 of the imaging device 200 .
In S452, the CPU 310 and the auxiliary processing unit 317 of the image processing device 300 extract the affected area from the received image data. The details of this process are similar to those of S442, and therefore will not be described here.

In S453, the arithmetic unit 311 of the CPU 310 calculates the area of the affected area as information on the size of the extracted affected area. Details of this process are similar to those of S443, and therefore will not be described.
In S454, the arithmetic unit 311 performs image analysis. Specifically, the arithmetic unit 311 calculates the length of the major axis and the minor axis of the extracted affected area, and the area of a rectangle circumscribing the affected area, based on the length on the focal plane corresponding to one pixel on the image obtained in S453. In the DESIGN-R, an evaluation index for bedsores, it is stipulated that the size of a bedsore is measured as the product of the major axis and the minor axis. In the image processing system 1 of this embodiment, by analyzing the major axis and the minor axis, it is possible to ensure compatibility with data previously measured by DESIGN-R. Since DESIGN-R does not have a strict definition, mathematically, multiple methods for calculating the major axis and the minor axis are conceivable.

As a first example of a method for calculating the major axis and the minor axis, the arithmetic device 311 calculates the rectangle with the smallest area (minimum bounding rectangle) among the rectangles circumscribing the affected area. Next, the lengths of the long and short sides of this rectangle are calculated, and the length of the long side is calculated as the major axis, and the length of the short side is calculated as the minor axis. Next, the area of the rectangle is calculated based on the length on the focal plane corresponding to one pixel on the image obtained in S453.
As a second example of the method for calculating the major axis and the minor axis, the calculation device 311 selects the maximum Feret diameter, which is the maximum caliper length, as the major axis, and selects the minimum Feret diameter as the minor axis. Alternatively, the maximum Feret diameter, which is the maximum caliper length, may be selected as the major axis, and the length measured in a direction perpendicular to the axis of the maximum Feret diameter may be selected as the minor axis.
The method for calculating the major axis and minor axis can be selected arbitrarily based on compatibility with conventional measurement results.

The process of calculating the length of the long and short axes of the affected area and the area of the rectangle is not performed on the image data received in S441. During live view, the purpose is to allow the user to check the results of extraction of the affected area, so the processing time is reduced by omitting the image analysis process equivalent to S454 for the image data received in S441.

In S455, the calculation device 311 generates image data in which information indicating the extraction result of the affected area and information regarding the size of the affected area are superimposed on the image data from which the affected area is extracted.
7A to 7C are diagrams for explaining a method of superimposing information showing the extraction result of the affected area and information about the size of the affected area including the major axis and minor axis of the affected area on the image data. Since multiple pieces of information about the size of the affected area are assumed, the following description will be given with reference to FIGS. 7A to 7C.

The image 701 shown in Fig. 7(a) uses a minimum bounding rectangle as a method for calculating the major axis and minor axis. In the upper left corner of the image 701, a label 611 is superimposed, which displays a character string 612 indicating the area of the affected area in white characters on a black background, as information regarding the size of the affected area, in the same manner as in Fig. 6(b).
In addition, a label 712 displaying the major axis and minor axis calculated based on the minimum bounding rectangle is superimposed on the upper right corner of the image 701 as information on the size of the affected area. The label 712 includes a character string 713 and a character string 714. The character string 713 indicates the length of the major axis (unit: cm), and the character string 714 indicates the length of the minor axis (unit: cm). In addition, a rectangular frame 715 representing the minimum bounding rectangle is superimposed on the affected area of the image 701. By superimposing the rectangular frame 715 together with the lengths of the major axis and minor axis, the user can confirm which part in the image is being measured.

In addition, a scale bar 716 is superimposed on the lower right corner of the image 701. The scale bar 716 is for measuring the size of the affected area 102, and the size of the scale bar for the image data is changed according to the distance information. Specifically, the scale bar 716 is a bar marked with graduations in 1 cm increments up to 5 cm based on the length on the focal plane corresponding to one pixel on the image obtained in S453, and corresponds to the size on the focal plane of the imaging device 200, i.e., on the subject. The user can grasp the size of the subject or affected area 102 by referring to the scale bar 716.

In addition, the DESIGN-R size evaluation index 717 described above is superimposed on the bottom left corner of the image 701. The DESIGN-R size evaluation index 717 measures the major axis and minor axis (the maximum axis perpendicular to the major axis) of the skin damage area (units are cm), and classifies them into the seven levels described above based on the numerical value obtained by multiplying them together. In this embodiment, the index 717 obtained by replacing the major axis and minor axis with the values output by the respective calculation methods is superimposed.

Image 702 shown in FIG. 7(b) uses the maximum Feret diameter as the major axis and the minimum Feret diameter as the minor axis. A label 722 displaying a character string 723 indicating the length of the major axis and a character string 724 indicating the length of the minor axis is superimposed in the upper right corner of image 702. In addition, an auxiliary line 725 corresponding to the measurement position of the maximum Feret diameter and an auxiliary line 726 corresponding to the minimum Feret diameter are displayed in the affected area of image 702. By superimposing the auxiliary lines 725 and 726 together with the character strings 723 and 724 indicating the lengths of the major and minor axes, the user can confirm which part in the image is being measured.

Image 703 shown in FIG. 7(c) has the same long diameter as image 702 shown in FIG. 7(b), but the short diameter is measured as the length measured in a direction perpendicular to the axis of the maximum Feret diameter, not the minimum Feret diameter. A label 732 displaying a character string 723 indicating the length of the long diameter and a character string 734 indicating the length of the short diameter is superimposed in the upper right corner of image 702. In addition, an auxiliary line 725 corresponding to the measurement position of the maximum Feret diameter and an auxiliary line 736 corresponding to the length measured in a direction perpendicular to the axis of the maximum Feret diameter are displayed in the affected area of image 703.

The various pieces of information shown in Figures 7(a) to (c) that are superimposed on the image data may be any one or a combination of multiple pieces, and the user may be able to select the information to be displayed. Also, the images shown in Figures 6 and 7 are merely examples, and the display format, display position, size, font, font size, font color, or positional relationship of the information regarding the size of the affected area 102 and the affected area can be changed according to various conditions.

In S456, the communication device 313 of the image processing device 300 transmits information indicating the extraction result of the affected area and information regarding the size of the affected area to the imaging device 200. In this embodiment, the communication device 313 transmits the image data generated in S455 to the imaging device 200.

Next, the process proceeds to the imaging device 200.
In S<b>419 , the communication device 219 of the imaging device 200 receives the image data transmitted from the image processing device 300 .
In S420, the display device 223 displays the received image data for a predetermined time. Here, the display device 223 displays one of the images 701 to 703 shown in Fig. 7, and when the predetermined time has elapsed, the process returns to S402.

In this way, according to this embodiment, the system control circuit 220 controls the timing of photographing the affected area based on the acquired information on the size of the affected area. Therefore, since the affected area can be automatically photographed when the imaging device 200 is in a position that results in an appropriate size for the affected area, the affected area can be properly photographed even when the user is in an unnatural position.

Specifically, the system control circuit 220 issues a release signal to automatically capture an image of the affected area when the size of the newly received affected area is equal to or greater than the threshold value and the change in the area of the newly received affected area and the area of the affected area received immediately before is within a predetermined range. That is, when the size of the affected area is equal to or greater than the threshold value, it can be assumed that the posture of the imaging device 200 is appropriate for capturing an image of the affected area. Also, when the change in the area of the affected area is within a predetermined range, it can be assumed that the imaging device 200 is close to facing the affected area. Therefore, by controlling the imaging device 200 to automatically capture an image of the affected area when such conditions are met, the user can capture an image of the affected area appropriately. Note that the imaging device 200 may be configured to determine whether the size of the affected area is equal to or greater than the threshold value, but only whether the change in the area of the affected area is within a predetermined range, and assume that the posture of the imaging device 200 is appropriate for capturing an image of the affected area.

Note that since the process of displaying image data in S409 described above is necessary when the user manually issues a shooting instruction, S409 may be omitted when the system control circuit 220 issues a release signal. Similarly, the process of generating superimposed image data in S444 may be omitted. Also, information regarding the size of the affected area may be transmitted in S445, and image data may not be transmitted.

In the image processing system 1 of this embodiment, the user photographs the affected area with the imaging device 200, and information regarding the size of the affected area is displayed on the display device 223 of the imaging device 200. Therefore, when evaluating the size of the affected area of a bedsore, the burden on medical personnel and the burden on the patient being evaluated can be reduced. Furthermore, by calculating the size of the affected area based on a program, individual differences can be suppressed compared to when medical personnel measure manually, and the accuracy of the size evaluation of the bedsore can be improved. Furthermore, the area of the affected area can be calculated and displayed as an index to more accurately represent the scale of the bedsore. Note that there is no need for a function for the user to check whether the estimated area of the affected area is appropriate when displaying the live view, and steps S441 to S445 may be omitted.

The image processing device 300 may also store in the storage device 312 information indicating the extraction result of the affected area, information regarding the size of the affected area, and image data on which these pieces of information are superimposed. In this case, the output device 314 can output one or more pieces of information or image data stored in the storage device 312 to an output device such as a connected display. By displaying an image on the display, a user other than the user who photographs the affected area can obtain image data and information regarding the size of the affected area obtained in real time or in the past. The arithmetic device 311 of the image processing device 300 may be provided with a function to display a scale bar or the like that arbitrarily changes the position and angle of the image data transmitted from the output device 314 to the display. By displaying such a scale bar, a user looking at the display can measure the length of any part of the affected area. It is desirable that the width of the scale of this scale bar is automatically adjusted based on the distance information, zoom magnification information, and information on the size (number of pixels) of the resized image data received in S451.

It is preferable that the image processing device 300 is a stationary type and is constantly supplied with power. By being constantly supplied with power, the image processing device 300 can receive image data and information regarding the size of the affected area at any time, and can prevent the battery from running out. Furthermore, stationary types generally have a large storage capacity, so the image processing device 300 can store a large amount of image data.
In addition, although DESIGN-R (registered trademark) is used as an evaluation index for bedsores in this embodiment, it is not limited to this. Other evaluation indexes such as Bates-Jensen Wound Assessment Tool (BWAT), Pressure Ulcer Scale for Healing (PUSH), and Pressure Sore Status Tool (PSST) may be used.

(First Modification)
In the above-mentioned flowchart of FIG. 4, if it is determined in S411 that the area is not equal to or larger than the threshold, the process of S404 and subsequent steps is repeated without proceeding to S412. However, the affected area is not necessarily equal to or larger than a certain size. Therefore, after proceeding to No in S411 of the flowchart of FIG. 4, the system control circuit 220 may determine whether or not a certain period of time has passed in a state in which the area is not equal to or larger than the threshold. If the certain period of time has not passed, the process returns to S404, and if the certain period of time has passed, the process proceeds to S412, estimating that the size of the affected area is small. Note that the certain period of time may be defined as, for example, 5 seconds, 10 seconds, etc., or may be defined by the number of times data is received in S408. In this way, by issuing a release signal when a certain period of time has passed, the affected area can be photographed even if the size of the affected area is not equal to or larger than a certain size.

(Second Modification)
In the above-mentioned flowchart of FIG. 4, the process of S410 has been described as controlling the timing of shooting depending on whether the shooting conditions are satisfied, but the process may be a process of controlling the timing of prompting the user to perform a shooting operation depending on whether the shooting conditions are satisfied. Specifically, when the process proceeds from S412 to S413, the system control circuit 220 may notify the user to prompt the user to perform a shooting operation. For example, the system control circuit 220 prompts the user to press the release button included in the operation unit 224 by notifying the user by voice or alarm, or by displaying the notification on the display device 223. Note that, according to this modified example, after S413, the system control circuit 220 determines whether the user actually pressed the release button included in the operation unit 224, and if the user did press the release button, proceeds to the process of S414 and subsequent steps.

Second Embodiment
In the first embodiment, a case where a threshold value to be compared with the area of the affected region is set in advance has been described. In the present embodiment, a case where history information is used as the threshold value will be described.
Specifically, in S411, the system control circuit 220 compares the area of the received affected area with a predetermined threshold. If the area is equal to or larger than the threshold, the process proceeds to S412. If the change in the area is within a predetermined range in S412, the process proceeds to S413. In S413, the system control circuit 220 issues a release signal and records the received area in the external memory 222. When the process proceeds to S411 in the flowchart of FIG. 4 the next time the affected area is photographed, the system control circuit 220 reads out the area recorded in the external memory 222 as the threshold. The system control circuit 220 may set a margin to the read-out threshold as the threshold. For example, the system control circuit 220 may set the threshold as a value obtained by subtracting a certain margin from the read-out threshold, or may set the threshold as a value obtained by subtracting a larger margin as the interval from the previous photographing is longer.

When recording the area, the system control circuit 220 preferably records the area to be recorded in association with patient information. For example, the imaging device 200 or the image processing device 300 can acquire patient information by a user inputting or selecting patient information into the imaging device 200 or the image processing device 300. When the image processing device 300 acquires the patient information, in S445, the image processing device 300 can associate the patient information with the area and transmit it to the imaging device 200, so that the imaging device 200 can record the area in association with the patient information.

When patient information is used, in S411 the system control circuit 220 reads out the area associated with the patient information, sets it as a threshold, and compares it with the received area. If it is equal to or greater than the threshold, the process proceeds to S412. If the change in area is within a predetermined range in S412, in S413 the system control circuit 220 issues a release signal and updates and records the received area associated with the patient information. Note that if there is no area associated with the patient information, the system control circuit 220 compares the received area with a predetermined threshold (initial value), or causes the display device 223 to display a message to the user to perform manual shooting.

The image processing device 300 may record the area in association with the patient information. For example, before the imaging device 200 images the affected area, it images a barcode tag or the like containing the patient information attached to the patient, and transmits the image data of the barcode tag to the image processing device 300 when transmitting in the above-mentioned S406. The image processing device 300 acquires the patient information from the image data of the barcode tag, reads out the area associated with the patient information as a threshold, and transmits it to the imaging device 200 when transmitting in the above-mentioned S445. Therefore, the system control circuit 220 of the imaging device 200 can set a threshold according to the patient by controlling the imaging timing based on the received threshold. Furthermore, the CPU 310 of the image processing device 300 may update the area recorded in association with the patient information to the area of the affected area calculated in S453.

Thus, according to this embodiment, a threshold is set based on the area of the affected area photographed previously, and is compared with the received area. Since the size of a bedsore usually changes over time, setting the threshold based on the area of the affected area photographed previously allows the affected area to be photographed under optimal conditions. Also, setting the threshold based on the area of the affected area photographed previously for the same patient allows the affected area to be photographed under optimal conditions for the patient. Furthermore, to improve the accuracy of the threshold, if the area of the affected area is equal to or greater than the threshold in S411, S412 may be omitted and the process may proceed to S413, where a release signal is issued.

Third Embodiment
In the first embodiment, when the change in the area of the affected area is within a predetermined range, it is determined that the imaging device 200 is facing the affected area. In the present embodiment, a case will be described in which distance information is used to determine whether the imaging device 200 is facing a subject including the affected area, and a release signal is issued if the imaging device is facing the subject.
Fig. 8 is a flowchart showing an example of processing of the image processing system 1. Processing similar to that in Fig. 4 is given the same step numbers and description thereof will be omitted as appropriate. Specifically, the flowchart in Fig. 8 is obtained by changing S410 in the flowchart in Fig. 4 to S810.
S810 is a process in which the system control circuit 220 controls the timing of capturing an image based on the attitude of the image capturing device 200 relative to the subject, and includes S811.

In S811, the system control circuit 220 determines whether the imaging device 200 is facing the subject including the affected area. As described above, the distance measurement system 216 of the imaging device 200 can calculate distance information to the subject or a distance map showing the distribution of distance information. By using distance information or distance map information of any three or more points, information as a surface can be obtained. If the distance information of the relevant points matches or the difference is within a predetermined range, it can be determined that the subject and the imaging device 200 are facing each other.
Therefore, the system control circuit 220 can determine whether the image capturing device 200 faces the subject directly, based on the distance information or distance map information calculated by the distance measuring system 216. If it is determined that the image capturing device 200 faces the subject directly, the process proceeds to S413, and if it is determined that the image capturing device 200 does not face the subject directly, the process returns to S404.

In this way, according to this embodiment, the timing of capturing an image is controlled based on the orientation of the imaging device 200 relative to the subject. Therefore, when the imaging device 200 is directly facing the subject, a release signal can be issued, allowing the affected area to be captured appropriately.

The present invention has been described above in conjunction with various embodiments and modifications. However, the present invention is not limited to the above-described embodiments and modifications. Modifications and the like are possible within the scope of the present invention, and the above-described embodiments and modifications may be combined as appropriate.
For example, the first embodiment and the third embodiment may be combined. Specifically, the system control circuit 220 may determine whether the imaging device 200 faces the subject directly based on the distance information, and when it is determined that the imaging device 200 faces the subject directly, compare the received area with a threshold value, and issue a release signal if the area is equal to or greater than the threshold value. The system control circuit 220 may also determine whether the received area is equal to or greater than a threshold value, and when it is equal to or greater than the threshold value, determine whether the imaging device 200 faces the subject directly based on the distance information, and issue a release signal if the imaging device 200 faces the subject directly.

In the above embodiment, the information regarding the size of the affected region is the area of the affected region, but this is not limited to the case, and may be at least one of the area of the affected region, the length of the major axis of the affected region, and the length of the minor axis of the affected region. If the information regarding the size of the affected region is the length of the major axis of the affected region or the length of the minor axis of the affected region, in S408, the length of the major axis of the affected region or the length of the minor axis of the affected region is obtained from the image processing device 300. In S411, the system control circuit 220 can determine whether the length of the major axis of the affected region or the length of the minor axis of the affected region is equal to or greater than a predetermined threshold.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

1: Image processing system 200: Imaging device 211: Imaging unit 216: Distance measurement system 219: Communication device 300: Image processing device 310: CPU
313: Communication device

Claims (10)

An imaging means for imaging the affected area;
a control means for acquiring information on the size of the affected area in an image captured by the imaging means, and controlling a timing for the imaging means to capture an image of the affected area based on the information on the size of the affected area, or controlling a timing for prompting a user to perform an imaging operation;
An imaging device having
The control means
When the size of the affected area is equal to or larger than a threshold value, the imaging device controls the imaging device to capture an image of the affected area, or controls the imaging device to prompt a user to perform an imaging operation;
The imaging apparatus according to claim 1, wherein the threshold value is a value based on the size of the affected area previously photographed by the same patient . The control means
The imaging device according to claim 1, characterized in that even if the size of the affected area is equal to or larger than a threshold value, as long as the rate or difference of change in the size of the affected area relative to the previously acquired size of the affected area does not fall within a predetermined range, the imaging means is controlled not to photograph the affected area, or the user is not prompted to perform an imaging operation. The control means
The imaging device according to claim 2, characterized in that, when the size of the affected area is equal to or larger than a threshold value and the rate of change or difference in the size of the affected area relative to the previously acquired size of the affected area falls within a predetermined range, the imaging device is controlled to photograph the affected area, or the imaging device is controlled to prompt a user to perform a photographing operation. The control means
The imaging device according to any one of claims 1 to 3, characterized in that even if the size of the affected area is not equal to or larger than a threshold value, when a certain period of time has elapsed, the imaging means is controlled to photograph the affected area, or the imaging means is controlled to prompt a user to perform a photographing operation. The control means
5. The imaging device according to claim 1, further comprising: a communication means for transmitting an image including the affected area captured by the imaging means to an external device; and receiving information on the size of the affected area from the external device via the communication means in response to transmitting the image to the external device. The control means
6. The imaging apparatus according to claim 5 , wherein information on the size of the affected area is received from the outside via the communication means every time the image is transmitted to the outside via the communication means. The affected area is a bedsore,
7. The imaging device according to claim 1 , wherein the size of the affected area is at least one of an area of the affected area, a length of a major axis of the affected area, and a length of a minor axis of the affected area. taking an image of the affected area by an imaging means ;
acquiring information on the size of the affected area in the image captured by the imaging means;
a control step of controlling a timing for the imaging means to image the affected area or controlling a timing for prompting a user to perform an imaging operation based on information about the size of the affected area;
A method for controlling an imaging apparatus comprising:
In the control step,
When the size of the affected area is equal to or larger than a threshold value, the imaging device controls the imaging device to capture an image of the affected area, or controls the imaging device to prompt a user to perform an imaging operation;
2. A method for controlling an imaging apparatus , comprising: setting a threshold value based on a size of the affected area of a same patient previously photographed; A program for causing a computer to function as each of the means of the device according to any one of claims 1 to 7 . A computer-readable recording medium having recorded thereon a program for causing a computer to function as each of the means of the apparatus according to any one of claims 1 to 7 .

Images