JP2021049262A - Image processing system and method for controlling the same - Google Patents

Abstract

To improve operability in measuring an affected part.SOLUTION: An image processing system includes: acquisition means for acquiring information on a filmed moving image; detection means for detecting dots in an end in a radial direction of an affected part out of the trajectory of light moving in the affected part from the information acquired by the acquisition means; and providing means for providing information on an outer periphery of the affected part from the multiple dots detected by the detection means.SELECTED DRAWING: Figure 11

Description

The present invention relates to an image processing technique for estimating the size of an affected area of a subject from an image.

In medical and long-term care settings, it is required to evaluate pressure ulcers on a regular basis for patients who develop pressure ulcers, and the size of pressure ulcers is one index for grasping the progress of pressure ulcers. Patent Document 1 discloses that in order to measure the size of a pressure ulcer pocket, a light emitting portion is inserted into the pocket, a mark is made on the skin along the contour of the pocket, or a scale is read.

International Publication No. 2006/05/71138

In the method of Patent Document 1, the operator needs a process of marking on the skin or a process of reading the scale while holding the light at a position which is the outline of the pocket. Therefore, the worker performs the work for measuring the size of the pressure ulcer while keeping the position of the light unchanged, which may increase the work load.

In view of the above problems, an object of the present invention is to improve operability when measuring an affected area (such as a pressure ulcer pocket).

The image processing system of the present invention has an acquisition means for acquiring information about a captured moving image and a radial end of the affected portion among the trajectories of lights moving in the affected portion based on the information acquired by the acquisition means. It is characterized by having a detecting means for detecting the above-mentioned points and a providing means for providing information on the outer circumference of the affected portion from a plurality of points detected by the detecting means.

According to the present invention, it is possible to improve operability when measuring an affected area (such as a pressure ulcer pocket).

It is a figure for demonstrating a pressure ulcer. It is a figure for demonstrating the measurement of a pressure ulcer pocket. It is a block diagram of the image processing system which concerns on this embodiment. It is a figure which shows the subject which concerns on this embodiment. It is a block diagram which shows the structure of the image pickup apparatus which concerns on this embodiment. It is a block diagram which shows the structure of the image processing apparatus which concerns on this embodiment. It is a flowchart which shows the operation of the image processing system which concerns on this embodiment. It is a figure for demonstrating the calculation method of the area which concerns on this embodiment. It is a figure for demonstrating the superimposition method of the information which concerns on this embodiment. It is a figure which shows the moving image which concerns on this embodiment. It is a figure for demonstrating the moving image analysis processing which concerns on this embodiment. It is a figure which shows the display during the pocket measurement work which concerns on this embodiment. It is a figure which shows the display after the pocket measurement work which concerns on this embodiment. It is a figure which shows the superimposition image which concerns on this embodiment. It is a figure which shows the subject information which concerns on this embodiment. It is a flowchart which shows the moving image analysis processing which concerns on this embodiment. It is a figure which shows the pressure ulcer photograph image containing the predetermined marker which concerns on this embodiment. It is a figure which shows the synthesis result of the light region which concerns on this embodiment. It is a flowchart which shows the outer peripheral drawing processing which concerns on this embodiment. It is a flowchart which shows the modification of the moving image analysis processing which concerns on this embodiment. It is a figure which shows the UI for deleting the unnecessary light area which concerns on this embodiment.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of the components, etc. described in the following examples are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

FIGS. 1 (A) to 1 (C) show a method (evaluation method) for measuring a pressure ulcer size. FIG. 1 (A) is an example of a method for measuring a pressure ulcer on the ulcer surface only. The size of a pressure ulcer is often determined based on a value measured manually by applying a tape measure to the affected area (ulcer surface area 103). Specifically, two points having the longest straight line distance in the skin damage range (ulcer surface area 103) are measured, and the straight line distance is defined as the major axis a of the pressure ulcer. Further, two points of the longest straight line distance orthogonal to the major axis a in the skin damage range are measured, and the straight line distance is defined as the minor axis b of the pressure ulcer. Then, the value obtained by multiplying the major axis a and the minor axis b is defined as the pressure ulcer size. Hereinafter, for other regions, the longest straight line distance in the region is described as "major diameter", and the longest straight line distance orthogonal to the major diameter in the region is described as "minor diameter".

A typical pressure ulcer symptom / classification is a pressure ulcer with a pocket. The pocket is a wound cavity wider than the skin defect (ulcer surface; the part exposed to the outside), and may be deep and wide under the skin in the part not visible from the outside (the part not exposed to the outside). 1 (B) and 1 (C) are examples of pressure ulcers with pockets. FIG. 1 (B) is an example of a pocket including the ulcer surface, that is, a pocket that extends in all directions of the ulcer surface, and FIG. 1 (C) is a pocket that partially overlaps the ulcer surface, that is, one of the ulcer surfaces. This is an example of a pocket that spreads in the direction of the part. In pressure ulcers with pockets, the entire area including the ulcer surface area 103 and the pocket area 104 becomes the affected area 102. When evaluating such pressure ulcer pockets, it is necessary to measure how wide the wound cavity (pocket) is. For example, in the pocket scoring method of DESIGN-R (registered trademark), the size of the ulcer surface, that is, the ulcer surface region 103 is obtained from the value obtained by multiplying the major axis a and the minor axis b of the affected area 102 including the ulcer surface and the pocket. It is measured by subtracting the value obtained by multiplying the major axis c and the minor axis d of.

FIG. 2 shows an outline of the measurement work of the pocket 203 by the light 201. In the measurement operation, the tip (lighting portion) of the light 201 is inserted into the pocket 203 from the ulcer surface 202. Then, the tip of the light 201 is moved toward the end of the pocket 203, and when the tip of the light 201 reaches the deepest part (the end of the pocket 203), the light emitted from the light 201 is transmitted on the skin surface. Mark the position 204 with a magic pen or the like. After that, the light 201 is pulled out from the pocket 203. The arrow 205 indicates the movement of the light 201 at this time. As shown by the arrow 205, the light 201 moves in the affected area in the radial direction from a predetermined area near the center of the affected area to the position of the edge of the affected area, and further moves to a predetermined area. Such work is repeated. The states 200A and 200a are states in which one point is marked, the states 200B and 200b are states in which four points are marked, and the states 200C and 200c are states in which marking is performed along the entire circumference of the pocket 203. Is. The shape of the outer circumference of the pocket 203 can be determined from the plurality of markings along the entire circumference, and the pocket 203 can be evaluated.

[Example 1]
In Example 1, the procedure of measuring the area of the pressure ulcer ulcer surface from the captured image and further creating a composite image capable of measuring the size of the pocket area will be described.

Hereinafter, the image processing system according to the first embodiment of the present invention will be described with reference to FIGS. 3 and 4. FIG. 3 is a block diagram showing an example of the functional configuration of the image processing system according to the first embodiment. The image processing system 1 is composed of an image pickup device 2 which is a portable device that can be held by hand, and an image processing device 3. FIG. 4 is a diagram showing a subject measured by the image processing system 1. In Example 1, an example of the pathological condition of the affected area 402 generated in the buttocks of the subject 401 is described as a pressure ulcer.

The image processing system 1 photographs the affected area 402 of the subject 401, acquires the subject distance, extracts the image area corresponding to the affected area 402, detects the outer peripheral shape of the affected area 402, and detects the major axis of the affected area 402. And measure the minor axis to measure the size of the pressure ulcer. Here, the area per pixel is measured based on the subject distance and the angle of view of the imaging device 2, and the area of the affected area 402 is measured from the extraction result of the affected area 402 and the area per pixel. Good.

A barcode tag 403 on which a one-dimensional barcode (not shown) is drawn is attached to the subject 401 as information for identifying the subject, and the image data and the subject ID can be linked. The information for identifying the subject is not limited to the one-dimensional bar code, but may be a two-dimensional code such as a QR code (registered trademark) or a numerical value. Further, it may be data or an ID number attached to an ID card such as a medical examination card.

The functional configuration of the image pickup apparatus 2 will be described. The image pickup device 2 functions as an AF unit 10, an image pickup unit 11, an image processing unit 12, an information generation unit 13, a display unit 14, an output unit 15, and a second acquisition unit 16.

The AF unit 10 has an automatic focus adjustment function that automatically focuses on the subject. Further, the AF unit 10 has a function of outputting the distance to the subject (subject distance) from the amount of movement of the focus lens.

The image pickup unit 11 takes an image of the subject and generates image data of a still image or a moving image.

The image processing unit 12 performs image processing such as development and resizing on the image acquired by the imaging unit 11.

The information generation unit 13 generates distance information regarding the distance to the subject. For example, the information generation unit 13 generates distance information based on the distance output by the AF unit 10.

The display unit 14 displays an image or the like captured by the imaging unit 11. The display unit 14 can also display information output from the image processing device 3 (information indicating the extraction result of the affected area 402, information regarding the size of the affected area 402, and the like). Such information may be superimposed and displayed on the captured image. The display unit 14 can also display a composite image output from the image processing device 3 from which the size of the pocket region can be determined. The method of creating the composite image will be described later.

The output unit 15 outputs the image data and the distance information to an external device such as the image processing device 3. The image data includes image data obtained by imaging the affected part of the subject 401, image data of the entire subject 401, image data obtained by capturing identification information such as a one-dimensional barcode drawn on the barcode tag 403, and measurement work using a light. There is video data inside.

The second acquisition unit 16 acquires evaluation information indicating the evaluation results of the image, the ulcer surface region, and the pocket region from an external device such as the image processing device 3.

Next, the functional configuration of the image processing device 3 will be described. The image processing device 3 functions as an acquisition unit 21, an extraction unit 22, a superimposition unit 23, an analysis unit 24, a second output unit 25, and a storage unit 26.

The acquisition unit 21 acquires the image data output by the image pickup apparatus 2 and the distance information (subject distance).

The extraction unit 22 extracts an image region corresponding to the affected area 402 from the image (image data output by the imaging device 2) obtained by capturing the affected area 402. Extracting an area from an image is called area extraction or area division.

The analysis unit 24 analyzes the information regarding the size of the affected area 402 extracted by the extraction unit 22 based on the distance information (subject distance) generated by the information generation unit 13. In addition, the analysis unit 24 analyzes the moving image during the measurement operation by the light in order to create a composite image showing the size of the pocket region.

The superimposing unit 23 superimposes information indicating the extraction result of the affected area 402, information on the size of the affected area 402, and the like on the image corresponding to the image data used when extracting the affected area 402.

The second output unit 25 includes information indicating the affected area 402 extracted by the extraction unit 22, information regarding the size of the affected area 402 analyzed by the analysis unit 24, and image data (information is superimposed) obtained by the overlay unit 23. The image) and the like are output to an external device such as the image pickup device 2. The second output unit 25 can also output a composite image showing the size of the pocket area to an external device.

The reading unit 30 reads the one-dimensional barcode (not shown) drawn on the barcode tag 403 from the image obtained by capturing the barcode tag 403, and acquires the identification information (for example, the subject ID) for identifying the subject 401. To do. The target to be read by the reading unit 30 may be a two-dimensional code such as a QR code (registered trademark), a numerical value, or a character.

The recognition processing unit 31 collates the subject ID (identification information) read by the reading unit 30 with the subject ID registered in advance, and acquires the name of the subject 401.

The storage unit 26 includes an image of the affected area 402 (affected area image), information on the size of the affected area 402, a subject ID (identification information) of the subject 401, a name of the subject 401, a shooting date and time of the affected area image, and the like. A record based on the above is generated and recorded in the image processing device 3.

Next, an example of the hardware configuration of the image pickup apparatus 2 is shown in FIG. The imaging device 2 includes an AF control unit 225, an imaging unit 211, a zoom control unit 215, a distance measuring system 216, an image processing unit 217, a communication unit 218, a system control unit 219, a storage unit 220, an external memory 221 and a display unit 222. It is a camera having an operation unit 223 and a common bus 224.

The AF control unit 225 extracts a high-frequency component of the imaging signal (video signal), searches for the lens position (position of the focus lens included in the lens 212) at which this is maximized, controls the focus lens, and automatically controls the focus lens. Adjust the focus. This focus control method is also called TV-AF or contrast AF, and has a feature that high-precision focusing can be obtained. Further, the AF control unit 225 acquires and outputs the distance to the subject based on the amount of focus adjustment or the amount of movement of the focus lens. The focus control method is not limited to the contrast AF, and may be a phase difference AF or another AF method. The AF unit 10 in FIG. 3 is realized by operating the AF control unit 225.

The imaging unit 211 includes a lens 212, a shutter 213, and an image sensor 214. The image pickup unit 11 of the image pickup apparatus 2 of FIG. 3 is realized by operating the image pickup unit 211. The lens 212 forms an optical image of the subject on the image sensor 214. The image sensor 214 is composed of a charge storage type solid-state image sensor such as a CCD or a CMOS element that converts an optical image into an electric signal. The image pickup unit 211 includes a diaphragm in the lens 212 that determines a diaphragm value for adjusting the exposure amount. The shutter 213 controls the shutter speed by exposing or shading the image sensor 214 by opening and closing. The shutter is not limited to the mechanical shutter, and an electronic shutter may be used. In an image sensor using a CMOS sensor, the electronic shutter performs a reset scan to make the accumulated charge amount of the pixels zero for each pixel or for each region composed of a plurality of pixels (for example, for each line). After that, a scan for reading a signal is performed after a predetermined time has elapsed for each pixel or region for which the reset scan has been performed.

The zoom control unit 215 controls the drive of the zoom lens included in the lens 212. The zoom control unit 215 drives the zoom lens via a zoom motor (not shown) according to an instruction from the system control unit 219. This causes scaling.

The distance measuring system 216 is a unit that acquires the distance to the subject. The distance measuring system 216 may generate distance information based on the output of the AF control unit 225. When a plurality of blocks each consisting of one or more pixels in the screen (display surface) of the display unit 222 are set, the distance measuring system 216 repeatedly operates the AF block by block to block each block. Detect the distance of. As the distance measuring system 216, a system using a TOF (Time Of Flight) sensor may be used. The TOF sensor is a sensor that measures the distance to the object based on the time difference (or phase difference) between the transmission timing of the irradiation wave and the reception timing of the reflected wave that is the wave reflected by the object. is there. Further, the ranging system 216 may use a PSD method or the like using a PSD (Position Sensitive Device) as the light receiving element. The information generation unit 13 of the image pickup apparatus 2 of FIG. 3 is realized by operating the distance measuring system 216.

The image processing unit 217 performs image processing on the RAW image data output from the image sensor 214. The image processing unit 217 performs white balance adjustment, gamma correction, color interpolation or demosaiking, and filtering on the image (RAW image data) output from the imaging unit 211 or the image recorded in the storage unit 220 described later. , Etc., perform various image processing. Further, the image processing unit 217 performs a compression process on the image captured by the image capturing unit 211 according to a standard such as JPEG. The image processing unit 12 of the image pickup apparatus 2 of FIG. 3 is realized by operating the image processing unit 217.

The communication unit 218 is a communication interface for each configuration in the image pickup device 2 to communicate with an external device such as an image processing device 3 via a wireless network (not shown). The output unit 15 and the second acquisition unit 16 of the image pickup apparatus 2 of FIG. 3 are realized by the operation of the communication unit 218. A specific example of a network is a network based on the Wi-Fi® standard. Communication using Wi-Fi (registered trademark) may be realized via a router. Further, the communication unit 218 may be realized by a wired communication interface such as USB or LAN.

The system control unit 219 has a CPU (Central Processing Unit) and controls each unit of the image pickup apparatus 2 according to a program recorded (stored) in the storage unit 220 (overall control). For example, the system control unit 219 controls the AF control unit 225, the image pickup unit 211, the zoom control unit 215, the distance measurement system 216, the image processing unit 217, and the like.

The storage unit 220 includes various setting information (information on the focus position at the time of imaging, etc.) necessary for the operation of the imaging device 2 and various images (images captured by the imaging unit 211, images processed by the image processing unit 217, etc.). Etc. are temporarily memorized. The storage unit 220 may temporarily store image data, analysis data (information on the size of the subject, etc.) received by the communication unit 218 in communication with the image processing device 3. The storage unit 220 is composed of a rewritable non-volatile memory such as a flash memory or SDRAM.

The external memory 221 is a non-volatile storage medium that can be loaded into the main body of the image pickup apparatus 2 or is fixed inside, and is, for example, an SD card, a CF card, or the like. The external memory 221 stores image data processed by the image processing unit 217, image data and analysis data received by the communication unit 218 communicating with the image processing device 3. It is also possible to read out the image data, analysis data, etc. recorded in the external memory 221 and output them to the outside of the image pickup apparatus 2.

The display unit 222 displays an image temporarily stored in the storage unit 220, an image or information stored in the external memory 221, a setting screen of the image pickup device 2, and the like. The display unit 222 is a TFT (Thin Film Transistor) liquid crystal display, an organic EL display, an EVF (electronic viewfinder), or the like. The display unit 14 of the image pickup apparatus 2 of FIG. 3 is realized by operating the display unit 222.

The operation unit 223 is a reception unit that accepts user operations, and includes, for example, buttons, switches, keys, and a mode dial provided in the image pickup apparatus 2. The operation unit 223 may include a touch panel or the like that is also used as the display unit 222. Commands such as various mode settings and shooting operations by the user are transmitted to the system control unit 219 via the operation unit 223.

The common bus 224 includes the AF control unit 225, the imaging unit 211, the zoom control unit 215, the distance measuring system 216, the image processing unit 217, the communication unit 218, the system control unit 219, the storage unit 220, the external memory 221 and the display. The unit 222 and the operation unit 223 are connected. The common bus 224 is a signal line for transmitting and receiving signals between each block.

Next, FIG. 6 shows an example of the hardware configuration of the information processing device which is the image processing device 3. The image processing device 3 is a computer having a central processing unit (CPU) 310, a storage unit 312, an input unit 313 (mouse, keyboard, etc.), an output unit 314 (display, etc.), and an auxiliary calculation unit 317. The CPU 310 includes a calculation unit 311. The storage unit 312 is composed of a main storage unit 315 (ROM, RAM, etc.) and an auxiliary storage unit 316 (magnetic disk, SSD (Solid State Drive), etc.). A part of the input unit 313 and the output unit 314 is configured as a wireless communication module for Wi-fi communication.

The auxiliary calculation unit 317 is an auxiliary calculation IC used under the control of the CPU 310. As an example, a GPU (Graphic Processing Unit) can be used for the auxiliary calculation unit 317. The GPU is a processor for image processing, but since it has a plurality of product-sum arithmetic units and is good at matrix calculation, it is often used as a processor for processing for signal learning. The GPU is also generally used in the process of performing deep learning. As the auxiliary calculation unit 317, FPGA (field-programmable gate array), ASIC, or the like may be used.

The calculation unit 311 included in the CPU 310 executes a program recorded (stored) in the storage unit 312 to execute the acquisition unit 21, the extraction unit 22, the superimposition unit 23, the analysis unit 24, and the second image processing device 3 of FIG. It functions as an output unit 25, a storage unit 26, a reading unit 30, and a recognition processing unit 31. Further, the calculation unit 311 controls the order in which the processes are executed.

The CPU 310 and the storage unit 312 included in the image processing device 3 may be one or a plurality. That is, when at least one or more processing units (CPU) and at least one storage unit are connected and at least one or more processing units execute a program recorded in at least one storage unit, image processing is performed. The device 3 may function as each of the above parts. The processing unit is not limited to the CPU, and may be an FPGA, an ASIC, or the like.

Next, the operation of the image processing system 1 according to the first embodiment will be described with reference to the flowchart of FIG. 7. In the flowchart of FIG. 7, the processing of the image pickup apparatus 2 is performed by expanding the program recorded in the ROM (a part of the storage unit 220) into the RAM (a part of the storage unit 220) and executing the processing by the system control unit 219. Realize. Similarly, the processing of the image processing apparatus 3 is realized by expanding the program recorded in the ROM (a part of the main storage unit 315) into the RAM (a part of the main storage unit 315) and executing the program by the CPU 310. In the flowchart of FIG. 7, as an evaluation of pressure ulcer on the ulcer surface, one of the image data (1 frame) taken as a moving image is analyzed and the size of the ulcer surface is measured. In addition, the image processing device 3 creates a composite image so that the size of the pocket can be seen, and transmits the composite image to the image pickup device 2. The process of FIG. 7 starts when the power of the image pickup device 2 and the image processing device 3 is turned on and an operation for connecting the image pickup device 2 and the image processing device 3 to each other is performed.

In step S701 and step S721, the image pickup apparatus 2 and the image processing apparatus 3 perform a connection process for connecting to each other so as to be able to communicate with each other. For example, the system control unit 219 of the image pickup apparatus 2 uses the communication unit 218 to connect to a Wi-fi standard network (not shown), which is a wireless LAN standard. The CPU 310 of the image processing device 3 is also connected to the same network by using the input unit 313 and the output unit 314. Then, in step S721, the CPU 310 performs a search process for the connected imaging device, and in step S701, the system control unit 219 performs a response process to the search process. For the search process, various device search techniques for searching (searching) devices via a network can be used. For example, the search process used by UPnP (Universal Plug and Play) is performed, and each device is identified using a UUID (Universally Unique Identifier).

In step S702, the system control unit 219 of the image pickup apparatus 2 uses the image pickup unit 211 to photograph the barcode tag 403 of the subject 401. The barcode tag 403 includes a subject ID (patient ID) that identifies the subject 401 (patient). By taking a picture of the affected area after taking a picture of the barcode tag, the shooting order is managed according to the shooting date and time, and the image from the bar code tag image to the image before the next bar code tag image is the image of the same subject. As a result, it becomes possible to identify by the subject ID.

After that, the system control unit 219 of the image pickup apparatus 2 uses the image pickup unit 211 and the display unit 222 to perform a live view process of displaying the live image of the subject 401 on the display unit 222. In the live view process, the image pickup apparatus 2 performs the processes of steps S703 to S710. Along with the live view processing, the image processing device 3 performs the processing of steps S722 to S726.

In step S703, the system control unit 219 of the image pickup apparatus 2 uses the AF control unit 225 to adjust the focus so as to match the subject 401 (AF process). Here, in the AF process, it is assumed that the screen of the display unit 222 is divided into a plurality of blocks and AF is performed in a predetermined block. Specifically, the image pickup device 2 is held so that the affected area 402 is arranged in the center of the screen, and AF is performed by the block located in the center of the screen. Further, the AF control unit 225 outputs the distance from the amount of focus adjustment or the amount of movement of the focus lens to the AF area (the portion focused by AF) of the subject 401, and the system control unit 219 acquires the distance. To do.

In step S704, the system control unit 219 of the image pickup apparatus 2 uses the image pickup unit 211 to image the affected area 402 of the subject 401.

In step S705, the system control unit 219 of the image pickup apparatus 2 uses the image processing unit 217 to develop the image acquired in step S704, compress the developed image with a standard such as JPEG, and obtain the JPEG. Perform image resizing. The image generated in step S705 is transmitted to the image processing device 3 by wireless communication in step S707 described later. The larger the size of the image to be transmitted, the longer it takes for wireless communication. Therefore, the image size after resizing is selected in consideration of the allowable communication time. Further, the image generated in step S705 is subject to the extraction process of extracting the affected area 402 from the image in step S723 described later. Since the image size after resizing depends on the processing time of the extraction process and the extraction accuracy, these requirements are also taken into consideration when selecting the image. Further, since step S705 is a part of the live view processing, if the processing time of step S705 is long, the frame rate of the live video becomes low, which adversely affects usability. For this reason, it is desirable to perform resizing with the same or smaller image size as compared with image processing (resizing) at the time of actual shooting during live view processing. In step S705, it is assumed that 720 pixels × 540 pixels and 8-bit RGB color are resized so that the data size is about 1.1 megabytes. The image size, data size, bit depth, color space, etc. after resizing are not particularly limited.

In step S706, the system control unit 219 of the image pickup apparatus 2 uses the distance measurement system 216 to generate distance information regarding the distance to the subject. Specifically, the distance information is generated based on the distance output by the AF control unit 225 in step S703.

In step S707, the system control unit 219 of the image pickup apparatus 2 uses the communication unit 218 to transmit (output) the image (image data) generated in step S705 and the distance information generated in step S706 to the image processing apparatus 3. ). At the time of initial transmission, the system control unit 219 transmits the tag information image captured in step S702 to the image processing device 3 only once.

In step S722, the CPU 310 of the image processing device 3 uses the input unit 313 to transmit the image (image of the affected area 402) and the distance information (subject in the image (affected area 402)) transmitted by the image pickup device 2 in step S707. ) Corresponding distance information) is received (acquired). At the time of the first reception, the CPU 310 receives the tag information image captured in step S703 only once.

In step S723, the CPU 310 of the image processing device 3 extracts the affected area 402 of the subject 401 from the image acquired in step S722. Here, region division (region extraction) is performed only on the ulcer surface that can be extracted by image analysis. As a method of domain division, it is assumed that semantic domain division by deep learning is performed. That is, a trained model is generated by training a neural network model in advance using a plurality of actual images of pressure ulcer affected areas as teacher data on a learning computer (not shown). Then, the CPU 310 estimates the pressure ulcer area from the input image based on the generated trained model. Further, it is assumed that a complete convolutional network FCN (Full Convolutional Network), which is a segmentation model using deep learning, is applied as a model of a neural network. Deep learning inference is performed using a GPU (included in the auxiliary calculation unit 317), which is good at parallel execution of multiply-accumulate operations. The inference process may be executed by FPGA, ASIC, or the like. It should be noted that the region division may be realized by using another deep learning model. Further, the segmentation method is not limited to deep learning, and for example, graph cut, region growth, edge detection, governing division method, or the like may be used.

In step S724, the CPU 310 of the image processing apparatus 3 converts the image size (size on the image) of the ulcer surface region extracted in step S723 to obtain information on the actual size (actual size) of the ulcer surface region. Analyze (acquire). The image size of the ulcer surface region is converted into an actual size based on the information regarding the angle of view or the pixel size of the image acquired in step S722 and the distance information acquired in step S722.

A method of calculating the area (actual size) of the ulcer surface region will be described with reference to FIG. A general camera can be treated as a pinhole model as shown in FIG. The incident light 800 passes through the principal point of the lens 212 and is incident on the imaging surface of the image sensor 214. The distance from the imaging surface to the principal point of the lens is the focal length F. Here, when the thin lens approximation is used, it can be considered that the two principal points, the front principal point and the rear principal point, coincide with each other. Further, in the pinhole model, the lens 212 is represented by a single lens having no thickness, but the actual lens is composed of, for example, a plurality of thick lenses including a focus lens or a zoom lens. Then, by adjusting the focus lens of the lens 212 so that an image is formed on the image pickup surface of the image sensor 214, the focus is adjusted so as to match the subject 801. Further, in the case of a zoom lens, the angle of view θ changes by changing the focal length F. At this time, the width W of the subject 801 on the focal plane is geometrically determined from the relationship between the angle of view θ of the image pickup apparatus 2 and the subject distance D, and the width W of the subject 801 can be calculated using a trigonometric function. That is, the width W of the subject 801 is determined by the relationship between the angle of view θ with the focus position and the zoom amount as parameters and the subject distance D. Then, by dividing the width W of the subject 801 by the number of pixels in one line of the image sensor 214, the length on the focal plane corresponding to one pixel of the image can be obtained. Further, from the length on the focal plane corresponding to 1 pixel, the area on the focal surface corresponding to 1 pixel can be obtained. The area of the ulcer surface region can be calculated by multiplying the number of pixels of the ulcer surface region extracted in step S723 by the area on the focal surface corresponding to one pixel.

In step S725, the CPU 310 of the image processing device 3 superimposes information (processing result of step S724) regarding the area (actual size) of the ulcer surface region on the image acquired in step S722. Information on the extraction result of the ulcer surface region may be superimposed.

A state in which information on the area (actual size) of the ulcer surface region is superimposed will be described with reference to FIG. Image 910 of FIG. 9 is an image before the superimposition treatment, and includes the ulcer surface region (affected portion region 402) of the subject 401. Then, the image 913 is an image after the superimposition processing, and the label 911 in which the white character string 912 indicating the estimated area value is described above on the black background is superposed on the upper left corner of the image 913. As the information regarding the extraction result of the ulcer surface region, for example, a frame indicating the ulcer surface region is superimposed.

In step S726, the CPU 310 of the image processing device 3 transmits (outputs) information regarding the actual size of the ulcer surface region (processing result of step S724) to the imaging device 2 using the output unit 314. Specifically, the CPU 310 outputs the image after the superimposition processing in step S725 (superimposition processed image) to the image pickup apparatus 2 by wireless communication. Information regarding the extraction result of the ulcer surface region may be transmitted.

In step S708, the system control unit 219 of the image pickup apparatus 2 uses the communication unit 218 to receive (acquire) the information (superimposed processed image) transmitted by the image processing apparatus 3 in step S726.

In step S709, the system control unit 219 of the image pickup apparatus 2 displays the information (superimposed image) received in step S708 on the display unit 222. As a result, the live view image captured by the imaging unit 211 is displayed, and information on the actual size of the ulcer surface region is superimposed and displayed on the live view image. It is sufficient that at least one of the information on the extraction result of the ulcer surface region and the information on the actual size of the ulcer surface region is superimposed and displayed on the live view image, and the information is transmitted from the image processing device 3 to the image pickup device 2 for imaging. The superimposition processing may be performed by the device 2.

In step S710, the system control unit 219 of the image pickup apparatus 2 determines whether or not there is a main shooting operation (an operation for instructing the main shooting) on the operation unit 223. If there is a main shooting operation, the live view process is exited and the process proceeds to step S711. If not, the process returns to step S703 and the live view process is repeated.

In step S711, the system control unit 219 of the image pickup apparatus 2 determines whether or not the pressure ulcer to be imaged has a pocket, specifically, whether or not pocket evaluation by a light as described with reference to FIG. 2 is necessary. To do. Whether or not there is a pocket (whether or not pocket evaluation by a light is necessary) may be specified by the user (evaluator) using the operation unit 223, or the system control unit 219 analyzes the live view image. You may judge. If there is a pocket (if pocket evaluation by light is required), the process proceeds to step S712, and if not, the process proceeds to step S713.

In step S712, the system control unit 219 of the image pickup apparatus 2 uses the image pickup unit 211 to take a moving image of the measurement operation (FIG. 2) by the light. Further, the system control unit 219 also captures a still image (for example, a still image before the light is inserted into the pocket in the measurement operation by the light). In the first embodiment, since the pocket shape is detected by image analysis of the movement path of the light, marking using a magic pen or the like is omitted. Here, FIG. 10 schematically shows each frame of the moving image that can be acquired in step S712. In FIG. 10, a plurality of frames are arranged in chronological order, and the first frame 1000 shows the ulcer surface 1001 of the pressure ulcer and the light 1002 emitted from the light. Further, the position of the light 1002 moves as time advances with the frames 1003, 1004, and 1005.

In step S713, the system control unit 219 of the image pickup apparatus 2 uses the image pickup unit 211 to take a still image for evaluating a pressure ulcer without a pocket. Specifically, AF processing similar to step S703, imaging similar to step S704, and image processing (development, resizing, etc.) similar to step S705 are performed. Step S713 is not a part of the live view process but a process of main shooting. Therefore, in step S713, the large image size and the accuracy of the pressure ulcer size measurement are prioritized over the short processing time, and the image is resized to an image size equal to or larger than the image obtained in step S705. Here, it is assumed that resizing is performed so that the data size is approximately 4.45 megabytes with 1440 pixels × 1080 pixels and 4-bit RGB color. The image size, data size, bit depth, color space, etc. after resizing are not particularly limited.

In step S714, the system control unit 219 of the image pickup apparatus 2 uses the communication unit 218 to obtain image data of the image (the moving image and still image taken in step S712 or the still image taken in step S713) obtained in the main shooting. It is transmitted (output) to the image processing device 3. The system control unit 219 also transmits the distance information (subject distance) generated in step S706 to the image processing device 3. The distance information at the time of the main shooting may be generated again, and the distance information at the time of the main shooting may be transmitted to the image processing device 3.

In step S727, the CPU 310 of the image processing device 3 uses the input unit 313 to receive (acquire) the image and the distance information transmitted by the image pickup device 2 in step S714.

In steps S728 to S730, the CPU 310 of the image processing apparatus 3 measures the size of the pressure ulcer ulcer surface. In step S728, similarly to step S723, the CPU 310 of the image processing device 3 extracts the ulcer surface region of the subject 401 from the image (still image) acquired in step S727. When a moving image is acquired, one frame of the moving image (for example, one frame before the light is inserted into the pocket in the measurement work with the light) may be selected and the ulcer surface region may be extracted from the selected frame. Good.

In step S729, similarly to step S724, the CPU 310 of the image processing device 3 analyzes (acquires) information on the actual size of the ulcer surface region extracted in step S728 based on the distance information acquired in step S727.

In step S730, the CPU 310 of the image processing device 3 evaluates the ulcer surface using the image (still image) acquired in step S727. When the moving image taken in step S712 is acquired, one frame of a plurality of frames of the moving image (for example, one frame before the light is inserted into the pocket in the measurement work by the light) may be selected and used. ..

The evaluation of the ulcer surface will be specifically described. The CPU 310 of the image processing device 3 analyzes the information regarding the actual size of the ulcer surface region extracted in step S728 based on the distance information acquired in step S727, and calculates the major axis and the minor axis of the ulcer surface region and the rectangular area. To do. In the pressure ulcer evaluation index "DESIGN-R (registered trademark)", it is stipulated that the size of a pressure ulcer is evaluated by the product of the major axis and the minor axis. In the image processing system 1 according to the first embodiment, by analyzing the major axis and the minor axis, it is possible to obtain an evaluation result having compatibility with the evaluation result according to "DESIGN-R (registered trademark)". Since "DESIGN-R (registered trademark)" does not have a strict definition, mathematically, a plurality of calculation methods for major axis and minor axis can be considered. For example, among the rectangles circumscribing the ulcer surface region, the rectangle having the smallest area (Minimum bounding rectangle) is calculated, the lengths of the long side and the short side of the Minimum bounding rectangle are calculated, and the length of the long side is the major axis. The length of the short side can be calculated as the short diameter. The maximum ferret diameter, which is the maximum caliper length as the major diameter, may be selected as the minor diameter, and the length measured in the direction orthogonal to the axis of the maximum ferret diameter may be selected. Any method can be selected for the calculation method of the major axis and the minor axis based on the compatibility with the conventional measurement results. The ulcer surface region is not evaluated during the live view process. During the live view processing, it is sufficient if the extraction result of the affected area 402 (ulcer surface area) can be confirmed. By omitting the evaluation of the ulcer surface area, the processing time of the image analysis can be reduced and the frame rate of the live view can be increased. The convenience of the image pickup apparatus 2 can be improved.

The process of step S731 is performed when the moving image (moving image taken in step S712) is acquired in step S727. In step S731, the CPU 310 of the image processing device 3 analyzes the acquired moving image (image) in order to create a composite image showing the size of the pressure ulcer pocket, and acquires various information about the moving image (image). .. Specifically, the information about the locus of movement of the light is acquired. The method of acquiring information related to moving images is not particularly limited, and for example, the image processing device 3 may acquire information from the outside.

The moving image analysis process of step S731 executed by the image processing device 3 will be described with reference to FIG. FIG. 11 corresponds to the positions of the pocket 1100, the ulcer surface 1101 which is the entrance of the pocket 1100, the path 1102 of the light tip, and the position of the light tip when reaching the deepest part (end) of the pocket 1100, as in FIG. Point 1103 is shown. As the pocket 1100, the surface that cannot be seen in the subcutaneous part is conceptually illustrated. The points on the path 1102 indicate a plurality of positions of the light tips corresponding to the plurality of timings.

In the moving image analysis process of step S731, the CPU 310 detects the position of the tip of the light (point 1103) when it reaches the deepest part of the pocket from the acquired information. This point (position) can also be said to be "a point at the radial end of the affected area in the trajectory of the light moving in the affected area" or "the position of the boundary between the affected area and a different area from the affected area". For example, in moving images, the apex (the point where the light changes from being inserted into the pocket to being pulled out) when the light moves in the affected area in the radial direction can be detected as an end point. On the upper side of FIG. 11, the outline of the pocket measurement work is shown in four stages in a time sequence, and three points 1103 are detected in each stage. At the bottom of FIG. 11, all the detected points 1103 (12 points 1103) are shown. In the moving image analysis process, information on the outer circumference of the affected area is acquired from these points 1103. Specifically, the line 1104 that combines (connects) these points 1103, for example, a smooth free curve that connects with a spline curve or a Bezier curve, is determined (estimated) as the outer circumference of the pocket. Then, the pocket shape is determined by the shape analysis of the obtained line 1104. Information on the outer circumference of the affected area (shape of the outer circumference of the affected area, area of the affected area, major axis of the affected area, minor axis of the affected area, etc.) can be provided to the user by display or the like, or can be provided to other devices as data.

FIG. 12 is an example of the live view display during the pocket measurement work by the light, and shows the detected marking position (the position of the tip of the light when the deepest part of the pocket is reached) and the pocket generated (formed) from the marking position. Display the shape.

The screen 1201 is a live view display screen when the tip 1203 of the light 1202 reaches the deepest part of the pocket. From the screen 1201, it can be seen that the tip 1203 of the light 1202 is shining inside the pocket. The position 1204 is a marking position obtained by analyzing the movement of the light 1202 in the moving image captured by the live view, and four marking positions 1204 are displayed on the screen 1201. Line 1205 shows a line (a part of the pocket shape) detected by analyzing the marking positions 1204 at these four points.

The screen 1211 is a live view display screen when the tip 1203 of the light 1202 is slightly pulled out from the deepest part of the pocket after the state of the screen 1201. At this time, the position of the tip 1203 of the light 1202 on the screen 1201 is newly obtained as the marking position 1204 by the moving image analysis. In this way, by immediately displaying the marking position 1204 newly obtained by the video analysis on the live view screen, the pocket measurement worker confirms the shape around the pocket and whether the pocket measurement work is performed correctly. You can proceed with the work. When the marking position 1204 is newly displayed in the moving image analysis, the marking position 1204 may be blinked or made a sound to notify that the new marking position 1204 has been added. By displaying the marking position 1204 and pocket shape 1205 obtained by video analysis during the pocket measurement work by the light 1202 in live view, it is possible to add a desired marking position or delete an apparently incorrect marking position. it can.

FIG. 13 is an example of a live view display after the pocket measurement work by the light is completed, and displays the detected marking position and the pocket shape generated from the marking position. Further, the marking position can be additionally displayed by the editing operation.

The screen 1301 is a screen for displaying the live view at the end of the pocket measurement work (immediately after the end of the pocket measurement work). The marking position 1204 and the pocket shape 1205 obtained by the moving image analysis are displayed. Further, adjacent to the screen 1301, a marking position editing menu 1302 for editing the marking position is displayed. The marking position edit menu 1302 includes a plurality of items 1303, and the user can select any one of the plurality of items 1303. Here, it is assumed that the plurality of items 1303 include three items of "addition", "move", and "delete". On screen 1301, "Add" is selected.

With "Add" selected, the user can add any position as a marking position (not the position obtained by video analysis). The screen 1311 is a screen of the live view display at the time when the user selects "add" and specifies the marking position 1312 to be added. As shown on the screen 1311, when the user specifies the marking position 1312, the marking position 1312 is additionally displayed. Further, the pocket shape 1205 is updated to the shape obtained by analyzing the plurality of marking positions after the addition.

When "Move" is selected, the user can move the selected marking position by arbitrarily selecting and dragging and dropping the marking position on the screen. In this case as well, the pocket shape 1205 is updated to the shape obtained by analyzing the marking position after movement. In addition, when "Delete" is selected, the user can delete the specified marking position by arbitrarily designating (selecting) the marking position on the screen. In this case as well, the pocket shape 1205 is updated to the shape obtained by analyzing the remaining marking positions after deletion. In this way, the pocket shape 1205 is updated to a shape connecting the changed marking positions according to the operation.

Returning to the description of FIG. In step S732, the CPU 310 of the image processing device 3 superimposes information indicating the extraction result of the affected area and information on the size of the affected area on the image (still image) acquired in step S727. In the case of a pressure ulcer with a pocket, not only the information superimposed in step S725 but also the result of the moving image analysis in step S731 are superimposed. When the moving image is acquired in step S727 (in the case of a pressure ulcer with a pocket), one frame of the moving image (for example, one frame before the light is inserted into the pocket in the measurement work by the light) is selected, and the 1 frame is selected. Information may be superimposed on the frame.

The superimposition process of step S732 will be described with reference to FIGS. 14 (A) to 14 (C). Here, it is assumed that information including a major axis and a minor axis is superimposed as information indicating the extraction result of the affected area. Further, it is assumed that information on the marking position around the pocket, the shape of the pocket, and the size of the affected area is superimposed. 14 (A) to 14 (C) show an example of a superposed image (composite image) obtained by the superimposing process in step S732.

FIG. 14 (A) shows an example of a superimposed image in the case of a pressure ulcer without a pocket. In FIG. 14A, a label 1401 in which a white character string 1402 indicating the size (area value) of the ulcer surface region is described above on a black background is superimposed on the upper left corner of the superimposed image 1400. A label 1403 having a white character string 1404 indicating the major axis of the ulcer surface region and a white character string 1405 indicating the minor axis of the ulcer surface region described above on a black background is superimposed on the upper right corner of the superimposed image 1400. ing. A label 1406 in which a white character string indicating an index of the Size evaluation of "DESIGN-R (registered trademark)" is described above on a black background is superimposed on the lower left corner of the superimposed image 1400. A scale bar 1407 is superimposed on the lower right corner of the superimposed image 1400.

FIG. 14B shows an example of a superimposed image in the case of a pressure ulcer with a pocket. In the superimposed image 1410 of FIG. 14 (B), as in FIG. 14 (A), the label 1403 indicating the major axis and the minor axis of the ulcer surface region and the label indicating the index of the size evaluation of "DESIGN-R (registered trademark)" are shown. 1406 and scale bar 1407 are superimposed. However, the superimposed image 141 of FIG. 14 (B)
At 0, the label 1411 is superimposed instead of the label 1401 in FIG. 14 (A). Label 1411 describes not only the character string 1402 indicating the area value of the ulcer surface region but also the character string 1412 indicating the area value of the pocket. The area value of the pocket is also calculated based on the subject distance and the like, like the area value of the ulcer surface area. Further, in the superimposed image 1410 of FIG. 14B, the pocket region 1413 and the ulcer surface region 1414 are filled with different colors. By color-coding in this way, the pocket region 1413 and the ulcer surface region 1414 can be visually and accurately distinguished.

In the case of decubitus without pockets (in the case of FIG. 14A), a character indicating that no pocket exists instead of the character string indicating the area value of the pocket (character string 1412 in FIG. 14B). Columns (“Pocket 0”, “No Pocket”, etc.) may be superimposed. In addition, the pocket region and the ulcer surface region may be visually and accurately discriminated by superimposing frames (lines) indicating the contours of the regions. In the case of pressure ulcers without pockets (in the case of FIG. 14A), the ulcer surface region may be filled or a frame showing the outline of the ulcer surface region may be superimposed. Further, the display of only the pocket region, the display of only the ulcer surface region, and the display of both the pocket region and the ulcer surface region may be switched by operating the imaging device. By doing so, it is possible to confirm the image by focusing on the area of only one of the pocket and the ulcer surface.

FIG. 14C shows another example of a superimposed image in the case of a pressure ulcer with a pocket. In the superimposed image 1420 of FIG. 14C, the label 1411, the label 1403, the label 1406, and the scale bar 1407 are superimposed as in the case of FIG. 14B. In FIG. 14C, the pocket area and the ulcer surface area are not filled, and a plurality of points 1421 indicating a plurality of marking positions around the pocket and a line 1422 indicating the pocket shape are superimposed. Here, it is assumed that the major axis and the minor axis are calculated using the Minimum bounding rectangle. In the superimposed image 1420 of FIG. 14C, a rectangular frame 1423 representing a Minimum bounding rectangle surrounding the ulcer surface region 1414 is superimposed. In the case of pressure ulcers without pockets (in the case of FIG. 14A), a rectangular frame representing a Minimum bounding rectangle may be superimposed.

Returning to the description of FIG. In step S733, the CPU 310 of the image processing device 3 uses the output unit 314 to transmit the composite image (superimposed image) created in step S732 to the image pickup device 2. Information about the affected area may be transmitted from the image processing device 3 to the image pickup device 2, and the image pickup device 2 may create a composite image.

In step S734, the CPU 310 of the image processing device 3 reads a subject ID that identifies the subject from the one-dimensional barcode (not shown) included in the image captured in step S702. The transmission timing of the image captured in step S702 is not particularly limited. For example, in step S714, the image processing device 2 may output the captured image of step S702 to the image processing device 3, and in step S727, the image processing device 3 may acquire the captured image of step S702 from the image processing device 2.

In step S735, the CPU 310 of the image processing device 3 collates the subject ID read in step S734 with the subject ID registered in advance, and acquires (determines) the name of the current subject. When the current subject name or subject ID is not registered, the CPU 310 causes the user to register and acquire the current subject name or subject ID.

In step S736, the CPU 310 of the image processing device 3 records the subject information including the evaluation result of the affected area (analysis result of step S730 and step S731) in the auxiliary storage unit 316 as the data of the subject determined in step S735. To do. If the data associated with the current subject (subject ID) is not recorded, create new subject information, and if the data associated with the current subject (subject information) is recorded, Update subject information.

The data structure of the subject information 1500 stored in the image processing apparatus 3 will be described with reference to FIG. The subject information 1500 includes the subject ID 1501, the subject name 1502, and the affected area information 1510 corresponding to the subject ID 1501 and the name 1502. In the affected area information 1510, information is managed for each imaging date and time. Specifically, the affected area information 1510 includes one or more combinations of date information 1503, affected area image 1504, affected area evaluation information 1505, and pocket evaluation information 1506. The date information 1503 is information indicating the shooting date of the affected area, and the affected area image 1504 is an image (an image obtained by photographing the affected area) used for evaluation of the affected area. The affected area evaluation information 1505 includes the value obtained by evaluating the affected area including both the ulcer surface and the pocket. In the example of FIG. 15, the affected area evaluation information 1505 includes the size of the affected area including both the ulcer surface and the pocket, the major axis of the affected area, the minor axis of the affected area, and the evaluation value of DESIGN-R. The pocket evaluation information 1506 includes a value obtained by evaluating the pocket. In the example of FIG. 15, the pocket evaluation information 1506 includes pocket state information indicating the state of the pocket, the size of the pocket, the major axis of the pocket, and the minor axis of the pocket. For example, for pressure ulcers with pockets that cover the entire ulcer surface, the letters "all pockets with pockets", for pressure ulcers with pockets that partially overlap the ulcer surface, "partial outer capsules with pockets", and for pressure ulcers without pockets, "no pockets". The information is registered as pocket status information. The pocket state information may be registered by inputting information by the user, or may be automatically registered by image analysis. In this way, the affected area evaluation information 1505 and the pocket evaluation information 1506 are separately generated (calculated) and recorded. The subject information 1500 can be provided to the user by display or the like, or can be provided to other devices as data.

In step S715, the system control unit 219 of the image pickup apparatus 2 uses the communication unit 218 to receive (acquire) the composite image (superimposed image) transmitted by the image processing apparatus 3 in step S733.

In step S716, the system control unit 219 of the image pickup apparatus 2 displays the composite image received in step S715 on the display unit 222.

Next, an example of the moving image analysis process in step S731 of FIG. 7 will be described with reference to the flowchart of FIG. 16 (A). Here, an example of generating a composite image showing the size of a pressure ulcer pocket by moving image analysis processing will be described. In this case, in step S732 of FIG. 7, information may be superimposed on the composite image generated by the flowchart of FIG. 16 (A).

In step S1600, the CPU 310 of the image processing device 3 selects a reference frame (reference image) from a plurality of frames of the moving image. In step S1605, which will be described later, a light region (light emitting region; position of the tip of the light; position where light is emitted) is synthesized with respect to this reference image. Since human hands and lights are reflected in the frame during measurement with a light (Fig. 2), it is desirable to select a frame before measurement that does not show such extraneous objects as a reference image. For example, by starting movie shooting before putting the light in the pocket and selecting the first frame of the movie as the reference image, it is possible to obtain a reference image that does not show any unnecessary things. In addition, the shape and color information of the light is acquired in advance, it is analyzed whether or not the area corresponding to the light is included in the frame of the moving image, and the frame not including the area corresponding to the light is used as the reference image. May be selected as.

In step S1601, the CPU 310 of the image processing apparatus 3 detects the ulcer surface region from the reference image. Here, the ulcer surface region is detected in order to synthesize the light region based on the detection result of the ulcer surface region. If the image pickup device 2 and the subject do not move at all during the measurement with the light, it is not necessary to synthesize the light region with reference to the ulcer surface region, but since it is practically difficult, a reference region such as the ulcer surface region is provided. And synthesize the light area. Detection of the ulcer surface area is shown in FIG.
This is performed in the same manner as in step S728 of the above. Here, it is conceivable that the ulcer surface area may be hidden by the light or the operator's hand during the measurement with the light. In consideration of such a case, for example, markers 1701 and 1702 may be placed near the ulcer surface as shown in FIG. 17, and the placed markers may be detected as a reference for synthesizing the light region. In the example of FIG. 17, two markers 1701 and 1702 are arranged in consideration of the case where the markers are hidden during the measurement. In addition, three or more markers may be arranged. In addition, if there is something characteristic of the body of the patient (subject), that may be used as a reference.

The processing of steps S1602 to S1605 described below is repeated frame by frame so as to be performed for all frames of the moving image. In step S1602, the CPU 310 of the image processing device 3 detects a light region from the target image (frame to be processed). Here, it is assumed that the light region has a characteristic of being red and round. In step S1602, a region having such a feature is detected from the target image as a light region. The position of the light may be a red dot that is moving in the moving image with the predetermined size (the change in size in the moving image is within the predetermined size). In step S1603, the ulcer surface region is detected from the target image. As described above, it is necessary to detect the ulcer surface region in order to synthesize the light region with respect to the ulcer surface region. In step S1604, the projective transformation of the target image is performed. Since the relative orientation and position of the image pickup device 2 with respect to the subject may change during the measurement with the light, the light region can be accurately combined by performing the projective conversion. The specific method of projective transformation will be described later. In step S1605, the light region after the projective transformation is combined with the reference image. By performing the processing of steps S1602 to S1605 for all the frames, the composite image 1800 of FIG. 18A can be obtained. Alternatively, the point of the locus of the light may be obtained by using the frame at predetermined time intervals. At this time, the frame in which the light reaches the edge of the affected portion may be included in the composite image even if it is not the frame corresponding to the frame at predetermined time intervals. For example, a composite image is created by synthesizing a light region from a frame image every T = 0.01 seconds, 0.02 seconds, or 0.03 seconds. At this time, the position of the light with respect to the affected area may be detected from each image, and an item indicating the position of the light may be displayed. That is, only the light of the reference image may be displayed as the actual light, and the lights of the other images may display items such as red dots and black dots at the position of the reference image corresponding to the position of the light.

The light at the radial end of the affected area may be displayed in a display form different from that of other lights so that the end points can be easily understood. For example, the light at the edge position may be increased in brightness to generate a composite image. Alternatively, when the position of the light is indicated by an item, the color of the item may be changed and displayed. A line or the like indicating the trajectory of the light may be displayed. In this way, by making it easier for the user to understand the position of the edge and the trajectory of the light, it becomes easier for the user to draw the outer circumference.

Further, even if it is not every predetermined time, it may be acquired as an image for compositing every time the light moves a predetermined distance.

A still image taken together with the moving image may be used as a reference image, or the processing result of step S728 may be used instead of the processing result of step S1601.

Next, the process (projection transformation) of step S1604 of FIG. 16A will be described with reference to the flowchart of FIG. 16B. In step S1610, the CPU 310 of the image processing device 3 extracts the feature points of the ulcer surface region (the ulcer surface region detected in step S1601) of the reference image. Here, it is assumed that an arbitrary point on the outer periphery of the ulcer surface region is extracted as a feature point. In step S1611, similarly to step S1610, the CPU 310 of the image processing device 3 extracts feature points from the ulcer surface region of the target image. In step S1612, the CPU 310 of the image processing apparatus 3 includes the feature points extracted in step S1610 (feature points in the ulcer surface region of the reference image) and the feature points extracted in step S1611 (feature points in the ulcer surface region of the target image). Matching. By this matching, feature points corresponding to each other are identified between the reference image and the target image. In step S1613, the CPU 310 of the image processing apparatus 3 reverses the projective transformation so that the ulcer surface region of the target image becomes the same region (plane) as the ulcer surface region of the reference image based on the matching result of step S1612. Calculate the matrix. In step S1614, the CPU 310 of the image processing device 3 performs a projective transformation of the target image using the inverse matrix calculated in step S1613. By performing such a projective conversion, it is possible to obtain an image in which the change in the orientation of the image pickup apparatus 2 with respect to the subject is suppressed.

The composite image 1800 of FIG. 18A can be transmitted and received in steps S733 and S715 of FIG. 7 and displayed on the image pickup apparatus 2 in step S716 (providing the composite image). In that case, in step S717 of FIG. 7, the system control unit 219 performs an outer circumference drawing process for drawing the outer circumference of the pocket. The outer circumference drawing process after displaying the composite image 1800 on the image pickup apparatus 2 will be described with reference to the flowchart of FIG. Since the movement locus of the light can be visually recognized in the composite image 1800, the user can easily grasp the pocket area. As long as information on the trajectory of the movement of the light can be provided, the method of providing the information is not particularly limited.

In step S1900, the system control unit 219 of the image pickup apparatus 2 prompts the user to input the outer circumference of the pocket. The outer circumference of the pocket may be input by the user tracing the screen of the image pickup apparatus 2 (display unit 222) with a finger, or may be input by using an input device such as a touch pen. FIG. 18B shows a display example after the user inputs the outer circumference of the pocket. In the image 1810 of FIG. 18 (B), the pocket outer circumference 1811 is input along the apex (outside) of the light region, and the pocket outer circumference 1811 is superimposed and displayed on the composite image 1800 of FIG. 18 (A). ..

In step S1901, the system control unit 219 of the imaging device 2 uses the communication unit 218 to transmit the composite image 1810 on which the pocket outer periphery 1811 is drawn and the pocket outer periphery information regarding the pocket outer periphery 1811 to the image processing device 3.

In step S1910, the CPU 310 of the image processing device 3 uses the input unit 313 to receive the composite image 1810 and the pocket outer circumference information transmitted by the image pickup device 2 in step S1901.

In step S1911, the CPU 310 of the image processing device 3 calculates the area (size) of the pocket area based on the composite image 1810 received in step S1910 and the pocket outer circumference information. Here, it is assumed that the area of the pocket region is calculated by subtracting the area of the ulcer surface region 1812 from the area of the region surrounded by the outer circumference of the pocket 1811. That is, the area of the region 1821 in FIG. 18C is calculated. The area may be calculated according to the calculation method of DESIGN-R.

In step S1912, the CPU 310 of the image processing device 3 superimposes information such as a pocket area and its area (calculated in step S1911) on the reference image (the image that is the source of the composite image 1800). By doing so, a composite image as shown in FIGS. 14 (B) and 14 (C) can be obtained.

In step S1913, the CPU 310 of the image processing device 3 uses the output unit 314 to transmit the composite image created in step S1912 to the image pickup device 2.

In step S1902, the system control unit 219 of the image pickup apparatus 2 uses the communication unit 218 to receive the composite image transmitted by the image processing apparatus 3 in step S1913.

In step S1903, the system control unit 219 of the image pickup apparatus 2 displays the composite image received in step S1902. By doing so, the size of the pocket area can be measured without directly marking the pocket area on the skin of the patient (subject) with magic.

In FIGS. 16A and 16B, a composite image in which the position of the light region is accurately reflected is created by synthesizing the light region after performing the projective transformation. As another method, a method of using the focal length at the time of shooting can be considered. Since it takes time to measure the pocket area with the light, it is conceivable that the distance between the patient and the imaging device 2 changes during imaging (during measurement). Here, since the focal length information can also be acquired at the time of shooting, it is possible to enlarge or reduce the image by using this information.

A method of creating a composite image using the focal length at the time of shooting will be described with reference to the flowchart of FIG. Step S2000 is the same as step S1600 in FIG. 16A, and step S2001 is the same as step S1601. In step S2002, the CPU 310 of the image processing device 3 acquires the focal length of the reference image. The processing of steps S2003 to S2007 is repeated frame by frame so as to be performed for all frames of the moving image. In step S2003, the CPU 310 of the image processing device 3 acquires the focal length of the target image. In step S2004, the CPU 310 of the image processing apparatus 3 enlarges or reduces the target image so as to match the focal length of the reference image. Step S2005 is the same as step S1602 in FIG. 16A, step S2006 is the same as step S1603 in FIG. 1, and step S2007 is the same as step S1605. By using the focal length in this way, it is possible to create a composite image that accurately reflects the position of the light region.

In FIGS. 16A, 16B, and 20, when all the frames of the captured moving image are used as the target images, the light areas of the frames before and after the light is put in the pocket may also be combined. Since it is difficult to understand the locus of the light region in the composite image obtained in this way, it is easier to use if the frame (light region) of the specified period can be deleted. 21 (A) and 21 (B) show UIs (screens 2100) capable of such operations.

The screen 2100 includes a composite image and control items 2102-2104 for deleting unnecessary frames (unnecessary light areas) from the composite image. Item 2102 is a slider bar indicating a time axis, and items 2103 and 2104 are sliders for deleting unnecessary frames. By moving the sliders 2103 and 2104 left and right, unnecessary frames can be deleted. In the state of FIG. 21A, sliders 2103 and 2104 are placed at both ends of the slider bar 2102, and a composite image 2101 in which all frames of the moving image are combined is displayed. In the composite image 2101, the frames (light regions) before and after the light is put in the pocket are also composited, and the trajectory of the light region is difficult to understand. On the other hand, in the state of FIG. 21 (B), the range from the slider 2103 to the slider 2104 is narrowed from that of FIG. 21 (A). In this case, the frame before the frame corresponding to the slider 2103 and the frame after the frame corresponding to the slider 2104 are not combined. As a result, it is possible to display the composite image 2111 in which the frames before and after the light is put in the pocket are not composited and the locus of the light region is easy to understand. By adjusting the positions of the sliders 2103 and 2104 in this way, it is possible to exclude the light region during an unnecessary period from the synthesis target.

As described above, according to the first embodiment, the image pickup device 2 can image the pocket measurement work by the light as a moving image, and the image processing device 3 can perform the moving image analysis to create a composite image in which the pocket shape can be understood. Further, the composite image can be transmitted to the image pickup apparatus 2 so that the user can easily specify the pocket area.

[Example 2]
In the first embodiment, the image pickup device 2 and the image processing device 3 are separate devices, but the functional configuration of the image processing device 3 may be provided in the image pickup device 2 (the image pickup device 2 and the image processing device 3 are integrally configured). May be done). By doing so, it becomes unnecessary for the image pickup apparatus 2 to communicate with the image processing apparatus 3, and the processing load can be reduced. Further, in the first embodiment, a composite image showing the pocket area is transmitted to the image pickup device 2, and the user inputs the outer circumference of the pocket on the image pickup device 2. However, the input of the outer circumference of the pocket is not necessarily applied to the image pickup device 2. You don't have to do it. For example, it is conceivable that the composite image is stored in the image processing device 3, an input / output device such as a display or a mouse is connected to the image processing device 3, and the user inputs the outer periphery of the pocket to the image processing device 3. Further, the composite image is stored in the image processing device 3, and the user inputs the outer periphery of the pocket to another image processing device (PC, smartphone, tablet, etc.) different from the image processing device 3, and the other image processing device sends the composite image. It is also conceivable to notify the image processing device 3 of the outer periphery of the pocket.

[Example 3]
In Example 1, the calculation of the area of the ulcer surface region and the creation of the composite image showing the size of the pocket region were performed at the same timing (same flowchart), but they may be performed at different timings. For example, due to hospital circumstances, it is conceivable to measure the ulcer surface area and the pocket area using a light at different timings. In such a situation, it is attempted to superimpose the ulcer surface region and the pocket region (filled image) as shown in the superimposed image 1410 (composite image) of FIG. 14 (B). In such a case, the distance between the imaging device 2 and the patient changes due to a large change in the posture of the patient at the time of measurement between the timing of measurement of the ulcer surface region and the timing of measurement of the pocket region. There is. Therefore, the ulcer surface area or the pocket area cannot be overlapped with the correct size. In such a case, it is possible to obtain a composite image in which the ulcer surface area and the pocket area are superimposed with the correct size by enlarging or reducing one image (area) using the focal length at the time of shooting. Become.

Even if the measurement of the ulcer surface area and the measurement of the pocket area using the light are performed at different timings, if the shooting distance can be kept unchanged between those timings, the ulcer surface area and the pocket area can be measured. Superimposition becomes easier. For example, when the ulcer surface area is measured first and the pocket area is measured at a later date, if the measurement is performed at the same shooting distance, the scale of the ulcer surface area and the pocket area will be the same, and the image (the ulcer surface area and the pocket) will be the same. Area) superimposition becomes easier. Therefore, the imaging distance at the time of measuring the ulcer surface region is maintained, and the imaging distance at the time of photographing the pocket region is the same as that at the time of measuring the ulcer surface region (at the time of imaging for measuring the ulcer surface region). It is advisable to control the shooting so that the shooting starts at the same timing. Since it is also necessary for the inspector to start measuring the pocket area with a light when the imaging is started, the imaging device 2 may be notified that the imaging has started. By automatically determining the timing of starting shooting in this way, it is possible to adjust the shooting distance between a plurality of measurements.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1: Image processing system 2: Image processing device 211: Imaging unit 217: Image processing unit 219: System control unit 222: Display unit 223: Operation unit 3: Image processing device 310: Central processing unit (CPU) 313: Input unit 314 : Output section

Claims (20)

An acquisition method for acquiring information about the recorded video,
From the information acquired by the acquisition means, a detection means for detecting a point at the radial end of the affected portion in the locus of the light moving in the affected portion.
An image processing system comprising a providing means for providing information on the outer circumference of an affected portion from a plurality of points detected by the detecting means. The image processing system according to claim 1, wherein the acquisition means acquires information regarding a locus in which a light has moved. The image processing system according to claim 1 or 2, wherein the detection means detects an apex when the light moves in the affected portion in the radial direction as a point at the end in the moving image. The image processing system according to any one of claims 1 to 3, wherein the point at the end is a position of a boundary between a region of the affected portion and a region different from the affected portion. The claim is characterized in that the light moves in the affected portion in the radial direction from a predetermined region near the center of the affected portion to the position of the edge of the affected portion, and then repeatedly moves to the predetermined region. The image processing system according to any one of 1 to 4. The image processing system according to any one of claims 1 to 5, wherein the information regarding the outer circumference of the affected portion includes the shape of the outer circumference of the affected portion formed by connecting the plurality of points. The image processing system according to any one of claims 1 to 6, wherein the information regarding the outer circumference of the affected portion includes an area of the affected portion formed by connecting the plurality of points. The image according to any one of claims 1 to 7, wherein the information regarding the outer circumference of the affected portion includes a major axis or a minor axis length of the affected portion formed by connecting the plurality of points. Processing system. The image processing system according to any one of claims 1 to 8, wherein the plurality of points are connected by a spline curve or a Bezier curve. Any one of claims 1 to 9, wherein the providing means controls the plurality of points and a line connecting the plurality of points so as to superimpose and display the image of the affected portion. The image processing system described in the section. The providing means controls to display the moving image in a live view, and controls to superimpose and display a line connecting a plurality of points detected by the detecting means so far on the moving image. The image processing system according to any one of claims 1 to 10. Further having a receiving means capable of accepting an operation instructing deletion of a point for forming the outer circumference, movement of a point for forming the outer circumference, or addition of a point for forming the outer circumference.
The image processing system according to any one of claims 1 to 11, wherein the providing means updates information on the outer circumference from a plurality of points after the change according to the operation. The providing means is based on the affected portion formed by connecting the plurality of points and the portion of the affected portion exposed to the outside, as at least a part of the information regarding the outer circumference of the affected portion. The image processing system according to any one of claims 1 to 12, wherein the size of a portion not exposed to the outside is calculated. The providing means calculates the size of the portion not exposed to the outside by the calculation method of DESIGN-R from the affected portion formed by connecting the plurality of points and the portion exposed to the outside. 13. The image processing system according to claim 13. The providing means further calculates the size of the affected part including the part exposed to the outside and the part not exposed to the outside as a part of the information about the outer circumference.
The image processing system according to claim 13, wherein the image processing system separately records the size of the affected portion and the size of the portion not exposed to the outside. The image processing system according to any one of claims 1 to 15, wherein the acquisition means, the detection means, and the providing means are provided in an imaging device or an information processing device. It is characterized by further having a control means for controlling so that the shooting of the moving image is automatically started at the timing when the focal length becomes the same as the time of shooting for measuring the portion of the affected part exposed to the outside. The image processing system according to any one of claims 1 to 16. The acquisition step to get information about the video that was shot, and
From the information acquired in the acquisition step, a detection step of detecting a point at the radial end of the affected portion in the locus of the light moving in the affected portion, and a detection step.
A control method of an image processing system, comprising a providing step of providing information on the outer circumference of an affected portion from a plurality of points detected in the detection step. A program for causing a computer to function as each means of the image processing system according to any one of claims 1 to 17. A computer-readable storage medium containing a program for causing the computer to function as each means of the image processing system according to any one of claims 1 to 17.

Images