JP2023087562A - Processing device, processing program and processing method - Google Patents

Abstract

To provide a processing device, a processing program, and a processing method suitable for analyzing fluid attributes.SOLUTION: There is provided a processing device including at least one processor. The at least one processor is configured to perform the processes of: acquiring image data of a captured image captured by an imaging device included in the processing device or connected to the processing device via a communication interface; generating fluid region identification information that identifies a fluid region in which a fluid is present in the captured image based on the acquired image data; and analyzing attribute information of the fluid being present in the fluid region identified based on the fluid region identification information.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a processing device, a processing program, and a processing method for analyzing fluid attribute information from an image captured by a camera.

2. Description of the Related Art Conventionally, it has been known to identify an object included in an image captured by a camera or the like by inputting the image into a trained model. For example, in Patent Document 1, an image captured by a camera is input to a first classifier that has been pre-learned to detect a predetermined object, so that the image contains the object and has a predetermined shape. An object state identifier is described for detecting object regions having However, such a device has only been able to identify objects with little change in shape.

Japanese Patent Application Laid-Open No. 2021-128705

Therefore, based on the above technology, an object of the present disclosure is to provide a processing apparatus, a processing program, and a processing method suitable for analyzing attribute information of a fluid according to various embodiments.

According to one aspect of the present disclosure, "a processing device comprising at least one processor, the at least one processor being included in the processing device or connected to the processing device via a communication interface Acquiring image data of a captured image captured by an imaging device, generating fluid region specifying information specifying a fluid region in which fluid exists in the captured image based on the acquired image data, and specifying the fluid region. A processing device configured to perform processing for analyzing attribute information of the fluid present in the fluid region specified based on the information is provided.

According to one aspect of the present disclosure, "by being executed by at least one processor provided in a processing device, the at least one processor is included in the processing device or via a communication interface to the processing device. acquiring image data of a captured image captured by an imaging device connected to the device, and generating fluid region specifying information specifying a fluid region in which the fluid exists in the captured image based on the acquired image data; A processing program that analyzes the attribute information of the fluid existing in the fluid region specified based on the fluid region specifying information” is provided.

According to one aspect of the present disclosure, "a processing method executed by at least one processor provided in a processing device, the processing method being included in the processing device or connected to the processing device via a communication interface Acquiring image data of a captured image captured by an imaging device; generating fluid region specifying information specifying a fluid region in which fluid exists in the captured image based on the acquired image data; and analyzing the attribute information of the fluid present in the fluid region identified based on the fluid region identification information.

According to the present disclosure, it is possible to provide a processing device, a processing program, and a processing method suitable for analyzing fluid attribute information.

It should be noted that the above effect is merely an example for convenience of explanation, and is not limiting. In addition to or instead of the above effects, any effects described in the present disclosure or effects obvious to those skilled in the art can be achieved.

FIG. 1 is a diagram illustrating an example of use of a processing system 1 according to one embodiment of the present disclosure. FIG. 2 is a schematic diagram of a processing system 1 according to one embodiment of the present disclosure. FIG. 3 is a block diagram showing the configuration of the processing device 100 according to one embodiment of the present disclosure. FIG. 4 is a diagram conceptually showing an image table stored in the memory 113 of the processing device 100 according to an embodiment of the present disclosure. FIG. 5 is a diagram showing a processing flow performed by the processing device 100 according to one embodiment of the present disclosure. FIG. 6 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure; FIG. 7 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure; FIG. 8 is a diagram showing a processing flow for generating a trained model according to an embodiment of the present disclosure. FIG. 9A is a diagram showing an example of a captured image processed by the processing device 100 according to an embodiment of the present disclosure. FIG. 9B is a diagram illustrating an example of a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. FIG. 10 is a diagram showing an example of a screen output from the processing device 100 according to an embodiment of the present disclosure.

Various embodiments of the present disclosure are described with reference to the accompanying drawings. In addition, the same reference numerals are attached to common components in the drawings.

1. Overview of processing system 1 The processing system 1 according to the present disclosure identifies an area where a predetermined subject exists in an image captured by an imaging device such as a camera, and analyzes the attribute information of the subject existing in the area. used to A subject used in the processing system 1 is a fluid that can be easily deformed by external pressure. Examples of such fluids include gases, liquids, mixtures of gases and liquids, and mixtures of gases or liquids with trace amounts of solids. Such fluids are generally viscous, move due to external forces, and have irregular boundaries with other bodies.

FIG. 1 is a diagram illustrating an example of use of a processing system 1 according to one embodiment of the present disclosure. Specifically, FIG. 1 is a diagram showing an example of a captured image in which a fluid used in the processing system 1 is captured as a subject. According to FIG. 1, a floor 11, a magnetic stirrer 12 placed on the floor 11, a cup 13 placed on the magnetic stirrer 12, a fluid 14 poured into the cup 13, etc. are used as subjects. there is Then, by applying a captured image obtained by capturing the subject with a camera or the like to the processing system 1 according to the present disclosure, a region where the fluid 14 exists in the captured image is specified, and the fluid 14 existing in the region is specified. Attribute information is parsed.

In the present disclosure, the fluid includes gas, liquid, mixture of gas and liquid, and mixture of gas or liquid and a small amount of solid, as described above. Examples of such fluids include water, beverages, jellies, cosmetics, pharmaceuticals, foodstuffs, cement, foams, oils for edible, fuel, industrial and the like purposes.

In addition, in the present disclosure, attribute information may be any information that indicates the type, characteristics, properties, etc. of a fluid. Typical examples of attribute information include fluid classification information (kind), viscosity, flow velocity, mass, pressure, temperature, or a combination thereof. It is not necessary for the processing system 1 to analyze all of these pieces of attribute information, and it is sufficient if at least one of them can be analyzed.

In the present disclosure, the captured image may be one or more moving images or one or more still images. In order to obtain such a captured image, an imaging device such as a camera is used. As an example of its operation, when a power button is pressed, a through image is captured by the imaging device, and the captured through image is captured. is displayed on the display of the imaging device. After that, when the user presses the imaging button, one or more still images are captured by the imaging device, and the captured images are displayed on the display. Alternatively, when the user presses the imaging button, the imaging of a moving image is started, and the image captured by the imaging device is displayed on the display during that time. Then, when the imaging button is pressed again, imaging of the moving image is terminated. In this way, in a series of operations, various images such as through images, still images, and moving images are captured by the imaging device and displayed on the display. It may include all of the images captured by the image capture device. Furthermore, in the present disclosure, image data simply means data representing a captured image in a predetermined data format, and does not only refer to data represented in a specific data format.

2. Configuration of Processing System 1 FIG. 2 is a schematic diagram of the processing system 1 according to one embodiment of the present disclosure. According to FIG. 2, the processing system 1 includes a processing device 100 and an imaging device 200 communicably connected to the processing device 100 via a wired or wireless network. The processing device 100 receives an operation input by the user and controls imaging by the imaging device 200 . In addition, the processing device 100 processes the captured image captured by the imaging device 200, identifies a region in which the fluid exists in the captured image, and analyzes the attribute information of the fluid.

Such an imaging device 200 is typically a camera, but it does not necessarily have to be connected to the processing device 100 via a wired or wireless network as shown in FIG. For example, even the imaging device 200 provided as a component in the processing device 100 can appropriately execute the processing according to the present disclosure. Also, the imaging device 200 does not need to exist alone, and may exist in combination with other devices. For example, the imaging device 200 can be placed in a manufacturing line apparatus for manufacturing fluids, products containing fluids, or products made from fluids.

FIG. 3 is a block diagram showing the configuration of the processing device 100 according to one embodiment of the present disclosure. According to FIG. 3, processing device 100 includes output interface 111 , processor 112 , memory 113 , input interface 115 and communication interface 114 . Each of these components are electrically connected to each other via control lines and data lines. Note that the processing apparatus 100 does not need to include all of the components shown in FIG. 3, and may be configured by omitting some or adding other components. For example, processing system 1 may include a battery or the like for driving each component.

A laptop PC, a desktop PC, a smart phone, a tablet, or the like can be used as such a processing device 100 . However, not limited to these, other devices can be suitably used as the processing device 100 . Moreover, the processing device 100 does not need to execute all the processes by itself, and it is possible for the processing device 100 to execute some of the processes together with other devices. For example, it is possible to jointly execute analysis processing using a learned algorithm in a cloud server device.

First, the processor 112 functions as a control unit that controls other components of the processing device 100 based on programs stored in the memory 113 . The processor 112 acquires image data of a captured image from the imaging device 200 based on a program stored in the memory 113, and controls processing related to analysis of fluid attribute information. Specifically, the processor 112 performs processing for acquiring image data of a captured image captured by the imaging device 200 included in the processing device 100 or connected via the communication interface 114, and performs processing for acquiring image data of a captured image to a learned fluid region identification model. A process of inputting image data of an image, a process of generating fluid area specifying information specifying a fluid area in which a fluid exists in a captured image based on the acquired image data, and a process of specifying a fluid area based on the fluid area specifying information. Based on the program stored in the memory 113, a process of analyzing the classification information of the existing fluid, a process of analyzing the characteristic information of the fluid existing in the fluid region based on the analyzed classification information, and the like are executed. The processor 112 may be mainly composed of one or more processors, and such processors may be a combination of CPU, GPU, FPGA, and the like.

The memory 113 is composed of RAM, ROM, non-volatile memory, HDD, etc., and functions as a storage unit. The memory 113 stores instruction commands for various controls of the processing device 100 according to this embodiment as programs. Specifically, the memory 113 performs processing for acquiring image data of a captured image captured by the imaging device 200 included in the processing device 100 or connected via the communication interface 114, and performs processing for acquiring image data for a learned fluid region identification model. A process of inputting image data of an image, a process of generating fluid area specifying information specifying a fluid area in which a fluid exists in a captured image based on the acquired image data, and a process of specifying a fluid area based on the fluid area specifying information. It stores programs for the processor 112 to execute, such as processing for analyzing classification information of existing fluids and processing for analyzing characteristic information of fluids existing in the fluid region based on the analyzed classification information. In addition to the program, the memory 113 stores image data of images captured by the imaging device 200, an image table for managing the images, and the like. The memory 113 also stores programs related to each learned model such as a learned segmentation determination model, a learned characteristic analysis model, a learned fluid classification model, and a learned characteristic analysis model, which are used for analysis of captured images.

The input interface 115 functions as an input unit that receives a user's instruction input to the processing device 100 . An example of the input interface 115 is an "imaging button" for transmitting an instruction to capture an image of a subject in the imaging device 200, a "confirmation button" for making various selections, and a button for returning to the previous screen or canceling an input confirmation operation. a "return/cancel button" to move the pointer or the like displayed on the display via the output interface 111, an on/off key to turn on/off the power of the processing device 100, various characters physical key buttons such as character input key buttons for inputting . For the input interface 115, it is also possible to use a touch panel that is superimposed on the display functioning as the output interface 111 and has an input coordinate system corresponding to the display coordinate system of the display. In this case, icons corresponding to the physical keys are displayed on the display, and the user inputs instructions via the touch panel to select each icon. Any method such as a capacitive method, a resistive film method, or the like may be used to detect a user's instruction input through the touch panel. The input interface 115 does not always have to be physically provided in the processing device 100, and may be connected as necessary via a wired or wireless network.

The output interface 111 functions as an output unit for outputting the captured image captured by the imaging device 200, the segmentation image processed by the processing device 100, the attribute information of the analyzed fluid, and the like. An example of the output interface 111 is a display composed of a liquid crystal panel, an organic EL display, a plasma display, or the like. However, the processing device 100 itself does not necessarily have to have a display. For example, an interface for connecting a display or the like that can be connected to the processing device 100 via a wired or wireless network can function as the output interface 111 that outputs display data to the display or the like.

The communication interface 114 serves as a communication unit for transmitting/receiving various commands related to start of imaging and image data of an image captured by the imaging device 200 to/from the imaging device 200 connected via a wired or wireless network. Function. Examples of the communication interface 114 include connectors for wired communication such as USB and SCSI, transmitting/receiving devices for wireless communication such as wireless LAN, Bluetooth (registered trademark) and infrared rays, and various connection terminals for printed mounting boards and flexible mounting boards. and so on.

Although not specifically illustrated, the imaging device 200 includes a camera, light source, processor, memory, display panel, input interface, communication interface, and the like. Each of these components are electrically connected to each other via control lines and data lines. Note that the imaging apparatus 200 does not need to include all of these components, and may be configured by omitting some or adding other components.

In the imaging device 200, the camera functions as an imaging unit that detects reflected light reflected from a subject and generates a subject image. A camera includes, for example, a CMOS image sensor to detect the light, and a lens system and a drive system to achieve desired functions. The image sensor is not limited to a CMOS image sensor, and other sensors such as a CCD image sensor can also be used.

The light source is driven by an instruction from the processing device 100 or the imaging device 200 and functions as a light source unit for irradiating light into the oral cavity. A light source includes one or more light sources. In this embodiment, the light source is composed of one or a plurality of LEDs, and each LED emits light having a predetermined frequency band toward the oral cavity. For the light source, for example, light having a desired band from among an ultraviolet light band, a visible light band, and an infrared light band, or a combination thereof is used.

The processor functions as a control unit that controls other components of the imaging device 200 based on programs stored in the memory. Specifically, the processor controls the driving of the camera and the driving of the light source based on the program stored in the memory, and also controls the storage in the memory of the image captured by the camera and the imaging conditions in the imaging. . The processor also controls the output of the captured image stored in the memory to the display panel and the transmission to the processing device 100 . A processor is mainly composed of one or more processors, and a CPU is used as an example of such a processor.

The memory is composed of RAM, ROM, nonvolatile memory, HDD, etc., and functions as a storage unit. The memory stores instruction commands for various controls of the imaging device 200 as programs. In addition to the program, the memory also stores captured images captured by the camera.

The display panel functions as a display unit for displaying the subject image captured by the imaging device 200 . The display panel is composed of a liquid crystal panel, but it is not limited to the liquid crystal panel, and may be composed of an organic EL display, a plasma display, or the like.

The input interface functions as an input unit that receives a user's instruction input to the processing device 100 and the imaging device 200 . Examples of the input interface 210 include an “imaging button” for instructing the start/end of recording by the imaging device 200, a “power button” for turning on/off the power of the imaging device 200, and a “power button” for making various selections. Confirmation button", "Return/Cancel button" for returning to the previous screen or canceling the input confirmation operation, and physical key buttons such as a cross key button for moving icons displayed on the display panel. . These various buttons and keys may be physically prepared, or may be displayed as icons on the display panel and selected using a touch panel or the like superimposed on the display panel and arranged as an input interface. It may be made possible. Any method such as a capacitive method, a resistive film method, or the like may be used to detect a user's instruction input through the touch panel.

The communication interface functions as a communication unit for transmitting and receiving information to and from the imaging device 200 and/or other devices. Examples of communication interfaces include connectors for wired communication such as USB and SCSI, transmitting/receiving devices for wireless communication such as wireless LAN, Bluetooth (registered trademark) and infrared rays, and various connection terminals for printed mounting boards and flexible mounting boards. , various ones.

Further, the processing system 1 can include various sensor devices, not shown. Any such sensor device may be used as long as it can measure the pressure applied to the subject, the gravity applied to the subject, the surface condition of the subject, the ambient temperature where the subject is placed, and the like. . Specifically, an image sensor that analyzes the surface condition of a subject, a temperature sensor that measures the ambient temperature of the subject, a temperature sensor that measures the internal temperature of the subject, a pressure sensor, a distance sensor, or a combination thereof. is mentioned. The sensor device may exist by itself and transmit the measured output values to the processing device 100 . Alternatively, the sensor device may be provided inside the processing device 100 or the imaging device 200 as one of the constituent elements, and may transmit the measured output values so that they can be processed by the processor 112 of the processing device 100 .

3. Information Stored in Memory 113 of Processing Device 100 FIG. 4 is a diagram conceptually showing an image table stored in the memory 113 of the processing device 100 according to an embodiment of the present disclosure. The information stored in the image table is updated and stored as needed according to the progress of the processing of the processor 112 of the processing device 100 .

According to FIG. 4, the image table stores image data, imaging condition information, imaging environment information, segmentation image data, classification information, property information, etc. in association with the image ID information. “Image ID information” is information for specifying each captured image captured by the imaging device 200 . The image ID information is information that can be generated each time a new captured image is captured by the imaging device 200 or each time a new captured image is acquired by the processing device 100 . The “image data” is information of the image data itself of the captured image captured by the imaging device 200 or information specifying the storage location thereof. “Imaging condition information” is information indicating imaging conditions when the captured image is captured by the imaging device 200 . Such imaging condition information includes aperture value, shutter speed, ISO sensitivity, exposure, white balance, or a combination thereof when the captured image is captured. The imaging condition information is stored in the imaging device 200 with the conditions at the time of imaging, and is acquired together with the image data of the captured image from the imaging device 200 . In addition, in the image capturing apparatus 200, such image capturing conditions can be set automatically, or according to an instruction input from the user or another user accepted by the processing apparatus 100 or the image capturing apparatus 200. can also be set. The “imaging environment information” is information related to the subject itself or the environment around the subject when the captured image was captured by the imaging device 200 . Such imaging environment information includes output values measured by various sensor devices, such as pressure (atmospheric pressure), gravity applied to the subject, surface condition of the subject, ambient temperature where the subject is placed, subject Information on the distance to The imaging environment information is measured by various sensors, in which measurement instructions are received by various sensors in synchronism with reception of imaging instructions by the imaging apparatus 200 . The measured output values are then acquired by the processing device 100 via the communication interface 114 .

"Segmentation image data" is image data obtained as a result of inputting image data of an acquired captured image into a trained segmentation determination model. As an example, each region can be identified by adding color information to the result of segmentation (partitioning) of the fluid region where the fluid exists, other regions other than the fluid region, and the boundary region that serves as a boundary between these regions. image data. The “classification information” is one of the attribute information of the fluid, and is information indicating the type of fluid obtained as a result of inputting the image data of the captured image and the segmentation image data into the learned fluid classification model. Such fluid types include water, beverages, jellies, cosmetics, pharmaceuticals, foods, cements, foams, oils for edible, fuel, industrial, and the like purposes. The “property information” is one type of attribute information of the fluid, and is information indicating the characteristics of the fluid obtained as a result of inputting the image data of the captured image, the segmentation image data, and the classification information into the learned characteristic analysis model. Such fluid characteristic information includes viscosity, flow velocity, mass, pressure, temperature, and the like.

4. Processing Flow Executed by Processing Device 100 FIG. 5 is a diagram showing a processing flow executed by the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 5 is a diagram showing a processing flow from acquisition of a captured image from the imaging device 200 in the processing device 100 to analyzing attribute information and outputting the result. The processing flow is mainly performed by the processor 112 of the processing device 100 reading and executing a program stored in the memory 113 .

According to FIG. 5 , prior to execution of the processing flow, the processor 112 receives image data of the captured image of the subject captured by the imaging device 200 via the communication interface 114 and stores the image in the image table of the memory 113 . Stored in association with ID information. Also, at this time, output values measured from various sensor devices that are measured in synchronization with imaging by the imaging device 200 are received via the communication interface 114 and stored in the image table of the memory in association with the image ID information. do. Further, the imaging condition information when the imaging device 200 captures the captured image is received via the communication interface 114 and stored in the image table of the memory in association with the image ID information.

Here, FIG. 9A is a diagram showing an example of a captured image processed by the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 9A is a diagram showing an example of a captured image captured by the imaging device 200. FIG. According to FIG. 9A, a floor surface 11, a magnetic stirrer 12 placed on the floor surface 11, a cup 13 placed on the magnetic stirrer 12, a fluid 14 poured into the inside of the cup 13, and the like are imaged as subjects. there is That is, in this captured image, there is an area where the fluid 14 exists as the fluid area, and other areas where the floor surface 11, the magnetic stirrer 12, the cup 13, etc. other than the fluid exist. In the following, a case where such captured images are used for processing will be described.

Returning to FIG. 5 again, the processor 112 refers to the image table in the memory 113 and, if there is unprocessed image data, reads the image data (S111). The processor 112 inputs the read image data to a learned segmentation determination model (creation of the learned model will be described later) to perform fluid segmentation processing (S112). At this time, only image data is input to the trained segmentation determination model, but in addition to this, imaging condition information, imaging environment information, or a combination thereof associated with the image ID information of the captured image can also be used. It is possible.

Here, FIG. 9B is a diagram showing an example of a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. Specifically, it is a diagram showing an example of segmentation image data obtained as a result of inputting the image data of the captured image to the trained segmentation determination model in S112 of FIG. According to FIG. 9B, as shown in FIG. 9A, in the captured image, the area where the fluid 14 exists as the fluid area, and the floor surface 11 other than the fluid as other areas, the magnetic stirrer 12, the cup 13, etc. The existing area is included. By inputting the image data of such a captured image into the trained segmentation determination model, the fluid region 15 corresponding to the region where the fluid 14 existed, and the other region where something other than the fluid 14 existed Other areas 16 and boundary areas 17 indicating boundaries between the fluid area 15 and the other areas 16 are identifiably specified corresponding to the areas (areas such as the floor surface 11, the magnetic stirrer 12 and the cup 13). Thus, according to the segmentation image data shown in FIG. 9, any one of the fluid region, the other region, and the boundary region is specified for each pixel, and color information corresponding to the specified region is added for each pixel. , each region is identifiably specified. It should be noted that the identification by color information is only used to enable the user to identify the image when it is output to a display or the like via the output interface 111 . Therefore, it suffices here for each pixel (each coordinate position) to specify which region it is, and of course other information may be used for identification, and color information and the like may not be added. good. For example, it may be identified by gray scale, or may be identified by adding a specific code.

Returning to FIG. 5 again, the processor 112 stores the segmentation image data of the captured image obtained as a result of processing in the image table of the memory 113 in association with the image ID information of the image data of the processed captured image ( S113). Processor 112 then determines whether a fluid region exists based on the obtained segmentation image data (S114). As a result, if it is determined that a fluid region exists, the processor 112 combines the image data of the captured image and the segmentation image data stored in S113 with a learned fluid classification model (creation of the learned model will be described later). ) to perform fluid classification processing (S115). At this time, image data and segmentation image data are input to the learned fluid classification model. may

Next, the processor 112 classifies classification information (water, beverages, jelly, cosmetics, medicines, food, cement, foam, oil for food, fuel, industrial use, etc.) are stored in the image table of the memory 113 in association with the image ID information of the image data of the processed captured image (S116). Then, the processor 112 inputs the image data of the captured image, the segmentation image data stored in S113, and the classification information stored in S116 to a learned characteristic analysis model (creation of the learned model will be described later). Then, fluid characteristic analysis processing is performed (S117). At this time, image data, segmentation image data, and classification information are input to the learned fluid classification model. Combinations may also be used.

Next, the processor 112 obtains characteristic information (viscosity, flow velocity, mass, pressure, temperature, etc.) analyzed as a result of input to the learned characteristic analysis model with respect to the fluid existing in the fluid region specified based on the segmentation image data. ) is stored in the image table of the memory 113 in association with the image ID information of the image data of the processed captured image (S118). Then, the processor 112 receives a request from the user through the input interface 115, and outputs the analysis result to a display or the like through the output interface 111 based on the request (S119). With the above, the processing flow ends.

Here, FIG. 10 is a diagram showing an example of a screen output from the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 10 is a diagram showing an example of the analysis result image output to the display via the output interface 111 in S119 of FIG. According to FIG. 10, the captured image 22 captured by the imaging device 200 is displayed on the upper left of the display. A segmentation image 21 that reproduces the segmentation image data is displayed to the right of the captured image 22 so that the positional relationship of each region can be compared. This allows the user to visually identify the fluid region, the other region and the boundary region identified based on the segmentation image data.

Also, an imaging environment display area 25 is displayed below the captured image 22 . The imaging environment display area 25 includes, as an example, the pressure (atmospheric pressure) and temperature stored as the imaging environment information in the image table of the memory 113, but may naturally include other imaging environment information. Further, below the segmentation image 21, segmented fluid classification information 23 and fluid characteristic information (flow velocity information 24a and viscosity information 24b) are displayed. This allows the user to visually recognize the fluid attribute information analyzed by each learned model. The specific information that can be displayed here is not limited to the flow velocity and viscosity, and may be other characteristic information such as the temperature of the fluid itself.

Although FIG. 5 does not explicitly show that the image data of the captured image read out in S111 is subjected to special image processing, it is possible to perform predetermined preprocessing as desired. Such preprocessing includes filter processing such as bandpass filters including high-pass filters and low-pass filters, averaging filters, Gaussian filters, Gabor filters, Canny filters, Sobel filters, Laplacian filters, median filters, bilateral filters, etc. Trimming processing, defogging processing, super-resolution processing, and combinations thereof are selected according to purposes such as high definition, area extraction, noise removal, edge enhancement, image correction, and image conversion. By executing preprocessing in this way, it is possible to improve analysis accuracy in segmentation, fluid classification, and characteristic analysis.

Further, in the fluid segmentation process of S112, the fluid classification process of S115, and the fluid characteristic analysis process of S117 of FIG. may be processed. By using these pieces of information together, it is possible to further improve the processing accuracy. For example, the flow velocity of a fluid is generally affected by the ambient temperature of the fluid or the temperature inside the fluid. As a result, fluid movement and morphology changes are severe at high temperatures and relatively mild at low temperatures, which can affect segmentation accuracy. Also, fluids having different flow velocities under the same conditions may have similar flow velocities if one is imaged at a high temperature and the other is imaged at a low temperature. In such cases, the fluid classification and characterization processes may be affected by temperature. Therefore, a trained model trained using images captured under high-temperature conditions and a trained model trained using images captured under low-temperature conditions using a predetermined temperature as a threshold are prepared. Then, a learned model to be input may be selected depending on whether the temperature is higher or lower than the threshold based on the imaging environment information when the input captured image was captured. In addition, the imaging environment information and the imaging condition information can also be used as parameters that are input when learning each trained model. For example, in the above temperature example, since the temperature can have a considerable effect on the properties of the fluid, by learning the temperature information together with the temperature It is possible to output the result considering

The image data of the captured image read in S111 of FIG. 5 may be a single image, a group of still images, or a moving image composed of a group of images. It is preferable to use a plurality of still image groups or moving images. In these cases, by performing each process described in FIG. 5 on each read image data or a part of the image data and performing the segmentation process of the fluid region, not only the flow velocity information can be obtained as the characteristic information, It is possible to obtain temporal movement information and change information of the fluid region. In particular, when analyzing flow velocity information as fluid characteristic information, it is preferable to use a plurality of temporally different still images or moving images captured during a certain period of time. This makes it possible to more accurately reflect changes in the fluid over time and improve analysis accuracy. Furthermore, at this time, the processor 112 can calculate numerical values such as averages, maximum values, and minimum values for characteristic information such as flow velocity information and viscosity information obtained from each image data. Thus, it is possible to obtain more accurate characteristic information.

5. Processing Flow for Generating Each Trained Model FIG. 6 is a diagram showing a processing flow for generating a trained model according to an embodiment of the present disclosure. Specifically, FIG. 6 is a diagram showing a processing flow relating to generation of a learned segmentation determination model used for the fluid segmentation processing of S112 of FIG. The processing flow may be executed by the processor 112 of the processing device 100 or by a processor of another processing device.

According to FIG. 6, the processor executes a step of acquiring learning image data of a captured image captured by, for example, the imaging device 200 (S211). Next, the processor executes a processing step of performing segmentation processing on the acquired image data for learning and adding correct label information (S212). Then, the processor executes a step of storing the assigned correct label information as learning segmentation information in association with the learning image data (S213).

When the training image data and the learning segmentation information associated therewith are obtained, the processor executes a step of performing machine learning of segmentation patterns using them (S214). As an example, the machine learning provides a set of information to a neural network that combines neurons, and learns while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. is performed by repeating Then, a step of obtaining a trained segmentation determination model is executed (S215). The acquired learned segmentation determination model is stored in the memory 113 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

In addition, in FIG. 6, the case where the captured image captured by the imaging device 200 is used as the learning image data, and the segmentation processing is performed on the learning image data by the user, for example, has been described. However, without being limited to this, it is also possible to create a CG (computer graphics) image in which a fluid region in which fluid exists is specified in advance and use it as learning image data and learning segmentation information.

In FIG. 6, the case of learning using image data for learning and segmentation information for learning has been described. In addition, the imaging environment information, which is output values measured by various sensor devices, may be used for learning.

FIG. 7 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure; Specifically, FIG. 7 is a diagram showing a processing flow relating to generation of a learned fluid classification model used for the fluid classification processing of S115 of FIG. The processing flow may be executed by the processor 112 of the processing device 100 or by a processor of another processing device.

According to FIG. 7, the processor executes a step of acquiring learning image data of an image captured by the imaging device 200, for example, and learning segmentation information in which the fluid region is segmented using the image data (S311). ). Next, the processor performs a process of classifying the fluid present in the fluid region included in the obtained learning image data into types, and executes a processing step of assigning correct label information (S312). Then, the processor executes a step of storing the assigned correct label information as learning classification information in association with learning image data or the like (S313).

When the learning image data and the like and the learning classification information associated therewith are obtained, the processor executes a step of performing machine learning of classification patterns using them (S314). As an example, the machine learning provides a set of information to a neural network that combines neurons, and learns while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. is performed by repeating A step of obtaining a trained fluid classification model is then performed (S315). The acquired learned fluid classification model is stored in the memory 113 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

Note that FIG. 7 describes a case where an image captured by the imaging device 200 is used as the learning image data, and the type of fluid included in the image input by the user is used as the correct label information. However, not limited to this, create a CG (computer graphics) image in which the fluid region where the fluid exists and the type of the fluid are specified in advance, and use it as image data for learning, segmentation information for learning, and classification information for learning It is also possible to

In FIG. 7, the case of learning using image data for learning, segmentation information for learning, and classification information for learning has been described. The imaging environment information, which is output values measured by various sensor devices when an image is captured, may also be used for learning. By using these information, further ○○○.

FIG. 8 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure; Specifically, FIG. 8 is a diagram showing a processing flow relating to generation of a learned characteristic analysis model used in the fluid characteristic analysis processing of S117 of FIG. The processing flow may be executed by the processor 112 of the processing device 100 or by a processor of another processing device.

According to FIG. 8, the processor includes, for example, learning image data of a captured image captured by the imaging device 200, learning segmentation information obtained by segmenting a fluid region using the image data, and information about the fluid existing in the fluid region. A step of acquiring classification information whose type is specified is executed (S411). Next, the processor performs a process of analyzing the properties of the fluid present in the fluid region in the obtained learning image data, and executes a process step of adding correct label information (S412). Then, the processor executes a step of storing the assigned correct label information as learning characteristic information in association with learning image data or the like (S413).

When the learning image data and the associated learning characteristic information are obtained, the processor executes a step of performing machine learning of the characteristic analysis pattern using them (S414). As an example, the machine learning provides a set of information to a neural network that combines neurons, and learns while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. is performed by repeating Then, a step of acquiring a learned characteristic analysis model is performed (S415). The acquired learned characteristic analysis model is stored in the memory 113 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

Here, the Navier-Stoke equation shown in Equation 1 below is used as an equation that describes the behavior of the fluid. Conversely, in order to be a fluid, each piece of characteristic information analyzed using the learned characteristic analysis model or the like needs to satisfy the Navier-Stoke equation shown in Equation 1 below.

Therefore, in the learning of the learned characteristic analysis model, the loss function to which the Navierstoke equation is applied as shown in Equation 1 is used as the characteristic information analyzed by the learned characteristic analysis model and the correct label information. The parameters of each neuron are adjusted to minimize the loss (L _n ) between the characteristic information used. That is, for each piece of characteristic information output from the learned characteristic analysis model, the parameter of each neuron is adjusted so that when input to the Navier-Stoke equation of Equation 1 above, it becomes a value that satisfies the equation.

In Equation 1 above, pressure (p), acceleration field (g), and fluid density (ρ) can be ignored as fixed values. Flow velocity information obtained from the learned characteristic analysis model is used as the velocity vector (ν), and viscosity information obtained from the learned characteristic analysis model is used as the kinematic viscosity coefficient (ν).

In addition, in FIG. 8, a case has been described in which an image captured by the imaging device 200 is used as the image data for learning, and the characteristics of the fluid measured by the user are used as the correct label information. However, not limited to this, a CG (computer graphics) image is created in which the fluid region where the fluid exists, the type and characteristics of the fluid are specified in advance, and the image data for learning, segmentation information for learning, and classification information for learning And it can also be used as characteristic information for learning.

Further, in FIGS. 6 to 8, the case of learning using image data for learning, segmentation information for learning, classification information for learning, and characteristic information for learning has been described. The imaging condition information at the time and the imaging environment information, which is output values measured by various sensor devices at the time of imaging, may be further used for learning. By using these pieces of information, it becomes possible to output the result of further consideration of the imaging environment information and the imaging condition information when outputting the analysis result of the characteristics from the trained model.

Furthermore, each trained model described in FIGS. 6 to 8 was generated using a neural network. However, not limited to these, it is also possible to generate using machine learning such as the nearest neighbor method, decision tree, regression tree, random forest, or the like. Furthermore, each trained model described in FIGS. 6 to 8 was generated using a neural network. Specific examples include machine learning models such as FCN and Mask R-CNN. However, the present invention is not limited to these, and any machine learning model capable of inputting images or capable of outputting pixel data can be used.

As described above, in one embodiment of the present disclosure, it is possible to provide a processing device, a processing program, and a processing method suitable for analyzing attribute information of a fluid.

The processes and procedures described herein can be implemented not only by those explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described herein are implemented by implementing logic corresponding to the processes in media such as integrated circuits, volatile memories, non-volatile memories, magnetic disks, and optical storage. be done. Further, the processes and procedures described in this specification can be implemented as computer programs and executed by various computers including processing devices and server devices.

Although the processes and procedures described herein are described as being performed by a single device, software, component, module, such processes or procedures may be performed by multiple devices, multiple software, It may be performed by multiple components and/or multiple modules. In addition, even if it is explained that various information described in this specification is stored in a single memory or storage unit, such information may be stored in a plurality of memories provided in a single device or It can be distributed and stored in a plurality of memories arranged in a plurality of devices. Furthermore, the software and hardware elements described herein may be implemented by integrating them into fewer components or by decomposing them into more components.

1 processing system 100 processing device 200 imaging device

Claims (11)

A processing device comprising at least one processor,
the at least one processor
Acquiring image data of a captured image captured by an imaging device included in the processing device or connected to the processing device via a communication interface;
generating fluid region identification information that identifies a fluid region in which the fluid exists in the captured image based on the acquired image data;
analyzing the attribute information of the fluid present in the fluid region specified based on the fluid region specifying information;
A processing device configured to process for The fluid region is a learning generated by machine learning using at least image data of a learning image and learning fluid region specifying information in which a fluid region in which a fluid exists in the learning image is specified in advance. 2. The processing device according to claim 1, specified by inputting image data of said captured image into a completed fluid region specifying model. 3. The processing device according to claim 1, wherein said attribute information includes classification information indicating a type of said fluid. 4. The processing device according to claim 3, wherein said attribute information further includes property information indicating properties of said fluid. 5. The processing apparatus according to claim 4, wherein said characteristic information is at least one of viscosity and flow velocity of said fluid. the at least one processor
analyzing the classification information of the fluid present in the fluid region based on the fluid region identification information;
analyzing the property information of the fluid present in the fluid region based on the analyzed classification information;
6. A processing apparatus according to claim 4 or 5, configured to process for. The classification information uses at least learning fluid region identification information that preliminarily identifies a fluid region in which a fluid exists in a learning image, and learning classification information that preliminarily classifies the type of fluid in the learning image. 7. The analysis is performed by inputting the fluid region identification information generated based on the captured image into a learned fluid classification model generated by machine learning with Processing equipment as described. The characteristic information is a trained image generated by performing machine learning using at least image data of the learning image and learning characteristic information obtained by analyzing characteristics of a fluid present in the learning image in advance. 7. The processing device according to any one of claims 4 to 6, wherein analysis is performed by inputting image data of said captured image into a characteristic analysis model. The processing apparatus according to any one of claims 1 to 8, wherein the fluid is at least one of gas, liquid, a mixture of gas and liquid, and a mixture of gas or liquid mixed with a small amount of solid. By being executed by at least one processor provided in the processing device,
the at least one processor;
Acquiring image data of a captured image captured by an imaging device included in the processing device or connected to the processing device via a communication interface;
generating fluid region identification information that identifies a fluid region in which the fluid exists in the captured image based on the acquired image data;
analyzing the attribute information of the fluid present in the fluid region specified based on the fluid region specifying information;
A processing program that functions as A processing method executed by at least one processor provided in a processing device, comprising:
acquiring image data of a captured image captured by an imaging device included in the processing device or connected to the processing device via a communication interface;
generating fluid region identification information that identifies a fluid region in which fluid exists in the captured image based on the acquired image data;
analyzing attribute information of the fluid existing in the fluid region specified based on the fluid region specifying information;
processing methods, including;

Images