JP7265672B2 - Information processing method, program and information processing device - Google Patents

Description

The present invention relates to an information processing method, a program, and an information processing apparatus.

Deep learning has produced impressive results in the areas of computer vision and speech recognition. For example, in aquaculture farms, automatic counting of the number of fish is a problem, but it is expected that such a problem can be improved by using a deep learning method.

WO2019/045091

The accuracy of deep learning analysis depends on the training data. However, creating accurate training data is not easy. In addition, when targeting a school of fish, etc., there are many individuals that overlap and cannot be seen, so it is difficult to accurately estimate the necessary information using only deep learning.

Accordingly, one object of the present disclosure is to propose an information processing method, a program, and an information processing apparatus that can easily generate accurate training data. Another object of the present disclosure is to propose an information processing method, a program, and an information processing apparatus capable of accurately estimating information included in actual data.

According to one aspect of the present disclosure, a simulation image as input data is generated by computer graphics, label information is generated based on parameter information used in the simulation, and correct data including the simulation image and the label information is generated. A computer-implemented information processing method is provided comprising: generating a Further, according to the present disclosure, there are provided a program that causes a computer to execute the information processing method, and an information processing apparatus that executes the information processing method.

According to the present disclosure, accurate training data is easily generated.

FIG. 1 is a diagram for explaining an outline of information processing. FIG. 2 is a diagram showing variations of image elements. FIG. 3 is a diagram showing variations of image elements. FIG. 4 is a diagram showing variations of image elements. FIG. 5 is a diagram explaining machine learning for removing noise caused by a reflected image. FIG. 6 is a diagram illustrating machine learning for removing noise caused by shadows. FIG. 7 is a diagram showing an example of the flock model 23. As shown in FIG. FIG. 8 is a simulation image of a flock without feeding. FIG. 9 is a simulation image of a flock during feeding. FIG. 10 is a photographed image of a flock during non-feeding. FIG. 11 is a photographed image of a flock during feeding. FIG. 12 is a diagram showing an example of an individual model. FIG. 13 is a diagram showing a motion simulation of an individual. FIG. 14 is a diagram showing a sturgeon reared in a cage. FIG. 15 is a diagram illustrating an example of an information processing apparatus; FIG. 16 is a diagram illustrating an example of the process of estimating the number of individuals in a herd. FIG. 17 is a diagram showing an example of individual tracking. FIG. 18 is a diagram showing an example of an individual identification method and an abnormal behavior detection method. FIG. 19 is a flow chart showing an example of an information processing method. FIG. 20 is a diagram illustrating an example of a hardware configuration of an information processing apparatus;

Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Overview of information processing]
FIG. 1 is a diagram illustrating an overview of information processing according to the present disclosure.

The present disclosure proposes a new method of generating training data using CG (Computer Graphics) and a Data Driven method that fills in the gap between CG images and actual data (captured images). By generating training data from CG images, it is possible to generate a large amount of highly accurate training data. Further, by generating a CG image that approximates actual data, necessary information can be accurately estimated based on CG parameters. Details of the present disclosure are described below.

In the present disclosure, the target TG is analyzed by an image analysis method based on the captured image CI of the target TG. The information processing techniques of the present disclosure are applicable to a wide range of fields such as fisheries, agriculture, forestry and industry. The subject of analysis is appropriately set according to the technical field, purpose of analysis, and the like. For example, in the fisheries industry, the number of farmed fish, growth conditions, feeding behavior, abnormal behavior, and the like are subject to analysis. In the field of agriculture, the growth conditions and harvest numbers of crops are analyzed. In the field of forestry, the growth status of trees, the amount of wood, and the like are analyzed. In the field of industry, the quantity of products, the presence or absence of defective products, etc. are subject to analysis.

Examples in which the information processing method of the present disclosure is applied to farmed fish tracking, abnormal behavior detection, fish number counting, and the like will be described below.

[2. Generate training data]
In the present disclosure, computer simulation is used to generate three-dimensional CG images of a flock FLK and an individual OB. The generated CG images are used as training data for deep learning.

Training data includes multiple datasets. Each data set includes input data TID and correct answer data TOD. In the present disclosure, a simulation image SI as input data TID is generated by computer graphics. The correct data TOD is generated by adding label information to the input data TID. The label information is information indicating the correct answer (Ground Truth). Label information is automatically generated based on parameter information used in the simulation.

The object of simulation is determined according to the purpose of learning. For example, when counting the number of farmed fish, a flock FLK simulation is performed. Individual OB simulations are performed when individual farmed fish are tracked.

The example in FIG. 1 is an example in which a simulation image SI of an individual OB is generated for the purpose of tracking farmed fish. The label information includes classification information for classifying the individual OB and position information of the individual OB. The classification information includes, for example, information such as the type of individual OB (fish species, etc.) and morphological characteristics (fin deformation, loss, body color, pattern, etc.). The position information includes, for example, information indicating the center position of the individual OB and the existence range of the individual OB. Position information is generated using, for example, three-dimensional coordinate information. In the example of FIG. 1, the coordinates (b _x , b _y , b _z ) of the center of the individual OB, and the depth b _d , width b _w and Information about the height b _h is shown as position information. Information on the size of the individual OB is extracted based on the position information (information indicating the existence range) of the individual OB.

The correct answer data TOD can include an image element IE for identifying the individual OB. The image element IE is, for example, a frame element representing a boundary cube. The image element IE is automatically generated based on the position information of the individual OB used in the simulation. In the example of FIG. 1, the image element IE is indicated by the boundary cube, but the image element IE is not limited to this. The image element IE can include, for example, a frame element surrounding the individual OB or a color element for identifying the individual OB by color.

2 to 4 are diagrams showing variations of the image element IE.

The example in FIG. 2 is a reproduction of the boundary cube shown in FIG. The example in FIG. 3 is an example in which the individual OB is replaced with a silhouette, or an example in which the outline of the individual OB is outlined with a thick line. The example in FIG. 4 is an example in which a plurality of individuals OB are color-coded. The boundary cube in FIG. 2 and the outline in FIG. 3 correspond to frame elements surrounding the individual OB. The color coding in FIG. 4 corresponds to the addition of color elements. When there are a plurality of individual OBs, the color of the frame element may be different for each individual OB, or may be different according to the classification information of the individual OB.

[3. deep learning]
Returning to FIG. 1, in the present disclosure, machine learning (supervised learning) using input data TID and correct answer data TOD as training data is performed in a DNN (Deep Neural Network). When the input data RID is input to the learned DNN, the output data ROD is generated by adding the label information and the image element IE to the input data RID.

For example, when a captured image CI is input to the DNN, the DNN extracts individual individuals OB from the captured image CI. The DNN generates classification information and location information for each individual OB based on the characteristics of each individual OB. The DNN generates label information including classification information and location information for each individual OB. The DNN generates an image element IE for each individual OB based on the positional information of each individual OB. The DNN generates a corrected image CCI by adding the image element IE of each individual OB to the photographed image CI. The DNN adds label information of each individual OB to the corrected image CCI to generate output data ROD.

[4. [Remove noise]
5 and 6 are diagrams for explaining machine learning for removing noise.

In order to improve the accuracy of image analysis by DNN, machine learning can be performed to remove noise contained in the captured image CI. For example, when a camera is placed in a fish tank and a photograph is taken, a reflected image MIR of an individual OB moving near the water surface WS and the wall surface WL is projected on the water surface WS and the wall surface WL. The shadow SH of the individual OB is projected on the bottom of the fish tank. Such reflected images MIR and shadows SH become noise and may reduce the accuracy of image analysis. Therefore, in the present disclosure, for example, a simulation image SI of the reflection image MIR and the shadow SH is generated, and training data using the simulation image SI is generated.

In the example of FIG. 5, a simulation image SI including the individual OB and the reflected image MIR is generated as the input data TID. The correct data TOD is a simulation image SI obtained by removing the reflected image MIR from the input data TID. By using the input data TID and the correct data TOD as training data, a DNN that removes the reflected image MIR from the captured image CI is obtained. In the example of FIG. 5, the individual OB is included in the input data TID and the correct data TOD, but these data do not necessarily include the individual OB. For example, even if training data in which individuals OB are deleted from the input data TID and correct data TOD described above is used, a DNN that removes the reflected image MIR is generated.

In the example of FIG. 6, a simulation image SI including an individual OB and a shadow SH is generated as the input data TID. The correct data TOD is a simulation image SI obtained by removing the shadow SH from the input data TID. By using the input data TID and the correct data TOD as training data, a DNN that removes the shadow SH from the captured image CI is obtained. In the example of FIG. 6, the individual OB is included in the input data TID and the correct data TOD, but these data do not necessarily include the individual OB. For example, a DNN that removes the shadow SH is generated using training data in which the individuals OB are deleted from the input data TID and correct data TOD described above.

[5. simulation model]
A flock FLK and an individual OB simulation image SI can be generated using a flock model 23 (see FIG. 15) and an individual model 24 (see FIG. 15). The herd model 23 is a simulation model based on the statistical properties of the herd FLK. The individual model 24 is a simulation model based on the morphological features of the individual OB. An outline of the simulation model will be described below.

[5-1. Herd model]
FIG. 7 is a diagram showing an example of the flock model 23. As shown in FIG.

The flock model 23 is generated, for example, by combining a Boids model, which defines three rules of behavior for classification, alignment, and association, with a predator model that indicates predatory behavior during feeding. In the predator model, a repulsive area RA is set outside the congested area CA to which the segregation operation rule is applied to encourage movement in the direction opposite to the dense direction. A predator model is generated by simulating how an individual OB obtains food reflected in the field of view of the individual OB (Fish Vision Simulation). The predator model can also be used to optimize feeding according to individual OB type and cage density.

In the flock model 23, for example, the degree of alignment of individuals OB, the degree of flock formation, the viewing angle, the top speed, the food recognition speed, the degree of attachment to food, the time to continue eating food (the amount of food that satisfies), and the amount of food to feed at one time. The time of picking, the speed of movement while eating, the size of the congestion area CA, the size of the repulsion area RA, and the like are set as parameters of the flock FLK. These parameters change depending on the type and size of the individual OB.

FIG. 8 is a simulation image SI of the flock FLK during non-feeding. FIG. 9 is a simulation image SI of a flock FLK during feeding. FIG. 10 is a photographed image CI of the flock FLK during non-feeding. FIG. 11 is a photographed image CI of the flock FLK during feeding.

As shown in FIGS. 8 and 10, when not feeding, the flock FLK generally operates according to three rules of operation: separation, alignment and union. Therefore, a large number of individuals OB are concentrated in a narrow range. As shown in FIGS. 9 and 11, during feeding, the flock FLK operates according to four rules of operation: separation, alignment, association and repulsion. The density of the flock FLK decreases because the flock FLK spreads due to the rules of action of repulsion.

As shown in FIG. 10, in a dense flock FLK, many individuals OB overlap and cannot be seen, so it is difficult to accurately count the number of individuals. Therefore, the number of individuals is estimated based on the simulation image SI as shown in FIG. The shape and size of the flock FLK can be changed by adjusting simulation parameters. Therefore, by adjusting the parameter information until the simulation image SI approximates the actual data, it is possible to estimate the number of individuals with high accuracy.

As shown in Figure 11, the density of flock FLK decreases during feeding. Therefore, the number of individuals may be estimated based on the images captured during feeding. In this case, the number of overlapping individuals OB that cannot be seen is reduced, which facilitates counting. Also when comparing with the simulation image SI shown in FIG. 9, the flock FLK is not dense, so the comparison is easy.

[5-2. individual model]
FIG. 12 is a diagram showing an example of the individual model 24. As shown in FIG.

The individual model 24 is generated based on the body structure of the individual OB. As the individual model 24, for example, a musculoskeletal model is used. A musculoskeletal SBM is obtained by dissecting an individual OB of standard shape and size and analyzing the skeleton of the individual OB and the attachment of muscles to the skeleton. In FIG. 12 the sturgeon musculoskeletal SBM is shown. A reference model RM of the individual OB is reproduced from the musculoskeletal SBM. One or more mutation models MM1 to MMn (n is an integer of 1 or more) are derived and generated from the reference model RM. The reference model RM and the mutant models MM1 to MMn are individual models 24, respectively. The individual model 24 is used as classification information for the individual OB.

A mutation model is a three-dimensional model of an individual OB with morphological abnormalities. It is known that most of the farm-born OB individuals have physical abnormalities (eg, curved fins, altered fin shape, visceral disease). Therefore, mutation models of individual OB include, for example, pectoral fin abnormalities (e.g., pectoral fins are bent), dorsal fin abnormalities (e.g., dorsal fins are bent), and tail fin abnormalities (e.g., tail fins are bent). curved), and contour variation (eg, the contour of the head is mutated compared to a normal individual). Note that the mutation model is not limited to the above. What kind of mutation model is to be generated is determined individually according to the type of individual OB and the environment in which the individual OB is placed.

FIG. 13 is a diagram showing a motion simulation of an individual OB.

The individual model 24 is associated with motion feature amounts obtained by motion simulation. In the example of sturgeon, a swimming feature amount (movement feature amount) is extracted for each individual model 24 by fluid simulation. The swimming style of the individual OB changes depending on the morphological characteristics of the individual OB. For example, an individual OB with a physical abnormality swims to compensate for the abnormality. Therefore, the feature amount of the swimming style differs for each individual model 24 . The swimming style feature amount can be used as parameter information for identifying the individual OB. The parameter information indicating the behavior feature amount of the individual OB is linked to the individual model 24 and used as classification information.

FIG. 14 is a diagram showing a sturgeon reared in a cage.

Most sturgeons on farms have some physical abnormalities. For example, individuals OB1 to OB4 do not have uniform pectoral fins, dorsal fins, and tail fins, and have slightly different contours. Therefore, each sturgeon can be assigned a mutation model according to its physical abnormalities. Sturgeons assigned the same mutation model show varying degrees of physical abnormalities. Therefore, the parameter information indicating the motion feature amount of the individual OB also differs for each sturgeon. Therefore, if the parameter information of each individual OB is detected by motion simulation, each individual OB can be identified based on the detected parameter information.

[6. Configuration of information processing device]
FIG. 15 is a diagram showing an example of the information processing device 1 to which the information processing of the present disclosure is applied.

The information processing device 1 is a dedicated or general-purpose computer system such as a server. The DNN for image analysis described above is included in the analysis model 21 . The information processing device 1 has a processing device 10 and a storage device 20 . The processing device 10 includes, for example, an input data generation unit 11, a correct data generation unit 12, an image analysis unit 13, a simulation unit 14, and an estimation unit 15.

The simulation unit 14 uses the simulation model defined in the model information 22 to generate a three-dimensional simulation image SI. The simulation image SI is obtained by reproducing the simulation result as an image viewed from a certain camera angle using CG technology. The model information 22 defines, for example, a group model 23, an individual model 24, an abnormal behavior model 25, and the like. The abnormal behavior model 25 is a simulation model that indicates abnormal behavior of the individual OB. The simulation unit 14 generates a simulation image SI of the herd FLK and the individual OB based on these simulation models.

The input data generation unit 11 generates parameter information necessary for simulation and outputs it to the simulation unit 14 . The input data generation unit 11 uses the simulation unit 14 to generate a simulation image SI as the input data TID by computer graphics.

The correct data generation unit 12 generates label information based on the parameter information used in the simulation. The correct data generation unit 12 generates correct data TOD including the simulation image SI and label information. The correct data generation unit 12 can generate an image element IE for identifying an individual OB based on the position information of the individual OB used in the simulation. In this case, the correct data generation unit 12 generates a corrected image CSI by adding the image element IE to the simulation image SI, and adds label information to the corrected image CSI to generate correct data TOD.

A lot of training data is generated by the input data generator 11 and the correct data generator 12 . The analysis model 21 is generated based on machine learning using the above-described input data TID and correct data TOD as training data. The analysis model 21 includes, for example, a DNN for analyzing classification information and position information of an individual OB, a DNN for analyzing parameter information for a flock FLK, a DNN for analyzing parameter information for an individual OB, and a reflection image MIR. DNN for removing and DNN for removing shadow SH. These DNNs are trained appropriately according to the required functions. There are various machine learning methods, and RNN (Recurrent Neural Network), Deep Reinforcement Learning, Transformer, and the like can be used.

The image analysis unit 13 performs image analysis of the captured image CI using the analysis model 21 . The image analysis unit 13 extracts label information from the captured image CI by image analysis. The image analysis unit 13 extracts necessary parameter information from the extracted label information.

The estimation unit 15 outputs parameter information extracted by the image analysis unit 13 to the simulation unit 14 . The estimation unit 15 uses the simulation unit 14 to generate a simulation image SI based on the extracted parameter information. The estimation unit 15 calculates the difference between the simulation image SI and the captured image CI. If the difference does not satisfy the preset condition, the estimation unit 15 corrects the parameter information and generates the simulation image SI. The estimation unit 15 repeats the correction of the parameter information and the generation of the simulation image SI until the difference satisfies the above conditions. The estimation unit 15 estimates necessary information based on the parameter information corrected until the above conditions are satisfied.

The storage device 20 stores, for example, a program 29 executed by the processing device 10, an analysis model 21, and model information 22. FIG. The program 29 is a program that causes a computer to execute the information processing of the present disclosure. The processing device 10 performs various processes according to programs 29 stored in the storage device 20 . The storage device 20 may be used as a work area that temporarily stores the processing results of the processing device 10 . Storage device 20 includes, for example, any non-transitory storage media such as semiconductor storage media and magnetic storage media. The storage device 20 includes, for example, an optical disc, a magneto-optical disc, or a flash memory. The program 29 is stored, for example, in a non-transitory computer-readable storage medium.

The processing device 10 is, for example, a computer configured with a processor and memory. The memory of the processing device 10 includes RAM (Random Access Memory) and ROM (Read Only Memory). The processing device 10 functions as an input data generation unit 11 , a correct data generation unit 12 , an image analysis unit 13 , a simulation unit 14 and an estimation unit 15 by executing the program 29 .

[7. Estimation process of the number of individuals]
FIG. 16 is a diagram showing an example of the process of estimating the number of individuals in a flock FLK.

The analytical model 21 has DNNA, DNNB, DNNC and DNND. DNN is a DNN for removing reflected image MIR. DNNB is a DNN for removing shadow SH. DNNC is a DNN for extracting location information and classification information of an individual OB. DNND is a DNN for extracting parameter information of the flock FLK.

A simulation image SI for DNND training is a simulation image of the flock FLK generated using the flock model 23 . The label information of the DNND training data is generated based on the parameter information of the flock FLK applied to the flock model 23 . When the photographed image CI of the flock FLK is input to the DNND, the parameter information of the flock FLK is extracted. The DNNC extracts information such as the type, position and size of each individual OB from the captured image CI. DNNA and DNNB separate noise components (reflection image MIR, shadow SH) included in the captured image CI.

The image analysis unit 13 uses the analysis model 21 to extract label information from the photographed image CI of the flock FLK. The image analysis unit 13 extracts the parameter information of the flock FLK based on the extracted label information. The estimation unit 15 estimates the number of individuals OB included in the flock FLK based on the parameter information of the flock FLK.

In the estimation step described above, the estimation unit 15 first generates a simulation image SI of the flock FLK based on the parameter information of the flock FLK and the information on the type, position and size of the individual OB. The estimation unit 15 corrects the parameter information of the flock FLK until the difference diff between the simulation image SI and the captured image CI of the flock FLK satisfies a preset condition. The difference diff is calculated based on the difference between the feature amount of the captured image CI and the feature amount of the simulation image SI. The estimation unit 15 estimates the number of individuals OB based on the corrected parameter information of the flock FLK until the above condition is satisfied.

The simulation image SI generated in the estimation step includes the reflected image MIR and shadow SH of the individual OB. The number of reflected images MIR and the density of shadows SH may be clues for obtaining information about the flock FLK in the blind spots of the camera. Therefore, the difference diff also includes the difference between the reflected image MIR and shadow SH included in the simulation image SI and the reflected image MIR and shadow SH included in the captured image CI. It is possible to estimate the number of individuals with high accuracy by also considering the difference information about the reflected image MIR and the shadow SH.

In the example of FIG. 16, the images during non-feeding are used as the photographed image CI and the simulation image SI. However, the photographed image CI and the simulated image SI can also be images of the flock FLK during feeding. During feeding when the predator model is applied, the flock FLK density is lower than during non-feeding when the predator model is not applied. As a result, the number of individuals OB that overlap and cannot be seen is reduced. Therefore, the difference between the simulation image SI and the photographed image CI can be detected with high accuracy, and the estimation accuracy of the number of individuals also increases.

Note that the DNNC detects the type and size of the individual OB. Therefore, the information processing apparatus 1 may include input means for inputting the types and sizes to be counted. This allows the user to count only individuals OB of a specific type or size.

[8. individual tracking]
FIG. 17 is a diagram showing an example of individual tracking.

The analytical model 21 has DNNA, DNNB, DNNC and DNNE. A DNNE is a DNN that adds an image element IE to each individual OB based on the location information and classification information of each individual OB extracted by the DNNC.

The image analysis unit 13 applies the photographed image CI to the analysis model 21 and adds an image element IE to each individual OB appearing in the photographed image CI. In the example of FIG. 17, a different color element is attached to each individual OB as the image element IE. The color of the individual OB differs depending on the size, shape, etc. of the individual OB. The size of the individual OB is extracted from the position information of the individual OB. The shape of the individual OB is extracted from the classification information (morphological features) of the individual OB. The image analysis unit 13 detects a plurality of individuals OB based on the image elements IE attached to the individuals OB, and individually tracks each individual OB.

[9. Identification of Individuals and Recognition of Abnormal Behavior]
FIG. 18 is a diagram showing an example of an individual identification method and an abnormal behavior detection method.

The analytical model 21 has DNNA, DNNB, DNNC and DNNF. DNNF is a DNN for extracting parameter information of an individual OB.

A simulation image SI for DNNF training is a simulation image of an individual OB generated using the individual model 24 . The label information of the DNNF training data is generated based on the parameter information of the individual OB applied to the individual model 24 . When the captured image CI of the individual OB is input to the DNNF, parameter information corresponding to the morphological features (individual model 24) of the individual OB is extracted.

The image analysis unit 13 uses the analysis model 21 to extract the label information of each individual OB from the captured image CI. The image analysis unit 13 extracts the parameter information of each individual OB based on the label information of the individual OB. The extracted parameter information of the individual OB includes, for example, parameter information of muscle parameters applied to the musculoskeletal model of the individual OB, and parameter information linked to the individual model 24 and indicating the motion feature amount of the individual OB. is included. The estimation unit 15 identifies each individual OB based on the parameter information of each individual OB.

In the identification step described above, the estimation unit 15 first generates a simulation image SI of the individual OB based on the parameter information of the individual OB. The estimation unit 15 corrects the parameter information of the individual OB until the difference between the simulation image and the captured image CI of the individual OB satisfies a preset condition. The difference diff is calculated based on the difference between the feature amount of the captured image CI and the feature amount of the simulation image SI. The estimation unit 15 identifies the individual OB based on the parameter information of the individual OB that has been corrected until the above conditions are satisfied.

Abnormal behavior can be recognized by simulation using the retinal parameters of the individual OB. The retina parameter is, for example, a parameter that indicates the relationship between the direction of occurrence of an event seen from the individual OB and the direction of movement of the individual OB. For example, when food falls in front of an individual OB, if the individual OB moves in a direction completely unrelated to the direction in which the food is present, it can be estimated that the individual OB has an abnormality. In addition, sturgeon with visceral disease may exhibit abnormal behavior such as unnaturally drifting near the surface of the water without showing interest in food. In such a case, by applying the photographed image CI to the abnormal behavior model 25, extracting the retinal parameters, and comparing the retinal parameters with preset reference values, the abnormality of the individual OB can be detected.

Detecting abnormal behavior is also useful for detecting poor physical condition due to overfeeding. Therefore, it is possible to adjust or optimize the feeding amount based on the abnormal behavior detection result.

[10. Information processing method]
FIG. 19 is a flow chart showing an example of the information processing method of the present disclosure.

In step S1, the input data generation unit 11 generates a simulation image SI as input data TID by CG.

In step S2, the correct data generation unit 12 generates label information based on the parameter information used in the simulation.

In step S3, the correct data generation unit 12 generates correct data TOD including the simulation image SI and label information.

In step S4, the processing device 10 generates the analysis model 21 based on machine learning using the input data TID and the correct data TOD as training data.

In step S5, the image analysis unit 13 uses the analysis model 21 to extract label information from the captured image CI.

In step S6, the image analysis unit 13 extracts necessary parameter information from the extracted label information.

In step S<b>7 , the parameter information extracted by the image analysis unit 13 is output to the simulation unit 14 . The estimation unit 15 uses the simulation unit 14 to generate a simulation image SI based on the extracted parameter information.

In step S8, the estimation unit 15 calculates the difference between the simulation image SI and the captured image CI, and if the difference does not satisfy a preset condition, corrects the parameter information and generates the simulation image SI. The estimation unit 15 repeats the correction of the parameter information and the generation of the simulation image SI until the difference satisfies the above conditions.

In step S9, the estimation unit 15 estimates necessary information based on the corrected parameter information until the above conditions are satisfied.

[11. Hardware configuration]
FIG. 20 is a diagram showing an example of the hardware configuration of the information processing device 1. As shown in FIG.

The information processing apparatus 1 is implemented by, for example, a computer system 1000 as shown in FIG. Computer system 1000 has CPU 1100 , RAM 1200 , ROM 1300 , HDD 1400 , communication interface (I/F) 1500 , input/output interface (I/F) 1600 and media interface (I/F) 1700 .

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer system 1000 is started, a program depending on the hardware of the computer system 1000, and the like.

HDD 1400 stores programs executed by CPU 1100 and data used by such programs. Communication interface 1500 receives data from other devices via communication network NW, sends the data to CPU 1100, and transmits data generated by CPU 1100 to other devices via communication network NW.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, through an input/output interface 1600 . CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200 . CPU 1100 loads such a program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer system 1000 functions as the information processing device 1 of the present disclosure, the CPU 1100 of the computer system 1000 implements the functions of the processing device 10 by executing programs loaded onto the RAM 1200 . Data in the storage device 20 is stored in the HDD 1400 . CPU 1100 of computer system 1000 reads and executes these programs from recording medium 1800, but as another example, these programs may be obtained from another device via communication network NW.

[12. effect]
As described above, the information processing method of the present disclosure has an input data generation step and a correct data generation step. The input data generation step generates a simulation image SI as input data TID by computer graphics. The correct data generation step generates label information based on the parameter information used in the simulation, and generates correct data TOD including the simulation image and the label information. The program 29 of the present disclosure causes the computer to implement the information processing described above. The information processing device 1 of the present disclosure performs the information processing described above.

According to this configuration, the label information of the correct data TOD is generated based on the parameter information used in the simulation. Therefore, the accuracy of label information is high. Therefore, highly accurate training data can be easily generated.

The information processing method of the present disclosure has an analysis model generation step. The analysis model generation step generates the analysis model 21 based on machine learning using the input data TID and the correct data TOD as training data.

According to this configuration, CG coordinate information and the like used to generate the input data TID are reflected in the label information. Therefore, the analysis model 21 obtained by training accurately extracts the position, shape and size of the individual OB captured in the photographed image CI, parameter information of the flock FLK, and the like.

The information processing method of the present disclosure includes a label information extraction step, a parameter information extraction step, a simulation image generation step, a parameter information correction step, and an estimation step. The label information extraction step uses the analysis model 21 to extract label information from the captured image CI. The parameter information extraction step extracts parameter information based on the label information. The simulation image generation step generates a simulation image SI based on the parameter information. The parameter information correcting step corrects the parameter information until the difference between the simulation image and the captured image CI satisfies a preset condition. The estimation step estimates information based on the corrected parameter information until the above conditions are satisfied.

According to this configuration, the parameter information extracted using the analysis model 21 is corrected based on the difference information between the simulation image SI and the captured image CI. Therefore, parameter information that matches well with the actual data (captured image CI) can be obtained. Then, since the necessary information is estimated based on the corrected parameter information, the information estimation accuracy is enhanced.

The simulation image SI is a simulation image of the flock FLK generated using the flock model 23 based on the statistical properties of the flock FLK. The label information is generated based on the flock FLK parameter information applied to the flock model 23 .

According to this configuration, the parameter information of the flock FLK is accurately extracted from the photographed image CI of the flock FLK.

The information processing method of the present disclosure has a label information extraction step, a parameter information extraction step, and an estimation step. The label information extraction step uses the analysis model 21 to extract label information from the photographed image CI of the flock FLK. The parameter information extraction step extracts parameter information of the flock FLK based on the extracted label information. The estimation step estimates the number of individuals OB included in the flock FLK based on the parameter information of the flock FLK.

According to this configuration, the population of the flock FLK can be estimated with high accuracy.

The estimation step has a simulation image generation step, a parameter information correction step, and a fixed number estimation step. The simulation image generation step generates a simulation image SI of the flock FLK based on the parameter information of the flock FLK. The parameter information correcting step corrects the parameter information of the flock FLK until the difference between the flock FLK and the simulation image SI photographed image CI satisfies a preset condition. The fixed number estimation step estimates the number of individuals OB based on the corrected parameter information of the flock FLK until the above conditions are satisfied.

According to this configuration, the parameter information extracted using the analysis model 21 is corrected based on the difference information between the simulation image SI and the captured image CI. Therefore, parameter information that matches well with the actual data (captured image CI) can be obtained. Since the number of individuals is estimated based on the corrected parameter information, the number of individuals can be estimated with high accuracy.

A simulation image SI of the flock FLK is a simulation image of the flock FLK during feeding.

According to this configuration, the density of the flock FLK is lower during feeding when the predator model is applied than during non-feeding when the predator model is not applied. As a result, the number of individuals OB that overlap and cannot be seen is reduced. Therefore, the difference between the simulation image SI and the photographed image CI can be detected with high accuracy, and the estimation accuracy of the number of individuals also increases.

The simulation image SI is a simulation image of the individual OB generated using the individual model 24 based on the morphological features of the individual OB. Label information is generated based on parameter information of the individual OB applied to the individual model 24 .

According to this configuration, the parameter information of the individual OB is accurately extracted from the photographed image CI of the individual OB.

The information processing method of the present disclosure has a label information extraction step, a parameter information extraction step, and an individual identification step. The label information extraction step uses the analysis model 21 to extract the label information of each individual OB from the photographed image CI. The parameter information extraction step extracts the parameter information of each individual OB based on the label information of the individual OB. The individual identification step identifies each individual OB based on the parameter information of each individual OB.

According to this configuration, the individual OB is accurately identified.

The identification step has a simulation image generation step, a parameter information correction step, and an individual identification step. The simulation image generation step generates a simulation image SI of the individual OB based on the parameter information of the individual OB. The parameter information correcting step corrects the parameter information of the individual OB until the difference between the simulation image SI and the photographed image CI of the individual OB satisfies a preset condition. The individual identification step identifies the individual OB based on the corrected parameter information of the individual OB until the above conditions are satisfied.

According to this configuration, the parameter information extracted using the analysis model 21 is corrected based on the difference information between the simulation image SI and the captured image CI. Therefore, parameter information that matches well with the actual data (captured image CI) can be obtained. Since the individual OB is identified based on the corrected parameter information, the identification accuracy of the individual OB is high.

The parameter information of the individual OB includes parameter information of muscle parameters applied to the musculoskeletal model of the individual OB and parameter information indicating the motion feature amount of the individual OB.

According to this configuration, individual OB can be identified with high accuracy based on both the morphological features and movement features of individual OB.

The information processing method of the present disclosure has an anomaly detection step. In the anomaly detection step, the photographed image CI of the individual OB is applied to the abnormal behavior model 25 to detect an anomaly of the individual OB.

According to this configuration, it is possible to grasp the health condition of the identified individual OB.

The correct data generation step has an image element generation step, a corrected image generation step, and a data generation step. The image element generation step generates an image element IE for identifying the individual OB based on the positional information of the individual OB used in the simulation. The corrected image generation step generates a corrected image CSI by adding the image element IE to the simulation image SI. The data generation step adds label information to the corrected image CSI to generate correct data TOD.

According to this configuration, the image element IE is accurately added to an appropriate position of the simulation image SI. In addition, since the work of adding the image element IE is automated, the work of generating the correct data TOD becomes easier.

The image element includes a frame element surrounding the individual OB or a color element for identifying the individual OB by color.

With this configuration, the individual OB is clearly distinguished from other image regions.

The information processing method of the present disclosure has an image element adding step and a tracking step. The image element addition step applies the photographed image CI to the analysis model 21 and adds image elements IE to individual individuals OB captured in the photographed image CI. The tracking step individually tracks each individual OB based on the image element IE attached to each individual OB.

This configuration facilitates tracking using the image element IE as a clue.

[13. others]
Each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

As described above, the embodiments of the present application have been described in detail based on several drawings, but these are examples, and various modifications and It is possible to carry out the invention in other forms with modifications.

Although the method of generating training data using CG has been described in the above embodiment, this method contributes to improving the recognition rate of all types of computer vision, not limited to the examples described above. The elaborate musculoskeletal simulation method using CG can be applied not only to sturgeon but also to humans and horses. As a result, the above technique using AI (Artificial Intelligence) can be widely applied to various fields.

1 Information processing device 21 Analysis model 23 Group model 24 Individual model 25 Abnormal behavior model 29 Program CI Photographed image CSI Corrected image FLK Group IE Image element OB Individual SI Simulation image TID Input data TOD Correct data

Claims (9)

generating a simulation image of the flock viewed from a camera angle by simulating the flock with input of parameter information indicating properties of the flock to be analyzed;
generating label information associated with the simulation image based on the parameter information input when the simulation image was generated;
generating an analysis model based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data;
extracting parameter information from the photographed image based on the photographed image of the flock and the analysis model;
performing the simulation based on the parameter information extracted from the captured image to generate a comparison simulation image for comparison with the captured image;
correcting the parameter information extracted from the captured image until the difference between the comparison simulation image and the captured image satisfies a predetermined condition;
estimating the number of individuals included in the herd in the captured image based on the corrected parameter information;
A computer-implemented information processing method. generating a simulation image of the school of fish viewed from a camera angle by simulating the school of fish with input of parameter information indicating properties of the school of fish;
generating label information associated with the simulation image based on the parameter information input when the simulation image was generated;
generating an analysis model based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data;
extracting label information from the captured image based on the captured image of the fish school and the analysis model;
performing an estimation step of estimating the number of individuals included in the fish school based on the label information extracted from the captured image;
The simulation includes at least the perspective of how the fish that make up the fish school react to food,
A computer-implemented information processing method. generating a simulation image of the school of fish viewed from a camera angle by simulating the school of fish with input of parameter information indicating properties of the school of fish;
generating label information associated with the simulation image based on the parameter information input when the simulation image was generated;
generating an analysis model based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data;
extracting label information from the captured image based on the captured image of the fish school and the analysis model;
performing an estimation step of estimating the number of individuals included in the fish school based on the label information extracted from the captured image;
The simulation includes at least the viewpoint of the viewing angle of the fish that make up the school of fish,
A computer-implemented information processing method. generating a simulation image of the school of fish viewed from a camera angle by simulating the school of fish with input of parameter information indicating properties of the school of fish;
generating label information associated with the simulation image based on the parameter information input when the simulation image was generated;
generating an analysis model based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data;
extracting parameter information from the captured image based on the captured image of the fish school and the analysis model;
performing an estimation step of estimating the number of individuals included in the fish school based on the parameter information extracted from the captured image;
The parameter information includes at least the size of the crowded area in the school of fish, and the size of the crowded area is a parameter that varies depending on the species of fish that make up the school of fish.
A computer-implemented information processing method. A program that causes a computer to execute the information processing method according to any one of claims 1 to 4. a simulation unit that generates a simulation image of the flock viewed from a camera angle by simulating the flock with input of parameter information indicating properties of the flock to be analyzed;
a correct data generation unit that generates label information associated with the simulation image based on the parameter information that was input when the simulation image was generated;
Based on an analysis model generated based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data , and a photographed image of the flock, the photographed an image analysis unit that extracts parameter information from an image;
an estimating unit that estimates the number of individuals included in the herd in the captured image based on the parameter information extracted from the captured image;
has
The estimation unit
performing the simulation based on the parameter information extracted from the captured image to generate a comparison simulation image for comparison with the captured image;
correcting the parameter information extracted from the captured image until the difference between the comparison simulation image and the captured image satisfies a predetermined condition;
estimating the number of individuals included in the herd in the captured image based on the corrected parameter information;
Information processing equipment. a simulation unit that generates a simulation image of the school of fish viewed from a camera angle by simulating the school of fish with input of parameter information indicating the properties of the school of fish;
a correct data generation unit that generates label information associated with the simulation image based on the parameter information that was input when the simulation image was generated;
Based on an analysis model generated based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data, and a captured image of the school of fish, an image analysis unit that extracts label information from an image;
an estimation unit that estimates the number of individuals included in the fish school based on the label information extracted from the captured image;
has
The simulation includes at least the perspective of how the fish that make up the fish school react to food,
Information processing equipment. a simulation unit that generates a simulation image of the school of fish viewed from a camera angle by simulating the school of fish with input of parameter information indicating the properties of the school of fish;
a correct data generation unit that generates label information associated with the simulation image based on the parameter information that was input when the simulation image was generated;
Based on an analysis model generated based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data, and a captured image of the school of fish, an image analysis unit that extracts label information from an image;
an estimation unit that estimates the number of individuals included in the fish school based on the label information extracted from the captured image;
has
The simulation includes at least the viewpoint of the viewing angle of the fish that make up the school of fish,
Information processing equipment. a simulation unit that generates a simulation image of the school of fish viewed from a camera angle by simulating the school of fish with input of parameter information indicating the properties of the school of fish;
a correct data generation unit that generates label information associated with the simulation image based on the parameter information that was input when the simulation image was generated;
Based on an analysis model generated based on machine learning using a bundle of correct data generated based on the simulation image and the label information as training data, and a captured image of the school of fish, an image analysis unit that extracts parameter information from an image;
an estimation unit that estimates the number of individuals included in the fish school based on the parameter information extracted from the captured image;
has
The parameter information includes at least the size of the crowded area in the school of fish, and the size of the crowded area is a parameter that varies depending on the species of fish that make up the school of fish.
Information processing equipment.

Images