JP7166951B2 - Learning request device, learning device, inference model utilization device, inference model utilization method, inference model utilization program, and imaging device - Google Patents

Description

TECHNICAL FIELD The present invention relates to a learning request device, a learning device, an inference model utilization device, an inference model utilization method, an inference model utilization program, and an imaging device, which pay attention to consistency with usage conditions.

2. Description of the Related Art Conventionally, a rule-based control device that performs processing according to a rule written by a person using a logic circuit may be employed. Various problems can be solved by making inferences based on rule-based control. Also, in recent years, computer systems that solve various problems using inference models generated by machine learning such as deep learning have come into use. In particular, devices using deep learning can make extremely effective inferences, for example, for image analysis, speech analysis, natural language processing, and the like.

For example, Patent Literature 1 discloses a technique for determining a threshold value of the degree of abnormality for determining the presence or absence of an abnormality when detecting an abnormality. In the device of Patent Document 1, the degree of abnormality is calculated by using the learning result obtained by learning the data collected from the detection target of abnormality.

JP 2018-148350 A

In order to obtain an effective inference model by machine learning, it is necessary to adopt appropriate learning methods and training data. The effectiveness of inference by machine learning is affected by learning, and even with sufficient learning, correct inference may not always be obtained. In some cases, rule-based inference can yield more correct inference results than machine learning inference.

Therefore, it is important to have a mechanism that considers the advantages of rule-based inference and machine learning inference. .

The present invention considers the advantages of rule-based logic circuits as in the past, and provides a learning request device, a learning device, and an inference model that can enable devices that utilize inference by machine learning to function effectively. An object of the present invention is to provide a utilization device, an inference model utilization method, an inference model utilization program, and an imaging device.

A learning request device according to an aspect of the present invention includes a specification setting unit that generates specification information describing settings of a plurality of specification items for requesting specification of an inference model; and a control unit for controlling transmission to the learning device to be generated, and the specification setting unit includes, in the specification information, the inference engine using the inference model in cooperation with other combined inference devices. Include a specification item that sets whether to perform inference.

A learning device according to an aspect of the present invention has specification information describing settings of a plurality of specification items for determining specifications of an inference model, and an inference engine using the inference model cooperates with other concurrent inference devices. and an inference modeling unit that constructs the inference model based on the specification information for performing inference, and the inference setting information on how to cooperate with the combined inference device based on the specification information to construct the inference model. and a control unit that performs control for adding to and transmitting inference model information for.

An inference model utilization device according to an aspect of the present invention is inference model information for constructing an inference model on the premise that an inference engine and a combined inference device cooperate to perform inference, the inference engine and the The communication unit that receives the inference model information to which the inference setting information on how to cooperate with the combined inference device is added, the inference engine that performs inference using the inference model configured based on the inference model information, and a control unit that causes the inference engine and the combined inference device to cooperate and execute inference based on the inference setting information.

An inference model utilization method according to an aspect of the present invention is inference model information for constructing an inference model on the premise that an inference engine and a combined inference device cooperate to perform inference. receiving the inference model information to which the inference setting information about how to cooperate with the combined inference device is added, constructing the inference engine using the inference model configured based on the inference model information, and the inference setting information Based on, the inference engine and the combined inference device are linked to execute inference.

An inference model utilization program according to one aspect of the present invention is inference model information for building an inference model on the premise that an inference engine and a combined inference device cooperate to perform inference in a computer, receiving the inference model information to which inference setting information on how to link the engine and the combined inference device is added, constructing the inference engine using the inference model configured based on the inference model information; Based on the inference setting information, the inference engine and the combined inference device are linked to execute a procedure for executing inference.

An imaging apparatus according to an aspect of the present invention includes inference model information for constructing an inference model on the assumption that an imaging unit that acquires a captured image of a subject, an inference engine, and a combination inference device cooperate to perform inference. using a communication unit that receives the inference model information to which inference setting information on how to link the inference engine and the combination inference device is added, and an inference model configured based on the inference model information. The inference engine and the combined inference device perform inference by using the inference engine, the combined inference device, and the inference setting information. and a control unit for detecting an object.

ADVANTAGE OF THE INVENTION According to this invention, it has the effect that it can enable the apparatus using the inference by machine learning to function effectively.

1 is a block diagram showing a learning request device, a learning device, and an imaging device according to a first embodiment of the present invention; FIG. FIG. 10 is an explanatory diagram showing processing using a rule-based engine; FIG. 10 is an explanatory diagram showing processing using an inference engine; FIG. 10 is an explanatory diagram showing processing using a rule base engine and an inference engine in cooperation; 3 is an explanatory diagram showing an example of a specification setting menu display DM displayed on a display screen 14a of a display unit 14; FIG. FIG. 10 is an explanatory diagram for explaining an example of how the inference engine and the combined inference device are linked; FIG. 10 is an explanatory diagram for explaining an example of how the inference engine and the combined inference device are linked; FIG. 10 is an explanatory diagram for explaining an example of how the inference engine and the combined inference device are linked; FIG. 2 is an explanatory diagram showing how an image is captured by the imaging device 10; 4 is a flowchart for explaining the operation of the learning requesting device 30; 4 is a flowchart for explaining the operation of the learning device 20; FIG. 11 is an explanatory diagram for explaining learning by the learning device 20 in the first technique and an inference model obtained as a result of the learning; 4 is a flowchart for explaining the operation of the imaging device 10; FIG. 4 is an explanatory diagram for explaining inference by the rule base engine 18; 4 is an explanatory diagram for explaining inference by an inference engine 17; FIG. FIG. 10 is an explanatory diagram showing an example of image display based on a series of inference detection results; FIG. 11 is an explanatory diagram for explaining learning by the learning device 20 in the second method and an inference model obtained as a result of the learning; 4 is a flowchart for explaining the operation of the imaging device 10; 4 is a flowchart for explaining the operation of the imaging device 10; It is a block diagram showing a second embodiment. It is a block diagram showing a second embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
The present invention is for effectively utilizing an inference model based on machine learning, and considers compatibility with hardware and devices that use the inference model.
When an inference model is obtained by machine learning, it is necessary to fully consider the difference between the performance of the training computer and the training data given during learning, and the performance of the input data and the electronic circuit during actual use. I can't get the correct inference result. Therefore, it is important to have a learning step that fully considers the constraints when using the inference model. In other words, by devising ways to standardize the "required specifications of the inference model", it is possible to utilize a learning environment with a high degree of freedom and a wealth of training data, enabling even more accurate output under the usage environment. It is possible to obtain a reasonable inference model.

FIG. 1 is a block diagram showing a learning request device, a learning device, and an imaging device according to the first embodiment of the present invention. As an example of effectively utilizing an inference model based on machine learning, the present embodiment will describe an example of utilization of an inference model in a device having an inference device used in addition to an inference engine that uses the inference model. That is, in the present embodiment, in order to realize more effective inference in an apparatus having a rule-based engine, which is a combined inference device that executes inference using a rule base, and an inference engine that executes inference by machine learning, Create specification information for learning, generate inference model information including inference setting information indicating how to use the inference model of the learning result, etc., and use the inference model information to effectively operate the rule-based engine and the inference engine. It is possible to operate Note that FIG. 1 illustrates an imaging device as an example of a device having a rule base engine and an inference engine, but the device is not limited to the imaging device. Moreover, although an example in which a rule-based engine is employed as a combined inference device will be described, an inference engine that executes inference by machine learning, for example, an inference engine whose inference model cannot be updated may be employed.
As an example of using an inference model, an inference engine that uses the inference model performs inference in cooperation with a combined inference device. It is clear from the following explanation that the inference model can be effectively used by creating specification information that describes and generating inference model information that includes inference setting information that indicates how to use the inference model created based on the specification information. is.

First, with reference to FIGS. 2A to 2C, an example of cooperative use of the rule-based engine and the inference engine will be described. FIG. 2A is an explanatory diagram showing processing using the rule-based engine, FIG. 2B is an explanatory diagram showing processing using the inference engine, and FIG. 2C is processing using the rule-based engine and the inference engine in cooperation. It is an explanatory view showing . FIGS. 2A to 2C show a flow for obtaining the white balance gain (WB gain) of the imaging device. In addition, in FIGS. 2A to 2C, processing by the inference engine is indicated by hatching.

In the example of FIG. 2A, light source color determination and subject color determination are performed for image input. Using these determination results, the rule-based engine makes a comprehensive determination according to a preset rule, and obtains a white balance gain. In the example of FIG. 2B, for the image input, the inference engine estimates the white balance gain (WB estimation).

In the example of FIG. 2C, the inference engine first obtains a light source map from the input image, rather than directly outputting the white balance gain inference result for the image input. The rule-based engine, on the other hand, determines the light source color. The inference engine corrects the light source color determined by the rule-based engine using the light source map. Then, the white balance gain is obtained by comprehensive determination based on the correction result of the light source color and the determination result of the subject color by the inference engine.

The imaging apparatus 10 of FIG. 1 includes an inference engine 17 and a rule base engine 18 that cooperate to perform inference in this way. The imaging device 10 records an image obtained by imaging a subject. As the imaging device 10, not only a digital camera or a video camera but also a camera built into a smartphone or a tablet terminal may be adopted. Note that if the inference engine covers the functions of the rule-based engine, the rule-based engine may be omitted.

The imaging device 10 includes a control section 11 that controls each section of the imaging device 10 . The control unit 11 may be configured by a processor using a CPU (Central Processing Unit) or the like, and may operate according to a program stored in a memory (not shown) to control each unit, or may be a hardware electronic circuit. may implement some or all of the functions.

The imaging unit 12 of the imaging device 10 has an imaging element 12a and an optical system 12b. The optical system 12b includes a lens, a diaphragm, and the like (not shown) for zooming and focusing. The optical system 12b includes a zoom (variable magnification) mechanism, focus and diaphragm mechanism (not shown) for driving these lenses.

The imaging device 12a is composed of a CCD, CMOS sensor, or the like, and an optical image of a subject is guided to an imaging surface of the imaging device 12a by an optical system 12b. The imaging device 12a photoelectrically converts the subject optical image to acquire a captured image (image capturing signal) of the subject.

The imaging control unit 11a of the control unit 11 drives and controls the zoom mechanism, focus mechanism, and aperture mechanism of the optical system 12b to adjust zoom, aperture, and focus. The imaging unit 12 performs imaging under the control of the imaging control unit 11 a and outputs an imaging signal of the captured image (moving image and still image) to the control unit 11 .

An operation unit 13 is provided in the imaging device 10 . The operation unit 13 includes a release button (not shown), function buttons, various switches such as shooting mode setting and parameter operation, dials, ring members, and the like, and outputs operation signals to the control unit 11 based on user operations. The control section 11 controls each section based on an operation signal from the operation section 13 .

The control unit 11 captures captured images (moving images and still images) from the imaging unit 12 . The image processing unit 11b of the control unit 11 performs predetermined signal processing such as color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing on the captured image.

The imaging device 10 is provided with a display section 14, and the control section 11 is provided with a display control section 11f. The display unit 14 is, for example, a display device having a display screen such as an LCD (liquid crystal display device). The display control unit 11f causes the display unit 14 to display the picked-up image signal-processed by the image processing unit 11b. The display control unit 11f can also cause the display unit 14 to display various menu displays, warning displays, and the like of the imaging device 10. FIG.

The imaging device 10 is provided with a communication section 15, and the control section 11 is provided with a communication control section 11e. The communication unit 15 is controlled by the communication control unit 11e so that information can be transmitted and received between the learning device 20 and the learning request device 30. FIG. The communication unit 15 is capable of short-range wireless communication such as Bluetooth (registered trademark) and wireless LAN communication such as Wi-Fi (registered trademark). It should be noted that the communication unit 15 is not limited to Bluetooth and Wi-Fi, and can employ various communication methods. The communication control unit 11 e can receive the inference model information from the learning device 20 via the communication unit 15 . This inference model information is for constructing a desired inference model by the network 17 a of the inference engine 17 .

The control unit 11 is provided with a recording control unit 11c. The recording control unit 11c can compress the picked-up image after the signal processing, and supply the compressed image to the recording unit 16 for recording. The recording unit 16 is composed of a predetermined recording medium, and can record information given from the control unit 11 and output the recorded information to the control unit 11 . For example, a card interface may be used as the recording unit 16. In this case, the recording unit 16 can record image data on a recording medium such as a memory card.

The recording section 16 has an image data recording area 16a, and the recording control section 11c records image data in the image data recording area 16a. The recording control unit 11c can also read and reproduce information recorded in the recording unit 16. FIG.

The recording unit 16 also has an inference setting recording area 16b. The recording control unit 11 c can record the received inference model information in the inference setting recording area 16 b of the recording unit 16 . In this way, the inference model information regarding the setting of the network 17a that constitutes the inference engine 17 is recorded in the inference setting recording area 16b. Note that the inference model information also includes the inference setting information as described above. Test data for verifying the operation of the inference engine 17 is recorded in the test data recording area 16 c of the recording unit 16 .

The inference setting information is generated based on specification information to be described later, and defines how the inference engine 17 and the rule base engine 18, which is a concurrent inference device, cooperate to obtain a desired inference result. be. As the inference setting information, the specification information may be used as it is, or it may be generated based on the specification information. For example, the inference setting information indicates the input/output relationship between the inference engine 17 and the rule base engine 18 .

The rule base engine 18 comprises a logic circuit 18a. The logic circuit 18a makes inferences according to preset rules, and obtains outputs by inferences with respect to inputs. In the example of FIG. 1, the logic circuit 18a receives the captured image from the imaging unit 12 and outputs an inference result. In this embodiment, the inference result from the rule base engine 18 may be supplied to the inference engine 17 as well as to the control unit 11 .

The inference engine 17 has a network 17a. The network 17a is constructed using setting values included in the inference model information recorded in the inference setting recording area 16b, and constitutes a network obtained by completing learning in machine learning, that is, an inference model. . In the example of FIG. 1, the network 17a receives a captured image from the imaging unit 12 and outputs an inference result. In this embodiment, the inference result from the inference engine 17 may be supplied to the control unit 11 and also to the rule base engine 18 .

That is, in the present embodiment, the inference engine 17 may be given not only the captured image from the imaging unit 12 but also the inference result of the rule base engine 18 to perform inference and obtain the inference result. Similarly, the rule-based engine 18 may be given not only the captured image from the imaging unit 12 but also the inference result of the inference engine 17 to perform inference and obtain the inference result.

The linking relationship between the inference engine 17 and the rule base engine 18 is described in the inference setting information in the inference model information. An inference setting unit 11d is provided in the control unit 11, and the inference setting unit 11d reads inference setting information from the inference setting recording area 16b and controls the inference engine 17 and the rule base engine . The inference setting unit 11d uses the inference result of at least one of the inference engine 17 and the rule base engine 18 to make comprehensive determination.

In the present embodiment, the specification information that is the basis of the inference setting information is generated by the specification setting unit 31a of the learning requesting device 30, which will be described later. Further, in the present embodiment, the control section 11 is provided with a specification setting section 11g having the same function as the specification setting section 31a. The specification setting units 31 a and 11 g can generate specification information describing the specifications of the inference model information required to generate the inference model adopted by the inference engine 17 . The specification setting unit 11g transmits the generated specification information to the learning request device 30 via the communication unit 15. FIG.

The specification setting units 31a and 11g automatically generate part of the information in the specification information and also generate part of the information based on the user's input operation. The specification setting section 11g can control the display control section 11f to display a specification setting menu display for checking and changing specification information. The specification setting unit 31a can also display a specification setting menu display for checking and changing specification information on a display unit (not shown) provided in the learning requesting device 30. FIG.

When an inference model is obtained by machine learning, it is necessary to fully consider the difference between the performance of the training computer and the training data given during learning, and the performance of the input data and the electronic circuit during actual use. I can't get the correct inference result. Therefore, it is important to have a learning step that fully considers the constraints when using the inference model. The kind of information shown in this figure is what makes it possible to standardize the so-called "required specifications of the inference model." Various organizations and individuals use various means to improve the efficiency and performance of learning. It is difficult. In other words, in order to make it possible to use a learning environment with an extremely high degree of freedom and a wealth of teacher data, and to make it possible to acquire an inference model that can output even more accurate output under the usage environment, we have to meet these required specifications. Clear settings are important. These data may be set by manual input, information recorded in a recording unit mounted on the device, or information obtained by communication from the outside. In this way, meticulous consistency is important for effective use of inference models based on machine learning. be. Therefore, some of the required specifications shown here may not be necessary, and there may be situations where additional items should be added, such as environmental constraints such as temperature, humidity, and atmospheric pressure, circuit power supply voltage, and clock specifications. Needless to say. If necessary, conditions for not using this inference model may be described.
Thus, in this embodiment, it is possible to provide an inference model utilization device that has a communication unit for requesting the creation of an inference model from the outside, and that can display a list of the required specifications of the inference model to be installed on the display unit. , and this inference model utilization device becomes a high-precision inference device that can request an inference model that is highly consistent with the device from an external device.

FIG. 3 is an explanatory diagram showing an example of the specification setting menu display DM displayed on the display screen 14a of the display section 14. As shown in FIG. In the example of FIG. 3, the specification information includes 15 specification items, but the specification items can be set as appropriate. The example of FIG. 3 shows specification items of specification information for creating an inference model for detecting a predetermined detection target in an image.

The detection target item is information indicating that detection target information indicating a detection target is attached to the teacher data as an annotation. The hardware information item indicates information about the device using the inference model, and designates the device name, number of network layers, clock frequency and memory capacity. The response time item specifies the time from image input to inference output. In the example of FIG. 3, the accuracy rate and reliability items specify that the reliability of inference of 90% or more is ensured, and the input image size and other items indicate that the input image size is FHD (full high definition ) designates that the process A is adopted as the image process.

Also, the input image system number item designates that the input image is switched between two systems. It is possible. In other words, this designation makes it possible to give the output of the rule base engine 18 to the inference engine 17 for cooperation between the inference engine 17 and the rule base engine 18 .

In addition, in the present embodiment, combined detectors and auxiliary information items are set. This item specifies the combined inference device to be used together. In the example of FIG. 1, the rule-based engine 18 that performs inference in cooperation with the inference engine 17 is designated as the combined inference device. In the example of FIG. 3, it can be seen that the rule-based engine 18 is a device having a function of performing face detection according to specifications**. The auxiliary information item designates what output of the rule base engine 18 is to be used as auxiliary information when the output of the rule base engine 18 is used in the inference engine 17 . For example, when performing face detection, it is possible to designate the coordinates of the face detected by the rule base engine 18, the image of the face, etc. as auxiliary information and to input the information to the inference engine 17. FIG. That is, this designation defines the input from the rule base engine 18 to the inference engine 17. FIG.

The delivery date item specifies the delivery date of the inference model information. The teacher data and target items specify the storage folder and detection target (file) of the teacher data, and in the example of FIG. The other data use item specifies use of data other than an image when the input is an image. For example, when detecting a building in an image, it is possible to use the tilt angle of the image as other data. It can be seen that the test sample item specifies sample data for testing the inference model and is not provided in the example of FIG. Activation conditions and restriction items specify the conditions and restrictions for activating inference. For example, when acquiring an image, if the brightness of the image is below a predetermined value, the inference is not executed. It is possible.

The supplementary information item specifies supplementary information. For example, it is possible to specify to display a frame or text on the detected image portion. Also, the history information item specifies the history of previous inference models, for example, by version information.

An item priority item is set in the present embodiment. The item priority item specifies which specification item in the specification setting information should be given priority to generate an inference model. In the example of FIG. It shows that the priority is set in the order of reliability item, response time item, input image size, other items, and input image system number item.

The specification setting units 31a and 11g automatically generate predetermined specification items among these specification items, and generate other specification items by user input. For example, the specification setting units 31 a and 11 g may automatically generate predetermined specification items based on the specifications of the imaging device 10 . For example, the specification setting units 11g and 31a may automatically generate information about predetermined specification items by executing a specification information generation program stored in a memory (not shown). Further, the specification setting units 11g and 31a may automatically generate predetermined specification items using an inference engine (not shown) for generating specification information.

The specification setting unit 11g transmits the generated specification information to the learning request device 30. FIG. The learning request device 30 requests learning to the learning device 20 based on the specification information. The learning requesting device 30 requests learning to the learning device 20 based on the specification information generated by the specification setting unit 31a or the specification information generated by the specification setting unit 11g.

The learning requesting device 30 has a communication section 32 and the learning device 20 has a communication section 22 . These communication units 22 and 32 have the same configuration as the communication unit 15, and communication is possible between the communication units 22, 32 and 15. FIG.

The learning requesting device 30 has a control section 31 that controls each section of the learning requesting device 30 , and the learning device 20 has a control section 21 that controls each section of the learning device 20 . The control units 21 and 31 may be configured by a processor using a CPU, FPGA, or the like, may operate according to a program stored in a memory (not shown) to control each unit, or may be implemented by hardware. A part or all of the functions may be realized by an electronic circuit.

Note that the learning unit 20 as a whole may be configured by a processor using a CPU, GPU, FPGA, or the like, and may operate according to a program stored in a memory (not shown) to control learning. A part or all of the functions may be realized by an electronic circuit.

The learning requesting device 30 has an image classification recording unit 33 that records a large amount of learning data. The image classification recording unit 33 is composed of a recording medium (not shown) such as a hard disk or a memory medium, and classifies and records a plurality of images according to the types of objects included in the images. In the example of FIG. 9, the image classification recording unit 33 stores a target object image group 34, and the target object image group 34 includes teacher data 34a and test data 34b for each type of target object. The control section 31 also has a specification setting section 31a having the same function as the specification setting section 11g. The control unit 31 of the learning request device 30 controls the communication unit 32 to transmit the specification information and the teacher data 34a to the learning device 20. FIG.

Although the test data 34b is data similar to the teacher data 34a, it is data that is not provided to the learning device 20. In the learning requesting device 30 and the imaging device 10, an inference model obtained as a result of learning by the learning device 20 is used. It is used for the test of The learning requesting device 30 can transmit the test data 34b to the imaging device 10 . The imaging device 10 records the test data from the learning requesting device 30 in the test data recording area 16c.

The mother set creation unit 24 of the learning unit 20 records the teacher data transmitted from the learning request device 30 in the teacher data recording unit 23 . The mother set creating unit 24 has an input data setting unit 24a and an output item setting unit 24b. The input data setting unit 24a sets input data used for learning, and the output item setting unit 24b sets outputs to be obtained as a result of inference. The settings of the input data setting unit 24 a and the output item setting unit 24 b are performed based on the specification information received from the learning request device 30 .

The input/output modeling unit 25 determines a network design so as to obtain an expected output from a large amount of teacher data, and generates inference model information as its setting information. The input/output modeling unit 25 is provided with a specification matching unit 25a. The specification matching unit 25a has a memory (not shown) for storing specification information, and determines whether or not the inference model information obtained by the input/output modeling unit 25 corresponds to the specification information. The input/output modeling unit 25 constructs the network design until the inference model information corresponds to the specification information.

In the present embodiment, the input/output modeling unit 25 includes information as to whether or not the constructed inference model performs inference in cooperation with the combined inference device, If it is to be performed, inference setting information including information on how to link it is generated, and the generated inference setting information is added to the inference model information. For example, the input/output modeling unit 25 may add the inference setting information to the header information of the inference model information. For example, the input/output modeling unit 25 uses, as header information of the inference model information, information indicating the network structure, information on weights and biases, information on cooperating inference devices, information on how to cooperate with cooperating inference devices, and the like. Generate information containing

The input/output modeling unit 25 may add the inference setting information as meta information to the inference model information instead of the header information. You may make it associate. Further, the input/output modeling unit 25 may add the specification information as it is to the inference model information as the inference setting information.

The input/output modeling unit 25 transmits the inference model information including the generated inference setting information to the control unit 11 of the imaging device 10 via the communication unit 22 . As described above, the control unit 11 records the inference model information in the inference setting recording area 16b. The inference setting unit 11d controls the operations of the inference engine 17 and the rule base engine 18 based on the inference setting information, thereby making it possible to obtain an inference result using the inference of the inference engine 17 and the rule base engine 18. do.

Next, the operation of the embodiment configured in this way will be described with reference to FIGS. 4 to 17. FIG. FIGS. 4 to 6 are explanatory diagrams for explaining an example of how the inference engine and the combined inference device are linked.

Three methods shown in FIGS. 4 to 6 will be described as methods for inference by the inference engine 17 and the rule base engine 18 in cooperation. In the examples of FIGS. 4 to 6, inference is performed to detect a predetermined detection target from an input image.

The example of FIG. 4 shows a technique (hereinafter referred to as the first technique) in which the inference by the inference engine 17 and the inference by the rule base engine 18, which is a combined inference device, are executed independently of each other. An input image is given to both the inference engine 17 and the rule base engine 18, and the inference engine 17 and the rule base engine 18 perform inference independently. The inference engine 17 outputs the position information of the detection target on the image and its reliability as an inference result, and the rule base engine 18 also outputs the position information of the detection target on the image and its reliability as an inference result. The control unit 11 selects the position information based on the reliability information and determines the detection target.

FIG. 5 shows a method (hereinafter referred to as the second method) of giving priority to the inference by the inference engine 17 and giving the inference result of the inference engine 17 to the rule base engine 18, which is a combined inference device, to improve the inference accuracy. . An input image is first provided to the inference engine 17 . The inference engine 17 outputs an inference result to the control unit 11, and outputs an inference result or a determination result of inference impossibility to the rule base engine 18, for example, when an inference result with sufficient reliability cannot be obtained. Upon receiving this result, the rule base engine 18 makes an inference and outputs the inference result to the control unit 11 . The control unit 11 selects the position information based on the reliability information and determines the detection target.

FIG. 6 shows a method of giving priority to the inference by the inference engine 17 and giving the inference result of the rule-based engine 18, which is a combined inference device, to the inference engine 17 to improve the degree of inference by the inference engine 17 (hereinafter referred to as the third method). ). The input image is provided to both the inference engine 17 and a companion inference appliance, the rule-based engine 18 . The rule base engine 18 outputs an inference result to the control unit 11 and outputs auxiliary information to the inference engine 17 for improving the inference accuracy of the inference engine 17, for example. For example, the rule base engine 18 detects a face to be detected and outputs the coordinates of the face, the face image, etc. to the inference engine 17 as auxiliary information. Upon receiving this result, the inference engine 17 performs inference and outputs the inference result to the control unit 11 . The control unit 11 selects the position information based on the reliability information and determines the detection target. It should be noted that the method of cooperation is not limited to the above first to third methods. For example, the inference result of the combined inference device may be preferentially used.

Next, specific examples of the first to third techniques will be described.

FIG. 7 is an explanatory diagram showing how the imaging device 10 captures an image. The imaging apparatus 10 according to the present embodiment includes each circuit shown in FIG. 1 housed in a housing 10a shown in FIG. 7, and a display screen 14a of the display unit 14 is provided on the rear surface of the housing 10a. For example, the user 41 holds the housing 10a with the right hand 42, and captures the subject within the visual field range while looking at the display screen 14a of the display unit 14. FIG. A release switch 43 constituting the operation unit 13 is provided on the upper surface of the housing 10a. In the example of FIG. 7, the subject is a cat 46 on top of a block wall 45 . The user 41 takes a picture by pressing the release switch 43 with the index finger 42a of the right hand 42. FIG.

In the example of FIG. 7, the inference engine 17 and the rule base engine 18 of the imaging device 10 are configured to detect the cat 10 in the captured image by inference. The rule-based engine 18 is configured, for example, to enable detection of a cat's face by a known method.

(First method)
FIG. 8 is a flow chart for explaining the operation of the learning requesting device 30, and FIG. 9 is a flow chart for explaining the operation of the learning device 20. As shown in FIG.

The control unit 31 of the learning requesting device 30 determines whether or not to set the required specification of the inference model to be set in the inference engine 17 in step S1 of FIG. If the required specifications are not set, the control unit 31 determines in step S2 whether or not creation of a teacher data folder has been instructed, and if instructed, creates a teacher data folder in step S3. do. That is, the control unit 31 collects and annotates images that serve as teacher data for inference for cat detection. The collected training data is recorded in the object image group 34 of the image classification recording section 33 as training data 34a. Test data 34b is also recorded in the object image group 34. FIG.

When setting the required specifications, the control unit 31 proceeds from step S1 to step S4 and sets the specification items shown in FIG. 3 by the specification setting unit 31a. As described above, the specification setting unit 31a automatically generates some of the specification items, and creates others based on the user's input operation. In step S5, the control unit 31 determines whether or not all specification items have been input. If not, the process returns to step S1. Specification information by item and teacher data are sent to the learning device 20 to request learning.

FIG. 10 is an explanatory diagram for explaining learning by the learning device 20 in the first method and an inference model obtained as a result of the learning. The control unit 21 of the learning device 20 is in a waiting state for a learning request in step S11 of FIG. When the learning request is generated, the control unit 21 shifts the process to step S12, receives the specification information from the learning request device 30, and acquires the requested specification. Further, the control unit 21 acquires teacher data from the learning request device 30 and records it in the teacher data recording unit 23 (step S13). The control unit 21 additionally records teacher data according to specifications and necessity (step S14). The mother set creating unit 24 sets inputs and outputs using the input data setting unit 24a and the output item setting unit 24b according to the required specifications based on the specification information. The input/output modeling unit 25 performs learning based on teacher data according to the set inputs and outputs.

As shown in FIG. 10, the input/output modeling unit 25 provides a predetermined network N1 with a large number of images corresponding to inputs and outputs as teacher data. In the example of FIG. 10, an image of a cat whose face is viewed from the side is input as teacher data. In this case, the image of the cat is annotated so that the face is surrounded by a dashed frame. The orientation of the face is described based on the imaging device 10, and in the following description, the face viewed from the side may also be referred to as the sideways face. That is, the sideways face means that the cat does not turn its face toward the optical axis direction of the imaging device 10 . Further, a front-facing face, which will be described later, means that the face faces the optical axis direction of the imaging device 10 .

By performing learning using a large amount of teacher data, the network design is determined so that the network N1 can obtain an output corresponding to the input. That is, in the example of FIG. 10, when an image of a cat is input, if the cat's face is turned sideways, information for displaying a dashed frame at a position surrounding the image portion of the face is the reliability level. obtained with information. That is, in the example of FIG. 10, an inference model for detecting an image of a cat with a sideways face is constructed.

“Deep learning” is a multi-layered process of “machine learning” using neural networks. A typical example is a "forward propagation neural network" that sends information from front to back and makes decisions. In the simplest case, it consists of an input layer consisting of N1 neurons, an intermediate layer consisting of N2 neurons given by parameters, and N3 neurons corresponding to the number of classes to be discriminated. It suffices if there are three output layers. The neurons of the input layer and the intermediate layer, and the neurons of the intermediate layer and the output layer are respectively connected by a connection weight, and the intermediate layer and the output layer are added with a bias value, thereby facilitating the formation of logic gates. Three layers may be sufficient for simple discrimination, but if a large number of intermediate layers are used, it becomes possible to learn how to combine a plurality of feature quantities in the process of machine learning. In recent years, those with 9 to 152 layers have become practical due to the relationship between the time required for learning, judgment accuracy, and energy consumption.

Various known networks may be employed as the network N1 employed for machine learning. For example, R-CNN (Regions with CNN features) using CNN (Convolution Neural Network) or FCN (Fully Convolutional Networks) may be used. It involves a process called "convolution" that compresses image features, works with minimal processing, and is robust to pattern recognition. In addition, a "recurrent neural network" (fully-connected recurrent neural network), which can handle more complicated information and can handle information analysis whose meaning changes depending on the order and order, may be used in which information flows in both directions.

In order to realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs (Field Programmable Gate Arrays) may be used, but much of the neural network processing is matrix multiplication. For this reason, GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) that are specialized for matrix calculations are sometimes used. In recent years, artificial intelligence (AI) dedicated hardware "neural network processing unit (NPU)" is designed to be integrated and embedded with other circuits such as CPU, and it may be part of the processing circuit. be.

In addition, the inference model may be acquired by employing not only deep learning but also various known machine learning techniques. For example, there are methods such as support vector machines and support vector regression. The learning here is to calculate the classifier weights, filter coefficients, and offsets, and there is also a method using logistic regression processing. When a machine judges something, a human needs to teach the machine how to judge something. A rule-based technique that applies rules acquired by humans through empirical rules and heuristics may be applied and used.

In step S16, the input/output modeling unit 25 determines whether the generated inference model satisfies the required specifications. If not satisfied, in the next step S17, the input/output modeling unit 25 determines whether or not the specifications satisfy the conditions specified by the priority in the specification information. For example, if the item priorities are as shown in FIG. It is determined whether or not the specifications of each item of the item, input image size, other items, and input image system number item are satisfied. If satisfied, the input/output modeling unit 25 resets the teacher data (step S18), determines that the predetermined number of times or more has been reached (step S19), If so, the process returns to step S15 to repeat the inference modeling.

If the input/output modeling unit 25 determines in step S17 that the generated inference model does not satisfy the condition specified by the priority, or in step S19, determines that inference modeling has been repeated a predetermined number of times or more. If so, the process proceeds to step S20, and weak image information indicating that the image is difficult to construct an effective inference model is transmitted to the learning requesting device 30. FIG.

When the input/output modeling unit 25 determines in step S16 that the generated inference model satisfies the required specifications, the process proceeds to step S21, and the learning request device 30 and the device designated as necessary Inference model information is transmitted to (for example, the imaging device 10).

In this embodiment, the input/output modeling unit 25 adds the inference setting information generated based on the specification information to the inference model information at the time of this transmission. In the example of FIG. 10, the inference setting information includes inference by the inference engine and inference by the combined inference device that are executed independently of each other, and inference results are obtained based on the reliability of each inference. It contains information indicating that detection by the method is to be performed.

Upon receiving the inference model information, the control unit 31 of the learning requesting device 30 uses the test data 34b to determine the quality of the learning result in step S7 of FIG. When determining that the test is good (OK), the control unit 31 requests the learning device 20 to transmit the inference model information to be tested to the imaging device 10, and determines that the test is bad. If the determination is (NG), the learning device 20 is requested to re-learn. In this case, the required specifications may be added or modified. Note that the learning requesting device 30 may directly transmit the OK-determined inference model information to the imaging device 10 .

FIG. 11 is a flowchart for explaining the operation of the imaging device 10. FIG.

The control unit 11 of the imaging device 10 determines whether or not the shooting mode is specified in step S31 of FIG. If the shooting mode is not specified, the control unit 11 determines whether or not acquisition of the inference model is instructed in step S32. If the inference model acquisition is not instructed, the process returns to step S31, and if it is instructed, the specifications of the inference model are confirmed in the next step S33. That is, the control unit 11 receives the inference model information from the learning device 20 or the learning request device 30, and constructs the network 17a of the inference engines 17 based on the received inference model information. In this way, an inference model similar to N1 in FIG. 10 is constructed in the network 17a. The inference setting unit 11d reads the inference setting information added to the inference model information, considers the function of the built-in rule base engine 18, and confirms whether or not the necessary specifications are satisfied. .

In step S34, the control unit 11 determines whether or not the result of the specification confirmation is good (OK). If not, the control unit 11 transmits information for requesting resetting of the required specifications to the learning device 20 or the learning requesting device 30 in step S38. If the result of the specification confirmation is good, the control section 11 reads the test data from the test data recording area 16c and executes the inference test. As a result of the test, the control unit 11 determines whether or not sufficient reliability has been obtained. If so, in step S38, a request is made to reset the required specifications.

When the photographing mode is designated, the control unit 11 shifts the process from step S31 to step S41 and captures the captured image. A captured image from the imaging unit 12 is provided to the control unit 11 and also to the inference engine 17 and the rule base engine 18 . Based on the inference setting information, the inference setting unit 11d of the control unit 11 instructs both the inference engine 17 and the rule base engine 18 to perform inference independently by the first method.

As shown in FIG. 7, the user 41 takes a picture. The display control unit 11f of the control unit 11 gives the captured image from the imaging unit 12 to the display unit 14 to display a live view image. Also, the inference engine 17 and the rule base engine 18 perform inference on the captured images that are sequentially input (step S42).

FIG. 12 is an explanatory diagram for explaining inference by the rule base engine 18, and FIG. 13 is an explanatory diagram for explaining inference by the inference engine 17. FIG.

Assume that the captured image PI1 shown in FIG. 12 is input to the rule base engine 18 at a certain moment. The rule-based engine 18 detects the part of the cat's front facing face by a known face detection process. The rule-based engine 18 outputs the information of the solid-line frame PO1a surrounding the cat's face as the detection result. This information is supplied from the rule base engine 18 to the control unit 11 . Based on the detection result of the rule base engine 18, the display control unit 11f displays the display image PO1 in which the solid line frame PO1a is superimposed on the live view image.

13 is input to the inference engine 17 at a certain moment. The inference engine 17 uses the network 17a based on the inference model information provided by the learning device 20 or the learning requesting device 30 to detect the part of the cat's sideways face. The inference engine 17 outputs, as a detection result, the information of the broken-line frame PO2a surrounding the sideways face of the cat. This information is supplied from the inference engine 17 to the control unit 11 . Based on the detection result of the inference engine 17, the display control unit 11f displays a display image PO2 in which a dashed frame PO2a is superimposed on the live view image.

FIG. 14 is an explanatory diagram showing an example of image display based on a series of inference detection results. FIG. 14 shows a live view image displayed on the display screen 14a of the display unit 14. As shown in FIG. Images P1 to P6 are shown among the images that are sequentially captured over time and displayed in live view. Image P1 is obtained by imaging a relatively wide range, and includes an image of cat 46 on block wall 45 . In step S43, the display control unit 11f of the control unit 11 causes the inference engine 17 and the rule base engine 18 to operate to display a “activation” display P1a on the display screen 14a, indicating that the cat image is being detected. display.

Here, it is assumed that the user 41 performs a zoom operation so that mainly only cats are photographed within the field of view. Images after the image P2 show live view images in this state. Image P2 is an image of the cat's face facing forward. In this case, the rule base engine 18 detects the cat's face, and supplies the control section 11 with the information of the solid line frame surrounding the face. When determining that the detection has been performed, the control unit 11 determines whether the detection by the inference engine 17 is used or the detection by the rule base engine 18 is used (step S45). For example, the control unit 11 may make this determination based on the reliability output from the inference engine 17 and rule base engine 18 . Since the detection result of the rule base engine 18 is adopted for the image P2, the process proceeds to step S46, and the display control unit 11f displays a solid line frame display P2b on the live view image. In addition, the display control unit 11f displays a display P2a of "R activation" indicating that the rule base engine 18 has been activated and detection of the cat's face, and characters of "face detection".

The next image P3 is an image of the cat's face taken sideways. In this case, the inference engine 17 detects the sideways face of the cat and supplies the control unit 11 with the information of the broken-line frame surrounding the face. In this case, the display control unit 11f displays a dashed frame display P3b on the live view image in step S47 following step S45. In addition, the display control unit 11f displays a display P3a of "A activation" indicating that the inference engine 17 has been activated and detection of the cat's face and characters of "AI inference".

The next image P4 is also taken with the cat's face facing sideways. In this case, the dashed frame display P4b based on the information from the inference engine 17, the "A activation" display P4a indicating that the inference engine 17 was activated and the cat's face was detected, and the "AI inference" display. Characters are displayed on the live view image.

Further, the next image P5 is an image of the cat's face facing forward again. In this case, the rule base engine 18 detects the cat's face, the solid line frame display P5b based on the information from the rule base engine 18, and the rule base engine 18 is activated to detect the cat's face. A display P5a of "R activation" and characters of "Face detection" are displayed on the live view image.

The final image P6 shows that the cat detection by the inference of the inference engine 17 and the rule base engine 18 is activated by the display P6a of "execution". However, the cat's face shows a non-detected state.

In step S48, the control unit 11 determines whether or not a moving image shooting operation or a still image shooting operation has been performed. When these operations are performed, shooting and recording are performed (step S49). It should be noted that, at the time of moving image shooting, the captured image is made into a file by the end operation of moving image shooting. If the photographing operation has not been performed, the control unit 11 returns the processing to step S31.

As described above, in the first method, the inference engine 17 and the rule-based engine 18, which is a combined inference device, operate independently of each other.

(Second method)
The operations of the learning device 20 and the learning request device 30 are the same for the first to third methods, but the specification information and teacher data are different for each of the first to third methods.

FIG. 15 is an explanatory diagram for explaining learning by the learning device 20 in the second method and an inference model obtained as a result of the learning.

In FIG. 15, a large number of images corresponding to inputs and outputs are given as training data to a predetermined network N2. The example in FIG. 15 is an image of a cat, and an image of a cat whose face is facing forward and an image of a cat whose face is facing sideways are input. In the image of the cat facing forward, the position information of the face (solid line frame) is set as an annotation, and in the image of the cat facing sideways, the annotation is set so that the face part is surrounded by the dashed frame. It is Furthermore, in the second method, an annotation is set for an image of a cat whose face is facing forward to encourage the use of a combination inference device.

Thereby, the network design of the network N2 is determined so that the output corresponding to the input can be obtained. That is, in the example of FIG. 15, when an image of a cat is input, if the cat's face is facing forward, use of the combined inference device is urged, and the positional information of the image portion of the face (solid line frame) is displayed. ) is obtained along with the reliability information, and when the cat's face is turned sideways, information for displaying a dashed frame around the image portion of the face is obtained along with the reliability information.

FIG. 16 is a flowchart for explaining the operation of the imaging device 10. FIG. In FIG. 16, the same steps as in FIG. 11 are denoted by the same reference numerals, and descriptions thereof are omitted. Also in the second method, the rule-based engine 18 detects the face of a cat facing forward, as shown in FIG.

When the photographing mode is designated, the control unit 11 shifts the process from step S31 to step S41 and captures the captured image. A captured image from the imaging unit 12 is provided to the control unit 11 and also to the inference engine 17 and the rule base engine 18 . Based on the inference setting information added to the inference model information, the inference setting unit 11d of the control unit 11 controls the inference engine 17 and the rule base engine 18 to perform inference by the second method.

In the next step S52, the control unit 11 determines whether or not the inference result by the inference engine 17 prompts the use of the combined inference device.

Assume that the captured image PI2 shown in FIG. 13 is input to the inference engine 17 at a certain moment. The inference engine 17 uses the network 17a based on the inference model information provided by the learning device 20 or the learning requesting device 30 to detect the part of the cat's sideways face. The inference engine 17 outputs, as the detection result, the information of the broken-line frame PO2a (see FIG. 13) surrounding the sideways face of the cat. This information is supplied from the inference engine 17 to the control unit 11 . In this case, the control unit 11 proceeds from step S52 to step S53 and determines whether or not the reliability of the inference is high. The control unit 11 displays the detection result when the reliability of the inference is high. That is, based on the detection result of the inference engine 17, the display control unit 11f displays the display image PO2 in which the dashed frame PO2a is superimposed on the live view image.

12 is inputted to the inference engine 17 and the rule base engine 18 at a certain moment. The inference engine 17 uses the network 17a based on the inference model information to detect the part of the cat's front facing face. The inference engine 17 outputs to the control unit 11 information on the image position of the cat's front facing face and information prompting the rule base engine 18 to detect the cat as the detection result. In this case, the control unit 11 shifts the process from step S52 to step S55, and provides the inference result of the inference engine 17, that is, the information of the image position of the cat's face to the rule base engine 18. Let the engine 18 detect the face of the cat. The rule-based engine 18 detects the part of the cat's front facing face by a known face detection process. In this case, the rule base engine 18 is provided with information on the image position of the cat's face, and the rule base engine 18 can detect the cat's face with higher accuracy.

The rule-based engine 18 outputs the information of the solid-line frame PO1a (see FIG. 12) surrounding the cat's face as the detection result. This information is supplied from the rule base engine 18 to the control unit 11 . The display control unit 11f displays the display image PO1 in which the solid line frame PO1a is superimposed on the live view image based on the detection result of the rule base engine 18 (step S56).

As described above, in the second method, the inference engine 17 gives the inference result to the rule base engine 18 to make the rule base engine 18 perform inference. As a result, it is possible to provide the rule base engine 18 with information for further improving the accuracy of images and the like for which high inference accuracy can be obtained in the rule base engine 18, and as a result, detection with high inference accuracy is possible. .

(Third method)
In the third method as well, the operations of the learning device 20 and the learning request device 30 are the same as those of the other methods, and the specification information and teacher data are different from those of the other methods.

In the third method, the inference model created in the learning device 20 is set so that the detection result of the combined inference device is used as input in addition to the captured image. For example, when an image of a cat's face and information such as the position of the image of the cat's face are input from the combined inference device, and an image of the cat is input from the imaging unit 12, an inference model that detects the cat's face Learning takes place so that

FIG. 17 is a flow chart for explaining the operation of the imaging device 10. FIG. In FIG. 17, the same steps as in FIG. 11 are denoted by the same reference numerals, and descriptions thereof are omitted. Also in the third method, the rule-based engine 18 detects the face of a cat facing forward, as shown in FIG.

When the photographing mode is designated, the control unit 11 shifts the process from step S31 to step S41 and captures the captured image. A captured image from the imaging unit 12 is provided to the control unit 11 and also to the inference engine 17 and the rule base engine 18 . Based on the inference setting information added to the inference model information, the inference setting unit 11d of the control unit 11 controls the inference engine 17 and the rule base engine 18 to perform inference by the third method.

In step S61 following step S41, the control unit 11 causes the rule-based engine 18 to execute inference and detect the cat's face. A detection result from the rule base engine 18 is given to the control unit 11 . The control unit 11 determines the detection result of the rule base engine 18 (step S62).

Next, the control unit 11 causes the inference engine 17 to perform inference in step S63. In this case, the control unit 11 gives the detection result of the rule base engine 18 to the inference engine 17 . As a result, the inference engine 17 receives the input image from the imaging unit 12 and the inference result of the rule base engine 18, and obtains the inference result by inference using these inputs. The inference result of the inference engine 17 is output to the control unit 11 .

The control unit 11 determines the inference result of the inference engine 17 in step S64. Furthermore, in step S65, the control unit 11 obtains a detection result by comprehensively determining the inference results of the inference engine 17 and the rule base engine 18. FIG.

Thus, in the third method, the rule base engine 18 provides the inference result to the inference engine 17, and the inference accuracy of the inference engine 17 can be further improved.

As described above, in the present embodiment, it is possible to realize more effective inference by performing learning for constructing an inference model assuming the use of a combined inference device. The learning requesting device creates specification information for learning on the assumption that such combined reasoning devices will be used, thereby enabling construction of an effective inference model in the learning device. Further, the learning device adds inference setting information indicating how to use the inference model to the inference model information and transmits the inference setting information in the device using the inference model. As a result, in a device that uses an inference model, it is possible to use the combined inference device and the inference engine that uses the inference model in cooperation based on the inference setting information added to the received inference model information. In this way, it is possible to effectively operate joint inference devices, such as a rule base engine and an inference engine, to obtain more effective inference results.

(Second embodiment)
18 and 19 are block diagrams showing the second embodiment. 18 and 19 show a configuration added to the learning device 20 of FIG.

An image database (DB) 51a and a correct answer database (DB) 51b in FIGS. 18 and 19 correspond to the training data recording unit 23 in the learning device 20 in FIG. An image DB 51a is shown, and a collection of annotation data in the teacher data is shown as a correct answer DB 51b.

In addition to the configuration of the learning device 20, the learning device of the present embodiment includes a combination reasoning device 53, a detection accuracy measuring device 54, a correct information output unit 55, a detection accuracy database (DB) 56, and a detection accuracy information output unit 57. I have it. As shown in FIG. 18, the control unit 21 (see FIG. 1) uses a combined inference device 53 and provides each image data 52 in the image DB 51a to the combined inference device 53 to execute inference processing. . The detection result of the combined inference device 53 is given to the detection accuracy measuring device 54 .

Further, the correct answer information output unit 55 acquires correct answer information corresponding to the image data input to the combined inference device 53 from the correct answer DB 51 b and outputs the correct answer information to the detection accuracy measuring device 54 . The detection accuracy measuring device 54 measures the detection accuracy of the combined reasoning device 53 by comparing the detection result of the combined reasoning device 53 and the correct answer information, and outputs the measurement result to the detection accuracy DB 56 . The detection accuracy DB 56 stores the detection accuracy of inference of the combined inference device 53 for each image.

During learning of the inference model N, the input/output modeling unit 25 provides each image data 52 to the inference model N, and provides correct information from the correct information output unit 55 to the inference model N, as shown in FIG. Further, the detection accuracy information output unit 57 reads information on detection accuracy corresponding to the image data 52 input to the inference model N from the detection accuracy DB 56 . The input/output modeling unit 25 is controlled by the control unit 21 and uses the detection accuracy information from the detection accuracy information output unit 57 when constructing the inference model N. FIG.

For example, when constructing the inference model N, the control unit 21 uses the detection accuracy information corresponding to the images input to the inference model N as teacher data, so that the combined inference device can be used to make inferences for images with high detection accuracy. You can learn as follows. In addition, if the detection accuracy of an image is higher than a predetermined threshold, it is determined that no further learning is necessary for this image, and learning is performed only for images whose detection accuracy is lower than the predetermined threshold. good. In this case, it is possible to sufficiently learn images with relatively low reliability, and it is possible to improve the detection accuracy as a whole.

As described above, in the present embodiment, it is possible to use the information of the detection accuracy of the combined inference device at the time of learning, and to construct an inference model that enables more effective inference.

In the above embodiments, a digital camera was used as a device for imaging, but the camera may be a digital single-lens reflex camera, a compact digital camera, a video camera, a movie camera, or a mobile phone. A camera built in a personal digital assistant (PDA: Personal Digital Assist) such as a smart phone may be used.

The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the present invention at the implementation stage. Also, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components of all components shown in the embodiments may be omitted. Furthermore, components across different embodiments may be combined as appropriate.

Regarding the operation flow in the claims, the specification, and the drawings, even if explanations are made using "first," "next," etc. for the sake of convenience, it is essential to implement them in this order. does not mean Further, it goes without saying that each step constituting these operation flows can be appropriately omitted as long as it does not affect the essence of the invention.

It should be noted that, among the techniques described here, the control described mainly in the flow charts can often be set by a program, and may be stored in a recording medium or a recording unit. The method of recording in the recording medium and the recording unit may be recorded at the time of product shipment, using a distributed recording medium, or downloading via the Internet.

It should be noted that, in the embodiments, portions described as "parts" (sections or units) may be configured by combining a dedicated circuit or a plurality of general-purpose circuits, and if necessary, pre-programmed software A processor such as a microcomputer, a CPU, or a sequencer such as an FPGA may be combined to operate according to the above. It is also possible to design a part or all of the control by an external device, in which case a wired or wireless communication circuit is interposed. Communication may be performed by Bluetooth, WiFi, telephone line, or the like, and may be performed by USB or the like. A dedicated circuit, a general-purpose circuit, and a control unit may be integrated into an ASIC.

DESCRIPTION OF SYMBOLS 1... Imaging device, 10... Control part, 11... Image processing part, 11a... Image quality improvement part, 12... Display control part, 13... Recording control part, 14... Action detection part, 20... Imaging part, 21... Optical system, 22... Imaging device 23... Displacement applying unit 30... Specific range estimating unit 31... Scene determining unit 32... Good composition storing unit 33... Improvement predicting unit 41... Display unit 42... Operation unit 43... Sensor Part 44... Recording part.

Claims (13)

a specification setting unit that generates specification information describing settings of a plurality of specification items for requesting specification of an inference model;
a control unit that performs control for transmitting the specification information together with teacher data to a learning device that generates the inference model;
The specification setting unit includes, in the specification information, a specification item for setting whether or not the inference engine using the inference model performs inference in cooperation with another combination inference device. request device. The specification setting unit may include, in the specification information, specification items for setting information on functions of the other combination inference device and information on input/output between the inference engine and the other combination inference device. 2. The learning request device according to claim 1. 2. The learning request device according to claim 1, wherein said specification setting unit includes specification items for setting priorities of said plurality of specification items in said specification information. Specification information that describes the settings of multiple specification items for determining the specifications of the inference model, and is specification information for the inference engine that uses the above inference model to perform inference in cooperation with other concurrent inference devices. an inference modeling unit that builds the inference model based on
a control unit for performing control for adding inference setting information regarding a method of cooperation with the combined inference device to the inference model information for constructing the inference model based on the specification information and transmitting the inference setting information; A learning device characterized by: 5. The learning device according to claim 4, wherein the control unit adds the specification information to the inference model information as the inference setting information. 5. The learning device according to claim 4, wherein the control unit adds the inference setting information to the inference model information as header information of the inference model information. Inference model information for constructing an inference model on the premise that the inference engine and the combined inference device cooperate to perform inference, and which is inference setting information related to how the inference engine and the combined inference device are linked. a communication unit that receives the inference model information to which is added;
the inference engine that performs inference using an inference model configured based on the inference model information;
the combined inference device;
An inference model utilization apparatus, comprising: a control unit that causes the inference engine and the combined inference device to cooperate and execute inference based on the inference setting information. The control unit operates the inference engine and the combined inference device independently of each other based on the inference setting information, and determines the reliability of the first inference result of the inference engine and the first inference result of the combined inference device. 8. The inference model utilization apparatus according to claim 7, wherein the first or second inference result is obtained based on the second reliability of the inference result. 3. The control unit, based on the inference setting information, provides the first inference result of the inference engine to the combined inference device to obtain the second inference result of the combined inference device. 8. The inference model utilization device according to 7. 8. The controller obtains the first inference result of the inference engine by giving the second inference result of the combined inference device to the inference engine based on the inference setting information. The inference model utilization device described in . Inference model information for constructing an inference model on the premise that the inference engine and the combined inference device cooperate to perform inference, and which is inference setting information related to how the inference engine and the combined inference device are linked. receives the above inference model information with
building the inference engine using an inference model configured based on the inference model information;
An inference model utilization method characterized by causing the inference engine and the combined inference device to cooperate to execute inference based on the inference setting information. to the computer,
Inference model information for constructing an inference model on the premise that the inference engine and the combined inference device cooperate to perform inference, and which is inference setting information related to how the inference engine and the combined inference device are linked. receives the above inference model information with
constructing the inference engine using an inference model configured based on the inference model information;
An inference model utilization program for executing a procedure for executing inference by linking the inference engine and the combination inference device based on the inference setting information. an imaging unit that acquires a captured image of a subject;
Inference model information for constructing an inference model on the premise that the inference engine and the combined inference device cooperate to perform inference, and which is inference setting information related to how the inference engine and the combined inference device are linked. a communication unit that receives the inference model information to which is added;
the inference engine that performs inference using an inference model configured based on the inference model information;
the combined inference device;
a control unit that detects a predetermined object from the captured image by causing the inference engine and the combined inference device to perform inference based on the inference setting information. Imaging device.

Images