JP2020188449A - Image analyzing program, information processing terminal, and image analyzing system - Google Patents

Abstract

To provide an image analyzing program, an information processing terminal, and an image analyzing system that allow users to easily install cameras without adjusting installation positions and installation angles of the cameras by a specialized technical staff.SOLUTION: A method includes measuring inference processing accuracy by NN model (S4) and notifying a user of instructions to improve the inference processing accuracy based on the measured inference processing accuracy (S5, S6, and S8). The user adjusts an installation position and installation angle of a camera of a smart phone based on the instructions, and thereby appropriate inference accuracy of the NN model can be obtained without a specialized technical staff visiting the installation position of the camera and adjusting the installation position and installation angle of the camera.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image analysis program, an information processing terminal, and an image analysis system.

Conventionally, a surveillance camera or a camera such as a so-called AI (Artificial Intelligence) camera is arranged indoors of a store or the like, and an object such as a person reflected in a frame image taken by these cameras is detected as a learned object. There are known devices and systems that detect an object by using a neural network for recognizing an object and recognize the detected object by using a neural network for recognizing a learned object (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2018-093283

However, in a device or system that detects and recognizes an object such as a person photographed by a conventional camera as described in Patent Document 1, there is a problem that it takes a lot of time and effort to install a camera such as an AI camera. It was. Specifically, when introducing this type of device or system, specialized technical staff will go to the camera installation location (taking into account the light source to avoid backlighting) and the camera installation position (shooting). It was necessary to adjust the position) and installation angle (shooting direction) so that the accuracy (inference accuracy) of the inference processing of the appropriate trained object detection neural network or trained object recognition neural network could be ensured. ..

The present invention solves the above-mentioned problems, and a specialized technical staff goes to the installation location of the camera to appropriately detect or learn the learned object without adjusting the installation position and the installation angle of the camera. An object of the present invention is to provide an image analysis program, an information processing terminal, and an image analysis system that enable a user to easily install a camera by making it possible to obtain the inference accuracy of a neural network for finished object recognition. To do.

In order to solve the above problem, the image analysis program according to the first aspect of the present invention uses an information processing terminal equipped with a camera to detect a trained object for detecting an object reflected in an input image from the camera. One of the image analysis unit that analyzes the input image from the camera and the one or more trained neural network models using one or more trained neural network models including the neural network model for Based on the inference accuracy measurement unit that measures the accuracy of the inference processing by the trained neural network model and the inference processing accuracy measured by the inference accuracy measurement unit, the user provides instruction information for improving the accuracy of the inference processing. It functions as a notification unit that notifies the user.

In this image analysis program, the object detected by the trained object detection neural network model is a person or a face, and the instruction information is used to detect the person or face by the trained object detection neural network model. It may be instructional information that prompts the user to move the camera to a suitable installation position or installation angle.

In this image analysis program, the notification unit may notify the user of the instruction information by displaying the instruction information on the display device.

In the image analysis program according to the second aspect of the present invention, an information processing terminal provided with a camera is superposed by an image superimposing unit that superimposes an image of a person or a face on an input image from the camera and the image superimposing unit. Inference processing by an image analysis unit that analyzes a person or face image using one or more trained neural network models and one of the trained neural network models of the one or more trained neural network models. Based on the inference accuracy measurement unit that measures the accuracy of the inference accuracy and the inference processing accuracy measured by the inference accuracy measurement unit, it functions as a notification unit that notifies the user of instruction information for improving the accuracy of the inference processing. ..

In this image analysis program, the instruction information is instruction information that prompts the user to move the camera to an installation position or an installation angle suitable for performing inference processing by any of the learned neural network models. You may.

In this image analysis program, the notification unit may notify the user of the instruction information by displaying the instruction information on the display device.

In this image analysis program, the image superimposing unit changes the attributes and numbers of people in the image of a person or face superimposed on the input image with time to various attributes and numbers, and the image analysis unit is said to be the same. Various images obtained by superimposing an image of a person or a face with an attribute and a number changed with time on the input image are analyzed one after another, and the notification unit is a person with an attribute and a number changed with time. Alternatively, for various images in which a face image is superimposed on the input image, instruction information for improving the accuracy of the inference processing is notified to the user based on the accuracy of the inference processing measured by the inference accuracy measuring unit. You may.

The information processing terminal according to the third aspect of the present invention includes a camera and one or more trained neural networks including a trained object detection neural network model for detecting an object reflected in an input image from the camera. An image analysis unit that analyzes an input image from the camera using a network model, and an inference that measures the accuracy of inference processing by one of the trained neural network models of the one or more trained neural network models. It includes an accuracy measuring unit and a notification unit that notifies the user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measuring unit.

In this information processing terminal, the object detected by the trained object detection neural network model is a person or face, and the instruction information is used to detect the person or face by the trained object detection neural network model. It may be instructional information that prompts the user to move the camera to a suitable installation position or installation angle.

In this information processing terminal, the information processing terminal further includes a display device for displaying an input image from the camera, and the notification unit displays the instruction information on the display device to display the instruction information. May be notified to the user.

In this information processing terminal, the information processing terminal is derived from a camera position direction estimation unit that estimates the installation position and installation angle of the camera based on an input image from the camera, and the camera displayed on the display device. A pointing device for designating the detection position of the person or face is further provided with respect to the input image of the camera, and the notification unit is provided with the camera in addition to the accuracy of the inference processing measured by the inference accuracy measurement unit. The instruction information may be obtained based on the installation position and installation angle of the camera estimated by the position direction estimation unit and the detection position of the person or face designated by the pointing device.

The information processing terminal according to the fourth aspect of the present invention has a camera, an image superimposing unit that superimposes an image of a person or a face on an input image from the camera, and an image of a person or a face superimposed by the image superimposing unit. An image analysis unit that analyzes using one or more trained neural network models, and an inference that measures the accuracy of inference processing by one of the trained neural network models of the one or more trained neural network models. It includes an accuracy measuring unit and a notification unit that notifies the user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measuring unit.

In this information processing terminal, the instruction information is instruction information that urges the user to move the camera to an installation position or an installation angle suitable for performing inference processing by any of the learned neural network models. You may.

In this information processing terminal, the information processing terminal further includes a display device for displaying an input image from the camera, and the notification unit displays the instruction information on the display device to display the instruction information to the user. May be notified to.

In this information processing terminal, the image superimposing unit changes the attributes and numbers of people in the person or face image superimposed on the input image with time to various attributes and numbers, and the image analysis unit is said to be said. Various images obtained by superimposing an image of a person or a face with an attribute and a number changed with time on the input image are analyzed one after another, and the notification unit is a person with an attribute and a number changed with time. Alternatively, for various images in which a face image is superimposed on the input image, instruction information for improving the accuracy of the inference processing is notified to the user based on the accuracy of the inference processing measured by the inference accuracy measuring unit. You may.

The image analysis system according to the fifth aspect of the present invention is a management server that manages the information processing terminal, including any of the above information processing terminals and installation of the learned neural network model on the information processing terminal. And.

According to the image analysis program according to the first and second aspects of the present invention and the information processing terminal according to the third and fourth aspects, any one of the trained neural network models of one or more trained neural networks The accuracy of the inference processing by the model is measured, and the instruction information for improving the accuracy of the inference processing is notified to the user based on the measured accuracy of the inference processing. Based on this notified instruction information, the user adjusts the installation position (shooting position) and installation angle (shooting direction) of the camera of the information processing terminal, and the specialized technical staff goes to the camera installation location. Therefore, it is possible to obtain an appropriate inference accuracy of the trained neural network model (learned object detection neural network or trained object recognition neural network) without adjusting the installation position and installation angle of the camera. Therefore, the user can easily install the camera.

Further, according to the image analysis system according to the fifth aspect of the present invention, in addition to the above effects, a trained neural network model for an information processing terminal (a trained neural network for detecting an object and learning) using a management server. It is possible to manage information processing terminals, including the installation of a neural network for learning objects.

The block block diagram which shows the schematic structure of the image analysis system including the smartphone of 1st Embodiment of this invention. A block diagram showing a schematic hardware configuration of the smartphone. Configuration diagram of the main software on the smartphone. Functional block configuration diagram of SoC in the smartphone. Explanatory drawing which shows an example of the installation method of the smartphone (camera). The flowchart which shows an example of the adjustment process of the installation position and the installation angle of a camera using an instruction in the smartphone. An example of an instruction for improving the accuracy of inference processing by a face detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. A flowchart showing another example of the camera installation position and installation angle adjustment process using instructions in the smartphone. An example of an instruction for improving the accuracy of inference processing by a person detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a person detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a person detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a person detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a person detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a person detection model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. The functional block block diagram of SoC in the smartphone of the 2nd Embodiment of this invention. A block diagram showing each functional block of SoC in the same smartphone and a file necessary for creating an image of a person in 3D computer graphics (3DCG) by a 3D model superimposition part. The flowchart which shows the example of the adjustment process of the installation position and the installation angle of the camera using the instruction in the smartphone. An example of an instruction for improving the accuracy of the inference processing (gender / age estimation processing) by the face recognition model on the smartphone, and an explanatory diagram of the adjustment processing of the camera installation position and installation angle using this instruction. The figure which shows the situation selection menu screen displayed on the touch panel of the smartphone. An example of an instruction for improving the accuracy of inference processing by a face recognition model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a face recognition model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a face recognition model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction. An example of an instruction for improving the accuracy of inference processing by a face recognition model on the smartphone, and an explanatory diagram of an adjustment process of a camera installation position and an installation angle using this instruction.

Hereinafter, an image analysis program, an information processing terminal, and an image analysis system according to an embodiment embodying the present invention will be described with reference to the drawings. FIG. 1 shows information on which a smartphone 1 which is an information processing terminal according to the first embodiment of the present invention (that is, an image analysis application (see FIG. 2) which is an image analysis program according to the first embodiment of the present invention is installed). It is a schematic block block diagram of the image analysis system 10 including a processing terminal). As shown in FIG. 1, the image analysis system 10 includes a smartphone 1 installed in each installation area such as a residence, an AI analysis server 3 on the cloud C, and a smartphone management server 4 (“management server” in the claim). It has. Each smartphone 1 includes a camera 2 which is a digital camera capable of shooting a moving image.

The AI analysis server 3 analyzes the behavior of a person in each area based on the object recognition result using the image analysis application on the smartphone 1, and uses the analysis result information for various purposes such as marketing and crime prevention. Converts the data into data that is easy for the application to use and outputs it. The object recognition result from the smartphone 1 (the image analysis application in the above) is converted into character string data that does not include personal information and is sent to the AI analysis server 3. In addition, the smartphone management server 4 manages the smartphone 1 installed in each installation area. Specifically, the smartphone management server 4 includes an image analysis application storage 5 such as a hard disk that stores a large number of image analysis applications according to various types of object recognition (including object detection), and each smartphone 1 It is possible to install the image analysis application on the smartphone 1 and manage the execution of the image analysis application on each smartphone 1.

Next, with reference to FIG. 2, the hardware configuration of the smartphone 1 and the image analysis application 21 installed on the smartphone 1 will be described. In addition to the above-mentioned camera 2, the smartphone 1 includes a SoC (System-on-a-Chip) 11, a touch panel 14 (“display device” and “pointing device” in the claim), a speaker 15, and various types. It includes a memory 16 for storing data and programs, a communication unit 17, a secondary battery 18, and a charging terminal 19. The SoC 11 includes a CPU 12 that controls the entire device and performs various operations, a trained object detection neural network model (hereinafter referred to as “object detection NN model”), and a trained object recognition neural network model (hereinafter, “object”). It is equipped with a GPU 13 used for inference processing and the like of "recognition NN model"). Further, the touch panel 14 is used as a pointing device for the user to specify a detection position of a person or a face with respect to an input image from the camera 2 displayed on the touch panel 14.

The program stored in the memory 16 includes an image analysis application 21 downloaded and installed from the smartphone management server 4 to the smartphone 1 and a smartphone analysis OS program 20 described later. The communication unit 17 includes a communication IC and an antenna. The smartphone 1 is connected to the AI analysis server 3 and the smartphone management server 4 via the communication unit 17 and the network. Further, the secondary battery 18 is a battery such as a lithium ion battery that can be used repeatedly by charging, and stores power from a commercial power source after being converted into DC power by an AC / DC converter. It is supplied to each part of the smartphone 1.

Further, the image analysis application 21 of the smartphone 1 shown in FIG. 2 is an example of various types of image analysis applications stored in the image analysis application storage 5 of the smartphone management server 4. Each of the image analysis applications stored in the image analysis application storage 5 includes one or more NN models including an object detection NN model for detecting an object reflected in an input image from the camera 2, and these NN models. It is a package program that includes a control script for the NN model. In this script, how to use one or more of the above NN models (processing order), accuracy measurement processing of inference processing by one of the above one or more NN models, and this inference processing. The display processing of the instruction (“instruction information” in the claims) for improving the accuracy of the script is described. The example image analysis application 21 shown in FIG. 2 includes a face detection model 22, a face recognition model 23, a script 24, and an AR application 25.

The face detection model 22 is an object detection NN model that detects a face included in an image acquired from the camera 2. The face recognition model 23 is an object recognition NN model that estimates the gender and age of a person having a detected face based on the face detected by the face detection model 22. The script 24 is a processing procedure of the face detection model 22 and the face recognition model 23, an accuracy measurement process of the inference process by the face detection model 22, and an instruction display process for improving the accuracy of the inference process by the face detection model 22. It is a simple program for control that describes such things. The AR application 25 is a program for an AR engine using existing technologies such as ARKit (a framework for AR-compatible applications for iPhone (registered trademark) and iPad (registered trademark)).

Next, the configuration of the main software in the smartphone 1 will be described with reference to FIG. As shown in FIG. 3, the main software in the smartphone 1 is an object detection recognition model 31, an easy installation UI (User Interface) 32, an AR engine 33, and a smartphone analysis OS 34. FIG. 3 shows the process of each software. The process of the object detection recognition model 31 in FIG. 3 corresponds to the programs of the face detection model 22 and the face recognition model 23 in the example of the image analysis application 21 shown in FIG. The process of the easy installation UI 32 in FIG. 3 corresponds to a part of the program of the script 24 in FIG. The process of the AR engine 33 in FIG. 3 corresponds to the program of the AR application 25 in FIG. The smartphone analysis OS 34 in FIG. 3 corresponds to the smartphone analysis OS program 20 in FIG.

FIG. 4 shows a functional block of the SoC 11 in the smartphone 1 described above. The smartphone 1 (SoC11) includes an image analysis unit 41, an inference accuracy measurement unit 42, a notification unit 43, and a camera position direction estimation unit 44 as functional blocks. Among the functional blocks in the SoC11, the functions of the inference accuracy measuring unit 42, the notification unit 43, and the camera position direction estimation unit 44 are realized by the CPU 12, and the functions of the image analysis unit 41 are realized by the CPU 12 and the GPU 13. ing.

The image analysis unit 41 includes one or more (learned) NN models including an object detection NN model for detecting an object (person, face, etc.) reflected in the input image from the camera 2. In the second example, the input image from the camera 2 is analyzed by using the face detection model 22 and the face recognition model 23). The inference accuracy measuring unit 42 measures the accuracy of the inference processing by any one of the above-mentioned one or more NN models. The notification unit 43 notifies the user of an instruction for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measurement unit 42. More specifically, the notification unit 43 notifies the user of the instruction by displaying the instruction for improving the accuracy of the inference processing by the NN model on the touch panel 14. The above-mentioned instruction is an instruction for urging the user to move the camera 2 to an installation position or an installation angle suitable for detecting a person or a face by an object detection NN model.

Further, the camera position direction estimation unit 44 estimates the installation position and installation angle of the camera 2 based on the input image from the camera 2. In addition to the accuracy of the inference processing measured by the inference accuracy measurement unit 42, the notification unit 43 includes the installation position and installation angle of the camera 2 estimated by the camera position direction estimation unit 44, and a person or a person designated by the touch panel 14. Obtain (determine) the above instructions based on the detection position of the face. In addition to the image from the camera 2, the camera position / direction estimation unit 44 uses an acceleration sensor, an angular acceleration sensor, and other sensors capable of detecting the movement of the smartphone 1 to improve the accuracy of the camera position / direction calculation. It may be improved. Further, the camera position direction estimation unit 44 does not use the image from the camera 2, but uses only the output values of the acceleration sensor, the angular acceleration sensor, and other sensors capable of detecting the movement of the smartphone 1, and the camera position direction. Estimates may be calculated.

FIG. 5 shows an example of the installation method of the smartphone 1 (camera 2). In this example, an example in which the smartphone 1 (camera 2) is installed at the entrance of the event venue is shown. In the example shown in FIG. 5, the smartphone 1 is held by a smartphone holder 52 having a flexible arm 51 whose shape can be easily changed, and a clip 53 provided at one end of the flexible arm 51 is used to hold an event. It is attached to the guide plate 54. By deforming the flexible arm 51, the camera 2 of the smartphone 1 can change its installation position and installation angle. Further, although not shown in FIG. 5, a power cord connected to the charging terminal 19 (see FIG. 2) of the smartphone 1 is connected to a commercial power source via an AC / DC converter. In the example shown in FIG. 5, an example is shown in which the smartphone holder 52 has a clip 53 for attachment at one end of the flexible arm 51, but the smartphone holder 52 has an attachment tool such as a magnet at one end of the flexible arm 51. It may be a configuration having. Further, a wide-angle lens may be attached to the camera 2 of the smartphone 1 with a clip or the like.

Next, with reference to FIGS. 6 and 7, an example of an instruction for improving the accuracy of the inference processing by the above NN model and the adjustment processing of the installation position and the installation angle of the camera 2 using this instruction will be described. explain. In this example, the above instruction is a simple instruction for improving the accuracy of the inference processing by the face detection model 22 (see FIG. 2).

When the user selects a desired process (estimation of human gender / age) from the candidates for the analysis process (recognition process) displayed on the touch panel 14 of the smartphone 1 (S1), the smartphone 1 (mainly the CPU 12) Downloads the NN model (face detection model 22 and face recognition model 23 (see FIG. 2)) corresponding to the selected process and the script 24 for these NN models from the smartphone management server 4 (S2). Next, the user installs the camera 2 of the smartphone 1 at an approximate position (a position where the original human face detection position (for example, the entrance of a building) can be photographed to some extent), and the image analysis application 21 (FIG. 2) is started (S3). Then, the CPU 12 of the smartphone 1 executes the script 24 of the image analysis application 21, for example, displays a message "Please stand a person at a desired detection position" on the touch panel 14, and then displays the current installation position and installation. The measurement (S4) of the accuracy of the inference processing by the face detection model 22 for the input image (captured image) from the camera 2 at the angle is started.

As a result of the measurement processing of S4, when the accuracy of the measured inference processing is less than a predetermined value (when an object whose probability of being classified as a human face is equal to or more than a predetermined value cannot be detected), the smartphone 1 The CPU 12 (notifying unit 43) determines that the inference by the face detection model 22 has failed (NO in S5), and moves the camera 2 to a position suitable for detecting the human face (more accurately, the human). The user is instructed to move the camera 2 to an installation position and an installation angle suitable for detecting the face of the user (S6). Specifically, when the CPU 12 of the smartphone 1 determines that the inference by the face detection model 22 has failed, the face detection is performed on the touch panel 14 as shown in the upper part of FIG. 7 according to the script 24 of the image analysis application 21. A red frame 61 (indicated by a broken line in FIG. 7) indicating a failure and a message 64 "Please move the camera so that the face fits in the frame" are displayed.

The CPU 12 (notification unit 43) of the smartphone 1 continues to display the above message 64 and the red frame 61 on the touch panel 14 until the inference by the face detection model 22 is successful. On the other hand, the user adjusts the installation position and the installation angle of the camera 2 so that the face 62 fits within the red frame 61 (S7), and the accuracy of the measured inference processing is a fixed time (for example, 3 seconds). When the value exceeds a predetermined value (when an object whose probability of being classified as a human face is equal to or higher than a predetermined value can be detected), the CPU 12 (notification unit 43) of the smartphone 1 states that the inference by the face detection model 22 is successful. After making a judgment (YES in S5), as shown in the lower part of FIG. 7, a green frame 63 (indicated by a solid line in FIG. 7) indicating successful face detection and a message "Adjustment completed" are displayed on the touch panel 14. 65 is displayed (S8). In this way, the user desires while looking at the instructions (red frame 61, green frame 63, message 64, and message 65) for improving the accuracy of the inference processing by the face detection model 22 displayed on the touch panel 14. In order to install the camera 2 at a position where the face of a person standing in the detection position can be accurately detected, the installation position and the installation angle of the camera 2 can be adjusted by trial and error. The CPU 12 (notification unit 43) of the smartphone 1 displays the current inference processing accuracy in the touch panel 14 (for example, a bar-shaped level meter in which the filled area becomes larger as the accuracy is higher). The user may be intuitively shown how to improve the inference processing accuracy.

Next, in addition to the flowchart of FIG. 8, with reference to FIGS. 9 to 14, another example of the instruction for improving the accuracy of the inference processing by the above NN model and the camera 2 using this instruction. The process of adjusting the installation position and installation angle will be described. In this example, the above instruction is an instruction for improving the accuracy of the inference processing by the NN model for detecting a person for counting the number of visitors in the store, and is compared with the examples shown in FIGS. 6 and 7 above. It is a polite instruction (including not only the result of inference processing but also a message indicating how to adjust the installation position and installation angle of the camera 2). Further, in this example, the NN model included in the image analysis application 21 is a person detection NN model (hereinafter referred to as “person detection model”) and a vectorization model. The vectorization model is an object recognition NN model that performs vectorization processing on a person image detected by the person detection model. By using this vectorization model to perform vectorization processing on the image of the person detected by the person detection model, it is determined whether or not the people photographed at different timings by the camera 2 are the same person. Can be done. That is, the camera 2 captures an image five times every second, for example, and performs a person detection process and a vectorization process for each captured image to track the person reflected in the captured image (same as the same person). By assigning an ID, it is possible to observe the movement of the person concerned).

When the user selects a desired process (visitor count) from the analysis process candidates displayed on the touch panel 14 of the smartphone 1 (S11), the CPU 12 of the smartphone 1 determines the NN model (NN model) corresponding to the selected process. The person detection model and the vectorization model) and the scripts for these NN models are downloaded from the smartphone management server 4 (S12). Next, the user fixedly installs the camera 2 of the smartphone 1 at an approximate position (a position where the visitor can be photographed to some extent (a position where the entrance of the store can be photographed to some extent)) and starts the image analysis application 21. (S13). Then, the CPU 12 of the smartphone 1 executes the script of the image analysis application 21 and displays a message on the touch panel 14 for the user to specify the detection position (the position where the person to be detected appears) (S14). .. For example, the CPU 12 of the smartphone 1 acquires the current camera image as a (frame) image, estimates the camera installation position and installation angle, and the three-dimensional coordinates of the floor surface in the image, and then shows it in FIG. The message 70 "Please draw a line (1)" and the message 73 "Please draw a line (2)" shown in FIG. 10 are displayed on the touch panel 14 by superimposing the image on the image of the camera image. Display above.

To estimate the camera installation position and installation angle, the AR application 25 (see FIG. 2) is used, and the camera position direction estimation unit 44 extracts feature points from each frame image acquired by the camera 2, and each frame image. The three-dimensional installation position and installation angle of the camera 2 are estimated from the movements of the feature points between them. This feature point is a point corresponding to an object edge or the like in the frame image, and the feature is hard to change even if the direction of the light source (illumination) changes. Further, the three-dimensional coordinates of the floor surface in the above image are also estimated from the movement of the feature points between the frame images by extracting the feature points from each frame image acquired by the camera 2 using the AR application 25.

As shown in FIG. 9, when the message 70 "Please draw a line (1)" is displayed, the user traces the line (1) on the touch panel 14 with his / her finger F according to this message. The line 71 corresponding to is written in the input image from the camera 2 displayed on the touch panel 14.

Further, as shown in FIG. 10, when the message 73 "Please draw a line (2)" is displayed, the user traces the line (on the touch panel 14 with his / her finger F according to this message). The line 72 corresponding to 2) is written on the input image from the camera 2 displayed on the touch panel 14.

When the writing process of the lines 71 and 72 to the above image is completed, the CPU 12 (notification unit 43) of the smartphone 1 displays, for example, a message "Please adjust the installation position and the installation angle of the camera". , Prompt the user to adjust the installation position and installation angle of the camera 2. After the user adjusts the installation position and installation angle of the camera 2, the CPU 12 of the smartphone 1 acquires the (frame) image of the current camera image, and estimates the installation position and installation angle of the camera 2 again from the image. (S15). It is assumed that the following processing will be repeated while fine-tuning the installation position and installation angle of the camera 2, and reacquiring the image of the camera image and re-acquiring the installation position and installation angle of the camera 2. The estimation is performed at each iteration.

The CPU 12 of the smartphone 1 executes the script of the image analysis application 21, for example, displays the message "Please stand on the line (1)" on the touch panel 14, and then the camera at the current installation position and installation angle. The accuracy of the inference processing by the person detection model for the image on the line (1) in the input image (photographed image) from 2 is measured (S16).

As a result of the measurement process of S16, when the accuracy of the measured inference process is less than a predetermined value (when an object whose probability of being classified as a person is equal to or more than a predetermined value cannot be detected), the CPU 12 of the smartphone 1 is used. , It is determined that the inference by the person detection model has failed (NO in S17), the installation position and installation angle of the camera 2 estimated in S15, and the detection position of the person (position of the line 71) specified by the touch panel 14. Based on the above, the user is instructed to move the camera 2 to an installation position and an installation angle suitable for detecting a person (considering the influence of the light source) (S18). Specifically, as shown in FIG. 11, the notification unit 43 of the smartphone 1 can shoot only the feet of the person 74a on the line (1) at the current installation position and angle of the camera 2. If the person 74a on the line (1) cannot be detected by the person detection model, the message 76 "Please raise the camera position and tilt it horizontally" is displayed as shown in FIG. , Displayed on the touch panel 14. In addition to the accuracy of the inference processing measured in S16, the notification unit 43 of the smartphone 1 includes the installation position and angle of the camera 2 estimated in S15 and the detection position of the person (position of the line 71) designated by the touch panel 14. ) And the above instruction (message 76).

According to the above message 76, the user raises the installation position of the camera 2 and tilts the installation angle in the horizontal direction so that the camera 2 can photograph the entire person 74a on the line (1) (S19). When the accuracy of the measured inference processing exceeds a predetermined value (when an object whose probability of being classified as a person is equal to or higher than a predetermined value can be detected), the CPU 12 of the smartphone 1 subsequently moves "on the line (2)". The message "Please stand" is displayed on the touch panel 14. Then, the CPU 12 of the smartphone 1 measures the accuracy of the inference processing by the person detection model for the image on the line (2) in the input image (photographed image) from the camera 2 at the current installation position and installation angle (S16). ). As a result of this measurement process, when the accuracy of the measured inference process is less than a predetermined value, the same process as in S18 and S19 is performed.

As a result of the above measurement processing, when the accuracy of the inference processing by the person detection model for both the image on the line (1) and the image on the line (2) becomes a predetermined value or more (on the line (1) shown in FIG. 13). (When both the person 74a and the person 74b on the line (2) can be detected), the CPU 12 (notification unit 43) of the smartphone 1 determines that the inference by the person detection model is successful (YES in S17). ), As shown in FIG. 14, the message 77 "adjustment is completed" is displayed on the touch panel 14 (S20). In this way, the notification unit 43 of the smartphone 1 gives instructions (messages 70, 73) for improving the accuracy of the inference processing by the person detection model (to support the user in adjusting the installation position and the installation angle of the camera 2). , 76, 77, etc.) are displayed on the touch panel 14. Note that 75a and 75b in FIG. 13 are a bounding box corresponding to the person 74a in the line (1) and a bounding box corresponding to the person 74b in the line (2), respectively.

After the display of the adjustment completion message 77 in S20, when the user instructs the touch panel 14 of the smartphone 1 to execute the store visitor count process, the image analysis unit 41 of the smartphone 1 uses the person detection model to display a line. After detecting the person 74a in the line (1) and the person 74b in the line (2), the person 74a detected at the position of the line (1) and the person 74b detected at the position of the line (2) using a vectorized model. Determine if they are the same person. Then, when the person 74a detected at the position of the line (1) and the person 74b detected at the position of the line (2) are the same person, the CPU 12 of the smartphone 1 outputs the (recognition) result to AI. It is transmitted to the analysis server 3. When the AI analysis server 3 receives the recognition result that the same person is detected at the position of the line (1) and the position of the line (2) from the smartphone 1, the AI analysis server 3 counts up the number of visitors to the store.

As described above, according to the image analysis application 21 and the smartphone 1 of the present embodiment, any of the NN models included in the image analysis application 21 (in the example of "face detection", the face detection model 22). , In the example of "visitor count", the accuracy of the inference processing by the person detection model) is measured, and the user is notified of the instruction for improving the accuracy of the inference processing based on the measured accuracy of the inference processing ( (Displayed on the touch panel 14). Based on this notified instruction, the user adjusts the installation position (shooting position) and installation angle (shooting direction) of the camera 2 of the smartphone 1, and the specialized technical staff goes to the installation location of the camera 2. An appropriate NN model (face detection model 22 and face recognition model 23 in the "face detection" example, and a person in the "visitor count" example without adjusting the installation position or angle of the camera 2. The inference accuracy of the detection model and vectorization model) can be obtained. Therefore, the user can easily install the camera 2.

Further, according to the smartphone 1 of the present embodiment, the notification unit 43 of the SoC 11 installs the camera 2 estimated by the camera position direction estimation unit 44 in addition to the accuracy of the inference processing measured by the inference accuracy measurement unit 42. Based on the position and installation angle and the detection position of a person or the like designated by the touch panel 14, an instruction for improving the accuracy of the inference processing of the NN model is obtained. As a result, not only the notification of the result of the inference processing but also the notification indicating how to adjust the installation position and the installation angle of the camera 2 can be performed. Therefore, the user can install the camera 2 more easily.

Further, according to the image analysis system 10 of the present embodiment, in addition to the above effects, the smartphone 1 includes the installation of the image analysis application 21 (including the NN model) on the smartphone 1 by using the smartphone management server 4. Can be managed.

Next, the smartphone 1 of the second embodiment of the present invention will be described with reference to FIGS. 15 to 23. FIG. 15 shows a functional block of the SoC 11 in the smartphone 1 of the second embodiment. The SoC11 in the smartphone 1 of the second embodiment is the same as the SoC11 of the first embodiment except that it includes a 3D model superimposing unit 81 (“image superimposing unit” in the claim). The 3D model superimposition unit 81 superimposes the image of a person created by 3D computer graphics (3DCG) on the frame image (input image from the camera 2) acquired by the camera 2. Here, 3D computer graphics refers to a method of inputting information such as the shape of an object, the orientation and position of a camera, and the intensity and position of a light source into a computer, and causing the computer to calculate and generate an image by a program. In the first embodiment, the image analysis unit 41 directly analyzes the input image (the person or face reflected in the camera 2) from the camera 2, but in the second embodiment, the image analysis unit 41 superimposes the 3D model. The 3DCG person image superimposed on the input image (frame image) from the camera 2 is analyzed by the unit 81. The details of the processing performed by the 3D model superimposing unit 81 will be described later. Further, the functional blocks other than the 3D model superimposing unit 81 in FIG. 15 are designated by the same reference numerals as the blocks in FIG. 4, and the description thereof will be omitted.

FIG. 16 shows each functional block in the SoC11 shown in FIG. 15 and a file required for creating an image of a 3DCG person by the 3D model superimposing unit 81.

Next, in addition to the flowchart of FIG. 17, referring to FIGS. 16 and 18 to 23, the camera 2 of the smartphone 1 of the second embodiment uses the same instructions as those of the first embodiment. The adjustment process of the installation position and the installation angle will be described. In this example, the above instruction is an instruction for improving the accuracy of the inference processing by the face recognition model 23 (see FIG. 2). Here, the face recognition model 23 is an object recognition that estimates the gender and age of a person having a detected face based on the face detected by the face detection model 22, as in the case of the examples shown in FIGS. 6 and 7. NN model (NN model for gender / age recognition).

Although it is omitted in FIG. 17 because the flowchart becomes long, in this adjustment process, the processes corresponding to S1 to S3 in FIG. 6 or S11 to S13 in FIG. 8 are performed before the process of S21 is performed. That is, when the user selects a desired process (gender / age estimation of a person) from the candidates for the analysis process (recognition process) displayed on the touch panel 14 of the smartphone 1 (corresponding to S1 in FIG. 6), the smartphone 1 (mainly the CPU 12) transfers the NN model (face detection model 22 and face recognition model 23 (see FIG. 2)) corresponding to the selected process and the script 24 for these NN models from the smartphone management server 4. Download (corresponds to S2 in FIG. 6). Next, the user installs the camera 2 of the smartphone 1 at an approximate position (a position where the face of the visitor can be photographed (a position where the entrance of the store can be photographed)), and the image analysis application 21 (FIG. 2). Refer to))) (corresponding to S3 in FIG. 6). Then, the CPU 12 of the smartphone 1 executes the script of the image analysis application 21 to send a message on the touch panel 14 for the user to specify the position of the person (the position of the person having the face to be detected and recognized). It is displayed (S21 in FIG. 17). For example, the CPU 12 of the smartphone 1 acquires the current camera image as a (frame) image, estimates the camera installation position and installation angle, and the three-dimensional coordinates of the floor surface in the image, and then shows it in FIG. The message 88 "Please draw a line at the position of a person" is displayed on the touch panel 14.

As in the case of the first embodiment, the camera installation position and the installation angle are estimated by the camera position direction estimation unit 44 using the AR application 25 (see FIG. 2) for each frame acquired by the camera 2. This is performed by a method of extracting feature points from an image and estimating the three-dimensional installation position and installation angle of the camera 2 from the movement of the feature points between each frame image.

As shown in FIG. 18, when the message 88 "Please draw a line of the position of the person" is displayed, the user follows this message and traces the position of the person on the touch panel 14 with his / her finger F. Line 89 is written on the input image from the camera 2 displayed on the touch panel 14.

When the process of writing the line 89 to the input image is completed, the CPU 12 of the smartphone 1 selects (the situation) of the customer group on the touch panel 14 that the user expects to pass through the shooting location of the camera 2. The status selection menu screen 86 (see FIG. 19) is displayed. As shown in FIG. 19, on the situation selection menu screen 86, “many families”, “many young people”, “many children”, “many old people”, “including mothers holding children”, A selection button 87 (87a to 87f, etc.) for selecting (a situation) of various customer groups such as "miscellaneous" is provided. The above-mentioned "miscellaneous" represents the customer base (situation) in which many people with various attributes come (for example, 10 to 20 men and women in their 10s and 60s are reflected in the image taken by the camera 2).

When the user selects a plurality of selection buttons 87 according to the customer base (situation) of the location where the camera 2 of the smartphone 1 is installed from among the various selection buttons 87 displayed on the status selection menu screen 86 (S22). ), The 3D model superimposing unit 81 of the SoC11 is an image of a person's 3D model (a person's model image created by 3D computer graphics (3DCG)) according to (the situation) of the customer group selected by the user. ) Is read from the 3D model DB83 shown in FIG. Then, the 3D model superimposition unit 81 (data of the 3D model image read from the 3D model DB83 and the light environment (“light source” and “shadow”) (data) randomly (appropriately) read from the environment data file 84. Based on the orientation and position of the camera 2 estimated by the camera position direction estimation unit 44, the images of the 3D model of one or more people read from the 3D model DB83 are obtained according to the orientation and position of the camera 2. After adjustment, the image of the above-adjusted 3D model in the random light environment read from the environment data file 84 is superimposed on the input image from the camera 2, and this superimposed image is displayed on the touch panel 14. (S23).

Next, the details of the image superimposition processing of the 3D model described in S23 will be described, but before that, it is stored in each of the 3D model DB 83, the environment data file 84, and the situation data file 85 shown in FIG. The data obtained will be described.

First, in the 3D model DB 83, image data of 3D models of various kinds of people having various attributes such as gender, age, and body shape are stored for each person. That is, each of the image data of the 3D model stored in the 3D model DB83 is, for example, the image data of the 3D model of one "thin and tall woman in her thirties". On the other hand, the status data file 85 stores the status data of each customer group corresponding to each selection button 87 on the status selection menu screen 86. In the situation data file 85, for example, as situation data (of the customer base) expressing "family", for example, "one woman in her 30s to 50s with an average body shape + 1 person in her 30s to 50s" The status data of "male of average body shape + 1 male of average body shape of 5 to 15 years old + 1 female of average body shape of 5 to 15 years old" is stored. Further, the above environment data file 84 stores data of the light environment (“light source” and “shadow”). Of these, the data of the "light source" is the data of the light emitting source defined by the "position / direction / light intensity / color" of the light source. For example, when the light source is a "fluorescent lamp", it is the data of the light emitting source defined as "downward from overhead / light intensity is ~ cd (candela) / white". Further, the "shadow" is the data of the shadow corresponding to the above "light source". However, among the optical environment data stored in the above environment data file 84, the "shadow" (data) has a high processing capacity of the CPU 12 and GPU 13 of the smartphone 1, and the above "light source" (data). ), If it is possible to render the shadow corresponding to the "light source" in real time, it is not necessary to hold it as data. Here, the "real-time rendering" of "shadow" means that the 3DCG image is drawn by calculating the shadow at high speed in real time, instead of drawing the 3DCG image based on the shadow information prepared in advance. To do.

Therefore, for example, in the selection process of S22, when the user selects the selection button 87a of the situation of "many families" on the situation selection menu screen 86, the details of the 3D model image superimposition processing described in S23 above. Is as follows. That is, from the situation data file 85, the 3D model superimposition unit 81 shows that "one woman in her thirties to fifties with an average body shape + one woman in her thirties to fifties" corresponding to "family". Read the situation data of "male + 1 male with average body shape of 5 to 15 years + 1 female with average body shape of 5 to 15 years". Then, the 3D model superimposition unit 81 is based on the above situation data, from the 3D model DB83, an image of a 3D model of one "woman with an average body shape in her thirties to fifties" and one "30". An image of a 3D model of "a man with an average body shape in his teens to 50s", an image of a 3D model of one "boy with an average body shape of 5 to 15 years old", and an image of one "average of 5 to 15 years old" The image of the 3D model of the "girl with a body shape" is read out, and the images of these 3D models are combined to create the image of the 3D model of the "family". Next, the 3D model superimposition unit 81 adjusts the image of the above-mentioned “family-friendly” 3D model according to the orientation and position of the camera 2 estimated by the camera position direction estimation unit 44, and then adjusts the environment data file. A 3D model image of the above-adjusted "family" in a light environment randomly read from 84 ("position / direction / light intensity / color" and shadow of a random light source) is taken from the camera 2. It is superimposed on the input image, and as shown in FIG. 20, this superimposed image is displayed at the position of line 89 on the touch panel 14. The orientation of the person (drawing direction of the person) in the (3D model image) of the superimposed image changes according to the orientation of the line 89. In addition, the above-mentioned orientation of people (drawing direction of people) is not the same when there are a plurality of people in the image of the 3D model, and the orientation of each person is randomly about several tens of degrees. different. For example, when the lines 89 in FIGS. 18 and 20 are drawn in the vertical direction in these figures, the people 91a to 91d in the above superimposed image (image of the 3D model) are drawn in the direction viewed from the side. However, not everyone is in the same orientation, and there is some variation in the orientation of each person.

The first superimposed image of S23 (the image of the 3D model corresponding to the situation of the customer base of the selection button 87 first selected by the user on the situation selection menu screen 86 (for example, the image of the 3D model of "family") is used as the environment. When the display of the image generated in the first light environment randomly read from the data file 84 and the image of this 3D model superimposed on the input image from the camera 2) is completed, the CPU 12 (notification unit 43) of the smartphone 1 is completed. Displays, for example, the message "Please adjust the installation position and installation angle of the camera" to prompt the user to adjust the installation position and installation angle of the camera 2. After the user adjusts the installation position and installation angle of the camera 2, the CPU 12 of the smartphone 1 acquires the (frame) image of the current camera image, and estimates the installation position and installation angle of the camera 2 again from the image. (S24).

Then, the image analysis unit 41 of the smartphone 1 executes the script of the image analysis application 21 for the person of the 3D model in the above-mentioned superimposed image at the current installation position and installation angle of the camera 2 (face detection model 22). The face detection process (according to the face recognition model 23) and the gender / age estimation process (according to the face recognition model 23) are executed (S25). After that, the CPU 12 (inference accuracy measuring unit 42) of the smartphone 1 measures the accuracy of the inference processing for gender / age estimation by the face recognition model 23 (S26).

As a result of the measurement processing in S26, when the accuracy of the measured inference processing is less than the predetermined value (when the probability of being classified as either male or female is less than the predetermined value, or when any age (or age group) If the probability of classification is also less than a predetermined value), the CPU 12 (notification unit 43) of the smartphone 1 determines that the inference by the face recognition model 23 has failed (NO in S27), and affects the influence of the light source. In consideration of this, the user is instructed to move the camera 2 to an installation position and an installation angle suitable for face recognition (gender / age estimation) (S28). When the accuracy of the inference processing measured above is less than a predetermined value, the faces of the 3D model people (for example, the 3D model people 91a to 91d in FIG. 20) in the superimposed image are faced. The case where the result of gender / age estimation by the recognition model 23 is different from the gender / age set for each of the people of these 3D models is included.

Specifically, as shown in FIG. 20, the notification unit 43 of the smartphone 1 detects and recognizes the face of the person 91c on the line 89 (gender / age) at the current installation position and installation angle of the camera 2. (Estimation) is possible, but if the faces of people 91a, 91b, and 91d on line 89 cannot be detected and recognized, as shown in FIG. 21, "recognition of all faces in consideration of the influence of the light source". The message 93 "Please move the camera to a position and angle suitable for the camera" is displayed on the touch panel 14. Each of 92a to 92d in FIGS. 20 to 23 is a bounding box corresponding to each face of the above-mentioned 3D model person 91a to 91d. Further, when the inference by the face recognition model 23 fails (NO in S27), the instruction given to the user by the notification unit 43 is the installation position and angle of the camera 2 estimated in S24 and the person specified by the touch panel 14. It may be an instruction to move the camera 2 to a specific installation position and installation angle suitable for face detection (considering the influence of the light source) based on the position of (the position of the line 89). In this case, the instruction given to the user by the notification unit 43 is a message similar to the message 76 shown in FIG. 12 of the first embodiment (for example, "the position of the camera is raised in consideration of the influence of the light source". , Please tilt horizontally ").

According to the instruction in S28 above, the user moves the camera 2 to an installation position and an installation angle suitable for face recognition (gender / age estimation) in consideration of the influence of the light source (adjusts the position and angle of the camera 2). ) And (S29), the CPU 12 of the smartphone 1 estimates the installation position / angle of the camera 2 in S24, the face detection process in S25, the gender / age estimation process (according to the face recognition model 23), and the S26. The accuracy measurement of the inference processing of gender / age estimation by the face recognition model 23 is executed again. Then, when the accuracy of the inference processing for gender / age estimation by the face recognition model 23 exceeds a predetermined value (probability of being classified as either male or female, and probability of being classified as any age (or age group)). The result of gender / age estimation by the face recognition model 23 for each (face) of the 3D model person in the above superimposed image, which is equal to or more than a predetermined value, is the gender set for each of these 3D model people. (When it becomes the same as the age), the CPU 12 of the smartphone 1 determines that the inference by the face recognition model 23 for the person of the 3D model in the currently displayed superimposed image is successful (YES in S27), and determines S30. Move on to processing.

In the determination process of S30, it is determined whether or not the inference by the face recognition model 23 for the person of the 3D model in the superimposed image of all patterns is successful. Here, the "superimposed image of all patterns" means that the image of the 3D model corresponding to the situation of the customer base of all the selection buttons 87 selected by the user on the situation selection menu screen 86 is randomly read from the environment data file 84. It means all the images generated in the environment (data) of 4 to 5 lights and superimposing the images of these 3D models on the input images from the camera 2. That is, the “superimposed image of all patterns” means that each of the images of the 3D model corresponding to the situation of a plurality of customer groups assumed at the installation location of the camera 2 is the environmental data of light stored in the environment data file 84. Generated in 4 to 5 random light environments (“position / direction / light intensity / color” and shading of the light source) included in, and use these 3D model images as input images from camera 2. It means all of the superimposed images.

In the determination process of S30, if the inference by the face recognition model 23 for the person of the 3D model in the superimposed image of all patterns is not completed (NO in S30), the 3D model overlay portion of SoC11 81 modifies at least one of the 3D model image of the person in the superimposed image and the light environment (randomly read from the environment data file 84) (S31). That is, the 3D model superimposition unit 81 randomly reads a superimposition image based on the image of the 3D model corresponding to the situation of the customer base of a certain selection button 87 selected by the user on the situation selection menu screen 86 from the environment data file 84. If a predetermined number (for example, 4 to 5) of light environments have not yet been generated in all the environments, the image of the human 3D model in the above superimposed image is not changed, and the above Change only the light environment. On the other hand, the 3D model superimposition unit 81 randomly reads a superimposition image based on the image of the 3D model corresponding to the situation of the customer base of a certain selection button 87 selected by the user on the situation selection menu screen 86 from the environment data file 84. If it has already been generated in all of the predetermined number of light environments, the image of the 3D model of the person in the above superimposed image is changed.

That is, the 3D model superimposition unit 81 sets the attributes such as the gender and age of the person and the number of people in the image of the 3D model of the person to be superimposed on the input image to the situations of various customer groups selected by the user on the situation selection menu screen 86. A predetermined number (for example, 4 to 5) of lights randomly read from the environment data file 84 as the light environment used to generate images of these 3D models while changing the corresponding attributes and number of people over time. The inference process of gender / age estimation by the face recognition model 23 for the superimposed image based on the image of a certain 3D model in a certain light environment is repeated while changing the environment over time.

In the determination process of S30, the CPU 12 of the smartphone 1 repeats the processes of S24 to S31 until the inference by the face recognition model 23 for the person (face) of the 3D model in the superimposed image of all patterns is completed (successful). .. Then, when the inference by the face recognition model 23 for the person (face) of the 3D model in the superimposed image of all patterns is completed (successful) (YES in S30), the CPU 12 (notification unit 43) of the smartphone 1 is shown in FIG. 23. As shown in (S32), the message 94 "adjustment is completed" is displayed on the touch panel 14.

As described above, the notification unit 43 of the smartphone 1 is an instruction (message 88) for improving the accuracy of the inference processing by the face recognition model 23 (to support the user in adjusting the installation position and the installation angle of the camera 2). , 93, etc.) is displayed on the touch panel 14.

The above instruction may be displayed by any of the following real-time type or command type. In the case of the real-time type, as shown in the flowchart of FIG. 17 above, "display the superimposed image in which the image of the 3D model is superimposed on the input image is displayed on the touch panel"-> "recognize the superimposed image (in)"-> " Since the process of "displaying instructions (instructions to the user)" is executed in real time (automatically repeated), the user can follow the instructions (instructions) updated in real time to install the camera 2 at the installation position and angle. Should be adjusted. On the other hand, in the case of the command type, when the user sets the camera 2 of the smartphone 1 to the approximate position and angle and presses the check button of the smartphone 1, the above "3D model image is used as the input image". The process of "displaying the superimposed superimposed image on the touch panel"-> "recognizing the superimposed image (in)"-> "displaying the instruction" is executed. When the user presses the check button after adjusting the installation position and installation angle of the camera 2 according to this instruction (instruction), the above processing (3D model image superimposition display, image (face) recognition, and instruction) Since the display) is repeated, the user presses the check button again after readjusting the installation position and installation angle of the camera 2 according to this instruction, and the message 94 "Adjustment is completed" is displayed. Until then, it will be repeated.

Although the above real-time method is more user-friendly, it consumes a considerable amount of computer resources of the smartphone 1, so in the case of the smartphone 1 whose processing power is not high, the above command method is easier to set the system. I think that the.

As described above, according to the smartphone 1 of the second embodiment and the image analysis application 21 (hereinafter, abbreviated as "smartphone 1 of the second embodiment"), the smartphone 1 and the image of the first embodiment. In addition to the effects of the analysis application 21, it has the following effects. That is, according to the smartphone 1 or the like of the second embodiment, the image analysis unit 41 superimposes one 3DCG person image superimposed on the input image (frame image) from the camera 2 by the 3D model superimposition unit 81. After analysis using the above trained NN models (face detection model 22 and face recognition model 23), the inference accuracy measuring unit 42 uses one of the above trained NN models to be trained. The accuracy of the inference processing by the model (face recognition model 23) is measured, and the notification unit 43 notifies the user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing described above. did. As a result, according to the smartphone 1 and the like of the second embodiment, unlike the first embodiment, the person does not actually stand at the shooting position or move on the line for the person of 3DCG. Based on the accuracy of the inference processing by the learned NN model, it is possible to notify the user of instruction information for improving the accuracy of the inference processing. Therefore, according to the smartphone 1 or the like of the second embodiment, the user does not have to actually stand at the shooting position or move on the line, but the user can install the camera 2 of the smartphone 1 (shooting position). ) And the installation angle (shooting direction) can be easily adjusted.

Further, according to the smartphone 1 or the like of the second embodiment, the 3D model superimposing unit 81 allows the user to determine the attributes and the number of people such as the gender and age of the person in the image of the 3D model of the person to be superimposed on the input image. The attributes and the number of people corresponding to the situations of various customer groups selected on the selection menu screen 86 are changed over time, and the image analysis unit 41 uses the above-mentioned images of the attributes and the number of people changed over time as input images. The various superimposed images were analyzed one after another. As a result, the notification unit 43 can urge the user to move the camera 2 to an installation position and an installation angle that is strong in image analysis (face detection or recognition) of various patterns of attributes and the number of people.

Further, according to the smartphone 1 and the like of the second embodiment, the 3D model superimposing unit 81 has attributes such as gender and age of the person and the number of people in the 3D model image of the person to be superposed on the input image as described above. A predetermined number (for example, 4 to 5) of randomly reading the light environment used to generate images of these 3D models from the environment data file 84, as well as changing the number of people to various attributes over time. The light environment is changed over time. Then, the image analysis unit 41 sequentially superimposes various superimposed images on which images of various attributes and numbers of people are superimposed on the input image in the above-mentioned predetermined number (for example, 4 to 5) light environments. I tried to analyze it. As a result, the notification unit 43 can urge the user to move the camera 2 to an installation position and an installation angle that is strong in analyzing images of various patterns of attributes and numbers of people in various light environments.

Modification example:
The present invention is not limited to the configuration of each of the above embodiments, and various modifications can be made without changing the gist of the invention. Next, a modification of the present invention will be described.

Modification 1:
In the first embodiment, the inference accuracy measuring unit 42 measures the inference processing accuracy of the object detection NN model (face detection model 22 and the person detection model), and the notification unit 43 measures the object detection NN model. An example of notifying the user of instructions for improving the accuracy of inference processing has been shown, but the inference accuracy measurement unit determines the accuracy of inference processing of NN models for object recognition (face recognition model, vectorization model, etc.). The measurement may be performed and the notification unit may notify the user of an instruction for improving the accuracy of the inference processing of the NN model for object recognition.

Modification 2:
In each of the above embodiments, the instruction is notified to the user by displaying the instruction for improving the accuracy of the inference processing of the NN model on the touch panel 14, but the instruction for improving the accuracy of the inference processing of the NN model is performed. May be notified to the user by voice using the speaker of the smartphone.

Modification 3:
In each of the above embodiments, the instruction (instruction information) displayed on the touch panel 14 in order to improve the accuracy of the inference processing of the NN model is a frame (red frame 61 and green frame 63) or a message in the display image. (Messages 64, 65, 70, 73, 76, 77, 88, 93, etc.) have been shown, but the instructions displayed on a display device such as a touch panel are not limited to this. For example, the mark or the like on the display screen may be blinked instead of the red frame, and the mark or the like on the display screen may be lit instead of the green frame. Further, instead of displaying a message (of text information) indicating the direction in which the camera is moved as shown in FIG. 12, an arrow indicating the direction in which the camera is moved may be displayed.

Modification 4:
In each of the above embodiments, an example is shown in which the smartphone 1 uses the built-in GPU 13 to perform inference processing of each NN model, but the smartphone 1 replaces the built-in GPU 13 with an external DNN. The inference processing of each NN model may be performed using the inference expansion chip, or the inference processing of each NN model may be performed using both the built-in GPU 13 and the external DNN inference expansion chip. ..

Modification 5:
In each of the above embodiments, the case where the information processing terminal in the claim is the smartphone 1 is shown, but the information processing terminal is not limited to this, and for example, even a tablet computer provided with a camera is used. Good.

Modification 6:
In the second embodiment described above, an example is shown in which the "human attribute" in the claim is mainly gender, age, and body shape (physical constitution such as height), but the "human attribute" is this. For example, clothes and belongings may be included. Further, in the second embodiment, the attributes and the number of people in the image of the 3D model superimposed on the input image and the light environment used for generating the image of the 3D model are changed (changed) one after another. ) However, even if the attributes and number of people in the 3D model image superimposed on the above input image and the direction of the person in the 3D model image superimposed on the input image are changed one after another in addition to the light environment. good. As a result, the notification unit 43 can urge the user to move the camera 2 to an installation position and an installation angle that are strong against patterns of various human orientations.

Modification 7:
In the second embodiment, as shown in the flowchart of FIG. 17, "displaying a superimposed image in which the image of the 3D model is superimposed on the input image is displayed on the touch panel"->"recognizing the face (in) of the superimposed image". -> Repeat the process of "Display instructions (instructions to the user)". Therefore, for example, the image of a person in the 3D model displayed every 200 milliseconds and the light environment change, so that the image displayed on the touch panel flickers considerably. In order to avoid the flickering of this screen, the superimposed image in which the image of the 3D model in the random light environment is superimposed on the input image from the camera is not actually displayed on the touch panel, but this superimposed image is displayed. The image analysis unit (face recognition model) recognizes the superimposed image (in) that is output to the off-screen memory (off-screen buffer) in the graphics memory 82 shown in FIG. 16 and stored in the off-screen memory. You may do so.

1 Smartphone (information processing terminal)
2 Camera 4 Smartphone management server (management server)
10 Image analysis system 14 Touch panel (display device, pointing device)
21 Image analysis application (image analysis program)
22 Face detection model (learned object detection neural network model)
41 Image analysis unit 42 Inference accuracy measurement unit 43 Notification unit 44 Camera position / direction estimation unit 61 Red frame (part of "instruction information")
63 Green frame (part of "instruction information")
64, 65, 70, 73, 76, 77, 88, 93 messages (part of "instruction information")
81 3D model superimposition part (image superimposition part)
91a-91d 3D model person (image of person or face)

In order to solve the above problems, the image analysis program according to the first aspect of the present invention uses an information processing terminal equipped with a camera to detect a trained object for detecting an object reflected in an input image from the camera. One of the image analysis unit that analyzes the input image from the camera and the one or more trained neural network models using one or more trained neural network models including the neural network model for the camera. An inference accuracy measurement unit that measures the accuracy of inference processing by a trained neural network model, a camera position direction estimation unit that estimates the installation position and installation angle of the camera, and a display device for displaying an input image from the camera. When the the input image from the camera which is displayed on the display device, and an operation section for designating the detection position of the object, the inference result of the measurement that by the precision measurement unit, the measured inference processing When the accuracy is less than a predetermined value, the inference process is based on the camera installation position and installation angle estimated by the camera position direction estimation unit and the detection position of the object specified by the operation unit. It functions as a notification unit that notifies the user of instruction information for improving the accuracy of the camera.

In the image analysis program, the notification unit, by displaying the instruction information on the display device, may be notified of the instruction information to the user.

The information processing terminal according to the third aspect of the present invention includes a camera and one or more trained neurals including a trained object detection neural network model for detecting an object reflected in an input image from the camera. An image analysis unit that analyzes an input image from the camera using a network model, and an inference that measures the accuracy of inference processing by one of the trained neural network models of the one or more trained neural network models. An accuracy measurement unit, a camera position direction estimation unit that estimates the installation position and installation angle of the camera, a display device for displaying an input image from the camera, and an input from the camera displayed on the display device. the image, an operation section for designating the detection position of the object, the inference result of the precision measuring portion that by the measurement, if the accuracy of the measured inference processing is less than a predetermined value, the camera position direction Based on the installation position and installation angle of the camera estimated by the estimation unit and the detection position of the object specified by the operation unit , the user is notified of instruction information for improving the accuracy of the inference processing. It is equipped with a notification unit.

In the information processing terminal, wherein the notification unit, by displaying the instruction information on the display device, may be notified of the instruction information to the user.

In this information processing terminal, the operation unit is a pointing device, and the notification unit is the camera estimated by the camera position direction estimation unit in addition to the accuracy of the inference processing measured by the inference accuracy measurement unit. The instruction information may be obtained based on the installation position and the installation angle of the above and the detection position of the person or face designated by the pointing device.

The second image analysis program according to aspects of the present invention, according to the information processing terminal according to the fourth aspect及beauty, the one or more trained neural network model, the inference processing with either learned neural network model The accuracy is measured, and based on the measured accuracy of the inference processing, instruction information for improving the accuracy of the inference processing is notified to the user. Based on this notified instruction information, the user adjusts the installation position (shooting position) and installation angle (shooting direction) of the camera of the information processing terminal, and the specialized technical staff goes to the camera installation location. Therefore, it is possible to obtain an appropriate inference accuracy of the trained neural network model (learned object detection neural network or trained object recognition neural network) without adjusting the installation position and installation angle of the camera. Therefore, the user can easily install the camera.
Further, according to the image analysis program according to the first aspect of the present invention and the information processing terminal according to the third aspect, inference processing by one of the trained neural network models among one or more trained neural network models. If the accuracy of the measured inference processing is less than a predetermined value, the inference processing is performed based on the estimated installation position and angle of the camera and the detection position of the object specified by the operation unit. The user is notified of the instruction information for improving the accuracy of the information processing. Based on this notified instruction information, the user adjusts the installation position (shooting position) and installation angle (shooting direction) of the camera of the information processing terminal, and the specialized technical staff goes to the camera installation location. Therefore, it is possible to obtain an appropriate inference accuracy of the trained neural network model (learned object detection neural network or trained object recognition neural network) without adjusting the installation position and installation angle of the camera. Therefore, the user can easily install the camera.

Claims (16)

An information processing terminal equipped with a camera
An image that analyzes an input image from the camera using one or more trained neural network models, including a trained object detection neural network model for detecting an object reflected in the input image from the camera. With the analysis department
An inference accuracy measuring unit that measures the accuracy of inference processing by any of the trained neural network models among the one or more trained neural network models.
An image analysis program for functioning as a notification unit for notifying a user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measurement unit. The object detected by the trained object detection neural network model is a person or a face.
The instruction information is characterized in that it is instruction information that prompts the user to move the camera to an installation position or an installation angle suitable for detecting the person or face by the learned object detection neural network model. The image analysis program according to claim 1. The image analysis program according to claim 1 or 2, wherein the notification unit notifies the user of the instruction information by displaying the instruction information on a display device. An information processing terminal equipped with a camera
An image superimposing unit that superimposes a person or face image on the input image from the camera, and
An image analysis unit that analyzes a person or face image superimposed by the image superimposition unit using one or more trained neural network models, and an image analysis unit.
An inference accuracy measuring unit that measures the accuracy of inference processing by any of the trained neural network models among the one or more trained neural network models.
An image analysis program for functioning as a notification unit for notifying a user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measurement unit. The claim is characterized in that the instruction information is instruction information for prompting the user to move the camera to an installation position or an installation angle suitable for performing inference processing by any of the learned neural network models. Item 4. The image analysis program according to item 4. The image analysis program according to claim 4 or 5, wherein the notification unit notifies the user of the instruction information by displaying the instruction information on a display device. The image superimposing unit changes the attributes and numbers of people in the image of a person or face superimposed on the input image with time to various attributes and numbers.
The image analysis unit analyzes various images obtained by superimposing the images of people or faces with the attributes and numbers changed over time on the input images one after another.
The notification unit is based on the accuracy of inference processing measured by the inference accuracy measurement unit for various images in which images of people or faces with attributes and numbers changed over time are superimposed on the input image. The image analysis program according to any one of claims 4 to 6, wherein the user is notified of instruction information for improving the accuracy of the inference processing. With the camera
An image that analyzes an input image from the camera using one or more trained neural network models, including a trained object detection neural network model for detecting an object reflected in the input image from the camera. With the analysis department
An inference accuracy measuring unit that measures the accuracy of inference processing by any of the trained neural network models among the one or more trained neural network models.
An information processing terminal including a notification unit that notifies a user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measurement unit. The object detected by the trained object detection neural network model is a person or a face.
The instruction information is characterized in that it is instruction information that prompts the user to move the camera to an installation position or an installation angle suitable for detecting the person or face by the learned object detection neural network model. The information processing terminal according to claim 8. The information processing terminal further includes a display device for displaying an input image from the camera.
The information processing terminal according to claim 9, wherein the notification unit notifies the user of the instruction information by displaying the instruction information on the display device. The information processing terminal is
A camera position direction estimation unit that estimates the installation position and installation angle of the camera based on the input image from the camera, and
A pointing device for designating the detection position of the person or face with respect to the input image from the camera displayed on the display device is further provided.
In addition to the accuracy of the inference processing measured by the inference accuracy measuring unit, the notification unit includes the installation position and the installation angle of the camera estimated by the camera position direction estimation unit, and the pointing device designated by the pointing device. The information processing terminal according to claim 10, wherein the instruction information is obtained based on the detection position of a person or a face. With the camera
An image superimposing unit that superimposes a person or face image on the input image from the camera, and
An image analysis unit that analyzes a person or face image superimposed by the image superimposition unit using one or more trained neural network models, and an image analysis unit.
An inference accuracy measuring unit that measures the accuracy of inference processing by any of the trained neural network models among the one or more trained neural network models.
An information processing terminal including a notification unit that notifies a user of instruction information for improving the accuracy of the inference processing based on the accuracy of the inference processing measured by the inference accuracy measurement unit. The claim is characterized in that the instruction information is instruction information for prompting the user to move the camera to an installation position or an installation angle suitable for performing inference processing by any of the learned neural network models. Item 12. The information processing terminal according to item 12. The information processing terminal further includes a display device that displays an input image from the camera.
The information processing terminal according to claim 13, wherein the notification unit notifies the user of the instruction information by displaying the instruction information on the display device. The image superimposing unit changes the attributes and numbers of people in the image of a person or face superimposed on the input image with time to various attributes and numbers.
The image analysis unit analyzes various images obtained by superimposing the images of people or faces with the attributes and numbers changed over time on the input images one after another.
The notification unit is based on the accuracy of inference processing measured by the inference accuracy measurement unit for various images in which images of people or faces with attributes and numbers changed over time are superimposed on the input image. The information processing terminal according to any one of claims 12 to 14, wherein instruction information for improving the accuracy of the inference processing is notified to the user. The information processing terminal according to any one of claims 8 to 15.
An image analysis system including a management server that manages the information processing terminal, including installation of the trained neural network model on the information processing terminal.