JP7455889B2 - Image evaluation device, image processing system, user terminal, image evaluation method, and image evaluation program - Google Patents

Description

The present invention relates to an image evaluation device, an image processing system, a user terminal, an image evaluation method, an image evaluation program, and an image display support method.

For example, Patent Document 1 listed below describes a device that detects abnormalities in image data using a trained model.

Japanese Patent Application Publication No. 2022-26016

The inventor considered that image data in which a predetermined part of the body is exposed is considered abnormal. In that case, if the learned model outputs a determination result of the presence or absence of an abnormality in image data without paying attention to a predetermined region, the determination accuracy may be lowered.

Below, means for solving the above problems and their effects will be described.
1. an execution device and a storage device, the storage device stores location information mapping data and evaluation mapping data, the location information mapping data is data for defining a location information mapping, and the storage device stores location information mapping data and evaluation mapping data; The position information mapping is a mapping that outputs position information data, the position information data is data indicating position information of a predetermined part of a person in an image shown by image data to be evaluated, and the evaluation mapping data is The evaluation mapping is data for defining an evaluation mapping, and the evaluation mapping is a mapping that inputs evaluation input data and the position information data and outputs an evaluation result of the image data, and the evaluation input data is a mapping that outputs an evaluation result of the image data. The execution device is configured to execute a position information generation process and an evaluation process, and the position information generation process It is a process of generating the position information data by inputting input data into the position information mapping, and the position input data is data that corresponds to the image data and is data that is input to the position information mapping. In the image evaluation apparatus, the evaluation process is a process of evaluating the image data by inputting the position information data and the evaluation input data to the evaluation mapping.

In the above configuration, not only the evaluation input data corresponding to the image data but also the position information data is input to the evaluation mapping. The position information data is data indicating information on a predetermined region. Therefore, the evaluation mapping can output an evaluation result using information about the position of a predetermined part in the image data. Therefore, abnormalities such as exposure of a predetermined region can be evaluated with higher precision than when position information data is not used.

2. The image evaluation device according to 1 above, wherein the predetermined region includes at least one of the following three parts: a person's chest, buttocks, and front lower abdomen.
The presence or absence of exposure of the three parts of a person's chest, buttocks, and front lower abdomen is particularly important in determining whether or not it violates public order and morals. Therefore, with the above configuration, by using the position information of the predetermined part, it is possible to evaluate with high accuracy whether or not the image data violates public order and morals.

3. The position information data is data that provides information as to whether each pixel forming the evaluation input data indicates the predetermined region, and the evaluation mapping is data that provides information on whether each pixel forming the evaluation input data indicates the predetermined region, and the evaluation mapping is data that provides information on whether each pixel forming the evaluation input data indicates the predetermined region. The image evaluation device according to the above 1 or 2, wherein the image evaluation device is a mapping that outputs the evaluation result by inputting the position information data corresponding to the pixels in association with each other.

In the evaluation mapping, evaluation input data and position information data are input into the evaluation mapping in association with each other. Therefore, it is possible to increase the reliability of calculating an evaluation result while paying attention to a pixel corresponding to a predetermined region among each pixel of image data.

4. The evaluation mapping includes a provisional evaluation mapping, the provisional evaluation mapping is a mapping that inputs the evaluation input data and outputs a provisional evaluation result of the image data, and the evaluation processing includes provisional evaluation processing; a validity evaluation process, the provisional evaluation process includes a process of outputting the provisional evaluation result by inputting the evaluation input data into the provisional evaluation mapping, and the validity evaluation process includes the The above-described process includes a process of taking position information data as input and evaluating the validity of the tentative evaluation result according to the degree to which a region indicating the predetermined part out of the regions indicated by the image data has contributed to the tentative evaluation result. The image evaluation device according to any one of items 1 to 3.

In the above configuration, the execution device evaluates the validity of the tentative evaluation result according to the degree to which the region indicating a predetermined part of the region indicated by the image data has contributed to the tentative evaluation result output by the tentative evaluation mapping. Execute processing. Therefore, compared to the case where the final evaluation result is determined from the provisional evaluation result output by the provisional evaluation mapping without evaluating the validity, the accuracy of the final evaluation result can be improved.

5. 4. The execution device is configured to execute a notification process, and the notification process is a process of notifying that it is not valid when it is determined that it is not valid by the validity evaluation process. It is an image evaluation device.

In the above configuration, when the execution device determines that the validity evaluation process is not valid, it notifies the execution device to that effect. Therefore, it becomes possible for a person to ultimately judge the validity of the provisional evaluation results.

6. The evaluation mapping includes a feature layer, and the feature layer is a layer that quantifies the feature amount of the image data by dividing the region indicated by the image data into a plurality of regions and assigning a numerical value to each region. 1 above, wherein the evaluation mapping is a mapping that outputs the evaluation result including a process of combining at least some values of the plurality of regions indicated by the feature amount layer and the position information data. This is an image evaluation device.

In the above configuration, the execution device performs a process of synthesizing at least part of the values of the plurality of regions indicated by the feature amount layer and the position information data. Therefore, the feature amount that has been subjected to this processing can be a feature amount that focuses more on a predetermined region than the feature amount that has not been corrected. Therefore, it is possible to obtain an evaluation result that reliably focuses on a predetermined region.

7. The image evaluation device according to any one of items 1 to 6 above includes the execution device, the storage device, and a plurality of user terminals, and the execution device is configured to execute provision processing and restriction processing. and the provision process is configured to provide the image data in order to enable the user terminal to display an image indicated by the image data that has been evaluated by the evaluation process to the effect that the display does not need to be restricted. The restriction process is a process of transmitting to the user terminal, and the restriction process is a process of transmitting the image to the user terminal, and the restriction process is a process of transmitting the image to the user terminal, and the restriction process is a process of transmitting the image to the user terminal, and the restriction process is a process of transmitting the image to the user terminal. This is an image processing system that limits processing.

According to the above-mentioned provision processing and restriction processing, it is possible to appropriately support selectively displaying the image indicated by the image data on the user terminal according to the evaluation result of the evaluation processing.
8. The user terminal is configured to execute an instruction process, and the instruction process is a process for instructing a permissible range for the degree of body exposure, and the evaluation that there is no need to limit the degree of exposure means that the degree of exposure is The image processing system according to 7 above, wherein the evaluation that the degree of exposure is within the permissible range and that displaying by the user terminal should be restricted is an evaluation that the degree of exposure is outside the permissible range. It is.

With the above configuration, it is possible to appropriately support selectively displaying an image indicated by image data on a user terminal according to the user's intention.
9. The restriction process includes a prohibition process that prohibits transmission of the image data that has been evaluated to be restricted from being displayed by the user terminal to the user terminal, and a prohibition process that prohibits the image data from being displayed by the user terminal. and a restriction transmission process of transmitting the image data evaluated as such along with a restriction command, the restriction command causing an image indicated by the image data to be displayed on the user terminal along with a warning. 7 above, which is any one of a command, a command to mask the exposure of a predetermined part of the image represented by the image data on the user terminal, and a command not to display the image represented by the image data on the user terminal. or the image processing system described in 8.

According to the above-mentioned prohibition process, it is possible to reliably suppress the image indicated by the image data evaluated to be restricted from being displayed on the user terminal from being displayed on the user terminal. Further, according to the above-mentioned limited transmission processing, one of the following becomes possible.

- A warning is issued by the user terminal to the effect that display on the user terminal should be restricted.
- The exposure of a predetermined part in the image displayed on the user terminal is masked.

- Prohibiting the user terminal from displaying an image indicated by image data that has been evaluated as being restricted from being displayed by the user terminal.
10. The user terminal in the image processing system described in 8 above.

11. An image evaluation method comprising the steps of executing each of the processes in the image evaluation apparatus according to any one of items 1 to 6 above.
12. This is an image evaluation program that causes a computer to execute each of the processes in the image evaluation apparatus described in any one of 1 to 6 above.

13. An image display support method for supporting selective display of images on a user terminal, the method comprising: selecting any one of the above items 1 to 6 among image data indicating image candidates to be displayed by the user terminal; In order to be able to display, on the user terminal, an image indicated by the image data that has been evaluated to the effect that displaying does not need to be restricted by the evaluation process in the image evaluation device described above, the image data is displayed on the user terminal. and a restricting step of restricting display of an image represented by the image data that has been evaluated by the evaluation process as being restricted from being displayed by the user terminal. This is an image display support method.

According to the above-mentioned providing step and restriction step, it is possible to appropriately support selectively displaying the image indicated by the image data on the user terminal according to the evaluation result of the evaluation process.
14. An image display support method for supporting selective display of images on a user terminal, wherein the user terminal is configured to execute instruction processing, and the instruction processing includes determining a permissible degree of body exposure. This is a process of instructing a range, and the exposure degree is determined by the evaluation process in the image evaluation apparatus according to any one of 1 to 6 above among image data indicating image candidates to be displayed by the user terminal. a providing step of transmitting the image data to the user terminal so that the image indicated by the image data evaluated to be within an acceptable range can be displayed on the user terminal; and a candidate image to be displayed by the user terminal. a limiting step of restricting display of an image represented by the image data whose exposure degree is evaluated to be outside the permissible range by the evaluation process from being displayed by the user terminal. This is an image display support method.

The above method can appropriately support selectively displaying images indicated by image data on a user terminal according to the user's intention.
15. An image display support method for supporting selective display of images on a user terminal, the method comprising: an acquisition step of acquiring image data indicating image candidates to be displayed by the user terminal; The image represented by the image data that has been evaluated by the evaluation process in the image evaluation device according to any one of the above items 1 to 6 to the effect that it is not necessary to restrict display of the image data. a providing step of transmitting the image data to the user terminal so that the image data can be displayed on the user terminal; and a step of restricting display of the image data acquired in the acquisition step by the evaluation process. The image display support method includes the step of restricting display of an image indicated by the evaluated image data by the user terminal.

According to the above-mentioned providing step and restricting step, it is possible to appropriately support selectively displaying the image indicated by the image data on the user terminal according to the evaluation result of the evaluation process.

1 is a diagram showing the overall configuration of an image processing system according to a first embodiment. FIG. 3 is a diagram defining displayable images according to the same embodiment. 2 is a flowchart showing a procedure for processing related to display support for an image indicated by commercial video data according to the same embodiment. FIG. 3 is a diagram illustrating a detailed procedure of frame evaluation processing according to the embodiment. It is a figure for explaining the heat map concerning the same embodiment. 12 is a flowchart illustrating a process procedure related to display support for an image indicated by CM video data according to a second embodiment. 12 is a flowchart showing detailed steps of frame evaluation processing according to the third embodiment. FIG. 7 is a diagram showing detailed procedures of frame evaluation processing according to the fourth embodiment. It is a figure showing the detailed procedure of final evaluation processing concerning the same embodiment. FIG. 7 is a diagram showing detailed procedures of frame evaluation processing according to the fifth embodiment. 2 is a flowchart showing a procedure for processing related to display support for an image indicated by commercial video data according to the same embodiment.

<First embodiment>
The first embodiment will be described below with reference to the drawings.
"Prerequisite configuration"
FIG. 1 shows the overall configuration of the image processing system.

The plurality of merchant terminals 10(1), 10(2), . . . are terminals owned by merchants who sell certain products to consumers. Note that hereinafter, when the vendor terminals 10(1), 10(2), . . . are collectively referred to as the vendor terminal 10. The vendor terminal 10 includes a PU 12, a storage device 14, and a communication device 16. The PU 12 is a software processing device including arithmetic units such as a CPU, a GPU, and a TPU. The storage device 14 includes an electrically rewritable nonvolatile memory and a storage medium such as a disk medium. The storage device 14 stores programs executed by the PU 12.

The communication device 16 of the vendor terminal 10 is enabled to communicate with the image evaluation device 30 via the network 20. The vendor terminal 10 transmits CM video data regarding a product that the vendor wants to sell to the image evaluation device 30. Here, the commercial video data corresponds to, for example, a commercial video for advertising a product. There are no restrictions on the playback time, number of frames, resolution, or data format of commercial videos.

The image evaluation device 30 evaluates the commercial video data, and if there is no problem, transmits the data to the user terminals 50(1), 50(2), . . . In the following, when the user terminals 50(1), 50(2), . . . are collectively referred to as the user terminal 50.

The image evaluation device 30 includes a PU 32, a storage device 34, and a communication device 36. The PU 32 is a software processing device that includes arithmetic units such as a CPU, a GPU, and a TPU. Specifically, the PU 32 may include a hardware accelerator specialized for predetermined processing such as image processing, inference processing, and parallel processing as an arithmetic unit. Further, the PU 32 may include an arithmetic unit that takes the form of a homogeneous architecture or a heterogeneous architecture. Further, the PU 32 may include an arithmetic unit monolithically integrated on a single chip, or may include an arithmetic unit to which a plurality of chips such as chiplets are connected. The storage device 34 includes an electrically rewritable nonvolatile memory and a storage medium such as a disk medium. The storage device 34 stores an image evaluation program 34a, position information mapping data 34b, and evaluation mapping data 34c. The communication device 36 is a device that enables communication with the vendor terminal 10 and the user terminal 50 via the network 20.

The user terminal 50 is a terminal that plays back commercial video data. The user terminal 50 may have a function of executing a purchasing operation for a product indicated by a commercial video.
The user terminal 50 includes a PU 52, a storage device 54, a communication device 56, and a user interface 58. The PU 52 is a software processing device that includes arithmetic units such as a CPU, a GPU, and a TPU. PU52 may include an arithmetic unit in a manner that can be included in PU32. The storage device 54 includes an electrically rewritable nonvolatile memory and a storage medium such as a disk medium. The storage device 54 stores an application program 54a. The application program 54a is a program that plays back commercial video data transmitted from the image evaluation device 30. The communication device 56 is a device that enables communication with the image evaluation device 30 via the network 20. The user interface 58 includes a display device and the like.

"Image evaluation processing"
The PU 32 of the image evaluation device 30 evaluates whether the CM video data transmitted from the vendor terminal 10 is data that can be reproduced by the user terminal 50 without any problem.

FIG. 2 shows an example of the definition of data that can be reproduced without any problem. In FIG. 2, markings are made at predetermined parts of the body. That is, markings are made on the chest defined by portions "3" to "5". Furthermore, in a front view of the human body, markings are made on the "front lower abdomen" defined by portions "10", "11", and "18". Also, markings are made on the "buttocks" defined by parts "12 to 14" and "18" on the back of the human body. The PU 32 evaluates an image of a person in which the portion marked in FIG. 2 is exposed as an image that poses a problem in reproduction.

FIG. 3 shows the procedure of a process executed by the image evaluation device 30 to support selective display of CM video data by the user terminal 50 based on evaluation in accordance with the criteria shown in FIG. The process shown in FIG. 3 is realized by the PU 32 repeatedly executing the image evaluation program 34a stored in the storage device 34, for example, at a predetermined period. Note that in the following, the step number of each process is expressed by a number prefixed with "S".

In the series of processes shown in FIG. 3, the PU 32 first obtains CM video data transmitted from the vendor terminal 10 (S10). Note that here, "obtaining" means selectively reading out one of the CM video data stored in the storage device 34. Before this process, the PU 32 receives CM video data transmitted from the vendor terminal 10 via the communication device 36. Then, the PU 32 stores the received CM video data in the storage device 34.

Next, the PU 32 initializes a variable i that specifies a frame of CM video data (S12). This process is a process for setting the variable i to a value that specifies the first frame of the commercial video data. Next, the PU 32 samples the frame data FD three times at N intervals (S14). That is, the PU 32 samples frame data FD(i), FD(i+N), and FD(i+2N). Note that frame data FD(1), FD(2), FD(3), . . . indicate a time series of frame data FD according to the reproduction order of CM video data. Specifically, the larger the number in parentheses after the frame data FD, the later the frame data FD is reproduced in chronological order. Incidentally, the frame data FD is two-dimensional data having the number of pixels of "w×h" indicating the brightness of each of the three primary colors of red, green, and blue. Here, "w" and "h" are natural numbers. In addition, below, the said three primary colors will be expressed as "R, G, B" suitably.

Then, the PU 32 executes a frame evaluation process for evaluating each of the frame data FD(i), FD(i+N), and FD(i+2N) (S16).
FIG. 4 shows details of the frame evaluation process.

In the series of processes shown in FIG. 4, the PU 32 inputs one frame data FD to a regression model that outputs positional information of the chest, front lower abdomen, buttocks, and face (S30). The regression model is a model that outputs a two-dimensional distribution indicating the existence probability of each of the chest, front lower abdomen, buttocks, and face in at least part of the "w x h" image area indicated by the frame data FD. be. Here, the two-dimensional distribution is, for example, a two-dimensional Gaussian distribution.

The PU 32 outputs, for example, the value of the xy coordinate component and the dispersion parameter in which the two-dimensional Gaussian distribution regarding the existence probability of each of the chest, frontal lower abdomen, buttocks, and face shows the average value. In this embodiment, the covariance matrix of a two-dimensional Gaussian distribution is regarded as a diagonal matrix. Furthermore, in this embodiment, two diagonal components are considered to be equal. Therefore, the two-dimensional Gaussian distribution of this embodiment has one dispersion parameter. In the regression model, the upper limit of the number of people that can be handled within the image area is a predetermined number k. Then, in the regression model, the values of the xy coordinate components and the values of the dispersion parameters are output for each person regarding the Gaussian distribution that expresses the distribution of the existence probability of the chest, front lower abdomen, buttocks, and face. do. Therefore, the regression model has, by way of example, "12k" output values.

The regression model is, for example, a trained model learned by supervised learning.
FIG. 5 illustrates training data for the regression model. The training data is data in which the coordinates (xi, yi: i=1 to 4) of representative points of the chest, front lower abdomen, buttocks, and face, and the values of the dispersion parameters are defined. The dispersion parameter is set depending on the size of the person occupying the image area.

Note that the regression model may be a model including a neural network (hereinafter referred to as NN). Specifically, it may be a convolutional neural network (hereinafter referred to as CNN). However, it is not limited to CNN.

The regression model is defined by the position information mapping data 34b stored in the storage device 34 shown in FIG. The position information mapping data 34b is data that includes values of learned parameters of a regression model that is a parametric model. That is, when the regression model is CNN, the position information mapping data 34b includes the values of each filter used in the convolution process. Furthermore, when the regression model includes a fully connected layer or the like, the position information mapping data 34b includes the value of a parameter indicating the weight of the fully connected layer. Note that when the regression model is a NN, the position information mapping data 34b may include data that defines an activation function. However, for example, data defining the activation function may be included in the image evaluation program 34a. Note that the regression model may be a trained model learned by unsupervised learning, or a trained model learned by semi-supervised learning.

Returning to FIG. 4, when completing the process of S10, the PU 32 generates a heat map for each of the chest, front lower abdomen, buttocks, and face based on the output value of the regression model (S32). Each of these heat maps has "w×h" areas. The heat map is a map in which a value is determined for each of "w×h" areas according to the existence probability. For example, in the heat map of the chest, the largest value is determined in the region corresponding to the coordinates (x1, y1) of the average value of the two-dimensional Gaussian distribution corresponding to the chest. A smaller value is determined for an area farther away from the area corresponding to the coordinates (x1, y1). The relationship between the distance from the area corresponding to the coordinates (x1, y1) and the value is defined by the dispersion parameter.

Next, the PU 32 calculates the values of evaluation variables yok and yng for evaluating the frame data FD by inputting the frame data FD and the heat map generated from the frame data FD into the identification model 70a (S34 ). The evaluation variable yok is a variable that indicates the probability that the frame data FD is data representing an image in which the chest, front lower abdomen, and buttocks are not exposed. Furthermore, the evaluation variable yng is a variable that indicates the probability that the frame data FD is data representing an image in which at least one of the chest, front lower abdomen, and buttocks is exposed.

The identification model 70a includes CNN, for example. In the identification model 70a, what are the three "w x h" data of R, G, B and the "w x h" heat maps 60 to 66 of the chest, front lower abdomen, buttocks, and face? , one or more filters fl. The filter fl consists of "a×b×7" numerical values. However, "a" is a natural number smaller than "w". Further, "b" is a natural number smaller than "h". This processing means that each pixel region constituting the frame data FD and the corresponding regions of the heat maps 60 to 66 are inputted into the identification model 70a in correspondence with each other.

The identification model 70a has several convolution layers, pooling layers, etc. Note that the CNN of the identification model 70a may have a residual block. In the discriminative model 70a, two output values are output by combining the feature maps in the downstream fully connected layer M10. Then, the values of the evaluation variables yok and yng are calculated using the two-output softmax function M12 as an output activation function.

The identification model 70a is, for example, a trained model that has been trained by supervised learning. The identification model 70a is a model trained using the following training data. That is, there are image data showing images in which the chest, front lower abdomen, and buttocks are not exposed, and image data showing images in which at least one of the chest, front lower abdomen, and buttocks is exposed. Through learning, numerical values of the filter that define the identification model 70a, weight parameters of the fully connected layer M10, etc. are learned. Note that the identification model 70a may be a learned model learned by unsupervised learning, or may be a learned model learned by semi-supervised learning.

The numerical values of the trained filter, the weight parameters of the fully connected layer M10, and the like are included in the evaluation mapping data 34c stored in the storage device 34 shown in FIG. The evaluation mapping data 34c defines the identification model 70a by including filter values, weight parameters of the fully connected layer M10, and the like. Note that the evaluation mapping data 34c may include data that defines an activation function of the CNN. However, the data defining the CNN activation function may be included in the image evaluation program 34a.

When completing the process of S34, the PU32 determines whether the value of the evaluation variable yok is larger than the value of the evaluation variable yng (S36). This process is a process of determining whether the frame data FD represents an image in which the chest, front lower abdomen, and buttocks are not exposed, based on the values of the evaluation variables yok and yng.

When determining that the value of the evaluation variable yok is greater than the value of the evaluation variable yng (S36: YES), the PU 32 evaluates that the chest, front lower abdomen, and buttocks are not exposed (S38). Hereinafter, this evaluation will be referred to as OK determination. On the other hand, when determining that the value of the evaluation variable yok is less than or equal to the value of the evaluation variable yng (S36: NO), the PU 32 evaluates that at least one of the chest, front lower abdomen, and buttocks is exposed ( S40). Hereinafter, this evaluation will be referred to as NG determination.

Note that when the PU 32 completes the processing of S38 and S40 regarding the frame data FD(i), FD(i+N), and FD(i+2N), it completes the processing of S16 in FIG. 3.
Then, the PU 72 determines whether or not two or more of the frame data FD(i), FD(i+N), and FD(i+2N) have been determined to be OK (S18). If the PU 32 determines that there is one or less OK determinations (S18: NO), the PU 32 prohibits distribution of the CM video data acquired through the process of S10 (S20). A negative determination in the process of S18 means that a final determination has been made that at least one of the chest, front lower abdomen, and buttocks is exposed in the CM video data.

On the other hand, if the PU 32 determines that there are two or more OK determinations (S18: YES), it adds "N" to the variable i (S22). The PU 32 then determines whether frame data FD(i+2N) exists (S24). This process is a process for determining whether the process of S16 has been completed for all of the CM video data. If the PU 32 determines that frame data FD (i+2N) exists (S24: YES), the process returns to S14. On the other hand, when the PU 32 determines that the frame data FD (i+2N) does not exist (S24: NO), the PU 32 enables distribution of the CM video data (S26). That is, the PU 32 distributes CM video data in response to a request from the user terminal 50.

Note that when the PU 32 completes the processes of S20 and S26, it temporarily ends the series of processes shown in FIG.
Here, the functions and effects of this embodiment will be explained.

The PU 32 calculates the values of the evaluation variables yok and yng by inputting the frame data FD into the identification model 70a. Then, the PU 32 evaluates the frame data FD according to the values of the evaluation variables yok and yng. This process is a process that uses an evaluation mapping that inputs the frame data FD and outputs an evaluation result of the frame data FD, which is an OK judgment or a NG judgment.

The identification model 70a is a learned model. In particular, the identification model 70a is, for example, a deep neural network (hereinafter referred to as DNN) having a large number of intermediate layers. DNN automatically extracts feature amounts. Therefore, when the evaluation mapping is configured using a DNN, it is simple to input only the frame data FD. However, in that case, the DNN does not necessarily output evaluation results focusing on the chest, front lower abdomen, and buttocks.

Therefore, in this embodiment, the identification model 70a is a model that inputs heat maps 60 to 64 indicating the existence probability of each of the chest, front lower abdomen, and buttocks in addition to the frame data FD. In other words, heat maps 60 to 64 indicating the respective existence probabilities of the chest, front lower abdomen, and buttocks are added to the input of the evaluation map. As a result, compared to the case where the heat maps 60 to 64 are not added, there is a possibility that the feature map showing the feature values extracted in the identification model 70a expresses the respective features of the chest, front lower abdomen, and buttocks. increases. Therefore, the values of the evaluation variables yok and yng can be values that more accurately represent whether at least one of the chest, front lower abdomen, and buttocks is exposed.

It is also conceivable to uniformly mask the areas where the existence probability of the chest, front lower abdomen, and buttocks shown by the heat maps 60 to 64 among the areas shown by the frame data FD is low, and then input the masked areas to the evaluation mapping. However, in that case, information regarding the skin color shown in exposed areas other than the chest, front lower abdomen, and buttocks, information regarding human behavior, etc. will be omitted. However, such information can be useful information in evaluating whether at least one of the chest, front lower abdomen, and buttocks is exposed.

Therefore, according to the above-mentioned identification model 70a, while increasing the possibility of focusing on the chest, front lower abdomen, and buttocks in the image, evaluation variables are The values of yok and yng can be calculated with high precision.

According to the present embodiment described above, the following effects and effects can be obtained.
(1-1) A heat map 66 indicating the existence probability of a face part is included in the input of the identification model 70a. The facial part of the body is most likely to be exposed. Therefore, by focusing on the face, it is possible to increase the possibility of focusing on the person's skin color. Therefore, the values of the evaluation variables yok and yng can be values that more accurately evaluate whether or not at least one of the chest, front lower abdomen, and buttocks is exposed.

(1-2) If OK is determined for two or more of three frame data FDs that are chronologically adjacent at N intervals, it is assumed that there is no problem in the section of 2N+1 frame data. Thereby, even if an erroneous determination is made regarding one frame data FD, the influence of the erroneous determination can be suppressed.

<Second embodiment>
The second embodiment will be described below with reference to the drawings, focusing on the differences from the first embodiment.

FIG. 6 shows the procedure of a process executed by the image evaluation device 30 according to the present embodiment to support selective display of CM video data by the user terminal 50 based on evaluation in accordance with the above criteria. . The process shown in FIG. 6 is realized by the PU 32 repeatedly executing the image evaluation program 34a stored in the storage device 34, for example, at a predetermined period. In addition, in FIG. 6, for convenience, the same step numbers are given to the processes corresponding to the processes shown in FIG. 3, and the description thereof will be omitted.

As shown in FIG. 6, when the PU 32 makes a negative determination in the process of S18 (S18: NO), it gives a restriction command to the CM video data (S20a). The PU 32 then proceeds to the process of S26.

The restriction command is one of the following commands.
- When commercial video data is played back on the user terminal 50, this is a command to issue a warning at the beginning. This may be a process of displaying visual information on a display included in the user interface 58. Alternatively, for example, the process may be performed to output audio information from a speaker included in the user interface 58.

- When commercial video data is played back on the user terminal 50, this is a command to mask the playback image when a scene in which a predetermined part is exposed is played back. The command to mask may be, for example, a command to perform masking by superimposing a predetermined image on the reproduced image. Further, for example, the command to mask may be a command to perform masking by applying an effect such as blurring to the reproduced image. For example, the command to mask may be a command to perform masking by superimposing a predetermined image on a part of the reproduced image that includes at least an exposed predetermined region. Alternatively, for example, it may be a command to mask a part of the reproduced image that includes at least an exposed predetermined region by applying an effect such as a mask.

- This is a command issued to the user terminal 50 to the effect that commercial video data should not be played back. This can be realized, for example, by setting the application program 54a not to reproduce the commercial video data when the application program 54a detects the command.

<Third embodiment>
The third embodiment will be described below with reference to the drawings, focusing on the differences from the first embodiment.

In this embodiment, the identification model 70b is used instead of the identification model 70a.
FIG. 7 shows details of the frame evaluation process. The process shown in FIG. 7 is realized by the PU 32 repeatedly executing the image evaluation program 34a stored in the storage device 34, for example, at a predetermined period. Note that in FIG. 7, the same step numbers are given to the processes corresponding to the processes shown in FIG. 4 for convenience.

In the series of processes shown in FIG. 7, the PU 72 generates the heat maps 60 to 66 (S32) and then reduces the heat maps 60 to 66 (S42). That is, the heat map with the number of regions of "w×h" generated in the process of S32 is reduced to a heat map with the number of regions of "w1×h1". Here, "w1<w, h1<h". For example, when reducing a heat map with a number of regions of "w×h" to a heat map of "1/4", this process may be performed as follows. That is, the average value of the "4 x 4" areas of the heat map with the number of areas of "w x h" may be taken as the value of one area of the heat map with the number of areas of "w1 x h1".

Next, the PU 32 calculates a coefficient matrix MK with the number of regions of “w1×h1” from the reduced heat map (S44). Coefficient matrix MK defines the value of coefficient K for each of “w1×h1” regions. The value of the coefficient K is greater than "1" when the value of the region in the reduced heat map is equal to or greater than the threshold value. The value of the coefficient K is "1" when the value of the area in the reduced heat map is less than the threshold value. Here, the threshold value is a value greater than "0" and less than "1". There are four coefficient matrices MK generated in the process of S44, which correspond to heat maps 60 to 66. For example, when the coefficient matrix MK corresponding to the heat map 60 has a value of a component greater than "1", it means that there is a high probability that a chest exists in that region. Further, for example, when the value of a component of the coefficient matrix MK corresponding to the heat map 60 is "1", it means that the probability that a chest exists in that region is small.

Next, the PU 72 calculates the values of the evaluation variables yok and yng by inputting the frame data FD and the coefficient matrix MK to the identification model 70b (S34a).
The identification model 70b inputs the frame data FD and the coefficient matrix MK, and outputs the values of evaluation variables yok and yng. However, the identification model 70b is not a model in which the coefficient matrix MK is input from the most upstream side.

The identification model 70b includes a feature extractor M20. The feature extractor M20 receives the frame data FD as input and extracts the feature amount of the frame data FD. Feature extractor M20 includes a CNN. This feature amount is a feature map 80 of “h1×w1”. A plurality of feature maps 80 exist. The identification model 70a has an integration process M22. The integration process M22 is a process of calculating Hadamard products with some of the feature maps 80 for each of the four coefficient matrices MK. Here, the coefficient matrix MK for which the Hadamard product is calculated with one feature map 80 may be any one of the four coefficient matrices MK. Furthermore, some of the feature maps 80 output by the feature extractor M20 are not subject to calculation of the Hadamard product with the coefficient matrix MK.

Specifically, for example, when the number of feature maps 80 is "5n" where "n" is a natural number, the feature map 80 to be multiplied with the coefficient matrix MK corresponding to each of the heat maps 60 to 66 is , "n" pieces. In that case, there are "n" feature maps 80 that are not subject to integration with any coefficient matrix MK.

The identification model 70b has an identification unit M24. The feature map subjected to the integration process M22 is input to the identification unit M24. The identification unit M24 may be, for example, a CNN whose output activation function is a softmax function.

Note that the identification model 70b is a learned model learned by supervised learning. The training data here is the same as that based on the identification model 70a. In this embodiment, the evaluation mapping data 34c includes parameters obtained by learning the discriminative model 70a. Incidentally, the parameters defining the feature extractor M20 were obtained through learning using image data representing images depending on whether or not at least one of the chest, front lower abdomen, and buttocks is exposed. It doesn't have to be. That is, a feature extractor obtained by existing image recognition processing may be used by transfer learning.

According to this embodiment described above, the following actions and effects can be obtained.
(3-1) The plurality of feature maps 80 include a map that is selectively combined with any one of the four coefficient matrices MK. As a result, the combined feature map becomes a map in which the values of the corresponding parts of the chest, front lower abdomen, buttocks, and face are amplified, so it becomes a map that extracts the features of the corresponding parts. Cheap.

(3-2) The plurality of feature maps 80 include a feature map that is not corrected by any of the four coefficient matrices MK. As a result, features that are not specific to the chest, front lower abdomen, buttocks, and face but are effective in determining whether at least one of the chest, front lower abdomen, and buttocks are exposed are extracted. Easy to generate maps.

<Fourth embodiment>
The fourth embodiment will be described below with reference to the drawings, focusing on the differences from the first embodiment.

In this embodiment, an identification model 70c is used instead of the identification model 70a.
FIG. 8 shows details of the frame evaluation process. The process shown in FIG. 8 is realized by the PU 32 repeatedly executing the image evaluation program 34a stored in the storage device 34, for example, at a predetermined period. Note that in FIG. 8, the same step numbers are given to the processes corresponding to the processes shown in FIG. 4 for convenience.

In the series of processes shown in FIG. 8, the PU 32 first calculates the values of the evaluation variables yok and yng by inputting the frame data FD to the identification model 70c (S34b). The identification model 70c is a model that outputs the values of evaluation variables yok and yng by inputting the frame data FD. Discrimination model 70c includes a feature extractor M30. The feature extractor M30 inputs image data of "w×h" for each of R, G, and B, and outputs a "w2×h2" feature map 82. Here, "w2<w, h2<h". Feature extractor M30 includes a CNN. The number of feature maps 82 that the feature extractor M30 outputs at one time is J. However, "J>2".

The identification model 70c includes a fully connected layer M32. The fully connected layer M32 combines "w2×h2×J" numerical values of the J feature maps 82 and outputs two output values. These two output values are input to the softmax function M34. The softmax function M34 outputs the values of evaluation variables yok and yng.

The identification model 70c is, for example, a learned model learned by supervised learning. The training data for the identification model 70c is similar to the training data for the identification model 70a. Note that the identification model 70c may be a learned model learned by unsupervised learning, or may be a learned model learned by semi-supervised learning.

Then, when determining that the evaluation variable yok is larger than the evaluation variable yng (S36: YES), the PU 32 executes the final evaluation process (S50).
FIG. 9 shows details of the final evaluation process.

In the series of processes shown in FIG. 9, the PU 32 generates an activation map indicating the area that the identification model 70c focuses on when the evaluation variable yok is large (S60). Here, the above J feature maps are written as f1 (x, y), f2 (x, y), ..., fJ (x, y). Among the parameters of the fully connected layer M32, those corresponding to the evaluation variable yok are assumed to be wok1, wok2, . . . , wokJ. However, "1≦x≦w2, 1≦y≦h2".

In that case, the degree to which each coordinate (x, y) contributed to the large evaluation variable yok is:
wok1・f1(x,y)+wok2・f2(x,y)+…+wokJ・fJ(x,y)
It is.

The PU 32 generates an activation map by calculating the above values for each coordinate (x, y) of "w2×h2".
Next, the PU 32 generates a binarized map MACT by binarizing the activation map (S62). Here, the PU 32 first assigns each of the "w2 x h2" values indicated by the activation map to the logistic sigmoid function, thereby calculating the "w2 x h2" values from "0" to "1". be the value of Then, the PU 32 sets it as "1" when the output value of the logistic sigmoid function is larger than a predetermined value, and sets it as "-1" when it is less than or equal to the predetermined value. Here, the predetermined value is set to a value greater than "0" and smaller than "1". Note that the predetermined value may be "1/2" or more.

The binarized map MACT is a map in which "1" is assigned to the region that greatly contributed to the OK determination in the "w2×h2" region obtained by reducing the region indicated by the frame data FD. Furthermore, the binarized map MACT is a map in which "-1" is assigned to an area that does not contribute much to the OK determination in the area of "w2×h2" which is a reduced area of the area indicated by the frame data FD.

Next, the PU 32 inputs the frame data FD into the regression model (S64). The regression model is the same as that used in the process of S30, except that there is no output that defines a two-dimensional distribution such as a two-dimensional Gaussian distribution of the face. Next, the PU 32 generates heat maps 60 to 64 (S66). Next, the PU 32 reduces each of the three heat maps 60 to 64 to "w2×h2" (S68). This process is similar to the process at S42. Next, the PU 32 generates one binarized map Mbody from the three reduced heat maps (S70). In the process of S70, the PU 32 first generates three provisional maps by comparing the values of the respective components of the three heat maps with a threshold value. That is, when the value of the component of the heat map is larger than the threshold value, the PU 32 sets the value of the corresponding component of the provisional map to "1". Further, when the value of a component of the heat map is less than or equal to the threshold value, the PU 32 sets the value of the corresponding component of the provisional map to "-1". Then, for each of the three provisional map components, if all three values are "-1", the PU 32 sets the corresponding value of the binarized map Mbody to "-1". Furthermore, when at least one of the three values for each of the three provisional map components is "1", the PU 32 sets the corresponding value of the binarized map Mbody to "1".

In the binarized map Mbody, in the "w2 x h2" area which is the reduced area indicated by the frame data FD, "1" is assigned to the area where the existence probability of any of the chest, front lower abdomen, and buttocks is large. This is a map. Furthermore, in the binarized map Mbody, in the “w2×h2” area which is a reduced area of the area indicated by the frame data FD, “-1” is placed in the area where the existence probability of all of the chest, front lower abdomen, and buttocks is small. This is the attached map.

Then, the PU 32 generates the gaze map MATT by calculating the Hadamard product of the binarized map MACT and the binarized map Mbody (S72). The gaze map MATT is a map that assigns "1" to regions that contribute to a large evaluation variable yok and that have a high probability of existence of the chest, front lower abdomen, and buttocks. Furthermore, the gaze map MATT is a map in which regions that do not contribute much to the large evaluation variable yok and in which the presence probability of the chest, front lower abdomen, and buttocks are small are set as "1". Furthermore, the gaze map MATT is a map that sets "-1" to regions that contribute to a large evaluation variable yok and that have a low probability of existence of the chest, front lower abdomen, and buttocks. Furthermore, the gaze map MATT is a map that sets "-1" to regions that do not contribute much to the large evaluation variable yok and have a high probability of existence of the chest, front lower abdomen, and buttocks.

Next, the PU 32 determines whether an index value (denoted as GAP(MATT) in the figure) indicating the average value of each component of the gaze map MATT is equal to or greater than the determination value gth (S74). Here, when the index value is a simple average value, the index value is a value greater than or equal to "-1" and less than or equal to "1". The larger the index value, the more likely the region is determined to be OK, such as the chest, front lower abdomen, and buttocks. Note that it is not essential that the index value be a simple average value. For example, for an area where only one area is ``1'' surrounded by an area where the surrounding area is ``-1'', filter processing such as rewriting the value to ``-1'' and taking the average value is performed. Good too.

If the PU 32 determines that the index value is equal to or greater than the threshold value gth (S74: YES), the PU 32 makes an OK determination (S38).
On the other hand, when determining that the index value is less than the threshold value gth (S74: NO), the PU 32 makes an NG determination (S76). Then, the PU 72 stores the frame data FD at that time in the storage device 34 (S78). Then, the PU 32 notifies the person that the validity of the large value of the evaluation variable yok is low by operating the user interface 40 shown in FIG. 1 (S80). Here, for example, the user interface 40 may be provided with a display device to display visual information indicating that the validity is low.

Note that when the processes of S38 and S80 are performed, the PU 32 once completes the process of S50 shown in FIG.
In this manner, in this embodiment, the PU 32 calculates the values of the evaluation variables yok and yng only from the frame data FD. However, when the value of the evaluation variable yok is large, the PU 32 does not immediately determine OK, but analyzes where in the image the evaluation variable yok was calculated to be a large value. Then, the PU 32 determines OK when the evaluation variable yok is calculated to be a large value by focusing on the chest, front lower abdomen, and buttocks in the image. This allows you to accidentally
It is possible to prevent judgments from being made.

According to this embodiment described above, the following effects and effects can be obtained.
(4-1) If the PU 32 determines that the validity of the large value of the evaluation variable yok is low, it saves the frame data FD at that time and notifies the person. This allows people to make the final decision. Therefore, if none of the chest, front lower abdomen, or buttocks are exposed, the NG determination can be canceled and the CM video data can be distributed. Furthermore, if none of the chest, front lower abdomen, or buttocks are exposed, the identification model 70c can be retrained using the saved frame data FD.

<Fifth embodiment>
The fifth embodiment will be described below with reference to the drawings, focusing on the differences from the first embodiment.

In this embodiment, an identification model 70d is used instead of the identification model 70a. The identification model 70d identifies the following three states.
Condition 1: None of the chest, front lower abdomen, and buttocks are exposed.

State 2: A part of the chest part "3" shown in FIG. 2, a part of the front lower abdomen part "10" shown in FIG. 2, and a part of the buttocks part "14" shown in FIG. ” is exposed. State 2 is a state in which, for example, the chest, front lower abdomen, and buttocks are covered with clothing, but the clothing is highly revealing to some extent. State 2 is a state in which, for example, the chest, front lower abdomen, and buttocks are covered with clothing, but the clothing is transparent to some extent.

State 3: A state in which at least one of the chest, front lower abdomen, and buttocks is significantly exposed compared to state 2 or higher.
FIG. 10 shows details of the frame evaluation process. The process shown in FIG. 10 is realized by the PU 32 repeatedly executing the image evaluation program 34a stored in the storage device 34 at, for example, a predetermined period. Note that in FIG. 10, the same step numbers are given to the processes corresponding to the processes shown in FIG. 4 for convenience.

In the series of processes shown in FIG. 10, the PU 32 first inputs the frame data FD into the semantic segmentation model (S30a). The semantic segmentation model is a model that outputs the value of a label variable for each "w×h" number of pixels indicated by the frame data FD. Here, the label variable is a value that distinguishes at least each part of "3 to 5, 10 to 14, and 18" in FIG. 2 from each other. Therefore, a label variable can take on ten or more different values.

A semantic segmentation model is a discriminative model. The semantic segmentation model is a trained model learned by supervised learning. The semantic segmentation model is data that has been trained on various image data using data in which label variable values are previously assigned to pixels of each portion as training data. The location information mapping data 34b according to this embodiment includes parameters that define a semantic segmentation model. That is, for example, when the semantic segmentation model includes CNN, the position information mapping data 34b includes each value of the filter, the weight parameter of the fully connected layer, and the like.

Next, the PU 32 generates a physical information map using the value of the label variable of each pixel output by the semantic segmentation model (S32a). The physical information map consists of the following six items.

(bm1) This is a map that identifies the part of the chest where part "3" shown in FIG. 2 is located and the other part. This can be realized, for example, by setting the part of the chest where part "3" shown in FIG. 2 is located as "1" and setting the other parts as "0".

(bm2) This is a map that identifies the portion of the front lower abdomen where the portion “10” shown in FIG. 2 is located and the other portions.
(bm3) This is a map that identifies the portion of the buttocks where the portion “14” shown in FIG. 2 is located and the other portions.

(bm4) This is a map that identifies parts of the chest other than part "3" shown in FIG. 2 and other parts.
(bm5) This is a map that identifies a portion of the front lower abdomen where a portion other than the portion “10” shown in FIG. 2 is located and the other portions.

(bm6) This is a map that identifies a portion of the buttocks where a portion other than the portion “14” shown in FIG. 2 is located and other portions.
The above (bm1) to (bm3) are map data indicating the positions of parts of particular interest in the above state 2. On the other hand, the above (bm4) to (bm6) are map data indicating the positions of parts of particular interest in the above state 3.

The positional information mapping that outputs the six physical information maps from the frame data through the processing in S30a and S32a is a mapping defined according to the above-mentioned semantic segmentation model.

Next, the PU 32 calculates the values of the evaluation variables yok1, yok2, and yng by inputting the six physical information maps 84 and the frame data FD into the identification model 70d (S34c). The identification model 70d includes CNN. The six physical information maps 84 input to the identification model 70d and the frame data FD are subjected to convolution processing by a plurality of filters fl having dimensions of "a2 x b2 x 9". Here, "a2" is a natural number smaller than "w". Moreover, "b2" is a natural number smaller than "h". The number of convolution layers is arbitrary. Additionally, the CNN may include a pooling layer. Additionally, the CNN may include residual blocks. Then, the feature maps output by the CNN are combined in a fully connected layer M40. The fully connected layer M40 has three outputs. Then, they are input to a softmax function M42 which is an output activation function. The softmax function M42 outputs the values of three evaluation variables yok1, yok2, and yng.

The identification model 70d is, for example, a learned model learned by supervised learning. The training data for learning the identification model 70d is image data of each of the states 1, 2, and 3 described above. For the image data in state 1, the value of the target variable of the evaluation variable yok1 is set to "1", and the values of the other target variables are set to "0". Furthermore, for the image data in state 2, the value of the target variable of the evaluation variable yok2 is set to "1", and the values of the other target variables are set to "0". Further, for the image data in state 3, the value of the target variable of the evaluation variable yng is set to "1", and the values of the other target variables are set to "0".

Each parameter obtained by learning the discriminative model 70d is included in the evaluation mapping data 34c.
The PU 32 determines whether the maximum value ymax among the values of the evaluation variables yok1, yok2, and yng is the value of the evaluation variable yok1 (S36a). When determining that the maximum value ymax is the value of the evaluation variable yok1 (S36a: YES), the PU32 determines that the state is 1 (S38a). Hereinafter, this will be referred to as OK1 determination. On the other hand, when determining that the maximum value ymax is not the value of the evaluation variable yok1 (S36a: NO), the PU32 determines whether the maximum value ymax is the value of the evaluation variable yok2 (S36b). When determining that the maximum value ymax is the value of the evaluation variable yok2 (S36b: YES), the PU32 determines that the state is 2 (S38b). Hereinafter, this will be referred to as OK2 determination. On the other hand, when determining that the maximum value ymax is not the value of the evaluation variable yok2 (S36b: NO), the PU32 makes an NG determination (S40).

FIG. 11 shows a process procedure for supporting display of an image that satisfies the user's desired criteria on the user terminal 50. The process shown in FIG. 11 consists of two processes. One is a process realized by the PU 52 repeatedly executing the application program 54a stored in the storage device 54 of the user terminal 50 at a predetermined period. The other process is realized by the PU 32 repeatedly executing the image evaluation program 34a stored in the storage device 34 of the image evaluation device 30 at a predetermined period.

As shown in FIG. 11, the PU 52 of the user terminal 50 receives a request from the user regarding criteria for commercial videos that are allowed to be played back (S90). Here, the PU 52 displays on the display device included in the user interface 58 that it is possible to select whether to allow only state 1 or to allow state 1 and state 2. The process of S90 is a process of accepting a request input to the user interface 58 by the user. When completing the process of S90, the PU 52 transmits the user ID, which is the user's identification symbol, and the requested criteria by operating the communication device 56 (S92).

In response, the PU 32 of the image evaluation device 30 receives the user ID and the requested criteria (S100). Then, the PU 32 stores the user ID and the requested criteria in the storage device 34 (S102). Then, the PU 32 evaluates the CM video data (S104). The process in S104 is similar to S10 to S18 and S22 to S26. However, here, the process in FIG. 10 is executed instead of the process in S16. Further, the processing in S22 to S26 is replaced with a process in which an OK2 determination is made if there is even one OK2 determination in the CM video data that does not have an NG determination.

Then, by operating the communication device 36, the PU 32 transmits a CM so that an image that satisfies the requested criteria can be displayed on each of the user terminals 50(1), 50(2), ... according to the evaluation result. The video data is transmitted (S106). That is, the PU 32 unconditionally transmits the CM video data determined to be OK1 to all user terminals 50 . On the other hand, the PU 32 unconditionally transmits the CM video data determined to be OK2 to the user terminal 50 of the user who allows state 2. On the other hand, the user terminal 50 of a user who does not allow state 2 may be prohibited from transmitting CM video data determined to be OK2. Alternatively, the PU 32 may perform a process similar to the process in S20a. On the other hand, the PU 32 may prohibit the transmission of CM video data judged as NG to the user terminals of all users. Alternatively, the PU 32 may execute the process of S20a. Note that when the PU 32 completes the process of S106, it temporarily ends the process executed by the PU 32 among the series of processes shown in FIG.

On the other hand, the PU 52 of the user terminal 50 receives the CM video data transmitted from the image evaluation device 30 (S94). Here, the PU 52 unconditionally plays back commercial video data that meets the criteria. Furthermore, when receiving commercial video data that does not meet the standards, it issues a warning, masks it, or prohibits playback, depending on the restriction command given to the commercial video data.

Note that when the PU 52 completes the process of S94, it temporarily ends the process executed by the PU 52 among the series of processes shown in FIG.
<Correspondence>
The correspondence relationship between the matters in the above embodiment and the matters described in the column of "Means for solving the problem" above is as follows. Below, the correspondence relationship is shown for each solution number listed in the "Means for solving the problem" column.

[1, 2, 11] The execution device corresponds to the PU32. The storage device corresponds to the storage device 34. The position information mapping data corresponds to the position information mapping data 34b. The evaluation mapping data corresponds to evaluation mapping data 34c.

In FIG. 4, the position information mapping corresponds to the mapping realized by the processing in S30 and S32. In other words, inputting the frame data FD corresponds to mapping that outputs a heat map. In FIG. 7, the position information mapping corresponds to the mapping realized by the processes of S30, S32, S42, and S44. In other words, it corresponds to a mapping that outputs the coefficient matrix MK by inputting the frame data FD. In FIG. 9, the position information mapping corresponds to the mapping realized by the processing of S64 to S70. In other words, it corresponds to mapping that outputs the binarized map Mbody by inputting the frame data FD. In FIG. 10, the position information mapping corresponds to the mapping realized by the processing in S30a and S32a. In other words, inputting the frame data FD corresponds to mapping that outputs the physical information map 84.

The position information calculation process corresponds to the processes in S30 and S32 in FIG. 4, the processes in S30, S32, S42, and S44 in FIG. 7, the processes in S64 to S70 in FIG. 9, and the processes in S30a and S32a in FIG.

In FIG. 4, the evaluation mapping corresponds to the mapping realized by the processing of S34 to S40. In other words, it corresponds to a mapping that receives frame data FD and heat maps 60 to 66 as input and outputs an OK determination or NG determination. In FIG. 7, this corresponds to the mapping realized by the processing of S34a and S36 to S40. In other words, it corresponds to a mapping that receives frame data FD and coefficient matrix MK as input and outputs an OK determination or NG determination. In FIG. 8, this corresponds to the mapping realized by the processing of S34b, S36, S40, S60, S62, S72 to S76, and S38. In other words, it corresponds to mapping that inputs the frame data FD and the binarized map Mbody and outputs an OK determination or NG determination. In FIG. 10, this corresponds to the mapping realized by the processing of S34c, S36a, S36b, S38a, S38b, and S40. In other words, it corresponds to mapping that inputs the frame data FD and the physical information map 84 and outputs an OK1 determination, OK2 determination, or NG determination.

The evaluation process corresponds to the processes of S34 to S40 in the processes of FIGS. 3 and 4. In FIG. 7, this corresponds to the processes of S34a and S36 to S40. In FIG. 8, this corresponds to the processes of S34b, S36, S40, S72 to S76, and S38. In FIG. 10, this corresponds to the processes of S34c, S36a, S36b, S38a, S38b, and S40.

The image data corresponds to frame data FD. The position input data corresponds to frame data FD. The evaluation input data corresponds to frame data FD.
[3] The location information data corresponds to the heat maps 60 to 66 and the physical information map 84. The evaluation mapping corresponds to the mapping realized by the processing of S34 to S40 and the mapping realized by the processing of S34c, S36a, S36b, S38a, S38b, and S40. [4] The tentative evaluation mapping corresponds to the mapping realized by the processing in S34b and S36. In other words, it corresponds to a mapping that takes one frame data FD as input and outputs the result of comparing the values of evaluation variables yok and yng. The provisional evaluation process corresponds to the processes of S34b and S36. The validity evaluation process corresponds to the processes of S60, S62, and S72 to S74. [5] The notification process corresponds to the process in S80. [6] The feature layer corresponds to the output layer of the feature extractor M20.

[7] User terminals correspond to user terminals 50(1), 50(2), . . . The provision process includes the process of S26 in FIG. 3, the process of S26 when a negative determination is made in the process of S24 in FIG. 6, and the process of S106 when it is determined that the request criteria are met in S104 of FIG. handle. The restriction process corresponds to the process of S20 in FIG. 3, the process of S20a in FIG. 6, and the process of S106 when it is determined in S104 of FIG. 11 that the required criteria are not satisfied. [8, 10] The instruction process corresponds to the processes of S90 and S92. [9] The prohibition process corresponds to the process in S20 and the process in S106 when it is determined in S104 of FIG. 11 that the required criteria are not satisfied. The restriction process corresponds to the process in S20a and the process in S106 when it is determined in S104 of FIG. 11 that the required criteria are not satisfied. [12] The image evaluation program corresponds to the image evaluation program 34a.

[13-15] The providing step corresponds to the step of executing the process of S26 in FIG. 3, and the step of executing the process of S26 when a negative determination is made in the process of S24 in FIG. Further, the providing step corresponds to the step of executing the process of S106 when it is determined in S104 of FIG. 11 that the request criteria are met. The restriction steps are a step of executing the process of S20 in FIG. 3, a step of executing the process of S20a of FIG. 6, and a step of executing the process of S106 when it is determined in S104 of FIG. 11 that the required standard is not satisfied. corresponds to The acquisition step corresponds to the process in S10.

<Other embodiments>
Note that this embodiment can be implemented with the following modifications. This embodiment and the following modified examples can be implemented in combination with each other within a technically consistent range.

"About designated parts"
The one or more body parts (predetermined parts) that are the basis for evaluating that the image is problematic when reproduced are not limited to the chest, front lower abdomen, and buttocks, but include, for example, the body parts in Figure 2. It may be appropriately defined in a manner including at least one. Further, in one embodiment, the above-mentioned basis may be appropriately defined, for example, in some aspects of one body part in FIG. 2 . Specifically, part of the body parts such as lips, eyes, and hair that constitute the face may serve as the basis.

"About location information mapping"
- The positional information mapping used in the process of FIG. 4 is not limited to mapping that inputs the frame data FD and outputs heat maps of the face, chest, lower abdomen, and buttocks. For example, it may be a mapping that receives frame data FD as input and outputs heat maps of the chest, lower abdomen, and buttocks.

- In the process of FIG. 4, it is not essential to use a mapping that outputs a heat map. For example, a semantic segmentation model may be used to output a map that specifies pixels corresponding to each of the chest, lower abdomen, and buttocks.

The position information mapping used in the processing of FIG. 7 is not limited to a mapping that takes frame data FD as input and outputs a coefficient matrix MK corresponding to the heat maps of the chest, lower abdomen, buttocks, and face. For example, it may be a mapping that takes frame data FD as input and outputs a coefficient matrix MK corresponding to the heat maps of the chest, lower abdomen, and buttocks.

- In the process of FIG. 7, it is not essential to use a mapping that outputs a heat map. For example, a mapping may be used that outputs coefficient matrices corresponding to pixels corresponding to each of the chest, lower abdomen, buttocks, and face identified by a semantic segmentation model. In that case, in the pixel area reduced according to the output layer of the feature extractor M20, the coefficient K of the area indicating each of the chest, lower abdomen, buttocks, and face may be set larger than "1".

- In the process of FIG. 9, it is not essential to use a mapping that outputs the binarized map Mbody according to the heat map. For example, a mapping may be used that outputs a binarized map Mbody corresponding to each region of the chest, lower abdomen, and buttocks identified by the semantic segmentation model.

・In the process of FIG. 10, the part "3", the part "4, 5", the part "10", the part "11, 18", the part "14", and the part "12, 13, 18" shown in FIG. Although a mapping that outputs a physical information map for specifying and identifying is used, the present invention is not limited to this. For example, physical information that specifies part "3", part "3, 4, 5", part "10", part "10, 11, 18", part "14", part "12, 13, 14, 18" A mapping that outputs a map may also be used.

- The mapping that outputs the heat map is not limited to the mapping that outputs the existence probability of each of the chest, lower abdomen, and buttocks. For example, a mapping that outputs the probabilities of each of the parts "3", "4, 5", "10", "11, 18", "14", and "12, 13, 18". It's okay.

- The mapping for outputting heat maps is not limited to mapping for outputting separate maps for each of the chest, lower abdomen, and buttocks. For example, it may be a mapping that outputs the maximum existence probability of each of the chest, lower abdomen, and buttocks for each pixel.

- It is not essential that the mapping that outputs the heat map be the mapping that outputs the existence probability of each of the chest, lower abdomen, and buttocks. For example, if it is guaranteed that the image is a frontal image, the mapping may be such that only the existence probabilities of the chest and frontal lower abdomen are output. That is, depending on the location of the predetermined region whose exposure is a problem, a heat map regarding the existence probability of each of the predetermined regions whose exposure is a problem may be output.

- The probability distribution shown by the heat map is not limited to an isotropic distribution. The anisotropic distribution can be realized, for example, by using a regression model that outputs two or three dispersion parameters of a Gaussian distribution, instead of the regression model used in the process of S30 in FIG.

- The semantic segmentation model is not limited to the model that identifies three or more parts as described above. For example, if it is guaranteed that the image is a frontal image, a model may be used that outputs label variables that identify only the chest and frontal lower abdomen, respectively, and the others.

- Mappings constructed using the semantic segmentation model are not limited to mappings that output different maps for each specified region. For example, it may be a mapping that outputs one map having different label variable values for the chest, front lower abdomen, buttocks, and other regions.

- It is not essential that the position information mapping is a mapping that outputs a map in which values are determined for unit areas in which the image area is subdivided. For example, it may be a mapping that outputs the respective coordinates of a representative point of the chest, a representative point of the lower abdomen, and a representative point of the buttocks.

- The image data that is input to the position information mapping is not limited to the above frame data FD. For example, monochrome image data may be used instead of R, G, and B image data.

"About input data for position"
- Positional input data to be input to positional information mapping is not limited to the image data itself to be evaluated. For example, the position input data may be image data in which a portion related to a person is extracted from each pixel indicated by the image data to be evaluated and embedded in a predetermined background image. Here, the extraction process of the parts related to people is performed by the PU 32 using a semantic segmentation model. Note that the preprocessing for using the image data to be evaluated as positional input data is not limited to this. For example, it may be a process of adjusting brightness.

"About evaluation mapping"
(a) A mapping in which each pixel of the input data for evaluation and the position information data corresponding to each pixel are input in association with each other. In S34 of FIG. 4, the frame data FD and the four heat maps 60 to 66 are An example of the input mapping is shown. Further, FIG. 10 illustrates a mapping in which frame data FD and six physical information maps 84 are input. However, map data to which each pixel is attached with information as to whether it is a predetermined part of the body or not is not limited to these, and map data to which the same information is attached may be Any map data etc. described in the column may be used.

- Although the frame data FD is input in S34 of FIG. 4, the present invention is not limited to this. For example, monochrome image data may be used instead of R, G, and B image data.
- Although the identification model 70a using CNN is illustrated in FIG. 4, the invention is not limited to this. For example, a model using an attention mechanism may be used. Specifically, it may be a model that uses a multi-head attention mechanism, such as a Transformer-based (equipped with a Transformer encoder) model. In that case, vectors obtained by suitably linearly transforming the frames data FD and patches obtained by dividing the heat maps 60 to 66 are input to the transformer encoder. Here, each patch is data regarding a part of the pixel area of the frame data FD and the corresponding area of the heat maps 60 to 66. This also allows each pixel of the evaluation input data to be input in association with the position information data corresponding to each pixel.

- It is not essential that the output activation function of the discrimination model 70a is a softmax function. For example, it may be a logistic sigmoid function.
(b) Regarding provisional evaluation mapping - In S34b of FIG. 8, mapping using frame data FD as input is illustrated, but the mapping is not limited to this. For example, monochrome image data may be used instead of R, G, and B image data.

- In FIG. 8, the identification model 70c using CNN is illustrated, but the invention is not limited to this. For example, a model using an attention mechanism may be used. Specifically, it may be a model that uses a multi-head attention mechanism, such as a Transformer-based (equipped with a Transformer encoder) model. In other words, the data that identifies the portion that contributed to the evaluation result is not limited to the CNN feature map. When a Transformer-based model is used as the identification model 70c, the above-described validity evaluation may be performed using the attention map MATT based on the attention map and attention mask.

- It is not essential that the output activation function of the discrimination model 70c is a softmax function. For example, it may be a logistic sigmoid function.
- It is not essential that the identification model that defines the tentative evaluation mapping not use the output of the location information mapping. For example, the identification model that defines the provisional evaluation mapping may be the identification model 70a, or the above-mentioned modification of the identification model 70a may be used. Further, for example, the identification model that defines the provisional evaluation mapping may be the identification model 70b, or the above-mentioned modification example regarding the identification model 70b may be used. Further, for example, the identification model that defines the provisional evaluation mapping may be the identification model 70d, or the above-mentioned modification example regarding the identification model 70d may be used.

(c) Regarding the validity evaluation process - In the process of FIG. 8, the condition for executing the process of S50 may include a condition that a predetermined part of the person is included. Thereby, appropriate evaluation can be performed when the image indicated by the frame data FD does not include a predetermined part of a person.

- In the process of FIG. 8, the process of S50 is executed only when an affirmative determination is made in the process of S36, but the process is not limited to this. For example, even when a negative determination is made in the process of S36, a process similar to the process of S50 may be executed. Here, processing similar to the processing of S50 can be realized, for example, as follows.

That is, instead of the process in S70, the PU 32 generates a binary map Mbody (1) in which the probability of being the chest is greater than or equal to the threshold, a binary map Mbody (2) in which the probability of being the front lower abdomen is greater than or equal to the threshold, A binarized map Mbody (3) whose probability of being the butt is equal to or higher than a threshold is generated. Here, the binarized map Mbody (1) is a label variable of a pixel for which the logical sum of the probability that it is the front lower abdomen is greater than or equal to the threshold and the probability that it is the buttocks is greater than or equal to the threshold is true. This is a map with "0". In addition, the binarized map Mbody (2) sets the label variable of a pixel for which the logical sum of the probability of being a chest is greater than or equal to a threshold and the probability of being a buttocks being greater than or equal to a threshold to be "0". This is the map. In addition, the binarized map Mbody (3) sets the label variable of a pixel for which the logical sum of the probability of being a chest is greater than or equal to a threshold and the probability of being a frontal lower abdomen being greater than or equal to a threshold to be true. ” This is a map with the following. Then, in place of the process in S72, the PU 32 calculates the Hadamard product of each of the binarized maps Mbody(1) to (3) and the binarized map MACT, and calculates the Hadamard product of each of the binarized maps Mbody(1) to (3) to ) is calculated. Then, the PU 32 evaluates that the process of S36 is appropriate if any of the average values of each component of the gaze maps MATT(1) to (3) is equal to or greater than the threshold value gth.

This process is performed in consideration of the fact that an NG determination is made by exposing at least one of the chest, lower abdomen, and buttocks. That is, the PU 32 evaluates that the NG determination is appropriate if the region gazed at when making the NG determination is at least one of the chest, lower abdomen, and buttocks.

- It is not essential that the validity evaluation process includes the process of generating the binarized map Mbody. For example, you may do the following: That is, the PU 32 identifies the chest, lower abdomen, and buttocks indicated by the heat map reduced in the process of S68 out of the activation maps generated in the process of S60. Then, the PU 32 compares the sum of the values of each of these areas with the sum of the values of other areas having a predetermined area. Note that in that case, it is desirable that the value of each pixel of the activation map be a value that has been converted in advance using ReLU, a logistic sigmoid function, or the like.

(d) A process of synthesizing at least some values of a plurality of regions indicated by the feature layer with position information data - In the integration process M22 of S34a in FIG. 7, the coefficients from which the Hadamard product is taken with one feature map Although the number of matrix MK is one, the number is not limited to this. There may be two or more.

- In the integration process M22 of S34a in FIG. 7, a feature map is provided for which the Hadamard product with the coefficient matrix MK is not calculated, but this is not essential.
- As suggested in the column "About position information mapping", the number of coefficient matrices MK is not limited to four.

- The process of combining at least part of the values of the plurality of regions indicated by the feature amount layer and the position information data is not limited to the integration process M22 of the coefficient matrix MK and the feature map. For example, it may be a convolution process of the coefficient matrix MK and the feature map.

(e) Regarding the value of the evaluation variable according to the degree of exposure - Although FIG. 10 illustrates the values of the evaluation variable that can take three different values depending on the degree of exposure, the present invention is not limited thereto. For example, the evaluation variable may be a variable that can take four or more values.

(f) Others - Not limited to the identification models 70a and 70c, for example, a model using an attention mechanism may be used in place of the identification models 70b and 70d. Specifically, a model using a multi-head attention mechanism such as a Transformer-based model (equipped with a Transformer encoder) may be used.

- The positional information data that is input to the evaluation mapping is not limited to data that includes information regarding a predetermined part in each two-dimensional image area. For example, it may be the coordinates of a representative point of a predetermined part, as described in the column "About position information mapping."

- For example, the frame data FD and the heat maps 60 to 66 may be input to a fully connected forward propagation type NN. In that case, the first layer performs weighted averaging processing using the weighting coefficient of NN for the respective values of the map data and frame data FD, to which each pixel is given information as to whether it belongs to a predetermined part of the body or not. It is also possible to perform processing using

"About the output of the model that defines the evaluation map"
- For example, when there are a plurality of predetermined parts where exposure is a problem, a model may be used that outputs the presence or absence of exposure for each part. That is, for example, when three predetermined parts are the chest, the front lower abdomen, and the buttocks, a variable value indicating the determination result of the presence or absence of exposure may be output for each of them. This can be realized, for example, by preparing three output activation functions, which correspond to the number of predetermined parts, of logistic sigmoid functions. Note that in such a case, when evaluating the validity of the NG determination in the validity evaluation process, it is desirable to make the following changes instead of the ones described above. That is, for example, if only the chest is determined to be NG, the process may be replaced with a process in which the chest is determined to be appropriate. Further, for example, if an NG determination is made for both the chest and the front lower abdomen, the process may be replaced with a process that determines that it is appropriate when both the chest and the front lower abdomen are focused.

"About input data for evaluation"
- The evaluation input data that is input to the evaluation mapping is not limited to the image data itself to be evaluated. For example, the input data for evaluation may be image data in which a portion related to a person is extracted from each pixel indicated by the image data to be evaluated and embedded in a predetermined background image. Here, the extraction process of the parts related to people is performed by the PU 32 using a semantic segmentation model. Note that the preprocessing for using the image data to be evaluated as input data for evaluation is not limited to this. For example, it may be a process of adjusting brightness.

"About location information mapping data"
- It is not essential that the location information mapping data consists only of learned parameters in the parametric model. For example, it may consist of data defining a feature extractor and support vectors. Here, the feature extractor is a trained model, such as a CNN, that receives frame data FD, etc. as input and outputs a feature vector. On the other hand, the support vector is a vector selected by support vector regression learning. That is, support vectors are extracted in the learning process from feature vectors output by inputting training data to a feature extractor. Then, the parameters defining the feature extractor and the support vector are stored in the storage device 34 as position information mapping data.

“About evaluation mapping data”
- When using a model that uses an attention mechanism instead of the identification models 70a to 70d, the evaluation mapping data 34c may include the following data. That is, the evaluation mapping data 34c may include values of components of a transformation matrix that transforms each patch obtained by dividing the frame data FD into a one-dimensional vector. Furthermore, for example, the evaluation mapping data 34c may include values of a matrix that converts each patch converted into the one-dimensional vector described above into each vector of a key, a query, and a value. Furthermore, for example, the evaluation mapping data 34c may include positional information data to be embedded in a one-dimensionally converted patch. Note that the transformation matrix and position information data may be already learned data. However, this does not necessarily have to be the value learned in the learning process described above. For example, the above learning process may be fine tuning, and the values may be learned only in the previous learning.

- It is not essential that the evaluation mapping data consists only of learned parameters in the parametric model. For example, it may consist of data defining a feature extractor and support vectors. Here, the feature extractor is a trained model such as CNN that receives frame data FD and the like as input and outputs a feature vector. On the other hand, the support vector is a vector selected by learning of the support vector machine. That is, support vectors are extracted in the learning process from feature vectors output by inputting training data to a feature extractor. Then, the parameters defining the feature extractor and the support vector are stored in the storage device 34 as evaluation mapping data.

Further, the parameters of the fully connected layer and the function that outputs the posterior distribution of the parameters constituting the convolution filter may be stored in the storage device 34 as evaluation mapping data. Here, for example, the output activation function in the discrimination model 70c is replaced with a logistic sigmoid function. For example, a prior distribution such as a normal distribution is assumed for each of the above parameters. Then, in the learning process, a posterior distribution is generated by Bayesian estimation. Note that the PU 32 may calculate the average value of the above-mentioned parameters based on a sampling method or the like based on the posterior distribution, and may replace this with the parameters used in FIG. 8 . In this case, the posterior distribution may be updated by performing re-learning using data that has been evaluated as not being valid by the validity evaluation process.

"About evaluation processing"
- If the evaluation process determines OK for two or more of the three frame data FD sampled at N cycles, the image data for 2N+1 frames is determined to be OK. However, it is not limited to this. For example, in the above-described modified example of the process shown in FIG. 8, the standard for determining OK may be changed depending on the determination result of the validity evaluation process. Specifically, if an NG determination is made in the process of S36, but it is determined that the NG determination is not valid in the validity evaluation process, the number of times the NG determination is made is counted as "0.5". Good too. In that case, for example, by making a final OK determination when the number of NG determinations is less than "1", the standard for OK determination can be changed according to the determination result of the validity evaluation process.

- It is not essential to perform an OK determination when a predetermined number or more of a plurality of frames are determined to be OK. For example, the final OK determination may be made only when all of the frame data FD sampled at a predetermined period is determined to be OK by the frame evaluation process. Note that the predetermined cycle may be equal to the frame cycle.

“About restriction processing and restriction processes”
- The process of S20a may be, for example, a command to mask a part of the reproduced image that includes at least an area in which a predetermined part of the reproduced image is exposed by superimposing a predetermined image. Alternatively, for example, the command may be a command to mask by applying an effect such as blurring to a part of the reproduced image that includes at least an area in which a predetermined part of the reproduced image is exposed. That is, the process of S20a is not limited in its form as long as it is a command that hides a predetermined part that is determined to be exposed and is determined to be exposed. Note that the part of the reproduced image that includes at least the area in which the predetermined part whose exposure is a problem is actually exposed may be determined based on at least the above-mentioned two-dimensional distribution, for example. Note that when a plurality of predetermined parts exist in an image, in order to mask only the actually exposed regions, it is desirable to specify the actually exposed regions. This may be, for example, a process of identifying an exposed area based on the gaze area of the identification model.

- For example, the PU 32 prohibits distribution (transmission) of CM video data when the number of frame data FDs for which an OK determination has not been made among a predetermined number of consecutive frame data FDs is less than a predetermined number. It's okay. In that case, the PU 32 may distribute (transmit) the CM video data on the condition that OK determination is made for all of a predetermined number of consecutive frame data FD.

For example, the PU 32 prohibits the distribution (transmission) of commercial video data when the number of frame data FDs for which an OK determination is not made is a predetermined number among a predetermined number of consecutive frame data FDs. Good too. In that case, the PU 32 may distribute (transmit) the CM video data on the condition that an OK determination is made for at least part of a predetermined number of consecutive frame data FD.

For example, among a predetermined number of consecutive frame data FDs, if there are consecutive frame data FDs that have been determined to be OK with less than the predetermined number of frame data FDs that have not been determined to be OK, the PU 32 may Video data may also be distributed (transmitted).

Note that, among a predetermined number of consecutive frame data FDs, if there are consecutive frame data FDs for which an OK decision has not been made sandwiching frame data FDs that are less than the predetermined number for which an OK decision has been made, Distribution (transmission) of data may be prohibited.

- For the process of restricting the display of an image on the user terminal 50 by the process of S20a, it is not essential that the frame evaluation process shown in FIG. 4 be performed. For example, the process of S20a may be applied when evaluation is performed by the process shown in FIG. 7 or the frame evaluation process shown in the modified example thereof. Further, for example, the process of S20a may be applied when evaluation is performed by the process shown in FIG. 8 or the frame evaluation process shown in the modified example thereof.

- In the process of FIG. 11, the processes of S92, S100, and S102 may be omitted. In that case, in the case of an OK2 determination, the PU 32 may add an OK2 determination to the CM video data as a restriction command in the process of S106. In that case, the PU 52 of the user terminal 50 may decide whether or not to reproduce the CM video data based on the criteria received in the process of S90 and the restriction command.

- For example, CM video data from which frames judged as NG may be deleted may be transmitted to the user terminal 50. This can also prevent inappropriate images from being displayed on the user terminal 50.

- It is not essential that the user terminal 50 stores the dedicated application program 54a. For example, the commercial video data distributed from the image evaluation device 30 may be played back using a general-purpose browser.

"About the execution device"
- The execution device is not limited to the PU32. For example, the execution device may be a dedicated hardware circuit such as an ASIC or an FPGA. That is, the execution device may include a processing circuit having any of the following configurations (a) to (c). (a) A processing circuit that includes a processing device that executes all of the above processing according to a program, and a program storage device such as a storage device that stores the program. (b) A processing circuit that includes a processing device and a program storage device that execute part of the above processing according to a program, and a dedicated hardware circuit (hardware accelerator) that executes the remaining processing. (c) A processing circuit that includes a dedicated hardware circuit that executes all of the above processing. Here, there may be a plurality of software execution devices including a processing device and a program storage device. Further, there may be a plurality of dedicated hardware circuits.

"About the image evaluation device"
- It is not essential for the image evaluation device to perform both the process of evaluating an image and the process of distributing the evaluation results to the user terminal 50. For example, a device other than the image evaluation device may execute the process of distributing the evaluation results to the user terminal 50. Furthermore, it is not essential that the image evaluation device receives image data transmitted from the vendor terminal 10. For example, a separate device may be provided that receives image data transmitted from the vendor terminal 10 and transmits it to the image evaluation device. Furthermore, the image evaluation device and the vendor terminal 10 may be integrated.

"About computers"
- The computer that executes the image evaluation program 34a is not limited to the PU 32 included in the image evaluation device 30. For example, by installing the image evaluation program 34a on the user terminal 50, the PU 52 of the user terminal 50 may be used as a computer that executes the image evaluation program 34a.

"others"
- The PU 12 may include an arithmetic unit that can be included in the PU 32.
- In the above embodiment, it is assumed that the CM video data is distributed (transmitted) to the user terminal 50 in the form of streaming video, but the present invention is not limited to this. For example, it may be distributed (transmitted) to the user terminal 50 in real time using a live streaming service. The image data to be evaluated is not limited to commercial video data. For example, there are no restrictions on the type of video or the purpose of abnormality determination, as long as the video is not limited to videos used for advertising products or the like, but can include at least part of a person's body. Further, the image data to be evaluated may be, for example, any still image data.

- The PU 32 distributes (sends) the commercial video data to the vendor terminal 10 and/or the user terminal 50, with information indicating an explicitly predetermined part that is treated as the basis for an OK or NG determination. may be notified in advance. Further, the PU 32 may notify the vendor terminal 10 and/or the user terminal 50 of information indicating the predetermined region during distribution (transmission) of the CM video data. Further, the PU 32 may notify the vendor terminal 10 and/or the user terminal 50 of information indicating a predetermined portion that is the basis for the NG determination regarding the frame data FD of the commercial video data that is being distributed (transmitted). . Note that the "notification" here may be a mode of sending a message to the vendor terminal 10 and/or the user terminal 50, and a display superimposed on the commercial video data being displayed on the vendor terminal 10 and/or the user terminal 50. It may be a mode of processing.

10...Business terminal 14...Storage device 16...Communication device 20...Network 30...Image evaluation device 34...Storage device 34a...Image evaluation program 34b...Position information mapping data 34c...Evaluation mapping data 36...Communication device 40...User interface 50... User terminal 54...Storage device 54a...Application program 56...Communication device 58...User interface 60-66...Heat map 70a-70d...Identification model

Claims (11)

includes an execution device and a storage device,
The storage device stores position information mapping data and evaluation mapping data,
The location information mapping data is data for defining a location information mapping,
The location information mapping is a mapping that outputs location information data,
The position information data is data indicating position information of a predetermined part of a person in an image shown by image data to be evaluated,
The evaluation mapping data is data for defining an evaluation mapping,
The evaluation mapping is a mapping that receives evaluation input data and the position information data as input and outputs an evaluation result of the image data,
The evaluation input data is data that corresponds to the image data and is data that is input to the evaluation mapping,
The execution device is configured to execute a position information generation process and an evaluation process,
The position information generation process is a process of generating the position information data by inputting position input data into the position information mapping,
The position input data is data that corresponds to the image data and is data that is input to the position information mapping,
The evaluation process is a process of evaluating the image data by inputting the position information data and the evaluation input data into the evaluation mapping,
The evaluation mapping includes a feature layer,
The feature amount layer is a layer that quantifies the feature amount of the image data by dividing the region indicated by the image data into a plurality of regions and giving a numerical value to each of the regions,
The evaluation mapping is a mapping that outputs the evaluation result by including a process of combining at least some values of the plurality of regions indicated by the feature amount layer with the position information data. includes an execution device and a storage device,
The storage device stores position information mapping data and evaluation mapping data,
The location information mapping data is data for defining a location information mapping,
The location information mapping is a mapping that outputs location information data,
The position information data is data indicating position information of a predetermined part of a person in an image shown by image data to be evaluated,
The evaluation mapping data is data for defining an evaluation mapping,
The evaluation mapping is a mapping that receives evaluation input data and the position information data as input and outputs an evaluation result of the image data,
The evaluation input data is data that corresponds to the image data and is data that is input to the evaluation mapping,
The execution device is configured to execute a position information generation process and an evaluation process,
The position information generation process is a process of generating the position information data by inputting position input data into the position information mapping,
The position input data is data that corresponds to the image data and is data that is input to the position information mapping,
The evaluation process is a process of evaluating the image data by inputting the position information data and the evaluation input data into the evaluation mapping,
The evaluation mapping includes a provisional evaluation mapping,
The provisional evaluation mapping is a mapping that receives the evaluation input data as input and outputs a provisional evaluation result of the image data,
The evaluation process includes a provisional evaluation process and a validity evaluation process,
The provisional evaluation process includes processing to output the provisional evaluation result by inputting the evaluation input data to the provisional evaluation mapping,
The validity evaluation process uses the position information data as input and determines the validity of the provisional evaluation result according to the degree to which an area indicating the predetermined part out of the area shown by the image data has contributed to the provisional evaluation result. An image evaluation device that includes processing for evaluating gender . includes an execution device and a storage device,
The storage device stores position information mapping data and evaluation mapping data,
The location information mapping data is data for defining a location information mapping,
The location information mapping is a mapping that outputs location information data,
The position information data is data indicating position information of a predetermined part of a person in an image shown by image data to be evaluated,
The evaluation mapping data is data for defining an evaluation mapping,
The evaluation mapping is a mapping that receives evaluation input data and the position information data as input and outputs an evaluation result of the image data,
The evaluation input data is data that corresponds to the image data and is data that is input to the evaluation mapping,
The execution device is configured to execute a position information generation process and an evaluation process,
The position information generation process is a process of generating the position information data by inputting position input data into the position information mapping,
The position input data is data that corresponds to the image data and is data that is input to the position information mapping,
The evaluation process is a process of evaluating the image data by inputting the position information data and the evaluation input data into the evaluation mapping,
The execution device is configured to execute a provision process and a restriction process,
The provision process includes transmitting the image data to the user terminal so that the image represented by the image data, which has been evaluated by the evaluation process as not having to be restricted from being displayed, can be displayed on the user terminal. It is a process to
The restriction process is a process of restricting the user terminal from displaying an image indicated by the image data that has been evaluated by the evaluation process to be restricted from being displayed;
The evaluation that there is no need to restrict is an evaluation that the degree of body exposure is within a permissible range based on the instruction from the user terminal,
The image evaluation device is characterized in that the evaluation that the display by the user terminal should be restricted is an evaluation that the exposure level is outside the permissible range . The execution device is configured to execute a notification process,
The image evaluation device according to claim 2 , wherein the notification process is a process of notifying that the image is not valid when the validity evaluation process determines that the image is not valid. The image evaluation device according to any one of claims 1 to 4, wherein the predetermined region includes at least one of three parts: a person's chest, buttocks, and front lower abdomen. The position information data is data that provides information as to whether each pixel forming the evaluation input data indicates the predetermined region,
The evaluation mapping is a mapping that outputs the evaluation result by inputting each pixel of the evaluation input data and the position information data corresponding to each pixel in a corresponding manner. The image evaluation device according to any one of the above . An image processing system comprising the image evaluation device according to claim 1 or 2 and a plurality of user terminals. An image evaluation device having an execution device and a storage device, and a plurality of user terminals,
The storage device stores position information mapping data and evaluation mapping data,
The location information mapping data is data for defining a location information mapping,
The location information mapping is a mapping that outputs location information data,
The position information data is data indicating position information of a predetermined part of a person in an image shown by image data to be evaluated,
The evaluation mapping data is data for defining an evaluation mapping,
The evaluation mapping is a mapping that receives evaluation input data and the position information data as input and outputs an evaluation result of the image data,
The evaluation input data is data that corresponds to the image data and is data that is input to the evaluation mapping,
The execution device is configured to execute a position information generation process and an evaluation process,
The position information generation process is a process of generating the position information data by inputting position input data into the position information mapping,
The position input data is data that corresponds to the image data and is data that is input to the position information mapping,
The evaluation process is a process of evaluating the image data by inputting the position information data and the evaluation input data into the evaluation mapping,
The execution device is configured to execute a provision process and a restriction process,
The provision process includes transmitting the image data to the user terminal so that the image represented by the image data, which has been evaluated by the evaluation process as not having to be restricted from being displayed, can be displayed on the user terminal. The process of sending
The restriction process is a process of restricting the user terminal from displaying an image indicated by the image data that has been evaluated by the evaluation process to be restricted from being displayed ;
The user terminal is configured to perform instruction processing,
The instruction process is a process of instructing a permissible range for the degree of body exposure,
The evaluation that there is no need to limit is an evaluation that the degree of exposure is within the permissible range,
The image processing system is characterized in that the evaluation to the effect that display by the user terminal should be restricted is an evaluation to the effect that the exposure level is outside the permissible range . The user terminal in the image processing system according to claim 8. An image evaluation method comprising the step of executing each of the processes in the image evaluation apparatus according to any one of claims 1 to 3 . An image evaluation program that causes a computer to execute each of the processes in the image evaluation apparatus according to any one of claims 1 to 3 .

Images