JP2024011398A - Program, image processing method, image processing device, model generation method, and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program and the like that can sort images with high precision without requiring time for sorting processing.

SOLUTION: A computer acquires a captured image obtained by photographing a subject. The computer generates a subject image by removing a background region from the acquired captured image. The computer inputs the subject image in which the background region is removed from the captured image into a learning model that has been trained to output information regarding the suitability of the captured image when the subject image in which the background region is removed from the captured image is input, and outputs information regarding the suitability of the captured image from the learning model. Then, the computer sorts the captured image in terms of suitability based on the information regarding the suitability output from the learning model.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present application relates to a program, an image processing method, an image processing device, a model generation method, and an image processing system.

Patent Document 1 discloses a sales system that sells images taken by photographers at sports tournaments, events, etc. as photographs and image data. In a sales system such as that disclosed in Patent Document 1, a large number of images taken by a photographer are sorted into images to be sold and images not to be sold, and such sorting processing requires manual labor. It is often done in

JP 2021-18534 Publication

When sorting images manually, sorting may take a long time depending on the number of images, and the sorting results may vary depending on the sensitivity of the person doing the sorting. Therefore, it is conceivable to sort images using a learning model generated by machine learning. However, if the images used for learning a learning model are biased, it may not be possible to generate a learning model that can classify images with high accuracy. Therefore, there is a problem in that it is difficult to sort images with high precision without requiring time for the sorting process. Patent Document 1 does not mention the process of sorting images.

An object of the present disclosure is to provide a program and the like that can sort images with high precision without requiring time for sorting processing.

A program according to one aspect of the present invention acquires a captured image of a subject, removes a background area from the acquired captured image, and when a subject image from which the background area is removed is input, the captured image A subject image obtained by removing the background region from the acquired photographed image is input to a learning model that has been trained to output information regarding the suitability of the photographed image, and information regarding the suitability of the photographed image is outputted from the learning model; A computer is caused to perform a process of classifying the photographed images as appropriate or inappropriate based on information regarding the suitability of the photographed images.

In one aspect of the present invention, images can be sorted with high precision without requiring time for sorting processing.

FIG. 1 is an explanatory diagram showing a configuration example of an image processing system. FIG. 2 is a block diagram showing a configuration example of a server and an administrator terminal. FIG. 2 is an explanatory diagram showing an example of a record layout of a DB stored in a server. It is an explanatory diagram of a learning model. 3 is a flowchart illustrating an example of a learning model generation process procedure. 3 is a flowchart illustrating an example of a procedure for sorting photographic images. It is an explanatory view showing an example of a screen of an administrator terminal. 7 is a flowchart illustrating an example of a sorting process procedure according to the second embodiment.

Below, a program, an image processing method, an image processing device, a model generation method, and an image processing system of the present disclosure will be described in detail based on drawings showing embodiments thereof.

(Embodiment 1)
FIG. 1 is an explanatory diagram showing an example of the configuration of an image processing system. In this embodiment, an image processing system will be described in which images taken by a photographer are sorted to determine whether they are to be sold or not, and the photographed images that have been classified as to be sold are sold. The image processing system of this embodiment can be applied to a system that sells images taken by a photographer on the event organizer's side to event participants as photos and image data at events such as professional sports matches and concerts. be. The image processing system of this embodiment includes a server 10, a camera 20, an administrator terminal 30, a photo vending machine 40, a user terminal 50, etc., and these devices are communicatively connected via a network N. The network N may be the Internet or a public telephone line network, or may be a LAN (Local Area Network) constructed within a facility where the image processing system is installed. Further, the server 10 and any one of the camera 20, the administrator terminal 30, or the photo vending machine 40 may be configured to directly transmit and receive information through wired communication via a cable or wireless communication.

The camera 20 is a photographing device that includes an imaging unit having a lens, an image sensor, etc., a communication unit for connecting to the network N, and the like. The camera 20 performs a process of taking a picture with the imaging unit in accordance with an operation on a photographing button and acquiring image data (hereinafter referred to as a photographed image), and a process of transmitting the acquired photographed image to the server 10 from a communication unit. The camera 20 is configured to perform processing to obtain one image (still image) according to one operation on the shooting button, and processing to obtain, for example, 30 or 15 images (video) per second. has been done. Further, the camera 20 may be configured to sequentially transmit images acquired through photography to the server 10, or may be configured to accumulate captured images in a storage section and transmit them to the server 10 according to an operation by a photographer. Note that the camera 20 may be a camera that takes pictures while being held by a photographer, or may be a camera that takes pictures while the shooting position is fixed using a tripod or a fixing device, and may be a camera that takes pictures while being held by a photographer. A plurality of cameras 20 may be provided at the event venue.

The server 10 is an image processing device capable of various information processing and transmission/reception of information, and is a server computer, a personal computer, or the like. The server 10 acquires photographed images taken by the camera 20, and performs sorting processing on the acquired photographed images to determine whether they are to be sold. The server 10 of this embodiment uses the learning model 12M (see FIG. 2) when sorting whether or not a photographed image is to be sold. The administrator terminal 30 is a terminal used by an administrator who manages the captured images sold by the server 10, and is a personal computer, a tablet terminal, a smartphone, or the like. Note that the administrator performs sorting processing on the captured images that have been sorted into sales items by the server 10 using the learning model 12M to determine whether they should truly be sold.

The server 10 also performs a process of selling, via the network N, captured images that have been sorted into sales targets by the server itself or by the administrator. Photographed images sold by the server 10 are purchased via the photo vending machine 40 or the user terminal 50. The photo vending machine 40 is a terminal installed at a venue or the like where an event was held where photography was performed using the camera 20, and includes a communication section, a touch panel, a payment processing section, and a printing section for communicating with the server 10. Equipped with etc. The photo vending machine 40 displays the photographed images sold by the server 10 on the touch panel, and performs a process of accepting selection of a photographed image to be purchased. The photo vending machine 40 also has a payment processing section that performs payment processing related to the purchase of a photographed image, and a printing section that prints the photographed image for which the payment processing has been performed and provides it to the purchaser. The payment process may be any payment method such as cash payment, electronic money payment, credit card payment, or app payment. The user terminal 50 is a terminal used by a user who purchases captured images, and is a smartphone, a tablet terminal, or the like. The user terminal 50 has a browser installed therein for viewing websites via the network N, and uses the browser to perform payment processing for viewing and purchasing captured images sold by the server 10. Note that when purchasing a photographed image using the user terminal 50, the photographed image is downloaded from the server 10 to the user terminal 50, and can be printed by transmitting the photographed image to a printer with which the user terminal 50 can communicate. Note that, in addition to being downloaded to the user terminal 50, captured images purchased using the user terminal 50 may be downloaded from the server 10 and printed on a printer installed at a convenience store, for example, and may be printed in a predetermined manner. It may also be configured such that the photo is sent to a printing company and the printed photo is delivered to the user's home or the like.

FIG. 2 is a block diagram showing a configuration example of the server 10 and the administrator terminal 30. The server 10 includes a control section 11, a storage section 12, a communication section 13, an input section 14, a display section 15, etc., and these sections are connected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or an AI chip (semiconductor for AI). The control unit 11 executes the information processing and control processing that the server 10 should perform by appropriately executing the program 12P stored in the storage unit 12.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores a program 12P (program product) executed by the control unit 11 and various data. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The program 12P and various data may be written into the storage unit 12 during the manufacturing stage of the server 10, or may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12. . Furthermore, the storage unit 12 stores a learning model 12M that has been trained using training data, for example, by machine learning. The learning model 12M determines whether or not the captured image is suitable for sale when an image captured by the camera 20 (strictly speaking, a subject image from which the background area has been removed) is input. This is a trained model that has been trained to output the information shown. Images that are suitable for sale are images that people would want to purchase, such as images where the subject is in focus, images with good composition or angles, and images that people generally like. In the following, images that are suitable for sale will be referred to as "good images," and images that are not suitable for sale will be referred to as "bad images." The quality of an image means whether or not it is suitable for sale. . The learning model 12M is assumed to be used as a program module that constitutes artificial intelligence software. The learning model 12M performs predetermined calculations on input values and outputs the calculation results, and the storage unit 12 stores data such as coefficients and threshold values of functions that define this calculation as the learning model 12M. be done.

The storage unit 12 also stores a photographed image DB 12a, a sorted image DB 12b, a determination result DB 12c, and a sales image DB 12d. The photographed image DB 12a is a database in which photographed images photographed by the camera 20 are accumulated. The sorted image DB 12b is a database in which the results of sorting captured images into good images or bad images using the learning model 12M are stored. The determination result DB 12c is a database in which the results of the administrator's determination of the quality of photographic images classified into good images using the learning model 12M are stored. The sales image DB 12d is a database that stores photographed images that are determined to be good images by the administrator and are available for sale. A part or all of the learning model 12M, photographed image DB 12a, sorted image DB 12b, judgment result DB 12c, and sales image DB 12d may be stored in another storage device connected to the server 10, with which the server 10 can communicate. It may also be stored in other storage devices.

The communication unit 13 is a communication module for performing processing related to wired communication or wireless communication, and sends and receives information to and from other devices via the network N. The input unit 14 accepts operation input by the user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be integrated into a touch panel, or the touch panel may be externally attached to the server 10.

In this embodiment, the server 10 may be a multicomputer consisting of a plurality of computers, a virtual machine virtually constructed by software, or a cloud server. Further, the input unit 14 and the display unit 15 are not essential to the server 10, and the server 10 may have a configuration that accepts operations through a connected computer, or a configuration that outputs information to be displayed to an external display device. The server 10 also includes a reading section that reads the non-temporary computer-readable portable storage medium 10a, and the reading section may be used to read the program 12P from the portable storage medium 10a and store it in the storage section 12. good. Note that the program 12P may be executed on a single computer, or may be executed on multiple computers interconnected via the network N.

The administrator terminal 30 includes a control section 31, a storage section 32, a communication section 33, an input section 34, a display section 35, etc., and these sections are connected via a bus. Each of the control unit 31, storage unit 32, communication unit 33, input unit 34, and display unit 35 is similar to the control unit 11, storage unit 12, communication unit 13, input unit 14, and display unit 15 of the server 10. Since the present invention has a configuration, a description of the configuration will be omitted. Since the user terminal 50 has the same configuration as the administrator terminal 30, illustration and description of the configuration will be omitted. The photo vending machine 40 has a payment processing section and a printing section in addition to the same configuration as the administrator terminal 30, but a detailed explanation of the configuration will be omitted.

FIG. 3 is an explanatory diagram showing an example of the record layout of the DBs 12a to 12d stored in the server 10. 3A shows the photographed image DB 12a, FIG. 3B shows the sorted image DB 12b, FIG. 3C shows the determination result DB 12c, and FIG. 3D shows the sales image DB 12d. The photographed image DB 12a, the sorted image DB 12b, the judgment result DB 12c, and the sales image DB 12d are provided for each event to be photographed, and are stored in the storage unit 12 in association with the event ID assigned to each event. .

The photographed image DB 12a shown in FIG. 3A includes an image ID column, a file name column, a photographing date and time column, and a photographing location column, and stores information regarding the photographed image in association with the image ID. The image ID column stores identification information (image ID) uniquely assigned to an image taken by the camera 20. The file name column stores folder names and file names for reading captured images stored in the storage unit 12. Note that the photographed images acquired from the camera 20 are stored in a predetermined area (image folder) of the storage unit 12. The photographing date and time column stores the date and time when the photographed image was photographed, and the photographing location column stores information on the photographing location. The information on the shooting location may include the address of the shooting location, the name of the building at the shooting location, the name of the event to be shot, information indicating a location within the event venue, and the like. The storage contents of the photographed image DB 12a are not limited to the example shown in FIG. 3A, and for example, information about the camera 20 that took the photograph and the cameraman may be stored.

The sorting image DB 12b shown in FIG. 3B includes an image ID column and a sorting result column, and the server 10 uses the learning model 12M to associate each captured image with the image ID of each captured image registered in the captured image DB 12a. The result of classifying images into good or bad images (good or bad) is stored. The judgment result DB 12c shown in FIG. 3C includes an image ID column and an administrator judgment result column, and includes each photographed image whose sorting result stored in the sorting image DB 12b is "good", that is, the image is sorted into a good image by the server 10. The result (determined) of the administrator's classification (determination) of each photographed image as a good image or a bad image is stored in association with the image ID of each photographed image. The sales image DB 12d shown in FIG. 3D includes an image ID column and a file name column, and includes each photographed image for which the administrator judgment result stored in the judgment result DB 12c is "good," that is, the image is judged to be a good image by the administrator. A folder name and a file name for reading the photographed image from the storage unit 12 are stored in association with the image ID of each photographed image. The storage contents of the sorting image DB 12b, determination result DB 12c, and sales image DB 12d are not limited to the examples shown in FIGS. 3B to 3D.

FIG. 4 is an explanatory diagram of the learning model 12M, FIG. 4A shows a configuration example of the learning model 12M, and FIG. 4B shows an example of a subject image that is input data of the learning model 12M. The learning model 12M shown in FIG. 4A receives as input a subject image from which the background region has been removed from a photographed image as shown on the left side of FIG. It has been trained to perform calculations to determine whether or not it is suitable for sale, and to output the results of the calculations. The subject extracted as the subject image can be a main subject, such as a subject that appears large in the captured image or a subject that appears close to the center of the captured image. The learning model 12M may be configured using algorithms such as CNN (Convolution Neural Network), SVM (Support Vector Machine), Transformer, etc., or may be configured by combining a plurality of algorithms.

The learning model 12M has an input layer into which a subject image is input, an intermediate layer which extracts feature amounts from the input subject image, and a process that determines whether captured images including the subject image are suitable for sale based on the calculation results of the intermediate layer. and an output layer that outputs information regarding whether or not the output layer exists. The input layer has an input node into which the pixel value of each pixel included in the subject image is input. The intermediate layer uses various functions, threshold values, and the like to calculate an output value based on each pixel value input from the input layer. The output layer (output section) has two output nodes respectively associated with images that are suitable for sale (i.e., good images) and images that are not suitable for sale (i.e., bad images). Output node 0 outputs the probability (confidence) that the captured image should be determined to be a good image, and output node 1 outputs the probability (confidence) that the captured image should be determined to be a bad image. Output. The output value from each output node is, for example, a value between 0 and 1, and the sum of the probabilities output from each output node is 1.0 (100%). In this embodiment, the output value from output node 0 is used as a score indicating the degree to which the photographed image is a good image, that is, the suitability of the image as a sales target.

With the above-described configuration, when a subject image is input, the learning model 12M generates an output value (confidence level) indicating whether the photographed image before the background region is removed is a good or bad image for sale. ) is output. In the learning model 12M described above, the server 10 acquires the output value from the output node 0 as a score indicating the extent to which the photographed image is suitable for sale. Note that the output layer of the learning model 12M may have a configuration having only output node 0 instead of having two output nodes.

The learning model 12M includes training data including a subject image for training and information (correct label) indicating whether the photographed image of the subject image before the background region is removed is a good image or a bad image. It can be generated by machine learning using The training data includes, for example, a correct label indicating the result of the administrator's determination of whether the image is suitable for sale (whether the image is good or not) for a photographed image, and a background area removed from the photographed image. It is generated by adding it to the subject image.

In the learning model 12M, when a subject image included in the training data is input, the output value from the output node corresponding to the correct label (good image or bad image) included in the training data approaches 1, and the output value from the other output node approaches 1. Learn so that the output value from the node approaches 0. In the learning process, the learning model 12M performs calculations in the intermediate layer and the output layer based on the input object image, and calculates an output value from each output node. The learning model 12M compares the calculated output value of each output node with the value corresponding to the correct label (1 for the output node corresponding to the correct label and 0 for the other output node), and The parameters used for calculation processing in the intermediate layer and output layer are optimized so that The parameters include weights (coupling coefficients) between nodes in the intermediate layer and the output layer. The parameter optimization method is not particularly limited, but an error backpropagation method, steepest descent method, etc. can be used. As a result, when a subject image is input, a learning model 12M that predicts whether a captured image before the background region is removed is a good image or a bad image and outputs a prediction result is obtained.

The server 10 prepares such a learning model 12M in advance and uses it when classifying images taken by the camera 20 into good images or bad images. Learning of the learning model 12M may be performed by another learning device. A trained learning model 12M generated by learning performed by another learning device is downloaded from the learning device to the server 10 via the network N or the portable storage medium 10a, and is stored in the storage unit 12, for example.

Below, a process of learning the above-mentioned training data to generate the learning model 12M will be described. FIG. 5 is a flowchart showing an example of the procedure for generating the learning model 12M. The following processing is executed by the control unit 11 of the server 10 according to the program 12P stored in the storage unit 12, but may be executed by another learning device. In the following processing, the control unit 11 first generates training data based on the captured images stored in the storage unit 12, and uses the generated training data to perform learning of the learning model 12M. It is assumed that the photographed images used for the training data are judged to be good or bad by an administrator or the like, and the judgment results are added to the photographed images and stored in a predetermined area (predetermined DB) of the storage unit 12.

The control unit 11 of the server 10 reads out the photographed image stored in the storage unit 12 and the judgment result (information regarding the suitability of the photographed image) determined by the administrator etc. for the photographed image (S11 ). The control unit 11 performs a background removal process to remove a background area on the read photographic image (S12). Background removal processing may be any type of processing, such as processing using OpenCV's background extraction class (BackgroundSubtrator), processing using the image background removal tool (remove.bg) provided by Kaleido, or U2- It may also be a process using a deep learning learning model such as Net. In addition, when using a learning model, a learning model that has been trained to output a subject image when a captured image is input is used by using a captured image and a foreground image (subject image) from which the background has been removed. do. Furthermore, when there is a photographed image in which only the background is photographed (background image), a configuration may be adopted in which a subject image is generated by using the background image and removing the background region from the photographed image. Through such background removal processing, the control unit 11 generates a subject image from which the background region has been removed from the photographed image.

The control unit 11 generates training data by adding a correct label to the generated subject image according to the good/bad determination result read in step S11, and stores it in the storage unit 12 (S13). Specifically, when the determination result is good, the control unit 11 attaches a correct label of good to the photographed image, and when the determination result is bad, it attaches a correct label of bad to the photographed image. The control unit 11 stores the generated training data in a training DB (not shown) provided in the storage unit 12, for example.

The control unit 11 determines whether or not there are unprocessed captured images that are not used in the training data generation process among the captured images stored in the storage unit 12 (S14). If it is determined that there is an unprocessed photographed image (S14: YES), the control unit 11 returns to the process of step S11 and performs the processes of steps S11 to S13 on the unprocessed photographed image. The control unit 11 repeats the processing of steps S11 to S14 until it determines that there are no unprocessed captured images. As a result, training data used for learning the learning model 12M is generated based on the photographed image stored in the storage unit 12 and the determination result for the photographed image, and is stored in the training DB. In the above-described process, the photographed images used to generate the training data are stored in the storage unit 12. However, the control unit 11 of the server 10 receives each photographed image from another device via the network N, for example. It may also be configured to acquire the determination result.

When the control unit 11 determines that there are no unprocessed captured images (S14: NO), the control unit 11 performs learning of the learning model 12M using the training data accumulated in the training DB as described above. The control unit 11 reads out one of the training data accumulated in the training DB through the above-described processing (S15). Then, the control unit 11 performs a learning process on the learning model 12M based on the read training data (S16). Here, the control unit 11 inputs a captured image included in the training data to the learning model 12M, and acquires an output value output from the learning model 12M by inputting the captured image. The control unit 11 outputs the output value of each output node output from the learning model 12M and a value corresponding to the correct label included in the training data (1 for the output node corresponding to the correct label, and 1 for the other output nodes). 0), and the learning model 12M is trained so that the two approximate each other. In the learning process, the learning model 12M optimizes parameters used for arithmetic processing in the intermediate layer and the output layer. For example, the control unit 11 optimizes parameters such as weights (coupling coefficients) between nodes in the intermediate layer and the output layer using an error backpropagation method that sequentially updates parameters from the output layer to the input layer of the learning model 12M. .

The control unit 11 determines whether or not there is unprocessed training data that has not been subjected to learning processing among the training data stored in the training DB (S17). If it is determined that there is unprocessed training data (S17: YES), the control unit 11 returns to the process of step S15, and performs the processes of steps S15 to S16 on the training data that has not been subjected to the learning process. If it is determined that there is no unprocessed training data (S17: NO), the control unit 11 ends the series of processes.

Through the learning process described above, when a subject image is input, an output value indicating the possibility that the captured image before the background area is removed is a good image and an output value indicating the possibility that it is a bad image are determined. A learning model 12M to be output is generated. Therefore, the server 10 can obtain information regarding whether a photographed image is a good image or a bad image (information regarding suitability) based on the output value from the learning model 12M. Note that in the above-described process, the training data generation process in steps S11 to S14 and the learning model 12M generation process in steps S15 to S17 may be performed in separate devices. The learning model 12M can be further optimized by repeatedly performing learning processing using training data as described above. In addition, by relearning the learning model 12M that has already been trained through the above-described learning process, it is possible to generate a learning model 12M with further improved discrimination accuracy.

Below, in the image processing system of this embodiment, a process of sorting photographic images taken using the camera 20 into images to be sold and images not to be sold will be described. FIG. 6 is a flowchart showing an example of a photographed image sorting process procedure, and FIG. 7 is an explanatory diagram showing an example of a screen of the administrator terminal 30. In FIG. 6, the processing performed by the camera 20 is shown on the left, the processing performed by the server 10 is shown in the center, and the processing performed by the administrator terminal 30 is shown on the right.

In the image processing system of this embodiment, a cameraman uses the camera 20 to photograph performers, participants, and the like of an event to obtain photographed images. Note that the camera 20 may be configured to automatically take pictures according to a preset shooting timing. The camera 20 performs photographing using the imaging section (S21), and transmits the acquired photographed image to the server 10 through the communication section (sending section) (S22). Note that the camera 20 may have a configuration in which it transmits the obtained captured image to the server 10 each time it performs photography, or it may have a configuration in which it transmits a plurality of captured images together to the server 10. In the following, a configuration will be described in which steps S22 to S35 are performed each time one photograph is taken, but steps S22 to S35 may be performed after a plurality of photographs are taken. In this case, in steps S22 to S35, each process is performed on each of the plurality of captured images.

The control unit 11 (acquisition unit) of the server 10 acquires the captured image transmitted from the camera 20, and stores the acquired captured image in the storage unit 12 (S23). At this time, the control unit 11 stores the photographed image in a predetermined area (image folder) of the storage unit 12, and also stores information regarding the photographed image in the photographed image DB 12a. Note that the information on the photographing date and time and the photographing location may be acquired from the camera 20 together with the photographed image, or may be registered in advance. Next, the control unit 11 (removal unit) performs background removal processing on the acquired photographed image (S24), and generates a subject image with the background region removed from the photographed image. As the background removal process, a process similar to step S12 in FIG. 5 can be used. The control unit 11 calculates a score indicating the degree to which the photographed image before the background region is removed is a good image, based on the generated subject image (S25). Specifically, the control unit 11 inputs the subject image to the learning model 12M, and obtains the output value from the output node 0 as a score for the captured image. The control unit 11 (sorting unit) sorts the captured images into good images or bad images based on the acquired scores (S26). For example, the control unit 11 classifies the photographed image as a good image when the acquired score is equal to or higher than a predetermined threshold (for example, 0.7), and classifies the photographed image as a bad image when the acquired score is less than the predetermined threshold. A threshold value for classifying captured images into good images or bad images is set in advance and stored in the storage unit 12, and can be changed according to an operation via the input unit 14. For example, when the control unit 11 receives an instruction to change the setting of the threshold value via the input unit 14, the setting of the threshold value is changed by updating the threshold value stored in the storage unit 12 to the threshold value for which the change instruction has been given. Ru.

The control unit 11 stores the sorted results (good or bad) in the sorted image DB 12b in association with the image ID of the captured image (S27). Next, the control unit 11 reads out the captured images that have been sorted into good images from the storage unit 12 based on the stored content of the sorted image DB 12b (S28), and transmits the read out captured images to the administrator terminal 30 (S29). . The control unit 31 of the administrator terminal 30 displays the captured images transmitted by the server 10 and sorted into good images on the display unit 35 (S30). For example, the control unit 31 displays a screen as shown in FIG. 7 on the display unit 35, and presents the captured image acquired from the server 10 to the administrator. Note that the server 10 may transmit a subject image obtained by removing the background area from the photographed image to the administrator terminal 30 together with or instead of the photographed image, and the administrator terminal 30 may transmit the subject image with the photographed image or in place of the photographed image. Instead of the image, a subject image may be displayed and presented to the administrator. The screen shown in FIG. 7 is configured to accept a determination result as to whether the captured image being displayed is a good image or a bad image. button, and a "Hold" button. The administrator determines whether the photographed image is a good image or a bad image by operating a "good" button or a "bad" button via the input unit 34 on the screen shown in FIG. Furthermore, if the administrator cannot determine whether the photographed image is good or bad, the administrator inputs that the determination cannot be made by operating the "Hold" button. When one of the buttons is operated, the control unit 31 receives the determination result by the administrator (S31), and transmits the received determination result by the administrator to the server 10 (S32).

The control unit 11 of the server 10 receives the judgment result sent by the administrator from the administrator terminal 30, and stores the received judgment result (good or bad) in the judgment result DB 12c in association with the image ID of the photographed image. (S33). The control unit 11 identifies the captured images classified as good images by the administrator based on the stored contents of the determination result DB 12c (S34), associates the image ID and file name of the identified captured images, and sells the images. It is stored in the DB 12d (S35). The photographed images stored in the sales image DB 12d are sold by the server 10 via the network N via the photo vending machine 40 or the user terminal 50. Note that the server 10 transmits a thumbnail list of photographed images to be sold to the photo vending machine 40 or user terminal 50, and acquires from the photo vending machine 40 or user terminal 50 a purchase request for the photographed images received via the thumbnail list. Then, the photographed image that the user wishes to purchase is output to the photo vending machine 40 or the user terminal 50. The photo vending machine 40 prints the captured image that the user desires to purchase obtained from the server 10 using the printing unit, and provides the image to the purchaser. Note that the process of selling photographed images is a common process, so a description thereof will be omitted here.

The administrator's judgment result for the photographed image stored in the judgment result DB 12c through the above-described processing can be used when relearning the learning model 12M. Specifically, the control unit 11 of the server 10 generates training data based on the stored contents of the determination result DB 12c by executing the process shown in FIG. 5 on the captured images stored in the determination result DB 12c. processing and learning processing of the learning model 12M can be executed. Thereby, the discrimination accuracy of the learning model 12M can be further improved. Furthermore, in the present embodiment, the learning model 12M is trained using a subject image from which the background region has been removed from a photographed image and a determination result of the quality of the photographed image as training data. Not limited. For example, for a learning model that is trained using training data that includes a captured image and a judgment result for the captured image, training data that includes a subject image and a judgment result for the captured image is used. It may also be generated by fine-tuning (transfer learning).

Through the above-described processing, the image processing system of the present embodiment uses the learning model 12M to calculate a score indicating the extent to which the photographed image taken by the camera 20 is suitable for sale, and based on the calculated score, Photographed images are classified into good images and bad images. Therefore, there is no need for manual image sorting work, and the workload can be reduced. Furthermore, when images are sorted manually, there may be differences in the sorting results depending on the sensitivity of the person doing the sorting, but in this embodiment, sorting is done based on the scores calculated by the learning model 12M. Therefore, objective sorting results can be obtained. Furthermore, since the learning model 12M calculates a score for the photographed image based on the subject image from which the background region has been removed, it is possible to improve the discrimination accuracy without being affected by the background region. For example, when learning captured images, if the images used as training data are biased, learning accuracy may decrease. For example, if a learning model is trained using training data in which a captured image that is determined to be a good image contains a large amount of a predetermined first color, and a captured image that is determined to be a bad image contains a large amount of a predetermined second color. For example, even an out-of-focus image may be determined to be a good image if it contains a large amount of the first color. However, in the present embodiment, since the quality of the image is determined based on the subject image obtained by removing the background region from the captured image, a decrease in the determination accuracy is suppressed.

In this embodiment, it is possible to change the threshold value used when classifying captured images into good images or bad images. Therefore, the administrator performs sorting by changing the threshold according to the number of cameras 20, the number of images taken by the cameras 20, the number of administrators performing sorting, the time that can be spent on sorting work, etc. It becomes possible to adjust the number of images. Furthermore, in this embodiment, the server 10 sorts the captured images using the learning model 12M, and the administrator further sorts the captured images that have been sorted into good images. As a result, captured images whose scores output by the learning model 12M are less than the threshold are excluded from sales targets without being checked by the administrator. Therefore, the number of images to be sorted by the administrator can be reduced, and the burden of sorting work on the administrator can be reduced. Note that the captured images that have been sorted into good images using the learning model 12M may be sold as they are without being sorted by the administrator. In this case, the workload of the administrator can be further reduced. For example, if a high value (for example, 0.8 or 0.9) is set for the threshold, captured images that are classified as good images based on the scores output by the learning model 12M will be judged as good images by the administrator as well. This is highly likely, and in this case, the configuration may be such that the administrator does not perform sorting.

In this embodiment, the sorting process for determining whether or not a photographed image is to be sold is performed based on the score for a subject image obtained by removing the background area from the photographed image. In addition to this, in addition to the score for the subject image, the score for the photographed image before the background area is removed may also be taken into consideration in the sorting process to determine whether the image is to be sold. For example, the control unit 11 determines whether the score for the subject image acquired using the learning model 12M is less than a predetermined value (for example, 0.6), and when determining that it is less than the predetermined value, A learning model is used to obtain a score for a photographed image before the background region of the subject image is removed. Then, the control unit 11 determines whether or not the score for the acquired photographed image is equal to or higher than a predetermined value (for example, 0.8), and if it is determined that the score is equal to or higher than the predetermined value, the control section 11 determines whether or not the score for the acquired photographed image is a target for sale. It may be configured to do so. At this time, the learning model that calculates the score for the photographed image may be the same model as the learning model 12M that calculates the score for the subject image, or may be a different model.

In this embodiment, the administrator terminal 30 performs one or more of the processes performed by the server 10 among the training data generation process, the learning process of the learning model 12M using the training data, and the process shown in FIG. It can also be configured to be performed locally. For example, the administrator terminal 30 may generate training data by executing the process shown in FIG. 5, and the learning model 12M may be generated using the generated training data and stored in the storage unit 32. Thereby, the process performed by the server 10 in FIG. 6 can be executed by the administrator terminal 30. Even in the case of such a configuration, the same processing as in this embodiment is possible and the same effects can be obtained.

(Embodiment 2)
An image processing system that generates still images from moving images captured by the camera 20 and sorts the generated still images to determine whether or not to sell them will be described. Since the image processing system of this embodiment is realized using the same devices as the image processing system of Embodiment 1 shown in FIGS. 1 and 2, description of the configuration of each device will be omitted.

FIG. 8 is a flowchart showing an example of the sorting processing procedure of the second embodiment. The process shown in FIG. 8 is the process shown in FIG. 6 with steps S41 to S42 added before step S23. Description of the same steps as in FIG. 6 will be omitted.

In the image processing system of this embodiment, the camera 20 executes the same processing as steps S21 to S22 in FIG. Note that the image taken by the camera 20 may be a still image or a moving image. When the control unit 11 of the server 10 acquires the captured image transmitted from the camera 20, it determines whether the acquired captured image is a moving image (S41). If it is determined that the photographed image is not a moving image (S41: NO), that is, if it is a still image, the control unit 11 moves to the process of step S23 and executes the same process as in the first embodiment. If it is determined that the captured image is a moving image (S41: YES), the control unit 11 generates a still image from the captured image that is a moving image (S42). For example, if the photographed image is a moving image containing 30 frames per second, the control unit 11 generates a still image from the photographed image every second, and generates 30 still images per second. Note that the control unit 11 does not need to generate still images from all frames, and may generate still images by extracting frames at predetermined intervals (every 0.1 seconds, every 0.5 seconds, etc.), for example. good.

The control unit 11 stores the still image generated from the moving image in the storage unit 12 as a photographed image (S23), and executes the processing from step S24 onwards. Note that since a plurality of captured images (still images) are generated from one moving image, each process is performed on the plurality of captured images (still images) in steps S23 to S35. As a result, in this embodiment as well, it is possible to perform sorting processing on the photographed images taken with the camera 20 to determine whether or not they should be sold. The still images generated from the still images can be sorted to determine whether they should be sold.

In this embodiment, still images generated from videos are sold, but videos themselves may be sold. For example, the control unit 11 generates a plurality of still images from a video, inputs each generated still image to the learning model 12M, obtains a score for each still image, and sets the average value of the scores of each still image to a predetermined value. (for example, 0.8) or more, the video may be made available for sale. In addition, when each frame included in a video is input, it is learned to output information indicating whether the video is suitable for sale (a score indicating the degree to which the video is suitable for sale). The learned model may be used to determine whether each video is to be sold. In this case, the control unit 11 inputs the video into the learning model, acquires a score for the video from the learning model, and sells the video when the acquired score is equal to or higher than a predetermined value (for example, 0.8). It may be configured to be targeted.

In this embodiment, the same effects as in the first embodiment described above can be obtained. Furthermore, in this embodiment, based on the video captured by the camera 20, still images generated from the video can be sold. Therefore, by photographing a moving image using the camera 20, still images generated from frames included in the moving image can be sold, and a large amount of objects for sale can be collected. Also in this embodiment, the modifications described in the above-described first embodiment can be applied.

Regarding the embodiments including Embodiments 1 and 2 above, the following additional notes are further disclosed.

(Additional note 1)
Obtain a photographed image of the subject,
Remove the background area from the captured image,
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. and outputting information regarding the suitability of the photographed image from the learning model,
A program that causes a computer to perform a process of classifying the photographed images as appropriate or inappropriate based on information regarding the suitability of the photographed images.

(Additional note 2)
Outputting the photographed images that have been appropriately sorted;
Accepting a determination of suitability for the outputted photographed image,
The program according to supplementary note 1, which causes the computer to execute a process of acquiring training data including the photographed image and the received information regarding suitability.

(Additional note 3)
The information regarding the suitability of the photographed image is a score indicating the suitability of the photographed image,
Accepting a threshold value for classifying the suitability of the photographed image,
The program according to supplementary note 1 or 2, which causes the computer to perform a process of classifying the acquired captured images as appropriate or inappropriate based on a received threshold value.

(Additional note 4)
Obtain a video of the subject,
Remove the background area from the frames included in the acquired video,
A subject image obtained by removing a background region from the frame is input to the learning model, and information regarding the suitability of the frame is output from the learning model. program.

(Appendix 5)
Obtain a photographed image of the subject,
Remove the background area from the captured image,
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. and outputting information regarding the suitability of the photographed image from the learning model,
An image processing method in which a computer performs a process of classifying the photographed images as appropriate or unsuitable based on information regarding the suitability of the photographed images.

(Appendix 6)
an acquisition unit that acquires a photographed image of the subject;
a removal unit that removes a background area from the acquired captured image;
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. an output unit that outputs information regarding the suitability of the photographed image from the learning model;
An image processing device comprising: a sorting unit that sorts the captured images into suitability based on information regarding the suitability of the captured images.

(Appendix 7)
Obtaining training data including a subject image from which a background region has been removed from a photographed image of the subject, and information regarding the suitability of the photographed image;
A model generation method in which a computer executes a process of generating a learning model that outputs information regarding the suitability of the photographed image when the subject image is input using acquired training data.

(Appendix 8)
Obtaining a photographed image of the subject and information regarding the suitability of the photographed image,
Remove the background area from the captured image,
The model generation method according to appendix 7, wherein the computer executes a process of acquiring training data including a subject image obtained by removing the background region from the photographed image and information regarding the suitability of the acquired photographed image.

(Appendix 9)
An image processing system including a photographing device and an image processing device,
The photographing device is
comprising a sending unit that sends a photographed image of a subject to the image processing device,
The image processing device includes:
an acquisition unit that acquires the photographed image from the photographing device;
a removal unit that removes a background area from the acquired captured image;
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. an output unit that outputs information regarding the suitability of the photographed image from the learning model;
An image processing system, comprising: a sorting unit that sorts the captured images into suitability based on information regarding the suitability of the captured images.

The embodiments disclosed herein are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the claims rather than the above-mentioned meaning, and is intended to include meanings equivalent to the claims and all changes within the scope.

10 Server 11 Control Unit 12 Storage Unit 13 Communication Unit 14 Input Unit 15 Display Unit 20 Camera 30 Administrator Terminal 31 Control Unit 32 Storage Unit 33 Communication Unit 40 Photo Vending Machine 50 User Terminal 12M Learning Model

Claims (9)

Obtain a photographed image of the subject,
Remove the background area from the captured image,
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. and outputting information regarding the suitability of the photographed image from the learning model,
A program that causes a computer to perform a process of classifying the photographed images as appropriate or inappropriate based on information regarding the suitability of the photographed images. Outputting the photographed images that have been appropriately sorted;
Accepting a determination of suitability for the outputted photographed image,
The program according to claim 1, which causes the computer to execute a process of acquiring training data including the photographed image and the received information regarding suitability. The information regarding the suitability of the photographed image is a score indicating the suitability of the photographed image,
Accepting a threshold value for classifying the suitability of the photographed image,
The program according to claim 1 or 2, which causes the computer to perform a process of classifying the obtained captured images as appropriate or inappropriate based on the received threshold value. Obtain a video of the subject,
Remove the background area from the frames included in the acquired video,
The program according to claim 1 or 2, which causes the computer to execute a process of inputting a subject image obtained by removing a background region from the frame to the learning model, and outputting information regarding suitability of the frame from the learning model. Obtain a photographed image of the subject,
Remove the background area from the captured image,
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. and outputting information regarding the suitability of the photographed image from the learning model,
An image processing method in which a computer performs a process of classifying the photographed images as appropriate or unsuitable based on information regarding the suitability of the photographed images. an acquisition unit that acquires a photographed image of the subject;
a removal unit that removes a background area from the acquired captured image;
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. an output unit that outputs information regarding the suitability of the photographed image from the learning model;
An image processing device comprising: a sorting unit that sorts the captured images into suitability based on information regarding the suitability of the captured images. Obtaining training data including a subject image from which a background region has been removed from a photographed image of the subject, and information regarding the suitability of the photographed image;
A model generation method in which a computer executes a process of generating a learning model that outputs information regarding the suitability of the photographed image when the subject image is input using acquired training data. Obtaining a photographed image of the subject and information regarding the suitability of the photographed image,
Remove the background area from the captured image,
8. The model generation method according to claim 7, wherein the computer executes a process of acquiring training data including a subject image obtained by removing the background region from the photographed image and information regarding the suitability of the acquired photographed image. An image processing system including a photographing device and an image processing device,
The photographing device is
comprising a sending unit that sends a photographed image of a subject to the image processing device,
The image processing device includes:
an acquisition unit that acquires the photographed image from the photographing device;
a removal unit that removes a background area from the acquired captured image;
A subject image from which the background area has been removed from the captured image is input to a learning model that has been trained to output information regarding the suitability of the captured image when a subject image from which the background area has been removed is input. an output unit that outputs information regarding the suitability of the photographed image from the learning model;
An image processing system, comprising: a sorting unit that sorts the captured images into suitability based on information regarding the suitability of the captured images.

Images