JP7448271B1 - Information processing system, program and information processing method - Google Patents

Abstract

The present invention provides an information processing system and the like that can easily grasp information about a group of images used for machine learning. According to one aspect of the present invention, an information processing system including a processor is provided. In this information processing system, in the information acquisition step, the processor acquires related information related to a group of images used for machine learning. In the output step, metadata indicating related information acquired for the image group is output in association with an image generated by an artificial intelligence module that performs machine learning using the image group. [Selection diagram] Figure 1

Description

The present invention relates to an information processing system, a program, and an information processing method.

Patent Document 1 discloses that a combination of each image of a sub-learning image group, which is a group of multiple images in which a subject is photographed, and posture information of the specific subject in each image of the sub-learning image group is used as training data. A technique is disclosed for newly generating an image of the specific subject based on parameters of an image generator obtained by performing a learning process using the above-described method.

Japanese Patent Application Publication No. 2020-86869

In order for the artificial intelligence module to generate images, machine learning is performed using existing images as training data. In this case, understanding the images used for machine learning may be important when using the generated images.

In view of the above circumstances, the present invention provides an information processing system and the like that can easily grasp information about a group of images used for machine learning.

According to one aspect of the present invention, an information processing system including a processor is provided. In this information processing system, in the information acquisition step, the processor acquires related information related to a group of images used for machine learning. In the output step, metadata indicating related information acquired for the image group is output in association with an image generated by an artificial intelligence module that performs machine learning using the image group.

According to this aspect, information about the image group used for machine learning can be easily grasped.

1 is a diagram showing an example of the overall configuration of an image generation system 1. FIG. 1 is a diagram showing an example of a hardware configuration of a server device 10. FIG. 2 is a diagram illustrating an example of a hardware configuration of an operator terminal 20. FIG. FIG. 2 is an activity diagram showing an example of data set creation processing. It is a figure which shows an example of the displayed data set creation screen. It is a diagram showing an example of an image database DB1. FIG. 3 is a diagram showing an example of displayed candidate images. It is a figure showing an example of generated data set information. FIG. 3 is an activity diagram showing an example of learning processing. It is a figure which shows an example of the displayed screen for machine learning. FIG. 3 is a diagram showing an example of stored learning model information. FIG. 3 is an activity diagram showing an example of image generation processing. FIG. 3 is a diagram showing an example of a displayed image generation screen. It is a diagram showing an example of a displayed AI image screen. FIG. 3 is a diagram showing an example of a displayed DS image screen. FIG. 3 is a diagram showing an example of a displayed contract information screen. It is a figure which shows an example of displayed metadata. It is a figure which shows an example of displayed metadata. FIG. 3 is a diagram illustrating an example of a screen for selecting displayed rights information. It is a figure which shows an example of the screen for changing the displayed DS image. FIG. 7 is a diagram showing another example of stored learning model information. FIG. 7 is a diagram showing another example of displayed metadata. It is a figure which shows an example of the displayed screen for adding a user image. FIG. 7 is a diagram showing another example of stored learning model information. FIG. 3 is a diagram illustrating an example of learning model information managed in a node format.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments described below can be combined with each other.

By the way, the program for implementing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be downloaded from an external server. The program may be provided in a manner that allows the program to be started on an external computer and the function thereof is realized on the client terminal (so-called cloud computing).

Furthermore, in this embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by circuits in a broad sense and software information processing that can be concretely implemented by these hardware resources. . In addition, various information is handled in this embodiment, and these information includes, for example, the physical value of a signal value representing voltage and current, and the signal value as a binary bit group consisting of 0 or 1. It is expressed by high and low levels or quantum superposition (so-called quantum bits), and communication and calculations can be performed on circuits in a broad sense.

Further, a circuit in a broad sense is a circuit realized by at least appropriately combining a circuit, a circuit, a processor, a memory, and the like. That is, Application Specific Integrated Circuit (ASIC), programmable logic device (for example, Simple Programmable Logic Device (SPLD)), Complex Programmable Logic Device (Complex Pr) ogrammable Logic Device: CPLD), and field This includes a field programmable gate array (FPGA) and the like.

<Embodiment 1>
1. System Configuration The system configuration according to the first embodiment will be described below.
FIG. 1 is a diagram showing an example of the overall configuration of an image generation system 1. As shown in FIG. FIG. 1 shows an overview of each device included in the image generation system 1 and the users who use those devices. Each outline will be explained from time to time with reference to other figures. The image generation system 1 is an information processing system that generates images using an artificial intelligence module that has a function of generating images. The image generation system 1 executes a learning process that causes an artificial intelligence module to perform machine learning, an image generation process that causes a learned artificial intelligence module to generate an image, and the like.

The image generation system 1 includes a communication line 2, a server device 10, an operator terminal 20, and a user terminal 30. The communication line 2 is configured by, for example, the Internet network, although it is not particularly limited. Furthermore, the communication line 2 may include a local area network, a mobile communication network, a VPN (Virtual Private Network), and the like. The communication line 2 mediates the exchange of data between devices connected to the communication line. In the example of FIG. 1, a server device 10 is connected to the communication line 2 by wire, and an operator terminal 20 and a user terminal 30 are connected wirelessly. Note that the connection of each device to the communication line 2 may be wired or wireless.

The server device 10 is an information processing device that performs learning processing, image generation processing, and the like. The server device 10 stores an image database DB1, a dataset database DB2, and a learning model database DB3. The image database DB1 stores images and the like used for machine learning. The dataset database DB2 stores information regarding datasets (hereinafter referred to as "dataset information") including a group of images selected from the images stored in the image database DB1.

The dataset here refers to a collection of data used as training data for machine learning. Data set information is, for example, information that allows identification of images included in a data set (hereinafter referred to as "DS images"). The learning model database DB3 stores information regarding learning models (hereinafter referred to as "learning model information") generated by machine learning performed using data sets. Data set information and learning model information will be explained in detail later.

The server device 10 includes an image generation AI module 100. The image generation AI module 100 is an artificial intelligence module that is tuned to generate images using AI (Artificial Intelligence) technology. The image generation AI module 100 performs machine learning using a dataset including a DS image specified by dataset information stored in the dataset database DB2, and generates an image having characteristics similar to the DS image. Hereinafter, the image generated by the image generation AI module 100 will be referred to as an "AI image."

The operator terminal 20 is a terminal whose user is an operator who operates the image generation system 1, and is, for example, a notebook computer. The user terminal 30 is a terminal whose user is an image user who uses an AI image on a web page or the like, and is, for example, a notebook computer. The server device 10 executes display processing for displaying images on the operator terminal 20 and user terminal 30, and authentication processing for authenticating the users of the operator terminal 20 and user terminal 30.

For example, the server device 10 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file as display processing, and displays a web page showing a system screen on user terminals such as the operator terminal 20 and the user terminal 30. let Note that the user terminal may install an application program for using the image generation system 1, and the server device 10 may perform processing such as generation and transmission of display data in the application as display processing. The server device 10 controls the display of the user terminal by performing these display processes.

The server device 10 stores authentication information (user ID, password, etc.) for authenticating users such as operators and image users, and authenticates users who input the authentication information. By authenticating the user, the server device 10 can restrict access to data and make it possible to identify data input by the user.

2. Hardware Configuration The hardware configuration according to the first embodiment will be described below.
FIG. 2 is a diagram showing an example of the hardware configuration of the server device 10. The server device 10 includes a control section 11, a storage section 12, a communication section 13, and a bus 14. The bus 14 electrically connects each part of the server device 10.

(Control unit 11)
The control unit 11 has at least one processor. The at least one processor may include, for example, a central processing unit (CPU), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), one or more Integrated Circuits (not shown), or one or more integrated circuits (not shown). The above Discrete Circuit and these It may be configured by a combination of.

The control unit 11 is a computer that implements various functions related to the image generation system 1 by reading predetermined programs stored in the storage unit 12. That is, information processing by software stored in the storage unit 12 is specifically implemented by the control unit 11, which is an example of hardware, so that it can be executed as each functional unit included in the control unit 11. Note that the control section 11 is not limited to a single control section, and may be implemented so as to have a plurality of control sections 11 for each function. It may also be a combination thereof.

(Storage unit 12)
The storage unit 12 stores various information defined by the above description. This may be, for example, a storage device such as a solid state drive (SSD) or an HDD (Hard Disk Drive) that stores various programs related to the image generation system 1 executed by the control unit 11, or It can be implemented as a memory such as a random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program operations. The storage unit 12 stores various programs, variables, etc. related to the image generation system 1 executed by the control unit 11.

(Communication Department 13)
The communication unit 13 is configured by a communication module. The communication module may be a wireless communication module that complies with standards such as IEEE802.11a/b/g/n/ac/ax, LTE, 5G, 6G, etc., or a wired communication module that complies with standards such as IEEE802.3. It may be. The communication unit 13 is configured to be capable of transmitting various electrical signals from the server device 10 to external components. Furthermore, the communication unit 13 is configured to be able to receive various electrical signals from external components to the server device 10 . More preferably, the communication unit 13 has a network communication function, so that various information can be communicated between the server device 10 and external devices via the communication line 2.

FIG. 3 is a diagram showing an example of the hardware configuration of the operator terminal 20. The operator terminal 20 includes a control section 21 , a storage section 22 , a communication section 23 , an input section 24 , an output section 25 , and a bus 26 . The bus 26 electrically connects each part of the operator terminal 20. The control unit 21, the storage unit 22, and the communication unit 23 are the same hardware as the control unit 11, the storage unit 12, and the communication unit 13 shown in FIG. 2, although specifications, models, etc. may be different.

(Input section 24)
The input unit 24 includes keys, buttons, a touch screen, a mouse, etc., and accepts input from the user. The input unit 24 also includes a microphone and receives voice input from the user.

(Output section 25)
The output unit 25 has a display, a speaker, etc., and displays visual information generated in a manner visible to the user, such as a screen, image, icon, text, etc., on the display surface of the display, and outputs sound including audio. do.

The user terminal 30 shown in FIG. 3 has the same hardware configuration as the operator terminal 20. Regarding the user terminal 30, only the control section 31 will be described with a different reference numeral from the control section 21 of the operator terminal 20.

3. Information Processing Information processing according to the embodiment will be described below. In the following description, the server device 10, the operator terminal 20, and the user terminal 30 will be described as the main bodies of each information process, but these information processes are executed by a processor included in the control unit of each device. The information processing executed by each processor includes, in addition to the above-described learning processing and image generation processing, a data set creation process for creating a data set for use in machine learning.

FIG. 4 is an activity diagram showing an example of data set creation processing. The data set creation process shown in FIG. 4 is started when the user of the operator terminal 20 performs an operation to display a data set creation screen. First, the server device 10 generates screen data indicating a data set creation screen, and transmits the generated screen data to the operator terminal 20 (A11). The operator terminal 20 displays the data set creation screen indicated by the transmitted screen data (A12).

On the dataset creation screen, a DS image (an image to be included in the dataset) is selected by the operator's operation. Methods for selecting a DS image include a method of individually inputting the file name and path name of an image to be selected as a DS image, and a method of specifying a folder in which an image to be selected as a DS image is stored. Now, a method of narrowing down images having common features from a large number of images as images that are candidates for DS images (hereinafter referred to as "candidate images") and then selecting images to be used as DS images will be described.

FIG. 5 is a diagram showing an example of a displayed data set creation screen. The dataset creation screen G1 shown in FIG. A condition input field E13, an image right condition input field E14, an image display field E15, a select all button B11, a deselection button B12, and a decision button B13 are displayed.

As described above, a person who uses the AI image is input as an image user in the input field E11. In the input field E12, the name of the data set to be created is input. In the input fields E13 and E14, narrowing conditions for narrowing down candidate images are input. These narrowing conditions are conditions that can be determined based on information associated with each image in the image database DB1, which will be described later.

In the input field E13, conditions for attributes of candidate images are input as narrowing conditions. Image attribute conditions include, for example, image size, resolution, and subject. The subject refers to the main object in the photograph when the image is a photograph, and includes people, scenery, and the like. Note that an image whose subject is a "person" is an image in which it is possible to identify the person, and the person has portrait rights. The attribute conditions listed here are just examples, and any attributes (for example, shooting date and time, shooting location, name of the subject, etc.) may be used as long as they are conditions for narrowing down candidate images.

For example, when "large", "medium", and "small" are input as image sizes in the input field E13, only one size or multiple sizes may be input. , a condition that does not narrow down the size, such as "all sizes", may be input. In this way, it is sufficient that one or more conditions are input as the image attribute condition. Note that when no conditions are input, the conditions may not be narrowed down in the same way as when "all sizes" is input.

In the input field E14, conditions regarding the rights that the image user has with respect to the candidate images are input as conditions for narrowing down the candidate images. Rights held by image users include, for example, the copyright granted to the person who created the candidate image, the copyright held to the creator (author) of the work depicted in the candidate image, and the person depicted in the candidate image. These include portrait rights that can be claimed by a person, or rights to use candidate images that are granted by the owner of those rights (hereinafter simply referred to as "permission").

In the input field E14, information such as availability, purpose of use, and expiry date of use are input. Usage availability indicates whether or not the image user can currently use the image, in other words, whether or not the image user currently has any rights that enable the image to be used. In the following, among the rights that enable the use of images, copyright or its permission will be referred to as "work-related rights," and portrait rights or its permission will be referred to as "portrait-related rights."

Below, if the subject is an image of a person, it can be used if the image user has both copyright-related rights and portrait-related rights. If the applicant does not have both copyright-related rights and portrait-related rights, it will be shown as “×”. In addition, if the subject is an image other than a person, it will be indicated as "○" if the image user has copyright-related rights so that it can be used, and "×" if the image user does not have copyright-related rights. . In addition, even if it is an image of "△" or "x", if it is an image that you want to use for machine learning, it is possible that the image user will acquire copyright-related rights or portrait-related rights and use it, so the narrowing conditions It is now possible to do so.

If the image user owns copyrights and portrait rights, there is no particular restriction on the purpose of using the image. If you are a licensor, the purpose of use may be limited by your contract with the licensor. In the input field for the purpose of use, such a limited purpose of use is input. The purposes of use include, for example, use on web pages, use in advertisements, use within the company, use outside the company, etc.

If the image user owns copyright and portrait rights, the image can be used indefinitely as long as the image user holds those rights, but if the image user is a licensee, the license In some cases, the end of the period of use, that is, the deadline for use, may be determined by the contract with the person. In the input field for the usage period, the period during which the image user can use the image is entered. For example, if you enter a deadline of "until the end of 2025," the images will be narrowed down to images that can be used at least until the end of 2025.

In the input field E14, for example, only "○" may be entered as information indicating whether or not the image can be used, multiple conditions such as "○" and "△" may be entered, or "all" may be entered. ”, a condition for not narrowing down images based on availability may be input. In this way, one or more conditions may be input as the image right condition, similar to the image attribute condition. Note that when no conditions are input, the conditions may not be narrowed down in the same way as when "all" is input.

The operator terminal 20 accepts input of conditions for narrowing down candidate images by the user (A13). The operator terminal 20 transmits the input narrowing conditions to the server device 10. The server device 10 reads out images that meet the transmitted narrowing-down conditions as candidate images from the images stored in the image database DB1 (A14).

FIG. 6 is a diagram showing an example of the image database DB1. The image database DB1 stores the image file name of each image, image attribute information, and contract information in association with each other. In addition to the file name of the image data, the image file name may include a path to a folder in which the image data is stored. As image attribute information, image size, resolution, subject, etc. are stored. The subjects of the image include people, scenery, and the like. In addition to these objects, objects may also include buildings, vehicles, animals, and the like.

The image contract information includes the image copyright holder, copyright licensee, copyright license conditions, portrait right owner, portrait right licensee, portrait right license conditions, etc. . Note that portrait rights do not occur if the subject is not a person, so if the subject is a landscape, portrait rights, etc. are set to "NA" (not applicable). The permission conditions include the purpose of use of the image and the expiration date for use of the image described in FIG. 5.

In addition to these conditions, the licensing conditions include conditions that clearly define the scope of copyrighted works and portraits in images, restrictions on alteration of copyrighted works and portraits in images, and usage fees (copyright owners and portrait rights). (including remuneration or royalties to the person), period of use (including not only the period of use but also the start of use), obligation to display the copyright holder or portrait right holder, obligation to protect the privacy and reputation of the portrait right holder, or use. Conditions such as possible areas may also be included. Further, the contract information may include information such as the contract holder, contract date, contract number, and image data of the contract. The server device 10 reads out images whose attribute information and contract information satisfy the narrowing conditions, and transmits them to the operator terminal 20. The operator terminal 20 displays the transmitted image as a candidate image (A15).

FIG. 7 is a diagram showing an example of displayed candidate images. In FIG. 7, the data set creation screen G1 shown in FIG. 5 is shown. In the example of FIG. 7, "Company C1" is input as the image user, and "DSv1.1 for Company C1" is input as the dataset name. The operator terminal 20 displays the transmitted candidate images in the image display field E15 shown in FIG. 5, and displays check boxes E16 in association with these candidate images. It is assumed that only a portion of the candidate images are displayed in the display field E15, and other candidate images are also displayed by scrolling or other operations.

The operator checks the checkbox E16 of the candidate image to be included in the data set. Note that when the select all button B11 is operated, all the check boxes E16 are checked, and when the deselection button B12 is operated, all the check boxes E16 are unchecked. Every time the operator changes the conditions input in the image attribute condition input field E13 and the image right condition input field E14, the operations from A13 to A15 are performed to update the displayed candidate images.

The operator, for example, roughly narrows down the candidate images using the narrowing conditions, checks all checkboxes E16, and then selects candidate images to be included in the dataset by canceling the checkboxes E16 of unnecessary candidate images. do. Note that there may be a function to display candidate images that do not satisfy the narrowing down conditions in a separate display area so that candidate images that do not satisfy the narrowing down conditions can be added. Once the candidate images to be included in the data set are determined, the operator operates the determination button B13.

The operator terminal 20 accepts the input operation of the narrowing condition, the check operation, and the operation on the decision button B13 as the selection operation and decision operation of candidate images (A16), and sends selection result data indicating the selection result to the server device 10. . The server device 10 determines the candidate image indicated by the transmitted selection result data as a DS image, and generates information regarding a dataset including the determined DS image as dataset information (A17). Then, the server device 10 stores and saves the generated data set information in the data set database DB2 shown in FIG. 1 (A18).

FIG. 8 is a diagram showing an example of generated data set information. In the dataset database DB2 shown in FIG. 8, narrowing conditions, image information, and rights information are stored as dataset information in association with dataset names. The narrowing-down conditions store the conditions that have been input in the image attribute condition input field E13 and the image right condition input field E14 on the data set creation screen G1. Although not all candidate images narrowed down by the narrowing down conditions are included in the dataset, the narrowing down conditions are stored as reference information representing the characteristics of the images included in the dataset.

The image information includes information such as an image file name and a subject. The image file name indicates the file names (which may also include path names) of all DS images included in the data set. The server device 10 can read the DS image included in the data set by referring to this image file name.

For the subject, the percentage of the subject appearing in all DS images included in the data set is shown. For example, in the case of the data set "DSv1.1 for Company C1", the proportions of subjects are shown as 80% people and 10% landscapes. The server device 10 calculates this ratio by counting the number of subjects in all DS images stored in the image database DB1. In addition to the subject, the image information may also store information such as the number of images, image size, and resolution, for example. In short, it is sufficient that image information includes information representing the characteristics of those DS images so that it is easy to get an idea of what kind of DS images are included in the data set.

The rights information includes the image user ("user" in the diagram), permission information, copyright ownership, copyright permission, conditions for that permission, ownership of portrait rights, permission for portrait rights, and conditions for that permission, etc. . The image user is a user of the DS image indicated by the right information.

The availability information is information indicating whether or not all DS images included in the data set can be used. As mentioned above, if the image user has both copyright-related rights and portrait-related rights (if the subject does not include a person, only copyright-related rights), the availability of DS images is determined by ``○''; If you have only one of them, it will be shown as "△", and if you don't have both of them (if the subject does not include a person, only copyright-related rights), it will be shown as "×". The availability information is indicated as "○" if all DS images are "○", "△" if there is even one "△", and "×" if there is even one "×". shall be.

"Copyright Ownership" indicates the percentage of images for which the image user owns the copyright among all DS images. "Copyright permission" indicates the percentage of images for which the image user has copyright permission among all DS images. The "license condition" of copyright indicates the percentage of images that share the same license condition among all DS images. In the example of FIG. 8, the percentage of images that have a common usage period among the permission conditions is shown. In the case of the data set "DSv1.1 for company C1", 50% is "indefinitely", 25% is "~20xx", and 10% is "~20yy".

“Portrait rights ownership” indicates the percentage of images for which the image user owns portrait rights among all DS images in which the subject is a person. "Portrait Rights Permission" indicates the percentage of images for which portrait rights have been granted to image users among all DS images in which the subject is a person. The "permission conditions" for portrait rights indicates the percentage of images that share the same permission conditions among all DS images in which the subject is a person. In the example of FIG. 8, the percentage of images that have a common usage period among the permission conditions is shown. In the case of the data set "DSv1.1 for company C1", 40% are "indefinitely", 30% are "~20xx", and 15% are "~20yy".

The server device 10 refers to the contract information of each DS image stored in the image database DB1, calculates the above-mentioned rights information, and stores it in the dataset database DB2. Note that the rights information shown in FIG. 8 is an example, and the information is not limited to this information. The rights information may include, for example, license conditions other than the expiration date, or may include only license conditions with the highest proportion of common images. In short, it is only necessary that information representing the characteristics of the rights of the DS images included in the data set is included as the rights information so that it is easy to get an idea of the status of the rights of the DS images included in the data set.

Next, a learning process for performing machine learning using the data set created by the data set creation process described above will be described.
FIG. 9 is an activity diagram showing an example of learning processing. The learning process shown in FIG. 9 is started when the user of the operator terminal 20 performs an operation to display a machine learning screen. First, the server device 10 generates screen data indicating a machine learning screen, and transmits the generated screen data to the operator terminal 20 (A21). The operator terminal 20 displays the machine learning screen indicated by the transmitted screen data (A22).

FIG. 10 is a diagram showing an example of a displayed machine learning screen. The machine learning screen G2 shown in FIG. A display field E23 for a list of and a machine learning execution button B21 are displayed. In the input field E21, a person who uses the AI image is input as an image user. In the example of FIG. 10, "Company C1" is input as the image user.

When the operator terminal 20 receives the input from the image user (A23), the operator terminal 20 transmits user data indicating the input image user to the server device 10. When the user data is transmitted, the server device 10 refers to the dataset database DB2 and reads out the dataset information associated with the image user indicated by the user data (A24). The server device 10 transmits the read data set information to the operator terminal 20. The operator terminal 20 displays the transmitted data set information in the data set list display column E23 (A25).

In the example of FIG. 10, the operator terminal 20 displays the dataset name, availability ("availability" in the diagram), subject, copyright permission conditions, and portrait rights permission conditions as dataset information. . In the example of FIG. 10, "DSv1.1 for company C1", "DSv1.2 for company C1", "DSv1.3 for company C1", "DSv2.1 for company C1", and "DSv2.2 for company C1" Data set information of a data set created with company C1 as the image user is displayed.

Note that the data set information displayed in the display field E23 is not limited to that shown in FIG. 10. For example, the number of images, the resolution, the ratio between copyright and its permission, or the ratio between portrait right and its permission, etc. may be displayed in the display column E23. In short, as a criterion when selecting a dataset to be used for machine learning, information representing the characteristics of the DS image and the characteristics of the rights included in the dataset may be displayed as dataset information.

The operator selects a dataset to use for machine learning by referring to the displayed dataset information. In the example of FIG. 10, the operator has selected "DSv1.3 for company C1". Note that in the example of FIG. 10, only one data set is selected, but two or more data sets may be selected. The operator inputs the name of the learning model generated by machine learning. In the example of FIG. 10, the operator inputs "model for company C1 v1.3". The operator causes the image generation AI module 100 to execute machine learning on the selected data set by operating the machine learning execution button B21.

The operator terminal 20 accepts the above operations as an operation for selecting a dataset (A26) and an operation for instructing machine learning (A27), and indicates the selected dataset and the name of the input learning model. The instruction data is transmitted to the server device 10. The server device 10 refers to the file name of the DS image included in the data set indicated by the transmitted instruction data in the data set database DB2, and reads out the image with the referenced file name as a data set (A31).

Next, the server device 10 instructs the image generation AI module 100 to perform machine learning using the read data set (A32). The image generation AI module 100 executes machine learning using the instructed data set (A33). As a result of performing machine learning, the image generation AI module 100 generates a learning model that can output an image similar to a DS image included in the dataset. The server device 10 stores the generated learning model in the learning model database DB3 in association with learning model information regarding the learning model (A34).

FIG. 11 is a diagram showing an example of stored learning model information. The learning model database DB3 shown in FIG. 11 stores learning model information in association with learning models. The learning model information includes a dataset name, image information, and rights information. The dataset name indicates the name of the dataset used for machine learning when generating the learning model. For example, "model v1.1 for company C1" is associated with "DSv1.1 for company C1". Furthermore, the “model v1.2 for company C1” is associated with two data sets, “DSv1.1 for company C1” and “DSv2.1 for company C1”.

In the image information and rights information, image information and rights information stored in association with the corresponding data set in the data set database DB2 are read out and stored. For example, "model v1.1 for company C1" includes image information including the proportion of the subject such as "person: 80%", image user "company C1", availability of use "○", and "unlimited period". Rights information including copyright permission conditions such as ``: 50%'' and portrait rights permission conditions such as ``unlimited: 40%'' are stored in association with each other.

The server device 10 refers to the image information and rights information stored in the dataset database DB2, reads out information associated with the dataset used for machine learning, and stores it in the learning model database DB3. In addition, for a learning model that has two or more datasets used for machine learning, such as "model for company C1 v1.2," the server device 10 supports images included in those datasets in the image database DB1. Referring to the attached image information and contract information, the proportion of the subject and the proportion of permission conditions, etc. are recalculated and stored as image information and rights information.

The learning model information shown in FIG. 11 is an example, and the information is not limited to this information. The learning model information may include image information such as the number of images or resolution, and may include only license conditions other than the expiration date or license conditions with the highest proportion of common images as rights information. It's okay. In short, the characteristics of the DS images included in the dataset used for machine learning of the learning model and the rights status of those DS images are displayed so that it is easy to get an idea of the rights status of the DS images. It is only necessary that the information to be learned is included as the learning model information.

Next, an image generation process for generating an AI image using the learning model generated by the learning process described above will be described.
FIG. 12 is an activity diagram showing an example of image generation processing. The image generation process shown in FIG. 12 is started when the user of the operator terminal 20 performs an operation to display an image generation screen. First, the server device 10 generates screen data indicating an image generation screen, and transmits the generated screen data to the operator terminal 20 (A41). The operator terminal 20 displays the image generation screen indicated by the transmitted screen data (A42).

FIG. 13 is a diagram showing an example of a displayed image generation screen. On the image generation screen G3 shown in FIG. 13, there is a character string "Please select a learning model to be used for image generation.", an input field E31 for image users, a display field E32 for a list of learning models, and a prompt. An input field E33 and an image generation execution button B31 are displayed. In the input field E31, a person who uses the AI image is input as an image user. In the example of FIG. 13, "Company C1" is input as the image user.

When the operator terminal 20 receives the input from the image user (A43), the operator terminal 20 transmits user data indicating the input image user to the server device 10. When the user data is transmitted, the server device 10 reads learning model information associated with the image user indicated by the user data (A44). The server device 10 transmits the read learning model information to the operator terminal 20. The operator terminal 20 displays the transmitted learning model information in the learning model list display column E32 (A45).

In the example of FIG. 13, the operator terminal 20 displays the learning model name, availability ("availability" in the diagram), subject, copyright permission conditions, and portrait rights permission conditions as learning model information. . In the example of FIG. 13, "Model for company C1 v1.1", "Model for company C1 v1.2", "Model for company C1 v1.3", "Model for company C1 v2.1", and "Model for company C1 v1.3" Learning model information of a learning model called "Model v2.2" generated by machine learning executed with company C1 as the image user is displayed.

Note that the learning model information displayed in the display field E32 is not limited to that shown in FIG. 13. For example, the number of images, their resolution, the ratio between copyright and their permission, or the ratio between portrait rights and their permission, etc. may be displayed in the display column E32. In short, information representing the characteristics of the DS image and the characteristics of rights included in the dataset used for machine learning of the learning model is used as learning model information as a basis for selecting a learning model to be used for AI image generation. It is fine as long as it is displayed.

The operator refers to the displayed learning model information and selects a learning model to be used for generating the AI image. In the example of FIG. 13, the operator has selected "Model v1.3 for company C1". The operator inputs a text (so-called prompt) instructing the AI image to be generated in the prompt input field E33. Then, by operating the image generation execution button B31, the operator causes the image generation AI module 100 to generate an AI image using the selected learning model using the input prompt.

The operator terminal 20 accepts the above operations as an operation for selecting a learning model (A46), an operation for inputting a prompt (A47), and an operation for instructing generation of an AI image (A48), and selects the selected learning model. and the input prompt are transmitted to the server device 10. The server device 10 executes an AI image generation process based on the transmitted instruction data (A51). Specifically, the server device 10 executes a process of instructing the image generation AI module 100 to generate an AI image using the learning model and prompt indicated by the instruction data. The image generation AI module 100 generates an AI image based on this instruction.

Next, the server device 10 reads learning model information associated with the learning model used to generate the AI image in the learning model database DB3 (A52). Subsequently, the server device 10 generates metadata of the generated AI image based on the read learning model information (A53). Then, the server device 10 generates screen data indicating an AI image screen that displays the generated AI image and metadata, and transmits the generated screen data to the operator terminal 20 (A54). The operator terminal 20 displays the AI image screen indicated by the transmitted screen data (A55).

FIG. 14 is a diagram showing an example of the displayed AI image screen. On the AI image screen G4 shown in FIG. 14, there is a character string "AI image has been completed.", an AI image display field E41, a metadata display field E42, a back button B41, and an image regeneration button. B42 is displayed. The generated AI image F41 is displayed in the display field E41. The generated metadata M41 is displayed in the display column E41.

In the example of FIG. 14, the server device 10 stores the "image ID", "learning model name", "model safety", "dataset name", the link L41 to the DS image, and the contract information. The metadata M41 including the link L42 is generated. “Image ID” indicates identification information of the AI image issued by the image generation AI module 100 when generating the AI image F41. “Learning model name” indicates the name of the learning model used to generate the AI image F41. “Dataset name” indicates the name of the dataset used for machine learning of the learning model used to generate the AI image F41.

"Model Safety" indicates the legal safety of the learning model used to generate the AI image F41. The server device 10 indicates "usability" included in the learning model information of the learning model used to generate the AI image F41 as "model safety." In other words, if the server device 10 has both copyright-related rights and portrait-related rights for all DS images included in the dataset used for machine learning of the learning model used to generate the AI image F41, indicates "○", if there is at least one image missing one of them, indicate "△", and if there is at least one image missing both of them, indicate "×". There is.

When the return button B41 is operated, the server device 10 returns to the previous screen, the image generation screen G3 shown in FIG. 13, and displays the screen immediately before the image generation execution button B31 was operated. . The operator can select the learning model again on the displayed image generation screen G3. When the image regeneration button B42 is operated, the server device 10 executes the process from A51 onwards again without changing the learning model etc., and displays the regenerated AI image and metadata on the operator terminal 20. let

The link L41 to the DS image indicates a link to an image included in the data set indicated by "data set name." When the link L41 is operated, the operator terminal 20 accepts the operation and sends the name of the dataset linked by the operated link L41 and instruction data for instructing the display of the DS image to the server device 10. (A61). The server device 10 reads the DS images included in the data set with the name indicated by the transmitted instruction data from the image database DB1 (A62), and generates a DS image screen showing a list of the read DS images (A63). The server device 10 generates screen data indicating the generated DS image screen, and transmits the generated screen data to the operator terminal 20. The operator terminal 20 displays the DS image screen indicated by the transmitted screen data (A64).

FIG. 15 is a diagram showing an example of a displayed DS image screen. On the DS image screen G5 shown in FIG. 15, there is a character string "This is an image included in a dataset used for machine learning.", an image user display field E51, a dataset name display field E52, and an image. A display field E53 for attribute conditions, a display field E54 for image rights conditions, and a display field E55 for DS images are displayed. The server device 10 refers to the image database DB1, the dataset database DB2, and the learning model database DB3 to control the display of these display columns.

The server device 10 displays the name of the image user who will use the generated AI image in the display field E51, and displays the name of the dataset used for machine learning in the display field E52. In addition, the server device 10 displays the narrowing-down conditions (attribute conditions and rights conditions) that are satisfied by the DS images used for machine learning in the display columns E53 and E54, and displays the narrowing-down conditions (attribute conditions and rights conditions) that are satisfied by the DS images used for machine learning in the display column E55. A DS image is being displayed. It is assumed that the DS image displayed in the display field E55 is only a part, and other DS images are also displayed by scrolling or the like.

FIG. 15 shows an example of a display of a DS image when candidate images are selected after being narrowed down using the narrowing conditions as described in FIG. 5 and the like. In the example of FIG. 15, by displaying the narrowing down conditions, it is possible to understand what common features the DS images have. Note that if the selection of DS images is performed by specifying a file name, path name, or folder name, those file names, etc. may be displayed instead of the narrowing down conditions.

Link L42 to contract information indicates a link to contract information of an image included in the data set indicated by "data set name." When the link L42 is operated, the operator terminal 20 accepts the operation and sends the name of the data set linked by the operated link L42 and instruction data instructing to display the contract information to the server device 10. (A71). The server device 10 reads the contract information of the DS image included in the data set with the name indicated by the transmitted instruction data from the image database DB1 (A72), and generates a contract information screen showing a list of the read contract information (A72). A73). The server device 10 generates screen data indicating the generated contract information screen, and transmits the generated screen data to the operator terminal 20. The operator terminal 20 displays the contract information screen indicated by the transmitted screen data (A74).

FIG. 16 is a diagram showing an example of a displayed contract information screen. On the contract information screen G6 shown in FIG. 16, there is a character string "This is contract information for images included in the dataset used for machine learning.", an image user display field E61, and a data set name display field E62. and a display field E63 for a list of contract information for DS images are displayed. The server device 10 refers to the image database DB1, the dataset database DB2, and the learning model database DB3 to control the display of these display columns.

The server device 10 causes the display field E61 to display the name of the image user who will use the generated AI image, and the display field E62 to display the name of the dataset used for machine learning. Furthermore, the server device 10 displays the file names of the DS images used for machine learning and the contract information of those DS images in association with each other in the display column E63. The contract information includes the copyright holder of the image, the copyright licensee, the copyright license conditions, the portrait right owner, the portrait right licensee, the portrait right license conditions, and the like. Portrait-related rights for DS images where the subject is not a person are indicated as "NA" (not applicable).

As described above, in the image generation system 1, the server device 10 functions as an example of an information acquisition unit that acquires related information related to a group of images used for machine learning. In other words, the image group used for machine learning is a group of images included in a dataset, that is, a plurality of DS images. The server device 10 acquires learning model information as related information, for example, in A52 shown in FIG. 12. The learning model information, as shown in FIG. 11, is information indicating image information and rights information of a plurality of DS images. In this way, the related information (learning model information) acquired by the server device 10 includes rights information regarding the image group used for machine learning.

The rights information is information indicating the right to use images (DS images) included in the image group used for machine learning. In the example of FIG. 11, the rights information includes the image user who has the right to use the DS image, whether or not the DS image can be used, copyright permission conditions, portrait right permission conditions, and the like. The server device 10 then associates the acquired images with AI images generated by an artificial intelligence module (the image generation AI module 100 is an example) that performs machine learning using these image groups. It functions as an example of an output unit that outputs metadata indicating related information.

In the example of FIG. 14, the server device 10 outputs the metadata M41 in association with the AI image F41 to the operator terminal 20 for display. The metadata M41 includes information about the DS image used for machine learning by the image generation AI module 100 that generated the AI image F41, that is, the DS image included in the dataset "DSv1.3 for company C1". The right information indicates that the "gender" (the same information as the availability of the DS image) is "○".

According to this aspect, compared to the case where metadata as shown in FIG. 14 is not output, the image group used for machine learning, that is, the plurality of images used by the artificial intelligence module that generated the AI image Information about the DS image can be easily grasped. In the example of FIG. 14, it is possible to easily grasp the rights information regarding the plurality of DS images used for machine learning of the artificial intelligence module in generating the AI image F41.

If machine learning is performed using a group of images including DS images that the user of the AI image does not have the right to use, that is, a group of images with defective rights, there is an intention to infringe on the rights of others. or may be judged to have failed to take precautions to avoid infringing on the rights of others. On the other hand, if machine learning is performed using a group of images that includes only DS images that the user of the AI image has the right to use, that is, a group of images that have no copyright defects, there is no intention to infringe on the rights of others. It is easy to see that they are taking sufficient care not to infringe on the rights of others.

By indicating the rights information in the metadata, the artificial intelligence module that generated the AI image corresponding to the metadata can perform machine learning using either the image group with the defective rights or the group of images without the defective rights. Since the user of the AI image can understand the security of the AI image in terms of rights.

Further, the server device 10 functions as an example of a selection reception unit that receives selections of images (DS images) to be included in the image group used for machine learning. In the examples of FIGS. 5 and 7, the server device 10 accepts the selection of a DS image by displaying the data set creation screen G1. As shown in FIG. 6, the selected DS images are associated with contract information representing the content of the contract that defines the right to use those DS images.

The server device 10 functions as an example of a generation unit that generates rights information for an image group based on contract information associated with a DS image included in the image group by the above selection. For example, in the example shown in FIG. 8, the server device 10 uses contract information indicating the copyright-related rights and portrait-related rights associated with the DS images to store DS images that have the same usability and licensing conditions. It generates rights information called percentage. According to this aspect, compared to a case where a user manually prepares similar rights information, it is possible to reduce the effort required to prepare rights information.

Further, the server device 10 functions as an example of an output unit that outputs metadata in a displayable manner to a user terminal. The operator terminal 20 is an example of a user terminal, and in the example of FIG. 14 displays metadata M41. Then, when an operation for displaying a group of images used for machine learning is performed on the user terminal on which the metadata is displayed, the server device 10 displays images included in the group of images whose related information is indicated by the metadata. functions as an example of an image display unit that displays on a user terminal. According to such an aspect, it is possible to easily verify the actual image that has been subjected to machine learning, compared to the case where the display of the DS image is not controlled by the above-described display operation.

Further, when an operation for displaying details of rights is performed on a user terminal on which metadata is displayed, the server device 10 causes the user terminal to display contract information that is the source of the rights information indicated by the metadata. It functions as an example of a contract display section. In the example of FIG. 14, the server device 10 performs the following operations when the operation to the link L42 to the contract information included in the metadata M41 is performed as an operation to display the details of the right to use the image group used for machine learning. As shown in FIG. 16, the contract information of the DS image whose rights information is indicated by the metadata M41 is displayed on the operator terminal 20. According to this aspect, it is possible to more easily verify the contents of the contract indicating the rights to the image than in the case where the display of the contract information is not controlled by the display operation described above.

<Variation example: Rights information>
The rights information included in the metadata is not limited to the information shown in FIG. 14 (model safety=usability).
FIG. 17 is a diagram showing an example of displayed metadata. In the example of FIG. 17, the server device 10 displays metadata M42 in the metadata display column E42 shown in FIG. In addition to the rights information (model safety) shown in FIG. 14, the metadata M42 includes "authorship-related rights" and "portrait-related rights."

Specifically, the copyright-related rights include "right of use" of "copyright = 4%" and "copyright permission = 96%" and "unlimited: 45%, ~20xx: 35%, ~20yy" It includes a "usage period" such as ": 5%, ..." and a "purpose of use" such as "web page, advertisement, internal material, external material, ...". In addition, as portrait-related rights, there are "rights of use" such as "portrait rights = 2%" and "portrait rights permission = 98%", and "~20xx: 30%, ~20yy: 25%, indefinitely: 20%" ,...'' and the "purpose of use" such as "web page, advertisement, internal material, external material,..." are included.

In this way, the rights information may include information indicating the conditions under which the right to use the DS image is valid. Conditions under which the right to use a DS image is valid include conditions such as the usage period and purpose of use shown in FIG. 17, the area where the DS image can be used, license renewal conditions, or license fee. By including these conditions in the metadata, it is possible to alert users of AI images to ensure that their rights to use DS images are not invalidated.

For example, if a user of an AI image is alerted to the fact that it contains a DS image with a usable period of up to 20xx, the user may decide whether to use the generated AI image until 20xx or not. It is possible to take measures such as extending the usable period of the corresponding DS image. Additionally, the user of the AI image can be careful not to use the generated AI image for purposes other than the permitted purpose of use.

Further, the rights information may include information indicating the amount of DS images that share a common condition under which the right to use the DS images is valid. In the example in FIG. 17, the rights information includes the percentage of DS images that have a common usage period, such as "Unlimited: 45%, ~20xx: 35%, ~20yy: 5%, ..." . Note that the information indicating the amount of DS images is not limited to the ratio, and may be expressed by the number of images, data size, or the like. According to such an aspect, it is possible to grasp the degree of influence when it is assumed that the right to use some DS images is invalidated.

Furthermore, when a condition for validating the right to use a DS image is likely to no longer be satisfied, the server device 10 may display rights information indicating the condition in a manner different from other rights information. For example, if there is a DS image whose expiration date is less than one year, the server device 10 displays the expiration date in bold red text, or displays a message such as "One year has passed until the expiration date!" A character string may be added and displayed to encourage the user to pay special attention. According to this aspect, compared to a case where only uniform display is performed, it is possible to make it less likely that the right to use the DS image will be invalidated.

Further, the rights information may include information about the type of rights to use the DS image. The types of rights to use DS images are, for example, "authorship-related rights" and "portrait-related rights" shown in FIG. 17. A copyright-related right is an example of a right related to a person who created a DS image, and a portrait-related right is an example of a right related to a person represented in a DS image. Note that it is sufficient that at least one of these rights is included.

Rights related to the person who created the DS image are not limited to copyright, and may include, for example, design rights or trademark rights. Furthermore, the rights related to the person represented in the DS image may include publicity rights, privacy rights, and the like. Regardless of which of these rights is included in the rights information, the safety regarding the rights can be evaluated in more detail than in the case where, for example, only the permission to use the DS image is included in the rights information.

Further, the rights information may include information indicating the amount of DS images that have a common type of right to use the DS images. In the example of FIG. 17, the rights information includes the percentage of DS images that have the same type of rights, such as "copyright = 4%" and "copyright permission = 96%." Note that the information indicating the amount of DS images is not limited to the ratio, and may be expressed by the number of images, data size, or the like. According to this aspect, the balance between the types of rights is understood, and if necessary, measures are taken to adjust the balance (for example, measures to lower the percentage of permissions and increase the percentage of copyright ownership). be able to.

Furthermore, if the subject of the DS image is a person, the portions protected by the granted portrait rights may be limited. For example, if the DS image is a close-up photo of the face, the AI image showing the face close-up is likely to be protected by the rights granted for the portrait rights of the DS image, but the AI image showing the whole body is likely to be protected. There is a high possibility that the protection provided by that right will not extend. On the other hand, if the DS image is a full-body photo, there is a high possibility that the AI image showing the whole body will be protected by the rights granted for the portrait rights of the DS image, but the AI image showing the close-up of the face will be protected. There is a high possibility that such rights will not be protected.

Therefore, the rights information may include information indicating the range of objects that are covered by the rights among the objects shown in the images included in the image group. The "right" referred to here refers to the right to use the DS image, and is, for example, the right granted by the owner of portrait rights. In this case, in the image database DB1, information indicating the range covered by the rights (hereinafter referred to as "range information") is stored in association with each image. If the subject in the DS image is a person, "face", "whole body", "upper body", "sideways figure", etc. are stored as range information.

When DS images to be included in the data set are selected, the server device 10 refers to the image database DB1, reads range information associated with those DS images, and creates rights information based on the read range information. generate. For example, the server device 10 generates, as rights information, information indicating the amount of DS images that have a common range of rights. Then, when the server device 10 generates the AI image, it outputs metadata including rights information based on the range information.

FIG. 18 is a diagram showing an example of displayed metadata. In the example of FIG. 18, the server device 10 displays metadata M43 in the metadata display field E42 shown in FIG. In addition to the rights information (model safety) shown in Figure 14, the metadata M43 includes, as "portrait-related rights", "Range covered by rights: face = 65%, upper body = 30%, ..." It is.

In this case, the user of the AI image only has to decide the composition of the AI image based on the rights information indicated by the metadata. For example, in the example shown in FIG. 18, the range indicated by the rights information, that is, the range of the rights granted for the DS image is almost limited to the upper part of the body, so when an AI image representing a person is generated. In this case, the user may be prompted to generate an AI image with a composition that greatly represents the upper part of the body. According to this aspect, it is possible to generate an AI image with a composition that has little risk in terms of rights. Furthermore, by including in the metadata rights information indicating the amount of DS images that have a common range of rights, it is possible to intuitively understand a composition with less rights risk.

In addition to the above, the rights information may also include information on items such as the holder of the right to use the DS image, the owner of copyright-related rights and portrait-related rights, and usage fees. Metadata allows users of AI images to understand the status of rights to DS images that are the source of AI images, judge rights risks, and take measures to avoid such risks. It is desirable that the rights information includes information on items that can be used as reference in case of a case.

Furthermore, the rights information may include a plurality of items, as in the examples of FIGS. 17 and 18. Multiple items include the presence or absence of rights (one example is "model safety"), the conditions under which the rights are valid (an example is "usage period"), and the type of rights ("author-related rights" and "portrait rights"). It is desirable to include two or more of the following items: the scope of the right (an example is "related rights"), the scope of the right, the type of title to the right (an example is "right to use"), or the name of the right. According to such an aspect, the rights safety of the generated AI image can be evaluated from multiple perspectives.

<Modified example: Selection of rights information>
The rights information included in the metadata may be selectable. In that case, the server device 10 displays a selection screen for selecting rights information. Since it is conceivable that both the operator and the user of the AI image may select the right information item, the selection screen may be displayed on either the operator terminal 20 or the user terminal 30. good.

FIG. 19 is a diagram showing an example of a screen for selecting displayed rights information. On the right information selection screen G7 shown in FIG. 19, a check field E71 for the right information item to be selected, a metadata image E72, and a decision button B71 are displayed. In the check column E71, rights information such as "existence of right", "type of right", "name of right", "limit of use", "area of use", "usage fee", and "renewal conditions" is provided. It includes items and their check boxes (“○” indicates that they are checked).

Image E72 shows an image of metadata when the items checked in check column E71 are included. In the example of FIG. 19, "Presence of rights", "Type of rights", "Usage period", and "Usage fee" are checked, and in image E72, "Presence of rights" is "Model security". The "types of rights" are represented by the names "work-related rights" and "portrait-related rights." Furthermore, the “usage period” and “usage fee” are expressed with the same names as the items.

The user (operator or AI image user) confirms the image E72 while checking the items in the check field E71, and when the items to be included in the metadata are determined, operates the enter button B71. In this way, the server device 10 functions as an example of a selection display unit that displays on a user terminal a selection image that accepts a selection operation of an item to be included in metadata among a plurality of items included in the rights information. The operation of checking the check field E71 and the operation of the enter button B71 are examples of selection operations, and the selection screen G7 is an example of a selection image. Then, the server device 10 (an example of an output unit) outputs data indicating the item selected by the selection operation as metadata. According to this aspect, the user can include desired items in the metadata.

<Modification example: Change of DS image>
The DS images included in the dataset may be modified after the dataset is created. In that case, in A11 and A12 of the data set creation process shown in FIG. 4, the server device 10 displays a change screen for changing the DS image instead of the data set creation screen. Since the DS image may be changed by both the operator and the user of the AI image, the change screen may be displayed on either the operator terminal 20 or the user terminal 30.

FIG. 20 is a diagram showing an example of a displayed DS image change screen. On the DS image change screen G8 shown in FIG. 20, there is a character string "Please reselect the images to be included in the dataset.", an input field E81 for the image user, and an input field E82 for the dataset name. DS image display field E83, check box E84, image attribute condition input field E85, image right condition input field E86, candidate image display field E87, check box E88, and dataset name after change An input field E89 and a decision button B81 are displayed.

When the user inputs an image user in the input field E81, the name of the dataset associated with the image user can be input in the input field E82, and the dataset name is input in the input field E82. Then, the DS images included in the data set are displayed in the display field E83. The displayed DS image has a corresponding check box E84 checked, but by removing the check, it can be removed from the data set.

Conditions for narrowing down candidate images to be added to the data set are input into the input field E85 for image attribute conditions and the input field E86 for image rights conditions. When the narrowing down conditions are input, the operations A13 to A15 shown in FIG. 4 are performed, and candidate images that satisfy the narrowing down conditions input by the user are displayed in the display field E87. Among the displayed candidate images, candidate images whose corresponding checkboxes E88 are checked are added to the data set.

The data set name after the DS image has been changed is input into the input field E89. When the enter button B81 is operated in this state, the operations A16 to A18 shown in FIG. 4 are performed, and data set information including the data set name input in the input field E89 is newly generated and saved. Ru. At this time, rights information regarding the newly generated data set is stored in the data set database DB2 shown in FIG.

In this way, the server device 10 functions as an example of a change reception unit that accepts changes to DS images to be included in the image group of the data set, based on the user's operation on the change screen G8, for example. Then, the server device 10 (an example of a generation unit) generates the rights information of the image group based on the contract information associated with the DS image included in the image group of the data set after the change is accepted. generate. According to this aspect, even after changing the data set, the effort required to prepare rights information can be reduced compared to when a user manually prepares similar rights information.

Note that the server device 10 may automatically generate the changed data set name and display it in the input field E89. For example, the server device 10 refers to the input field E82, and if the input data set name includes "v(N1).(N2)" (N1 and N2 are natural numbers), "v(N1). .(N2+1)'' is generated and displayed in the input field E89. For example, in the example of FIG. 20, the server device 10 generates a changed data set name of "DSv3.2 for company C1" based on the data set name of "DSv3.1 for company C1". Display it in the input field E89.

In addition, if the content of the change in the DS image satisfies the condition indicating a major change, the server device 10 generates a dataset name (v(N1+1), (N2)) that has been majorly upgraded, and updates the condition. If this is not the case, a dataset name (v(N1), (N2+1)) with a minor version upgrade may be generated. The condition indicating a significant change is, for example, a condition that is satisfied when a predetermined percentage (for example, 30% or 50%) or more of the DS images included in the data set changes. According to such an aspect, it is possible to save the effort of inputting the data set name after the change.

<Modified example: component>
An artificial intelligence module (eg, image generation AI module 100) may have multiple components. A component is a part that forms part of an artificial intelligence module. The artificial intelligence module generates one AI image by operating multiple components. These multiple components each perform machine learning using the same or different image groups.

When the artificial intelligence module has the plurality of components described above, the server device 10 stores learning model information different from the example of FIG. 11 in the learning model database DB3.
FIG. 21 is a diagram showing another example of stored learning model information. The learning model database DB3 shown in FIG. 21 stores learning models such as "model ML1,""modelML2," and "model ML3."

Each learning model is associated with three components, "CMP1," "CMP2," and "CMP3," and the data set name used for machine learning of each component. In the example of FIG. 21, datasets DS1, DS2, and DS3 are used for machine learning. For example, in the case of "model ML1", DS1 is associated with CMP1, DS1 and DS2 are associated with CMP2, and DS3 is associated with CMP3.

In addition, in the case of "model ML2", DS2 is associated with CMP1, DS3 is associated with CMP2, and DS2 is associated with CMP3, and in the case of "model ML3", DS1 and DS2 are associated with CMP1 and CMP2, and DS2 is associated with CMP3 is associated with DS2 and DS3. Furthermore, each component is associated with image information and rights information according to the dataset used for machine learning. These image information and rights information are information read out from the data set database DB2 or recalculated as in the example of FIG. 11.

When the generation of an AI image is instructed, the server device 10 (an example of an output unit) generates a plurality of components that are associated with the AI image generated by the artificial intelligence module, each acquired for a group of images used for machine learning. Output data indicating related information as metadata. For example, the server device 10 outputs metadata indicating, as related information, rights information about a group of images included in a data set used for machine learning, similar to the example of FIG. 14 and the like.

FIG. 22 is a diagram showing another example of displayed metadata. In the example of FIG. 22, the server device 10 displays metadata M44 in the metadata display field E42 shown in FIG. The metadata M44 includes the learning model name "Model ML1", the component names "CMP1", "CMP2" and "CMP3", and the dataset names "DS1", "DS1+DS2" and "DS3", and all The safety of the model "○" is included in correspondence with each other.

In the example shown in Figure 22, the image user can easily understand the learning content (data set) of each component compared to the case where no metadata is output, and each component is secure in terms of rights. It is easy to understand that machine learning is performed using DS images.

<Variation example: User-provided image>
The images included in the data set are images prepared in advance for the image generation system 1, but the invention is not limited to this, and images provided by the user may be added. In that case, the server device 10 displays an addition screen for adding an image provided by the user (hereinafter referred to as "user image"). Since it is conceivable that both the operator and the user of the AI image may add a user image, the addition screen may be displayed on either the operator terminal 20 or the user terminal 30.

FIG. 23 is a diagram illustrating an example of a displayed user image addition screen. On the user image addition screen G9 shown in FIG. 23, there is a character string "Please enter the user image to be added and its provision conditions." An image user input field E92, an input field E93 for copyright-related provision conditions, an input field E94 for portrait rights-related provision conditions, and a decision button B91 are displayed.

In addition to the file name of the user image, a path name may also be input in the input field E91, or the file names of two or more user images may be input. In the input field E92, a user who will use an AI image generated when the added user image is a DS image is input. The provision conditions entered in the input fields E93 and E94 are provision conditions for this user.

In the input field E93, provision conditions such as "copyright holder", "copyright transfer", "license of use", "usage period", "purpose of use", and "usage fee" are input. The name, etc. of the copyright holder of the user image is entered in the "copyright holder" field. In "Copyright transfer", "○" is input when the copyright of the user image is transferred to the image user. In "Usage Permission", "○" is input when usage is permitted without transferring the copyright. Conditions for permitting use are entered in the "usage period", "purpose of use", "usage fee", etc.

In the input field E94, when portrait rights exist for the user image, provision conditions are input in the same way as in the input field E93. When each input field is filled in and the OK button B91 is operated, the server device 10 inquires of the image user whether he or she agrees to the input provision conditions. The server device 10 stores the user image in the image database DB1 when the image user answers that he or she agrees to the provision conditions. At this time, the server device 10 stores the provision conditions input on the addition screen G9 in the image database DB1 as contract information.

For example, the server device 10 stores the image user as the "copyright holder" of the image database DB1 if the copyright has been transferred, and stores the input copyright holder if the copyright has not been transferred. Note that the information on the image user, etc. that is stored is a corporate name or an individual name. The server device 10 stores "NA" (not applicable) as the "copyright licensee" if the copyright has been transferred, and stores the image user if the copyright has been granted.

The server device 10 stores provision conditions such as "usage period," "purpose of use," and "usage fee" as copyright "license conditions." Furthermore, if there are conditions for providing portrait rights, the server device 10 similarly stores "portrait right holder", "portrait rights licensee", "license conditions", etc. for portrait rights. The provision conditions thus stored in the image database DB1 are used as contract information in the data set creation process shown in FIG. 4, etc. As a result, this provision condition is included in the rights information stored in the dataset database DB2 in the dataset creation process.

As described above, the server device 10 functions as an example of an image acquisition unit that acquires a user image provided by a user. As described above, this user image is acquired based on the provision conditions agreed to by the user and the image user. Then, when the artificial intelligence module performs machine learning using a group of images including the acquired user image, the server device 10 acquires information including the conditions for providing the user image as rights information regarding the group of images. It functions as an example of an information acquisition unit. According to such an aspect, the user can include an image that he or she likes in the data set as a DS image, so that the tendency of the generated AI image can be tailored to the user's preference.

<Variation: Multiple rights holders>
There may be multiple holders of rights such as copyright and portrait rights for DS images. For example, not only the photographer who took the DS image, but also the copyright holder of the copyrighted work if the DS image includes a copyrighted work, may have rights to the DS image. Further, if a plurality of people are individually identified in the DS image, all of those people may have portrait rights.

In those cases, the image database DB1 shown in FIG. 6 stores contract information about a plurality of rights holders in association with one image. As a result, the server device 10 allows a right holder to be input as an image right condition on the data set creation screen G1 shown in FIG. You can narrow it down. Further, the server device 10 may also store data set information and learning model information indicating the rights holder in the data set database DB2 and the learning model database DB3, respectively.

By doing so, the server device 10, for example, in the display field E23 of the list of data sets of the machine learning screen G2 shown in FIG. 10 and the display field E32 of the list of learning models of the image generation screen G3 shown in FIG. Information about the rights holders of DS images (for example, information indicating that the copyright of DS images is shared between companies C1 and C2) can be displayed when selecting datasets and learning models. It can be used as a reference.

<Modified example: Relearning>
The image generation AI module 100 may perform relearning to perform machine learning again using the once generated learning model and dataset (either a new dataset or a trained dataset). In that case, the server device 10 stores the learning model used for relearning as a learning source model in the learning model database DB3.

FIG. 24 is a diagram showing another example of stored learning model information. The learning model database DB3a shown in FIG. 24 stores learning model information including learning source models. The learning source model indicates the name of the learning model used to generate the corresponding learning model. In the example of FIG. 24, the first generated "model v1.1 for company C1" is not re-trained, so only the dataset "DSv1 for company C1" is associated, and the learning source model is " "NA" (not applicable).

The "Model for Company C1 v1.1" is used as the learning source model, and the learning model generated by relearning using the data set "DSv2 for Company C1" is the "Model for Company C1 v1.2." . In addition, the learning model that was generated by relearning using the "Company C1 Model v1.2" as the learning source model and the "Company C1 DSv3" dataset is the "Company C1 Model v1.3." Yes, the learning model that was created by relearning using the "Company C1 Model v1.2" as the learning source model and the "Company C1 DSv4" dataset is the "Company C1 Model v2.1". be.

In addition, the learning model that is generated by relearning using the "Company C1 Model v2.1" as the learning source model and the "Company C1 DSv5" dataset is the "Company C1 Model v2.2." Yes, the learning model that is generated by relearning using the "Model for Company C1 v1.3" as the learning source model and the dataset "DSv4 for Company C1" is the "Model for Company C1 v3.1". be. In this way, according to the example of FIG. 24, the learning model that is the learning source of the learning model that has been relearned can be traced over many generations. Although the learning model information including the learning source model is managed in a table format in the example of FIG. 24, it may also be managed in a node format.

FIG. 25 is a diagram illustrating an example of learning model information managed in a node format. In the example of FIG. 25, the server device 10 stores nodes Nm1 to Nm6 indicating learning models and nodes Nd1 to Nd5 indicating data sets. Nodes Nm1 to Nm6 use "v1.1", "v1.2", "v1.3", "v2.1", "v2.2" and "v3.1" of "model for company C1", respectively. It shows. Nodes Nd1 to Nd5 indicate "v1", "v2", "v3", "v4", and "v5" of the "DS for company C1", respectively. Rights information corresponding to each data set is associated with the nodes Nd1 to Nd5. Note that learning model information corresponding to each learning model may be associated with the nodes Nm1 to Nm6.

The server device 10 stores connection information indicating connections between nodes. In the example of FIG. 25, the connections between the learning models by relearning shown in FIG. 24 are shown. For example, node Nm2 (model v1.2 for company C1) is generated by relearning using node Nd2 (DSv2 for company C1) using node Nm1 (model v1.1 for company C1) as the learning source model. The connection information indicates that the learning model is

In addition, node Nm2 is branched into node Nm3 (model v1.3 for company C1) which has undergone a minor version upgrade, and node Nm4 (model v2.1 for company C1) which has undergone a major version upgrade. The connection information indicates that only node Nm5 (model v3.1 for company C1), which has been majorly upgraded, is connected to node Nm3. The server device 10 may be able to display an image visually representing the relationship between the nodes as shown in FIG. 25 based on the information on each node and the connection information. This makes it possible to intuitively understand the relationship between the learning model, learning source model, and dataset.

<Variation: Related information>
Related information related to a group of images used for machine learning is not limited to rights information (information indicating the right to use a DS image). For example, the related information may be version information of a dataset or image information regarding a DS image. The image information of the DS images includes the number of DS images included in the data set, the resolution of the DS images, the subject of the DS images, and the like. Image information such as the resolution of the DS image and the subject of the DS image may be information indicating the amount of DS images that have common image information.

<Modified example: DS image viewing function>
In the example of FIG. 15, an example of the function of viewing a DS image from metadata has been described, but the following aspect may also be used. AI images and metadata are viewed, for example, by image users, but some image users have the right to view, edit, or access DS images (hereinafter referred to as "viewing authority, etc."). This may include those who do not have one.

In that case, the server device 10 outputs metadata including a link L41 to the DS image shown in FIG. 15 to a user terminal used by a person who has the authority to view the DS image, etc. Metadata that does not include the link L41 to the DS image may be output to a user terminal used by a person who does not have the DS image. According to this aspect, it is possible to prevent unauthorized persons from viewing the DS image.

Further, on the DS image screen G5 shown in FIG. 15, the number of displayed DS images may be enormous. In that case, it is difficult to verify the DS images even if a list of DS images is simply displayed. Therefore, the server device 10 calculates, for example, feature amounts used in image recognition (HOG (Histograms of Oriented Gradients) or SIFT (Scaled Invariance Feature Transform), etc.) for the DS images included in the dataset. A representative image extracted from similar images whose feature amounts fall within a common range may be displayed preferentially. According to this aspect, compared to the case where DS images are displayed regardless of image characteristics, it is possible to compare DS images having different characteristics in a shorter time.

<Modified example: Contract information viewing function>
In the example of FIG. 16, an example of the function of viewing contract information from metadata has been described, but the following embodiment may also be used. Similar to the DS image described above, the server device 10 outputs metadata including a link L42 to the contract information shown in FIG. Metadata that does not include the link L42 to the contract information may be output to a user terminal used by an unauthorized person. According to this aspect, it is possible to prevent unauthorized persons from viewing the contract information.

<Example of change: Configuration variation>
The configuration shown in FIG. 1 and the like (overall configuration, hardware configuration, functional configuration, etc.) is an example, and other configurations may be adopted as long as there are no inconveniences in implementation. For example, the server device 10 may be distributed over two or more devices, or may be provided in the form of SaaS (Software as a Service), a cloud computing system, or the like. Further, the information processing executed by the server device 10 may be executed collectively by a user terminal such as the operator terminal 20. In short, as long as the necessary information processing is executed in the entire image generation system 1, the device that executes the information processing may have any configuration.

The information may be associated with each other in various ways. For example, each piece of information may be associated by being stored in a corresponding area in a database, a table, etc., or a table or the like may be created to show each piece of information in association with each other. Alternatively, one piece of information may be associated with information that identifies the other piece of information. In addition, the association may be performed using various well-known methods. The output destination of information or data (hereinafter referred to as "information etc.") may be another device, a display, a storage unit (including a built-in storage unit and an external storage unit), etc. Acquisition of information etc. includes a mode of acquiring information etc. generated by the own device in addition to a mode of acquiring information etc. transmitted from another device.

The aspects of the embodiment described above are information processing devices such as the server device 10, the operator terminal 20, and the user terminal 30, and an information processing system such as the image generation system 1 that includes these information processing devices. , it may be an information processing method. The information processing method includes the same steps as those performed by the information processing system. Moreover, the aspect of the embodiment described above may be a program. The program causes the computer to perform the same steps that an information handling system performs.

<Additional notes>
Furthermore, each aspect described below may be provided.

(1) An information processing system including a processor, wherein the processor acquires relevant information related to a group of images used for machine learning in an information acquisition step, and performs machine learning using the group of images in an output step. An information processing system that outputs metadata indicating the related information acquired for the image group in association with an image generated by an artificial intelligence module.

According to this aspect, information about the image group used for machine learning can be easily grasped.

(2) In the information processing system according to (1) above, the related information includes rights information regarding the image group, and the rights information indicates a right to use the images included in the image group. An information processing system that is information.

According to this aspect, it is possible to grasp the security of the generated image in terms of rights.

(3) The information processing system according to (2) above, wherein the right information includes information indicating a condition under which the right is valid.

According to this aspect, it is possible to call attention to the expiration of rights.

(4) In the information processing system according to (2) or (3) above, the rights information includes information about the type of the right, and the type of right includes information regarding the person who created the image. An information processing system including at least one of a right and a right related to a person represented in the image.

According to such an aspect, security regarding rights can be evaluated in detail.

(5) In the information processing system according to any one of (2) to (4) above, the right information includes a range to which the right applies among objects appearing in images included in the image group. An information processing system that includes information indicating.

According to this aspect, it is possible to generate an image having a composition with little risk in terms of rights.

(6) In the information processing system according to any one of (2) to (5) above, the rights information includes a plurality of items, and the plurality of items include the presence or absence of the right, the An information processing system that includes two or more of the following: a condition for a right to be valid, a type of the right, a scope of the right, a type of title of the right, or a name of the right.

According to such an aspect, rights security can be evaluated from multiple perspectives.

(7) In the information processing system according to any one of (2) to (6) above, the rights information includes a plurality of items, and the processor selects the plurality of items in the selection display step. An information processing system that displays a selection image on a user terminal that accepts a selection operation of an item to be included in the metadata, and in the output step outputs data indicating the item selected by the selection operation as the metadata.

According to such an aspect, desired items can be included in the metadata.

(8) In the information processing system according to any one of (2) to (7) above, in the selection acceptance step, the processor accepts a selection of images to be included in the image group, and includes the Contract information representing the content of a contract that defines the right to use the image is associated, and in the generation step, based on the contract information associated with the image included in the image group by the selection, An information processing system that generates the rights information regarding the image group.

According to this aspect, it is possible to reduce the effort required to prepare the rights information.

(9) In the information processing system according to (8) above, in the change accepting step, the processor accepts a change to an image to be included in the image group, and in the generating step, the processor receives the image after the change is accepted. An information processing system that generates the right information for a group of images based on the contract information associated with the images included in the group.

According to such an aspect, it is possible to reduce the effort required to prepare the rights information even after changing the data set.

(10) In the information processing system according to (8) or (9) above, in the output step, the processor outputs the metadata in a displayable manner to a user terminal, and in the contract display step, the An information processing system that causes the user terminal to display contract information, which is the source of the right information indicated by the metadata, when a display operation of the details of the right is performed on a user terminal on which data is displayed.

According to such an aspect, it is possible to easily verify the contents of a contract indicating rights to an image.

(11) In the information processing system according to any one of (1) to (10) above, in the output step, the processor outputs the metadata in a displayable manner to a user terminal, and displays an image. In the step, when a display operation for the image group is performed on the user terminal on which the metadata is displayed, images included in the image group for which the related information is indicated by the metadata are displayed on the user terminal. information processing system.

According to this aspect, it is possible to easily verify the learned actual image.

(12) In the information processing system according to any one of (1) to (11) above, the artificial intelligence module has a plurality of components, and each of the plurality of components is a group of images that are the same or different. The artificial intelligence module generates one image by each of the plurality of components operating, and in the output step, the processor corresponds to the image generated by the artificial intelligence module. The information processing system further includes outputting, as the metadata, data indicating the related information acquired for a group of images used for machine learning by each of the plurality of components.

According to such an aspect, the learning content of each component can be easily grasped.

(13) In the information processing system according to any one of (2) to (10) above, in the image acquisition step, the processor acquires a user image provided by a user, and the user image is acquired based on the provision conditions agreed to by the user, and in the information acquisition step, when the artificial intelligence module performs machine learning using a group of images including the acquired user image, the information acquisition step includes the provision condition of the user image. An information processing system that acquires information as the rights information regarding the image group.

According to this aspect, the tendency of the generated images can be adjusted to the user's preference.

(14) A program that causes a computer to execute each step executed by the information processing system according to any one of (1) to (13) above.

According to this aspect, information about the image group used for machine learning can be easily grasped.

(15) An information processing method, wherein a processor included in the information processing system executes each step executed by the information processing system according to any one of (1) to (13) above.

According to this aspect, information about the image group used for machine learning can be easily grasped.
Of course, this is not the case.
Furthermore, the embodiments and modifications described above may be implemented in any combination.

Finally, although various embodiments according to the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The new embodiment can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

1: Image generation system 2: Communication line 10: Server device 11: Control unit 20: Operator terminal 21: Control unit 30: User terminal 31: Control unit 100: Image generation AI module DB1: Image database DB2: Data set database DB3: Learning model database

Claims (6)

An information processing system comprising a processor,
The processor,
In the information acquisition step, relevant information related to the image group used for machine learning is acquired,
In the output step, output metadata indicating the related information acquired for the image group in association with an image generated by an artificial intelligence module that performs machine learning using the image group ;
The related information includes rights information regarding the image group,
The rights information is information indicating the right to use images included in the image group,
The rights information includes multiple items,
In the selection display step, displaying on a user terminal a selection image that accepts a selection operation of an item to be included in the metadata among the plurality of items;
In the output step, data indicating the item selected by the selection operation is output as the metadata.
Information processing system. An information processing system comprising a processor,
The processor,
In the information acquisition step, relevant information related to the image group used for machine learning is acquired,
In the output step, output metadata indicating the related information acquired for the image group in association with an image generated by an artificial intelligence module that performs machine learning using the image group ;
The related information includes rights information regarding the image group,
The rights information is information indicating the right to use images included in the image group,
In the selection reception step, a selection of an image to be included in the image group is accepted, and the image is associated with contract information representing the content of a contract that defines the right to use the image;
In the generation step, the rights information for the image group is generated based on the contract information associated with the image included in the image group by the selection.
Information processing system. The information processing system according to claim 2 ,
The processor includes:
The change acceptance step accepts changes to images to be included in the image group;
In the generation step, the right information of the image group is generated based on the contract information associated with the image included in the image group after the change is accepted.
Information processing system. The information processing system according to claim 2 ,
The processor includes:
The output step outputs the metadata in a displayable manner on a user terminal;
In the contract display step, when the display operation of the details of the rights is performed on the user terminal on which the metadata is displayed, the contract information that is the source of the rights information indicated by the metadata is displayed on the user terminal. display,
Information processing system. A program that causes a computer to execute each step executed by the information processing system according to any one of claims 1 to 4 . An information processing method,
The processor included in the information processing system is
executing each step executed by the information processing system according to any one of claims 1 to 4 ;
Information processing method.