JP2020043384A - Information processing apparatus, method, and program for generating composite image for user - Google Patents

Abstract

To provide an information processing apparatus, a method, and a program, capable of achieving a service that fulfills a user's desire to impersonate another person.SOLUTION: In a computer system 1000, an information processing apparatus for generating a composite image for a user includes: first acquisition means for acquiring at least one user image; second acquisition means for acquiring at least one base image for which modification is permitted; generation means for generating a composite image on the basis of the at least one base image and the at least one user image.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing apparatus, method, and program for generating a composite image for a user.

  Everyone has a desire to impersonate others. For example, they have a desire to impersonate a movie actor or an entertainer in advertising. There is no service that fulfills such a desire.

  The inventor of the present invention has realized that realizing a service that fulfills the desire to impersonate another user will lead to a new media experience.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an information processing apparatus or the like that can realize a service that fulfills a desire to impersonate another user.

The present invention provides, for example, the following items.
(Item 1)
An information processing apparatus for generating a composite image for a user,
First acquisition means for acquiring at least one user image;
Second acquisition means for acquiring at least one base image for which modification is permitted;
An information processing apparatus comprising: a generation unit configured to generate a composite image based on the at least one base image and the at least one user image.
(Item 2)
The information processing apparatus according to item 1, further comprising a providing unit for providing the composite image to a user.
(Item 3)
The information processing apparatus according to item 2, wherein the providing unit automatically provides the user with the composite image without receiving a request to provide the composite image from the user.
(Item 4)
The providing means,
Notifying the user that the composite image can be provided;
Receiving a request from the user to provide the composite image;
3. The information processing apparatus according to item 2, wherein in response to receiving a request to provide the composite image from the user, providing the composite image to the user.
(Item 5)
The second acquisition unit acquires a plurality of base images,
The providing means,
Providing the user with a choice of the plurality of base images;
Receiving from the user an input to select at least one of the plurality of base images;
Responsive to receiving input from the user to select at least one of the plurality of base images, the image generated based on the selected at least one base image and the at least one user image 3. The information processing apparatus according to item 2, wherein the information processing apparatus provides a composite image to the user.
(Item 6)
A plurality of persons are shown in the at least one base image;
The providing means,
Providing the user with a choice of a plurality of persons in the at least one base image;
Receiving from the user an input selecting at least one of a plurality of persons in the at least one base image;
Providing the composite image to the user in response to receiving from the user an input to select at least one of the plurality of persons in the at least one base image;
The composite image is a composite image obtained by converting at least a part of the selected at least one person in the at least one base image and at least a part of a person in the at least one user image. The information processing device according to any one of 2, 4 to 5.
(Item 7)
The generation means,
Generating a composite image obtained by converting at least a part of the at least one base image and at least a part of a person in the at least one user image. Information processing device.
(Item 8)
The generation means,
Generating a composite image in which a face of a person in the at least one base image and a face of a person in the at least one user image are converted, the information according to any one of items 1 to 7, Processing equipment.
(Item 9)
The base image and the user image include sound,
The generation means,
The information processing apparatus according to any one of items 1 to 8, further comprising: generating a combined image obtained by converting a voice of a person in the at least one base image and a voice of a person in the at least one user image.
(Item 10)
The generation means,
The method according to any one of items 1 to 9, further comprising: generating a combined image obtained by converting a body shape of a person in the at least one base image and a body shape of a person in the at least one user image. Processing equipment.
(Item 11)
The first obtaining means obtains user images of a plurality of persons including a first person and a second person,
The generation means,
Converting at least a portion of a first person in the at least one base image and at least a portion of a first person in the user image, and converting at least one of a second person in the at least one base image; The information processing apparatus according to any one of items 1 to 10, further comprising: generating a combined image obtained by converting a part and at least a part of a second person in the user image.
(Item 12)
The generation means,
The at least one at least a portion of the first person in the user image is to be synthesized based on a relationship between a first person in the user image and a second person in the user image. Determining a first person in the base image and determining a second person in the at least one base image to which at least a portion of the second person in the user image is to be combined; Item 12. The information processing device according to item 11.
(Item 13)
Each of the at least one base image includes a set of sub-base images, each sub-base image having the same content, but a different person in the image,
The generation means,
Determining a sub-base image that includes a person most similar to the person in the at least one user image;
Generating a combined image obtained by converting at least a part of the person in the determined sub-base image and at least a part of a person in the at least one user image. An information processing device according to claim 1.
(Item 14)
The information processing device according to any one of items 1 to 13, wherein the user image is a user's own image.
(Item 15)
The information processing device according to any one of items 1 to 14, wherein the composite image is an advertisement moving image.
(Item 16)
A program for generating a composite image for a user, wherein the program is executed in an information processing apparatus including a processor unit, and the program includes:
Obtaining at least one user image;
Obtaining at least one base image for which modification is permitted;
Generating a composite image based on the at least one base image and the at least one user image.
(Item 17)
A method for generating a composite image for a user, wherein the method is performed in an information processing apparatus including a processor unit, the method comprising:
The processor unit acquiring at least one user image;
The processor unit acquiring at least one base image for which modification is permitted;
The processor unit generating a composite image based on the at least one base image and the at least one user image.
(Item 18)
A terminal device for providing a composite image for a user, wherein the terminal device can communicate with a server device, wherein the terminal device includes:
Acquiring means for acquiring at least one user image;
Transmitting means for transmitting the at least one user image to the server device;
A terminal comprising: a receiving unit that receives, from the server device, a composite image generated based on at least one base image for which modification is permitted and the at least one user image; and an output unit that outputs the composite image. apparatus.
(Item 19)
Item 19. The terminal device according to item 18, further comprising receiving means for receiving permission from the user to generate the composite image.
(Item 20)
A program for providing a composite image for a user, wherein the program is executed in a terminal device including a processor unit, the terminal device can communicate with a server device, and the program includes:
Obtaining at least one user image;
Transmitting the at least one user image to the server device;
Executing a process including, from the server device, receiving a composite image generated based on at least one base image for which modification is permitted and the at least one user image; and outputting the composite image. A program that causes the processor unit to perform the operation.
(Item 21)
A method for providing a composite image for a user, the method being performed in a terminal device capable of communicating with a server device, the method comprising:
Obtaining at least one user image;
Transmitting the at least one user image to the server device;
Receiving, from the server device, a composite image generated based on at least one base image for which modification is permitted and the at least one user image;
A program for causing the processor unit to execute a process including outputting the composite image.
(Item 22)
A computer system for generating a composite image for a user, the computer system comprising: a server device; and at least one terminal device capable of communicating with the server device.
The terminal device,
Obtaining at least one user image;
And transmitting the at least one user image to the server device.
The server device,
Receiving from the server device acquiring the at least one user image;
Obtaining at least one base image for which modification is permitted;
Generating a composite image based on the at least one base image and the at least one user image;
Transmitting the composite image to the terminal device.
The terminal device,
Receiving the composite image from the server device;
Outputting the composite image.

  ADVANTAGE OF THE INVENTION According to this invention, it is possible to provide the server apparatus etc. which can implement | achieve the service which fulfills the desire to impersonate another user. Thereby, it is possible to provide a new media experience to the user.

The figure which shows roughly the flow of the new service which provides the synthetic | combination image for a user. FIG. 9 is a diagram illustrating a state in which an original image before conversion is reproduced and a state in which a composite image after conversion is reproduced. The figure which shows the terminal device 100 on which the example of the selection screen 10 for making a user select a base image was displayed. The figure which shows the terminal device 100 on which the example of the selection screen 20 for making a user select the person who wants to substitute is displayed. The figure which shows the terminal device 100 on which the example of the notification screen 30 which displays the notification to the effect that a synthetic image is viewable was displayed. FIG. 1 is a diagram illustrating an example of a configuration of a computer system 1000 for providing a composite image for a user. FIG. 2 is a block diagram showing an example of the configuration of the terminal device 100. FIG. 2 is a block diagram showing an example of a configuration of a server device 200. FIG. 3 is a block diagram showing an example of a configuration of a processor unit 230 of the server device 200. 9 is a flowchart showing an example of processing in the computer system 1000 for providing a composite image for a user. 9 is a flowchart showing an example of processing in the computer system 1000 for providing a composite image for a user. 9 is a flowchart showing an example of processing in the computer system 1000 for providing a composite image for a user. 9 is a flowchart showing an example of processing in the computer system 1000 for providing a composite image for a user.

(Definition)
As used herein, “image” includes a still image and a moving image. Still images and moving images may or may not include audio. A still image or a moving image that does not include audio is called a video. The image is preferably a moving image rather than a still image. This is because a moving image has a larger amount of information than a still image and has a wider range of expression. A moving image can represent more attractive content for a user than a still image.

  In the present specification, the “user image” includes an image in which the user himself or an image in which a person related to the user, such as the user's family, relatives, or friends, is shown. The person associated with the user may be, for example, a person who has a blood or marriage relationship with the user. A person associated with a user means, for example, a person for whom the user can be responsible for the portrait right of that person.

  In the present specification, the “base image” means an image serving as a base of the composite image. The base image can be, for example, an image in which a company, such as a movie, a program, or an advertisement, has a copyright. The base image may be, for example, an image in which the celebrity has a copyright, such as an image taken by the celebrity. The base image may be, for example, a user image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1. New service of providing composite images for users The inventor of the present invention has developed a new service of providing composite images for users. The service includes at least a part of an image of a person (for example, a face image) appearing in a movie, a program, an advertisement, and / or at least a part of an image of a user different from the person (for example, a face image). , A face image) and / or a synthesized image converted to the user's voice. With this new service, users can view images as if they appeared in the movie, program, advertisement, or the like.

  As mentioned above, everyone has a desire to impersonate others. According to this new service, a desire to take the place of an actor in a movie or an entertainer in an advertisement can be virtually fulfilled on an image. With this new service, users can enjoy a new media experience like never before.

  FIG. 1A is a diagram schematically illustrating a flow of a new service of providing a composite image for a user. A case in which a user using the terminal device 100 uses this service will be described as an example.

  First, in step S1, the user performs use registration in order to use a new service of providing a composite image for the user. For example, it is possible to perform use registration by starting a dedicated application using the terminal device 100 and inputting necessary information. At the time of use registration, the user can state that there is a desire to take the place of another person. This may be input to the dedicated application as permission to generate a composite image based on the user image, for example. This application may be, for example, a local application installed on the terminal device 100 or a web application available via a web browser. Although the terminal device 100 is depicted as a smartphone, it can be any terminal device that interacts with the user, such as a tablet, personal computer, smart glass, and the like.

  When the use registration is completed, the user uploads an image including the user to the server device 200 in step S2. The image may be, for example, an image captured in the past or an image captured at the time of upload. The user can upload, for example, his own voice in addition to the image. The sound may be, for example, a sound recorded in the past, a sound recorded at the time of upload, or a sound included in an image. When the sound is the sound included in the image, the upload of the image and the sound is achieved by uploading the image with sound. For example, instead of, or in addition to, uploading an image or sound, the user specifies the location of the image or sound that has already been uploaded on the SNS such as Facebook or Instagram, and thereby specifies the image or sound to the server device 200. You may make it acquire a sound.

  After the image (and sound) of the user is uploaded to the server device 200, in step S3, the server device 200 generates a composite image for the user.

  After the composite image for the user is generated, the composite image for the user is provided to the terminal device 100 in step S4.

  In step S5, the composite image for the user is reproduced on the terminal device 100.

  For example, as shown in FIG. 1B, a composite image obtained by converting an image of the face of actor A in the movie into an image of the face of the user is reproduced on terminal device 100. The figure on the left side of FIG. 1B shows a state where the original image before the conversion is reproduced, and the figure on the right side of FIG. 1B shows a state where the composite image after the conversion is reproduced. For example, the voice of actor A in the movie may be converted into the voice of the user. Thereby, the user can watch the movie appearing in place of actor A on terminal device 100.

  In step S5, for example, the composite image may be provided in response to the user selecting an image (base image) from which the composite image is based.

  FIG. 1C shows terminal device 100 on which an example of selection screen 10 for allowing a user to select a base image is displayed. In the selection screen 10 of FIG. 1C, a plurality of movies are displayed as options of the base image. On the selection screen 10, the user can select a movie that he or she wants to appear in place of the acting actor. When the user selects a movie to appear, the terminal device 100 reproduces a composite image obtained by converting an image and / or sound of an actor's face in the selected movie into a user's face and / or sound. For example, when the user selects the movie 11 on the selection screen, as shown in the diagram on the right side of FIG. 1B, a composite image obtained by converting the face image and the voice of the actor A in the movie into the face image and the voice of the user is obtained. It is reproduced on the terminal device 100.

  Alternatively, in step S5, for example, after selecting a base image on the selection screen 10, a composite image is provided in response to selecting a person who wants to replace the person appearing in the base image. Is also good.

  FIG. 1D shows the terminal device 100 on which an example of the selection screen 20 for allowing the user to select a person to be replaced is displayed. The selection screen 20 is, for example, a screen transited from the selection screen 10 of FIG. 1C. In the selection screen 20 of FIG. 1D, a plurality of actors are displayed as options of a person who can be impersonated. The user can select an actor to be replaced on the selection screen 20. When the user selects a movie to appear on the selection screen 10 of FIG. 1C and selects an actor to be replaced on the selection screen 20 of FIG. 1D, the face image and / or sound of the selected actor in the selected movie is displayed. The composite image converted into the image and / or the voice of the user's face is reproduced on the terminal device 100. For example, when the user selects the movie 11 on the selection screen 10 and selects the actor 21 on the selection screen 20, the image and / or sound of the face of the actor A in the movie is displayed as shown in the right-side diagram of FIG. 1B. The composite image converted into the image of the user's face and / or the voice is reproduced on the terminal device 100.

  Alternatively, in step S5, for example, a composite image may be provided in response to selecting a person to be replaced without selecting a base image.

  For example, the selection screen 20 of FIG. 1D can be displayed without passing through the selection screen 10 of FIG. 1C, and the user can select an actor to be replaced. When the user selects an actor to be replaced on the selection screen 20, a composite image obtained by converting an image and / or sound of the face of the selected actor into an image and / or sound of the user's face in a certain image (for example, a movie) is displayed. Is reproduced on the terminal device 100. At this time, the certain image may be, for example, a randomly determined base image, or may be an arbitrary base image provided by a base image provider or a service provider. For example, when the user selects the actor 21 on the selection screen 20, the face image and / or sound of the actor A in the movie is converted into the face image and / or sound of the user as shown in the figure on the right side of FIG. 1B. The converted composite image is reproduced on the terminal device 100.

  In the above three examples, the user selects a base image that the user wants to appear and / or an actor that the user wants to take over, so that the user can see a personalized composite image according to his or her preference, The user can have a new media experience.

  Alternatively, in step S5, for example, when the user is viewing a moving image on the terminal device 100, a notification that the composite image can be viewed is received, and the user provides a request to provide the composite image in response to the notification. When transmitted, a composite image may be provided.

  FIG. 1E shows the terminal device 100 on which an example of the notification screen 30 displaying a notification that the composite image can be viewed is displayed. The notification screen 30 is, for example, a screen that is displayed while the reproduction of the movie is paused while watching the movie shown on the left side of FIG. 1B. On the notification screen 30 of FIG. 1E, a message “You can watch an interesting advertisement. Would you like to watch it?” Is displayed, and respond to this message with “Yes” or “No”. Has become. When the user selects “Yes” on the notification screen 30, a composite image obtained by converting a face image and / or a voice of a person appearing in a certain image (for example, an advertisement) into a user face image and / or a voice is displayed. It is reproduced on the terminal device 100. When the user selects “No” on the notification screen 30, the reproduction of the paused moving image is started, or a certain image (for example, an advertisement) is reproduced on the terminal device 100 without conversion. .

  Alternatively, in step S5, for example, when the user is watching a moving image on the terminal device 100, the image and / or sound of the face of the person suddenly appearing in a certain image (for example, an advertisement) is displayed. The synthesized image converted into the image and / or the sound is reproduced by the terminal device 100. For example, when watching the movie shown on the left side of FIG. 1B, suddenly, the movie is paused and the face image and / or sound of the person appearing in the advertisement is changed to the user face image and sound. The converted composite image is reproduced on the terminal device 100. When the reproduction of the composite image ends, the reproduction of the paused moving image starts.

  In the above two examples, a personalized image can be accidentally encountered, thereby allowing the user to have a new media experience.

  In the example shown in FIG. 1A, the user logs in and actively uploads his / her own image to the server device 200. For example, an image captured by a camera (for example, a web camera) of the terminal device 100 is automatically stored in the server device. You may make it upload to the apparatus 200. As a result, a synthesized image obtained by converting the image and / or sound of the performer's face into the image and / or sound of his / her own face is generated irrespective of the user's intention. The generated composite image may be reproduced by the terminal device 100 in real time, for example, or may be reproduced by the terminal device 100 at a time interval. This results in the user seeing the personalized image quite unexpectedly, allowing the user a new media experience.

  The above-described new service of providing a composite image for a user can be realized by, for example, a computer system 1000 for providing a composite image for a user described below.

2. Configuration diagram of a computer system for providing a composite image for the user. 2 shows an example of the configuration of a computer system 1000 for providing a composite image for the user.

  The computer system 1000 includes at least one terminal device 100, a server device 200 connected to the at least one terminal device 100 via a network 400, and a database unit 300 connected to the server device 200.

  The terminal device 100 may be any terminal device that interacts with the user, such as a smartphone, a tablet, a personal computer, and a smart glass. The terminal device 100 may include, for example, a computer system including a camera installed in a facility such as a movie theater, an amusement park, a commercial facility, a station, and the like, and a screen that projects an image from the camera. The terminal device 100 can communicate with the server device 100 via the network 400. Here, the type of the network 400 does not matter. For example, the terminal device 100 may communicate with the server device 200 via the Internet, or may communicate with the server device 200 via a LAN. Although three terminal devices 100 are depicted in FIG. 2, the number of terminal devices 100 is not limited to this. The number of the terminal devices 100 may be any number of one or more.

  The server device 200 can communicate with at least one terminal device 100 via the network 400. For example, the server device 200 can acquire an image from the database unit 300 connected to the server device 200, and transmit the acquired image to at least one terminal device 100. For example, the server device 200 acquires a base image from the terminal device 500 of a base image provider that can be connected to the server device 200 via the network 400, and transmits the acquired base image to the database unit 300 for storage. Can be.

  The database unit 300 connected to the server device 200 stores, for example, a user image of a user of at least one terminal device 100, a base image, a composite image generated by the server device 200, and the like. The database unit 300 includes, for example, information indicating the physical characteristics of the user (for example, height, weight, chest circumference, waist circumference, waist circumference, skin texture, skin age, etc.), and user attributes (for example, age, gender, Nationality, hometown, etc.), characteristics of the user's behavior (eg, habit, gesture, etc.) Information on the user's human relationship (eg, lover information (eg, lover's attributes, physical characteristics, images, etc.), spouse's Information (eg, spouse attributes, physical characteristics, images, etc.), parents information (eg, parents attributes, physical characteristics, images, etc.), sibling information (eg, sibling attributes, physical characteristics, images) Etc.), kinship information (eg, kinship attributes, physical characteristics, images, etc.) may be stored. The database unit 300 includes, for example, information on whether or not modification of the base image is permitted, information on a period during which modification of the base image is permitted, and information on a person in the base image who is permitted to modify the base image. And may be stored in association with. The database unit 300 can be implemented by any storage means.

  FIG. 3A shows an example of the configuration of the terminal device 100.

  The terminal device 100 includes a communication interface unit 110, an input unit 120, a display unit 130, a memory unit 140, and a processor unit 150.

  The communication interface unit 110 controls communication via the network 400. The processor unit 150 of the terminal device 100 can receive information from outside the terminal device 100 via the communication interface unit 110, and can transmit information to the outside of the terminal device 100. The communication interface unit 110 can control communication by an arbitrary method.

  The input unit 120 enables information from a user to be input to the terminal device 100. The input unit 120 enables the terminal device 100 to input, for example, an account name and a password for login, information on a human relationship of the user, permission to generate a composite image based on the user image, and the like. It does not matter how the input unit 120 can input information from the user to the terminal device 100. For example, when the input unit 120 is a touch panel, the user may input information by touching the touch panel. Alternatively, when the input unit 120 is a mouse, the user may input information by operating the mouse. Alternatively, when the input unit 120 is a keyboard, the user may input information by pressing a key on the keyboard. Alternatively, when the input unit 120 is a microphone, the user may input information by inputting voice to the microphone.

  Display unit 130 can be any display for displaying information.

  The memory unit 140 stores a program for executing processing in the terminal device 100, data required for executing the program, and the like. The memory unit 140 includes, for example, a program for generating a composite image for a user (for example, a program for realizing a process illustrated in FIG. 4 described later) or a program for providing a composite image for a user ( For example, part or all of a program for realizing the processes shown in FIGS. 5, 6, and 7 described below is stored. The memory unit 140 may store an application that implements an arbitrary function. Here, it does not matter how the program is stored in the memory unit 140. For example, the program may be preinstalled in the memory unit 140. Alternatively, the program may be installed in the memory unit 140 by being downloaded via the network 400. The memory unit 140 can be implemented by any storage means.

  The processor unit 150 controls the operation of the terminal device 100 as a whole. The processor unit 150 reads out a program stored in the memory unit 140 and executes the program. This allows the terminal device 100 to function as a device that executes a desired step. The processor unit 150 may be implemented by a single processor, or may be implemented by a plurality of processors.

  The terminal device 100 may include, for example, an arbitrary camera capable of capturing an image in addition to the above configuration. The camera may be a camera built in the terminal device 100 or an external camera attached to the terminal device 100.

  In the example shown in FIG. 3A, each component of the terminal device 100 is provided in the terminal device 100, but the present invention is not limited to this. Any of the components of the terminal device 100 may be provided outside the terminal device 100. For example, the display unit 130 can be configured as separate hardware (for example, a television). For example, when each of the input unit 120, the display unit 130, the memory unit 140, and the processor unit 150 is configured by separate hardware components, each hardware component may be connected via an arbitrary network. . At this time, the type of network does not matter. Each hardware component may be connected, for example, via a LAN, may be connected wirelessly, or may be connected by wire. The terminal device 100 is not limited to a specific hardware configuration. For example, it is within the scope of the present invention to configure the processor unit 150 with an analog circuit instead of a digital circuit. The configuration of the terminal device 100 is not limited to the above as long as the function can be realized.

  FIG. 3B shows an example of the configuration of the server device 200.

  The server device 200 includes a communication interface unit 210, a memory unit 220, and a processor unit 230.

  The communication interface unit 210 controls communication via the network 400. The communication interface unit 210 also controls communication with the database unit 300. The processor unit 220 of the server device 200 can receive information from outside the server device 200 via the communication interface unit 210, and can transmit information to the outside of the server device 200. For example, the processor unit 220 of the server device 200 receives a user image from at least one terminal device 100 via the network 400. For example, the processor unit 220 of the server device 200 transmits the composite image to at least one terminal device 100 via the network 400. For example, the processor unit 220 of the server device 200 can receive the base image from the terminal device 500 of the base image provider via the network 400. The communication interface unit 210 can control communication by an arbitrary method.

  For example, the communication interface unit 210 may be configured to receive only an image whose alteration is permitted by the base image provider. The image whose alteration is permitted includes, for example, an image in which a person whose alteration is permitted is shown. This can be achieved, for example, by causing the base image provider's terminal device 500 to transmit an image to the server device 200 and also transmit information indicating that the base image provider has permitted image modification. The information indicating that the modification is permitted may include information of a person who permits the modification during a period during which the modification is permitted. An example of the information indicating that the modification is permitted is information indicating that modification of the person A in the base moving image is permitted for one year.

  The memory unit 220 stores a program required to execute the processing of the server device 200, data required to execute the program, and the like. For example, a program for generating a composite image for the user (for example, a program for realizing the processing shown in FIG. 4 described later) or a program for generating a composite image for the user (for example, FIG. 5 described later) , A program for realizing the processing shown in FIGS. 6 and 7). The memory unit 220 can be implemented by any storage means.

  The processor unit 230 controls the operation of the entire server device 200. The processor unit 230 reads out a program stored in the memory unit 220 and executes the program. This allows the server device 200 to function as a device that executes a desired step. The processor unit 230 may be implemented by a single processor, or may be implemented by a plurality of processors.

  In the example shown in FIG. 3B, each component of the server device 200 is provided in the server device 200, but the present invention is not limited to this. Any of the components of the server device 200 may be provided outside the server device 200. For example, when each of the memory unit 220 and the processor unit 230 is configured by separate hardware components, each hardware component may be connected via an arbitrary network. At this time, the type of network does not matter. Each hardware component may be connected, for example, via a LAN, may be connected wirelessly, or may be connected by wire. The server device 200 is not limited to a specific hardware configuration. For example, it is within the scope of the present invention to configure the processor unit 230 with an analog circuit instead of a digital circuit. The configuration of the server device 200 is not limited to the above as long as the function can be realized.

  In the examples shown in FIGS. 2 and 3B, the database unit 300 is provided outside the server device 200, but the present invention is not limited to this. The database unit 300 may be provided inside the server device 200. At this time, the database unit 300 may be implemented by the same storage unit that implements the memory unit 220, or may be implemented by a storage unit that is different from the storage unit that implements the memory unit 220. In any case, the database unit 300 is configured as a storage unit for the server device 200. The configuration of the database unit 300 is not limited to a specific hardware configuration. For example, the database unit 300 may be configured by a single hardware component, or may be configured by a plurality of hardware components. For example, the database unit 300 may be configured as an external hard disk device of the server device 200, or may be configured as a storage on a cloud connected via a network.

  FIG. 3C illustrates an example of a configuration of the processor unit 230 of the server device 200.

  The processor unit 230 includes a user image processing unit 231, a base image processing unit 232, a face conversion model forming unit 233, an audio conversion model forming unit 234, a face converting unit 235, an audio converting unit 236, and a video / audio combining unit. Section 237. The audio conversion model forming unit 234, the audio converting unit 236, and the video / audio combining unit 237 are indicated by broken lines in FIG. 3C. When the user image and the base image do not include audio, the audio conversion model forming unit 234, the audio converting unit 236, and the video / audio combining unit 237 are unnecessary, and the processor unit 230 includes the audio conversion model forming unit 234, It is not necessary to provide the converter 236 and the video / audio combiner 237.

  The user image processing unit 231 includes a user image acquisition unit 2311, blocks related to video processing (video clipping unit 2313, face clipping unit 2315), and blocks related to audio processing (voice clipping unit 2314, audio adjustment unit 2316). And Blocks related to audio processing are indicated by dashed lines in FIG. 3C. When the user image does not include sound, blocks related to sound processing are unnecessary, and the user image processing unit 231 does not need to include blocks related to sound processing.

  The user image acquisition unit 2311 acquires a user image. The user image acquisition unit 2311 may acquire, for example, a user image transmitted from at least one terminal device 100, or may acquire a user image stored in the database unit 300. The base image obtaining unit 2322 may obtain, in addition to the base image, another image of the person shown in the base image.

  The video clipping unit 2313 performs a process of clipping a video from the user image received by the user image acquisition unit 2311, while the audio clipping unit 2314 performs a process of clipping a voice from the user image received by the user image acquisition unit 2311. . This allows video and audio to be processed separately in subsequent processing. The video clipping unit 2313 and the audio clipping unit 2314 can be implemented by software such as ffmpeg, for example.

  The face cutout unit 2315 performs a process of cutting out a face portion of a person appearing in the video. For example, when the user image is a moving image, the face cutout unit 2315 performs a process of cutting out a still image from a moving image, and then performs a process of cutting out a face portion of a person from the still image. The face portion cut out by the face cutout unit 2315 is used as learning data for the face conversion model forming unit 233. The face cutout unit 2315 can be implemented by software such as faceswap, for example.

  The sound adjusting unit 2316 performs a process of removing sounds other than sounds that can be included in the sound and adjusting the sound to a format that can be easily processed. The sound adjusting unit 2316 performs, for example, a process of adjusting the sample rate of the sound. The voice adjusting unit 2316 further performs a voice analysis process. This makes it possible to extract features (for example, frequency distribution and the like) of speech, and to perform speech recognition. The features of the voice extracted by the voice adjusting unit 2316 are used as learning data for the voice conversion model forming unit 234. The audio adjustment unit 2316 can be implemented by software such as ffmpeg, cycle GANvoice, or the like.

  The base image processing unit 232 includes a base image acquisition unit 2322, blocks related to video processing (video clipping unit 2323, face clipping unit 2325), and blocks related to audio processing (voice clipping unit 2324, audio adjustment unit 2326). And Blocks related to audio processing are indicated by dashed lines in FIG. 3C. When the base image does not include sound, blocks related to sound processing are unnecessary, and the base image processing unit 232 does not need to include blocks related to sound processing. The base image processing unit 232 can process, in addition to the base image, another image of a person appearing in the base image in the same manner as the base image. By processing the person in the base image in addition to the base image, the amount of data used for learning in the face conversion model forming unit 233 or the voice conversion model forming unit 234 at the subsequent stage increases. The accuracy of the conversion model is improved.

  The base image obtaining unit 2322 obtains a base image. The base image acquisition unit 2322 may acquire, for example, the base image transmitted from the terminal device 500 of the base image provider, or may acquire the base image stored in the database unit 300. The base image obtaining unit 2322 may be configured to obtain, for example, only the base image for which the modification is permitted. For example, before acquiring the base image, the base image acquiring unit 2322 determines whether the modification of the base image is permitted, and acquires the base image only when the modification is permitted. can do. For example, the base image acquisition unit 2322 can determine whether or not modification of the acquired base image is permitted, and can pass the base image to the next processing block only when modification is permitted. The determination as to whether modification of the base image is permitted includes, for example, referring to information attached to the base image, inquiring of a server device of a rights management organization that manages the rights of the base image, providing the base image, This can be done by requesting a person to explicitly give permission for modification.

  The video clipping unit 2323 and the audio clipping unit 2324 perform the same processing as the video clipping unit 2313 and the audio clipping unit 2314. The video cutout unit 2323 performs a process of cutting out a video from the base image received by the base image acquisition unit 2321, and performs a process of cutting out audio from the base image received by the base image acquisition unit 2321. This allows video and audio to be processed separately in subsequent processing. The video clipping unit 2323 and the audio clipping unit 2324 can be implemented by software such as ffmpeg, for example.

  The face clipping unit 2325 performs the same processing as the face clipping unit 2315. The face cutout unit 2325 performs a process of cutting out the face of a person appearing in the video. For example, when the base image is a moving image, the face cutout unit 2325 performs a process of cutting out a still image from a moving image, and then performs a process of cutting out a human face from the still image. The face portion cut out by the face cutout unit 2325 is used as learning data for the face conversion model forming unit 233. The face cutout unit 2325 may be implemented by software such as faceswap, for example.

  The audio adjustment unit 2326 performs the same processing as the audio adjustment unit 2316. The sound adjusting unit 2326 removes sounds other than sounds that may be included in the sound, and performs processing for adjusting the sound to a format that can be easily processed. The sound adjusting unit 2326 performs, for example, a process of adjusting the sample rate of the sound. The voice adjusting unit 2326 further performs a voice analysis process. This makes it possible to extract features (for example, frequency distribution and the like) of speech, and to perform speech recognition. The features of the voice extracted by the voice adjusting unit 2326 are used as learning data for the voice conversion model forming unit 234. The sound adjusting unit 2326 can be implemented by software such as ffmpeg, cycle GANvoice, or the like.

  The face conversion model forming unit 233 learns, as learning data, the face portion cut out by the face cutout unit 2315 and the face portion cutout by the face cutout unit 2325, so that the face of the person in the base image is learned. Is performed to form a face conversion model for converting into a face of a person in the user image. The face conversion model forming unit 233 uses, for example, a deep learning technique using a multilayer neural network. At this time, the process of learning using the learning data is performed by using the face part cut out by the face cutout unit 2315 and the face part cut out by the face cutout unit 2325 by using the node of each hidden layer of the multilayer neural network. In the face conversion model formed by this, when the face image of the person shown in the base image is input, the face image (the face image converted to the face of the person shown in the user image) For example, a video (as if a person appearing in the user image had an expression of the face of the person appearing in the base image) is output. The face conversion model forming unit 233 can form a face conversion model with higher accuracy as the number of face parts used for learning increases. The face conversion model forming unit 233 can be implemented by software such as faceswap, for example.

  Using the face conversion model formed by the face conversion model forming unit 233, the face conversion unit 235 generates a face image in which the face of the person shown in the base image is converted into the face of the person shown in the user image. Then, the face image is synthesized with the image of the base image. As a result, a composite image is generated as if the person appearing in the base image is a person appearing in the user image. When the base image is a moving image, for each of the still images constituting the moving image, generate a face image in which the face of the person shown in the base image is converted into the face of the person shown in the user image, The face image is synthesized with the still image image of the base image. Next, a moving image can be constructed by connecting the synthesized still image. The moving image is a moving image as if the person appearing in the base image (moving image) is the person appearing in the user image. The face conversion unit 235 can be implemented by software such as faceswap, for example.

  The speech conversion model forming unit 234 learns the features of the speech extracted by the speech adjustment unit 2316 and the features of the speech extracted by the speech adjustment unit 2326 as learning data, thereby converting the speech included in the base image. A process of forming a voice conversion model for converting into a voice included in the user image is performed. The speech conversion model forming unit 234 uses, for example, a deep learning technique using a multilayer neural network. At this time, the learning process using the learning data is performed by using the features of the speech extracted by the speech adjustment unit 2316 and the features of the speech extracted by the speech adjustment unit 2326 in each hidden layer of the multilayer neural network. In the speech conversion model formed by this, when the speech included in the base image is input, the speech converted into the speech included in the user image (for example, included in the base image) (A sound as if the person in the user image uttered the sound). The voice conversion model forming unit 234 can form a voice conversion model with higher accuracy as the number of voice features used for learning increases. The voice conversion model forming unit 234 can be implemented by software such as cycle GANvoice, for example.

  The audio conversion unit 236 uses the audio conversion model formed by the audio conversion model forming unit 234 to generate audio converted from audio included in the base image into audio included in the user image. Thereby, a sound is generated as if the sound included in the base image is the sound of a person included in the user image. The audio conversion unit 236 can be implemented by software such as cycle GANvoice, for example.

  The video / audio combining unit 237 combines the synthesized video generated by the face converting unit 235 and the voice generated by the voice converting unit 236 to generate a synthesized image with voice. The video / audio coupling unit 237 may be implemented by software such as ffmpeg, for example.

  The composite image generated by the video / audio combining unit 237 is output to the outside of the server device 200 (for example, at least one terminal device 100 and the database unit 300) via the communication interface unit 210.

  In the example illustrated in FIG. 3C, each component of the processor unit 230 is provided in the same processor unit 230, but the present invention is not limited to this. A configuration in which each component of the processor unit 230 is distributed to a plurality of processor units is also within the scope of the present invention.

3. Processing in Computer System for Providing a Composite Image for User FIG. 4 shows an example of processing in a computer system 1000 for providing a composite image for a user. In the example shown in FIG. 4, processing for generating a composite image in the computer system 1000 will be described. Hereinafter, generation of a composite image of one person shown in a user image will be described as an example.

  In step S401, the processor unit 150 of the terminal device 100 acquires at least one user image. The processor unit 150 may obtain a user image by controlling a camera that the terminal device 100 can have, or may store the user image stored in a storage unit (for example, a memory, a storage, an external storage device, or the like). The user image may be obtained, or a user image on the network 400 may be obtained via the communication interface unit 110.

  In step S402, the processor unit 150 of the terminal device 100 transmits at least one user image to the server device 200 via the communication interface unit 110, and the processor unit 230 of the server device 200 transmits the user image via the communication interface unit 210. Receiving at least one user image. Preferably, processor unit 150 of terminal device 100 transmits a plurality of user images to server device 200, and processor unit 230 of server device 200 receives the plurality of user images. This is because, by receiving more user images, more user images can be used in the processing described later, thereby improving the quality of the generated composite image. The received user image may be stored in the database unit 300 and used for subsequent processing, or may be temporarily stored in the memory unit 220 and used for subsequent processing, for example. .

  In step S402, instead of transmitting the user image to the server device 200 or in addition to transmitting the user image to the server device 200, the processor unit 150 of the terminal device 100 , URL) may be transmitted. When the server device 200 receives the location of the user image on the network, the server device 200 accesses the location and acquires the user image.

  In step S403, the user image acquisition unit 2311 of the processor unit 230 acquires at least one user image. The user image acquisition unit 2311 may acquire, for example, the user image received from at least one terminal device 100 in step S402, or may acquire the user image stored in the database unit 300 from the database unit 300. Good.

  Upon acquiring the user image, in step S404, the video clipping unit 2313, the voice clipping unit 2314, the face clipping unit 2315, and the voice adjusting unit 2316 of the processor unit 230 process at least one user image. For example, the video cutout unit 2313 cuts out a video from the user image, the face cutout unit 2315 cuts out a face portion from the cutout video, then the audio cutout unit 2314 cuts out audio from the user image, and the sound adjustment unit 2316 cuts out audio. And adjust the speech and extract features of the speech.

  In step S405, the processor unit 230 of the server device 200 receives at least one base image transmitted from the base image provider terminal device 500 via the communication interface unit 210. For example, the received base image may be stored in the database unit 300 and used for subsequent processing, or may be temporarily stored in the memory unit 220 and used for subsequent processing. . The server device 200 may receive another image of the person shown in the base image in addition to the base image. This is because, by receiving more images of a person in the base image, more images of the person can be used in the processing described later, thereby improving the quality of the generated composite image. . The image of the person appearing in the received base image may be stored in, for example, the database unit 300 and used for subsequent processing, or may be temporarily stored in the memory unit 220 and used for subsequent processing. May be used for

  In step S405, the processor unit 230 of the server device 200 receives the base image (and the image of a person appearing in the base image) transmitted from the terminal device 500 of the base image provider, or provides the base image. In addition to receiving the base image (and the image of the person shown in the base image) transmitted from the terminal device 500 of the user, the location of the base image (and the image of the person shown in the base image) on the network (For example, a URL). When server device 200 receives the location of the base image (and the image of the person shown in the base image) on the network, server device 200 accesses the location and accesses the base image (and the person in the base image). Image).

  In step S405, the processor unit 230 of the server apparatus 200 preferably receives and / or acquires only the base image for which the modification is permitted. The image for which the modification is permitted includes, for example, an image for which modification of the person in the base image is permitted. By receiving or acquiring only the base image for which the modification is permitted, the composite image generated by the server device 200 based on the base image infringes the rights (eg, copyright, portrait right, etc.) that can accompany the base image. This is because it becomes possible to eliminate the possibility of doing this.

  For example, the server apparatus 200 may obtain in advance from the base image provider an agreement that all the base images transmitted to the server apparatus 200 are permitted to be modified, or the terminal apparatus 500 of the base image provider. When the base image is received from, information indicating that modification of the base image is permitted may be received together. At this time, information indicating that modification of the base image is permitted may be stored in the database unit 300 in association with the base image.

  In step S406, the base image acquisition unit 2321 of the processor unit 230 acquires at least one base image. The base image obtaining unit 2321 may obtain the base image received from the terminal device 500 of the base image provider in step S403, or may obtain the base image stored in the database unit 300 from the database unit 300, for example. You may.

  Step S <b> 406 may include, for example, determining whether the modification of the acquired base image is permitted by the base image acquisition unit 2321 of the processor unit 230. The determination as to whether the modification of the base image is permitted may be made, for example, by referring to the information attached to the base image by the base image acquisition unit 2321 of the processor unit 230, or by referring to the database unit 300 in association with the base image. Refer to the information stored in the base image provider, send an inquiry via the communication interface unit 210 to the server of the rights management organization that manages the rights of the base image, and permit the base image provider's terminal device 500 to modify. Is transmitted through the communication interface unit 210 or the like. For example, the base image acquisition unit 2321 of the processor unit 230 may acquire the base image only when the modification is permitted, or may proceed to step S407 only when the modification is permitted. Good.

  After acquiring the base image, in step S407, the video clipping unit 2323, the voice clipping unit 2324, the face clipping unit 2325, and the voice adjusting unit 2326 of the processor unit 230 process at least one base image. For example, the video cutout unit 2323 cuts out a video from the base image, the face cutout unit 2325 cuts out a face part from the cutout video, and then the audio cutout unit 2324 cuts out audio from the base image, and the audio adjustment unit 2326 cuts out audio. And adjust the speech and extract features of the speech.

  When the processing of at least one user image and at least one base image is completed, in step S408, the face conversion model forming unit 233 and the voice conversion model forming unit 234 of the processor unit 230 form a conversion model. For example, the face conversion model forming unit 233 forms a face conversion model, and then the voice conversion model forming unit 234 forms a voice conversion model. The face conversion model and the voice conversion model are formed by using, for example, a deep learning technique.

  When the face conversion model and the voice conversion model are formed, in step S409, the face conversion unit 235, the voice conversion unit 236, and the video / audio combination unit 237 of the processor unit 230 generate a synthesized image. For example, using the face conversion model formed in step S408, the face conversion unit 235 generates a face video obtained by converting the face of the person shown in the base image into the face of the person shown in the user image, The face image is combined with the base image. For example, the audio conversion unit 236 generates audio obtained by converting audio included in the base image into audio included in the user image, using the audio conversion model formed in step S408. For example, the video / audio combining unit 237 generates a combined image with sound by combining the generated composite video and the generated audio.

  In this way, a composite image is generated in server device 200.

  In the example described above, the user image is acquired in step S401, but step S401 is not always necessary. This is because when transmitting the location of the user image on the network as described above in step S402, the user device 100 does not need to hold the user image.

  In the example described above, video and audio are processed in series in each step. However, video and audio may be processed in parallel. At this time, the process for video may proceed to the next step after waiting for the completion of the process for audio, or may proceed to the next step without waiting for the completion of the process for audio. The reverse is also true.

  In the example described above, an example in which steps S405 to S407 are performed after steps S402 to S404 has been described, but the order of steps S402 to S404 and steps S405 to S407 does not matter. For example, steps S402 to S404 may be performed after at least one of steps S405 to S407, or at least one of steps S402 to S404 and at least one of steps S405 to S407 may be performed in parallel. Good.

  For example, at any timing before the server device 200 generates the composite image in step S409, the processor unit 150 of the terminal device 100 generates the composite image based on the user image via the communication interface unit 110. May be transmitted to the server device 200, and the processor unit 230 of the server device 200 may receive the permission. This eliminates the possibility that the composite image generated by the server device 200 based on the user image infringes the rights (for example, copyright, portrait right, etc.) that can accompany the user image. It is. For example, permission to generate a composite image based on the user image may be made each time the user image is transmitted to the server device 200, or at a predetermined timing (for example, once a day, once a week, Times, once a month, once a year, etc.). For example, the registration of the use of the service for providing the composite image for the user described above with reference to FIG. 1A has permitted the generation of the composite image based on the user image. You may consider it. At this time, the use registration information is transmitted to the server device 200 as permission to generate a composite image based on the user image.

  For example, when a plurality of persons are shown in the base image and each of the plurality of persons is permitted to be modified, the face and voice of one of the plurality of persons shown in the base image and the user image are displayed. A synthesized image obtained by converting the face and voice of a person may be generated. Which one of a plurality of persons in the base image is to be converted to the face and voice may be set by the user, for example, or may be automatically determined by the server device 200. Good.

  For example, when the user sets, the server device 200 provides the terminal device 100 with options of a plurality of persons appearing in the base image, and the user designates one of the plurality of persons appearing in the base image. The input to be selected is performed on the terminal device 100, and the terminal device 100 transmits the input to the server device 200. Based on the input, the server device 200 converts the combined image obtained by converting the face and voice of the selected person among the plurality of persons shown in the base image and the face and voice of the person shown in the user image. Generate.

  For example, when the server device 200 automatically determines, based on the attributes (for example, gender, age), physical characteristics (for example, height, body type), and the like of the person shown in the user image, A person who is more similar to the person shown in the user image or a person who is less similar may be selected from the plurality of persons, or may be selected at random. For example, if the person in the user image is a male, a synthetic image is automatically generated by converting the male face and the voice in the base image and the male face and the voice in the user image. can do. For example, if the person appearing in the user image is a fat woman, the synthesis is performed by converting the face and voice of the slender woman appearing in the base image and the face and voice of the fat woman appearing in the user image. Images can be generated automatically. When the server device 200 automatically determines the person to be converted, it leads to the unexpected creation of a composite image and provides the user with a new media experience.

  In the above-described example, an example has been described in which a composite image of one person shown in a user image is generated, but the present invention is not limited to this. In the processing in the computer system 1000 for providing a composite image for the user, it is possible to generate a composite image of a plurality of persons appearing in the user image.

  For example, in the example of FIG. 4, in step S402, the user images of a plurality of persons (for example, at least one user image in which a plurality of persons are captured, or a plurality of user images in which each of a plurality of persons is captured) are displayed. Then, steps S403 to S404 are performed for each person in the user image. In step S405, at least one base image including a plurality of persons is received, and steps S406 to S407 are performed for each person in the base image. Next, step S408 is performed for each person shown in the user image and each person shown in the base image.

  For example, steps S403 to S404 are performed on the first person in the user image, steps S406 to S407 are performed on the first person in the base image, and the first person in the user image is performed. Step S408 is performed for the first person in the base image and the first person in the base image to convert the first person's face in the base image into the first person's face in the user image. And a voice conversion model for converting the voice of the first person in the base image into the voice of the first person in the user image.

  For example, steps S403 to S404 are performed on the second person in the user image, steps S406 to S407 are performed on the second person in the base image, and the second person in the user image is performed. Step S408 is performed on the image and the second person in the base image to convert the face of the second person in the base image into the face of the second person in the user image. And a voice conversion model for converting the voice of the second person in the base image into the voice of the second person in the user image.

  For example, steps S403 to S404 are performed on the nth person in the user image, steps S406 to S407 are performed on the nth person in the base image, and the nth person in the user image Step S408 is performed on the and the nth person in the base image to convert the face of the nth person in the base image into the face of the nth person in the user image. And a voice conversion model for converting the voice of the n-th person shown in the base image into the voice of the n-th person shown in the user image (where n is 2 or more) integer).

  In step S409, the face image obtained by converting the face of the first person in the base image into the face of the first person in the user image using the conversion model of each person formed in this manner. Is combined with the image of the base image, and the face image obtained by converting the face of the second person in the base image into the face of the second person in the user image is combined with the image of the base image. A face image obtained by converting the face of the n-th person in the base image into the face of the n-th person in the user image is synthesized with the image of the base image. Also, a voice is generated by converting the voice of the first person in the base image into the voice of the first person in the user image, and the voice of the second person in the base image is converted to the user image. , A voice converted from the voice of the second person in the base image, and a voice converted from the voice of the n-th person in the base image into the voice of the n-th person in the user image Generate Then, by combining the synthesized video and the generated audio, a synthesized image with audio is generated.

  When generating a composite image of a plurality of people, the face and voice of the person in the user image to be converted to the face and voice of the person in the base image can be set by the user, for example. Alternatively, the server device 200 may automatically determine it.

  For example, when set by the user, the server device 200 provides the terminal device 100 with a plurality of choices of a plurality of persons appearing in the base image, and the user selects the plurality of persons appearing in the user image and the base image. The terminal device 100 performs input for associating each of the plurality of persons in the image with the terminal device 100, and the terminal device 100 transmits the input to the server device 200. Based on the input, server device 200 generates a composite image obtained by converting the face and voice of each person associated with the base image and the user image.

  For example, when the server device 200 automatically determines, the attributes (for example, gender, age), physical characteristics (for example, height, body type), and the like of the person shown in the user image and the person shown in the base image are used. Based on the plurality of persons shown in the base image, a person who is more similar to the person shown in the user image or a person who is less similar may be selected, or may be selected at random. It may be. For example, the attributes and physical characteristics of the person appearing in the user image may be referred to information stored in the database unit 300, or may be recognized from the user image by using existing image recognition technology. It may be. The attributes and physical characteristics of a person appearing in the base image may be recognized from the user image by using, for example, an existing image recognition technology. For example, if the person in the user image is a male, a synthetic image is automatically generated by converting the male face and the voice in the base image and the male face and the voice in the user image. can do. For example, if the person appearing in the user image is a fat woman, the synthesis is performed by converting the face and voice of the slender woman appearing in the base image and the face and voice of the fat woman appearing in the user image. Images can be generated automatically. When the server device 200 automatically determines the person to be converted, it leads to the unexpected creation of a composite image, and provides the user with an interesting and new media experience.

  For example, when the server device 200 determines automatically, a human relationship (for example, lover, sibling, parent and child, etc.) between a plurality of persons appearing in a user image and a plurality of persons appearing in a base image. Based on the human relationship, a person who is more or less similar to the human relationship of the person shown in the user image may be selected from a plurality of people shown in the base image. The human relationship between a plurality of persons appearing in the user image may be estimated from the user image by using, for example, information stored in the database unit 300 or using an existing image recognition technique. You may do so. For example, from an image in which men and women of the same age are shown with smiles, it is estimated that the men and women are lovers or couples. The human relationship between a plurality of persons appearing in the base image may be recognized from the user image by using, for example, an existing image recognition technology. For example, when it is estimated that two men and women appearing in the user image are a couple, the hero face and voice shown in the base image and the male face and voice shown in the user image are converted. It is possible to automatically generate a composite image in which the face and voice of the heroine shown in the base image and the female face and voice shown in the user image are converted. For example, if the adult and child in the user image are estimated to be parents and children, convert the adult face and voice in the base image and the adult face and voice in the user image, It is possible to automatically generate a composite image obtained by converting the child's face and voice shown in the base image and the child's face and voice shown in the user image. Even if the human relationship estimated from the image is different from the actual human relationship, the composite image generated based on such a different human relationship is also an unexpected composite image, which is interesting and also , Leading to a new media experience for users.

  The base image may be a movie, a program, an advertisement, or the like in which an actor, an entertainer, an athlete, or the like appears, but the base image may be, for example, an image shot using a copyright-free model. The copyright free model is a model for photographing a base image used for a composite image. In other words, an image photographed using the copyright free model is a base image dedicated to the composite image. In the case of an actor, entertainer, athlete, etc., the image of the person may not be usable by the company due to the expiration of the contract or the crime, but in the case of the copyright-free model, the face and sound of the copyright-free model are the user image It is assumed that the face and sound of the copyright-free model will not be distributed as they are, and the risk of being unable to use the image due to expiration of contract or crime is extremely high. small.

  By photographing a large amount of images in advance for the copyright-free model and performing the processes of steps S406 and S407 in advance, the load in the process of generating a composite image can be reduced, and the speed of generating the composite image can be reduced. Can be improved. Further, since learning can be performed using a large number of images, the accuracy of the conversion model is also improved.

  The copyright of the image of the face and sound of the copyright-free model taken using the copyright-free model may belong to television stations, video production companies, photographers, etc. , May be considered as being licensed. Therefore, it is easy to handle in terms of rights.

  The copyright-free model is preferably, for example, a model having an average face and / or sound. This is because the accuracy of the conversion model is improved with an average face and / or voice. The definition of an average can vary, for example, from user to user. For example, the average may be defined based on the user's nationality, which may be the average face and / or voice of a Japanese if the user's nationality is Japan, and a Chinese if the user's nationality is Chinese. May be the average face and / or voice of a Chinese. For example, the average may be defined based on the user's hometown, which may be the average face and / or voice of a person from the Kansai region if the user's hometown is in the Kansai region, If the land is in the Kyushu region, it may be the average face and / or sound of a person from the Kyushu region.

  In the conversion model formed in step S408 described above, the face or voice of the person shown in the base image used to form the conversion model is similar to the face or voice of the person shown in the user image. The higher the accuracy. Therefore, it is preferably similar to the face of the copyright-free model and / or the face and / or the voice of the person in the user image used. For example, this can be achieved by preparing a plurality of sub-base images for one base image. Each sub-base image is an image having the same content but different persons in the image. For example, each sub-base image is an image in which only the person in the image is different. For example, each sub-base image is an image obtained by shooting a movie or an advertisement moving image of the same script using different copyright-free models. For example, each sub-base image is an image obtained by photographing an image of the same concept using a different copyright-free model.

  For example, each sub-base image is photographed using a plurality of copyright-free models having an average face and / or sound, and is most similar to the person shown in the user image before generating a composite image. It is possible to determine a sub-base image in which the copyright-free model is shown, and generate a composite image in step S409 based on the determined sub-base image. As a result, a composite image based on the highly accurate conversion model is generated, and the quality of the composite image is improved. The copyright-free model most similar to the person appearing in the user image can be determined by calculating the similarity between the images using, for example, a known image recognition technique.

  In the above-described example, the generation of the composite image in which the face and the voice are converted has been described, but the present invention is not limited to this. It is also within the scope of the present invention to generate a composite image obtained by converting at least a part of the person shown in the base image and at least a part of the person shown in the user image, which is not limited to the face and the voice. For example, instead of the face conversion model and the voice conversion model described above, a conversion model that converts at least a part of the person shown in the base image and at least a part of the person shown in the user image is constructed. This can be achieved by a process similar to the process described above with reference to FIG. A conversion model that converts at least a part of the person in the base image and at least a part of the person in the user image is, for example, at least a part of the person in the base image obtained from the base image. And a model learned using at least a part of the person in the user image obtained from the user image.

  For example, at least a part of the person appearing in the base image and the user image may be a figure of the person appearing. For example, by performing a process similar to the process described above with reference to FIG. 4 so as to construct a conversion model that converts the body shape of the person shown in the base image and the body shape of the person shown in the user image. Thus, it is possible to generate a composite image obtained by converting the figure of a person shown in the base image and the figure of a person shown in the user image. As a result, it is possible to generate a composite image in which the body shape of an actor appearing in a base image (for example, a movie) is converted into the body shape of the user.

  For example, at least a part of the person appearing in the base image and the user image may be the face, voice, and body of the person appearing. As a result, it is possible to generate a composite image in which the face, voice, and body of an actor appearing in a base image (for example, a movie) are converted into the user's own face and voice body, as if the user himself appeared in the movie This leads to providing the user with a new media experience of seeing a composite image as if he had done it.

  In the above example, a face image is generated by converting the face of the person in the base image into the face of the person in the user image, and the face image is synthesized with the image of the base image to generate a synthesized image. However, the present invention is not limited to this. It is also within the scope of the present invention to generate a face image in which a face of a person in a user image is converted into a face of a person in a base image, and generate a composite image by synthesizing the face image with the user image. Is within.

  This is, for example, to generate an image in which faces of actors, entertainers, athletes, and the like are combined with an image captured by the user. For example, by synthesizing an actor's face with an image of a user who is dancing, it is possible to generate a synthetic image as if the actor is dancing. For example, by combining the face of an entertainer with an image of a user who is a comic artist, it is possible to generate a composite image as if the entertainer is a comic artist. As a result, personalities of actors, entertainers, athletes, and the like can be distributed over time and / or physical, new content is created, and a new media experience can be provided to the user.

  The composite image generated by the above-described processing is provided to the user by the processing described below.

  FIG. 5 shows an example of processing in the computer system 1000 for providing a composite image for a user. In the example shown in FIG. 5, processing for providing a composite image in the computer system 1000 will be described. The example shown in FIG. 5 is a case where a composite image is automatically provided without a user requesting to provide the composite image.

  In step S501, the processor unit 230 of the server device 200 transmits the composite image generated by the processing described with reference to FIG. 4 to the terminal device 100 via the communication interface unit 210. The processor unit 230 of the server device 200 automatically transmits the composite image to the terminal device 100 without receiving a request to provide the composite image from the user. The terminal device 100 receives the composite image via the communication interface unit 110.

  For example, before step S501, the processor unit 230 of the server device 200 may determine the terminal device 100 to which the composite image is to be transmitted. For example, the processor unit 230 of the server device 200 determines the terminal device 100 to which the composite image is to be transmitted, randomly or according to an arbitrary rule. The processor unit 230 of the server device 200 transmits the synthesized image to the terminal device 100 determined at a predetermined timing, for example. The predetermined timing may be, for example, immediately before the terminal device 100 reproduces the predetermined image, immediately after the terminal device 100 reproduces the predetermined image, or while the terminal device 100 reproduces the predetermined image.

  When the combined image is received, the processor unit 150 of the terminal device 100 outputs the combined image via the display unit 130 in step S502.

  When the processing illustrated in FIG. 5 is realized, for example, when a user is watching a moving image on a moving image posting site such as YouTube, a composite advertisement moving image in which the face of the performer is suddenly converted to the user's face in a certain advertisement moving image Will be played. This can be achieved, for example, by moving from a video posting site such as YouTube to a site providing a composite advertisement video provided by the server device 200 at a predetermined timing and providing the composite advertisement video on the site. Alternatively, for example, in the case of a single information processing device described later, the information processing device locally generates a synthetic advertisement moving image without moving from a moving image posting site such as YouTube and reproduces the generated moving image. Can be achieved.

  When the processing shown in FIG. 5 is realized, for example, when the user is watching a TV program, a composite advertisement moving image in which the face of the performer is converted to the user's face in a certain advertisement moving image is suddenly reproduced. become. This can be achieved, for example, when the information processing apparatus is a single information processing apparatus described later, and the information processing apparatus locally creates a composite advertisement image based on the broadcast image and reproduces the composite advertisement image.

  This allows the user of the terminal device 100 to unexpectedly come across a personalized image unexpectedly, thereby allowing the user to have a new media experience. Further, by watching a personalized advertisement video in which the performer's face has been converted to the user's face in a certain advertisement video, the user can, for example, display the product (eg, cosmetics, hairdressing agent, clothes, etc.) of the advertisement. You can virtually experience what it will be like if you use it, and what you will do if you use the advertising service (eg, beauty treatment, training gym). This also leads to a new media experience.

  FIG. 6 shows an example of processing in the computer system 1000 for providing a composite image for a user. In the example shown in FIG. 6, a process for providing a composite image in the computer system 1000 will be described. The example shown in FIG. 6 is a case where the user receives a notification that the composite image can be viewed and requests to provide the composite image, and the composite image is provided.

  In step S <b> 601, the processor unit 230 of the server device 200 transmits a notification that the synthesized image can be provided to the terminal device 100 via the communication interface unit 210. The terminal device 100 receives the notification via the communication interface unit 110. The notification that the composite image can be provided may be, for example, transmitted directly from server device 200 to terminal device 100, or may be transmitted indirectly. What is transmitted indirectly may be, for example, an image being reproduced in the terminal device 100 or a notification embedded in an image to be reproduced. For example, the server device 200 transmits a request to embed a notification to the provider of the image, and the provider of the image embeds the notification in the image in response to the request. Sent to 100.

  Upon receiving the notification, the processor unit 150 of the terminal device 100 provides, via the display unit 130, an interface that allows a user to input a request to provide a composite image. The user can enter a request to provide a composite image via this interface.

  In step S602, when the processor unit 150 of the terminal device 100 receives a request to provide a composite image from the user, in step S603, the processor unit 150 of the terminal device 100 communicates the request to provide the composite image. The data is transmitted to the server device 200 via the interface unit 110. The processor unit 230 of the server device 200 receives this request via the communication interface unit 210.

  Upon receiving the request to provide the composite image, in step S604, the processor unit 230 of the server device 200 transmits the composite image generated by the processing described with reference to FIG. Send to 100. The terminal device 100 receives the composite image via the communication interface unit 110.

  When the combined image is received, the processor unit 150 of the terminal device 100 outputs the combined image via the display unit 130 in step S605.

  When the processing shown in FIG. 6 is realized, for example, when the user is watching a moving image on a moving image posting site such as YouTube, a screen notifying that the synthesized image can be viewed is displayed. When a request to provide an image is input, a composite advertisement moving image in which a performer's face has been converted to a user's face in a certain advertisement moving image is reproduced. This is, for example, by displaying a notification screen on a video posting site such as YouTube, and responding to the notification screen to move to a site that provides a composite advertisement video provided by the server device 200 and to display the composite advertisement video on the site. Can be achieved by providing. Alternatively, for example, in the case of a single information processing device described later, the information processing device locally generates a synthetic advertisement moving image without moving from a moving image posting site such as YouTube and reproduces the generated moving image. Can be achieved.

  When the processing shown in FIG. 6 is realized, for example, when the user is watching a TV program, a screen notifying that the composite image can be viewed is displayed, and the composite image is provided. Is input, a composite advertisement moving image in which the performer's face is converted to the user's face in a certain advertisement moving image is reproduced. This is, for example, in the case of a single information processing device described later, a notification screen is displayed by data broadcasting or the like, and when the notification screen is responded, the information processing device locally generates a synthetic advertisement image And play it back.

  Thereby, the user of the terminal device 100 can unexpectedly encounter a personalized image, and thereby the user can have a new media experience. Further, by watching a personalized advertisement video in which the performer's face has been converted to the user's face in a certain advertisement video, the user can, for example, display the product (eg, cosmetics, hairdressing agent, clothes, etc.) of the advertisement. You can virtually experience what it will be like if you use it, and what you will do if you use the advertising service (eg, beauty treatment, training gym). This also leads to a new media experience.

  In the example shown in FIG. 6, after step S601, an interface may be provided that allows to select a person in the base image that can be substituted in the composite image. Through this interface, the user can select an object to replace in the base image. At this time, the composite image transmitted to the terminal device 100 in step S604 is a composite image obtained by converting at least a part of the selected person in the base image into at least a part of the person in the user image. This composite image may be generated by performing at least a part of the processing illustrated in FIG. 4 after step S603, or may be generated by performing the processing illustrated in FIG. 4 in advance. .

  FIG. 7 shows an example of processing in the computer system 1000 for providing a composite image for a user. In the example illustrated in FIG. 7, a process for providing a composite image in the computer system 1000 will be described. The example shown in FIG. 7 is a case where the user selects a base image that he or she wants to appear on behalf of the user, and a composite image generated based on the selected base image is provided.

  In step S <b> 701, the processor unit 230 of the server device 200 transmits to the terminal device 100 via the communication interface unit 210 the options of a plurality of base images capable of generating a composite image. The terminal device 100 receives the option via the communication interface unit 110.

  Upon receiving the option, the processor unit 150 of the terminal device 100 provides, via the display unit 130, an interface that allows the user to select at least one of a plurality of base image options. Through this interface, the user can select at least one base image that the user wants to appear on behalf of the plurality of base images.

  In step S702, when the processor unit 150 of the terminal device 100 receives an input for selecting a base moving image from the user, in step S703, the processor unit 150 of the terminal device 100 transmits the input for selecting the base moving image to the communication interface unit 110. And transmits it to the server device 200 via the Internet. The processor unit 230 of the server device 200 receives this input via the communication interface unit 210.

  Upon receiving an input for selecting a base moving image, in step S601, the processor unit 230 of the server device 200 transmits a composite image generated based on the selected base image to the terminal device 100 via the communication interface unit 210. . The terminal device 100 receives the composite image via the communication interface unit 110. This composite image may be generated by performing at least a part of the processing illustrated in FIG. 4 after step S703, or may be generated by performing the processing illustrated in FIG. 4 in advance. .

  When the combined image is received, in step S705, the processor unit 150 of the terminal device 100 outputs the combined image via the display unit 130.

  When the processing shown in FIG. 7 is implemented, for example, a plurality of movie choices in which the user can appear are provided, and when the user selects one of the plurality of movies, the face of the performer in the selected movie is selected. Is converted to the face of the user.

  Thereby, the user of the terminal device 100 can see the personalized image according to his / her preference, and thus the user can have a new media experience.

  In the example shown in FIG. 7, after step S701, an interface may be provided that allows to select a person in the base image that can be substituted in the composite image. Through this interface, the user can select an object to replace in the base image. At this time, the composite image transmitted to the terminal device 100 in step S704 is a composite image obtained by converting at least a part of the selected person in the selected base image into at least a part of the person in the user image. This composite image may be generated by performing at least a part of the processing illustrated in FIG. 4 after step S703, or may be generated by performing the processing illustrated in FIG. 4 in advance. .

  In the above-described example, an example in which each step of the processing of each step shown in FIGS. 4, 5, 6, and 7 is executed in the terminal device 100 or the server device 200 has been described, but the present invention is not limited to this. Not done. Each step of the processing of each step shown in FIGS. 4, 5, 6, and 7 can be executed by at least one information processing device including a processor unit. That is, a single information processing apparatus capable of performing both the above-described steps for the terminal apparatus 100 and the above-described steps for the server apparatus 200 is also within the scope of the present invention.

  In the example described above with reference to FIGS. 4, 5, 6, and 7, the processing of each step illustrated in FIGS. 4, 5, 6, and 7 is stored in the processor unit 150 and the memory unit 140. It has been described that the present invention is implemented by a program stored in the processor unit 230 or the memory unit 220, but the present invention is not limited to this. At least one of the processes of each step shown in FIGS. 4, 5, 6, and 7 may be realized by a hardware configuration such as a control circuit.

  The present invention is not limited to the embodiments described above. It is understood that the scope of the present invention should be construed only by the claims. It is understood that those skilled in the art can implement equivalent ranges based on the description of the present invention and common general knowledge from the description of the specific preferred embodiments of the present invention.

  INDUSTRIAL APPLICABILITY The present invention is useful as providing a server device or the like capable of realizing a service that fulfills a desire to impersonate another user. Thereby, it is possible to provide a new media experience to the user.

Reference Signs List 100 terminal device 200 server device 300 database unit 400 network 500 base image provider terminal device 1000 computer system

Claims (22)

An information processing apparatus for generating a composite image for a user,
First acquisition means for acquiring at least one user image;
Second acquisition means for acquiring at least one base image for which modification is permitted;
An information processing apparatus comprising: a generation unit configured to generate a composite image based on the at least one base image and the at least one user image.   The information processing apparatus according to claim 1, further comprising a providing unit configured to provide the composite image to a user.   The information processing apparatus according to claim 2, wherein the providing unit automatically provides the user with the composite image without receiving a request to provide the composite image from the user. The providing means,
Notifying the user that the composite image can be provided;
Receiving a request from the user to provide the composite image;
The information processing apparatus according to claim 2, further comprising: providing the composite image to the user in response to receiving a request to provide the composite image from the user. The second acquisition unit acquires a plurality of base images,
The providing means,
Providing the user with a choice of the plurality of base images;
Receiving from the user an input to select at least one of the plurality of base images;
Responsive to receiving input from the user to select at least one of the plurality of base images, the image generated based on the selected at least one base image and the at least one user image The information processing apparatus according to claim 2, wherein the information processing apparatus performs: providing a composite image to the user. A plurality of persons are shown in the at least one base image;
The providing means,
Providing the user with a choice of a plurality of persons in the at least one base image;
Receiving from the user an input selecting at least one of a plurality of persons in the at least one base image;
Providing the composite image to the user in response to receiving from the user an input to select at least one of the plurality of persons in the at least one base image;
The composite image is a composite image obtained by converting at least a part of the selected at least one person in the at least one base image and at least a part of a person in the at least one user image. Item 6. The information processing apparatus according to any one of Items 2, 4 to 5. The generation means,
7. Generating a composite image obtained by converting at least a part of the at least one base image and at least a part of a person in the at least one user image. 8. Information processing device. The generation means,
Generating a composite image obtained by transforming a person's face in the at least one base image and a person's face in the at least one user image. Information processing device. The base image and the user image include sound,
The generation means,
The information processing apparatus according to claim 1, further comprising: generating a combined image obtained by converting voice of a person in the at least one base image and voice of a person in the at least one user image. The generation means,
The method according to any one of claims 1 to 9, further comprising: generating a combined image obtained by converting a body shape of a person in the at least one base image and a body shape of a person in the at least one user image. Information processing device. The first obtaining means obtains user images of a plurality of persons including a first person and a second person,
The generation means,
Converting at least a portion of a first person in the at least one base image and at least a portion of a first person in the user image, and converting at least one of a second person in the at least one base image; The information processing apparatus according to any one of claims 1 to 10, further comprising: generating a composite image obtained by converting a part and at least a part of a second person in the user image. The generation means,
The at least one at least a portion of the first person in the user image is to be synthesized based on a relationship between a first person in the user image and a second person in the user image. Determining a first person in the base image and determining a second person in the at least one base image to which at least a portion of the second person in the user image is to be combined; The information processing device according to claim 11. Each of the at least one base image includes a set of sub-base images, each sub-base image having the same content, but a different person in the image,
The generation means,
Determining a sub-base image that includes a person most similar to the person in the at least one user image;
Generating a combined image obtained by converting at least a part of the person in the determined sub-base image and at least a part of the person in the at least one user image. The information processing device according to claim 1.   The information processing apparatus according to claim 1, wherein the user image is a user's own image.   The information processing device according to claim 1, wherein the composite image is an advertisement moving image. A program for generating a composite image for a user, wherein the program is executed in an information processing apparatus including a processor unit, and the program includes:
Obtaining at least one user image;
Obtaining at least one base image for which modification is permitted;
Generating a composite image based on the at least one base image and the at least one user image. A method for generating a composite image for a user, wherein the method is performed in an information processing apparatus including a processor unit, the method comprising:
The processor unit acquiring at least one user image;
The processor unit acquiring at least one base image for which modification is permitted;
The processor unit generating a composite image based on the at least one base image and the at least one user image. A terminal device for providing a composite image for a user, wherein the terminal device can communicate with a server device, wherein the terminal device includes:
Acquiring means for acquiring at least one user image;
Transmitting means for transmitting the at least one user image to the server device;
A terminal comprising: a receiving unit that receives, from the server device, a composite image generated based on at least one base image for which modification is permitted and the at least one user image; and an output unit that outputs the composite image. apparatus.   The terminal device according to claim 18, further comprising a receiving unit configured to receive a permission to generate a composite image from a user. A program for providing a composite image for a user, wherein the program is executed in a terminal device including a processor unit, the terminal device can communicate with a server device, and the program includes:
Obtaining at least one user image;
Transmitting the at least one user image to the server device;
Executing a process including, from the server device, receiving a composite image generated based on at least one base image for which modification is permitted and the at least one user image; and outputting the composite image. A program that causes the processor unit to perform the operation. A method for providing a composite image for a user, the method being performed in a terminal device capable of communicating with a server device, the method comprising:
Obtaining at least one user image;
Transmitting the at least one user image to the server device;
Receiving, from the server device, a composite image generated based on at least one base image for which modification is permitted and the at least one user image;
A program that causes the processor unit to execute a process including outputting the composite image. A computer system for generating a composite image for a user, the computer system comprising: a server device; and at least one terminal device capable of communicating with the server device.
The terminal device,
Obtaining at least one user image;
And transmitting the at least one user image to the server device.
The server device,
Receiving from the server device acquiring the at least one user image;
Obtaining at least one base image for which modification is permitted;
Generating a composite image based on the at least one base image and the at least one user image;
Transmitting the composite image to the terminal device.
The terminal device,
Receiving the composite image from the server device;
Outputting the composite image.