JP7534706B2 - PROGRAM, IMAGE PROCESSING APPARATUS, AND IMAGE PROCESSING METHOD - Google Patents

Description

This technology relates to a program, an image processing device, and an image processing method, and in particular to a program, an image processing device, and an image processing method that enable efficient adjustment of the appearance of a subject in a captured image.

In photo taking devices such as passport photo machines and print sticker machines installed in game arcades, the user can adjust the appearance of the captured image. The user can adjust the appearance of the image that is printed on sticker paper or the like, one by one, to their liking.

JP 2015-118329 A

Adjusting the appearance of many images one by one is tedious.

This technology was developed in light of these circumstances, and makes it possible to efficiently adjust the appearance of subjects in captured images.

A program according to one aspect of the present technology is a program that causes a computer to execute a process of identifying people appearing in each captured image using a model generated by machine learning, adjusting the appearance of a first captured image selected as an adjustment target from among a plurality of captured images in response to an operation by a user, and, when an operation is performed to reflect the adjustment content of the first captured image to a second captured image in which the subject's composition is different from that of the first captured image, reflecting the adjustment content of a specific person appearing in the first captured image to a person identical to the specific person appearing in the second captured image, based on the subject's composition of the first and second captured images.

In one aspect of the present technology, the person appearing in each captured image is identified using a model generated by machine learning, and the appearance of a first captured image selected as an adjustment target from among multiple captured images is adjusted according to an operation by a user. Furthermore, when an operation is performed to reflect the adjustment content of the first captured image to a second captured image in which the subject's composition is different from that of the first captured image, the appearance is adjusted so that the adjustment content for a specific person appearing in the first captured image is reflected in a person identical to the specific person appearing in the second captured image based on the subject's composition of the first captured image and the second captured image.

This technology makes it possible to efficiently adjust the appearance of a subject in a captured image.

FIG. 1 is a diagram showing a workflow in a photography studio. FIG. 13 is a diagram showing an example of providing a captured image. FIG. 13 is a diagram showing an example of file management in a retouching PC. FIG. 13 is a diagram showing an example of an image selection screen. FIG. 13 is a diagram showing an example of a retouch screen. FIG. 13 is an enlarged view showing a display of a retouch selection area. FIG. 13 is a diagram showing a display example when batch retouching is selected. 13A and 13B are diagrams illustrating an example of a display when a photographed image to be retouched is switched FIG. 13 is a diagram illustrating an example of application of retouching parameters. FIG. 2 is a block diagram showing an example of the configuration of a retouching PC. FIG. 2 is a block diagram showing an example of the functional configuration of a retouching PC. 11 is a flowchart illustrating a retouching process. 2 is a block diagram showing an example of the configuration of an image processing unit; FIG. 13 is a diagram illustrating an example of extraction of a skin region. FIG. 11 is a diagram illustrating an example of retouching parameters. 13A and 13B are diagrams illustrating an example of a display of an area used for adjusting brightness. 13A to 13C are diagrams illustrating an example of taking over retouching contents. FIG. 18 is a diagram continuing from FIG. 17 showing an example of taking over of retouching contents. FIG. 13 is a diagram showing a first example of taking over retouching contents. FIG. 11 is a diagram showing a second example of the transfer of retouching contents. FIG. 11 is a diagram showing a third example of the takeover of retouching contents. FIG. 13 is a block diagram showing another example of the configuration of the image processing unit.

<<Work flow in the photography studio>>
FIG. 1 is a diagram showing the flow of work in a photography studio.

This technology is used, for example, in photography studios where commemorative photos are taken at weddings. This technology can also be applied to photography studios where commemorative photos are taken for various events, such as entrance/graduation photos, coming-of-age ceremonies, and Shichi-Go-San ceremonies.

As shown on the left side of Figure 1, photography in a photography studio is performed by a photographer. In the example of Figure 1, the subjects of the photo are a bride and groom standing in front of a background screen. The groom is wearing a tuxedo, and the bride is wearing a wedding dress. Around the bride and groom, equipment such as strobe devices that emit light in time with the photographer's photography is placed.

The photographer takes pictures by changing the composition and taking many still images (photographs), such as 50 or 100. The composition of the still images is determined by the poses of the bride and groom, the relative positions of the bride and groom and the photographer, and camera parameters such as focal length. The still images taken by the photographer are transferred to and saved in a PC (not shown) connected to the camera.

After the photo shoot is completed, the bride and groom move to the retouching space prepared in the photography studio and use retouching PC 1 to check the images taken by the photographer, as shown on the right side of Figure 1.

The bride and groom, as users of the retouching PC 1, select the number of images they ultimately want printed, and adjust the appearance of each image by retouching them themselves. Retouching is performed on a portion of the images, such as five or ten images, selected from the images taken by the photographer. Retouching (processing) includes various types of image processing to adjust the appearance of the images.

In the retouching space, one or more retouching PCs 1 are prepared. The captured images stored in the storage PC are transferred to the retouching PC 1 and displayed on the display 11 of the retouching PC 1. A retouching application 1A, which is an application used to retouch images taken by a photographer, is installed on the retouching PC 1.

In this way, in a photography studio where a retouching PC 1 to which the present technology is applied is installed, the users (the bride and groom) themselves retouch the images taken by the photographer. Using the functions of the retouching application 1A, the users can retouch the images to their liking. The retouching application 1A may be installed on a device other than a PC, such as a tablet device, and used for retouching.

Figure 2 shows an example of providing captured images.

The photographed images retouched by retouching application 1A are printed by the photography studio's operating company and later given to the user in the form of, for example, an album, as shown by arrow #1.

The retouched photographed images are uploaded from the retouching PC 1 to the image management server 2, as shown by arrow #2, and are managed by the image management server 2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the management company of the photography studio, or by a business entity different from the management company of the photography studio.

By operating their own mobile terminal 3 such as a smartphone and operating the image management app 3A, users can view and download images that have been taken of them as the subject, as shown by arrow #3. The image management app 3A is an application for viewing, etc., the captured images managed in the image management server 2. The image management app 3A is provided, for example, by the business entity that manages the image management server 2.

The images taken in the photography studio will be provided in the form of an album and, where appropriate, in the form of data. It may be possible to provide the images in the form of data for a fee separate from the fee for the album.

The provision of photographed images in the form of data is carried out so that not only photographed images that have been retouched by the user, but also photographed images that have not been selected by the user as subjects for retouching are provided. Photographed images that have not been selected by the user as subjects for retouching are automatically retouched by the retouching application 1A without any user operation, as described below.

The user can receive in the form of data even captured images that were not selected for retouching. Instead of using the image management app 3A, the user can use a browser to access a specific site and receive captured images managed in the image management server 2. For example, by registering as a member of an image providing service provided by the administrator of the image management server 2, the user can receive captured images managed in the image management server 2 using the image management app 3A or a browser.

Instead of the mobile terminal 3, it may be possible to receive captured images using other terminals such as a tablet terminal or a PC.

<<About the retouching app>>
<File Management>
FIG. 3 is a diagram showing an example of file management in the retouching PC 1. As shown in FIG.

As shown in the speech bubble in Figure 3, two folders, the all folder and the album folder, are prepared and linked to the user ID, which is the user's identification information. The all folder is a folder with the folder name "all," and the album folder is a folder with the folder name "album."

The captured image files transferred from the storage PC are stored in the all folder. The captured image files that were selected for retouching and have been retouched are stored in the album folder. The files stored in the album folder are the files used for printing.

When a user who has finished taking a photograph starts retouching, the user ID is entered on the screen of retouching application 1A. Based on the user ID, the captured image is read by retouching application 1A and processing is performed using the captured image. For example, the files of all captured images stored in the all folder linked to the user ID are read by retouching application 1A. After the captured images are read, the captured images to be retouched are selected.

<Screen display>
Image Selection Screen FIG. 4 is a diagram showing an example of an image selection screen.

A message is displayed at the top of the image selection screen instructing the user to select five captured images. The number of selected captured images is set, for example, by a staff member at the photography studio before image selection. If the user has signed up for a plan to receive five captured images printed as an album, five is set as the number of selected images.

A thumbnail display area 21 is formed on the right side of the image selection screen. The thumbnail display area 21 is an area in which thumbnail images of the captured images are displayed. For example, if 50 captured images have been taken, thumbnail images of each of the 50 captured images are displayed. Various operations using the screen of the retouching application 1A are performed using a mouse, keyboard, etc.

A photographed image selected using a thumbnail image is enlarged and displayed in a display area 22 formed to the left of the thumbnail display area 21. In the example of FIG. 4, the thumbnail image of photographed image P1 displayed at the upper left corner of the thumbnail display area 21 is selected, and photographed image P1 is enlarged and displayed in the display area 22.

A confirmation button 23 is displayed at the bottom right of the display area 22. The confirmation button 23 is pressed when deciding on the photographed image displayed in the display area 22 as the one to be retouched. The user selects thumbnail images to sequentially enlarge and display each photographed image in the display area 22, and when the user finds a photographed image that the user likes, the user presses the confirmation button 23 to select it as the one to be retouched.

Below the display area 22, a selected image display area 24 is formed in which a thumbnail image of the captured image selected by the user is displayed.

When the selection of five photographed images has been completed, the selection of photographed images to be retouched is completed. After the selection of photographed images to be retouched has been completed, retouching is performed on each photographed image selected as the subject of retouching.

Retouch Screen FIG. 5 is a diagram showing an example of a retouch screen.

A display area 31 is formed at the upper center of the retouch screen. The display area 31 is an area in which the photographed image currently being retouched is displayed in an enlarged manner. A thumbnail area 32 formed on the left side of the display area 31 displays a thumbnail image of the photographed image selected by the user. In the example of FIG. 5, thumbnail images of five photographed images P1 to P5 are displayed. The photographed image P1, which is selected using the thumbnail image, is displayed in an enlarged manner in the display area 31.

A horizontally long retouching selection area 33 is formed below the retouching screen. The buttons displayed in the retouching selection area 33 are used to select the retouching content. Each time a retouching content is selected using a button displayed in the retouching selection area 33, the selected content is reflected in the display in the display area 31. While viewing the display in the display area 31, the user can use the display in the retouching selection area 33 to select the retouching content to be applied to the captured image currently being retouched.

Figure 6 shows an enlarged view of the retouch selection area 33.

The retouching selection area 33 is made up of area 33-1, which is a display area for information used to adjust the entire face and body, area 33-2, which is a display area for information used to adjust the eyes and makeup, and area 33-3, which is a display area for information used to adjust brightness.

In area 33-1 formed to the left of retouch selection area 33, adjustment button 51 used to adjust the appearance of a smaller face, adjustment button 52 used to adjust the shape of the face, adjustment button 53 used to adjust the skin texture, adjustment button 54 used to adjust the shorter chin, and adjustment button 55 used to adjust the style.

Five different sizes can be selected to achieve a smaller face. A smaller face size of "0" indicates that the size of the face is not changed and remains at its original size. By selecting one of the sizes "1" to "4," the user can adjust the size of their face in the captured image displayed in the display area 31.

The user can select one of three face shapes: "round," "egg," or "slim." By selecting one of these shapes, the user can adjust the shape of their face that appears in the captured image displayed in the display area 31. By selecting "no retouching," it is also possible to leave the shape of the face unchanged.

It is possible to select from five levels of skin texture. Skin texture "0" indicates that the skin texture is not changed and remains the original skin texture. By selecting one of the levels, the user can adjust the degree of smoothness of their skin texture as it appears in the captured image displayed in the display area 31.

Three levels of chin shortening can be selected. Chin shortening "0" indicates no processing, i.e. the chin is left unchanged and the original chin is kept as is. By selecting one of the levels, the user can adjust the length of their chin that appears in the captured image displayed in the display area 31.

It is possible to select from three levels of style improvement. Style improvement "0" indicates no processing, i.e. the style is left unchanged and remains the original style. By selecting one of the levels, the user can adjust their own style as it appears in the captured image displayed in the display area 31.

In area 33-2 formed in the center of retouch selection area 33, adjustment button 56 used to adjust eye size, adjustment button 57 used to adjust eye shape, adjustment button 58 used to adjust makeup intensity, adjustment button 59 used to adjust nose bridge, and adjustment button 60 used to adjust facial hair removal are displayed.

It is possible to select from five different eye sizes. Eye size "0" indicates no processing, i.e. the eye size is left unchanged at its original size. By selecting one of the sizes, the user can adjust the size of their own eyes as they appear in the captured image displayed in the display area 31.

The user can select one of the following eye shapes: "round eyes," "downturned eyes," or "cat eyes." By selecting one of these shapes, the user can adjust the shape of their own eyes that appear in the captured image displayed in the display area 31. By selecting "no processing," it is also possible to leave the shape of the eyes unchanged.

The intensity of makeup can be selected from five levels. The level of makeup "0" indicates that the intensity of makeup is not changed and remains the original intensity. By selecting one of the levels, the user can adjust the intensity of their makeup that appears in the captured image displayed in the display area 31.

It is possible to select ON/OFF for the nose bridge. By selecting ON or OFF, the user can choose whether to emphasize the highlights of the bridge of their nose that appear in the captured image displayed in the display area 31, or to leave it in its original state.

Three levels of beard erasure can be selected. Beard erasure level "0" indicates that the beard is not changed and remains the original beard. By selecting one of the levels, the user can adjust the beard of the user that appears in the photographed image displayed in the display area 31. The beard erasure function is enabled when the person to be retouched is the groom.

In area 33-3 formed to the right of retouch selection area 33, adjustment button 61 used to adjust the brightness of only the skin, adjustment button 62 used to adjust the overall brightness, and adjustment button 63 used to adjust the brightness of only the background are displayed.

Three levels of brightness can be selected for adjusting the skin brightness. Skin brightness of "0" indicates that the skin brightness is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the brightness of their own skin that appears in the captured image displayed in the display area 31. Details of the skin brightness adjustment function will be described later.

Three levels of brightness can be selected to adjust the overall brightness. An overall brightness of "0" indicates that the overall brightness is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the overall brightness of the captured image displayed in the display area 31.

Three levels of brightness can be selected to adjust the background brightness. A background brightness of "0" indicates that the brightness of the background is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the brightness of the background that appears in the captured image displayed in the display area 31.

Returning to the explanation of FIG. 5, a batch retouch button 34 used for batch retouching is displayed on the left side of the display area 31. Batch retouching is a function for selecting a combination of various retouching contents as described above all at once.

In the example of FIG. 5, a user can select one of three levels of batch retouching, each of which has a different combination of retouching content levels. By using the batch retouching function, the user can select the retouching content with a single operation, without having to select each retouching content using the display in the retouching selection area 33.

Figure 7 shows an example of the display when batch retouching is selected.

When the batch retouch button 34 is operated, a rounded rectangular pop-up image 71 is displayed superimposed on the retouch screen. The pop-up image 71, which is displayed approximately in the center of the retouch screen, displays a circular button used to select the range of images to be subjected to batch retouching together with a message.

By operating the button displayed in the pop-up image 71, the user can select whether the batch retouching contents should be applied to only the one image currently being retouched, or to all (five) images including the one image currently being retouched.

Returning to the explanation of FIG. 5, a Before-processing button 35 is provided on the right side of the display area 31. The Before-processing button 35 is a button that is operated when switching the display of the display area 31 to the state before retouching. While the Before-processing button 35 is pressed, the captured image in its pre-retouched state is displayed in the display area 31. By using the Before-processing button 35, the user can compare the post-retouched state with the pre-retouched state to check whether the degree of retouching is too strong, for example.

Below the Before button 35 is a Person Selection button 36 used to select a person to be retouched. By using the Person Selection button 36, the user can select to retouch only the person appearing on the left side of the captured image, only the person appearing on the right side, or both people. Face recognition and other processes are performed on the captured image, and the person to be the subject of image processing for retouching is selected in response to the user's operation.

Using these various functions, the user can select the retouching content to be applied to the captured image displayed in the display area 31. In this way, retouching that can be performed using the retouching application 1A includes at least adjustments to the face and body as a whole, adjustments to the eyes and makeup, and adjustments to brightness.

The photographed image to be retouched is switched using the display in the thumbnail area 32, and retouching is performed on each of the photographed images P1 to P5 in sequence. The retouched files of the photographed images P1 to P5 are stored in the album folder.

Figure 8 shows an example of the display when the captured image to be retouched is switched.

When the photographed image to be retouched is switched, a pop-up image 72 in the shape of a speech bubble is displayed superimposed on the retouching screen. The pop-up image 72 is displayed at a position slightly below the center of the retouching screen, so that the speech bubble originates from the display area 31 formed approximately above the center of the retouching screen. The pop-up image 72 displays a horizontally long rectangular button together with a message, which is used to select whether or not the retouching contents set on the previous photo (photographed image) should be reflected in the switched photographed image.

In other words, when the captured image to be retouched is switched, an image with a different shape from the pop-up image 71 that is displayed when the batch retouch button 34 is operated is displayed as the pop-up image 72.

By displaying the pop-up image 72, the user can carry over the retouching content selected for the previous captured image to the new captured image. The retouching content carryover function will be described in detail later.

Figure 9 shows an example of applying retouching parameters.

As shown in FIG. 9, the retouching content applied to each of the captured images P1 to P5 by user operations is reflected in each unselected image stored in the all folder. Unselected images are captured images that were not selected as targets for retouching on the image selection screen. Captured images selected as targets for retouching on the image selection screen become selected images.

Which of the retouched images P1 to P5 should have its retouching applied to each unselected image is selected based on, for example, the state of the face in the selected image and the state of the face in the unselected image. In this case, each captured image is analyzed, and the state of the face in each captured image is recognized. A selected image in which a face appears in a state similar to the state of the face in each unselected image is selected, and the retouching applied to the selected selected image is reflected in the unselected image.

<<Configuration and operation of retouching PC 1>>
<Configuration of retouching PC 1>
FIG. 10 is a block diagram showing an example of the configuration of the retouching PC 1. As shown in FIG.

CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103 are interconnected by bus 104.

An input/output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a memory unit 108, a communication unit 109, and a drive 110 are connected to the input/output interface 105.

The input unit 106 is composed of a keyboard, mouse, etc.

The output unit 107 is composed of a display 11, etc.

The storage unit 108 is composed of a hard disk, a non-volatile memory, etc. The storage unit 108 stores various information such as programs executed by the CPU 101, such as the retouching application 1A, and captured images.

The communication unit 109 is an interface to a network such as the Internet or a LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a photographed image sent from the storage PC, in which the user about to begin retouching appears as a subject. The communication unit 109 also communicates with the image management server 2, and sends the retouched photographed image to the image management server 2.

The drive 110 controls writing of data to the removable medium 111 and reading of data from the removable medium 111. Import of captured images from a PC for storage may be performed via the removable medium 111.

Figure 11 is a block diagram showing an example of the functional configuration of retouching PC 1.

As shown in FIG. 11, in the retouching PC 1, the retouching application 1A is executed to realize an image processing unit 121. The image processing unit 121 is composed of an image acquisition unit 131, a retouching processing unit 132, an output control unit 133, and a display control unit 134.

The image acquisition unit 131 accepts a user's operation on the image selection screen, and acquires the captured image selected by the user as the target for retouching from, for example, the storage unit 108. The captured image to be retouched acquired by the image acquisition unit 131 is supplied to the retouching processing unit 132 and the display control unit 134.

In addition, when retouching of each captured image to be retouched is completed, the image acquisition unit 131 acquires an unselected image, for example, from the memory unit 108, and outputs it to the retouching processing unit 132.

The retouching processing unit 132 performs face recognition to recognize the face appearing in the captured image to be retouched. The retouching processing unit 132 analyzes the features of each part of the recognized face and performs various image processing related to retouching in response to the user's operation on the retouching screen. The analysis of the features of each part may be performed using an inference model generated by machine learning.

For example, when an operation is performed to adjust the size or shape of the eyes, the retouching processor 132 performs image processing such as shifting the position of the eye outline in accordance with the user's selection. When an operation is performed to adjust the intensity of makeup, the retouching processor 132 performs image processing such as compositing an image of a blush of a specified color onto the cheeks, or compositing an image of a lipstick of a specified color onto the lips.

When an operation related to adjusting the skin texture is performed, the retouching processing unit 132 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the face size (feeling of a small face) or face shape is performed, the retouching processing unit 132 performs image processing such as shifting the position of the face outline according to the content selected by the user.

Information representing the results of image processing by the retouching processing unit 132 is supplied to the display control unit 134 and used to display the retouched screen.

When retouching of each photographed image to be retouched is completed, the retouching processing unit 132 analyzes each photographed image including the unselected image, and reflects the retouching content applied to the retouched photographed image to the unselected image. The photographed images retouched by the retouching processing unit 132 are supplied to the output control unit 133.

The output control unit 133 saves the captured image that has been retouched in response to user operations in the album folder.

The output control unit 133 also outputs the retouched captured images (selected images and unselected images) to the communication unit 109 and transmits them to the image management server 2.

The display control unit 134 controls the display of various screens of the retouching application 1A, such as the image selection screen and the retouching screen.

<Operation of retouching PC 1>
The retouching process performed by the retouching application 1A will be described with reference to the flowchart of FIG.

In step S1, the display control unit 134 of the image processing unit 121 displays the image selection screen (Figure 4).

In step S2, the image acquisition unit 131 accepts the selection of the captured image to be retouched in response to a user's operation on the image selection screen. The image acquisition unit 131 acquires the captured image to be retouched.

In step S3, the image acquisition unit 131 determines whether or not the selection of the captured images to be retouched has been completed. If it is determined in step S3 that the selection of the captured images to be retouched has not been completed, the process returns to step S1, and image selection continues.

For example, if five captured images have been selected, it is determined in step S3 that the selection of captured images to be retouched has ended.

In step S4, the display control unit 134 displays a retouch screen (FIG. 5). The user uses the retouch screen to retouch the captured image.

In step S5, the retouching processing unit 132 performs image processing in response to the user's operation, and retouches the captured image selected as the target for retouching.

In step S6, the retouching processing unit 132 determines whether or not retouching is complete. If it is determined in step S6 that retouching is not complete, the process returns to step S4, and retouching continues for each captured image selected as the target for retouching.

For example, when all the captured images selected for retouching have been retouched, it is determined in step S6 that retouching is to end.

In step S7, the retouching processing unit 132 applies the same retouching content applied to the selected image to each of the unselected images.

In step S8, the output control unit 133 saves the retouched unselected image.

In step S9, the communication unit 109 transmits the retouched selected images and the unselected images to the image management server 2. After that, the processing of the retouching application 1A ends.

By performing the above process, the user can adjust the appearance of the images taken of themselves as the subject. The user can receive an album of the images that have been taken to their liking.

In addition, by using the mobile terminal 3, the user can receive, in the form of data, photographed images that have been manually adjusted and photographed images (unselected images) that have been automatically retouched in the same manner as the manual adjustments, which are managed by the image management server 2.

＜＜Skin brightness adjustment function＞＞
5, it is possible to adjust the brightness of the photographed image to be retouched. Here, a function for adjusting the brightness of only the skin, which is performed using the adjustment button 61 in the area 33-3, will be described.

<Configuration of image processing unit 121>
13 is a block diagram showing an example of the configuration of the image processing unit 121 that realizes the function of adjusting the brightness of only the skin. In FIG. 13, only a part of the configuration of the image processing unit 121 is shown.

In addition to the components described above, the image processing unit 121 is provided with a gender determination unit 135. In the retouching processing unit 132, a face recognition unit 151, a skin region extraction unit 152, and a brightness adjustment unit 153 are realized. The captured image to be retouched is input to the gender determination unit 135 and also input to each unit of the retouching processing unit 132.

The gender determination unit 135 determines the gender of people appearing in the captured image. The determination by the gender determination unit 135 is performed, for example, using an inference model that inputs the captured image and outputs the gender of each person appearing in the captured image. In this case, an inference model for gender determination generated by machine learning is prepared in advance in the gender determination unit 135. Gender determination may be performed based on the characteristics of a person obtained by analyzing the captured image to be retouched.

The determination result by the gender determination unit 135 is supplied to the brightness adjustment unit 153 of the retouching processing unit 132. For example, if the photographed image to be retouched contains only the bride, information indicating that the person in the photographed image is a woman is supplied as the determination result. Also, if the photographed image to be retouched contains only the groom, information indicating that the person in the photographed image is a man is supplied as the determination result. If the photographed image to be retouched contains both the bride and groom, information indicating that the people in the photographed image are a woman and a man, respectively, is supplied as the determination result.

The face recognition unit 151 of the retouching processing unit 132 recognizes the face of a person appearing in a captured image and extracts a face region. The extraction of the face region by the face recognition unit 151 may be performed using an inference model generated by machine learning, or may be performed based on features such as the contour points of each part of the body determined by analyzing the captured image. Information on the face region extracted by the face recognition unit 151 is supplied to the skin region extraction unit 152 and the brightness adjustment unit 153.

The skin region extraction unit 152 extracts the skin region of a person appearing in a captured image. For example, an inference model is prepared in the skin region extraction unit 152, which receives a captured image as input and outputs information about the skin region of a person appearing in the captured image. The skin region extraction unit 152 extracts skin regions other than the face region based on the skin region extracted using the inference model and the face region extracted by the face recognition unit 151.

Figure 14 shows an example of skin region extraction by the skin region extraction unit 152.

For example, a case will be described in which the photographed image P11 shown in the upper left of FIG. 14 is selected as the target for retouching. The photographed image P11 shows a person H1, a bride wearing a wedding dress. The skin of person H1 is exposed from the chest to the areas from the shoulders to the arms (upper arms).

When such a captured image P11 is to be retouched, the face recognition unit 151 extracts a face area as shown at the tip of arrow #11.

The skin region extraction unit 152 inputs the captured image P11 to the skin region extraction model M as indicated by the tip of the arrow #12, and extracts the skin region including the face region based on the output of the skin region extraction model M as indicated by the tip of the arrow #13. In the captured image P11, the face region and the region from the shoulders to the arms are captured as areas where the skin of person H1 is exposed.

In addition, as shown by the tips of arrows #14 and #15, the skin region extraction unit 152 extracts the skin region other than the face region, as shown by the tip of arrow #16, by removing the face region extracted by the face recognition unit 151 from the skin region extracted using the skin region extraction model M. Information on the skin region other than the face region extracted in this manner is supplied to the brightness adjustment unit 153.

The brightness adjustment unit 153 in FIG. 13 identifies the face area of a person appearing in a captured image to be retouched based on information supplied from the face recognition unit 151, and identifies the skin area other than the face area based on information supplied from the skin area extraction unit 152. The brightness adjustment unit 153 adjusts the face area and the skin area other than the face area using different retouching parameters. The retouching parameters are parameters that specify the amount of brightness adjustment.

For example, the brightness adjustment unit 153 selects retouching parameters in response to the user's operation using the adjustment button 61 such that the adjustment amount for the face region is greater than the adjustment amount for the skin region other than the face region, and adjusts the brightness of each region. That is, the retouching parameters for the brightness of the face region that are brighter than the retouching parameters for the brightness of the skin region other than the face region are used in image processing. As described with reference to FIG. 5, one of levels 1 or 2 is selected as the brightness level for only the skin using the adjustment button 61.

As a result, regardless of whether the user selects brightness level 1 or 2, image processing is applied to make the face area brighter than the skin area other than the face area, as shown in FIG. 15A. For example, if the brightness of the face area and the brightness of the skin area other than the face area are the same, the face area after the brightness adjustment will be brighter than the skin area other than the face area.

When the person in the captured image is a woman (the bride), makeup is usually applied to the face, so brightening the facial area does not look unnatural. By applying image processing to brighten the facial area more than the skin area other than the facial area, it is possible to prevent the skin area other than the facial area from becoming too bright and resulting in an unnatural appearance.

Such image processing, which brightens the face area more than the skin area other than the face area, may be performed based on the gender determination result by the gender determination unit 135.

For example, image processing that brightens the face area more than the skin area other than the face area is only applied to women who appear in the captured image. In this case, image processing is applied to men who appear in the captured image using retouching parameters that brighten the face area and the skin areas other than the face area, such as the hands, to the same extent. This makes it possible to prevent the face area of the man from becoming too bright and appearing unnatural.

Even if the user selects the same brightness level, retouching parameters may be selected that make the female skin area (including the facial area) brighter than the male skin area, as shown in FIG. 15B. In this way, different retouching parameters may be used to adjust the brightness of the female skin area and the male skin area.

<Displaying the retouch screen>
FIG. 16 is a diagram showing a display example of the area 33-3 on the retouch screen.

After adjusting the brightness of only the skin using adjustment button 61 as described above, the user can further adjust the overall brightness using adjustment button 62, or adjust the brightness of the background using adjustment button 63.

In contrast, after adjusting the overall brightness using adjustment button 62, adjustment of the background brightness using adjustment button 63 is restricted, as shown in FIG. 16A. In the example on the right side of FIG. 16A, the overall brightness level has been selected as "1," and adjustment button 63 is displayed grayed out to prevent selection of the background brightness level.

In addition, after adjusting the background brightness using the adjustment button 63, adjustment of the overall brightness using the adjustment button 62 is restricted, as shown in FIG. 16B. In the example on the right side of FIG. 16B, the background brightness level has been selected as "2," and therefore the adjustment button 62 is displayed grayed out, making it impossible to select the overall brightness level.

In other words, the adjustment of the overall brightness and the adjustment of the background brightness are mutually exclusive, and it is possible to adjust only one of them. After one of the adjustments of the overall brightness and the background brightness has been performed, the adjustment of the other is restricted. The adjustment of the overall brightness and the adjustment of the background brightness are also performed by, for example, the brightness adjustment unit 153.

For example, if you were to brighten the background after brightening the entire image, the image processing that adjusts the brightness would be applied to the background in a superimposed manner, increasing the likelihood of whiteout. By making it possible to adjust only one of the brightness adjustments, either the overall brightness or the background brightness, it is possible to prevent such whiteout from occurring.

Even after adjusting the overall brightness using adjustment button 62, or adjusting the background brightness using adjustment button 63, the display of adjustment button 61 remains (remains unchanged) so that the brightness level of only the skin remains selectable.

Adjustment buttons 61 to 63 are displayed as information used to select the brightness adjustment level, but other selection information such as a slide bar may also be displayed.

<< Transfer of retouching content >>
As described with reference to Fig. 8, when the photographed image to be retouched is switched, the retouching contents selected for the photographed image before the switch can be carried over to the photographed image after the switch. Here, the retouching contents carrying over function will be described. It is assumed that the retouching contents set for the previous photographed image are selected to be reflected in the photographed image after the switch using the display of the pop-up image 72.

<Transfer of retouching content>
17 and 18 are diagrams showing an example of taking over the retouching contents.

As shown by arrow #21 in FIG. 17, when the captured image P21 is the first image to be retouched, the user operates the camera to set the following retouching parameters for the captured image P21: "small face feeling" to "3," "face shape" to "egg," "skin texture" to "2," and so on. The appearance of the retouched captured image P21 shown in the lower part of FIG. 17 is different from the appearance of the captured image P21 before retouching shown in the upper part.

When retouching of the photographed image P21 is completed and the retouching target is switched from the photographed image P21 to the photographed image P22, the retouching parameters of the retouched photographed image P1 are inherited as the default retouching parameters of the photographed image P22, which is the second photographed image to be retouched, as shown at the end of arrow #22 in FIG. 18. In the display area 31, the photographed image P22 is displayed in place of the photographed image P21, with the same retouching applied to it as the retouched photographed image P21. The display of the retouching selection area 33 also changes according to the retouching applied to the photographed image P22.

The user can start retouching the photographed image P22 using the photographed image P22, which has been retouched in the same manner as the photographed image P21, as a reference. In the example of FIG. 18, the photographed image P22 has been adjusted to the contents represented by the retouching parameters, such as "small face feeling" is "4", "face shape" is "slim", "skin quality" is "1", and so on.

Similarly, when retouching of photographed image P22 is completed and a third photographed image to be retouched is selected, the retouching parameters of photographed image P22 are carried over as the default retouching parameters for the third photographed image to be retouched, as shown by arrow #24.

In this way, when the photographed image to be retouched is switched, the retouching contents of the photographed image to be retouched immediately before are carried over to the appearance of the next photographed image. Each time the photographed image to be retouched is switched, the carrying over of the appearance of the photographed images is repeated. This allows the user to efficiently retouch each photographed image.

If the captured image after switching is an unadjusted image for which no retouching adjustments have been made, the retouching details may be carried over as described above.

<About transfer target>
Retouching can be performed for each person by selecting the person to be retouched. The retouching content is passed on based on the gender determination result by the gender determination unit 135 so that the retouching content for the bride is passed on to the bride's image, and the retouching content for the groom is passed on to the groom's image.

Switching from an image showing the bride and groom to another image showing the bride and groom FIG. 19 is a diagram showing a first example of the handover of retouching contents.

Figure 19 shows an example of the transfer of retouching content when retouching of a photographed image P31 showing a bride H1 and a groom H2 is completed and the retouching target is switched from the photographed image P31 to a photographed image P32 also showing the persons H1 and H2.

In this case, the retouching content applied to person H1 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H1 in photographed image P32, as indicated by arrow #31. Based on the determination result by gender determination unit 135, the retouching content applied to the female person (person H1) in photographed image P31 is carried over to the appearance of the same female person (person H1) in photographed image P32.

In addition, the retouching content applied to person H2 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H2 in photographed image P32, as shown by arrow #32. Based on the determination result by gender determination unit 135, the retouching content applied to the male person (person H2) in photographed image P31 is carried over to the appearance of the same male person (person H2) in photographed image P32.

Switching from an image showing the bride and groom to an image showing either the bride or groom FIG. 20 is a diagram showing a second example of the handover of retouching contents.

Figure 20 shows an example of the transfer of retouching content when retouching of a photographed image P31 showing a person H1 who is the bride and a person H2 who is the groom is completed and the retouching target is switched from the photographed image P31 to a photographed image P33 showing only person H1 or a photographed image P34 showing only person H2.

When the retouching target is switched to photographed image P33 at the bottom right, the retouching content applied to person H1 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H1 in photographed image P33, as shown by arrow #33. Based on the determination result by gender determination unit 135, the retouching content applied to the female person (person H1) in photographed image P31 is carried over to the appearance of the same female person (person H1) in photographed image P33.

In this case, the retouching content applied to person H2 when photographed image P31 was selected as the retouching target will not be carried over.

When the retouching target is switched to the photographed image P34 at the bottom left, the retouching content applied to person H2 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H2 in photographed image P34, as shown by arrow #34. Based on the determination result by gender determination unit 135, the retouching content applied to the male person (person H2) in photographed image P31 is carried over to the appearance of the same male person (person H2) in photographed image P34.

In this case, the retouching content applied to person H1 when photographed image P31 was selected as the retouching target will not be carried over.

Switching from an image showing the groom or the bride to an image showing the groom and the bride FIG. 21 is a diagram showing a third example of the handover of retouching contents.

The right side of Figure 21 shows an example of the transfer of retouching content when retouching of a photographed image P33, which contains only a person H1 who is the bride, is completed and the retouching target is switched from the photographed image P33 to a photographed image P31, which contains person H1 and a person H2 who is the groom.

In this case, the retouching content applied to person H1 when photographed image P33 was selected as the retouching target is carried over to the appearance of person H1 in photographed image P31, as shown by arrow #35. Based on the determination result by gender determination unit 135, the retouching content applied to the female person (person H1) in photographed image P33 is carried over to the appearance of the same female person (person H1) in photographed image P31.

The left side of Figure 21 shows an example of the transfer of retouching content when retouching of photographed image P34, which contains only person H2, the groom, is completed and the retouching target is switched from photographed image P34 to photographed image P31, which contains person H2 and person H1, the bride.

In this case, the retouching content applied to person H2 when photographed image P34 was selected as the retouching target is carried over to the appearance of person H2 in photographed image P31, as shown by arrow #36. Based on the determination result by gender determination unit 135, the retouching content applied to the male person (person H2) in photographed image P34 is carried over to the appearance of the same male person (person H2) in photographed image P31.

This will prevent issues such as retouching intended for the bride being carried over to the groom's photo.

Example of Allowing User to Select Whether or Not to Carry Over Retouching Contents When the gender composition of people before and after switching of the photographic image to be retouched is different, information such as that described with reference to Fig. 8 may be presented. The user is allowed to select whether or not to carry over the retouching contents.

For example, as shown on the right side of Figure 21, when the retouching target is switched from a photographed image P33 that contains only women to a photographed image P31 that contains both women and men, if the gender composition of the people in the photographed image changes, the composition changes significantly and it may not be appropriate to carry over the retouching content. If inappropriate retouching content is carried over, it becomes cumbersome to correct it, but by making it possible to select not to carry over the retouching content as appropriate, it is possible to eliminate the need for correction.

- Transfer of retouching contents based on person identification results Although the transfer of retouching contents is based on the gender determination results in the above, it may be based on the person identification results. By transferring the retouching contents of a certain person to a photograph of the same person, the retouching contents of the bride are transferred to the photograph of the same person, and the retouching contents of the groom are transferred to the photograph of the same person, the groom.

Fig. 22 is a block diagram showing another example configuration of the image processing unit 121. The configuration shown in Fig. 22 differs from the configuration of the image processing unit 121 described above in that a person identification unit 136 is provided.

The person identification unit 136 identifies each person appearing in the captured image. The person identification unit 136 identifies the person using, for example, an inference model that uses the captured image as input and outputs the ID of the person appearing in the captured image. In this case, an inference model for person identification generated by machine learning is prepared in advance in the person identification unit 136. The person may be identified based on the characteristics of the person obtained by analyzing the captured image to be retouched.

The identification result by the person identification unit 136 is supplied to the retouching processing unit 132. For example, if only the bride appears in the photographed image to be retouched, the person ID assigned to the bride appearing in the photographed image is supplied as the identification result. Also, if only the groom appears in the photographed image to be retouched, the person ID assigned to the groom appearing in the photographed image is supplied as the identification result. If the bride and groom appear in the photographed image to be retouched, the person ID assigned to the bride and the person ID assigned to the groom are supplied as the identification result.

In the retouching processing unit 132, based on the identification result by the person identification unit 136, retouching is performed so that the retouching content applied to the person appearing in the captured image before the retouching target is switched is carried over to the appearance of the same person appearing in the captured image after the switch.

This also makes it possible to prevent problems such as retouching intended for the bride being carried over to the groom's photo.

If the composition of a person is different before and after the captured image to be retouched is switched, information such as that described with reference to FIG. 8 may be presented. The user is prompted to select whether or not to carry over the retouching content.

For example, as shown on the right side of Figure 21, when the configuration of people in a captured image changes, such as when the retouching target is switched from captured image P33, which contains only person H1, to captured image P31, which contains people H1 and H2, the composition changes significantly and it may not be appropriate to carry over the retouching content. If inappropriate retouching content is carried over, it becomes cumbersome to correct it, but by making it possible to select not to carry over the retouching content as appropriate, it is possible to eliminate the need for correction.

<<Modifications>>
- Examples of retouching locations In the above embodiment, retouching is performed using a PC installed in a photography studio, but retouching may also be performed using other devices, such as a mobile terminal 3 or a PC installed in the user's home.

When retouching using the mobile terminal 3 is possible, after a user has finished taking photos in a photography studio, the user operates the mobile terminal 3 to launch the image management app 3A and performs retouching on the screen of the image management app 3A. The image management app 3A has the same functions as the retouching app 1A.

The above-mentioned screen is displayed on the display of the mobile terminal 3 by the image management application 3A, and the photographed images to be retouched are selected and retouched from among the photographed images uploaded to the image management server 2. The photographed images that have been retouched using the functions of the image management application 3A are printed and provided to the user in the form of an album, or the retouched photographed images are provided to the user as is in the form of data.

This allows users to retouch photos in various locations, such as at home. Users can also retouch photos to their heart's content without worrying about time.

Regarding the program: The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware, or in a general-purpose personal computer.

The program to be installed is provided by being recorded on removable media 111 shown in FIG. 10, which may be an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. It may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in ROM 102 or storage unit 108.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

The effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiment of this technology is not limited to the above-mentioned embodiment, and various modifications are possible without departing from the gist of this technology.

For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

In addition, each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices.

Furthermore, when a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device, or can be shared and executed by multiple devices.

1 Retouching PC, 2 Image management server, 3 Mobile terminal, 121 Image processing unit, 131 Image acquisition unit, 132 Retouching processing unit, 133 Output control unit, 134 Display control unit, 135 Gender determination unit, 136 Person identification unit, 151 Face recognition unit, 152 Skin area extraction unit, 153 Brightness adjustment unit

Claims (8)

On the computer,
The people in each image are identified using a model generated by machine learning.
adjusting the appearance of a first captured image selected as an adjustment target from among the plurality of captured images in response to an operation by a user;
When an operation for reflecting the adjustment contents of the first photographed image to a second photographed image in which the composition of the subjects differs from that of the first photographed image, the program executes a process for reflecting the adjustment contents of a specified person appearing in the first photographed image to a person identical to the specified person appearing in the second photographed image, based on the composition of the subjects of the first photographed image and the second photographed image. When a female person and a male person are captured in the first photographed image and only a female person is captured as a subject person in the second photographed image, the adjustment content for the female person who is the specified person captured in the first photographed image is reflected in the female person who is the same person as the female person captured in the second photographed image,
2. The program according to claim 1, wherein, when the first captured image contains only a female person as a subject and the second captured image contains both a female person and a male person, adjustments made to the female person, who is the specified person, appearing in the first captured image , are reflected in the female person, who is the same person, appearing in the second captured image. The program according to claim 1 , further comprising: after adjusting the appearance of a plurality of captured images, reflecting the adjustment contents of the appearance according to a user's operation on each captured image that has not been adjusted in appearance by a user's operation. The program according to claim 3, wherein the captured image as the source and destination of the adjustments to appearance are selected based on the state of the faces appearing in each of the captured images that have been adjusted in appearance in accordance with user operation and the state of the faces appearing in each of the captured images that have not been adjusted in appearance. displaying a screen on which thumbnail images of the respective photographed images, the first photographed image selected as an adjustment target, and a plurality of pieces of information used for adjusting the appearance of the photographed image are arranged;
The program according to claim 1 , wherein an operation for reflecting the adjustment content of the first captured image in the second captured image is performed on the screen. 2. The program according to claim 1, wherein the adjustment of appearance includes at least one of adjustment of face size, adjustment of face shape, adjustment of chin length, adjustment of style, adjustment of eye size, adjustment of eye shape, adjustment of makeup intensity, adjustment of highlights on the bridge of the nose, adjustment of beard, adjustment of skin brightness, adjustment of overall image brightness, and adjustment of background brightness. a person identification unit that identifies people appearing in each captured image using a model generated by machine learning;
and a retouching processing unit that adjusts the appearance of a first captured image selected as an adjustment target from among a plurality of captured images in response to a user operation, and when an operation is performed to reflect the adjustment content of the first captured image to a second captured image in which the subject's composition is different from that of the first captured image, reflects the adjustment content for a specific person appearing in the first captured image to a person identical to the specific person appearing in the second captured image, based on the subject's composition of the first captured image and the subject's composition of the second captured image. The image processing device
The people in each image are identified using a model generated by machine learning.
adjusting the appearance of a first captured image selected as an adjustment target from among the plurality of captured images in response to an operation by a user;
an image processing method for, when an operation for reflecting adjustment contents of a first captured image to a second captured image in which the composition of subjects differs from that of the first captured image, reflecting adjustment contents for a specified person appearing in the first captured image to a person identical to the specified person appearing in the second captured image, based on the composition of subjects of the first captured image and the second captured image.

Images