JP6205599B2 - Image composition apparatus, image composition method, and computer program - Google Patents

Description

  The present invention relates to an image composition device that synthesizes an image of a user's face and a material image in which a person model is recorded.

  Various apparatuses have been proposed as apparatuses that perform image processing on a user's image captured by an imaging means such as a camera to generate images having different aspects. For example, an apparatus capable of synthesizing decorative elements such as lines and patterns input using a touch pen or the like on an image obtained by photographing a user is known (see, for example, Patent Document 1). There is also known an apparatus that detects a portion of an eye in an image obtained by photographing a user and performs correction to enlarge the eye (see, for example, Patent Document 2).

JP 2002-314909 A JP 2006-228036 A

  Attempts have also been made to transform a user by synthesizing an image of the user's face with an image other than the face of another person model. However, since the face has a direction, it is not easy to obtain a composite image with such unnaturalness or small uncomfortable feeling. For example, it has been necessary to perform adjustment such that the user's face is matched with the face direction of the person model in advance and then the user is photographed. There is an image processing technique for virtually rotating a photographed image to generate an image viewed from a different position. However, if the orientation deviation is large, the unnaturalness of the processed image increases. The technology described in the above-mentioned patent documents does not provide a solution to these problems.

  Therefore, the present invention suppresses unnaturalness that can occur when a face image of a user model in a material image is combined with an image of a user's face, and an image composition device that can obtain a natural composite image as much as possible. The purpose is to provide.

  An image composition device according to an aspect of the present invention includes a storage unit that stores a material image in which a person model is recorded in association with face orientation information that specifies a face orientation of the person model, and an image that captures a user Means, photographing control means for controlling the photographing means so that an image including at least the face of the user is repeatedly photographed, and face orientation determination for judging the orientation of the user's face when photographed by the photographing means And an image of the user's face that is suitable for combining with the material image based on the determination result of the means and the face orientation determination means and the face orientation information corresponding to the material image. Based on the image selection means for selecting a composite target image from a plurality of images and the composite target image and the material image, a composite image is generated by combining the face image of the user image with the face portion of the material image A synthesizing unit that is those with.

  An image composition method according to an aspect of the present invention includes a face of the person model in which an image of a user's face photographed by a photographing unit and a material image in which a person model is recorded are associated with the material image. An image composition method for compositing with reference to face orientation information for specifying the orientation of the user, the procedure for controlling the photographing unit so that an image including at least the face of the user is repeatedly photographed, and the photographing unit Suitable for combining with the material image based on the procedure for determining the orientation of the user's face when the image was taken, the discrimination result in the procedure for determining the orientation of the face, and the face orientation information corresponding to the material image The face of the material image is selected based on the procedure for selecting the image of the user's face as a compositing target image from the plurality of images of the user captured by the photographing means, and the compositing target image and the material image. A step of generating a composite image by combining an image of the face of the user in minutes, are those having a.

  A computer program according to an aspect of the present invention provides an image of a face of a person model in which an image of a user's face photographed by photographing means and a material image in which a person model is recorded are associated with the material image. A computer program for compositing by a computer with reference to face orientation information for specifying the orientation, wherein the computer controls the photographing means so that an image including at least the face of the user is repeatedly photographed. Means for determining the orientation of the face of the user when photographed by the photographing means, the material based on the discrimination result of the face orientation discrimination means and the face orientation information corresponding to the material image An image for selecting a user's face image suitable for combining with an image as a compositing target image from a plurality of images of the user captured by the photographing unit And a combining unit that generates a combined image by combining the face image of the material image with the face image of the user based on the combining target image and the material image. It is.

The figure which shows the whole structure of the trial fitting pseudo-experience system with which the image synthesizing | combining apparatus which concerns on one form of this invention was integrated. The figure which shows the outline | summary of the various services provided by the try-on simulation experience system. The perspective view which shows the structure of the front side seeing from the user inside an arcade machine. The perspective view which shows the structure inside a user's back side inside an arcade machine. The figure which shows the structure of the control system of an arcade machine. The figure which shows the structure of the control system of a center server and a user terminal device. The figure which shows the relationship of the various data preserve | saved in the system or produced | generated. The figure which shows an example of the data structure of contribution data. The figure which shows an example of the option of tag information. The flowchart which shows the outline | summary of the process sequence for the play control part of an arcade machine to make a user try on a plurality of coordination in a pseudo manner. The figure which shows the transition of the typical screen displayed on the display apparatus of an arcade machine when the process of FIG. 10 is performed. The figure which shows the transition of the screen following FIG. The flowchart which shows the procedure of the face detection confirmation process which the play control part of an arcade machine performs. The flowchart which shows the procedure of the face image storage process which the play control part of an arcade machine performs. The flowchart which shows the procedure of the raw material image selection process which the play control part of an arcade machine performs. The flowchart which shows the procedure of the try-on play process which the play control part of an arcade machine performs. The figure which showed the screen transition at the time of guiding shutter timing on the try-on screen in association with the elapsed time of the try-on play. The flowchart which shows the procedure of the shutter timing guidance process which the play control part of an arcade machine performs. The figure which shows the modification of FIG. The flowchart which shows the procedure of the synthetic | combination process which the play control part of an arcade machine performs. The figure which shows the example which made the mesh model the user's face image. The figure which shows the example of the blend process etc. which are performed at the time of image composition. The figure which shows the transition of the various screens displayed on the display apparatus of a user terminal device. The figure which shows the transition of the screen following FIG. The flowchart which shows the procedure of the my page browsing process which the browsing control part of a user terminal device performs when browsing a fitting image etc. The flowchart which shows the procedure of the my page access control process which the user service management part of a center server performs corresponding to the process of FIG. The flowchart which shows the procedure of the process which the browsing control part of a user terminal device and the user service management part of a center server perform when a user posts an own user coordinate image. The flowchart which shows the procedure of the process which the browsing control part of a user terminal device and the user service management part of a center server perform when a user evaluates other person's coordination. The flowchart which shows the procedure of the process which the browsing control part of a user terminal device and the user service management part of a center server perform when a user browses the evaluation given with respect to the self-coordination. The flowchart which shows the procedure of the material data addition process which the user service management part of a center server performs in order to add material data based on a user coordinate image.

  Hereinafter, a try-on simulation experience system in which an image composition device according to an embodiment of the present invention is incorporated will be described. First, with reference to FIG. 1, the overall configuration of a try-on simulation experience system will be described. The trial fitting simulated experience system 1 includes a center server 2, an arcade machine 3 as a client that can be connected to the center server 2 via a predetermined network 5, and a user terminal device 4. The center server 2 is configured as a single logical server by combining server units 2A, 2B... As a plurality of computer devices. However, the center server 2 may be configured by a single server unit. Alternatively, the center server 2 may be logically configured using cloud computing.

  The arcade machine 3 is configured as a business terminal device that provides a user with a predetermined service (predetermined play) in exchange for payment of a predetermined usage fee. The arcade machine 3 is installed in a predetermined facility such as an amusement store (sometimes referred to as an arcade facility) 6 for the purpose of increasing profits by utilizing it by a large number of users. One or more appropriate numbers of arcade machines 3 are installed in the store 6. A specific configuration of the arcade machine 3 will be described later.

  On the other hand, the user terminal device 4 is a computer device that can be connected to a network and is used for a user's personal use. For example, a stationary or book-type personal computer (hereinafter referred to as a PC) 4a or a mobile terminal device 4b such as a mobile phone (including a smartphone) is used as the user terminal device 4. In addition, various computer devices that can be connected to the network and are used for the personal use of the user, such as stationary home game machines, portable game machines, and portable tablet terminal devices, are used as the user terminal device 4. May be. The user terminal device 4 can allow the user to enjoy various services provided by the center server 2 by installing various computer software.

  The network 5 may be appropriately configured as long as the arcade machine 3 and the user terminal device 4 can be connected to the center server 2 respectively. As an example, the network 5 is configured to realize network communication using a TCP / IP protocol. Typically, it is constructed by connecting the Internet 5A as a WAN and the LANs 5B and 5C connecting the Internet 5A to the center server 2 and the arcade machine 3 via the router 5D. The user terminal device 4 is also connected to the Internet 5A with an appropriate configuration. A local server may be installed between the arcade machine 3 and the router 5D of the store 6, and the arcade machine 3 may be communicably connected to the center server 2 via the local server. The server units 2A, 2B,... Of the center server 2 may be connected to each other by the WAN 5A instead of or in addition to the LAN 5C.

  Next, with reference to FIG. 2, the outline | summary of the various services provided by the trial fitting simulated experience system 1 is demonstrated. First, in the arcade machine 3, an image P1 of a user (here, user A is taken up as an example) is photographed, and an image Pf of a predetermined region including the face portion of the user A and a material image P2 are combined and a try-on image. P3 is generated. The material image P <b> 2 is an image in which a person model of arbitrary clothes is recorded. The person model of the material image P2 takes an appropriate pose. However, the material image P2 does not include face components (organs such as eyebrows, eyes, nose, and mouth) of the person model. The person model may be a real person or a virtual person created using CG technology or the like. The material image P2 is stored in advance in the material database DB1 managed by the center server 2. In the material database DB1, a large number of material images P2 with different clothes and poses are stored in advance. The material image P2 of the material database DB1 is distributed from the center server 2 to the arcade machine 3 at an appropriate time and stored in a storage device (not shown) of the arcade machine 3.

  The generated fitting image P3 is stored in the center server 2 in a timely manner in association with the identification information of the user A. The arcade machine 3 charges a predetermined unit fee for one play of one user. In one play, the user can enjoy composition with a plurality of material images P2 having different coordinates, that is, a pseudo fitting experience, but FIG. 2 illustrates only composition with one material image P2. Yes. Moreover, in one play, a plurality of users can also enjoy a try-on experience sequentially. In that case, a charge for the number of players may be charged for one play, or a play fee may be charged for each group with one or more users as one group.

  In the arcade machine 3, access information for accessing the user's own fitting image P3 and the like is provided to the user A. For example, an access two-dimensional code CD1 in which a URL for accessing the fitting image P3 of the center server 2 is embedded as access information is issued to the user A. In the arcade machine 3, when the user A desires, a try-on image P <b> 3 and the like are printed out via the printer 7. Furthermore, the image P1 of the user A himself / herself photographed to generate the fitting image P3 is also stored as the user coordinate image P1a in association with the identification information of the user A. However, since many images P1 of the user A are photographed in the arcade machine 3, a predetermined number of images P1 selected from these images (for example, one in one play) are stored in the center server 2 as user coordinated images. It is stored in the center server 2 as P1a. In this embodiment, the user image P1 photographed by the arcade machine 3 and the user coordinated image P1a stored in the center server 2 are the same as the image, but are photographed by the arcade machine 3 in particular. Of the user images P1 that have been stored, those stored in the center server 2 may be referred to as user coordinate images P1a to be distinguished. The user coordinated image P1a stored in the center server 2 may be selected by the user from a plurality of images P1 of the user photographed by the arcade machine 3, or the arcade machine 3 or the center server 2 may select the random or predetermined You may select automatically according to the criteria.

  The user A reads the access information of the two-dimensional code CD1 using the user terminal device 4 and accesses the center server 2 in accordance with the access information, thereby browsing his / her coordinated image P1a and the fitting image P3. Note that incidental information is recorded in association with the user coordinate image P1a and the fitting image P3. The incidental information includes tag information to be used for coordinate classification and the like in addition to shooting information describing shooting conditions and the like. By accessing the center server 2 from the user terminal device 4, the user A can perform operations such as addition, change, or correction of tag information for his coordinated image P1a. A specific example of tag information will be described later.

  Further, the user A can post an evaluation target image P4 based on the coordinated image P1a in order to allow other users to evaluate the coordination of their clothes. The evaluation target image P4 corresponds to an image obtained by processing the user coordinated image P1a so that it is difficult to identify the user. For example, it is an image obtained by erasing the components (eyes, nose, mouth, etc.) on the user's face from the user coordinated image P1a. In order to obtain such an evaluation target image P4, for example, when the user's image P1 is transmitted from the arcade machine 3 to the center server 2, the arcade machine 3 uses the eyes, nose, mouth, etc. from the user's image P. A corrected image P1b is generated, and the corrected image P1b is transmitted to the center server 2 together with the user image P1. The modified image P1b is recorded in the center server 2 in association with the user coordinated image P1a. When the user instructs to post the user coordinated image P1a, the modified image P1b associated with the user coordinated image P1a is disclosed to other users as the evaluation target image P4. Other users B, C,... Can access the center server 2 from each user terminal device 4, browse the evaluation target image P4 of the user A, and evaluate the coordination. The user A can view other person's evaluation with respect to the evaluation target image P4 posted by himself / herself. When the evaluation target image P4 posted by the user A obtains a certain evaluation, the evaluation target image P4, in other words, the user coordinated image P1a stored in the center server 2 is posted by the user and has a certain evaluation. The modified image P1b corresponding to the obtained user coordinate image P1a is selected as the material candidate image P5. Therefore, the material candidate image P5 also corresponds to an image obtained by processing the face of the user coordinate image P1a so that it is difficult to identify the user. The selected material candidate image P5 is added to the material database DB1 together with the accompanying information. Thereby, the arcade machine 3 can also generate the fitting image P3 using the material image P2 having the user coordinate image P1 as an original image. Note that the process of generating the evaluation target image P4 is not limited to the above. For example, after the center server 2 receives the image P1 of the user, the modified image P1b may be generated when the image P1 is stored as the user coordinate image P1a, or the user coordinate image when the user instructs the posting. The corrected image P1b may be generated from P1a. Further, when the user submits, the modified image P1b may be selected or confirmed by the user in addition to or instead of the user coordinated image P1a.

  Next, with reference to FIGS. 3-9, the detail of each part of the trial fitting simulated experience system 1 is demonstrated. First, the physical configuration of the arcade machine 3 will be described. As is apparent from FIG. 1, the arcade machine 3 is an amusement device of a type having a substantially rectangular parallelepiped housing and allowing a user to enter and play inside the housing. As this type of amusement machine, a photo sticker printing machine is well known, but the arcade machine 3 in FIG. 1 also has the same basic configuration. FIG. 3 shows the configuration of the inside of the arcade machine 3 and the side facing the user (inside front side), and FIG. 4 shows the configuration of the back side of the user. As is clear from FIG. 3, the front unit 10 is installed on the inner front side of the arcade machine 3. A display device 11 is disposed on the front unit 10. The display device 11 is a flat panel display such as a liquid crystal display. When the user stands on the floor mat FM and faces the front unit 10, the display device 11 is arranged so as to look down somewhat at the center in the left-right direction and in the up-down direction. For this reason, the display device 11 is installed obliquely so that the display surface 11a faces somewhat upward.

  A camera 12 for photographing the user is attached above the frame 11b of the display device 11 so that the inside of the arcade machine 3 can be seen. The shooting direction of the camera 12 is generally horizontal. The shooting range (shooting angle of view) of the camera 12 is set so that almost the whole body of the user facing the camera 12 can be shot. Above the camera 12, an operation detection device 13 for detecting a user's operation is arranged so that its detection unit faces the device 3. The motion detection device 13 identifies the user's position, motion, face, etc. by combining a camera that captures the user, a distance measuring sensor that detects the distance to each part of the user, and what operation the user performs. In other words, it is a sensor device that outputs operation information describing what posture the user is taking. As this type of motion detection apparatus, for example, a sensor device provided by Microsoft Corporation using the trademark of Kinect can be used. Furthermore, illumination lamps 14, 15, 16 are arranged on the top, bottom, left and right of the front unit 10. Speakers 17 are arranged on the left and right of the upper illumination lamp 16. A coin insertion / return unit 18 for inserting and returning coins as a play fee is to the right of the lower illumination lamp 17, and to the left of the illumination lamp 17 is to print out printed matter such as a fitting image P 3. A printout unit 19 is provided.

  On the other hand, as shown in FIG. 4, a rear unit 20 is installed on the back side of the arcade machine 3. The rear unit 20 includes a background panel 21 of a single color (for example, green or blue) for performing chroma key processing on the user's image captured by the camera 12, and an illumination lamp disposed above the background panel 21. 22. In addition to the units 10 and 20, the arcade machine 3 is also equipped with various equipment such as a curtain that appropriately shields the user's entrance and exit, and an entrance sign that indicates whether the user is playing or not. However, those details are omitted.

  Next, the configuration of the control system of the arcade machine 3 will be described with reference to FIG. The arcade machine 3 is provided with a control unit 30 and a storage device 31. The control unit 30 is configured as a computer unit including a CPU and an internal storage device necessary for its operation. The storage device 31 functions as an external storage device for the control unit 30. The storage device 31 is a non-volatile storage device such as a semiconductor storage device using a semiconductor storage element such as a hard disk storage device or a flash memory as an example. The storage device 31 stores an arcade program PG1. The arcade program PG1 uses a function provided by a combination of the control unit 30 and an operating system (not shown) to perform a process necessary for causing the user to experience a try-on in a simulated manner, for example, the user's image P1. This is an application program for causing the control unit 30 to execute processing such as photographing and generating a fitting image P3.

  The control unit 30 is provided with a play control unit 32 for controlling the progress of play by the user. The play control unit 32 is a logical device realized by a combination of the hardware of the control unit 30 and the arcade program PG1. The play control unit 32 is connected to the camera 12 and the motion detection device 13 described above via an interface unit (not shown). The play control unit 32 controls the camera 12 to capture the user's image P1, and controls the motion detection device 13 to detect the user's motion. The play control unit 32 is connected to the display device 11 and the speaker 17 as output devices. The play control unit 32 controls display of images on the display device 11 and output of music and sound effects from the speaker 17. Further, the play control unit 32 is connected with a touch panel input device 35 and a coin selector 36 as input devices. The touch panel input device 35 is arranged so as to overlap the display surface 11 a of the display device 11, and outputs a signal corresponding to the user's touch operation to the play control unit 32. The coin selector 36 determines the authenticity of the coin 8 inserted by the user, and if it is authentic, outputs a signal corresponding to the number of inserted coins to the play control unit 32, and returns the coin 8 if illegal. Note that the touch panel input device 35 may be integrated with the display device 11.

  The play control unit 32 includes a face image extraction unit 37, a face orientation correction unit 38, and an image composition unit 39 as further logical devices. The face image extraction unit 37 extracts an image of the user's face portion from the user image P <b> 1 captured by the camera 12. The face orientation correcting unit 38 corrects the orientation of the face image extracted by the face image extracting unit 37. The image composition unit 39 synthesizes the corrected face image and the material image P2. These processes will be described in detail later.

  In addition to the above-described arcade program PG1, the storage device 31 also records arcade data D1 and user data D2. The arcade data D1 is data that should be referred to as appropriate in the arcade program PG1 when the arcade machine 3 allows the user to try on a try-on. For example, the arcade data D1 includes material data D11 and presentation data D12. The material data D11 includes the image data of the material image P2 described above and incidental information corresponding thereto. The effect data D12 is a music tune to be reproduced from the speaker 17 during play, sound effect data, and an image that provides decoration effects such as a background, pattern, stamp, etc. to be displayed on the display surface 11a of the display device 11. It includes image data such as a decoration image as a material or an object image (camera icon 150 shown in FIGS. 17 and 19 as an example). The user data D2 is data generated with the user's play. For example, the user data D2 includes data (hereinafter, also referred to as “my data”) D21 and a user's try-on image P3 related to the user's own photographing result including the user's image P1 photographed by the camera 12. Including fitting data D22 is recorded. Further, face image data D23 including the user face image Pf extracted from the user image P1 is also temporarily recorded in the user data D2. When the corrected image P1b shown in FIG. 2 is created by the arcade machine 3, the corrected image P1b may also be included as part of the user data D2. Various data other than the illustrated data may be recorded in the storage device 31.

  Next, the configuration of the control system of the center server 2 and the user terminal device 4 will be described with reference to FIG. The center server 2 includes an arcade service management unit 40, a user service management unit 41, and a storage unit 42. The arcade service management unit 40 and the user service management unit 41 are logical devices realized by a combination of the computer hardware resources of the center server 2 and the server program PG2 stored in the storage unit 42. The arcade service management unit 40 provides various services to the play control unit 32 of the arcade machine 3, and the user service management unit 41 provides various services to the user terminal device 4. The storage unit 42 stores arcade data D1 and user data D2. The arcade data D1 is the same as the original data D1 stored in the storage device 31 of each arcade machine 3, and is distributed to the arcade machine 3 by the arcade service management unit 40 in a timely manner. The user data D2 includes try-on data D22 and at least a part of my data D21 among the user data D2 stored in the storage device 31 of the arcade machine 3. These data are transmitted from the arcade machine 3 at an appropriate time, for example, at the end of play, and stored in the storage unit 42.

  The storage unit 42 also records post data D3, evaluation data D4, analysis data D5, and total data D6. The posted data D3 is data including the evaluation target image P4 posted by the user and the accompanying information, and is associated with the user identification information. The evaluation data D4 is data in which an evaluation given by a user as an evaluator is recorded for the evaluation target image P4. The evaluation data D4 is generated every time evaluation is performed. The analysis data D5 is data obtained by analyzing the evaluation described in the evaluation data D4, and is recorded in association with the user identification information for each evaluation target image P4. The total data D6 is data indicating the result of totaling the evaluation contents recorded in the analysis data D5. The storage unit 42 may record various types of data other than those illustrated.

  The user terminal device 4 includes a control unit 50 and a storage device 51. The control unit 50 includes a two-dimensional code reading unit 52 and a two-dimensional code display unit as logical devices realized by a combination of the hardware resources and a predetermined user terminal program PG3 recorded in the storage device 51. 53 and a browsing control unit 54 are provided. The user terminal program PG3 is so-called application software for causing the user terminal device 4 to execute arithmetic processing unique to the user terminal device 4 using functions provided by the operating system of the user terminal device 4.

  The two-dimensional code reading unit 52 reads the two-dimensional code CD1 as access information issued when the user plays on the arcade machine 3 via the camera (not shown) of the user terminal device 4, and the read access Information is provided to the browsing control unit 54. The two-dimensional code display unit 53 generates a two-dimensional code for authentication in which user identification information stored in the storage device 51 is embedded, and the two-dimensional code for authentication is displayed on the display surface of the user terminal device 4. Display. The two-dimensional code for authentication is different from the two-dimensional code CD1 for access shown in FIG. However, once the two-dimensional code for authentication is generated, the two-dimensional code itself is stored as image data in the storage device 51, and thereafter, the two-dimensional code display unit 53 displays the image data. Good.

  The browsing control unit 54 controls access to the center server 2, acquisition of information, or transmission of information using a Web browser function or the like implemented in the user terminal device 4. When the browsing control unit 54 accesses the center server 2 using the access information embedded in the two-dimensional code CD1 or the user identification information stored in the storage device 51, the user uses his user terminal device 4. Then, the user's user coordinate image P1a and the fitting image P3 can be browsed, or the evaluation target image P4 can be posted, and the evaluation on the evaluation target image P4 can be browsed.

  Next, various data will be described in more detail with reference to FIGS. First, the relationship between data will be described with reference to FIG. In the arcade machine 3, my data D21 and try-on data D22 are generated. In my data D21, a user image P1 photographed to include the user's own clothes and pose information IP are recorded in association with each other. The pose information IP is information (vector information) for specifying the position of the user's face, shoulder, arm, hand, etc. recorded in the user image P1, and is specified by the motion detection device 13. The pose information IP includes information for specifying the orientation of the user's face (referred to as face orientation information). The face direction information includes first angle information for specifying in what direction the user is facing the user with respect to an appropriate reference direction, for example, the shooting direction (optical axis direction) of the camera 12, and the user's head. Both of the second angle information specifying how the portion is inclined with respect to the vertical direction are included. On the other hand, the fitting data D22 corresponds to the incidental information of the material data D11, the fitting image P3 generated by combining the material image P2 included in the material data D11, the user's face image Pf, and the decoration image DP. It is the data that was recorded. The decoration image PD is an image appropriately selected from a background or pattern image group recorded in the effect data D12. However, in certain cases, the pose information IP corresponding to the user's image P1 includes the person of the material image P2 that is the origin of the fitting image P3 instead of or in addition to the vector information specified by the motion detection device 13 Pose information (hereinafter, this may be referred to as basic pose information) for specifying a model pose may be recorded. This point will be described later.

  My data D21 and fitting data D22 created by the arcade machine 3 are transmitted to the center server 2 and stored in the storage unit 42. The user image P1 stored in the my data D21 is positioned as the user coordinated image P1a by being stored in the storage unit 42. The user coordinate image P1a and the fitting image P3 of the data D21 and D22 stored in the storage unit 42 can be viewed from the user terminal device 4 by the user corresponding to the images P1a and P3. When the user instructs the user coordinate image P1a recorded in the my data D21 to be posted as the evaluation target image P4, post data D3 is generated using the modified image P1b associated with the user coordinate image P1a as the evaluation target image P4. Is done. The post data D3 inherits the incidental information of the user coordinate image P1a that is the original. In addition, the user can add tag information from the user terminal device 4 to the evaluation target image P4 posted by the user. The evaluation target image P4 included in the posting data D3 is a browsing target for other users. When another user gives an evaluation at the user terminal device 4, evaluation data D4 is generated for each evaluation. The evaluation data D4 includes information for specifying the evaluation target image P4 and information for specifying the evaluation content for the evaluation target image P4. The evaluation data D4 is differentiated for each evaluation target image P4 and aggregated in a timely manner, thereby generating analysis data D5 for each evaluation target image P4. Further, the analysis data D5 is aggregated across the lines to generate aggregate data D6. For example, the number of evaluations related to a specific evaluation element is aggregated, and the ranking of the evaluation target image P4 is created and recorded in the aggregate data D6. The analysis data D5 can be browsed by the user recorded in the evaluation target image P4 using the user terminal device 4. The total data D6 can be browsed by each user using the user terminal device 4.

  The posted data D3 and the total data D6 are also used for creating the material candidate data D7. That is, post data D3 for which a certain evaluation is obtained in the total data D6 is extracted, and the modified image P1b corresponding to the post data D3, tag information, pose information, etc. recorded in the post data D3. Material candidate data D7 is generated so as to include the accompanying information. The generated material candidate data D7 is added to the group of material data D11 (corresponding to the material database DB1 in FIG. 2), and is appropriately distributed to the arcade machine 3 as the material data D11.

  Next, the data structure will be described with reference to FIGS. In the following, post data D3 is taken as an example. As shown in FIG. 8, post data D3 includes, as an example, a data number, registration date and time, image designation information, pause information, and a plurality of sets of tag information 1, 2,. The data number is uniquely set for each data. The registration date and time is the date and time when the data is registered in the center server 2, and the image designation information is information for designating the evaluation target image P4 to be included in the post data D3. For example, when the user coordinated image P1a is managed with a unique file name or image ID for each image, the image designation information includes a file corresponding to the user coordinated image P1a that is the basis of the evaluation target image P4. A name or an image ID can be designated. However, instead of the file name of the user coordinated image P1a, the file name of the corresponding modified image P1b may be specified. The pose information is information (vector information) for specifying the position of the user's face, shoulder, arm, hand, etc. in the evaluation target image P4, and corresponds to the pose information IP in FIG. The tag information is information for classifying the evaluation target image P4 and is divided into a plurality of items. In the example of FIG. 8, one set of tag information is divided into items such as coordinated category, coordinated item, scene, style, posing, body shape, height, hairstyle / hair color, music, effect, and background.

  Regarding tag information, a plurality of options are prepared in advance for each item. Each item of tag information is input by specifying an appropriate option from these options. An example of the options is shown in FIG. For example, for the category item, options such as “girly”, “gal”, “natural”, “street casual”, and “beautiful” are prepared as the coordination category of the evaluation target image P4. Items such as color, item, and scene are also items for classifying the characteristics of coordinates in the evaluation target image P4. The “scene” is an item for evaluating what kind of situation the evaluation target image P4 is suitable for, such as date or travel. The items of style, body shape, and height are items for classifying the user's style, body shape, and height recorded in the evaluation target image P4. The posing item is an item expressing a pose that the user is taking in the evaluation target image P4. Furthermore, the hairstyle / hair color items are divided into hair styles and hair colors, and options are prepared. For example, options such as “short”, “medium”, and “long” are prepared for the hairstyle. For hair color, in addition to large classification options such as “beige” and “yellow”, for example, “beige” is further classified into “baby beige” and “milk beige”. That is, for the hair color item, options are prepared in a hierarchical structure.

  The items of the tag information described above include items specified by the user as the person to be evaluated who posted the evaluation target image P4, items specified by the user as the evaluator, and the administrator (or management) of the center server 2 There are items specified by the administrator etc.). For example, items such as “color”, “item”, “body shape”, and “height” are specified by the user of the evaluated person. For example, when the user instructs to post his / her user coordinated image P1a, the user can designate these items. The “scene” is designated by the user as an evaluator. The designation can be made when the user evaluates another person's evaluation target image P4. Accordingly, there is no value for the item specified by the user as the evaluator at the post data D3 stage. Items such as “music”, “effect”, and “background” are designated by the operator of the center server 2 or the like. Since these items are information that is referred to when the posted evaluation target image P4 is used as the material image P2 at the time of another person's play, it is necessary for the administrator or the like to set a value according to a certain standard. The specification is entrusted. However, the configuration of the tag information is merely an example, and the person who designates each item may be changed as appropriate.

  Although the posting data D3 is illustrated in FIG. 9, the evaluation data D4 also has a similar data structure. That is, when the user performs the evaluation, a copy of the post data D3 to be evaluated is generated, and the tag data is added according to the designation of the user as the evaluator, thereby generating the evaluation data D4. Material candidate data D7, material data D11, and try-on data D22 generated based on the posted data D3 are also generated as data having a similar data structure.

  Next, with reference to FIGS. 10 to 30, various processes performed in the trial fitting simulated experience system 1 will be described. 10 to 22 are diagrams related to processing in the arcade machine 3. First, processing in the arcade machine 3 will be described with reference to these drawings. FIG. 10 shows an outline of a processing procedure for causing the play control unit 32 of the arcade machine 3 to try on a plurality of coordinates in a pseudo manner. 11 and 12 show an example of the transition of the screen displayed on the display device 11 when the process of FIG. 10 is performed. Hereinafter, an outline of a play procedure performed in the arcade machine 3 will be described with reference to these drawings. The play control unit 32 of the arcade machine 3 monitors whether or not the motion detection device 13 has detected a user, and starts the process of FIG. 10 when the user is detected. However, the process of FIG. 10 may be started based on an instruction from the user. In the process of FIG. 10, first, the play control unit 32 causes the user to specify the number of players in order to confirm the number of players (step S1), and then confirms whether or not the motion detection device 13 can detect each user's face. (Step S2). At this stage, for example, the confirmation screen 100 of FIG. On the confirmation screen 100, a thumbnail display unit 101 in which thumbnail images for the number of users are arranged, and a main display unit 102 indicating the image of the user being photographed are displayed. A message is displayed informing that the user is to perform the operation and that the face should be faced directly.

  In the process of step S2 in FIG. 10, for example, when the user wears sunglasses or a mask or wears a muffler, the organ of the user's eyes and mouth, or the contour of the face is identified. May be difficult and face detection may be impossible. Therefore, when a face cannot be detected, an advice for successfully detecting the face is presented to the user. After that, when a face cannot be detected for a certain period of time, an error is given to the user and it is excluded from the number of players. Specific examples of these processes will be described later.

  After confirming the detection of the user's face, the play control unit 32 performs two-dimensional code authentication (step S3). In this process, a two-dimensional code for authentication corresponding to the user identification information stored in the user terminal device 4 is displayed on the screen of the user terminal device 4, and the two-dimensional code is read by the camera 12 of the arcade machine 3. In this process, the user identification information embedded therein is acquired. By this processing, the arcade machine 3 can acquire the identification information of the user as a repeater. In addition, the display of the two-dimensional code for authentication in the user terminal device 4 is executed by the two-dimensional code display unit 53 of FIG. 6 as described above. For users who have not acquired user identification information, the two-dimensional code for authentication cannot be displayed, and thus the process of step S3 is skipped. Here, the user is authenticated using a two-dimensional code. However, as long as unique identification information is used for each user, the user can be authenticated by a method different from the authentication of the user via the two-dimensional code for authentication. Good. For example, the user may be authenticated using identification information such as a user ID and a card ID shared by a plurality of types of game machines.

  When the two-dimensional code authentication is completed, the play control unit 32 collects a charge for the number of players from the user (step S4). Next, the play control unit 32 performs a commemorative photo (step S5). In this process, as shown in the commemorative photographing screen 110 as an example in FIG. 11, all the users to be played (four in the example of FIG. 11) are photographed together and the images are recorded in the storage device 31. It is processing. However, when there is one user, the process of step S5 may be skipped.

  When the commemorative photo is over, the play control unit 32 selects the first user as a play target (step S6), and then causes the user to select a predetermined number of material images P2 (step S7). For example, as shown in the selection screen 120 of FIG. 11, a candidate display unit 121 in which a large number of material images P2 stored in the storage device 31 are arranged as selection candidates is displayed, and the material image is displayed according to the user's scroll operation. A process of allowing the user to select a predetermined number (six images as an example) of material images P2 desired to be tried on is performed while replacing P2. The material image P2 is read from the material data D11 recorded in the storage device 31. In this case, the selection screen 120 includes a remaining selection image display unit 122 for each user indicating how many material images P2 can be selected, a confirmation button 123 for confirming the selected material image P2, and “Random”. The indicated button 124 is displayed. When the user touches the “Random” button 124, the play control unit 32 automatically selects a predetermined number of material images P2 without selecting a predetermined number of users one by one.

  Returning to FIG. 10, when the selection of the material image P2 is completed, the play control unit 32 executes a try-on play process (step S8). At this stage, effect elements such as reproduction of music and display of a decoration image are output, and the fitting screen 130 illustrated in FIG. 12 is displayed on the display device 11 in accordance with the output. On the try-on screen 130, as indicated by broken lines in FIG. 12, the pose images PP corresponding to the pose of the person model of the material image P2 selected by the user are sequentially displayed so as to be switched at an appropriate time interval. The pose image PP is an image that shows the pose of the person model specified by the pose information IP of the material image P2 recorded in the material data D11 in a predetermined manner. The plurality of pose images PP are sequentially displayed so that the images are switched at a predetermined timing according to the rhythm and tempo of a song output as a production element, for example. At this time, a decoration image PD such as a background or a pattern is superimposed and displayed on the pose image PP. Further, while the try-on play is being performed, the user is repeatedly photographed by the camera 12, and the user's image P1 obtained by the photographing is displayed as a moving image. The pose image PP and the user image P1 are displayed so as to overlap each other in such a manner that both images can be visually recognized by the user. For example, the user's image P1 photographed by the camera 12 is displayed as it is photographed, while the pose image PP shows a pose identified based on the pose information IP corresponding to the material image P2. The portion other than the line is displayed as a transparent image. Alternatively, the pose image PP obtained by making the material image P2 translucent may be superimposed on the user image P1, or the material image P2 itself is displayed as the pose image PP, and the user image P1 is made translucent. By superimposing in a state, both images P1 and PP may be made visible. The user is required to take the same pose as specified by the pose information of the material image P2. When the user image P1 satisfies a certain condition, a face image suitable for composition is selected from a large number of face images photographed during play, and the face image is pasted on the face portion of the person model in the material image P2. The fitting image P3 is generated. The procedure for generating the fitting image P3 will be described later in detail. The image 130 includes a main display unit 131 on which the user's image P1 and the pose image PP are displayed, a try-on result display unit 132 that displays a reduced display of the generated try-on image P3, and a material image that has not been tried on. A remaining number display section 133 for displaying the number of P2 is also displayed.

  When the try-on play for one user is finished in step S8 in FIG. 10, the play control unit 32 determines whether or not the try-on play has been finished up to the last user (step S9), and a user who has not played yet. If so, the next user is selected in step S10 and the process returns to step S7. On the other hand, when it is determined that the last user has been played in step S9, the play control unit 32 displays the play results of all the users on the display device 11 and browses the generated fitting image P3. The two-dimensional code in which the access information is embedded is displayed on the display device 11 (step S11). For example, as shown on the result display surface 140 of FIG. 12, the fitting result display unit 141 arranged with fitting images P3 displays the fitting result, and the access two-dimensional code CD1 is displayed below the fitting result. On the result display surface 140, an end button 142 for instructing the user to end the play and a printout button 143 for instructing the printout of the fitting image P3 are also displayed. The printout requires payment of an additional fee, and the amount of the additional fee is also displayed on the printout button 143. When a plurality of users play, try-on images P3 generated by each user's try-on play may be collectively printed on a single print mount, or try-on by changing the print mount for each user. The image P3 may be printed out.

  Returning to FIG. 10, when the display of the fitting image P3 and the like is completed, the play control unit 32 determines whether the user has touched the printout button 143 to instruct printing output (step S12). For example, an additional fee is collected (step S13), and then a try-on image P3 is printed out (step S14). However, at this stage, an image taken in commemorative photography (step S5) may also be added to the print target. After that, if the user does not instruct printing output in step S12, the play control unit 32 skips step S13 and step S14 and proceeds to step S15, and the user data of each user generated in the current process D2 (my data D21 and fitting data D22) is transmitted to the center server 2. Thereafter, the play control unit 32 ends the process of FIG. Note that image data obtained by the commemorative photographing in step S5 may also be transmitted to the center server 2 as part of the user data D2.

  The transmitted data D2 is recorded in the storage unit 42 by the arcade service management unit 40 of the center server 2. In this case, for the user who has authenticated the two-dimensional code in step S3, the user identification information embedded in the two-dimensional code is provided to the center server 2, so that the user is associated with the user identification information. Data D2 is recorded. For users who have not received the two-dimensional code authentication in step S3, the user identification information does not exist or cannot be specified by the arcade machine 3. Regarding such user data D2, the center server 2 stores user data D2 including a fitting image P3 and the like without being associated with the user identification information.

  Next, a more detailed procedure of the process executed by the play control unit 32 will be described with reference to FIGS. FIG. 13 shows a specific procedure of the face detection confirmation process executed by the play control unit 32 when confirming the detection of the user's face in step S2 of FIG. At the time of face detection confirmation, the play control unit 32 first starts photographing the user with the camera 12 and recording the obtained user face image Pf in the storage device 31 (step S21). Next, the play control unit 32 refers to the output of the motion detection device 13 to determine whether or not the motion detection device 13 detects the user's face, in other words, the motion detection device 13 can recognize the user's face. It is determined whether or not there is (step S22). If the face detection is successful, the play control unit 32 proceeds to step S23 and records that the confirmation result is success (OK). On the other hand, if it is determined in step S22 that the face has not been successfully detected, the play control unit 32 determines whether or not a time-out has occurred, that is, whether or not a fixed time has elapsed after the start of confirmation (step S24). . If not time-out, the play control unit 32 notifies the user of advice for making the face detection successful (step S25), and the process returns to step S22. If it is determined in step S24 that a time-out has occurred, the play control unit 32 determines that play is not possible for the user who is currently being photographed by the camera 12, and performs error processing such as notifying the user accordingly. Execute (step S26). After the process of step S23 or S26, the play control unit 32 finishes the process of FIG. Thereafter, the play control unit 32 proceeds to the process of step S3 in FIG.

  FIG. 14 shows a procedure of face image accumulation processing executed by the play control unit 32 using step S21 of FIG. 13 as a trigger. The process of FIG. 14 is a routine process that is repeatedly executed in parallel with other processes at a predetermined cycle (for example, the frame rate of the display device 11 or an integral multiple thereof). When the process of FIG. 14 is started, the play control unit 32 first captures the user with the camera 12 and acquires the image P1 (step S31). Next, the play control unit 32 acquires information indicating the user's motion detected by the motion detection device 13, that is, pause information IP (step S32), and from the detection result, the user's image P1 is captured. The orientation of the user's face is determined (step S33). In this case, the face orientation is a first direction that specifies in what direction the user is facing the face with respect to a predetermined reference direction, similar to the face orientation information in the pose information IP (FIG. 7) described above. Both angle information and second angle information for specifying how the user's face is inclined with respect to the vertical direction are included.

  Next, the play control unit 32 proceeds to step S34, and generates a mesh model MM (see FIG. 21) corresponding to the user's face image Pf based on the face orientation determined in step S33. The mesh model MM will be described later. Next, the play control unit 32 determines whether or not the face image Pf of the user having the face orientation that matches the face orientation specified in step S33 has already been recorded in the face image data D23 relating to the same user (step S31). S35). Since the face image Pf of the user is recorded in the face image data D23 in association with the face orientation information (first angle information and second angle information) specified by the pose information IP, the determination is made in step S33. It may be determined whether or not a face image Pf having the same information as the face orientation information already exists in the face image data D23. If the face image Pf of the user whose face direction matches does not exist, the play control unit 32 and the pose information IP acquired in step S32 and the face image Pf extracted from the user image P1 acquired in step S31. Is stored in the face image data D23 in association with the face orientation information based on the mesh model MM (step S36). On the other hand, if there is a face image Pf of the user whose face orientation matches in step S35, that is, if there is a face image Pf with matching face orientation information, the play control unit 32 determines the face image Pf and the face image Pf. The face image data Pf and the like are stored such that the corresponding mesh model MM is overwritten with the face image Pf extracted from the current user image P1 and the mesh model MM corresponding to the face image Pf (step S37). When the process of step S36 or S37 ends, the play control unit 32 ends the routine of FIG. With the processing in FIG. 14, while the user is playing on the arcade machine 3, the image P <b> 1 is repeatedly shot, and the latest face image Pf is accumulated in the storage device 31 for each face orientation. Note that when taking a commemorative photo in step S5 of FIG. 10, a plurality of users are simultaneously positioned in front of the camera 12, and therefore the process of FIG. 14 may be temporarily interrupted. Alternatively, the process of FIG. 14 may be started using the process start of step S7 as a trigger instead of step S2 of FIG. The face image accumulation process shown in FIG. 14 is performed at the timing of the end of play in the case of single-player play, and at the timing when the user changes or the play of all members is completed in the case of multiple-player play. It may be erased.

  FIG. 15 shows a specific procedure of the material image selection process executed by the play control unit 32 when the user selects the material image P2 in step S7 of FIG. When selecting the material image P2, the play control unit 32 first displays the selection screen 120 (FIG. 11) on the display device 11 to present the user selectable material image P2 to the user as a selection candidate (step) S41). When material data D11 of a large number of material images P2 is stored in the storage device 31, the selection candidates to be presented to the user are narrowed down by causing the user to specify items recorded in the tag information of the material data D11. You may do it.

  After the presentation of the material image P2, the play control unit 32 determines whether or not the user has performed an operation for changing the selection candidate (step S42). If there is an operation, the play control unit 32 returns to step S41 and responds to the user's operation. A new material image P2 is presented to the user. If there is no change operation, the play control unit 32 proceeds to step S43, and determines whether or not the user has touched the button 124 and selected "Random". If “Random” is not selected, the play control unit 32 further determines whether or not the user has touched and selected any material image P2 (step S44). When the selection is made, the play control unit 32 records the material image P2 selected by the user (step S45), and then determines whether or not the selection number has reached a predetermined number (step S46). If the predetermined number has been reached, the play control unit 32 displays all the selected material images P2 on the display device 11 (step S47) and asks the user for confirmation (step S48). If “Random” is selected in step S43, the play control unit 32 automatically selects a predetermined number of material images P2 (step S49), and then proceeds to step S47.

  In the case where the user refuses confirmation because the user wishes to reselect in step S48, the play control unit 32 returns to step S43. Note that the play control unit 32 also returns to step S44 if a negative determination is made in step S46. When the user confirms the selection of the material image P2 in step S48, the play control unit 32 analyzes the tag information associated with the selected material image P2 (step S50), and according to the analysis result, FIG. The effect elements (songs, backgrounds, etc.) to be output in the trial play in step S8 are determined (step S51).

  For example, as shown in FIG. 8, with reference to the items “music”, “effect”, and “background” in the tag information, recording is performed in effect elements in the try-on play, that is, effect data D12 (see FIG. 5). The decorative image PD such as the music to be reproduced during play, the background to be displayed on the fitting screen 130, the pattern, and the effect is determined from the music and the decorative image PD (FIG. 7). However, even when there is no tag information that directly specifies music, background, etc., by referring to tag information of items such as “category”, “color”, and “scene” that describe the characteristics of the coordinate, the material image P2 A song or decoration image PD suitable for coordination may be determined. In that case, it is only necessary to prepare in advance data associating each item option (see FIG. 9) with effect elements such as music and background, and select music corresponding to the tag information using the data as a clue. In addition, when the user selects a plurality of material images P2, tag information may be different for each material image P2. In this case, the music and decoration image PD may be sequentially switched and output for each material image P2. When one song or the same decoration image PD is used through all the selected material images P2, for example, the tag information of each material image P2 is analyzed, and the songs and decorations that can be applied to the most material images P2 are analyzed. The image PD may be selected, or may be selected at random from music pieces or decoration images PD extracted from tag information corresponding to the selected material image P2.

  After determining the effect element in step S51 in FIG. 15, the play control unit 32 finishes the material image selection process in FIG. Thereafter, the play control unit 32 proceeds to the try-on play process of step S8 in FIG. FIG. 16 shows a specific procedure of the try-on play process executed by the play control unit 32 in step S8. When the try-on play process is started, the play control unit 32 first reads the data of the song determined in step S51 of FIG. 15 from the storage device 31 and starts the reproduction of the song (step S61). The data of the decoration image PD determined in S51 is read from the storage device 31 and the display is started (step S62). Subsequently, the play control unit 32 selects the material image P2 to be tried on first from the material image P2 selected by the user (step S63), and decorates the pose image PP corresponding to the pose of the person model of the material image P2. The pose image PP is presented to the user by being displayed on the display device 11 together with the image PD (step S64). In this case, as illustrated in the try-on screen 130 of FIG. 12, the user's image P1 repeatedly photographed by the camera 12 is continuously displayed superimposed on the pose image PP. In addition, the display mode is controlled such that either the pause image PP or the user image P1 is made translucent, or either one is displayed as a diagram so that both the images P1 and PP can be visually recognized. The

  After presenting the pose image PP or the like, the play control unit 32 uses the display device 11 to start processing for guiding the shutter timing to the user (step S65). The shutter timing is determined for each material image P2 in advance in association with the elapsed time from the start of the process of step S8 in FIG. However, as is apparent from the description of FIG. 14, the user's shooting by the camera 12 is not limited to the shutter timing. The user's shooting by the camera 12 is repeatedly performed during the try-on play, and the shutter timing is determined to make the user aware of shooting. However, the user image P1 captured at the shutter timing may be recorded in the my data D21. Further, the shutter timing guidance is performed in order to guide the user's eyes to a predetermined target position outside the display surface (for example, a shooting position by the camera 12). Details of the guidance will be described later.

  After starting the shutter timing guidance, the play control unit 32 attempts to combine the user's face image Pf and the material image P2 (step S66). The processing is performed using the face orientation correction unit 38 and the image composition unit 39, details of which will be described later. Next, the play control unit 32 determines whether or not the image has been successfully combined (step S67). If the image has been successfully displayed, the combined fitting image P3 is displayed to the user via the display device 11. At the same time, the fitting image P3 is recorded in the storage device 31 (step S68). In this case, the fitting image P3 is stored in the storage device 31 as fitting data D22 associated with the accompanying information (including tag information and pose information) of the material image P2 that is the composition source. After recording the try-on image P3, the play control unit 32 determines whether it is the material image switching time for each material image P2 set during the try-on play (step S69). The material image switching time is set, for example, at a time when the break is good in the rhythm and tempo in the music output as part of the performance. However, an appropriate time interval is assigned between the material image switching times so as to ensure a sufficient time for composition.

  When the material image switching time has not come, the play control unit 32 returns to step S68 and continues displaying the result. When the material image switching time has arrived, the play control unit 32 proceeds to step S70, and determines whether or not the try-on has been completed for all material images P2 selected by the current play target user. If there is a material image P2 that has not yet been tried on, the play control unit 32 selects the next material image P2 (step S71), and then returns to the process of step S64.

  If it is determined in step S67 that the composition has failed, the play control unit 32 executes a predetermined substitution process (step S72), and then proceeds to the process of step S69. As a case where the composition fails, for example, the face direction coincides with the face direction of the user (first angle information and second angle information) specified by the face direction information corresponding to the material image P2. The face image Pf is not recorded in the storage device 31. Details will be described later. Further, as an alternative process, for example, the user is notified that the composition could not be performed, and the user's own image captured at the time of the process, that is, the user coordinated image P1 is replaced with the try-on image P3 to the user. Processing to display can be performed. Alternatively, an appropriate single face image Pf is selected from the face images Pf recorded in the face image data D23, and a material image P2 having a face orientation that matches the face orientation in the face image Pf is selected as the material data D11. May be extracted and the material image P2 may be combined with the face image Pf.

  If it is determined in step S69 that the processing has been completed for all the material images P2, the play control unit 32 ends the shooting of the user by the camera 12 and the recording of the face image Pf (step S73). That is, the face image accumulation process of FIG. 14 is terminated. Therefore, the camera 12 is controlled to repeatedly photograph the user during the image acquisition period from the start of the process of step S2 in FIG. 10 to the process of step S72 in FIG. After the user's shooting is finished, the play control unit 32 uses a predetermined number of images (as an example) to be transmitted to the center server 2 from a large number of user images P1 recorded as a part of the my data D21 in the storage device 31. One image P1 is selected (step S74). The selection of the image P1 may be left to the user, or the play control unit 32 may automatically select according to a predetermined condition. The condition in this case can be set such that the image P1 having a relatively high matching rate with the pose of the person model in the material image P2 is preferentially selected or randomly selected. As an example, the pose match rate can be determined based on the proximity between the pose information IP associated with the material image P2 and the user's pose detected by the motion detection device 13 at the time of shooting by the user. After the end of step S73, the play control unit 32 ends the process of FIG. Thereafter, the play control unit 32 proceeds to step S9 in FIG.

  17 to 19 are diagrams for explaining a shutter timing guide method. FIG. 17 shows the screen transition when guiding the shutter timing on the try-on screen 130 (see FIG. 11) in association with the elapsed time of the try-on play. First, the outline of the shutter timing guidance method will be described with reference to FIG. 17. In FIG. 17, a fitting screen 130 is shown with branch numbers -1, -2,. Separated. In the following description, the target position of the user's line of sight in the shutter tin mig is an example in which the shooting position of the camera 12 is set.

  When the shutter timing guidance is started, first, a camera icon 150 as a shutter timing guidance sign appears at the initial position of the fitting screen 130-1. The appearance time is set to a time preceding the shutter timing set during the try-on play by a certain time. For example, as shown in the first try-on screen 130-2, the initial position is set substantially at the center of the main display 131 where the material image P2 (not shown in FIG. 17) should be displayed. Thereafter, the camera icon 150 rises straight in the fitting screen 130 toward the shooting position SP by the camera 12, as indicated by an arrow M1 on the second fitting screen 130-2. As shown in the fitting screen 130-2, when the camera icon 150 rises to just before the upper end of the fitting screen 130, as shown by the arrow M2 on the third fitting screen 130-3 from the left, the camera icon The display mode of the camera icon 150 is gradually changed so that a state in which 150 bounces against the frame of the screen 130 is expressed. Thereafter, as shown by the arrow M3 on the fourth try-on screen 130-4, the camera icon 150 jumps out of the try-on screen 130 (that is, outside the display surface 11a of the display device 11) and disappears from the screen 130. As shown, the display mode of the camera icon 150 is changed. The time when the camera icon 150 disappears is set earlier than the shutter timing.

  After the camera icon 150 disappears upward from the fitting screen 130, the camera icon 150 ascends toward the shooting position SP as before, as indicated by the arrow M4 on the fitting screen 130-5 at the right end of FIG. Is assumed to continue, the camera icon 150 virtually reaches the shooting position SP at the shutter timing. That is, the timing when the camera icon 150 virtually reaches the shooting position SP matches the shutter timing. The appearance time of the camera icon 150 at the initial position and the moving speed of the camera icon 150 are controlled so that such a relationship is established.

  By guiding the shutter timing as described above, it is possible to attract the user's eyes to the camera icon 150 and guide the user's eyes to the shooting position SP so as to follow the camera icon 150. Then, the user's line of sight can be substantially matched with the shooting position SP at the shutter timing. As a range in which the arrival time and the shutter timing can be considered to coincide with each other, it is sufficient that the shutter timing arrives while a part of the camera icon 150 is virtually overlapped with the shooting position SP. As long as the purpose of moving to the position SP can be achieved, if the shutter timing arrives at a time when the camera icon 150 is assumed to almost reach the shooting position, it may be considered that the two match. In order to direct the user's line of sight toward the target position outside the display surface 11a at the shutter timing, the camera icon 150 moves toward the target position and the camera icon 150 is displayed at a time earlier than the shutter timing. The display of the camera icon 150 should just be controlled so that it may disappear from 11a. In other words, if the display mode is controlled so that the user moves the line of sight to the outside of the display surface as a result of the movement of the camera icon 150, the camera icon 150 remains within the display surface 11a at the shutter timing. Thus, an effect of bringing the user's eyes close to the target position outside the display surface 11a can be expected. The target position is not limited to the shooting position. For example, if the target position is set somewhat above the shooting position, the user can take an image with his eye line slightly up at the shutter timing. On the other hand, if the target position is set below the shooting position, an image in which the user drops some eyes can be shot at the shutter timing. At the shutter timing, a shutter sound is output to give the user a sense of shooting.

  FIG. 18 shows a specific procedure of shutter timing guidance processing executed by the play control unit 32 to realize the shutter timing guidance described above. When the processing of FIG. 18 is started, the play control unit 32 determines the material image P2 (that is, the material image P2 selected in step S63 or S71 in FIG. 16) that is the synthesis target at that time. The shutter timing is acquired (step S81). As described above, the shutter timing is determined in advance for each material image P2 in association with the elapsed time from the start of the try-on play process (the process of step S8 in FIG. 10). For example, the shutter timing is in the middle of the song so as to give the user a sense that shooting is performed for each material image P2 (or for each pause image PP in the screen 130) according to the rhythm and tempo of the song played during the try-on play. It is set together with the position of. However, the shutter timing needs to be set earlier than the material image switching time corresponding to each material image P2 (see step S69 in FIG. 16). As an example, the shutter timing setting value may be recorded in the effect data D12 in association with a song. When six material images P2 are displayed for one user in one try-on play, shutter timings are set at six positions in the song.

  After acquiring the shutter timing, the play control unit 32 causes the camera icon 150 to appear at the initial position of the fitting screen 130 (step S82). Next, the play control unit 32 acquires effect element data such as sound effects to be output when guiding the shutter timing from the effect data D12, and starts outputting the effect element (step S83). Thereafter, the play control unit 32 calculates the position of the camera icon 150 on the try-on screen 130 to be drawn in the next frame based on the elapsed time after the start of the try-on play (step S84). In this position calculation, the position of the camera icon 150 is calculated so that the moving speed at which the camera icon 150 virtually reaches the shooting position SP at the shutter timing is realized. The moving speed of the camera icon 150 may be constant or variable. For example, the moving speed may be set so that the camera icon 150 gradually accelerates or decelerates.

  In a succeeding step S85, it is determined whether or not the previously calculated position of the camera icon 150 has reached the upper end of the fitting screen 130. That is, it is determined whether or not the camera icon 150 has moved to the state of the fitting screen 130-2 in FIG. If the upper end has not been reached, the play control unit 32 causes the camera icon 150 to be in the normal state at the position calculated in step S84 on the try-on screen 130 of the next frame (that is, before bouncing like the try-on screen 130-3). The drawing of the try-on screen 130 of the next frame is controlled so as to be drawn in (state) (step S86). Thereafter, the play control unit 32 returns to step S84.

  On the other hand, if an affirmative determination is made in step S85, the play control unit 32 causes the camera to display an animation representing a state in which the camera icon 150 bounces and disappears upward on the try-on screen 130 of the next frame or later. The drawing of the icon 130 is controlled (step S87). Accordingly, the timing at which the camera icon 150 appears is the time required from the appearance at the initial position until the affirmative determination is made in step S85, that is, the time during which the camera icon 150 is drawn in the normal state, and the step S87. And the time required to reach the shooting position SP when it is assumed that the camera icon 150 has continuously moved up the area outside the display surface 11a as a required moving time. What is necessary is just to set to the time which preceded the movement required time from shutter timing. For example, the movement time of the camera icon 150 outside the display surface 11a can be obtained on the assumption that the movement of the camera icon 150 is continued as it is displayed in the normal state. For example, if the camera icon 150 moves at a constant speed, it is assumed that the camera icon 150 moves while maintaining its speed, and if the camera icon 150 performs an acceleration movement, it is assumed that the camera icon 150 moves while maintaining its acceleration. Good.

  After the process of step S87, the play control unit 32 determines whether or not the shutter timing acquired in step S81 has arrived (step S88). If the shutter timing has not arrived, the play control unit 32 repeats the determination in step S88. When the shutter timing arrives, the play control unit 32 proceeds to step S89 and causes the speaker 17 to output a shutter sound. Thereafter, the play control unit 32 ends the process of FIG. The camera icon 150 is not limited to the example displayed in the form of FIG. For example, as shown in FIG. 19, the camera icon 150 may be displayed with a numerical display for counting down the remaining time until the shutter timing. By adding a countdown display, the user can feel the remaining time until the shutter timing more realistically. In FIG. 19, the fitting screen 130 at the appearance time of the camera icon 150 is not shown, and screen transitions after the countdown start time are shown. Also, illustration of the user's image P1 (also a video because it is displayed continuously) and the pause image PP are omitted.

  FIG. 20 shows a more detailed processing procedure of image composition performed in step S66 of FIG. Hereinafter, the processing of FIG. 20 will be described with reference to FIGS. 21 and 22 as well. First, in the image composition process, the play control unit 32 acquires the material image P2 to be composed, that is, the pose information IP of the material image P2 corresponding to the pose image PP currently displayed on the try-on screen 130 (step S1). Subsequently, the user's current pose information is acquired based on the output from the motion detection device 13 (step S102). Next, the play control unit 32 determines whether or not the user's face is currently detected by the motion detection device 13 (step S103). When the user's face is detected, the play control unit 32 determines that the user's pose is the pose information of the material image P2 based on the user's pose information acquired from the motion detection device 13 (information acquired in step S101). It is determined whether or not the pose recorded in the above is equal to or higher than a predetermined level (step S104).

  If the pose matches at a predetermined level or more, the play control unit 32 determines whether or not an appropriate face image is recorded in the face image data D23 of the storage device 31 (step S105). At this stage, the face image Pf of the user whose face direction is the closest is the face image data D23 with reference to the face direction of the person model of the material image P2 specified by the pose information corresponding to the composition target material image P2. It is determined whether or not the deviation between the face orientation of the face image Pf and the face orientation of the material image P2 is within an allowable range. If it is within the allowable range, the face image Pf is determined as an appropriate image. In this case, the face image Pf to be selected is the face image Pf accumulated in the storage device 31 by the processing of FIG. The allowable range is set as a limit that does not cause unnaturalness when correcting the user's face orientation.

  When it is determined in step S105 that there is an appropriate face image Pf, the play control unit 32 acquires the face image Pf from the face image data D23 (step S106). Subsequently, the play control unit 32 stores the face image Pf in step S36 or S37 in FIG. 14 in order to correct the face image Pf so that the orientation of the face image Pf as the composition target image matches the face orientation in the material image P2. Among the mesh models MM, a mesh model MM corresponding to the orientation of the face image Pf is acquired (step S107). As shown in the example of many face images Pf in FIG. 21, the mesh model MM is a model generated by dividing the user's face image Pf into many polygonal small areas SA with the mouth or the like as a base point. is there. Subsequently, the play control unit 32 uses the face image correction unit 38 to mesh the mesh model MM so that the amount of deviation between the face direction in the compositing target face image Pf and the face direction in the material image P2 becomes zero. Is rotated to correct the orientation of the face image Pf (step S108). This process is realized by converting each vertex coordinate of the small area SA in accordance with the amount of deviation of the face direction. That is, by deforming the small area SA in accordance with the shift amount, the mesh model MM is virtually rotated to change the orientation of the face image Pf. At this time, if the amount of deviation exceeds a predetermined level, the degree of deformation of the small area SA increases, and a level of unnaturalness that cannot be overlooked in the corrected face image Pf may occur. From such a viewpoint, the allowable range of the deviation amount in step S105 of FIG. 20 may be set to any amount range in which unnaturalness does not occur. FIG. 21 shows face images of various orientations arranged in the same drawing for convenience. The compositing target image is one face image Pf, and a large number of face images as shown in FIG. 21 are not simultaneously selected as compositing target images.

  Returning to FIG. 20, when an image with the face orientation corrected is generated in step S108, the play control unit 32 uses the image composition unit 39 to convert the corrected face image Pf into the face portion of the material image P2. Both images are combined so as to be pasted together (step S109). Thereafter, the play control unit 32 uses the image composition unit 39 to perform a predetermined blending process on the boundary portion between the user's face image Pf and the material image P2 (step S110). For example, as shown in FIG. 22, the user's face outline and the outline of the person model face of the material image are smoothly blended so as not to feel uncomfortable (part A in FIG. 22), or The bangs included in the face image and the hair of the person model of the material image are blended so that the boundary portion is not conspicuous (part B in FIG. 22), or the person's face image and the person of the material image Blend processing is performed so that the difference in skin color between the two is not noticeable at the boundary with the model skin. The blending process here may be performed by a known method used when a set of images is synthesized in the image processing. Note that the motion detection device 13 is used to detect the user's height T and width W (see FIG. 22), and the person model of the material image P2 is expanded and contracted vertically and horizontally in accordance with the measurement results so that the user's face image. Processing for reducing or eliminating the sense of incongruity after synthesis due to the difference in height and body shape between Pf and the person model of the material image P2 may be further performed. However, without using the detection result of the motion detection device 13, the material image P2 or the pose image PP corresponding to the material image P2 and the user image P1 are visually recognized on the display surface 11a at an appropriate time such as the start of play. The height and width may be adjusted by overlapping and displaying in a manner that can be performed, and expanding and contracting the user's image P1 vertically and horizontally in accordance with the user's slide operation on the touch panel 35 and the like.

  When the blend process is completed in step S110, the play control unit 32 ends the process of FIG. Thereby, the generation of the fitting image P3 is completed. In this case, it is determined that the synthesis is successful in step S67 of FIG. In the processing of FIG. 20, as a precondition for synthesizing images, it is a requirement that the pose of the user and the pose of the person model of the material image P2 (that is, the pose of the pose image PP) match at a predetermined level or higher. When the fitting image P3 can be generated by the processing of FIG. 20, a predetermined effect may be displayed on the fitting screen 130 to notify the user of the success of the composition. For example, when the result is displayed in step S68 of FIG. 16, the effect can be output in such a manner that a fitting image P3 appears through a predetermined animation. Alternatively, as exemplified in the fitting screen 130 of FIG. 12, a mark 134 indicating that the composition is successful may be displayed. These effects are only superimposed on the display device 11, and are not recorded in the fitting image P3. Note that the screen 130 in FIG. 12 does not indicate a state in which the pose matches a predetermined level or higher.

  If it is determined in step S103 in FIG. 20 that no face has been detected, or if it is determined in step S104 that the poses do not match, the play control unit 32 determines whether or not a timeout has occurred, that is, FIG. After the start of the process, it is determined whether or not a predetermined waiting time has elapsed (step S111). In this case, it is desirable to set the waiting time so that a time-out occurs at a time earlier than the material image switching time determined in step S16 of FIG. 16 and immediately before the material image switching time. If not timed out in step S111, the play control unit 32 gives advice to the user (step S112), and then returns to step S102. Note that the advice in step S112 includes an instruction for the user to adjust the pose to the pose of the pose image PP. That is, the tag information of the material data D11 in which the pose information IP for displaying the pose image PP is included in the pose representing the pose characteristics of the person model in the image P2 as in the post data D3 in FIG. Since the item is included, the phrase described in the posing item is displayed on the screen of the display device 11 or output from the speaker 17 to make the user aware of the pose. For example, on the try-on screen 130 of FIG. 12, the image PM indicating the phrase “Leg cross” is specified in the pose item of the tag information of the material image P2, so that the user is in a pose. Instructions have been given. In the case of timeout in step S111, the play control unit 32 executes a predetermined error process (step S113), and then ends the process of FIG.

  Next, with reference to FIGS. 23 to 29, processing related to the service for the user terminal device 4 will be described. 23 and 24 show transitions of various screens displayed by the display device when the user terminal program PG3 is executed in the user terminal device 4. FIG. First, the outline of various functions implemented in the user terminal device 4 by the user terminal program PG3 will be described with reference to these drawings. In the following description, it is assumed that a mobile terminal device 4b such as a smartphone is used as the user terminal device 4.

  When the program PG3 is activated in the user terminal device 4, a title screen 200 shown in FIG. 23 is displayed, and then a my page screen 210 is displayed as a start screen. When the user terminal program PG3 is started for the first time, a screen for the first time such as a function explanation or a tutorial may be displayed.

  My page screen 210 includes a user profile display unit 211, a try-on image browsing unit 212, and a menu unit 213. The user profile display unit 211 displays a user image and profile information such as a name input by the user. The user's image may be at least a part of the user's own coordinated image P1a stored in the center server 2, or may be an image recorded by the user on the user terminal device 4. In the fitting image browsing unit 212, user fitting images P <b> 3 generated by the arcade machine 3 and stored in the storage unit 41 of the center server 2 are displayed in a list format. By specifying user identification information from the user terminal device 4 and accessing the center server 2, or by reading the two-dimensional code CD1 displayed on the arcade machine 3 and accessing the designated location, the user terminal can be accessed from the center server 2. Data necessary for displaying the My Page screen 210 is provided to the device 4. The menu part 213 is a part that is selected and operated to switch pages to be displayed on the user terminal device 4. When “My Page” is selected from the menu section 213, a My Page screen 210 is displayed. When “My Coordination” is selected and operated, a My Coordinate screen 220 is displayed. The menu unit 213 is displayed in common on each screen. However, in addition to the items prepared in the menu display unit 213, various functions by the program PG3 can be called by performing a predetermined operation. For example, functions such as reading the two-dimensional code CD1 for access or displaying the two-dimensional code CD2 for two-dimensional code authentication are called by a predetermined operation.

  On the my coordination screen 220, a my image display unit 221 that displays the user's own coordinated image P1a stored in the center server 2, and a tag information display unit 222 that displays the contents of the tag information corresponding to the image P1a. And are included. The user can add additional tag information to the tag information display unit 222 by selecting the tag addition button 223. The information added here is transmitted from the user terminal device 4 to the center server 2 and stored as a part of incidental information associated with the user's user coordinated image P1a. A post button 224 is further displayed on the my coordination screen 220. When the user selects the posting button 224, it is assumed that posting of the user coordinated image P1a displayed on the my image display unit 221 is instructed, and the modified image P1b corresponding to the user coordinated image P1a is the evaluation target image P4. Is posted to the center server 2.

  Referring to FIG. 24, when the user posts his / her user coordinate image P1a, the tag information display unit 222 and the evaluation result display unit 225 are alternatively displayed on the my coordinate screen 220 according to the user's selection operation. Is displayed. In the evaluation result display unit 225, evaluation data D4 (refer to FIG. 7) corresponding to the evaluation target image P4 posted by the user (that is, the correction image P1b corresponding to the user coordinate image P1a instructed by the user to be posted). The evaluation result is displayed in accordance with the analysis data D5 associated with. Further, when the evaluation result display unit 225 is displayed, the my image display unit 221 displays information such as the rank totaled by the total data D6 along with the user coordinated image P1. For example, when the user evaluates the evaluation target image P4, a popularity vote can be performed for each option with respect to the category of the coordinate, and the ranking is recorded in the aggregated data D6. It is possible to display on the my image display unit 221 how many ranks the category has been ranked.

  When “Minna no coordination” is selected and operated from the menu portion 213, the everyone coordination screen 230 is displayed. On everyone's coordination screen 230, evaluation target images P4 posted by each user are displayed in a list format. When the user selects an image P4 to be evaluated, the user can input an evaluation for the selected image P4. Here, as described above, the evaluation is input by a method in which the user specifies an option that seems appropriate for the item of tag information that is open to the user as an evaluator. Alternatively, the user as an evaluator may display a question on the user terminal device 4 asking whether or not he / she wants to try on the coordination of the evaluation target image P4 and obtain the answer as at least a part of the evaluation. Good. When the user inputs an evaluation on another person's evaluation target image P4, the evaluation content is transmitted to the center server 2 and recorded as evaluation data D4. The ranking described above may be determined based on the total number of users who answered “I want to try on”. The ranking may be determined according to the number of evaluated users or the number of evaluated users.

  Next, various processes executed by the user terminal device 4 and the center server 2 in order to realize the above-described functions of the user terminal device 4 will be described with reference to FIGS. FIG. 25 shows a procedure of my page browsing processing executed by the user terminal device 4 when the user reads the fitting image P3 by causing the user terminal device 4 to read the two-dimensional code CD1, and FIG. 6 shows a procedure of my page access control processing executed by the user service management unit 41 of the center server 2 in order to process access from the user terminal device 4. Here, it is assumed that the user terminal program PG3 is downloaded in advance to the user terminal device 4 and the function of the program PG3 is used.

  When the my page browsing process of FIG. 25 is started, the browsing control unit 54 of the user terminal device 4 first uses the two-dimensional code reading unit 52 to read the two-dimensional code CD1 displayed on the arcade machine 3. Reading is performed (step S201). Next, the browsing control unit 54 designates the storage location of the access information embedded in the two-dimensional code CD1 read in step S201, that is, the user data D2 including the fitting image P3, and accesses the center server 2. A request is made (step S202).

  On the other hand, when receiving the access request in step S202, the user service management unit 41 of the center server 2 starts the my page access control process of FIG. 26, and first determines the access contents from the user terminal device 4 (step S401). Next, the user service management unit 41 determines whether or not the user data D2 stored at the location specified by the user terminal device 4 is associated with the user identification information (step S402). If there is no association with the user identification information, the user service management unit 41 issues new user identification information (step S403). In this case, the user terminal device 4 may be requested to provide the user personal information necessary for issuing the user identification information, and the identification information may be issued on the condition that the information is obtained. Next, the user service management unit 41 associates the newly issued identification information with the user data D2 stored in the location specified by the access request (step S404). Thereafter, the user service management unit 41 transmits the newly issued identification information to the user terminal device 4 (step S405). Thereafter, the user service management unit 41 transmits the user data D2 requested in step S202 as browsing data to the user terminal device 4 (step S406). If it is determined in step S402 that the association has been made, the user service management unit 41 skips steps S403 to S405 and proceeds to step S406. After the process of step S406, the user service management unit 41 finishes the process of FIG.

  On the other hand, the browsing control unit 54 determines whether or not user identification information is stored in the storage device 51 after requesting access in step S202 of FIG. 25, in other words, whether or not user identification information has already been acquired. (Step S203). If the identification information has been acquired, the browsing control unit 54 proceeds to step S204, acquires data such as a fitting image P3 corresponding to the two-dimensional code CD1 from the center server 2, and displays this on the My Page screen 212 ( Step S204). On the other hand, if it is determined in step S203 that the identification information has not yet been acquired, the browsing control unit 54 waits for notification of the identification information from the center server 2, acquires the identification information after notification, and stores the storage device 51. (Step S205). Thereafter, the browsing control unit 54 proceeds to step S204. After the process of step S204, the browsing control unit 54 ends the process of FIG.

  FIG. 27 shows a procedure of processing when the user posts his / her user coordinated image P1a. When the user operates the posting button 224 in a state where the my coordination screen 220 of FIG. 23 is displayed, the browsing control unit 54 starts the posting processing of FIG. 27, and first, the user coordination image P1a ( Actually, it is the modified image P1b.) The user is asked for permission to use it as an original image of the material image P2 for others, or the disclaimer such as the administrator of the center server 2 is given to the user via the display device of the user terminal device 4. (Step S211). In this case, when the user is asked for consent and the consent is rejected, the subsequent processing may be skipped and the evaluation target image P4 may not be posted.

  Next, the browsing control unit 54 causes the user to select a category to be posted (step S212). For example, the user is allowed to select a posting destination category from options for the “category” item included in the tag information. Next, the browsing control unit 54 acquires tag information added by the user coordinated image P1a to be posted (step S213). For example, since the user can add tag information by selecting and operating the tag addition button 223 on the my coordination screen 220, the tag information added on the user terminal device 4 by the time point of the processing in step S213 is acquired. To do. Alternatively, the user may be requested to add tag information at the time of step S213.

  In subsequent step S214, the browsing control unit 54 generates post data D3 based on the information acquired in steps S212 and S213. As described with reference to FIGS. 8 and 9, since some of the items included in the tag information are specified by the user himself, here, it corresponds to the user coordinated image P <b> 1 a to be posted. The post data D3 may be generated by adding the category specified by the user as tag information to the my data D21. After the posting data D3 is generated, the browsing control unit 54 transmits the posting data D3 to the center server 2 (step S215), and then ends the processing of FIG.

When the post data D3 is transmitted from the user terminal device 4, the user service management unit 41 of the center server 2 starts the post acceptance process of FIG. 27, and first confirms the content of the received post data D3 (step S411). The posted data D3 is stored in the storage unit 42 (step S412). The post data D3 stored here is disclosed to each user and is subject to evaluation.
In addition, after the posting data D3 is transmitted to the center server 2, the release may be suspended until the tag information necessary for the administrator of the center server 2 is added.

  FIG. 28 shows a processing procedure when the user browses and evaluates the evaluation target image P4. When the user selects “everyone's coordination” from the menu unit 213 displayed on the user terminal device 4, the browsing control unit 54 starts the coordinate evaluation process of FIG. Data required for browsing P4 is requested (step S221). In response to this, the user service management unit 41 of the center server 2 starts the evaluation recording process of FIG. 28, and provides the posting data D3 as browsing data to the user terminal device 4 (step S421). The browsing control unit 54 of the user terminal device 4 acquires browsing data provided from the center server 2 (step S222), and displays the evaluation target image P4 included in the browsing data on the coordination screen 230 of everyone. (Step S223). When all the images P4 cannot be displayed at once, the images P4 may be switched according to the user's scrolling operation or the like. Or you may display, after narrowing down evaluation object picture P4 using a category etc. of coordination.

  Next, the browsing control unit 54 causes the user to select an image P4 to be evaluated (step S224), and obtains an evaluation for the selected image P4 from the user (step S225). In this case, as an example, as described above, as described above, the user as the evaluator adds tag information regarding the opened items, or whether the user as the evaluator wants to try on the coordination of the image P4. It may be performed by a method such as answering. Next, the browsing control unit 54 designates the evaluated image P4 and transmits the evaluation result to the center server 2 (step S226). Thereafter, the browsing control unit 54 ends the process of FIG.

  On the other hand, when receiving the evaluation result from the user terminal device 4, the user service management unit 41 of the center server 2 generates new evaluation data D4 and records the evaluation result in the data D4 (step S422). As described above, the structure of the evaluation data D4 may be the same as the posted data D3 illustrated in FIG. 8, but the data structure may be appropriately changed according to the evaluation contents to be recorded. Next, the user service management unit 41 updates the analysis data D5 so that the content of the newly generated evaluation data D4 is reflected in the analysis data D5 corresponding to the evaluated image P4 (step S423). For example, when the evaluator's user designates the “job” -oriented coordinate for the “scene” item of the tag information, 1 is added to the evaluation count of “work” recorded in the analysis data D5. Thus, the evaluation result is reflected in the analysis data D5. Alternatively, when the user of the evaluator answers “I want to try on”, the evaluation result is reflected in the analysis data D5 by adding 1 to the number of responses of “I want to try on” recorded in the analysis data D5. After completing the process of step S423, the user service management unit 41 ends the evaluation recording process of FIG.

  The evaluation result recorded in the evaluation data D4 may be reflected in the analysis data D5 each time the evaluation data D4 is created as described above, but batch processing is performed every time a certain period elapses. As may be done. In this case, the evaluation data D4 created from the previous processing to the current processing may be acquired, and the analysis data D5 may be updated according to the contents recorded in the evaluation data D4. Similarly, the process from the analysis data D5 to the aggregated data D6 may be executed as a batch process at regular intervals by the user service management unit 41.

  FIG. 29 shows a procedure of processing for causing the user of the person to be evaluated to browse the evaluation on the evaluation target image P4. When the user instructs the user terminal device 4 to display the evaluation result, the browsing control unit 54 starts the evaluation information browsing process of FIG. 29, and first evaluates the evaluation target image P4 posted by itself to the center server 2. A request for information transmission is made (step S231). In response to this, the user service management unit 41 of the center server 2 starts the evaluation information providing process of FIG. 29 and acquires the evaluation information requested from the user terminal device 4 (step S431). The evaluation information is, for example, information on the analysis data 5 corresponding to the evaluation target image P4 posted by the user, and information related to the evaluation target image P4 in the total data D6 (for example, the rank of the evaluation target image P4). . When obtaining the evaluation information, the evaluation information may be specified using the image ID of the user coordinated image P1a (or the corrected image P1b) that is the original image of the evaluation target image P4 as a clue, or the user of the person to be evaluated Evaluation information may be specified using identification information.

  When acquiring the requested evaluation information, the user service management unit 41 provides the evaluation information to the user terminal device 4 (step S432), and thereafter ends the evaluation information providing process. On the other hand, the browsing control unit 54 of the user terminal device 4 acquires the evaluation information provided from the center server 2 (step S232), and displays the obtained information for the user (step S233). For example, the browsing control unit 54 causes the evaluation result display unit 225 on the my coordination screen 220 in FIG. 24 to display the evaluation result according to the content of the analysis data D5, and the my image display unit 221 counts the total data D6. The evaluation information is displayed to the user by displaying information such as the ranking. By presenting the evaluation results to the user of the person being evaluated, the user as the person to be evaluated objectively grasps how his / her coordination is evaluated by others and serves as a reference for his / her coordination. Can be used.

  Next, referring to FIG. 30, the material data addition process executed by the user service management unit 41 of the center server 2 in order to select the material candidate image P5 from the evaluation target image P4 and add the material data D11. explain. The material data addition process in FIG. 30 is a process that is repeatedly executed at regular intervals. When the process of FIG. 30 is started, the user service management unit 41 first refers to the total data D6 and acquires evaluation information necessary for selecting an evaluation target image P4 that should be a candidate for the material data D11 ( Step S501). For example, when the number of responses that the user wants to try on is aggregated for each image P4, the ranking is stored in the aggregated data D6, and the aggregated result is disclosed to the user, it is ranked higher in the ranking. When it is desired to select the evaluation target image P4 as a candidate, the ranking information recorded in the total data D6 is acquired as evaluation information.

  After acquiring the evaluation information, the user service management unit 41 selects an evaluation target image P4 that satisfies a predetermined condition (step S502). The condition in this case is set so that the evaluation target image P4 for which a relatively high evaluation is obtained is selected. For example, the condition is set so that an image P4 that has acquired a certain ranking in the ranking is selected. The condition may be set for each category recorded as part of the tag information in the post data D3. In addition, conditions may be set so that a predetermined number of images P4 are selected in descending order of the number of evaluations for each option (see FIG. 9) prepared in the scene item of the tag information. In this case, based on the evaluation data D4, the number of evaluations for each option relating to the scene item of each evaluation target image P4 is aggregated and recorded in the analysis data D5. By comparing the number of evaluations for each option and recording the evaluation number ranking for each selection in the total data D6, an appropriate image P4 can be selected from the total data D6.

  Subsequently, the user service management unit 41 generates material candidate data D7 corresponding to the image P4 selected in step S502 (step S503). In this case, since the evaluation target image P4 corresponds to the corrected image P1b, the material candidate image P5 is also subjected to processing that makes it difficult to identify an individual on the face portion of the user (see FIG. 2). The candidate material image D7 is generated by associating and recording the candidate material image P5 and tag information and the like. As the pose information IP associated with the material candidate image P5, pose information detected by the motion detection device 13 when the user's image P1 that is the basis of the evaluation target image P4 is captured by the arcade machine 3 is recorded. However, instead of or in addition to this, pose information (basic pose information) associated with the material image P2 when the fitting image P3 is synthesized may be recorded. For example, when the evaluation target image P4 is generated from the user's image P1 captured when it is determined in step S104 in FIG. 20 that the pose matches a certain level or more, the motion detection device 13 at the time of capturing the image P1. The detected user pose approximates the pose of the person model of the material image P2 to be synthesized, but is not necessarily the same. In this case, if the user's own pose information is recorded as the pose information IP corresponding to the material candidate image P5, the pose information IP when using the material candidate image P5 as the material image P2 is the material image from which the composition is based. This is somewhat different from the pose information of the P2 person model. If such differences are accumulated, the pose of the material data D11 created by the processing of FIG. 30 may gradually depart from the pose information of the person model in the original material data D11. In order to prevent such a change, the pose information IP corresponding to the material image P2 to be synthesized may be recorded as the pose information IP corresponding to the material candidate image P5. In that case, the pose image PP displayed in step S64 of FIG. 16 is displayed based on the pose information IP corresponding to the material image P2 used in the synthesis of the try-on image P3, and also in step S104 of FIG. The suitability of the user's pose is determined based on the pose information IP corresponding to the material image P2 used in the synthesis of the image P3. In addition, in step S503, tag information may be added or modified according to the analysis data D5 or the like. For example, it is possible to determine what kind of evaluation the image P4 has obtained from the user according to the analysis data D5, and each item of the tag information may be changed to contents designated by more users. After generating the material candidate data D7, the user service management unit 41 adds the obtained material candidate data D7 as material data D11 to the material database DB1 (FIG. 2) (step S504). Thereafter, the user service management unit 41 finishes the process of FIG. The material data D11 added to the material database DB1 is provided to the arcade machine 3 in the same manner as the material data D11 prepared in advance by the administrator of the center server 2, and is used for the user.

  According to the processing in FIG. 30, regarding the evaluation target image P4 that has been relatively highly evaluated by the user, the material image P2 is generated based on the user image P1 photographed as the original image, and this is generated by other users. Can be used. Since there is a virtually unlimited number of coordination of clothes, the administrator of the center server 2 prepares all of the material data D11 in advance, or the administrator or the like simply adds the material data D11 to various users. It is difficult to respond to this request, and it imposes a great work burden on the administrator. On the other hand, by performing the process of FIG. 30, the material data D11 can be gradually increased based on the coordinated image P1 that has received a relatively high evaluation from the user. As a result, the system 1 can be gradually improved so as to be widely adapted to various user preferences and demands while reducing the burden on the administrator or the like.

  As described above, according to the trial-fitting simulated experience system 1 of this embodiment, the pose image PP corresponding to the pose of the person model in each of the plurality of material images P2 can be played back as a production element for the user. While synchronizing with the display of the decoration image PD, the images are displayed so that the images are sequentially switched at a predetermined time interval, and shooting is repeated while the user pauses in accordance with the pose images PP. Therefore, it is possible to give the user a feeling of taking a picture while changing clothes in order while riding a production. Then, the user visually recognizes various try-on images P3 that are displayed and output by the display device 11 or printed and output by the printer 7, and the user feels that the user has tried various coordination. Can taste. As a result, it is possible to greatly satisfy the user's needs regarding the coordination of clothes. Due to the synergistic effect of the effect during the try-on play and the sequential switching display of the pose image PP, the try-on play can be advanced rhythmically or at a good tempo, and the entertainment action is also high. In addition, since a plurality of try-on images P3 are sequentially generated in a short time, the arcade machine 3 that is a business terminal device is not occupied by a specific user for a long time, and the profit rate of the arcade machine 3 is not impaired.

  Displaying the user's image P1 together with the pose image PP creates an atmosphere in which the user performs an action that poses by imitating the person model of the material image P2 in synergy with the production effect, thereby creating an interest in try-on play. It is possible to increase. In addition, when compositing face images, face images with different face orientations are sequentially recorded, and an image closest to the face orientation of the material image P2 is selected as a compositing target image, so that the occurrence of unnaturalness due to compositing is also suppressed. Can do. Thereby, a high-quality fitting image P3 can be provided to the user, and the interest of the system 1 can be further enhanced. By performing composition after correcting the orientation of the face image Pf selected as the composition target image to match the face orientation of the material image P2, it is possible to synthesize a fitting image P3 that is more natural, such as face contour The effect can be further enhanced by the blending process. Since the user is repeatedly photographed while the user is trying to pose and the face image Pf is recorded for each face orientation, it is possible to record face images in various orientations, and thereby the try-on image P3. It is possible to further enhance the effect of suppressing the unnaturalness of the image. If the pose does not match, an instruction is given to the user as illustrated in the screen 130 of FIG. 12, thereby facilitating the user's action to accumulate images with different face orientations more easily and reliably, As a result, the quality of the composition of the fitting image P3 can be improved.

  On the other hand, since the shutter timing is set for each material image and is guided to the user by the method illustrated in FIGS. 17 to 19, the line of sight is directed to the target position outside the display surface such as the shooting position of the camera 12. The possibility that images can be acquired can be increased. Thereby, the possibility that the fitting image P3 whose line of sight is shifted from the target position is reduced, and the user's satisfaction with the fitting image P3 can be enhanced.

  Furthermore, in parallel with the acquisition of the face image for composition, the user's own image P1 is acquired and stored as a user coordinated image P1a on the server side, and many users are evaluated for coordination through user posting. The evaluation can be fed back to the user. Thereby, not only enjoying the coordination, but also allowing the user to grasp the evaluation of others for his / her coordination, thereby providing the user with an opportunity to hone the user's sense of coordination or deepen knowledge. Furthermore, since the material image P2 corresponding to the coordinated image P1a, which is relatively highly evaluated among users, can be generated and used for other users, the material image P2 can be additionally supplemented efficiently. In addition, the system 1 can be gradually improved so as to meet various user preferences regarding coordination.

  The present invention is not limited to the form described above, and various modifications are possible. Hereinafter, it demonstrates in order. First, in the above embodiment, a two-step process is adopted in which the face image of the user is corrected to match the face direction of the person model in the material image, and the corrected face image is combined with the material image. It is not always necessary to correct the face image. Depending on the level of unnaturalness allowed, it is possible to obtain a composite image of sufficient quality simply by selecting the face image with the smallest deviation in face orientation as the target image and compositing it without correcting the face orientation. May be. In such a case, the correction process may be omitted. It is sufficient that the blending process is performed as much as necessary, and the blending process may be omitted depending on the situation of the user's face image and the material image.

  In the above embodiment, the user's photographing is performed by the camera 12 while the user's face orientation is detected by using the motion detection device 13, but the face orientation is detected from the image photographed by the camera 12. If possible, the motion detection device 13 may be omitted. Alternatively, when the motion detection device 13 itself includes a camera, the camera may also be used as a photographing unit.

  In the above embodiment, the arcade machine 3 for generating the fitting image P3 is taken up as an example of the image synthesizing apparatus in the present invention, but the present invention is not limited to such an example, and the face obtained by photographing the user As long as the image is synthesized with the face part of the person model of the material image, it can be widely applied regardless of the purpose of obtaining the synthesized image. The image composition device according to the present invention is not limited to an example configured as at least a part of a business terminal device. For example, the image synthesizing apparatus according to the present invention may be configured using the user terminal device 4. On the other hand, the image composition device of the present invention may be configured as a server client type system. For example, the image synthesizing apparatus according to the present invention is configured as a so-called thin client type system by providing only the essential hardware on the client side, such as the user's photographing unit and display unit, on the client side and performing the control on the server side. May be. Alternatively, an image composition device system may be constructed by a server and a user terminal device without including a business terminal device. For example, information such as a captured image is acquired from the camera function by connecting to a user terminal device or a camera function having the functions of the camera 12 and the motion detection device 13 (hereinafter referred to as “camera function” in this paragraph). By using the user terminal device that can be used instead of the arcade machine 3, an image composition apparatus can be constructed even in a place where the arcade machine 3 is difficult to install, such as at home.

  Various aspects of the present invention derived from each of the above-described embodiments and modifications will be described below. In the following description, in order to facilitate understanding of each aspect of the present invention, reference numerals in the accompanying drawings are added in parentheses, but the present invention is not limited to the illustrated embodiments.

  The image synthesizing device (3) according to one aspect of the present invention stores a material image (P2) in which a person model is recorded in association with face direction information for specifying the face direction of the person model ( 31), photographing means (12) for photographing the user, and photographing control means (32, S21, S31, S73) for controlling the photographing means so that an image (P1) including at least the face of the user is repeatedly photographed. ), Face orientation determining means (32, S33) for determining the orientation of the user's face when photographed by the photographing means, the discrimination result of the face orientation discriminating means, and face orientation information corresponding to the material image Based on the image selection means (32, S105, image selection means (32, S105), which selects a user face image (Pf) suitable for composition with the material image from a plurality of images of the user photographed by the photographing means. 106) and a synthesizing unit (32, S107 to S110) for generating a synthesized image (P3) in which the face image of the user is synthesized with the face portion of the material image based on the synthesis target image and the material image. And.

  An image composition method according to an aspect of the present invention corresponds to an image of a user's face (Pf) photographed by the photographing means (12) and a material image (P2) in which a person model is recorded, corresponding to the material image. An image composition method for compositing with reference to face orientation information for specifying the face orientation of the person model attached, and controlling the photographing unit so that an image including at least the face of the user is repeatedly photographed Determination procedure (S21, S31, S73), determination procedure (S33) for determining the orientation of the face of the user when the image is taken by the imaging means, determination results in the procedure for determining the orientation of the face, and the material image A procedure for selecting a user's face image suitable for composition with the material image as a composition target image from a plurality of images of the user photographed by the photographing means based on the face orientation information corresponding to (S) 05, S106), and a procedure (S107 to S110) for generating a composite image (P3) in which the face image of the user is combined with the face portion of the material image based on the composition target image and the material image. , With.

  The computer program (PG1) according to an aspect of the present invention includes a material image (Pf) of a user's face photographed by the photographing means (12) and a material image (P2) on which a person model is recorded. A computer program for compositing by a computer (30) with reference to face orientation information specifying the face orientation of the person model associated with the image, the computer including an image including at least the face of the user A photographing control means (32, S21, S31, S73) for controlling the photographing means so that (P1) is repeatedly photographed, and a face orientation judgment for judging the orientation of the user's face when photographed by the photographing means. Means (32, S33), suitable for combining with the material image based on the discrimination result of the face orientation discrimination means and the face orientation information corresponding to the material image. Based on the image selection means (S32, S105, S106) for selecting the image of the user's face as a compositing target image from the plurality of images of the user captured by the photographing means, and the compositing target image and the material image The composition unit is configured to function as a synthesis unit (32, S107 to S110) that generates a composite image (P3) in which the face image of the user is combined with the face part of the material image.

  According to each aspect of the present invention, since the user's face moves while the user is repeatedly photographed, it is possible to acquire images of many faces with different face orientations. Since the orientation of the face when the user is photographed is determined, the image of the user's face can be associated with the orientation of the face. Furthermore, by comparing the orientation of the user's face with the orientation of the face of the person model in the material image, the face image having a face orientation suitable for composition, for example, the orientation of the face of the person model in the material image An image of a user's face having a face orientation with a relatively small deviation or a face image with the smallest deviation of the face orientation can be selected as a synthesis target image. Therefore, by synthesizing the face image with the face portion of the material image based on the selected composition target image and the material image, it is possible to suppress unnaturalness that may occur at the time of synthesis and obtain a natural synthesized image as much as possible. it can.

  Note that the computer program according to one embodiment of the present invention may be provided in a state of being stored in a storage medium. By using this storage medium, for example, by installing and executing the computer program according to the present invention in a computer, the game system of the present invention can be realized using the computer. The storage medium storing the computer program may be a non-transitory storage medium such as a CDROM.

  In one aspect of the present invention, the image selection unit is configured such that, from a plurality of images of the user, a deviation of a user's face relative to a face specified by face information corresponding to the material image is relatively A small face image may be selected as the synthesis target image. Accordingly, it is possible to select a face image that is advantageous in suppressing unnaturalness of the composite image caused by the face orientation shift.

  In one aspect of the present invention, the synthesizing unit is configured to determine a face direction in the compositing target image selected by the image selecting unit and a face direction of the person model specified by the face direction information corresponding to the material image. Image correcting means (32, S34, S108) for correcting the orientation of the face image selected by the selecting means so as to reduce the deviation, and composition for combining the corrected face image with the face portion of the material image. And execution means (32, S109, S110). According to this, since the face image selected as an image suitable for composition is corrected so that the deviation of the face direction is reduced and then combined with the material image, the unnaturalness of the combined image is further suppressed. Can do.

  Further, the image correction unit is configured to allow the user in the compositing target image so that the orientation of the user's face in the compositing target image matches the face direction of the person model specified by the face orientation information corresponding to the material image. The face orientation may be corrected. According to this, since the image selection means corrects the orientation so that the orientation of the face matches, it is possible to further suppress the unnaturalness of the composite image.

  In addition, the image correction unit generates a mesh model (MM) corresponding to the image of the user's face in the synthesis target image (S34), and the coordinates of a large number of small regions (SA) constituting the mesh model are You may correct | amend the said user's face direction by correct | amending according to the shift | offset | difference of the face direction in a synthetic | combination object image, and the face direction of the said person model. Thereby, it is possible to perform correction so that the image of the user's face is virtually rotated and the face orientations are matched. As a result, it is possible to generate a synthetic image that is as natural as possible.

  In one aspect of the present invention, the composition execution means performs a blending process on at least one element of a face contour, bangs, and skin color at a boundary portion between the user's face image and the material image. You may make it (S110). For facial contours, bangs, and skin color elements, differences are likely to occur between the user and the person model, so by blending at least one of these, the differences at the borders are processed inconspicuously, It is possible to generate a synthetic image closer to nature.

  In one aspect of the present invention, the material image stored in the storage unit may be an image in which the face portion of the person model is deleted. According to this, it is not necessary to process the eyes, nose and the like of the person model at the time of synthesis, and the labor of image synthesis can be reduced.

1 Try-on simulated experience system 2 Center server 3 Arcade machine (business terminal)
4 User terminal device 5 Network 7 Printer 11 Display device (display means)
11a Display surface 12 Camera 13 Motion detection device 30 Control unit 31 Storage device 32 Play control unit 40 Arcade service management unit 41 User service management unit 42 Storage unit 50 Control unit 51 Storage device 54 View control unit 150 Camera icon

Claims (9)

Material image storage means for storing a material image in which a person model is recorded in association with face orientation information for specifying a face orientation of the person model;
Photographing means for photographing the user;
Photographing control means for controlling the photographing means so that images including at least the face of the user are repeatedly photographed;
Face orientation determining means for determining the orientation of the user's face when imaged by the imaging means;
User image recording means for recording each of the images repeatedly photographed by the photographing means in association with information specifying the face orientation determined by the face orientation determining means in the user image storage means;
An image of the user's face suitable for combining with the material image based on the information specifying the face direction corresponding to the image recorded in the user image storage means and the face direction information corresponding to the material image Image selecting means for selecting as a synthesis target image from a plurality of images of the user recorded in the user image storage means ,
Based on the synthesis target image and the material image, a synthesis unit that generates a synthesized image obtained by synthesizing the face image of the user with the face portion of the material image;
An image synthesizing apparatus.   The image selection means is configured to combine, from the plurality of images of the user, a face image in which a deviation of the user's face direction relative to the face direction specified by the face direction information corresponding to the material image is relatively small. The image composition device according to claim 1, wherein the image composition device is selected as a target image. The synthesizing unit is configured to reduce a deviation between a face direction in the synthesis target image selected by the image selecting unit and a face direction of the person model specified by the face direction information corresponding to the material image. Image correcting means for correcting the orientation of the face image selected by the image selecting means;
Combining execution means for combining the corrected face image with the face portion of the material image;
The image composition device according to claim 1, further comprising:   The image correcting unit is configured to match the face of the user in the compositing target image so that the orientation of the user's face in the compositing target image matches the face direction of the person model specified by the face orientation information corresponding to the material image. The image synthesizing apparatus according to claim 3, wherein the orientation of the image is corrected.   The image correction unit generates a mesh model according to the image of the user's face in the synthesis target image, and determines the coordinates of a large number of small regions constituting the mesh model and the orientation of the face in the synthesis target image and the person model. The image composition device according to claim 4, wherein the orientation of the user's face is corrected by performing correction according to a deviation from the orientation of the user's face. 6. The blending unit according to any one of claims 3 to 5, wherein the composition execution unit performs a blending process on at least one element of a face outline, bangs, and skin color at a boundary portion between the user's face image and the material image. The image composition device according to one item. The image composition device according to claim 1, wherein the material image stored by the material image storage unit is an image in which a face portion of the person model is deleted. While referring to the face orientation information for identifying the face orientation of the person model, the face image of the user photographed by the photographing means and the material image in which the person model is recorded are associated with the material image. An image composition method to synthesize,
A procedure for controlling the photographing means so that an image including at least the face of the user is repeatedly photographed;
A procedure for determining the orientation of the user's face when photographed by the photographing means;
A procedure for recording each of the images repeatedly photographed in the procedure for controlling the photographing means in the user image storage means in association with information for identifying the face orientation determined in the procedure for determining the face orientation. When,
Based on the information for specifying the orientation of the face corresponding to the image recorded in the recording procedure and the face orientation information corresponding to the material image, an image of the user's face suitable for synthesis with the material image is obtained. A procedure for selecting as a synthesis target image from a plurality of images of the user recorded in the user image storage means ;
A procedure for generating a composite image in which a face image of a user is combined with a face portion of the material image based on the composition target image and the material image;
An image composition method comprising: While referring to the face orientation information for identifying the orientation of the face of the person model, the face image of the user photographed by the photographing means and the material image in which the person model is recorded are associated with the material image. A computer program for synthesis by a computer comprising:
Photographing control means for controlling the photographing means so that an image including at least the face of the user is repeatedly photographed;
Face orientation determining means for determining the orientation of the face of the user when imaged by the imaging means;
User image recording means for recording each of the images repeatedly photographed by the photographing means in association with information identifying the face orientation determined by the face orientation determining means in the user image storage means;
An image of the user's face suitable for combining with the material image based on the information specifying the face direction corresponding to the image recorded in the user image storage means and the face direction information corresponding to the material image On the face portion of the material image based on the image to be synthesized and the material image based on the image to be synthesized from the plurality of images of the user recorded in the user image storage means. A computer program configured to function as a combining unit that generates a combined image obtained by combining images of a user's face.