JP2023086000A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

To further improve quality of a service provision using a multi-viewpoint image.SOLUTION: An information processing device includes: a guide display part for displaying a predetermined photographing guide on a screen when photographing a multi-viewpoint image; and a guide change part for changing the photographing guide displayed on the screen according to the viewpoint every time the viewpoint is moved when shooting. The guide display part displays, on the screen, a guide relating to at least one of a pose, the facial expression, and a holding manner of a person to be photographed among subjects as the photographing guide. Also, the guide display part displays a silhouette, an outline, or a translucent display of the subject on the screen as the shooting guide.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

2. Description of the Related Art Conventionally, there has been provided a technique for generating a multi-viewpoint image (free-viewpoint image), which is a plurality of images of a subject (capturing target) captured from different viewpoints (see, for example, Patent Document 1).

JP 2016-119513 A

However, the conventional technology described above may not be sufficient to provide services using multi-viewpoint images. For example, in the conventional technology described above, users are allowed to utilize multi-viewpoint images, but there is room for improvement in terms of the quality of service provision using multi-viewpoint images.

The present application has been made in view of the above, and an object of the present application is to further improve the quality of service provision using multi-view images.

An information processing apparatus according to the present application includes a guide display unit that displays a predetermined shooting guide on a screen when shooting a multi-viewpoint image, and the shooting guide that is displayed on the screen every time a viewpoint is moved during shooting. and a guide changing unit that changes according to the viewpoint.

According to one aspect of the embodiment, it is possible to further improve the quality of service provision using multi-view images.

FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. FIG. 2 is an explanatory diagram showing an overview of the display of the shooting guide. FIG. 3 is an explanatory diagram showing an overview of replacement of a face image of a photographed model. FIG. 4 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 5 is a diagram illustrating a configuration example of a terminal device according to the embodiment; FIG. 6 is a diagram illustrating a configuration example of an information providing apparatus according to the embodiment; FIG. 7 is a diagram showing an example of a user information database. FIG. 8 is a diagram showing an example of the history information database. FIG. 9 is a diagram showing an example of an image information database. FIG. 10 is a flowchart illustrating a processing procedure according to the embodiment; FIG. 11 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Outline of information processing method]
First, an outline of an information processing method performed by an information processing apparatus according to an embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. Note that in FIG. 1, a case of providing a service using multi-view images will be described as an example.

As shown in FIG. 1 , the information processing system 1 includes a terminal device 10 and an information providing device 100 . The terminal device 10 and the information providing device 100 are communicably connected to each other by wire or wirelessly via a network N (see FIG. 4). In this embodiment, the terminal device 10 cooperates with the information providing device 100 .

The terminal device 10 is a smart device such as a smartphone or tablet used by a user U (user), and communicates with an arbitrary server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). It is a portable terminal device capable of performing The terminal device 10 has a screen such as a liquid crystal display and has a touch panel function, and the user U can perform a tap operation, a slide operation, a scroll operation, or the like with a finger, a stylus, or the like, and display data such as contents. Accepts various operations for . An operation performed on an area where content is displayed on the screen may be an operation on the content. In addition, the terminal device 10 may be an information processing device such as a desktop PC (Personal Computer) or a notebook PC as well as a smart device.

The information providing device 100 cooperates with the terminal device 10 of each user U, and provides the terminal device 10 of each user U with API (Application Programming Interface) services for various applications (hereinafter referred to as apps) and various It is an information processing device that provides data, and is realized by a server device, a cloud system, or the like.

Also, the information providing device 100 may be an information processing device that provides some web service online to the terminal device 10 of each user U. FIG. For example, the information providing apparatus 100 provides, as Web services, Internet connection, search service, SNS (Social Networking Service), electronic commerce (EC), fashion coordination (pictures (still images) and videos of wearing fashion items). Posting sites, electronic payments, online games, online banking, online trading, accommodation/ticket reservations, video/music distribution, news, maps, route searches, route guidance, route information, operation information, weather forecasts, etc. may be provided. In practice, the information providing apparatus 100 may cooperate with various servers that provide web services as described above, mediate web services, or take charge of web service processing.

Note that the information providing device 100 can acquire user information about the user U. FIG. For example, the information providing device 100 acquires information about attributes of the user U, such as the user's U gender, age, and area of residence. The information providing apparatus 100 stores and manages identification information indicating the user U (user ID, etc.) and information related to the attributes of the user U. FIG.

Further, the information providing apparatus 100 acquires various types of history information (log data) indicating actions of the user U from the terminal apparatus 10 of the user U or from various servers based on the user ID or the like. For example, the information providing device 100 acquires a location history, which is a history of the user U's location and date and time, from the terminal device 10 . The information providing apparatus 100 also acquires a search history, which is a history of search queries input by the user U, from a search server (search engine), an electronic commerce server, or a posting server. Further, the information providing apparatus 100 acquires a browsing history, which is a history of contents and products (fashion items) browsed by the user U, from a content server, an electronic commerce server, or a posting server. Further, the information providing apparatus 100 acquires a purchase history (payment history), which is a history of contents and products (fashion items) purchased or paid for by the user U, from an electronic commerce server or a payment processing server. Further, the information providing apparatus 100 may acquire the exhibition history and sales history, which are the history of the user U's exhibition in the marketplace, from the electronic commerce server or the payment processing server. The information providing apparatus 100 also posts a posting history, which is a history of fashion coordination (fashion items) posted by the user U, and a support history, which is a history of fashion coordination (fashion items) supported (liked) by the viewer. Acquire from a server or SNS server.

In this embodiment, the information providing apparatus 100 uses image analysis and AI ( Determined by Artificial Intelligence (AI) and displayed in the appropriate position. Here, the information providing apparatus 100 identifies fashion items included in the image by image analysis, and attaches annotation tags (hereinafter referred to as tags) corresponding to the fashion items. Note that adding a tag means displaying a tag at an appropriate position in the vicinity of the object to be tagged. For example, the information providing apparatus 100 may be used in an image of the user wearing a fashion item as a photograph or video for posting to a posting site or SNS, or in an image of another user that has been photographed and posted. , to assign tags corresponding to fashion items.

For example, the information providing device 100 attaches a tag displaying detailed information of the fashion item. In addition, the information providing apparatus 100 may display the product page of the fashion item (sales page/purchase page/advertisement page, etc.), or other fashion items related to the fashion item (products that are often purchased together, products of the same brand, etc.). , etc.) may be added.

Also, the information providing apparatus 100 may attach a characteristic tag to the fashion item. For example, the information providing apparatus 100 attaches tags that are highly compatible with fashion items in terms of the color wheel. Alternatively, a tag that is estimated to increase the support rate (like rate) of viewers on the posting site or SNS is added. Alternatively, a tag that is presumed to increase the purchase rate of fashion items by viewers is added. As a result, it is possible to encourage the viewer to purchase the fashion item.

Fashion information about user's fashion preferences can be obtained from each user U's attribute information, history information, and the like. The fashion information includes fashion items worn by the user U and information on fashion items viewed by the user U (fashion item information). In this embodiment, fashion information is behavior information (search, selection , viewed, purchased, posted). The fashion information includes information (fashion item information) related to fashion items searched, browsed, purchased, owned, posted, etc. by the user U.

Users' fashion preferences are collected from various history information (log data) such as e-commerce sites such as fashion mail-order sites and fashion coordination posting sites that connect accounts with this system (or are part of this system). determined by logic that calculates/estimates based on For example, the information providing apparatus 100 calculates/estimates the "combination of clothes and tags" from clothes purchased by the user at a fashion mail-order site (defined as clothes wearing clothes that match the user information) or browsed clothes. and assign the most appropriate tag to match with clothes. In addition, the information providing apparatus 100 calculates/estimates the “combination of clothes and tags” from the clothes browsed by the user on the fashion coordination posting site, and assigns the optimum tag to the combination with the clothes.

Further, the information providing apparatus 100 may add tags based on information such as lifestyle behavior/habits and mood of the user in addition to fashion information. That is, the information providing apparatus 100 may change the content and form of the tag for each user who views the same fashion item.

In the present embodiment, the information providing apparatus 100 is used in a service for taking and posting fashion photographs (images) such as a fashion coordination posting site (for example, "WEAR" (registered trademark)). bags, shoes, hats, etc.). For example, as tagging, a relationship with a product page of an electronic commerce site such as a fashion mail-order site (for example, "ZOZOTOWN" (registered trademark)) is set. The tag is superimposed on the photo. When a tag is clicked, a transition is made to a product page where the product (fashion item) is sold on an e-commerce site such as a fashion mail-order site (for example, "ZOZOTOWN" (registered trademark)).

In addition, the information providing apparatus 100 displays multi-viewpoint images on a fashion coordination posting site (for example, “WEAR” (registered trademark)). For example, when a photo (image) is displayed by clicking or scrolling on a fashion coordination posting site (for example, “WEAR” (registered trademark)), the information providing device 100 automatically/over time/user's operation switch to a different image depending on the

[1-1. Annotation tag]
In this embodiment, the information providing apparatus 100 assigns the same gazing point (tagging target) from each viewpoint image of the multi-view image when an annotation tag is attached to the shooting target in the multi-view image. It is detected, and a tag is set and displayed in the vicinity of the same fixation point.

As shown in FIG. 1, the information providing apparatus 100 acquires multi-viewpoint images from the terminal device 10 of the user U who is the poster via the network N (see FIG. 4) (step S1). For example, the information providing apparatus 100 receives images taken from various viewpoints used for creating multi-viewpoint images from the terminal device 10 of the user U who is the contributor via the network N (see FIG. 4). get. In FIG. 1, in the image, M denotes a person to be photographed (photographed model), B denotes a bag to be photographed, and T denotes a tag.

Next, the information providing apparatus 100 performs image recognition (Image Recognition) or machine learning on all of the acquired images to recognize, specify, and classify the shooting target (step S2). For example, the information providing apparatus 100 identifies imaging targets included in each image and classifies them by category. A plurality of subjects may be photographed.

Next, the information providing apparatus 100 estimates the position of the imaging target within the multi-viewpoint image (position within the image) (step S3). In the present embodiment, the information providing apparatus 100 estimates (or specifies) the three-dimensional position of the shooting target included in each image constituting the multi-viewpoint image (for each image of each viewpoint). The three-dimensional position of the object to be imaged may be an absolute position such as coordinates in an image, or may be a relative position from a reference point or another object to be imaged. Also, the three-dimensional position is merely an example.

Next, the information providing apparatus 100 selects, from the user U, who is the poster or the viewer, the object to be tagged among the shooting objects (candidates for annotation objects) whose three-dimensional position in the multi-view image is estimated. The selection of (annotation target) is accepted (step S4). For example, the information providing apparatus 100 receives a product (fashion item) to be tagged (annotated) from the terminal device 10 of the user U, who is a poster or a viewer, via the network N (see FIG. 4). , the specification of the product page of the corresponding product to be associated is accepted. At this time, the information providing device 100 searches each product page of an e-commerce site such as a fashion mail-order site (for example, “ZOZOTOWN” (registered trademark)) for similar images of the product by image recognition or machine learning, Based on this, the product page of the product may be automatically specified.

Note that if the information providing apparatus 100 does not receive a selection of a tagging target (annotation target) from the user U who is a poster or a viewer, the information providing apparatus 100 selects all shooting targets in the multi-view image as tagging targets ( Annotation target). In addition, the information providing apparatus 100 unconditionally tags all shooting targets in the multi-view image regardless of the selection of the tagging target (annotation target) by the user U who is the poster or the viewer. It may be a target (annotation target). In addition, the information providing apparatus 100 may determine a tagging target (annotation target) among shooting targets (annotation target candidates) according to presetting.

Further, the information providing apparatus 100 may use a learning model to estimate a tagging target (annotation target) among shooting targets whose three-dimensional positions in the multi-view image are estimated. For example, the information providing apparatus 100 may build a learning model by learning about combinations of shooting targets that have been selected as targets for tagging in the past and tags that have been assigned to the targets for tagging. Then, when the shooting target is input to the learning model, the information providing apparatus 100 may infer and output an appropriate tag if the shooting target is to be tagged.

Next, the information providing apparatus 100 sends, via the network N (see FIG. 4), to the terminal device 10 of the user U, who is a viewer, the tagging target selected from among the shooting targets in the multi-view image. (Annotation target) is displayed with an annotation tag (step S5).

Next, the information providing apparatus 100 changes the tag of the annotation displayed on the terminal device 10 of the user U, who is the viewer, in accordance with the change of the viewpoint of the multi-view image via the network N (see FIG. 4). The position is changed (step S6). For example, when the viewpoint of the multi-view image is changed, the information providing apparatus 100 automatically follows the tagging target (annotation target) in the multi-view image, attaches the annotation tag, and attaches the tag to the appropriate position. indicate. At this time, the information providing apparatus 100 may attach an annotation tag to a tagging target (annotation target) for each image from a different viewpoint (each time) and display the tag at an appropriate position.

At this time, the information providing apparatus 100 arranges (displays) the tags of the annotations so as not to overlap with other objects or other tags. In addition, the information providing apparatus 100 continues to place (display) the tag at a position such that the positional relationship between the tag of the annotation and the target of tagging (annotation target) is maintained.

As described above, in the present embodiment, the information providing apparatus 100 receives a multi-view image (a multi-view image posted by a poster, a multi-view image specified by a viewer, etc.) from a user who is a poster or a viewer. Annotation tagging targets are identified from among shooting targets included in the acquired multi-viewpoint images, and the three-dimensional positions of the tagging targets within the multi-viewpoint images are estimated. Then, the information providing apparatus 100 attaches a tag to the tagging target according to the three-dimensional position of the tagging target. That is, the information providing apparatus 100 notifies (presents) the tag to the user by displaying the tag on the screen of the terminal device 10 together with the object to be tagged.

In addition, the information providing apparatus 100 identifies and classifies a shooting target for each viewpoint image of the multi-viewpoint image by image recognition or machine learning. The information providing apparatus 100 also receives a selection of a tagging target from a user who is a poster or a viewer. Then, the information providing apparatus 100 identifies the target of annotation tagging from among the shooting targets included in the multi-view image according to the selection of the target of tagging.

For example, the information providing apparatus 100 receives a selection of a tagging target and a specification of a web page associated with the tagging target from a user who is a poster or a viewer. Alternatively, the information providing apparatus 100 searches a plurality of web pages on the network for images similar to the image to be tagged by image recognition or machine learning, and associates the web page containing the similar image with the tag target. automatically identified as

When the information providing apparatus 100 attaches a tag to an object to be tagged, the tag is attached so as not to overlap with other objects and other tags. Further, when the information providing apparatus 100 attaches a tag to an object to be tagged, the information providing apparatus 100 attaches the tag to the object to be tagged if the object to be tagged is not hidden by other objects. Note that when the information providing apparatus 100 attaches a tag to a target to be tagged, even if the target to be tagged is hidden by another target, the tag to be tagged is the tag of the other target. If the display priority is higher than the target, the tag is assigned to the tagging target without assigning the tag to the other target.

Note that each of the above processes may be performed by the terminal device 10, not by the information providing device 100, by a function such as an application. That is, the processing may be completed on the terminal device 10 .

[1-2. Change display position of tags]
In this embodiment, the information providing apparatus 100 changes the display position (display mode) of the content (annotation tag) indicating the shooting target in accordance with the positional relationship of the shooting target shot in the multi-view image. Note that there may be a plurality of contents (annotation tags) indicating the shooting target.

The information providing apparatus 100 changes the display position of the tag according to the positional relationship between the tag of the annotation and the target of tagging (annotation target).

The information providing apparatus 100 hides the tag attached to the annotation target when the annotation target is hidden behind the photographed model/user or another photographed target.

For example, when the imaging range/display range of the annotation target is equal to or less than a predetermined value, or when the annotation target is hidden behind the user or another target, the information providing apparatus 100 displays the tag of the annotation. do not.

Alternatively, the information providing apparatus 100 may change the anteroposterior relationship of tags between an annotation target and another target. When there are a plurality of tags, the information providing apparatus 100 normally displays the target tag arranged closest to the front of the screen unless otherwise specified (when nothing is set). In this embodiment, the information providing apparatus 100 does not display the tag of the target placed closest to the screen when an object having a higher priority than the target placed closest to the screen is hidden behind the target. , to display the target tag with its higher priority.

At this time, the information providing apparatus 100 may set a priority for each target, and determine the tags to be displayed according to the set priority. In addition, the information providing apparatus 100 assigns an object that is presumed to be of higher interest to the user than the object placed closest to the screen as an object with a higher priority than the object placed closest to the screen. may decide.

As described above, in the present embodiment, the information providing apparatus 100 identifies an object to be tagged with an annotation from among shooting objects included in a multi-view image, and attaches a tag according to the position of the object to be tagged. . After that, when the position of the target of tagging in the screen changes due to the change of the viewpoint of the multi-view image, the information providing apparatus 100 changes the display position of the tag according to the change of the position of the target of tagging. .

Further, the information providing apparatus 100 arranges the tags at positions such that the positional relationship between the tagging target and the tags is maintained even if the viewpoint of the multi-viewpoint image is changed.

Note that each of the above processes may be performed by the terminal device 10, not by the information providing device 100, by a function such as an application. That is, the processing may be completed on the terminal device 10 .

[1-3. Shooting guide display]
In the present embodiment, the information providing apparatus 100 captures an image of itself using a terminal device 10 (imaging device) having a camera (in-camera, etc.) in order for the user to post a multi-view image (in-camera shooting). When shooting, a predetermined shooting guide is displayed on the screen (photo shooting screen) of the user's terminal device 10 (or the display device mounted/connected to it) used for shooting, and each time an image is shot (the viewpoint is moved each time) to change the shooting guide. FIG. 2 is an explanatory diagram showing an overview of the display of the shooting guide. In FIG. 2, in the image, M denotes a person to be photographed (photographed model), and B denotes a bag to be photographed.

As shown in FIG. 2, the information providing apparatus 100 confirms that the terminal device 10 having a camera has started shooting multi-view images in order for the user to post the multi-view images (step S11). For example, the information providing device 100 receives a signal or data indicating the start of multi-view image shooting from the terminal device 10 of the user via the network N (see FIG. 4).

Next, the information providing device 100 cooperates with the user's terminal device 10 via the network N (see FIG. 4) (step S12). For example, the information providing device 100 may control the photography application of the terminal device 10 via an API. That is, the subsequent processing may be performed by the information providing device 100 in cooperation with the terminal device 10 .

Next, the terminal device 10 displays a shooting guide on the screen of the terminal device 10 when the user takes an image of himself/herself (in-camera shooting) (step S13).

Next, when the user's pose, bag, etc. carried by the user deviate from the photographing guide, the terminal device 10 notifies the location of the deviation from the photographing guide (step S14). Note that the notification method may be screen display or voice guidance.

Next, the terminal device 10 automatically takes a picture when the pose of the user, the bag the user carries, etc. match the picture-taking guide (step S15). Even if the pose of the user, the bag the user is carrying, etc. do not completely match the shooting guide, the terminal device 10 automatically determines that they match if they match at least a predetermined ratio. You can shoot at In addition, the terminal device 10 may automatically shoot when the pose of the user, the bag the user carries, etc. match the shooting guide. At this time, the information providing device 100 instructs the terminal device 10 in advance to automatically take a picture when the pose of the user, the bag carried by the user, etc. match the shooting guide. good too.

Next, the terminal device 10 identifies and notifies a photographing target (annotation target candidate) included in the photographed image (step S16). Details will be described later.

Next, the terminal device 10 changes the shooting guide according to the viewpoint change for generating the multi-viewpoint image (step S17). That is, the terminal device 10 displays the shooting guide corresponding to the changed viewpoint.

Next, the terminal device 10 posts the multi-viewpoint image after completing the shooting of all the viewpoint images necessary for generating the multi-viewpoint image (step S18). For example, the terminal device 10 generates a multi-view image from captured images and automatically posts the multi-view image to the information providing device 100 via the network N (see FIG. 4). Alternatively, the terminal device 10 may post the captured image as it is to the information providing device 100 via the network N (see FIG. 4). In this case, the information providing apparatus 100 may generate a multi-view image from the posted image after acquiring the posted image.

In the present embodiment, the terminal device 10 of the user used to capture the multi-view image displays, on the screen of the terminal device 10, a facial expression guide, a posture (pose) guide, a product holding guide, or a shooting guide. A guide for how to wear the device is displayed, and the photographing guide is changed each time an image is photographed at a different position or angle.

For example, the terminal device 10 displays, on the screen of the user's terminal device 10 used for photographing, as a photographing guide, facial expressions, postures (poses) that the user should take, and silhouettes and outlines of how to hold or wear the product. However, the silhouette or outline may be changed each time an image is taken at a different position or angle. In this case, the user adjusts the facial expression, posture (pose), how to hold the product, or how to wear the product displayed on the screen to match the silhouette or outline of the user. Or change the way you wear them.

In addition, the terminal device 10 may provide voice guidance (guidance) in accordance with the display/change of the shooting guide. In addition, the terminal device 10 displays a comment regarding the content of the change on the photographing screen at a place/part where the user needs to change the facial expression, posture (pose), how to hold the product, how to wear the product, or the like. You can add tags.

(For pose guide)
Here, a case of a pose guide will be described as an example of the shooting guide. A terminal device 10 of a user having a camera displays a pose guide on the screen of the terminal device 10 when capturing multi-view images. The terminal device 10 changes the shooting guide each time an image is shot (every time the viewpoint is moved). That is, the pose guide changes one after another each time the photograph is taken.

Since the camera of the terminal device 10 is fixed, the user changes the position and angle of the camera by moving/rotating the terminal device 10 . The shooting guide has contents related to poses, facial expressions, how to hold the camera, and the like. The pose includes face orientation, body orientation, and the like. The pose guide gradually (gradually) changes according to the position and angle of the camera. That is, the terminal device 10 changes the pose guide step by step according to the position and angle of the camera.

When the terminal device 10 determines that the pose guide matches the user's pose on the screen, the terminal device 10 automatically takes a picture. At this time, the terminal device 10 determines the suitability of the user's pose. When the terminal device 10 determines that the user's pose is not appropriate, the terminal device 10 gives specific advice to make the user's pose appropriate, and guides the user to an appropriate pose.

For example, the terminal device 10 can pull back the chin, open/close the legs, tilt the body, arch the back, shift XX steps/XX cm in a specific direction (back and forth, left and right), what kind of bags and pouches, etc. A pose guide such as Kaze no Tsuka is displayed, or voice guidance is provided.

The terminal device 10 superimposes the user and the pose guide on the screen when shooting with the in-camera. A pose guide may be, for example, a silhouette, an outline, or a translucent display. Also, it is possible to say "Please do this" by voice. The terminal device 10 identifies the difference between the current pose of the user and the pose guide, and identifies the part that is different from the pose, such as "Please raise your right hand more" or "Please raise your leg a little more". may be displayed/guided. Then, when the terminal device 10 determines that the pose of the user matches the pose guide, the terminal device 10 may automatically shoot the image.

Also, the pose guide may be a facial expression guide. For example, it may be ``If you make one turn, you will smile''. That is, the facial expression does not have to be the same each time the photograph is taken. Also, the terminal device 10 may specify a pose in the first image and display the specified pose guide.

As described above, in the present embodiment, the terminal device 10 displays a predetermined shooting guide on the screen when shooting a multi-view image. For example, the terminal device 10 displays on the screen a guide regarding at least one of the pose, facial expression, and manner of holding of the subject (photographed model) as the photographing guide. In addition, the terminal device 10 displays the silhouette, outline, or semi-transparent display of the subject on the screen as a shooting guide.

In addition, the terminal device 10 superimposes a shooting guide on the subject when shooting the subject. In addition, the terminal device 10 identifies the pose of the person to be photographed among the subjects in the first photographed image, and selects and displays a photographing guide corresponding to the identified pose.

In addition, the terminal device 10 changes the shooting guide displayed on the screen according to the viewpoint every time the viewpoint during shooting is changed. In addition, the terminal device 10 may change the shooting guide step by step as the viewpoint during shooting is moved.

In addition, the terminal device 10 instructs the terminal device 10 to automatically shoot when the subject matches the shooting guide. In addition, when the subject is out of alignment with the shooting guide, the terminal device 10 notifies (presents) the location where the subject is out of alignment with the shooting guide to the photographer (user U). Note that the subject may be the photographer himself/herself. That is, the photographer and the person to be photographed may be the same person.

Note that each of the above processes may be performed by the information providing apparatus 100 via an API instead of by the terminal apparatus 10 .

[1-4. Recognition of Candidates for Annotation]
In the present embodiment, the terminal device 10 specifies a shooting target (annotation target candidate) included in the shot image when shooting an image constituting a multi-view image, and notifies the user of the specified shooting target. do.

When registering a multi-view image, the terminal device 10 recognizes a shooting target included in the image by image recognition or machine learning each time shooting is performed, and notifies the user of information about the shooting target by screen display or voice. A plurality of subjects may be photographed. Identifying and notifying what to shoot makes tagging easier later. In addition, the trouble of tagging in advance is eliminated. Note that the terminal device 10 may notify the user of the tagging target (annotation target) in advance, and notify the user whether or not the tagging target (annotation target) has been captured each time an image is captured. good.

In addition, the terminal device 10 may notify the user of "a dress was taken", "how many cuts were taken in the background", etc., by screen display or voice. Also, each time the user takes a picture, the user may be notified of how many more pictures are to be taken by means of screen display or voice, such as "There are XX pictures left." That is, the terminal device 10 may notify the user of the number of shots (the number of shots) of the shooting target and the remaining number of images constituting the multi-view image before the required number every time shooting is performed. In addition, the terminal device 10 may notify the user of the position (position) and angle (angle) of the camera that has completed (or has not completed) shooting by screen display or voice. In addition, the terminal device 10 may notify the user by screen display or voice if there are any viewpoints (images) that have not been captured for the images of each viewpoint necessary for generating a multi-view image. At this time, the terminal device 10 may notify by attaching and displaying a tag indicating the content of notification.

In addition, the terminal device 10 may notify a shooting target that has become invisible (hidden behind the user or another target) during shooting due to a change in the position or angle of the camera. good. At this time, the terminal device 10 attaches and displays a tag indicating that the object to be photographed is hidden behind another object, so that the object becomes invisible during the photographing (hidden behind the user or another object). d) You may notify the shooting target.

Note that each of the above processes may be performed by the information providing apparatus 100 via an API instead of by the terminal apparatus 10 .

[1-5. Replacing face image of shooting model]
In this embodiment, the information providing apparatus 100 replaces the face of the subject (photographed model) of the multi-view image with the face of the user U who is the viewer. That is, the information providing apparatus 100 replaces the face of the subject (photographed model) of the multi-view image with the face of another person. FIG. 3 is an explanatory diagram showing an overview of replacement of a face image of a person (photographed model). In FIG. 3, in the image, M1 is the person to be photographed (photographing model) before replacement of the face image, B is the bag photographed with the person to be photographed, and B is the person to be photographed after replacing the face image with the face image of user U. is denoted as M+U.

As shown in FIG. 3, the information providing apparatus 100 acquires a multi-viewpoint face image of the face of the user U from the terminal device 10 of the user U (user) via the network N (see FIG. 4) (see FIG. 4). step S21). For example, the information providing apparatus 100 acquires, from the terminal device 10 of the user U, multi-viewpoint face images of the face of the user U browsing the posted multi-viewpoint images taken from a plurality of viewpoints.

Next, the information providing apparatus 100 identifies the face of the person to be photographed (photographed model) from the multi-viewpoint image to be browsed by the user U (step S22). For example, the information providing apparatus 100 identifies the face of the person (photographed model) for each viewpoint of the multi-view image. In the present embodiment, the multi-view image to be browsed is a multi-view image including the face of a person to be photographed (photographed model) different from the user U (user). In fact, it may be a multi-viewpoint image including the face of the user U (user) himself.

Next, the information providing apparatus 100 identifies the viewpoint at the time of shooting from the multi-viewpoint image to be browsed (step S23). For example, the information providing apparatus 100 identifies the position and angle of the face of the person (photographed model) for each viewpoint of the multi-view image.

Next, the information providing apparatus 100 changes the face of the person being photographed in the multi-viewpoint image to be browsed to the face of the user U (user) who is the browsing person, according to the viewpoint at the time of photographing (step S24). ). For example, the information providing apparatus 100 displays the face of the person (photographed model) according to the position and angle of the face of the person (photographed model) for each viewpoint of the multi-view image. Generate a face-replaced image. At this time, the information providing apparatus 100 makes the face of the person to be photographed in the multi-view image to be browsed look like the face of the user U (user) who is the browsing person, in a form that is as natural as possible (no sense of incongruity as much as possible). ) replace. In addition, the information providing apparatus 100 collectively displays the face of each person to be photographed (photographed model) in accordance with the viewpoint at the time of photographing each of a plurality of multi-viewpoint images that are viewed at the same time. Convert to face.

Next, when changing the face of the subject (photographed model) of the multi-view image to the face of the viewer U (user), the information providing apparatus 100 changes the face of the multi-view image as needed. The height of the person to be photographed (photographed model) is adjusted (step S25). That is, the information providing apparatus 100 may change not only the face of the person to be photographed (photographed model) but also the height thereof according to the user. For example, the information providing apparatus 100 may change the height of the subject (photographed model) of the multi-view image to the height of the user. Further, the information providing apparatus 100 may adjust the height of the person to be photographed based on the face of the person (photographed model) in the multi-view image and the face of the user. In addition, the information providing apparatus 100 may adjust the height of the person to be photographed according to the size of the object photographed together with the background and the person to be photographed.

Further, when the face of the person (photographed model) of the multi-view image is changed to the face of the user, or when the person (photographed model) of the multi-view image is the user himself/herself, the information providing apparatus 100 The user's hairstyle, hair color (including shading), facial expression, etc. in the image may be changed by image processing and editing. For example, long hair may be changed to short hair, black hair may be changed to brown, and the corners of the eyes and the corners of the mouth may be raised and lowered. Alternatively, the information providing apparatus 100 changes the current face image (such as the user's face image after conversion) displayed in the multi-view image to a hairstyle or hair color (including shading) according to a user's instruction or the like. , may be converted into a face image with a different facial expression or the like.

Next, the information providing device 100 displays the converted multi-view image on the terminal device 10 of the user U (user) via the network N (see FIG. 4) (step S26).

In the above description, the information providing apparatus 100 converts the face of the person being photographed in the multi-view image into the face of the user U who is the viewer. Not limited to faces. The information providing apparatus 100 may convert the face of an arbitrary person other than the face of the user U who is the viewer. For example, the information providing apparatus 100 may set the face of a friend of the user U as a gift recipient when purchasing a product corresponding to a tagging target (annotation target) among shooting targets included in a multi-view image. .

Also, in the above description, the information providing apparatus 100 converts the face of the subject of the multi-view image into the face of another person, but in reality, the face is not limited to the face of another person. The information providing apparatus 100 may convert to another face of the same person instead of the face of another person. For example, when the subject of a certain multi-view image is the user U who is the viewer (when the subject of the multi-view image and the viewer are the same person, etc.), the information providing apparatus 100 The face of the user U who is the subject of the viewpoint image may be converted into another face of the user U.

In addition, the information providing apparatus 100 may use techniques such as deepfake to transform the face of the person being photographed in the multi-viewpoint image to be browsed. In addition, the information providing apparatus 100 generates a multi-viewpoint image of the face to be replaced by using a known technique for generating multiple-viewpoint images from a single-viewpoint face image, and uses this to convert the face of the image. You may

Note that each of the above processes may be performed by the terminal device 10, not by the information providing device 100, by a function such as an application. That is, the processing may be completed on the terminal device 10 .

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 4 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 4, the information processing system 1 according to the embodiment includes a terminal device 10 and an information providing device 100. As shown in FIG. These various devices are communicatively connected via a network N by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Also, the number of devices included in the information processing system 1 shown in FIG. 4 is not limited to the illustrated one. For example, in FIG. 4, only one terminal device 10 is shown for simplification of illustration, but this is only an example and is not limited, and two or more devices may be provided.

The terminal device 10 is an information processing device used by the user U. FIG. For example, the terminal device 10 includes smart devices such as smartphones and tablet terminals, feature phones, PCs (Personal Computers), PDAs (Personal Digital Assistants), game machines and AV equipment with communication functions, car navigation systems, smart watches, Wearable devices such as head-mounted displays, smart glasses, and the like.

In addition, the terminal device 10 is compatible with wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local It is possible to communicate with the information providing apparatus 100 by connecting to the network N via short-range wireless communication such as Area Network).

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, or the like. Note that the information providing apparatus 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be described using FIG. FIG. 5 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 5, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, an imaging unit 15, a sensor unit 20, a control unit 30 (controller), and a storage unit 40 .

(Communication unit 11)
The communication unit 11 is connected to the network N (see FIG. 4) by wire or wirelessly, and transmits and receives information to and from the information providing apparatus 100 via the network N. FIG. For example, the communication unit 11 is implemented by a NIC (Network Interface Card), an antenna, or the like.

(Display unit 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic EL display (Organic Electro-Luminescent Display). Also, the display unit 12 is a touch panel display, but is not limited to this.

(Input unit 13)
The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons and the like for inputting characters, numbers, and the like. The input unit 13 may be an input/output port (I/O port), a USB (Universal Serial Bus) port, or the like. Moreover, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13 . Also, the input unit 13 may be a microphone or the like that receives voice input from the user U. FIG. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites, and based on the received signals, position information (for example, latitude and longitude). That is, the positioning unit 14 positions the position of the terminal device 10 . GPS is merely an example of GNSS (Global Navigation Satellite System).

Also, the positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position using various communication functions of the terminal device 10 as described below as auxiliary positioning means for position correction and the like.

(Imaging unit 15)
The imaging unit 15 is an image sensor (camera) that photographs a subject. For example, the imaging unit 15 is a CMOS image sensor, a CCD image sensor, or the like. Note that the imaging unit 15 is not limited to the built-in camera, and may be a wireless camera that can communicate with the terminal device 10 or an external camera such as a web camera.

(Wi-Fi positioning)
For example, the positioning unit 14 measures the position of the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the position of the terminal device 10 by measuring the distance to a nearby base station or access point.

(beacon positioning)
The positioning unit 14 may also use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 positions the position of the terminal device 10 by connecting with a beacon transmitter connected by the Bluetooth (registered trademark) function.

(geomagnetic positioning)
Further, the positioning unit 14 positions the position of the terminal device 10 based on the geomagnetism pattern of the structure measured in advance and the geomagnetic sensor provided in the terminal device 10 .

(RFID positioning)
Further, for example, if the terminal device 10 has an RFID (Radio Frequency Identification) tag function equivalent to a contactless IC card used at station ticket gates, shops, etc., or has a function of reading an RFID tag In this case, the location used is recorded together with the information that the payment was made by the terminal device 10 . The positioning unit 14 may measure the position of the terminal device 10 by acquiring such information. Also, the position may be measured by an optical sensor provided in the terminal device 10, an infrared sensor, or the like.

The positioning unit 14 may measure the position of the terminal device 10 using one or a combination of the positioning means described above, if necessary.

(Sensor unit 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10 . The connection may be wired connection or wireless connection. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices. In the example shown in FIG. 5, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, and a magnetic sensor 27.

The sensors 21 to 27 described above are only examples and are not limited. That is, the sensor unit 20 may be configured to include a part of the sensors 21 to 27, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 27. . The imaging unit 15 is also a kind of image sensor.

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10 such as movement direction, speed, and acceleration of the terminal device 10 . The gyro sensor 22 detects physical movements of the terminal device 10 such as inclination in three axial directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 includes the above-described acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc., techniques such as pedestrian dead-reckoning (PDR: Pedestrian Dead-Reckoning) using these sensors 21 to 23, etc. , the position of the terminal device 10 can be determined. This makes it possible to acquire indoor position information that is difficult to acquire with a positioning system such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. Further, by using the gyro sensor 22, it is possible to know the traveling direction, the direction of the line of sight, and the inclination of the body of the user U. Also, from the atmospheric pressure detected by the atmospheric pressure sensor 23, the altitude at which the terminal device 10 of the user U is present and the number of floors can be known.

The temperature sensor 24 detects the temperature around the terminal device 10, for example. The sound sensor 25 detects sounds around the terminal device 10, for example. The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, geomagnetism around the terminal device 10 . The imaging unit 15 captures an image around the terminal device 10 .

The atmospheric pressure sensor 23, the temperature sensor 24, the sound sensor 25, the optical sensor 26, and the imaging unit 15 described above detect the atmospheric pressure, the temperature, the sound, and the illuminance, respectively, or capture an image of the surroundings, so that the terminal device 10 It is possible to detect the surrounding environment and situations. In addition, it is possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and situation of the terminal device 10 .

(control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, and various circuits. Also, the control unit 30 may be configured by hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31 , a reception unit 32 , a processing unit 33 , a guide display unit 34 , a guide change unit 35 , a shooting determination unit 36 , a recognition unit 37 and a notification unit 38 . Note that the processing unit 33 may actually include a guide display unit 34 , a guide change unit 35 , a shooting determination unit 36 , a recognition unit 37 and a notification unit 38 .

(Sending unit 31)
The transmission unit 31 receives, for example, various information input by the user U using the input unit 13, various information detected by the sensors 21 to 27 mounted on or connected to the terminal device 10, and information measured by the positioning unit 14. The position information of the terminal device 10 and the like can be transmitted to the information providing device 100 via the communication unit 11 .

(Receiver 32)
The receiving unit 32 can receive various information provided by the information providing apparatus 100 and requests for various information from the information providing apparatus 100 via the communication unit 11 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10 including the display unit 12 and the like. For example, the processing unit 33 can output various types of information transmitted by the transmitting unit 31 and various types of information received by the receiving unit 32 from the information providing apparatus 100 to the display unit 12 for display.

(Guide display section 34)
The guide display unit 34 displays a predetermined shooting guide on the screen when shooting a multi-viewpoint image. For example, the guide display unit 34 displays on the screen a guide regarding at least one of the pose, facial expression, and manner of holding of the person to be photographed (photographing model) as a photographing guide. Further, the guide display unit 34 displays the silhouette, outline, or semi-transparent display of the subject on the screen as a shooting guide.

Further, the guide display unit 34 superimposes and displays a shooting guide on the subject when shooting the subject. In addition, the guide display unit 34 identifies the pose of the subject among the subjects in the first captured image, and selects and displays a photographing guide corresponding to the identified pose.

(Guide changing unit 35)
A guide changing unit 35 changes the shooting guide displayed on the screen according to the viewpoint every time the viewpoint during shooting is moved. Further, the guide changing unit 35 may change the shooting guide step by step as the viewpoint during shooting is moved.

(Photographing determination unit 36)
The shooting determination unit 36 automatically shoots the subject using the imaging unit 15 when the subject matches the shooting guide. Further, when the shooting of the multi-viewpoint image is completed, the shooting determination unit 36 uses the transmission unit 31 to post the multi-viewpoint image to the information providing apparatus 100 . At this time, the photographing determination unit 36 may generate a multi-viewpoint image from the photographed images.

(Recognition unit 37)
The recognition unit 37 recognizes a shooting target included in an image when shooting an image forming a multi-view image. For example, the recognition unit 37 recognizes the shooting target included in the image by image recognition or machine learning. Further, the recognition unit 37 recognizes the photographing target included in the image each time photographing is performed. Further, the recognition unit 37 recognizes each of the plurality of shooting targets included in the image each time shooting is performed. In addition, the recognition unit 37 recognizes an object to be photographed that is hidden behind other objects to be photographed and is no longer visible each time photographing is performed.

(Notification unit 38)
When the subject is out of alignment with the shooting guide, the notification unit 38 notifies the photographer of the location where the subject is out of alignment with the shooting guide. Note that the subject may be the photographer himself/herself. That is, the photographer and the person to be photographed may be the same person.

Also, the notification unit 38 notifies the user of the imaging target recognized by the recognition unit 37 . For example, the notification unit 38 notifies the user of the recognized shooting target each time shooting is performed. In addition, the notification unit 38 notifies the user of each of the plurality of recognized shooting targets each time shooting is performed. In addition, the notification unit 38 notifies the user of the number of images of the recognized object to be photographed each time photographing is performed. In addition, the notification unit 38 notifies the user of the remaining number of images constituting the multi-view image before the required number each time the image is captured. In addition, the notification unit 38 notifies the user of an object to be photographed that is hidden behind another object to be photographed and is no longer visible each time photographing is performed.

At this time, the notification unit 38 may notify the user of the specified imaging target by voice. In addition, the notification unit 38 may notify the user of the specified imaging target by displaying it on the screen.

(storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disk. be. Various programs, various data, and the like are stored in the storage unit 40 .

[4. Configuration example of information providing device]
Next, the configuration of the information providing device 100 according to the embodiment will be described using FIG. FIG. 6 is a diagram showing a configuration example of the information providing device 100 according to the embodiment. As shown in FIG. 6, the information providing device 100 has a communication section 110, a storage section 120, and a control section .

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Also, the communication unit 110 is connected to the network N (see FIG. 4) by wire or wirelessly.

(storage unit 120)
The storage unit 120 is implemented by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as an HDD, SSD, or optical disk. As shown in FIG. 6, storage unit 120 has user information database 121 , history information database 122 , and image information database 123 .

(User information database 121)
The user information database 121 stores user information about the user U. FIG. For example, the user information database 121 stores various information such as user U attributes. FIG. 7 is a diagram showing an example of the user information database 121. As shown in FIG. In the example shown in FIG. 7, the user information database 121 has items such as "user ID (Identifier)", "age", "sex", "home", "place of work", and "interest".

“User ID” indicates identification information for identifying the user U. The “user ID” may be the user U's contact information (telephone number, e-mail address, etc.), or may be identification information for identifying the user U's terminal device 10 .

"Age" indicates the age of the user U identified by the user ID. Note that the "age" may be information indicating a specific age of the user U (for example, 35 years old) or information indicating the age of the user U (for example, 30's). . Alternatively, the "age" may be information indicating the date of birth of the user U, or information indicating the generation of the user U (for example, born in the 80's). "Gender" indicates the gender of the user U identified by the user ID.

"Home" indicates location information of the home of the user U identified by the user ID. In the example shown in FIG. 7, "home" is represented by an abstract code such as "LC11", but may be latitude/longitude information or the like. Also, for example, "home" may be an area name or an address.

"Place of work" indicates location information of the place of work (school in the case of a student) of the user U identified by the user ID. In the example shown in FIG. 7, the "place of work" is illustrated as an abstract code such as "LC12", but may be latitude/longitude information or the like. Also, for example, the "place of work" may be an area name or an address.

"Interest" indicates the interest of the user U identified by the user ID. That is, "interest" indicates an object in which the user U identified by the user ID is highly interested. For example, the "interest" may be a search query (keyword) that the user U has entered into a search engine and searched for. In the example shown in FIG. 7, one "interest" is shown for each user U, but there may be more than one.

For example, in the example shown in FIG. 7, the age of the user U identified by the user ID "U1" is "twenties" and the gender is "male". Also, for example, the user U identified by the user ID "U1" indicates that the home is "LC11". Also, for example, the user U identified by the user ID "U1" indicates that the place of work is "LC12". Also, for example, the user U identified by the user ID "U1" indicates that he is interested in "sports".

Here, in the example shown in FIG. 7, abstract values such as “U1”, “LC11” and “LC12” are used, but “U1”, “LC11” and “LC12” are concrete values. It is assumed that information such as character strings and numerical values is stored. Hereinafter, abstract values may also be illustrated in diagrams relating to other information.

The user information database 121 is not limited to the above, and may store various types of information depending on the purpose. For example, the user information database 121 may store various information about the terminal device 10 of the user U. FIG. In addition, the user information database 121 stores user U's demographics (demographic attributes), psychographics (psychological attributes), geographics (geographical attributes), behavioral attributes (behavioral attributes), etc. Information about attributes may be stored. For example, the user information database 121 includes name, family structure, hometown (local), occupation, position, income, qualification, residence type (detached house, condominium, etc.), presence or absence of car, commuting time, commuting time, commuting time. Information such as routes, commuter pass sections (stations, lines, etc.), frequently used stations (other than the nearest station to your home or place of work), lessons (places, time zones, etc.), hobbies, interests, lifestyle, etc. may

(History information database 122)
The history information database 122 stores various types of information related to history information (log data) indicating user U's actions. FIG. 8 is a diagram showing an example of the history information database 122. As shown in FIG. In the example shown in FIG. 8, the history information database 122 has items such as "user ID", "location history", "search history", "browsing history", "purchase history", and "posting history".

“User ID” indicates identification information for identifying the user U. "Position history" indicates a position history that is a history of the user's U position and movement. Also, "search history" indicates a search history that is a history of search queries input by the user U. FIG. "Browsing history" indicates a browsing history that is a history of contents browsed by the user U. FIG. "Purchase history" indicates the purchase history of the user U's purchases. In addition, “posting history” indicates a posting history that is a history of posts by the user U. FIG. In addition, the “posting history” may include questions about user U's property.

For example, in the example shown in FIG. 8, the user U identified by the user ID “U1” moves along the “location history #1”, searches along the “search history #1”, It indicates that the content was browsed according to the "history #1", a predetermined product or the like was purchased at a predetermined store or the like according to the "purchase history #1", and the content was posted according to the "posting history".

Here, in the example shown in FIG. 8, abstract history such as "U1", "location history #1", "search history #1", "browsing history #1", "purchase history #1" and "posting history #1" "U1", "location history #1", "search history #1", "browsing history #1", "purchase history #1" and "posting history #1" , information such as specific character strings and numerical values are stored.

Note that the history information database 122 may store various types of information, not limited to the above, depending on the purpose. For example, the history information database 122 may store the user U's usage history of a predetermined service. In addition, the history information database 122 may store the user U's store visit history, facility visit history, and the like. In addition, the history information database 122 may store a history of payment (electronic payment) using the terminal device 10 of the user U, and the like.

(Image information database 123)
The image information database 123 stores various information regarding multi-view images. FIG. 9 is a diagram showing an example of the image information database 123. As shown in FIG. In the example shown in FIG. 9, the image information database 123 includes "multi-viewpoint image", "image", "viewpoint", "shooting target", "position", "annotation target", "tag", "face position", and "image". ” has items such as

“Multi-view image” indicates identification information for identifying a multi-view image. It should be noted that, in practice, it may be the storage location or the location of the data of the multi-view image. Also, "image" indicates identification information for identifying an image forming a multi-view image. It should be noted that, in practice, it may be the storage location or location of the data of the images forming the multi-viewpoint image.

Also, "viewpoint" indicates a viewpoint when an image forming a multi-viewpoint image is captured. In other words, the viewpoint indicates the position and angle of the object to be photographed that is included in the images forming the multi-viewpoint image.

In addition, "shooting target" indicates a shooting target included in the images forming the multi-viewpoint image. That is, it indicates an object to be photographed as a subject. For example, it indicates the classification (category) of the object to be photographed, the specific product name, the product code, and the like. Also, the object to be photographed may be a person (person) to be photographed. Also, the number of subjects to be photographed may be plural. That is, one image may include a plurality of shooting targets. For example, a person to be photographed and two fashion items worn by the person to be photographed may be photographed.

"Position" indicates the position of the imaging target within the multi-viewpoint image (position within the image). In this embodiment, the three-dimensional position of the object to be photographed in each image that constitutes the multi-viewpoint image is indicated. The three-dimensional position of the object to be imaged may be an absolute position such as coordinates in an image, or may be a relative position from a reference point or another object to be imaged. Also, the three-dimensional position is merely an example.

"Annotation target" indicates a tagging target (annotation target) among shooting targets (annotation target candidates). The target of tagging may be selected by the user, or may be automatically determined by presetting, machine learning, or the like.

"Tag" indicates a tag given to a tagging target (annotation target). For example, it may be identification information for identifying a tag registered in advance, or the content of the tag. For example, it may be information about a product page of a product (fashion item) to be tagged (annotated). At this time, similar images of the product are searched for by image recognition or machine learning from each product page of e-commerce sites such as fashion mail order sites (for example, "ZOZOTOWN" (registered trademark)), and based on the search results, the product of the product Pages may be automatically identified.

"Position of face" indicates the position of the face of the photographed person in the multi-view image (the position of the face in the image). In this embodiment, the three-dimensional position of the face of the person to be photographed in each image that constitutes the multi-viewpoint image is indicated. The position of the face may be an absolute position such as coordinates in the image, or may be a relative position from a reference point or another object to be imaged. Also, the outline of the face and the position of each part of the face (eyebrows, eyes, ears, nose, mouth, chin, etc.) may be used.

For example, in the example shown in FIG. 9, an image "A1" that forms a multi-viewpoint image "A" is shot at "viewpoint #A1", and the subject "bag" is at "position #A1" in the image. Yes, it is selected as an "annotation target" (target for tagging), a link to the website "Site #W1" related to the target bag is given as a tag, and the position of the photographed person's face in the image is " face position #A1”.

Here, in the example shown in FIG. 9, abstract values such as "A", "A1", "viewpoint #A1", "position #A1", "site #W1", and "face position #A1" are used. As shown, "A", "A1", "viewpoint #A1", "position #A1", "site #W1" and "face position #A1" contain information such as specific character strings and numerical values. shall be memorized.

The image information database 123 is not limited to the above, and may store various kinds of information depending on the purpose. For example, the image information database 123 may store identification information for identifying contributors or viewers of multi-view images. In addition, the image information database 123 may store detailed information about a subject (photographed object, photographed person, etc.). The image information database 123 may also store a list of candidate tags. In addition, the image information database 123 may store information about the shooting location and shooting date and time. The image information database 123 may also store information about the imaging device (user's terminal device, etc.) used for imaging and the imaging environment.

(control unit 130)
Returning to FIG. 6, the description is continued. The control unit 130 is a controller, and for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like controls the information providing apparatus 100. Various programs (corresponding to an example of an information processing program) stored in the internal storage device are executed by using a storage area such as a RAM as a work area. In the example shown in FIG. 6, the control unit 130 includes an acquisition unit 131, an identification unit 132, an estimation unit 133, a tagging unit 134, a tag changing unit 135, an image conversion unit 136, and a providing unit 137. have.

(Acquisition unit 131)
The acquisition unit 131 acquires a search query input by the user U. For example, the acquisition unit 131 acquires the search query via the communication unit 110 when the user U inputs a search query into a search engine or the like and performs a keyword search. That is, the acquisition unit 131 acquires, via the communication unit 110, a keyword input by the user U to a search window of a search engine, site, or application.

The acquisition unit 131 also acquires user information about the user U via the communication unit 110 . For example, the acquisition unit 131 acquires identification information (user ID, etc.) indicating the user U, location information of the user U, attribute information of the user U, and the like, from the terminal device 10 of the user U. FIG. Further, the acquisition unit 131 may acquire identification information indicating the user U, attribute information of the user U, and the like when the user U is registered as a user. Acquisition unit 131 then registers the user information in user information database 121 of storage unit 120 .

In addition, the acquisition unit 131 acquires various types of history information (log data) indicating actions of the user U via the communication unit 110 . For example, the acquisition unit 131 acquires various types of history information indicating actions of the user U from the terminal device 10 of the user U or from various servers based on the user ID or the like. The acquisition unit 131 then registers various types of history information in the history information database 122 of the storage unit 120 .

Also, the acquisition unit 131 acquires a multi-view image from the user U, who is a poster or a viewer, via the communication unit 110 . For example, the acquiring unit 131 acquires a multi-view image captured by the poster from the terminal device 10 of the user U who is the poster. In addition, the acquisition unit 131 acquires multi-viewpoint images shot by other contributors specified by the user U who is the viewer.

Also, the acquiring unit 131 acquires a multi-viewpoint image of the face of a person other than the person being photographed via the communication unit 110 . For example, the acquiring unit 131 acquires a multi-viewpoint face image obtained by photographing another person's face from a plurality of viewpoints. The face of another person may be the face of the user U who is the viewer. In this embodiment, the acquisition unit 131 acquires a multi-viewpoint image of the viewer's face via the communication unit 110 . Note that the acquisition unit 131 may acquire the multi-viewpoint image of the viewer's face in advance, or may acquire it at the time of viewing. Also, the multi-viewpoint image of the viewer's face may be a multi-viewpoint image including at least the viewer's face. For example, the acquiring unit 131 acquires multi-viewpoint face images obtained by photographing the face of the user U, who is a viewer, from a plurality of viewpoints.

The acquisition unit 131 also functions as a reception unit that receives selection of a tagging target from a poster via the communication unit 110 . For example, the acquiring unit 131 (receiving unit) receives, from the contributor, selection of a tagging target and designation of a web page to be associated with the tagging target.

(Specifying unit 132)
The identifying unit 132 identifies a shooting target included in the multi-viewpoint image. Then, the specifying unit 132 specifies targets to be tagged with annotations from shooting targets included in the multi-viewpoint images. At this time, the identifying unit 132 identifies and classifies the shooting target for each viewpoint image of the multi-viewpoint image by image recognition or machine learning.

For example, the specifying unit 132 specifies the target of annotation tagging from among the shooting targets included in the multi-view image according to the selection of the target of tagging by the contributor. Alternatively, the identifying unit 132 identifies, by image recognition or machine learning, an annotation tagging target from shooting targets included in the multi-viewpoint image.

In addition, the identifying unit 132 searches a plurality of web pages on the network for images similar to the image to be tagged by image recognition or machine learning, and uses the web page containing the similar image as a web page to associate with the tagging target. Identify automatically.

Further, the specifying unit 132 specifies the face of the person to be photographed in the multi-viewpoint image to be browsed by the user U. For example, the specifying unit 132 specifies the face of the person to be photographed in the multi-viewpoint image to be browsed by the user U and the viewpoint at the time of shooting the multi-viewpoint image. At this time, the specifying unit 132 may specify the face of the person to be photographed in each of the multiple multi-view images viewed at the same time, and the viewpoint at the time of shooting each of the multiple multiple-view images.

(Estimation unit 133)
The estimation unit 133 estimates the position of the shooting target in the multi-view image. That is, the estimation unit 133 estimates the position of the tagging target selected from the shooting targets in the multi-view image. In this embodiment, the estimation unit 133 estimates a three-dimensional position in the multi-view image to be tagged. The three-dimensional position in the multi-viewpoint image may be an absolute position such as coordinates in the image, or may be a relative position from a reference point or another imaging target. Note that the identifying unit 132 may actually function as the estimating unit 133 . At this time, the specifying unit 132 specifies a three-dimensional position in the multi-view image to be tagged.

(Tagging unit 134)
The tag assigning unit 134 assigns a tag according to the position of the object to be tagged. For example, the tagging unit 134 tags the object to be tagged according to the three-dimensional position of the object to be tagged. As a result, the attached tag is displayed on the screen. In addition, when the tagging unit 134 gives a tag to a target to be tagged, the tag is given so as not to overlap with other targets and other tags.

Further, when tagging a target to be tagged, the tagging unit 134 tags the target to be tagged if the target to be tagged is not hidden by other targets. Note that when the tagging unit 134 gives a tag to a target to be tagged, even if the target to be tagged is hidden by another target, the tag to be tagged is the tag of the other target. If the priority of display is higher than the target, the target to be tagged may be tagged without assigning the tag to the other target.

(Tag changing unit 135)
The tag changing unit 135 changes the tag display position in accordance with the change of the tagging target position when the tagging target position in the screen is changed due to the change of the viewpoint of the multi-viewpoint image. Also, the tag changing unit 135 arranges the tag at a position such that the positional relationship between the tagging target and the tag is maintained even if the viewpoint of the multi-viewpoint image is changed. Note that the tagging unit 134 may actually function as the tag changing unit 135 . In this case, the tagging unit 134 tags the target to be tagged in accordance with the change in the position of the target to be tagged each time the viewpoint of the multi-viewpoint image is changed.

(Image converter 136)
The image conversion unit 136 converts the face of the photographed person in the multi-view image into the face of another person for each image forming the multi-view image. For example, the image conversion unit 136 converts the face of the subject in the multi-viewpoint image into the face of another person in accordance with the viewpoint at the time of shooting the multi-viewpoint image, for each image forming the multi-viewpoint image. At this time, the image conversion unit 136 may generate a new multi-view image by replacing the face of the subject in the multi-view image with the face of another person in accordance with the viewpoint at the time of shooting the multi-view image.

In this embodiment, the image conversion unit 136 converts the face of the subject of the multi-view image into the face of the user U who is the viewer, for each image forming the multi-view image. For example, the image conversion unit 136 converts the face of the person to be photographed in the multi-viewpoint image into the face of the user U for each image forming the multi-viewpoint image in accordance with the viewpoint at the time of shooting the multi-viewpoint image. At this time, the image conversion unit 136 may generate a new multi-view image by replacing the face of the person being photographed in the multi-view image with the face of the user U according to the viewpoint at the time of shooting the multi-view image. good.

In addition, the image conversion unit 136 adjusts the height of the person to be photographed in the multi-view image when converting the face of the person to be photographed in the multi-view image to the face of another person according to the viewpoint at the time of photographing the multi-view image. Adjust for another person's height. At this time, the image conversion unit 136 may adjust the height of the person to be photographed based on the face of the person being photographed and the face of another person.

In the present embodiment, the image conversion unit 136 converts the face of the person being photographed in the multi-viewpoint image into the face of the user U who is the viewer, in accordance with the viewpoint at the time of shooting the multi-viewpoint image. The height of the person to be photographed in the viewpoint image is adjusted according to the height of the user U. At this time, the image conversion unit 136 may adjust the height of the person to be photographed based on the face of the person to be photographed and the face of the user U.

In addition, the image conversion unit 136 converts the face of the subject in the multi-viewpoint image into the face of another person according to the viewpoint at the time of shooting the multi-viewpoint image, and also converts the expression of the face of the other person in the image after conversion. change. In addition, the image conversion unit 136 converts the face of the photographed person in the multi-viewpoint image into another person's face and changes the hairstyle of the other person in the image after conversion in accordance with the viewpoint at the time of photographing the multi-viewpoint image. . In addition, the image conversion unit 136 converts the face of the person to be photographed in the multi-viewpoint image into the face of another person in accordance with the viewpoint at the time of shooting the multi-viewpoint image, and the hair color of the other person in the image after conversion ( (including shading).

In this embodiment, the image conversion unit 136 converts the face of the person being photographed in the multi-view image into the face of the user U who is the viewer, according to the viewpoint at the time of shooting the multi-view image. The facial expression of the subsequent user U is changed. In addition, the image conversion unit 136 converts the face of the person to be photographed in the multi-viewpoint image into the face of the user U in accordance with the viewpoint at the time of shooting the multi-viewpoint image. to change In addition, the image conversion unit 136 converts the face of the person to be photographed in the multi-viewpoint image into the face of the user U in accordance with the viewpoint at the time of shooting the multi-viewpoint image. change the color of

For example, the image conversion unit 136 may change the hairstyle, hair color (including shading), facial expression, etc. of the user U in the image by processing and editing the image. Alternatively, the image conversion unit 136 converts the current face image displayed in the multi-viewpoint image (face image of the user U after conversion, etc.) into a hairstyle or hair color ( (including shading), facial expressions, etc. may be converted into face images.

In addition, the image conversion unit 136 collectively converts the faces of the respective persons to be photographed into the faces of different persons according to the respective viewpoints of the plurality of multi-viewpoint images viewed at the same time. In this embodiment, the image conversion unit 136 collectively converts the faces of the persons to be photographed according to the viewpoints at the time of photographing of the plurality of multi-viewpoint images that are viewed at the same time. Convert to face.

(Providing unit 137)
The providing unit 137 provides the user U with the multi-viewpoint image converted to the face of another person. For example, the providing unit 137 provides the user U with the generated new multi-view image. In addition, the providing unit 137 provides the user U with each of the plurality of multi-viewpoint images after conversion into the face of another person.

In the present embodiment, the providing unit 137 provides the user U, who is a viewer, with the multi-viewpoint image converted into the user U's face. For example, the providing unit 137 provides the user U with the generated new multi-view image. Further, the providing unit 137 provides the user U with each of the plurality of multi-viewpoint images converted into the user U's face.

[5. Processing procedure]
Next, processing procedures by the terminal device 10 and the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 10 is a flowchart illustrating a processing procedure according to the embodiment; The processing procedure described below is repeatedly executed by the control unit 30 of the terminal device 10 and the control unit 130 of the information providing device 100 . In addition, the terminal device 10 and the information providing device 100 cooperate with each other.

As shown in FIG. 10, the guide display unit 34 of the terminal device 10 displays a predetermined shooting guide on the screen when shooting a multi-viewpoint image (step S101).

Subsequently, when the subject matches the shooting guide, the shooting determination unit 36 of the terminal device 10 automatically shoots the subject using the imaging unit 15 (step S102). At this time, if the subject is out of alignment with the shooting guide, the notification unit 38 of the terminal device 10 notifies the photographer of the location where the subject is out of alignment with the shooting guide. At this time, the notification unit 38 may notify the user of the specified imaging target by voice, or may notify the user of the specified imaging target by screen display.

Subsequently, the recognition unit 37 of the terminal device 10 recognizes the photographing target included in the photographed image. The notification unit 38 of the terminal device 10 notifies the user of the imaging target recognized by the recognition unit 37 (step S103). For example, each time the notification unit 38 takes a photograph, the number of images of the recognized object to be photographed, the remaining number of images constituting the multi-view image to the required number, and the number of images to be photographed that are hidden behind other objects to be photographed. Notify the user of the target etc.

Subsequently, the guide changing unit 35 of the terminal device 10 changes the shooting guide displayed on the screen according to the viewpoint every time the viewpoint during shooting is changed (step S104). At this time, the guide changing unit 35 may change the shooting guide step by step as the viewpoint during shooting is moved.

Subsequently, when the shooting of the multi-view image is completed, the shooting determination unit 36 of the terminal device 10 uses the transmission unit 31 to post the multi-view image to the information providing device 100 (step S105). At this time, the photographing determination unit 36 may generate a multi-viewpoint image from the photographed images. Further, the photographing determination unit 36 may confirm with the photographer whether or not it is permissible to post before posting.

Subsequently, the estimation unit 133 of the information providing device 100 estimates the position of the imaging target in the multi-view image (step S106). For example, the acquisition unit 131 of the information providing device 100 acquires multi-view images from the terminal device 10 via the communication unit 110 . The specifying unit 132 of the information providing device 100 specifies a shooting target in the multi-view image. Then, the estimation unit 133 of the information providing apparatus 100 estimates the three-dimensional position of the imaging target in the multi-view image.

Subsequently, the specifying unit 132 of the information providing apparatus 100 specifies targets to be tagged with annotations from among shooting targets included in the multi-view images (step S107). For example, the specifying unit 132 specifies the target of annotation tagging from among the shooting targets included in the multi-view image according to the selection of the target of tagging by the contributor. Alternatively, the identifying unit 132 identifies, by image recognition or machine learning, an annotation tagging target from shooting targets included in the multi-viewpoint image.

Subsequently, the tagging unit 134 of the information providing device 100 tags the object to be tagged (step S108). At this time, the estimation unit 133 of the information providing apparatus 100 estimates the position of the tagging target within the multi-view image. The tag assigning unit 134 assigns a tag according to the position of the object to be tagged. Note that the position to which the tag is attached may be specified in advance by the contributor who posted the multi-view image.

At this time, when the tagging unit 134 gives the tag to the object to be tagged, the tag is given so as not to overlap with other objects and other tags. Further, when tagging a target to be tagged, the tagging unit 134 tags the target to be tagged if the target to be tagged is not hidden by other targets. Note that when the tagging unit 134 gives a tag to a target to be tagged, even if the target to be tagged is hidden by another target, the tag to be tagged is the tag of the other target. If the priority of display is higher than the target, the target to be tagged may be tagged without assigning the tag to the other target.

Subsequently, the tag changing unit 135 of the information providing apparatus 100 changes the display position of the tag given to the target of tagging according to the image of each viewpoint of the multi-view image (step S109).
For example, when the position of the target of tagging in the screen changes due to the change of the viewpoint of the multi-view image, the tag changing unit 135 changes the display position of the tag according to the change of the position of the target of tagging. .

Subsequently, the image conversion unit 136 of the information providing apparatus 100 converts the face of the subject of the multi-viewpoint image into the face of the viewer according to the viewpoint at the time of shooting the multi-viewpoint image (step S110). At this time, the acquiring unit 131 of the information providing apparatus 100 acquires a multi-viewpoint image of the viewer's face via the communication unit 110 . Note that the acquisition unit 131 may acquire the multi-viewpoint image of the viewer's face in advance, or may acquire it at the time of viewing. In addition, the providing unit 137 of the information providing apparatus 100 provides the viewer with the multi-viewpoint image converted to the viewer's face via the communication unit 110 .

[6. Modification]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. So, below, the modification of embodiment is demonstrated.

In the above embodiment, part or all of the processing executed by the information providing device 100 may actually be executed by the terminal device 10 . For example, the processing may be completed stand-alone (by the terminal device 10 alone). In this case, it is assumed that the terminal device 10 has the functions of the information providing device 100 in the above embodiment. In addition, in the above-described embodiment, the terminal device 10 cooperates with the information providing device 100, so from the user U's point of view, it appears that the terminal device 10 is executing the processing of the information providing device 100 as well. That is, from another point of view, it can be said that the terminal device 10 includes the information providing device 100 .

Also, in the above-described embodiment, fashion items are used as an example of shooting targets (annotation target candidates) and tagging targets (annotation targets), but in reality they are not limited to fashion items. The object to be photographed and the object to be tagged (annotation object) may be an article photographed together with the user when the multi-view image is photographed. For example, it may be a wearable device worn by the user at the time of shooting, a terminal device held in the hand, or the like, home appliances (household appliances/electrical appliances) placed around the user at the time of shooting, or a background. It may be the interior of the room photographed together, the books on the bookshelf, the dishes and tableware on the kitchen or table, the art work, or the like.

Further, in the above embodiment, only a specific article may be photographed instead of the user when photographing multi-viewpoint images. For example, when an annotation tag is attached to a tagging target (annotation target) in a multi-view image, the user does not have to be included in the multi-view image.

[7. effect〕
As described above, the information processing device (the terminal device 10 and the information providing device 100) according to the present application includes the guide display unit 34 for displaying a predetermined shooting guide on the screen when shooting a multi-view image, a guide changing unit 35 for changing the photographing guide displayed on the screen according to the viewpoint every time the viewpoint is moved.

The guide display unit 34 displays on the screen a guide regarding at least one of the pose, facial expression, and manner of holding of the subject among the subjects as a photographing guide.

The guide display unit 34 displays the silhouette, outline, or translucent display of the subject on the screen as a shooting guide.

The guide display unit 34 superimposes a shooting guide on the subject when shooting the subject.

The guide display unit 34 identifies the pose of the person to be photographed among the subjects in the first photographed image, and selects and displays a photographing guide corresponding to the identified pose.

A guide changing unit 35 changes the shooting guide step by step as the viewpoint during shooting is moved.

The information processing apparatus according to the present application further includes a shooting determination unit 36 that automatically shoots when the subject matches the shooting guide.

The information processing apparatus according to the present application further includes a notification unit 37 that notifies the photographer of the location where the subject is out of alignment with the shooting guide when the subject is out of alignment with the shooting guide.

The information processing apparatus according to the present application can further improve the quality of service provision using multi-view images by one or a combination of the processes described above.

[8. Hardware configuration]
Also, the terminal device 10 and the information providing device 100 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 11, for example. The information providing apparatus 100 will be described below as an example. FIG. 11 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are buses. It has a form connected by 1090.

Arithmetic device 1030 operates based on programs stored in primary storage device 1040 and secondary storage device 1050, programs read from input device 1020, and the like, and executes various types of processing. The arithmetic unit 1030 is implemented by, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used for various calculations by the arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and other standardized connectors. Also, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by, for example, USB.

Also, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

Also, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 100 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 130 by executing a program loaded on the primary storage device 1040 . Further, arithmetic device 1030 of computer 1000 may load a program acquired from another device via network I/F 1080 onto primary storage device 1040 and execute the loaded program. Further, the arithmetic unit 1030 of the computer 1000 may cooperate with another device via the network I/F 1080, and call functions, data, etc. of the program from another program of the other device for use.

[9. others〕
Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the information providing apparatus 100 described above may be implemented by a plurality of server computers, and depending on the function, may be implemented by calling an external platform or the like using an API (Application Programming Interface), network computing, or the like. can be changed flexibly.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing content.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

1 information processing system 10 terminal device 34 guide display unit 35 guide change unit 36 photographing determination unit 37 recognition unit 38 notification unit 100 information providing device 110 communication unit 120 storage unit 121 user information database 122 history information database 123 image information database 130 control Section 131 Acquisition Section 132 Identification Section 133 Estimation Section 134 Tag Assignment Section 135 Tag Change Section 136 Image Conversion Section 137 Provision Section

Claims (10)

a guide display unit that displays a predetermined shooting guide on a screen when shooting a multi-view image;
a guide changing unit that changes the shooting guide displayed on the screen in accordance with the viewpoint each time the viewpoint during shooting is changed;
An information processing device comprising: 2. The information processing apparatus according to claim 1, wherein the guide display unit displays on the screen, as the shooting guide, a guide regarding at least one of a pose, an expression, and a way of holding the subject among the subjects. . 3. The information processing apparatus according to claim 1, wherein the guide display unit displays a silhouette, an outline, or a translucent display of a subject on the screen as the shooting guide. The information processing apparatus according to any one of claims 1 to 3, wherein the guide display unit superimposes the shooting guide on the subject when the subject is shot. 5. The guide display unit specifies a pose of a subject among subjects in an image that is first shot, and selects and displays a shooting guide corresponding to the specified pose. The information processing apparatus according to any one of 6. The information processing apparatus according to any one of claims 1 to 5, wherein the guide changing unit changes the shooting guide step by step as a viewpoint during shooting moves. a shooting determination unit that automatically shoots when the subject matches the shooting guide;
The information processing apparatus according to any one of claims 1 to 6, further comprising: a notification unit for notifying a photographer of a location where the subject is deviated from the shooting guide when the subject is deviated from the shooting guide;
The information processing apparatus according to any one of claims 1 to 7, further comprising: An information processing method executed by an information processing device,
a guide display step of displaying a predetermined shooting guide on a screen when shooting a multi-viewpoint image;
a guide change step of changing the shooting guide displayed on the screen in accordance with the viewpoint each time the viewpoint during shooting is changed;
An information processing method comprising: a guide display procedure for displaying a predetermined shooting guide on a screen when shooting a multi-viewpoint image;
a guide change procedure for changing the shooting guide displayed on the screen in accordance with the viewpoint each time the viewpoint during shooting is changed;
An information processing program for executing a computer.

Images