JP6656572B1 - Information processing apparatus, display control method, and display control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To increase the feeling of wearing and promote purchase. A user terminal displays a captured two-dimensional image, detects a predetermined subject that changes in time series from the two-dimensional image, and displays three-dimensional image data of a virtual object. Instruct the display posture. The user terminal determines a priority area to be displayed preferentially among the virtual objects displayed based on the display posture of the instructed virtual object of the three-dimensional image data. The user terminal displays the virtual object at a predetermined display position together with the two-dimensional image, and at least when the display of the displayed subject and the display of the virtual object overlap each other, the priority region in the overlapped region is set as the subject. And the virtual object area other than the priority area is hidden. [Selection diagram] FIG.

Description

  The present invention relates to an information processing device, a display control method, and a display control program.

  In recent years, services such as virtual fitting have been used. For example, a large display that looks like a mirror is placed in a store, the image of the dresser captured by the camera is captured on the large display, and the image data of the clothes is superimposed on the image of the dresser, as if trying on. Is known. There is also known a technique of calculating a time-series change of a skeleton position of a subject's body and controlling the attachment / detachment of virtual clothing items superimposed and displayed on the subject.

JP 2013-97699 A JP 2006-72669 A JP 2013-101527 A JP 2010-152556 A

  However, in the above technique, since there is a large difference in sense between real fitting and virtual fitting, it is hard to say that a satisfactory wearing feeling can be obtained, and it is hard to say that it is an effective method to promote purchase.

  In particular, in the case of small accessories such as rings and earrings, it is often the case that the user wears and removes the accessories repeatedly to check the wearing feeling. Virtual fitting of goods is rather difficult to see and is not effective. Also, going to the store and waiting for the turn or the like to perform virtual fitting is a burden on the user, and is not an effective method.

  In one aspect, an object is to provide an information processing device, a display control method, and a display control program that can enhance a feeling of wearing and promote purchase.

  According to a first aspect, an information processing apparatus includes a display unit that displays a captured two-dimensional image and a detection unit that detects a predetermined subject that changes in time series from the captured two-dimensional image. Means, storage means for storing three-dimensional image data of a virtual object to be displayed, and instructing means for instructing a display attitude for displaying the three-dimensional image data of the virtual object stored in the storage means on the display means. Deciding means for deciding a priority area to be displayed preferentially among the virtual objects displayed on the display means based on the display orientation of the three-dimensional data of the virtual object instructed by the instructing means; Displaying the virtual object at a predetermined display position of the display means together with the two-dimensional image, and displaying the virtual object at least on the display means. When the display of the virtual object overlaps with the displayed subject, the priority area determined by the determining unit is displayed in priority to the subject, and the overlapping area other than the priority area is displayed. Display control means for hiding an area of the virtual object.

  In one aspect, purchasing can be promoted by enhancing the feeling of wearing.

FIG. 1 is a diagram illustrating an example of the overall configuration of a system according to the first embodiment. FIG. 2 is a diagram illustrating a usage scene of the system according to the first embodiment. FIG. 3 is a functional block diagram illustrating the functional configuration of the product server according to the first embodiment. FIG. 4 is a diagram illustrating an example of information stored in the product information DB. FIG. 5 is a diagram illustrating an example of information stored in the image data DB. FIG. 6 is a diagram illustrating transition of a displayed Web page. FIG. 7 is a diagram illustrating an example of generating two-dimensional image data of a virtual object. FIG. 8 is a diagram illustrating generated two-dimensional image data. FIG. 9 is a diagram illustrating a fitting image. FIG. 10 is a diagram illustrating a configuration of fitting of a virtual object. FIG. 11 is a diagram illustrating the transition of fitting. FIG. 12 is a diagram for explaining another example of fitting. FIG. 13 is a sequence diagram illustrating a flow of the fitting process. FIG. 14 is a diagram illustrating an example of a hardware configuration.

  Hereinafter, embodiments of an information processing apparatus, a display control method, and a display control program disclosed in the present application will be described in detail with reference to the drawings. The present invention is not limited by the embodiment. In addition, the embodiments can be appropriately combined within a consistent range.

[System configuration]
FIG. 1 is a diagram illustrating an example of the overall configuration of a system according to the first embodiment. As shown in FIG. 1, in this system, a product server 10 and a user terminal 30 are communicably connected to each other via a network N. Note that the numbers of the product servers 10 and the user terminals 30 are merely examples, and are not limited to those illustrated. In addition, various communication networks such as the Internet and a dedicated line can be adopted as the network N regardless of whether it is wired or wireless.

  The product server 10 is an example of a server device that provides the user terminal 30 with various services such as product browsing, fitting, and purchasing. For example, when the product server 10 receives an access from the user terminal 5 via a Web browser or the like, the product server 10 displays a product site on which information of various products is posted. And the goods server 10 provides various services by each page displayed by the Web browser.

  The user terminal 30 is an example of a terminal device used by a user who considers purchasing accessories such as a ring and a scarf, and is, for example, a terminal device having an imaging unit such as a camera and a Web browser. For example, as the user terminal 30, a smartphone, a mobile phone, a tablet, a personal computer, or the like can be employed. The user terminal 30 accesses the product server 10 using a Web browser or the like, and uses various services provided by the product server 10.

  Here, the user terminal 30 acquires the information on the product from the product server 10 and executes the fitting service on the Web browser. Specifically, the product server 10 stores, for example, product information, which is information on the product such as the price and size of the product such as a ring, and three-dimensional image data of the product. Then, when receiving the access from the user terminal 30, the product server 10 displays a product site where the product information is posted on the user terminal 30. Subsequently, when the product server 10 receives a selection of a product to be tried on from the user terminal 30 on a Web browser that displays a product site, the product server 10 displays a fitting page incorporating the three-dimensional image data of the product and a control script. Responds to user terminal 30.

  In this way, the user tries on the product on the user terminal 30 using the fitting page in which the three-dimensional image data of the product to be tried on and the control script are incorporated. Specifically, the user terminal 30 displays an image of a subject (for example, a hand) photographed by the camera on a fitting page displayed by a Web browser. Then, the user terminal 30 generates the two-dimensional image data from the three-dimensional image data of the product and superimposes the two-dimensional image data on the video of the subject on the Web browser, thereby realizing the fitting.

[Use scene]
Next, the use scene of the try-on service executed using the user terminal 30 will be described. FIG. 2 is a diagram illustrating a usage scene of the system according to the first embodiment. As shown in FIG. 2, the user obtains a try-on page incorporating the three-dimensional image data of the product and a control script from the product server 10, so that the user is not limited to online, but also in a try-on service even in an offline state. Can be used.

  For example, as shown in FIG. 2, a ring fitting service can be used while performing wireless communication in a living room at home. In addition, when lovers consider an engagement ring, by using the fitting service provided by the product server 10, it is possible to repeatedly try on engagement rings of various brands in the living room. You can also narrow down the target. Also, a ring fitting service can be used on a train on the way to work.

  As described above, the fitting service provided by the commodity server 10 does not need to go to an actual store, and can be used using a Web browser. Therefore, the fitting service can be used in various places to select a place. .

[Function configuration]
Next, the functional configuration of each device shown in FIG. 1 will be described with reference to FIG. FIG. 3 is a functional block diagram illustrating the functional configuration of the system according to the first embodiment. Here, each of the product server 10 and the user terminal 30 will be described.

(Functional Configuration of Product Server 10)
As shown in FIG. 3, the product server 10 includes a communication unit 11, a storage unit 12, and a control unit 20. The communication unit 11 is a processing unit that controls communication with another device, and is, for example, a communication interface. For example, the communication unit 11 establishes a session with the user terminal 30 using a Web browser, and transmits and receives data via the Web browser.

  The storage unit 12 is an example of a storage device that stores various data and various programs executed by the control unit 20, and is, for example, a memory or a hard disk. The storage unit 12 stores a product information DB 13 and an image data DB 14.

  The product information DB 13 is a database that stores data on each product handled by the product server 10. Specifically, the product information DB 13 stores data on each piece of clothing. FIG. 4 is a diagram illustrating an example of information stored in the product information DB 13. As shown in FIG. 4, the product information DB 13 stores “product ID, product, price” and the like in association with each other.

  The “merchandise ID” stored here is an identifier for identifying the merchandise, the “merchandise” is a merchandise name or the like, and the “price” is the price of the merchandise. In the example of FIG. 4, the price of the product “Ring A” identified by the product ID “ID01” is “30,000 yen”.

  The image data DB 14 is a database that stores three-dimensional image data (hereinafter, may be simply referred to as “three-dimensional image data”) of each product. Specifically, the image data DB 14 stores three-dimensional image data transmitted as a virtual object to the user terminal 30 when using the fitting service.

  FIG. 5 is a diagram illustrating an example of information stored in the image data DB 14. As shown in FIG. 5, the image data DB 14 stores “product ID, product, three-dimensional image data” in association with each other. The “product ID” stored here is the same as the product ID in the product information DB 13 and is an identifier for identifying the product. “Product” is the same as the product in the product information DB 13, such as a product name. “Three-dimensional image data” is three-dimensional image data of a product. In the example of FIG. 5, the image data “3D image data A” is associated with the product “Ring A” identified by the product ID “ID01”. Note that the three-dimensional image data can be created from data obtained by imaging the product itself, created in a virtual space, or created using a known method.

  The control unit 20 is a processing unit that controls the entire product server 10, and is, for example, a processor. The control unit 20 includes a reception unit 21 and a response unit 22. The receiving unit 21 and the responding unit 22 are an example of an electronic circuit included in the processor and an example of a process executed by the processor. The control unit 20 also executes an authentication process for authenticating whether or not the accessing user is a legitimate user, a product purchase process executed when the user has indicated an intention to purchase, and the like. Detailed description is omitted.

  The receiving unit 21 is a processing unit that displays various Web pages on the user terminal 30 in response to a request from the user terminal 30 and receives various operations from the user terminal 30. Specifically, upon receiving access from the user terminal 30 using a Web browser or the like, the receiving unit 21 displays a product site on the user terminal 30. Then, when receiving the selection of a product on the product site, the receiving unit 21 displays a product page on which detailed information of the product is posted on the user terminal 30. In this way, the reception unit 21 displays a Web page or the like corresponding to the user's operation.

  The response unit 22 is a processing unit that responds to the user terminal 30 with various information for providing a try-on service when a request to try on a product is received from the user terminal 30. Specifically, when the user selects fitting on the product page displayed by the accepting unit 21, the responding unit 22 acquires, from the image data DB 14, three-dimensional image data of the product requested to be tried on. Then, the response unit 23 responds to the user terminal 30 with a try-on page incorporating the three-dimensional image data of the product used as the virtual object and a control script related to the operation of the three-dimensional image data of the product.

  Here, the transition of the displayed Web page will be described. FIG. 6 is a diagram illustrating transition of a displayed Web page. As shown in FIG. 6A, when receiving the access from the user terminal 30, the receiving unit 21 displays a product site 60 on which each product is posted on the user terminal 30. Note that a slider or the like for scrolling the screen is also displayed on the product site 60.

  Subsequently, as shown in FIG. 6B, when the product “ring A” is selected on the product site 60, the receiving unit 21 stores the detailed information of the selected product “ring A” in the product information DB 13. And displays on the user terminal 30 a product page 61 on which detailed information of the product “Ring A” is posted. The product page 61 also displays a “return” button for returning to the previous screen, a “purchase” button for proceeding to purchase of the product, a “try-on” button for proceeding to try on the product, and the like.

  Thereafter, as shown in FIG. 6C, when the “try on” button is selected on the product page 61, the response unit 22 converts the three-dimensional image data A of the displayed product “ring A” into image data. A try-on page 62 for trying on the product “Ring A”, which is obtained from the DB 14 and incorporates the three-dimensional image data A and various control scripts for performing the try-on, is displayed. The various processes executed by the control script provide the functions of various processing units described in the user terminal 30. The try-on page 62 includes an area to be photographed by the camera in cooperation with the camera of the user terminal 30 that displays the try-on page 62 on the display, and a virtual object of the product “Ring A” is displayed in the area. You. The fitting page 62 also displays a “return” button for returning to the previous screen, a “purchase” button for proceeding to purchase of a product, and the like.

(Functional Configuration of User Terminal 30)
As shown in FIG. 3, the user terminal 30 includes a communication unit 31, an imaging unit 32, a storage unit 33, and a control unit 40.

  The communication unit 31 is a processing unit that controls communication with another device, and is, for example, a communication interface. For example, the communication unit 31 establishes a session with the product server 10 using a Web browser, and transmits and receives data via the Web browser.

  The imaging unit 32 is a processing unit that acquires two-dimensional image data such as images and videos captured using a camera or the like. For example, the imaging unit 32 captures an image of a hand, a face, a body, and the like of the user performing the fitting, and outputs captured image data to the control unit 40.

  The storage unit 33 is an example of a storage device that stores various data and various programs executed by the control unit 40, and is, for example, a memory or a hard disk. For example, the storage unit 33 stores three-dimensional image data of a product received from the product server 10, image data captured by the imaging unit 32, and the like.

  The control unit 40 is a processing unit that controls the entire user terminal 30, and is, for example, a processor. The control unit 40 includes an access unit 41 and a try-on control unit 42. The access unit 41 and the try-on control unit 42 are an example of an electronic circuit included in the processor and an example of a process executed by the processor.

  The access unit 41 is a processing unit that executes various controls using the Web browser with the product server 10. For example, the access unit 41 activates a Web browser and accesses the product server 10 according to a user's instruction operation. Then, the access unit 41 displays the product site 60 or the product page 61 received from the product server 10 on a display or the like. Further, when receiving the fitting page 62 in which the three-dimensional image data of the product to be tried on and the control script are incorporated, the access unit 41 displays the fitting page on a display or the like.

  The fitting control unit 42 has a two-dimensional image generation unit 43 and a display control unit 44, and is a processing unit that executes a fitting service on the display of the user terminal 30. Specifically, when the fitting page 62 is acquired by the access unit 41, the fitting control unit 42 executes a control script incorporated in the fitting page 62, and executes the control script to execute the two-dimensional image generation unit 43 and the display control unit 44. By executing the processing described above, the control using the virtual space for generating the two-dimensional image data of the product to be tried on and the control for actually displaying the two-dimensional image data on the display are linked in real time. Perform the fitting service.

  The two-dimensional image generation unit 43 is a processing unit that generates, for each product, two-dimensional image data of a virtual object used in the try-on service. Specifically, the two-dimensional image generation unit 43 obtains three-dimensional image data of the product transmitted by the product server 10 from the fitting page 62 obtained by the access unit 41. Then, the two-dimensional image generation unit 43 arranges the three-dimensional image data of the product as a virtual object in a three-dimensional virtual space simulated on a computer. Then, the two-dimensional image generation unit 43 uses the virtual camera arranged in the three-dimensional virtual space to image the virtual object of the product, and generates two-dimensional image data. Further, the two-dimensional image generation unit 43 outputs two-dimensional image data of the product obtained by shooting to the display control unit 44 and the like. For simulating a three-dimensional virtual space, a virtual camera, and the like, known software can be used.

  FIG. 7 is a diagram illustrating an example of generating two-dimensional image data of a virtual object. FIG. 7 is a top view of a three-dimensional virtual space that generates two-dimensional image data of a virtual object of a product, and an object that becomes a product using a plurality of virtual cameras arranged in the virtual space. Is imaged. As shown in FIG. 7, the virtual camera 1 and the virtual camera 1 'are installed at the same position in the virtual space. Then, the virtual object (three-dimensional image data) is arranged on the boundary between the area P where the virtual camera 1 can shoot and the area P 'where the virtual camera 1' can shoot. The virtual object can be rotated in any direction. In addition, a portion of the virtual object that is out of the area that can be photographed by each virtual camera does not appear in the image of the virtual camera. Then, the virtual object is rotated, the virtual object is photographed by each virtual camera, and the photographed images are combined to generate two-dimensional image data of the virtual object.

  This will be described in more detail. The center coordinates of the virtual object in the virtual space are set to (x = 0, y = 0, z = 100), and the virtual object can rotate in various directions in the virtual space. The coordinates do not change. The two virtual cameras (virtual camera 1 and virtual camera 1 ') are located at (x = 0, y = 0, z = 0) in the virtual space (x = 0, y = 0, z = 100) direction is photographed.

  Then, the virtual camera 1 captures and images an area from z ≧ 10 to z ≦ 100, and does not show an area of z <10 or z> 100. The virtual camera 1 ′ captures and images an area from z> 100 to z ≦ 200, and does not show an area where z ≦ 100 or z> 200. That is, the virtual camera 1 obtains an image in which a part of the virtual object included in the region from z ≧ 10 to z ≦ 100 is reflected, and the other portions are transmitted. From the virtual camera 1 ′, a part of the virtual object included in the region from z> 100 to z ≦ 200 is reflected, and an image in which the rest is transmitted is obtained.

  Then, the images of the virtual camera 1 and the virtual camera 1 'are output to canvas elements in HTML (Hyper Text Markup Language), respectively, and displayed on the screen, so that the canvas and the virtual camera 1' which output the image of the virtual camera 1 are output. And the canvas that output the video of the image are laid out so as to be overlapped and displayed at the same position on the screen.

  As described above, the two-dimensional image generation unit 43 arranges the virtual object of the product selected as the fitting target on the simulated three-dimensional space, combines the images captured by the two virtual cameras, and generates a two-dimensional image of the product. Generate data. FIG. 8 is a diagram illustrating generated two-dimensional image data. As shown in FIG. 8, the two-dimensional image generation unit 43 combines the video obtained from the virtual camera 1 and the video obtained from the virtual camera 1 ′ to generate one two-dimensional image data. Note that, in one piece of two-dimensional image data, an area of a video (data) captured by the virtual camera 1 is a priority area that is always displayed with priority over the subject.

  As described above, each virtual camera in the virtual space always captures a virtual object in the virtual space and generates real-time two-dimensional image data of the virtual object, which is three-dimensional image data.

  Here, the generation timing of the two-dimensional image data will be specifically described. For example, when starting the try-on service, the two-dimensional image generation unit 43 generates a two-dimensional image data by arranging a virtual object at a predetermined position in a virtual space with a predetermined default posture.

  In addition, when the user operates the two-dimensional image data of the virtual object displayed on the fitting page 62, the two-dimensional image generation unit 43 responds to the operation by using the virtual object (three-dimensional image data) in the virtual space. Is changed to generate two-dimensional image data. More specifically, upon detecting a flick operation on the two-dimensional image data of the virtual object displayed on the fitting page 62, the two-dimensional image generation unit 43 rotates the virtual object in the virtual space by a distance corresponding to the flick operation. Let it. Then, the two-dimensional image generation unit 43 generates a two-dimensional image data by photographing the virtual object in the posture after the rotation.

  Returning to FIG. 3, the display control unit 44 is a processing unit that executes various processes related to display control of the fitting page 62 displayed on the display of the user terminal 30. Specifically, the display control unit 44 executes a display process of displaying two-dimensional image data of the virtual object, a detection process of detecting a subject to be tried on, a fitting process of performing a try on using the two-dimensional image data, and the like. .

(Display processing)
For example, the display control unit 44 is a processing unit that displays the two-dimensional image data of the virtual object generated by the two-dimensional image generation unit 43 at a predetermined position on the fitting page 62. That is, the display control unit 44 displays the two-dimensional image data generated by the two-dimensional image generation unit 43 as needed on the fitting page 62 displayed on the display in real time.

(Detection processing)
The display control unit 44 detects a subject from image data captured by a camera or the like. For example, the display control unit 44 displays image data that is a two-dimensional image acquired by the imaging unit 32 of the user terminal 30 on the fitting page 62, and detects a subject that changes in time series from the displayed image data. . Specifically, the display control unit 44 performs various methods such as an outline of a subject, an area of a designated color such as a skin color, an area similar to a predetermined type, and the like from image data captured by a camera. To detect a subject such as a hand.

  Here, as an example of a subject detection method, (1) a method of detecting a region similar to a specific color, (2) a method using feature amount extraction, and (3) a method using background difference will be described.

(1) Method of Detecting Region Similar to Specific Color For example, the display control unit 44 can determine only a pixel that matches a predetermined reference color as a subject. Specifically, the display control unit 44 holds in advance a range of RGB values (pixel values) to be detected. Then, the display control unit 44 refers to each pixel of the captured image data and determines whether or not the pixel value is within the range of the RGB values to be detected. Then, the display control unit 44 determines each pixel determined to be within the range of the RGB values to be detected as the subject.

  In addition, the display control unit 44 accepts a user's selection on the captured image data. Then, the display control unit 44 determines a predetermined range as a detection target with the RGB value of the selected pixel as a median value. Thereafter, the display control unit 44 refers to each pixel of the captured image data and determines each pixel within the range of the RGB values to be detected as the subject. In this way, the display control unit 44 can detect the contour of the subject (for example, a hand) with reference to each pixel of the image data. The predetermined range and the like can be arbitrarily changed.

(2) Method Using Feature Extraction For example, the display control unit 44 detects a feature from the image data, compares the similarity with a predetermined feature of the subject, and regards a similar part to the subject. You can also judge. More specifically, the display control unit 44 includes a method (Haar Like feature) that features a pattern of arrangement of brightness differences between pixels, a method (Local Binary Pattern feature) that features a distribution of luminance between pixels, and a luminance. (Histogram of Oriented Gradients feature) or the like, which is characterized by the combination of the directions (how to apply gradation in image data).

(3) Method Using Background Difference For example, the display control unit 44 can compare images in time series and detect a region having a difference as a subject. Specifically, the display control unit 44 sequentially acquires image data from the imaging unit 32, for example, every one second. Then, the display control unit 44 compares each of the preceding and following image data, and can determine an area in which the image data has moved as a subject.

(Try-on process)
The display control unit 44 also displays, on the fitting page 62, two-dimensional image data of the virtual object generated by the two-dimensional image generation unit 43 and a subject detected from image data captured in real time by a camera or the like. Display and try on the virtual object.

  For example, the display control unit 44 displays the two-dimensional image data corresponding to the product at a predetermined display position together with the image data of the subject that is occasionally recognized from the captured image data. When the displayed subject and the display of the two-dimensional image data overlap with each other, the display control unit 44 displays the two-dimensional image data of the two-dimensional image data captured by the virtual camera 1 to be preferentially displayed in the overlapping area. The area is displayed in preference to the subject, and the other areas of the two-dimensional image data are not displayed.

[Concrete example]
Next, a specific example of trying on the ring A will be described as an example with reference to FIGS. 9 to 12. Here, a user operation, display control, a screen transition example in fitting, and a fitting example will be sequentially described.

(User operation)
FIG. 9 is a diagram illustrating a fitting image. As illustrated in FIG. 9, the display control unit 44 of the user terminal 30 displays the two-dimensional image data Q of the ring A generated by the two-dimensional image generation unit 43 on the fitting page 62 displayed on the display. When the user shoots his / her own hand using the camera of the user terminal 30, the display control unit 44 detects the hand of the user as the subject and displays it on the fitting page 62. Then, when the user moves his or her hand while shooting with the camera, the display control unit 44 detects the user's finger from each image data shot in time series and displays it on the fitting page 62 as needed. As a result, the user performs the fitting by putting a ring (two-dimensional image data Q) on the finger on the fitting page 62 while photographing the hand with the camera.

(Display control)
Here, the try-on control unit 42 of the user terminal 30 acquires the state of the finger imaged in real time by a camera or the like via the Web browser, and implements the try-on on the display of the user terminal 30. FIG. 10 is a diagram illustrating a configuration of fitting of two-dimensional image data corresponding to a product.

  As illustrated in FIG. 10, the display control unit 44 cuts out a video (layer 2) indicating an area of only the detected subject (hand) from the video (layer 3) of the real object captured by the user terminal 30. Then, the display control unit 44 controls the image of the layer 1 ′ that is not preferentially displayed in the two-dimensional image data of the virtual object of the product generated by the two-dimensional image generation unit 43, and the subject (hand) cut out from the layer 3. The video (layer 2) showing only the area and the video of layer 1 which is preferentially displayed in the two-dimensional image data corresponding to the product are sequentially superimposed and projected on one screen. The display control unit 44 controls the layers 1 ′, 2, and 1 by making areas other than the display target object and the hand transparent so that the lower layer is reflected even when the layers are overlapped. I do. In addition, the display control unit 44 can display the background when the hand is shot with the camera as the background of the try-on page 62, or can display another background prepared in advance.

(Example of screen transition during fitting)
Next, an actual fitting example will be described. FIG. 11 is a diagram illustrating the transition of fitting. As shown in FIG. 11A, the try-on control unit 42 generates two-dimensional image data of the ring A obtained by combining the image of the layer 1 and the image of the layer 1 ′ (in the description of FIG. 11, simply described as the ring A). (Sometimes) is displayed on the try-on page 62 displayed in the area designated in advance, and the image captured by the camera of the user terminal 30 is displayed to detect the hand as the subject.

  Subsequently, as shown in FIG. 11B, when the hand approaches the ring A by the operation of the user and the finger of the user starts to overlap with the ring A, the try-on control unit 42 determines the layer 1 ′ portion of the ring A. Display with more priority on the hand. As a result, the user's finger is displayed on the layer 1 'of the ring A.

  After that, as shown in FIG. 11C, the try-on control unit 42 determines that the hand approaches the ring A by the user's operation, and when the user's finger completely overlaps the ring A, the layer 1 'portion of the ring A The hand is displayed with higher priority than the hand, and the layer A portion of the ring A is displayed with higher priority than the hand.

  As a result, the try-on control unit 42 can display as if the ring A was worn on the finger. Since the fitting controller 42 hides only the portion of the layer A 'of the ring A that overlaps the finger, the size of the non-displayed area changes depending on the thickness of the finger and the like. Therefore, a feeling of wearing suitable for each finger for each user can be provided.

(Example of fitting)
Next, the try-on control unit 42 can move the two-dimensional image data (which may be simply described as a ring in the description of FIG. 12) corresponding to a product such as a ring in various directions, so that the various directions can be used. Can be provided to the user. FIG. 12 is a diagram for explaining another example of fitting.

  As shown in FIG. 12, the display control unit 44 of the user terminal 30 displays, on the fitting page 62, a slider A for rotating the ring in the horizontal direction and a slider B for rotating the ring in the vertical direction. With these sliders, the ring can be changed in various directions. For example, when the slider is operated, the display control unit 44 detects an operation direction and an operation distance and outputs the operation direction and the operation distance to the two-dimensional image generation unit 43. The two-dimensional image generation unit 43 changes the posture of the virtual object arranged in the virtual space according to the input operation direction and operation distance, captures the virtual object having the changed posture, and converts the two-dimensional image data. Generate. Then, the display control unit 44 displays the new two-dimensional image data on the fitting page 62. As described above, the user operation of the fitting page 62 is notified to the virtual space in real time, and the posture of the virtual object can be dynamically changed. Therefore, the user can determine whether the product is moving on the fitting page 62. Can be recognized as follows. The operation such as rotation of the ring is not limited to the slider, and a flick operation in a state where the ring is selected can be adopted.

  The try-on control unit 42 of the user terminal 30 detects the user's hand with the ring turned sideways, and provides a try-on view from the side. Also in this case, the fitting control unit 42 displays the hand with priority over the layer 1 'portion of the ring, and also displays the hand with the layer 1 portion of the ring prior to the hand.

  In addition, the fitting control unit 42 detects the user's hand with the ring on the back side, that is, with the palm-side ring visible, and provides the fitting viewed from the back side. Also in this case, the fitting control unit 42 displays the hand with priority over the layer 1 'portion of the ring, and also displays the hand with the layer 1 portion of the ring prior to the hand. As shown in FIG. 12, in the case of the back side, the position of the image of the layer is opposite to that of the normal state, but the image of the layer 1 is always given priority, and the image of the layer 1 'is not always given priority. Similarly, it is possible to provide a video that is virtually correctly mounted.

  Here, an example in which a ring is rotated using a slider has been described, but the present invention is not limited to this. For example, the two-dimensional image generation unit 43 can also generate two-dimensional image data in which the posture and orientation of the ring are changed by a flick operation or the like on the displayed ring. Further, by performing a flick operation on an arbitrary area of the display, it is possible to generate two-dimensional image data in which the orientation of the ring is changed according to the direction of the operated flick. Also, the ring displayed on the fitting page 62 can be enlarged or reduced by the two-dimensional image generation unit 43 performing enlargement or reduction of the two-dimensional image data in response to a double click, a pinch out operation, or the like.

[Processing flow]
FIG. 13 is a sequence diagram illustrating a flow of the fitting process. As shown in FIG. 13, the access unit 41 of the user terminal 30 accesses a site of the product server 10 using a Web browser by a user operation (S101 and S102). Then, the reception unit 21 of the product server 10 displays the product site 60 on which each product is posted on the user terminal 30 (S103 and S104).

  Thereafter, when a product is selected on the product site 60 in the user terminal 30 (S105 and S106), the receiving unit 21 of the product server 10 displays the product page 61 of the selected product on the user terminal 30 (S107). And S108).

  Then, when the try-on button is selected on the product page 61 in the user terminal 30 (S109 and S110), the response unit 23 of the product server 10 transmits the three-dimensional image data of the selected product and the control script. The incorporated fitting page 62 is returned to the user terminal 30 (S111 and S112).

  In this way, when the try-on control unit 42 of the user terminal 30 acquires the try-on page 62 (S113), it generates two-dimensional image data corresponding to the product using the three-dimensional image data included in the try-on page 62. (S114).

  Thereafter, the try-on control unit 42 of the user terminal 30 uses the control script transmitted from the product server 10 to display a try-on page 62 displaying two-dimensional image data of the product on a display or the like (S115).

  Next, the try-on control unit 42 of the user terminal 30 executes the try-on service by executing an operation of imaging the hand (S116), rotating the two-dimensional image data of the product (S117), checking on the try-on of the product (S118), and the like. Execute

  Then, while the fitting is repeated on the user terminal 30 (S119: No), S116 and subsequent steps are repeated, and when the operation of ending the fitting on the user terminal 30 is performed (S119: Yes), the fitting service is ended (S119: Yes). S120).

[effect]
As described above, the user terminal 30 can execute the try-on service of the product regardless of the place and the time. In addition, the user terminal 30 can dynamically control the non-displayed part of the two-dimensional image data corresponding to the product according to the size of each user's finger, etc. Such a feeling of wearing can be provided. As a result, the user terminal 30 can enhance the feeling of wearing in the try-on service and promote purchase.

  [C] Third Embodiment Although the embodiments of the present invention have been described above, the present invention may be implemented in various different forms other than the above-described embodiments.

[Product example]
In the above example, a ring was described as an example, but the present invention is not limited to this. For example, apparel such as scarves, bracelets, anklets, earrings, earrings, watches, hats, etc. can be treated similarly. In addition, the user performs a try-on using a part of the hand or the like by himself / herself, but it is also possible to adopt a product such as clothes that performs a try-on by photographing the entire user. For example, an infant, a cared person, or the like can be photographed to try on clothes. Further, not only humans but also pet clothes and accessories can be adopted.

[Numerical values, etc.]
The data examples, numerical examples, display examples, and the like used in the above embodiments are merely examples, and can be arbitrarily changed. For example, the configuration of various Web pages is an example, and can be arbitrarily changed. The subject is not limited to the finger, and the subject to be detected can be changed depending on the product, such as the neck for a scarf and the wrist for a watch. For example, the merchandise server 10 or the try-on control unit 42 manages the merchandise in association with the subject to be detected, so that the detection target can be dynamically changed depending on the merchandise to be tried on.

[Product operation]
Further, even when the two-dimensional image data corresponding to the product is displayed together with the subject, the display orientation of the product can be instructed. For example, even when the ring (two-dimensional image data) overlaps with the finger, the fitting controller 42 performs rotation or the like in the virtual space when receiving an operation such as rotation of the ring on the fitting page 62. Thus, the rotated two-dimensional image data can be generated and displayed on the fitting page 62. Also in the rotated two-dimensional image data at this time, the image of layer 1 is displayed with priority. In addition, as a method of specifying an area operated by the user on the display, a known method can be adopted.

  Also, when the two-dimensional image data of the ring is displayed on the fitting page 62 in such a posture that the finger cannot be worn, the fitting controller 42 recognizes the finger (subject) recognized from each image data taken in chronological order. Using the recognition result of the above, the direction and the moving direction of the finger on the fitting page 62 are detected, and the posture of the virtual object in the virtual space is corrected to a posture that can be mounted, and then two-dimensional image data is generated. , Can be displayed on the fitting page 62. As a result, even when the posture of the ring (two-dimensional image data) is incorrect due to a user operation, it is possible to dynamically correct the posture to a try-on posture.

[Example of site]
A site or page described in HTML (HyperText Markup Language) or the like can be adopted as the Web site or Web page described in the above embodiment. In addition, as a control script, JavaScript (registered trademark) or the like can be adopted. In addition to the virtual space, two-dimensional image data obtained by photographing a product from each direction may be prepared in advance and displayed on the fitting page 62.

[Applicable system]
In the above embodiment, an EC (Electronic Commerce) site for browsing, trying on, and purchasing a product has been described as an example, but the present invention is not limited to this. For example, a service equivalent to the above-mentioned fitting service can be provided by generating three-dimensional image data as a virtual object at the stage of designing before making a prototype or an actual product. As a result, the design can be modified or consulted before the mass production of the product, so that unnecessary costs can be reduced and the manufacturing process can be made more efficient.

  Further, a fitting service can be provided on the product server 10. Specifically, the product server 10 includes a processing unit equivalent to the user terminal 30 and the try-on control unit 42, and provides a try-on service to the user terminal 30 using a Web browser. In addition, the user terminal 30 can execute the try-on service by acquiring the three-dimensional image data of the product from the product server 10 using a Web browser in which a control script is already installed.

  In the above example, the example in which the user terminal 30 acquires the three-dimensional image data of one product from the product server 10 is described. However, the present invention is not limited thereto. 10 can also be obtained. For example, the user terminal 30 selects three rings to be tried on and acquires the respective three-dimensional image data from the product server 10. The user terminal 30 can execute the try-on service by switching the target product by the instruction operation of the user.

[system]
The processing procedures, control procedures, specific names, and information including various data and parameters shown in the above-mentioned documents and drawings can be arbitrarily changed unless otherwise specified.

  Each component of each device illustrated is a functional concept, and does not necessarily need to be physically configured as illustrated. That is, the specific form of distribution and integration of each device is not limited to the illustrated one. That is, all or a part thereof can be configured to be functionally or physically distributed / integrated in arbitrary units according to various loads and usage conditions. For example, object generation, try-on service, product management, and the like can be realized by separate servers.

  Further, all or any part of each processing function performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic.

[hardware]
FIG. 14 is a diagram illustrating an example of a hardware configuration. Since the product server 10 and the user terminal 30 have the same hardware configuration, they will be described here as the computer 100. As shown in FIG. 14, the computer 100 includes a communication device 100a, a hard disk drive (HDD) 100b, a memory 100c, and a processor 100d. The units shown in FIG. 14 are mutually connected by a bus or the like.

  The communication device 100a is a network interface card or the like, and performs communication with another device. The HDD 100b stores programs and DBs for operating the functions shown in FIG.

  The processor 100d operates the process of executing each function described in FIG. 3 and the like by reading a program for executing the same processing as each processing unit illustrated in FIG. 3 from the HDD 10b or the like and expanding the program in the memory 10c. For example, taking the user terminal 30 as an example, this process performs the same function as each processing unit of the user terminal 30. Specifically, the processor 10d reads a program having the same functions as those of the access unit 41 and the try-on control unit 42 from the HDD 10b and the like. Then, the processor 10d executes a process of executing the same processing as the access unit 41 and the try-on control unit 42 and the like.

  As described above, the computer 100 operates as an information processing apparatus that executes the fitting control method by reading and executing the program. The computer 100 can also realize the same functions as those in the above-described embodiments by reading the program from a recording medium using a medium reading device and executing the read program. The programs referred to in the other embodiments are not limited to being executed by the computer 100. For example, the present invention can be similarly applied to a case where another computer or server executes a program, or a case where these execute a program in cooperation with each other.

10 product server 11 communication unit 12 storage unit 13 product information DB
14 Image data DB
Reference Signs List 20 control unit 21 reception unit 22 response unit 30 user terminal 31 communication unit 32 imaging unit 33 storage unit 40 control unit 41 access unit 42 try-on control unit 43 two-dimensional image generation unit 44 display control unit

Claims (11)

Display means for displaying the captured two-dimensional image;
Detecting means for detecting a predetermined subject that changes in time series from the photographed two-dimensional image;
Storage means for storing three-dimensional image data of a virtual object to be displayed;
Instructing means for instructing a display posture for displaying the three-dimensional image data of the virtual object stored in the storage means on the display means;
In the two-dimensional image data of the virtual object displayed on the display means based on the display orientation of the three-dimensional data of the virtual object instructed by the instructing means, a priority area to be displayed preferentially and a priority area other than the priority area Determining means for determining a non-priority area ;
A two-dimensional image of the photographed subject is displayed at a predetermined display position of the display means together with the priority area and the non-priority area of the two-dimensional image data of the virtual object , and is displayed on at least the display means. When the two-dimensional image of the subject and the display of the two-dimensional image data of the virtual object overlap each other, the priority area and the non-priority area in the overlapped area are replaced with the priority area of the subject . Display control means for displaying in priority to the three- dimensional image and for hiding the non-priority area,
Information processing device provided with. Said detecting means refers to each pixel of the captured two-dimensional image, the pixel value of each pixel, based the amount of change in feature amount or the pixel using each pixel, the time series varying The information processing apparatus according to claim 1, wherein a predetermined subject to be detected is detected. The instruction means is capable of instructing the display orientation of the three-dimensional image data of the virtual object even when the two-dimensional image data of the virtual object is displayed on the display means together with the subject.
The information processing apparatus according to claim 1, wherein the display control unit dynamically changes the priority area and the non-priority area according to a display orientation instruction from the instruction means. The virtual object is an accessory virtually worn on the subject,
2. The apparatus according to claim 1, wherein the determining unit determines a region other than a region that is hidden by the subject when attached to the subject and that is determined by a display orientation of the accessory, as the priority region that is displayed with priority over the subject. The information processing apparatus according to any one of claims 3 to 3. Simulating a three-dimensional virtual space having a first region and a second region, arranging three-dimensional image data of the virtual object at a boundary between the first region and the second region, Using a first virtual camera that captures only the first area, capturing a first partial image located in the first area of the three-dimensional image data of the virtual object, A second virtual camera that captures only the area is used to capture a second partial image located in the second area among the three-dimensional image data of the virtual object, and the first partial image and the second partial image are captured. Generating means for generating two-dimensional image data of the virtual object in combination with the two partial images;
2. The information processing apparatus according to claim 1, wherein the display control unit displays the first partial image as the priority area when displaying the two-dimensional image data of the virtual object on the display unit. apparatus. Displaying the captured two-dimensional image;
Detecting a predetermined subject that changes in time series from the two-dimensional image;
Instructing a display posture for displaying three-dimensional image data of the virtual object;
In the two-dimensional image data of the virtual object displayed based on the designated display orientation of the three-dimensional image data of the virtual object, a priority area to be displayed preferentially and a non-priority area other than the priority area are determined. Steps and
Together with the priority area and the non-priority area of the two-dimensional image data of the virtual object to the display position determined Me pre, display of two-dimensional image data of a two-dimensional image of an object that is at least displayed the virtual object is overlapped And when the priority area and the non-priority area in the overlapped area are displayed in preference to the two-dimensional image of the subject , and the non -priority area is hidden.
And a display control method. A process of displaying the captured two-dimensional image;
A process of detecting a predetermined subject that changes in time series from the two-dimensional image;
A process of instructing a display posture for displaying three-dimensional image data of the virtual object;
In the two-dimensional image data of the virtual object displayed based on the designated display orientation of the three-dimensional image data of the virtual object, a priority area to be displayed preferentially and a non-priority area other than the priority area are determined. Processing,
Together with the priority area and the non-priority area of the two-dimensional image data of the virtual object to the display position determined Me pre, display of two-dimensional image data of a two-dimensional image of an object that is at least displayed the virtual object is overlapped when the among the priority area and the non-priority area in the overlapping area, and displays with priority the priority area in the two-dimensional image of an object, a process to hide the non-priority area,
Display control program that causes a computer to execute Processing for detecting the object refers to each pixel of the captured two-dimensional image, the pixel value of each pixel, based the amount of change in feature amount or the pixel using each pixel, the time series The display control program according to claim 7, wherein a predetermined subject that changes in time is detected. The process of instructing the display posture can instruct the display posture of the three-dimensional image data of the virtual object even when the two-dimensional image data of the virtual object is displayed together with the subject,
9. The display control program according to claim 7, wherein the processing for controlling the display dynamically changes the priority area and the non-priority area according to an instruction of the display attitude. 10. The virtual object is an accessory virtually worn on the subject,
8. The processing of determining the priority area, wherein the priority area is displayed in priority to the subject other than an area that is hidden by the subject when attached to the subject and that is determined by a display orientation of the accessory. 10. The display control program according to any one of claims 9 to 9. Simulating a three-dimensional virtual space having a first region and a second region,
At the boundary between the first area and the second area, a three-dimensional image data of the virtual object corresponding to the accessory is arranged, and a first virtual camera for photographing only the first area is provided. A first partial image located in the first region in the three-dimensional image data of the virtual object;
Using a second virtual camera that captures only the second area, captures a second partial image located in the second area in the three-dimensional image data of the virtual object,
Causing the computer to further execute a process of generating two-dimensional image data of the virtual object, in which the first partial image and the second partial image are combined;
11. The display control program according to claim 10, wherein the displaying process displays the first partial image as the priority area when displaying the two-dimensional image data of the virtual object.

Images