JP2018180803A - Model arrangement program, model arrangement method and model arrangement device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more comfortably and efficiently perform net shopping.SOLUTION: A room model and interior model generation unit 422 generates a room model and an interior model on the basis of a plurality of picked-up images, a display processing unit 432 displays a room while allowing a viewpoint to be changeable on the basis of the room model, shows interior in the space of the displayed room in a display state or in a non-display state on the basis of the interior model, and displays the arrangement position of the displayed interior in a changeable manner, and a merchandise arrangement processing unit 434 acquires a merchandise model, shows merchandise in the space of the display room in a display state or in a non-display state on the basis of the merchandise model, and displays the displayed merchandise in the space of the room while allowing an arrangement position to be changeable.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a model placement program, a model placement method, and a model placement device.

  Conventionally, when trying to place an interior in a room, if the interior is actually placed in the room and it is not possible to check the situation, the interior and the dimensions of the room can be grasped more accurately. Then, based on the grasped dimensions, it is possible to guess the placement. In addition, it is only possible to intuitively grasp the sense of layout when the interior is arranged in a room.

  As an example, in an Internet shopping system that purchases a product using a network such as the Internet, an image (for example, a photo) of the product displayed on the display screen and product information (for example, text or drawing) including the dimensions of the product Based on the explanation by), grasp the dimensions etc. of the product, and imagine the situation where it is actually arranged in the room, decide the availability of the product and the propriety of the purchase of the product. ing.

  As related prior art, there is a technique for checking a collision with a peripheral part based on the arrangement position where the part picture representing the part to which the movable range is added is represented by the image on the floor plan by the user. (For example, refer to the following patent document 1).

  Also, as related prior art, referring to a furniture information file, a furniture image file, a room information file, and a room image file, the image size is adjusted according to the ratio between the size of the room in which the product is installed and the size of the product. There is a technique of adjusting the direction of the product image in the shooting direction of the room image by subjecting the product image to rotation processing according to the shooting direction in which the room was shot, and compositing the product image in the room image (for example, See Patent Document 2).

JP, 2004-126820, A JP 2004-318359 A

  However, in the prior art, it is necessary to determine the dimensions based on the flat display and the supplementary information, and it may not be possible to appropriately grasp the interior and the size and layout of the room in which the interior is arranged. Or, there is a problem that it takes time to properly grasp.

  In one aspect, the present invention aims to properly grasp the state of the size or layout of an interior and a room in which the interior is arranged.

  In one embodiment, a model of an interior and a model of a room included in the plurality of captured images are generated based on a plurality of captured images obtained by capturing the inside of a room, and a storage unit storing a model of a product is referred to. , Placing a model of the selected specific product in the model of the room, and individually moving or rotating the display of the model of the interior and the model of the product A model placement program, a model placement method, and a model placement apparatus are provided.

  According to one aspect of the present invention, it is possible to appropriately grasp the size of the interior and the size or layout of the room in which the interior is arranged.

FIG. 1 is an explanatory view showing an example of a system configuration of a net shopping system for realizing a model placement program, a model placement method, and a model placement apparatus according to the embodiment. FIG. 2 is an explanatory diagram of an example of the hardware configuration of the server. FIG. 3 is an explanatory view showing an example of the hardware configuration of the customer terminal device and the business operator terminal device. FIG. 4 is an explanatory view showing the flow of the entire processing of the Internet shopping system. FIG. 5 is a flowchart showing an example of the product model generation process. FIG. 6 is a flowchart of an example of a process of generating a room model / interior model. FIG. 7 is an explanatory view showing an image of the entire data configuration. FIG. 8 is a flow chart showing an example of the procedure for displaying a room model, an interior model and a product model. FIG. 9 is an explanatory view showing a display example (No. 1) of the display screen of the customer terminal device. FIG. 10A is an explanatory view showing a display example (No. 2) of the display screen of the customer terminal device. FIG. 10B is an explanatory view showing a display example (No. 3) of the display screen of the customer terminal device. FIG. 11 is an explanatory view showing a display example (No. 4) of the display screen of the customer terminal device. FIG. 12 is an explanatory view showing a display example (No. 5) of the display screen of the customer terminal device.

  Hereinafter, embodiments of a model placement program, a model placement method, and a model placement apparatus according to the present invention will be described in detail with reference to the drawings.

Embodiment
(System configuration example of the internet shopping system 100)
FIG. 1 is an explanatory view showing an example of a system configuration of a net shopping system for realizing a model placement program, a model placement method, and a model placement apparatus according to the embodiment. In FIG. 1, the Internet shopping system 100 includes a plurality of customer terminal devices 101, a plurality of business operator terminal devices 102, and a server 103 connected to the customer terminal devices 101 and the business operator terminal device 102. is there.

  In the Internet shopping system 100, the server 103 and the customer terminal device 101 and the business operator terminal device 102 are connected via a wired or wireless network 150. The network 150 may be, for example, the Internet, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), or the like.

  The server 103 controls the entire Internet shopping system 100, and performs product search, room space, interior, product display processing, and the like. The internet shopping system 100 shows a system for handling interiors such as furniture as an example of the internet shopping system.

  Therefore, the server 103 stores a product master 110 for storing information on interior products, customer information data 120 for storing information on a customer who performs online shopping, customer room data 121 for storing information on a customer room, and a customer room. It has various databases such as customer interior data 122 related to the interior of the arranged customer and purchase information data 123 including information related to the customer's purchase history. The storage contents of the product master 110 and the various databases 120 to 123 will be described later with reference to FIG.

  The customer terminal device 101 is a computer used by a customer who purchases a product in the internet shopping system 100. The customer terminal device 101 is, for example, a notebook PC, a desktop PC, a tablet PC, a smartphone, a mobile phone, a PHS, or the like.

  The business entity terminal device 102 is a computer used by a business that sells products in the internet shopping system 100. The business operator terminal device 102 is, for example, a notebook PC, a desktop PC, a tablet PC, a smartphone, a mobile phone, a PHS, or the like.

(Example of hardware configuration of server 103)
FIG. 2 is a block diagram showing an example of the hardware configuration of the server 103. As shown in FIG. In FIG. 2, the server 103 includes a central processing unit (CPU) 201, a memory 202, an interface (I / F) 203, a disk drive 204, and a disk 205. Further, each component is connected by a bus 200.

  Here, the CPU 201 is in charge of overall control of the server 103. The memory 202 includes, for example, a read only memory (ROM), a random access memory (RAM), and a flash ROM. Specifically, for example, a flash ROM or a ROM stores various programs, and a RAM is used as a work area of the CPU 201. The program stored in the memory 202 is loaded into the CPU 201 to cause the CPU 201 to execute the coded processing.

  The I / F 203 is connected to the network 150 through a communication line, and is connected to another device (for example, the customer terminal device 101 / business terminal device 102 shown in FIG. 1) via the network 150. Then, the I / F 203 controls the interface between the network 150 and the inside of the own apparatus, and controls input / output of data from other apparatuses. For example, a modem or a LAN adapter can be adopted as the I / F 203.

  The disk drive 204 controls read / write of data to the disk 205 according to the control of the CPU 201. The disk 205 stores data written under the control of the disk drive 204. Examples of the disk 205 include a magnetic disk and an optical disk.

  The server 103 may include, for example, a solid state drive (SSD), a keyboard, a pointing device, a display, and the like in addition to the above-described components.

(Hardware Configuration Example of Customer Terminal Device 101 / Business Operator Terminal Device 102)
FIG. 3 is a block diagram showing an example of the hardware configuration of the customer terminal device 101 and the business operator terminal device 102. As shown in FIG. In FIG. 3, the customer terminal device 101 and the business operator terminal device 102 have a CPU 301, a memory 302, an I / F 303, a display 304, a camera 305, and an input device 306. Also, each component is connected by a bus 300.

  Here, the CPU 301 is responsible for overall control of the customer terminal device 101 and the business operator terminal device 102. The memory 302 includes, for example, a ROM, a RAM, and a flash ROM. Specifically, for example, a flash ROM or a ROM stores various programs, and a RAM is used as a work area of the CPU 301. The program stored in the memory 302 is loaded into the CPU 301 to cause the CPU 301 to execute coded processing.

  The I / F 303 is connected to the network 150 through a communication line, and is connected to another device (for example, the server 103 illustrated in FIG. 1) through the network 150. Then, the I / F 303 controls the interface between the network 150 and the inside of the own apparatus, and controls input / output of data from other apparatuses.

  The display 304 displays a cursor, an icon, a toolbox, and data such as a document, an image, and function information. The display 304 can employ, for example, a liquid crystal display, an organic electroluminescence (EL) display, or the like. Also, the display 304 may be a head mounted display.

  The camera 305 is a device for capturing a control image or a moving image. Also, it may be a 3D camera capable of capturing 3D images. The camera 305 may be a camera dedicated to photographing. In that case, a captured image captured by a camera dedicated to capturing is captured into the customer terminal device 101 and the business operator terminal device 102. In addition, a camera incorporated in a tablet PC or a smartphone may be used. Further, the camera 305 may be a 3D camera system including a plurality of cameras for capturing 3D images.

  The input device 306 has keys for inputting characters, numbers, various instructions, and the like, and performs data input. The input device 306 may be a keyboard, a pointing device, or the like, or may be a touch panel input pad, a numeric keypad, or the like.

  The customer terminal device 101 and the business operator terminal device 102 may have various sensors such as an acceleration sensor and a gyro sensor, an HDD (Hard Disk Drive), an SSD, and the like in addition to the above-described components. As the input device 306, an acceleration sensor or a gyro sensor may be used to input information on a direction, an inclination, or the like.

(Overview of Internet shopping system)
Next, an outline of the internet shopping system 100 will be described with reference to FIG. FIG. 4 is an explanatory view showing the flow of the entire processing of the Internet shopping system. In FIG. 4, the Internet shopping system 100 includes a process 401 related to product data registration by an interior sales business, a process 402 related to customer room data registration by a new customer, and a process 403 related to interior purchase by an existing customer.

  The process 401 related to product data registration by the interior sales business is performed using the business operator terminal device 102 and the server 103. Further, a process 402 related to customer room data registration by a new customer and a process 403 related to interior purchase by an existing customer are performed using the customer terminal device 101 and the server 103.

  First, in processing 401 related to product data registration by the interior sales business, the interior sales business uses the business operator terminal device 102 to display the product image file 411 and information (including the price, type, etc. of the product) regarding the product. , And the product model generation unit 412 of the server 103.

  The product model generation unit 412 generates a product model based on the received product image file 411, and registers information (product model information) on the generated product model in the product master 110. Details of the content of the generation of the product model by the product model generation unit 412 will be described later using FIG.

  Specifically, for example, the product model generation unit 412 causes the CPU 201 to execute a program stored in a storage device such as the memory 202 or the disk 205 illustrated in FIG. 2 or the function of the I / F 203. To achieve.

  Although illustration is omitted, each process of the product model generation unit 412 may be performed by the business operator terminal device 102, and information on the generated product model may be sent to the server 103. Specifically, for example, by causing the CPU 301 to execute a program stored in a storage device such as the memory 302 of the business entity terminal device 102 shown in FIG. The function may be realized by F303.

  Next, in processing 402 related to customer room data registration by a new customer, the new customer sends the room image file 421 to the room model / interior model generation unit 422 of the server 103 using the customer terminal device 101. Also, the new customer sends customer information to the server 103 using the customer terminal device 101.

  The room model / interior model generation unit 422 generates a room model and an interior model based on the received room image file 421, and registers information (room model information) on the generated room model in the database 121 of the customer room data. The information on the generated interior model (interior model information) is registered in the database 122 of the customer interior data. Details of generation of a room model / interior model by the room model / interior model generation unit 422 will be described later with reference to FIG.

  Specifically, for example, the room model / interior model generation unit 422 causes the CPU 201 to execute a program stored in a storage device such as the memory 202 or the disk 205 of the server 103 illustrated in FIG. The function is realized by / F203.

  Although illustration is omitted, each process of the room model / interior model generation unit 422 may be performed by the customer terminal device 101, and information on the generated room model / interior model may be sent to the server 103. Specifically, for example, by causing the CPU 301 to execute a program stored in a storage device such as the memory 302 of the customer terminal device 101 shown in FIG. 3, the room model / interior model generation unit 422, or The function may be realized by the I / F 303.

  Further, the customer information registration unit 423 of the server 103 registers the received customer information in the database 120 of the customer information data.

  Next, in the process 403 related to interior purchase by an existing customer (a customer who has already registered as a new customer), the existing customer uses the customer terminal device 101 and is based on the information registered in the process 402 related to customer room data registration. And instructs the room model / interior model reading unit 431 to display the space of the customer's room.

  The room model / interior model reading unit 431 extracts room model information from the database 121 of customer room data, and interior information from the database 122 of customer interior data, and passes the information to the display processing unit 432.

  The display processing unit 432 displays on the customer terminal device 101 of the existing customer based on the received room model information and interior model information together with the shopping screen information. Also, according to the instructions from the customer, display the room space desired by the customer, change the viewpoint and display, arrange and display the interior in the customer room space, or change the arrangement position. Necessary information for executing display processing on the display 304 of the customer terminal device 101 so that the customer terminal device 101 can perform display processing (see FIG. 9, FIG. 10A, FIG. Send to

  Further, the existing customer uses the customer terminal device 101 to instruct the product search unit 433 to search for a product. The product search unit 433 searches the product master 110, extracts the product model information and product information of the searched product from the product master 110, and passes the product master 110 to the product arrangement processing unit 434.

  The product arrangement processing unit 434 can arrange and display the product in the space of the customer room, change the arrangement position, or hide it according to the instruction from the customer based on the received product model. As described above (see FIG. 11 and FIG. 12 described later), necessary information is transmitted to the customer terminal device 101 of the customer. Further, the product arrangement processing unit 434 transmits necessary information to the customer terminal device 101 of the customer so that the product information received on the shopping screen can be displayed on the display 304 of the customer terminal device 101.

  Specifically, the display processing unit 432, the product search unit 433, and the product arrangement processing unit 434 execute, for example, a program stored in a storage device such as the memory 202 and the disk 205 of the server 103 shown in FIG. The function is realized by causing or by the I / F 203.

  Although illustration is omitted, information on each model may be received from the server 103, and the processing of the display processing unit 432 and the product arrangement processing unit 434 may be performed by the customer terminal device 101. More specifically, the display processing unit 432 and the product arrangement processing unit 434 cause the CPU 301 to execute a program stored in a storage device such as the memory 302 of the customer terminal device 101 shown in FIG. 3, for example. Alternatively, the function may be realized by the I / F 303, the display 304, and the input device 306.

  After that, the existing customer instructs the purchase processing unit 435 to purchase a product using the customer terminal device 101 as necessary. The purchase processing unit 435 registers the received purchase information in the purchase information data database 123. Thereafter, various processing such as payment processing of the purchase price is executed.

  Specifically, for example, the purchase processing unit 435 and the processing unit (not shown) for performing payment processing may use the CPU 201 for storing a program stored in a storage device such as the memory 202 and the disk 205 of the server 103 shown in FIG. The function is realized by executing the program or by the I / F 203.

(Product model generation processing procedure)
FIG. 5 is a flowchart showing an example of the product model generation process. In the flowchart of FIG. 5, the product model generation unit 412 acquires a captured image of a product (step S501). The captured image acquired is obtained by capturing the product from different positions at different angles, and a plurality of captured images are required.

  Next, the product model generation unit 412 generates an imaging model from the plurality of acquired captured images (step S502). To create an imaging model, first, predetermined feature points of a product are extracted from a plurality of captured images using image recognition processing technology. The feature points of the product include the end portion of the product in the vertical direction, the horizontal direction, and the height direction, the feature point (protruding part or recessed part) concerning the shape of the product, the feature point by color, and the like. Extraction of feature points is performed for each of the acquired captured images.

  Then, for each captured image, the coordinates in the captured image of the extracted feature point are recognized. Next, the feature point common to each captured image is recognized, and based on the difference of the coordinates of the recognized common feature point, the dimensional ratio of the product in the vertical direction, horizontal direction, height direction, and three-dimensional shape are Obtain an understood imaging model. Thus, an imaging model of a product is created from a plurality of captured images.

  Here, when the information of the captured image is insufficient to create the captured image model, a notification may be sent to the business terminal device 102 indicating that the captured image needs to be added. At that time, comment information may be sent together regarding what kind of angle the captured image from which position is necessary.

  Next, the product is specified based on the imaging model of the product (step S503). The specification of the product is matching (for example, by a three-dimensional shape or the like) of the imaging model information, that is, the dimensions of the product according to the identified imaging model in the vertical direction, horizontal direction, height direction, and information on the product candidate. By which the item is the item (for example, a chair or a table, or a table if it is a table).

  Further, the product may be specified based on the product information (dimension information included in the product information, type information, and the like) for which the input is received together with the captured image. Further, the product may be specified based on both the result of the matching process and the product information. In addition, another method may be used to specify what kind of product the product according to the specified imaging model is.

  The imaging model creation and the specific series of processing of the product perform matching processing with the shape and the color to determine the optimal type. The matching process may be performed by, for example, deep learning (deep learning) using AI (Artificial Intelligence).

  Next, a product model is created (step S504). A product model is created by including specific dimensions, product type information, and the like in the imaging model created in step S502. This completes the product model. Furthermore, the product information may be completed as a product model including product information (material, color, product sale price, contact information, and other information necessary for purchase and sale of the product), or the product information as a product model You may register and manage separately.

  Then, the created product model / product information is registered in the product master 110 (step S505), and the series of processing of the flowchart of FIG. 5 is ended. The operator terminal device 102 may perform all or part of the above processes instead of the server 103.

(Room model / interior model generation processing procedure)
FIG. 6 is a flowchart of an example of a process of generating a room model / interior model. In the flowchart of FIG. 6, the room model / interior model generation unit 422 acquires a captured image of a product (step S601). The acquired captured image is obtained by capturing the target room at different angles from different positions of the product, and a plurality of captured images are required.

  Next, the room model / interior model generation unit 422 generates an imaging model (step S602). To create an imaging model, first, predetermined feature points displayed in an image are extracted from a plurality of captured images using an image recognition processing technique. The feature points of the room include the end in the vertical direction, the horizontal direction, the height direction, the feature point (protruding part or recessed part) of the shape, the feature point by color, and the like.

  Specifically, for example, any point on the wall-wall boundary, any point on the wall-floor boundary, or a point at which the interior is arranged. Also, the feature points may be extracted based on information registered in advance as feature point candidates as general room features. Extraction of feature points is performed for each of a plurality of acquired captured images.

  Then, for each captured image, the coordinates in the captured image of the extracted feature point are recognized. Next, the feature ratio common to each captured image is recognized, and based on the difference in coordinates of the recognized common feature point, the dimensional ratio of the vertical direction, horizontal direction, height direction of the room, and three-dimensional shape Obtain an understood imaging model. Thus, an imaging model is created from a plurality of captured images.

  Here, when the information of the captured image is insufficient to create the captured model, a notification may be sent to the customer terminal device 101 indicating that the captured image needs to be added. At that time, comment information may be sent together regarding what kind of angle the captured image from which position is necessary.

  Next, a predetermined member is specified from each captured image (step S603). The predetermined member is a member such as a door, a window, or the like, which has a predetermined size. The dimensions registered in advance for the predetermined member specified in step S603 are acquired from a dimension database (not shown). This dimension database may be in the server 103, or a database existing on the Internet may be used.

  The dimensions of each part of the room space are calculated by comparing the acquired dimensions of the predetermined member and the dimensions between the feature points of the room space of the imaging model (step S604). If a fraction appears in the dimensions by calculation, the calculation results may be rounded with reference to the dimensions of the standard.

  Next, the interior is identified (step S605). The imaging model is a mixture of a room model and an interior model. Therefore, an interior model is specified from the captured image. The difference with the background of the room is extracted from the color of the captured image and the like, and a predetermined feature point of the interior is extracted based on the difference. The feature points of the product include the end of the interior in the vertical direction, the horizontal direction, and the height direction, the feature points (protruding part or recessed part) relating to the shape of the interior, the feature points by color, and the like. Extraction of feature points is performed for each of the acquired captured images.

  Then, for each captured image, the coordinates in the captured image of the extracted feature point are recognized. Next, the feature point common to each captured image is recognized, and based on the difference in coordinates of the recognized common feature point, the dimensional ratio in the vertical direction, horizontal direction, height direction of the interior, and three-dimensional shape Obtain an understood imaging model. Thus, the imaging model of the interior is created from the plurality of captured images.

  Then, the interior is separated from the space of the room (step S606). The imaging model created in step S602 is compared with the imaging model of the interior created in step S605, and the space of the interior and the room is separated based on the comparison result.

  Next, a room model is created (step S607). A room model is created by including the dimensions and the like calculated in step S604 in the imaging model in which the interior is separated. This completes the room model. Furthermore, a wall or a floor may be recognized, and the recognized information may be registered.

  At this time, a portion of the room space where the interior model is excluded is corrected based on the peripheral image of the portion. Specifically, the floor and wall of the part that was hidden by the interior is corrected with the floor and wall of the same color.

  Next, an interior model is created (step S608). An interior model is created by including specific dimensions and type information, and which part of the room space the information model (for example, coordinates, direction information, etc.) is included in the imaging model created separately from the room model Do. Thereby, the arrangement position in the space of the interior room in the initial state can be known. Thus, the interior model is completed. The matching processing with the shape and color of the type candidate registered in advance is performed according to the shape and color of the interior according to the interior model, and the optimum type is determined.

  A series of processes of these imaging model creation, identification of a predetermined member, identification of an interior, separation of an interior model, creation of a room model, and creation of an interior model are, for example, deep learning (deep layer) using AI (Artificial Intelligence) It may be performed by learning).

  Further, in the case where the type can not be determined by the above method, the type may be input by inquiring the customer.

  Then, the created room model / interior model is registered (step S609), and the series of processes in the flowchart of FIG. 6 are ended.

  With regard to such creation processing, the captured image from the customer terminal device 101 may be received, the server 103 may execute each processing, and room model / interior model information may be registered, or the customer terminal device 101 may After installing application programs and the like necessary for processing in advance, each processing may be executed in the customer terminal device 101, and the created room model / interior model information may be transmitted to the server 103 and registered in the database. .

  Furthermore, in the customer terminal device 101, a room model / interior model may be created, and the created room model / interior model information may be stored in the memory 302 or the like of the customer terminal device 101 without transmitting it to the server 103. . The stored room model / interior model information may be transmitted to the server 103 as necessary, and without transmitting to the server 103, only the product models are received from the server 103, and they are transmitted. It may be matched and displayed on the display 304. By doing this, it is possible to prevent leakage of room model information and interior model information and to securely protect customer personal information.

(Overall data configuration)
Next, storage contents of various databases included in the server 103 illustrated in FIG. 1 will be described. The product master 110 and the various databases 120 to 123 are realized by, for example, a storage device such as the memory 202 and the disk 205 shown in FIG.

  FIG. 7 is an explanatory view showing an image of the entire data configuration. In FIG. 7, the product master 110 stores, for each product ID, information related to data of product name, price, type, and created product model.

  Further, the customer information data 120 stores, for each customer ID, information on the customer's name, address and the like. Multiple (1 to n) customer room data 121, multiple (1 to n) customer interior data 122, and multiple (1 to n) purchase information data 123 are associated with one customer information data 120 Are stored.

  The customer room data 121 is assigned a serial number for each customer ID, and corresponds to a plurality of rooms of the same customer, and a room name such as a living room, a bedroom, a child room, etc. is stored for each serial number. ing. By registering the room name, it becomes easy to specify the room when reading the room model. Information on data of a registered room model is stored for each serial number.

  Similar to the customer room data 121, the customer interior data 122 is assigned a serial number for each customer ID, and further, the customer room data is assigned a serial number. This makes it possible to know in which room the interior is located. In addition, information on the type of interior is registered. This is used for matching with the product of the product master 110. Furthermore, information on data of the registered interior model is stored.

  The purchase information data 123 stores a customer ID and status information (purchased product information, payment status, shipping status, etc.) related to purchase for each reception number uniquely assigned. These data structures manage the operation of the internet shopping system.

(Procedure of display processing of room model, interior model, product model)
Next, a procedure of display processing of a room model, an interior model, and a product model will be described with reference to FIG.

  FIG. 8 is a flow chart showing an example of the procedure for displaying a room model, an interior model and a product model. In the flowchart of FIG. 8, first, it is determined whether or not there is an instruction to display a room space (step S801). The display instruction of the room space is performed by the input from the input device 306 of the customer terminal device 101 by the customer. Specifically, a display instruction is issued by the customer launching an internet shopping site or pressing a display instruction button of a desired room space.

  In step S801, the process waits for an instruction to display a room space (step S801: No). Then, when there is a display instruction of the room space (step S801: Yes), the room space is displayed, for example, by perspective view (step S802), and the interior is arranged and displayed at a predetermined position of the room ( Step S803). This reproduces the space of the customer's room at the time of registration.

  Next, it is determined whether there is an instruction to change the viewpoint of the room space (step S804). The change instruction of the viewpoint is performed by the input from the input device 306 of the customer terminal device 101 by the customer. Specifically, the change instruction is given by the operation of the pointing device by the customer, the predetermined operation such as the direction key of the keyboard, the operation of the touch panel, or the like. Further, the change instruction may be made by tilting or changing the direction of the tablet or the smartphone.

  In step S804, when there is no view change instruction (step S804: No), the process proceeds to step S806 without doing anything. On the other hand, when there is a viewpoint change instruction (step S804: Yes), the viewpoint of the room space is changed and displayed (step S805).

  Specifically, the change display of the view point of the room space follows the drag by specifying an arbitrary position of the displayed room space by a pointing device or the like and dragging from there in a desired direction. The viewpoint of the room space may be changed. By changing the viewpoint of the room space, the room space appears to rotate around a certain viewpoint. Then, releasing the drag stops changing the viewpoint. It is preferable that the aspect of changing the viewpoint be different depending on the difference in the position for specifying the displayed room space and the difference in the dragging direction.

  By doing this, the customer can senseily move the display screen to view the room space from the desired viewpoint. The same applies to the case of operating the touch panel with a finger or the like and the case of using the direction key of the keyboard.

  Further, the viewpoint may be continuously changed in a predetermined direction at a predetermined speed. By doing this, it is possible to display the room space like a moving image taken while rotating the video camera. This allows the customer to view the display screen as a more realistic room space.

  In this way, after changing and displaying the viewpoint of the room space, the process proceeds to step S806.

  Next, it is determined whether there is an instruction to change the layout position of the interior (step S806). An instruction to change the arrangement position of the interior is given by an input from the input device 306 of the customer terminal device 101 by the customer. Specifically, the change instruction is given by the operation of the pointing device by the customer, the predetermined key operation such as the direction key of the keyboard, the operation of the touch panel, or the like.

  In step S806, when there is no change instruction of the arrangement position (step S806: No), the process proceeds to step S808 without doing anything. On the other hand, when there is a change instruction of the arrangement position (step S806: Yes), the arrangement position of the interior is changed and displayed (step S807).

  Specifically, the display for changing the arrangement position of the interior is such that the arbitrary position of the displayed interior is specified by a pointing device or the like, and dragging from there to a desired direction follows the drag to arrange the interior You may make it change the direction in which the position and interior are arranged. By changing the arrangement position and arrangement direction of the interior, it looks as if the interior was moved in the space of the room. And, by releasing the drag, the change of the arrangement position and the arrangement direction of the interior is stopped.

  It is preferable to make the mode of change different by moving the interior of the room in parallel or rotating the space according to the difference in the position where the displayed interior is specified and the difference in the dragging direction. Specifically, when the center of the interior is specified and dragged, the interior moves in parallel, and when other than the center of the interior is specified and dragged, the interior may be rotated.

  By doing this, the customer can sensibly change the display of the display screen on the display screen to place the interior at a desired position in the space of the room. The same applies to the case of operating the touch panel with a finger or the like and the case of using the direction key of the keyboard.

  Thus, after changing and displaying the viewpoint of the room space, the process proceeds to step S808.

  Next, it is determined whether or not there is an instruction to hide the interior (step S808). An instruction to hide the interior is given by an input from the input device 306 of the customer terminal device 101 by the customer. Specifically, the non-display instruction is given by the operation of the pointing device by the customer, the operation of a predetermined key of the keyboard, the operation of the touch panel, or the like.

  In step S806, when there is no non-display instruction (step S808: No), the process proceeds to step S810 without doing anything. On the other hand, when there is a non-display instruction (step S808: Yes), the interior is hidden (step S809). Thereafter, the process proceeds to step S810.

  Next, it is determined whether or not there is an instruction to display the interior that has been hidden in step S809 or has been hidden in step S814 described later (step S810). The instruction to display the non-displayed interior is given by the input from the input device 306 of the customer terminal device 101 by the customer.

  When there are a plurality of non-displayed interiors, although illustration is omitted, a list of non-displayed interiors may be displayed, and an interior to be displayed may be designated from among the interiors to give a display instruction. Specifically, a display instruction is given by an operation of a pointing device by a customer, an operation of a predetermined key of a keyboard, an operation of a touch panel, or the like.

  If there is no display instruction in step S810 (step S810: No), the process proceeds to step S812 without doing anything. On the other hand, when there is a display instruction (step S810: Yes), the hidden interior is displayed (step S811). When displaying the interior hidden in step S809, the interior may be displayed at the position where the interior was hidden.

  Further, when displaying the interior that has been hidden in step S814, the product displayed when the interior is hidden may be displayed after being hidden. In that case, the item may be displayed at the position where the product is displayed (the position after the change if the arrangement position is changed), and the position where it was arranged when it was not displayed. Alternatively, it may be displayed at the position indicated by the captured image.

  Also, the product may be displayed simultaneously with the product without hiding the product. In that case, the arrangement positions may be adjusted so as not to overlap each other.

  After displaying the hidden interior in this manner, the process moves to step S812.

  Next, it is determined whether or not there is a product display instruction (step S812). The display instruction of the product is performed by the input from the input device 306 of the customer terminal device 101 by the customer. Specifically, a display instruction is given by an operation of a pointing device by a customer, an operation of a predetermined key of a keyboard, an operation of a touch panel, or the like.

  In step S812, when there is no display instruction (step S812: No), the process proceeds to step S816 without doing anything. On the other hand, if a display instruction has been issued (step S812: YES), it is determined whether or not the interior of the same type is displayed (step S813). In addition to the case of the same type, whether it is a similar type or not, it is determined whether or not there is a target interior using parameters such as dimensions, shapes, and uses. This determination may be performed by, for example, deep learning (deep learning) using AI (Artificial Intelligence).

  In step S813, when the interior of the same type is not displayed (step S813: No), the process proceeds to step S815, and the product for which the display instruction is given is displayed (step S815).

  Thereafter, the process proceeds to step S816. On the other hand, when the interior of the same type is displayed in step S813 (step S813: Yes), the interior of the same type is hidden (step S814), and the product for which the display instruction is given is displayed (step S814) S815). At that time, the product to be displayed may be displayed at the position where the hidden interior is arranged, or may be arranged at another position. When arranging at another position, it is preferable to adjust the arrangement position so as not to overlap each other.

  After displaying the display of the product in this manner, the process proceeds to step S816.

  Next, it is determined whether or not there is an instruction to change the arrangement position of the product (step S816). The instruction to change the arrangement position of the product is given by the input from the input device 306 of the customer terminal device 101 by the customer. Specifically, the change instruction is given by the operation of the pointing device by the customer, the predetermined key operation such as the direction key of the keyboard, the operation of the touch panel, or the like.

  In step S816, when there is no change instruction of the arrangement position (step S816: No), the process proceeds to step S818 without doing anything. On the other hand, when there is a change instruction of the arrangement position (step S816: Yes), the arrangement position of the product is changed and displayed (step S817).

  Similarly to the instruction to change the arrangement position of the interior, the change indication of the arrangement position of the product specifically designates an arbitrary position of the displayed product with a pointing device or the like, and drags from there in a desired direction Thus, the position where the product is placed or the direction in which the product is placed may be changed following the drag. By changing the arrangement position and arrangement direction of the product, it looks as if the product has been moved in the space of the room. Then, by releasing the drag, the change of the arrangement position and the arrangement direction of the product is stopped.

  In addition, it is preferable that the mode of change be different by translating or rotating the space of the room depending on the difference in the position where the displayed product is specified and the difference in the dragging direction. Specifically, when the vicinity of the center of the product is specified and dragged, the product moves in parallel, and when the region other than the center of the product is specified and dragged, the product may rotate.

  By doing this, the customer can sensibly change the display of the display screen on the display screen to place the product at a desired position in the space of the room. The same applies to the case of operating the touch panel with a finger or the like and the case of using the direction key of the keyboard.

  Thus, after changing and displaying the arrangement position of goods, it transfers to step S818.

  Next, it is determined whether or not there is an instruction to hide the product (step S818). Here, when the non-display instruction has not been issued (step S 818: No), the process returns to step S 804 without doing anything. On the other hand, when there is a non-display instruction (step S 818: Yes), the product is not displayed (step S 819).

  At this time, if there is an interior that has been hidden in step S814, the interior may be displayed again. The position at which the interior is redisplayed may be the position at which the product to be hidden is located, or may be another position.

  In this manner, after the product is hidden, the process returns to step S804. Thereafter, steps S804 to S819 are repeatedly executed.

  Regarding such display processing, as shown in FIG. 4, an instruction from the customer terminal device 101 may be received, the server 103 may execute each processing, and the processing result may be transmitted to the customer terminal device 101. Although not shown, the customer terminal device 101 receives necessary information including each model information from the server 103, and the customer terminal device 101 installs in advance application programs and the like necessary for display processing, and then the customer terminal Each processing may be executed in the device 101, and the processing result may be displayed on the display 304 or the like of the customer terminal device 101.

(An example of a display screen of the customer terminal device 101)
9 to 12 are explanatory diagrams showing display examples of the display screen of the customer terminal device. In FIG. 9, on the display screen 900, the screen of the online shopping site is displayed. On the right side screen 901 of the display screen 900, the room of the customer is displayed by perspective based on the room model (step S802 in FIG. 8). By displaying in this way, the customer can grasp the space of the room in three dimensions.

  In the right side screen 901 of FIG. 9, a sofa 911 which is an interior, a sofa table 912, and an auxiliary table 913 attached to the sofa table 912 are displayed in the room of the customer (step S803 in FIG. 8). The interiors 911, 912, and 913 are shown in the captured image, and the arrangement of them is as shown in the captured image.

  Further, in the right side screen 901 of FIG. 9, in the room of the customer, there is a door 914 on the wall on the left side and a window 915 on the wall on the right side. The door 914 and the window 915 are members whose dimensions can be identified. The door 914 and the window 915 are also shown in the captured image, and their arrangement is also the same as the captured image.

  The left side screen 902 of the display screen 900 of FIG. 9 displays product information as a search result searched by the customer. Product information includes product image, product name, price and other product related information, "Purchase" button 903, "Return" button 904 for stopping the product and switching to another product, and input field 905 for inputting a quantity. Etc are displayed.

  The right side screen 1001 of FIG. 10A shows a state in which the interior is hidden in FIG. 9 (step S809 of FIG. 8). Since the interior model is stored separately from the room model, it can be hidden individually for each interior. As a result, the sofa 911 which is the interior, the sofa table 912, and the auxiliary table 913 are hidden.

  Also, in the room space, the part hidden in the interior is corrected based on the surrounding image. This enables the customer to intuitively and easily grasp the state of the room where the interior is not placed. The customer can confirm the feeling that "the room where the interior (furniture, etc.) has run out was unexpectedly large" without actually carrying out the interior. By giving such a sense to the customer, it is possible to motivate the purchase of a new interior.

  The right side screen 1002 in FIG. 10B shows a state in which the viewpoint is changed and displayed in FIG. 10A (step S805 in FIG. 8). Specifically, the right side screen 1002 is a view seen from the direction of the arrow A of 1001 in FIG. 10A. As described above, the viewpoint can be displayed changeably by changing the vanishing point position (or the number of vanishing points) of the screen displayed by perspective view.

  In FIG. 10B, the viewpoint is changed and displayed in a state in which the interior is hidden, but the viewpoint can also be changed in a state in which the interior (or a product) is displayed. In that case, the interior (or product) also changes following the change in the display of the room.

  FIG. 11 shows a state in which the product on the left side screen 902 is arranged and displayed on the right side screen 1101 (step S815 in FIG. 8). The sofa table 912 and the auxiliary table 913 displayed in FIG. 9 are not displayed, and the sofa table 1111 of the purchase candidate product is displayed at the position where the sofa table 912 was displayed. As described above, since the types of the sofa table 912 and the product sofa table 1111 are the same as the “sofa table”, the sofa table 912 and the auxiliary table 913 are not displayed. The auxiliary table 913 can be set not to be hidden by setting another type.

  By doing this, when displaying the sofa table 1111 of the product that is the purchase candidate, the customer can save time and effort to hide the sofa table 912 first, and can perform efficient product selection. Also, when returning to the original sofa table 912, the sofa table 1111 can be automatically hidden. The sofa table 1111 can also be automatically hidden when displaying a sofa table of another product (not shown).

  The right side screen 1201 of FIG. 12 shows a state in which only the arrangement position of the interior is changed and displayed with the viewpoint of the room unchanged in FIG. 11 (steps S807 and S817 of FIG. 8). It can be seen that the installation direction of the sofa 911 and the sofa table 1111 of the product is changed in the right screen 1201 compared to the right screen 1101. Thus, the layout can be easily changed by changing the arrangement position of only the interior and the product. There is no need to move and check the actual interior. And in the position so far, to make it possible to purchase new interiors by making it easy to confirm that furniture with dimensions that could not be placed can be placed by changing the placement position and orientation. Can.

  The change of the arrangement position of the interior and the product may be individually designated for the interior and the product and may be separately changed. Alternatively, a plurality of groups may be designated and changed in designated groups.

  In this way, it is possible to easily grasp whether the size of the product (for example, furniture) is suitable as compared to the size of the room, and furthermore, whether or not the product is suitable for the atmosphere of the room .

  This makes it possible to improve the purchase intention of the online shopping user. Also, customer room data and customer interior data can be used as marketing and product development information.

  In addition, although the example of the display by perspective drawing was shown in FIGS. 9-12, it is not limited to this example. For example, by using 3D displayable displays and special goggles, and by using mount head displays to show different images on the left and right eyes, virtual reality image display that actually looks three-dimensional can be realized. You can also

  As described above, in the embodiment, the model of the interior and the model of the room included in the plurality of captured images are generated based on the plurality of captured images obtained by capturing the inside of the room, and the model of the product is stored. An operation of placing a model of a specific selected product with reference to a storage unit in the model of the room, and individually moving or rotating the model of the interior and the model of the product Accept the operation to hide.

  By this, it is possible to confirm a room from a desired viewpoint of a user (for example, a customer), and therefore, it is possible to easily and surely confirm the interior and its interior without having to know the dimensions of the room, the interior and the product. It is possible to know the size or layout of the room to be placed.

  Further, based on a plurality of captured images, the dimensions of the room or interior shown in the captured image and the position in the interior of the interior can be displayed so as to be able to change the viewpoint. Therefore, the user only needs to provide a plurality of captured images of a desired room, and can easily grasp the dimensions of the desired room or the situation of the layout.

  As mentioned above, although the example of the internet shopping system of interior was explained, the model placement program, model placement method, and model placement device concerning the present invention are not limited to the application to the internet shopping system of interior. For example, the internet shopping system can be applied to the clothing field as well. Specifically, for example, virtual fitting can be realized by using a model placement program, a model placement method, and a model placement apparatus according to the present invention.

  Further, the model placement program, the model placement method, and the model placement apparatus according to the present invention are not limited to the application to the Internet shopping system. For example, a real estate agent generates a room model of a property to be sold or rented, and a customer who is considering purchasing or moving generates an interior model and displays the view point on the display screen so as to be changeable. In some cases, it can also be used to consider whether the interior you have can fit in the room of the property. As a result, it is possible to improve the search efficiency and the introduction efficiency as a function of searching for real estate properties.

  In addition, it can be applied to simulation such as remodeling of the house currently living (placement of the interior). Also, it can be applied to simulation of interior and coordination using a real room, and simulation games using a real room.

  The model layout method described in the present embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. The model placement program is a computer readable recording medium such as a hard disk, flexible disk, CD (Compact Disc) -ROM, MO (Magneto-Optical Disk), DVD (Digital Versatile Disk), USB (Universal Serial Bus) memory, etc. It is recorded and executed by being read from a recording medium by a computer. The model placement program may also be distributed via a network such as the Internet.

  The following appendices will be further disclosed regarding the embodiment described above.

(Supplementary Note 1) A model of an interior and a model of a room included in the plurality of captured images are generated based on a plurality of captured images obtained by capturing the inside of a room,
Referring to a storage unit storing a model of a product, and arranging a model of a selected specific product in the model of the room;
An operation to move individually or an operation to rotate or an operation to hide the model individually for each of the model of the interior and the model of the product
A model placement program that causes a computer to execute a process.

(Supplementary Note 2) The model of the interior is disposed in the room model based on the plurality of captured images.
The model placement program according to appendix 1, characterized in that

(Supplementary Note 3) In the arrangement of the model of the specific product, the model of the interior of the type corresponding to the type of the specific product among the models of the interior is not displayed.
The model placement program according to appendix 1, characterized in that

(Supplementary Note 4) The arrangement of the specific product model adjusts the size of the specific product model according to the size of a window or a door included in the plurality of captured images.
The model placement program according to appendix 1, characterized in that

(Supplementary Note 5) The process of generating includes a process of generating a model based on the plurality of captured images, and separating a part corresponding to the model of the interior from the model.
The model placement program according to appendix 1, characterized in that

(Supplementary Note 6) Based on a model of the room, the room is displayed so that the viewpoint can be changed.
The model placement program according to appendix 1, characterized in that

(Supplementary Note 7) An operation of changing the viewpoint is accepted.
The model arrangement | positioning program of Additional remarks characterized by the above-mentioned.

(Supplementary Note 8) The model of the product is generated based on a plurality of captured images obtained by capturing the product.
The model placement program according to appendix 1, characterized in that

(Supplementary Note 9) A model of an interior and a model of a room included in the plurality of captured images are generated based on a plurality of captured images obtained by capturing the inside of a room,
Referring to a storage unit storing a model of a product, and arranging a model of a selected specific product in the model of the room;
An operation to move individually or an operation to rotate or an operation to hide the model individually for each of the model of the interior and the model of the product
A model placement method characterized in that a computer executes processing.

(Supplementary Note 10) A model of an interior and a model of a room included in the plurality of captured images are generated based on a plurality of captured images obtained by capturing the inside of a room,
Referring to a storage unit storing a model of a product, and arranging a model of a selected specific product in the model of the room;
An operation to move individually or an operation to rotate or an operation to hide the model individually for each of the model of the interior and the model of the product
A model placement device comprising a control unit.

(Supplementary note 11) An online shopping system characterized in that any one of the model placement programs of supplementary notes 1 to 8 is executed.

101 customer terminal device 102 business terminal device 103 server 110 product master 120 customer information data 121 customer room data 122 customer interior data 123 purchase information data 411 product image file 412 product model generation unit 421 room image file 422 room model / interior model generation Part 423 Customer information registration part 431 Room model / interior model reading part 432 Display processing part 433 Product search part 434 Product placement processing part 435 Purchase processing part 911 Sofa (interior data)
912 Sofa table (Interior data)
914 Doors (members whose dimensions can be identified)
915 Window (members whose dimensions can be identified)
1111 Sofa table (product data)

Claims (9)

The model of the interior and the model of the room included in the plurality of captured images are generated based on the plurality of captured images obtained by capturing the inside of the room,
Referring to a storage unit storing a model of a product, and arranging a model of a selected specific product in the model of the room;
An operation to move individually or an operation to rotate or an operation to hide the model individually for each of the model of the interior and the model of the product
A model placement program that causes a computer to execute a process. The model of the interior is disposed in the room model based on the plurality of captured images.
The model placement program according to claim 1, characterized in that: In the arrangement of the model of the specific product, hide the model of the interior of the type corresponding to the type of the specific product among the models of the interior,
The model placement program according to claim 1, characterized in that: The arrangement of the specific product model adjusts the size of the specific product model according to the size of a window or a door included in the plurality of captured images.
The model placement program according to claim 1, characterized in that: The process of generating includes a process of generating a model based on the plurality of captured images and separating a portion corresponding to the model of the interior from the model.
The model placement program according to claim 1, characterized in that: Display the room changeable based on the model of the room,
The model placement program according to claim 1, characterized in that: The model of the product is generated based on a plurality of captured images obtained by capturing the product.
The model placement program according to claim 1, characterized in that: The model of the interior and the model of the room included in the plurality of captured images are generated based on the plurality of captured images obtained by capturing the inside of the room,
Referring to a storage unit storing a model of a product, and arranging a model of a selected specific product in the model of the room;
An operation to move individually or an operation to rotate or an operation to hide the model individually for each of the model of the interior and the model of the product
A model placement method characterized in that a computer executes processing. The model of the interior and the model of the room included in the plurality of captured images are generated based on the plurality of captured images obtained by capturing the inside of the room,
Referring to a storage unit storing a model of a product, and arranging a model of a selected specific product in the model of the room;
An operation to move individually or an operation to rotate or an operation to hide the model individually for each of the model of the interior and the model of the product
A model placement device comprising a control unit.

Images