JP2022181153A - Information processing system, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of highly precisely superimposedly displaying digital information of furniture, etc. on a prescribed space area such as a room by using augmented reality (AR).

SOLUTION: According to one embodiment of the present invention, an information processing system is provided. The information processing system is configured to execute the following steps. In a reception step, the information processing system receives first image data obtained by capturing an image of a real space. In an acquisition step, the information processing system acquires outline data relating to an outline of the real space. In a positioning step, the information processing system associates a coordinate system of the outline data and a coordinate data of the real space included in the first image data.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an information processing system, an information processing method, and a program.

In recent years, there has been disclosed a technology capable of creating and displaying a CG image of a building in real time based on CAD data (Patent Document 1).

JP-A-7-264545

However, when a customer actually visits a building for the purpose of viewing, etc., an image based on the actual state that the customer actually sees is used rather than the reproduction of the real world by the CG image of the conventional technology. There is a need for a technology that supports the inflation of the

In view of the above circumstances, the present invention uses augmented reality (AR) to make it possible to superimpose and display digital information of articles such as furniture with higher accuracy in a certain space area such as a room. We decided to provide the technology to

According to one aspect of the invention, an information processing system is provided. This information processing system is configured to perform the following steps. The receiving step receives first image data obtained by imaging the real space. The acquisition step acquires outline data relating to the outline of the real space. The alignment step associates the coordinate system of the outline data with the coordinate system of the real space included in the first image data.

According to the present disclosure, using augmented reality (AR), a technology that allows digital information of articles such as furniture to be superimposed and displayed with higher accuracy in a certain space area such as a room can be provided.

1 is a configuration diagram showing an information processing system 1 according to this embodiment; FIG. 2 is a block diagram showing the hardware configuration of the augmented reality presentation device 2; FIG. FIG. 11 is a block diagram showing functions realized by a control unit 23 and the like in an augmented reality presentation device 2 according to a third embodiment; 3 is a block diagram showing the hardware configuration of the information processing device 3; FIG. 3 is a block diagram showing functions realized by a control unit 33 and the like in the information processing device 3 according to the first embodiment and the second embodiment; FIG. It is a screen displayed on the host terminal 4 and is an example of the outline generation screen 5 . It is a screen displayed on the augmented reality presentation device 2 and is an example of the layout input screen 6 . It is a screen displayed on the augmented reality presentation device 2 and is an example of the layout input screen 6 . It is an example of the coordinate system CG of outline data. It is a part of the screen displayed on the augmented reality presentation device 2 and a part of the anchor setting screen. It is a part of the screen displayed on the augmented reality presentation device 2 and is part of the outline setting screen. It is an example of the coordinate system CR of the real space RS included in the first image data. It is part of the screen displayed on the augmented reality presentation device 2 and part of the object display screen. 3 is an activity diagram showing an example of the flow of information processing executed by the information processing system 1; FIG.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments shown below can be combined with each other.

By the way, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium (Non-Transitory Computer-Readable Medium), or may be downloaded from an external server. It may be provided as possible, or may be provided so that the program is activated on an external computer and the function is realized on the client terminal (so-called cloud computing).

Further, in the present embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by circuits in a broad sense and software information processing that can be specifically realized by these hardware resources. . In addition, various information is handled in the present embodiment, and these information include, for example, physical values of signal values representing voltage and current, and signal values as binary bit aggregates composed of 0 or 1. Represented by high and low, or quantum superposition (so-called qubits), communication and operations can be performed on circuits in a broad sense.

A circuit in a broad sense is a circuit realized by at least appropriately combining circuits, circuits, processors, memories, and the like. Application Specific Integrated Circuits (ASICs), programmable logic devices (e.g., Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and field It includes a programmable gate array (Field Programmable Gate Array: FPGA).

1. Hardware Configuration This section describes the hardware configuration of the first embodiment (hereinafter referred to as the present embodiment).

1.1 Information processing system 1
FIG. 1 is a configuration diagram showing an information processing system 1 according to this embodiment. The information processing system 1 includes an augmented reality presentation device 2, an information processing device 3, and a host terminal 4, which are connected via a network. These components are further described. Here, the system exemplified by the information processing system 1 consists of one or more devices or components. Therefore, even the single information processing device 3 serves as an example of a system.

1.2 Augmented reality presentation device 2
The type of the augmented reality presentation device 2 is not limited as long as it is a device capable of realizing Augmented Reality (AR). Augmented reality refers to technology that uses information processing equipment to add information to the real environment, such as actual scenery and topography. The augmented reality presentation device 2 may be a dedicated device equipped with AR, a personal computer, a mobile information terminal such as a smartphone or a tablet-type information terminal, and is configured to be worn by the user so as to cover the eyes. may be For example, it may be an AR wearable terminal. AR wearable terminals include, for example, glasses-type AR wearable terminals, head-mounted displays, watch-type wearable terminals, and the like.

FIG. 2 is a block diagram showing the hardware configuration of the augmented reality presentation device 2. As shown in FIG. The augmented reality presentation device 2 has a communication unit 21, a storage unit 22, a control unit 23, a display unit 24, an input unit 25, and an imaging unit 26, and these components constitute the augmented reality presentation device 2. are electrically connected via a communication bus 20 inside the . Descriptions of the communication unit 21, the storage unit 22, and the control unit 23 are omitted because they are substantially the same as those of the communication unit 31, the storage unit 32, and the control unit 33 in the information processing device 3. FIG.

The display unit 24 may be included in, for example, the augmented reality presentation device 2 or may be externally attached. When the augmented reality presentation device 2 is AR glasses, the display unit 24 is the graphic display portion of the AR glasses. The user can not only perceive the real world through the graphic display, but also perceive augmented reality by looking at characters and images displayed on the graphic display. The graphic display (display unit 24) of the AR glasses (augmented reality presentation device 2) includes, but is not limited to, a non-transmissive binocular type, a monocular type, and a transmissive binocular type.

If the augmented reality presentation device 2 is not AR glasses, the display unit 24 is a display device such as a CRT display, a liquid crystal display, an organic EL display, a plasma display, or the like, attached to or built into the augmented reality presentation device 2. It is preferable to selectively use these according to the type of the augmented reality presentation device 2 . Moreover, the specifications of the display unit 24, such as dimensions, resolution, and dot pitch, are not limited.

The input unit 25 receives various instructions and information input from the user. The input is transferred as a command signal to the control section 23 via the communication bus 20, and the control section 23 can execute predetermined control and calculation as necessary. The input unit 25 may be included in the housing of the augmented reality presentation device 2, or may be externally attached. For example, the input unit 25 may be integrated with the display unit 24 and implemented as a touch panel. With a touch panel, the user can input a tap operation, a swipe operation, or the like. Of course, a switch button, a mouse, a QWERTY keyboard, or the like may be employed instead of the touch panel.

The input unit 25 may use a graphical user interface (GUI) or a natural user interface. In the case of using the natural user interface, the user can send various instructions and information to the AR glasses (augmented reality presentation device 2) with natural actions that are normally performed. It should be noted that the motion target that serves as a natural user interface is not limited to hands or fingers, and can be all parts of the body such as the head, neck, and waist.

Also, the input unit 25 may be a microphone. The microphone (input unit 25) is built in or externally attached to the augmented reality presentation device 2, and is an acoustic device that converts sound into an electric signal. Specifically, the microphone (input unit 25) converts air vibrations into mechanical vibrations and converts them into electrical signals. The converted electrical signal may be a digital electrical signal or an analog electrical signal, and is temporarily stored in the storage section 22 as a digital electrical signal via the communication section 21 .

The imaging unit 26 is a so-called vision sensor (camera) configured to be capable of imaging information of the outside world. The imaging unit 26 is configured to generate image data by photographing an object to be photographed. The imaging unit 26 is connected to a communication unit 31 of the information processing device 3 to be described later via a network, and is configured to be capable of transferring captured image data to the information processing device 3 .

1.3 Information processing device 3
FIG. 4 is a block diagram showing the hardware configuration of the information processing device 3. As shown in FIG. The information processing device 3 includes a communication unit 31, a storage unit 32, and a control unit 33, and these components are electrically connected via a communication bus 30 inside the information processing device 3. . Each component will be further described.

The communication unit 31 preferably uses wired communication means such as USB, IEEE1394, Thunderbolt, wired LAN network communication, etc., but wireless LAN network communication, mobile communication such as 3G/LTE/5G, Bluetooth (registered trademark) communication, etc. is required. may be included depending on That is, it is more preferable that the communication unit 31 be implemented as a set of these communication means. That is, the information processing device 3 communicates various information with the augmented reality presentation device 2 and the host terminal 4 via the network via the communication unit 31 .

The storage unit 32 stores various information defined by the above description. The storage unit 32 is, for example, a storage device such as a solid state drive (SSD) that stores various programs related to the information processing device 3 executed by the control unit 33, or a storage device related to program calculation. It can be implemented as a memory such as a Random Access Memory (RAM) that temporarily stores necessary information (arguments, arrays, etc.). Moreover, the memory|storage part 32 may be these combinations. In particular, the storage unit 32 stores drawing data and position coordinate data, which are object information about the object O, outline data and anchor information about the outline GD of the real space RS, programs for generating these, and the like. The storage unit 32 also stores various programs related to the information processing device 3 that are executed by the control unit 33 .

The control unit 33 processes and controls overall operations related to the information processing device 3 . The control unit 33 is, for example, a central processing unit (CPU) (not shown). The control unit 33 implements various functions related to the information processing device 3 by reading a predetermined program stored in the storage unit 32 . That is, various functions related to the information processing device 3 are implemented by the control unit 33, which is an example of hardware, by specifically realizing information processing by software stored in the storage unit 32. It can be implemented as each function included. Each functional unit will be further detailed in the next section. Note that the control unit 33 is not limited to being single, and may be implemented so as to have a plurality of control units 33 for each function. Also, the controller 33 may be a combination of a plurality of controllers.

1.4 Host terminal 4
A description of the host terminal 4 is omitted because it is substantially the same as the augmented reality presentation device 2 described above. Note that the host terminal 4 may not be a device capable of realizing AR, but is preferably a device capable of realizing AR.

2.1 Functional configuration (augmented reality presentation device 2)
This section describes the functional configuration of the augmented reality presentation device 2 according to the third embodiment. As described above, various functions related to the augmented reality presentation device 2 are realized by the control unit 23, which is an example of hardware, by specifically realizing information processing by software stored in the storage unit 22. 23 can be implemented as each functional unit included in .

FIG. 3 is a block diagram showing functions realized by the control unit 23 and the like in the augmented reality presentation device 2 according to the third embodiment. Specifically, the augmented reality presentation device 2 includes an acquisition unit 231 , a management unit 232 and a display processing unit 233 .

The acquisition unit 231 is configured to acquire various information. For example, the acquisition unit 231 may acquire the arrangement information from the information processing device 3 via the network. Details will be described later.

The management unit 232 is configured to manage various information. For example, the management unit 232 may manage drawing data regarding a plurality of objects O that can be arranged in a room. Details will be described later.

The display processing unit 233 is configured to execute various display processes. For example, the display processing unit 233 may superimpose the object O, which is virtual information, on the room, which is the real space RS. Details will be described later.

2.2 Functional configuration (information processing device 3)
In this section, the functional configuration of the information processing device 3 according to the first embodiment and the second embodiment will be described. As described above, various functions of the information processing apparatus 3 are realized by the control unit 33, which is an example of hardware, by specifically realizing information processing by software stored in the storage unit 32. can be executed as each function unit included in the

FIG. 5 is a block diagram showing functions realized by the control unit 33 and the like in the information processing device 3 according to the first embodiment and the second embodiment. Specifically, the information processing device 3 as an example of the information processing system 1 includes a reception unit 331 , a screen display unit 332 , an outline generation unit 333 , an acquisition unit 334 and an alignment unit 335 .

The reception unit 331 is configured to receive various information. For example, the reception unit 331 may receive first image data obtained by imaging the real space RS as the reception step. Details will be described later.

The screen display unit 332 is configured to control to display various screens. The screen display unit 332, which is an example of the outline generation screen display unit, controls to display the outline generation screen 5 (see FIG. 6) as the outline generation screen display step. A screen display unit 332, which is an example of an anchor setting screen display unit, controls to display an anchor setting screen (see FIG. 10) as an anchor setting screen display step. The screen display unit 332, which is an example of the layout input screen display unit, controls to display the layout input screen 6 (see FIGS. 7 and 8) as the layout input screen display step. A screen display unit 332, which is an example of an object display screen, controls to display an object display screen (see FIG. 13) as an object display screen display step. Details will be described later.

The outline generator 333 is configured to generate outline data from the second image data. Details will be described later.

Acquisition unit 334 is configured to acquire various types of information. For example, the acquisition unit 334 may acquire outline data, anchor information, and object information. Details will be described later.

The alignment unit 335 is configured to associate the coordinate system CG of the outline data with the coordinate system CR of the real space RS included in the first image data. Details will be described later.

Management unit 336 is configured to manage various information. For example, the management unit 336 may manage outline data. Details will be described later.

3. First Embodiment In this section, an information processing method of the information processing system 1 according to the first embodiment will be described. FIG. 6 is a screen displayed on the host terminal 4, which is an example of the outline generation screen 5. As shown in FIG. FIG. 7 is a screen displayed on the augmented reality presentation device 2 and is an example of the layout input screen 6. As shown in FIG. FIG. 8 is a screen displayed on the augmented reality presentation device 2 and is an example of the layout input screen 6 . FIG. 9 is an example of the coordinate system CG of outline data. FIG. 10 is a part of the screen displayed on the augmented reality presentation device 2, which is part of the anchor setting screen. FIG. 11 is a part of the screen displayed on the augmented reality presentation device 2, which is part of the outline setting screen. FIG. 12 is an example of the coordinate system CR of the real space RS included in the first image data. FIG. 13 is a part of the screen displayed on the augmented reality presentation device 2, which is part of the object display screen. FIG. 14 is an activity diagram showing an example of the flow of information processing executed by the information processing system 1. As shown in FIG.

Here, the user is a person who uses the augmented reality presentation device 2 and who desires an AR experience targeting a certain spatial area such as a room. Also, a host is a person who uses the host terminal 4 and uses a service proposed by the information processing system 1 to provide the user with an AR experience. Note that the user and the host may be the same person, and in this case, the person uses the augmented reality presentation device 2 .

The real space RS refers to a certain spatial area such as a room for AR experience. Preferably, the real space RS is the interior space of a building. According to such a mode, it is possible to embody the image of the case where the installation object is arranged in a place such as a factory or an indoor exhibition hall by using AR. More preferably, the real space RS is a room. According to such a mode, it is possible to materialize an image when furniture and the like are arranged in a room or the like of a real estate property by using AR.

Also, an article that can be placed in the real space RS is called an object O. As shown in FIG. When the real space RS is a room, the objects O include, for example, storage furniture, shelves, chests of drawers, chairs, bedding, furniture such as desks, lighting fixtures such as ceiling lights and floor lights, air purifiers, fans, Examples include air conditioners, household appliances such as clocks, foliage plants, floor mats such as carpets, rugs, and mats, curtains, blinds, roll screens, printers, multifunction machines, office equipment such as shredders, and partitions.

The flow of information processing executed by the information processing system 1 will be described below with reference to the activity diagram of FIG. 13 .

First, the host uses the host terminal 4 to access the URL of the service proposed by the information processing system 1 using a browser. The service proposed by the information processing system 1 may be provided by a dedicated mobile application, in which case the host activates the application. Next, the host selects the function of editing and registering the outline GD of the real space RS on a menu screen (not shown) displaying a list of services (activity A001).

When the host selects the function of editing and registering the outline GD of the real space RS on a menu screen (not shown), information related to the selection is transmitted to the information processing device 3 via the network. After that, in the information processing device 3, after the communication unit 31 receives the information, the outline generation screen display unit 332 of the control unit 33 displays the outline generation screen 5 (see FIG. 6) on the display unit of the host terminal 4. Information to be displayed is transmitted to the host terminal 4 via the communication unit 31 and the network (activity A001).

That is, the outline generation screen display unit 332 controls to display the outline generation screen 5 . Here, as shown in FIG. 6, the outline generation screen 5 includes a plurality of grid points LP, and by connecting each grid point LP, outline data on the outline GD of the real space RS can be generated. configured to According to this aspect, outline data can be generated by a simple manual operation for the outline of a space such as a room having a certain shape (activity A001).

The outline generation screen 5 shown in FIG. 6 has, for example, a drawing area having a plurality of grid points LP on its right side, and a title 51 indicating that it is the outline generation screen 5 and a scale on its left side. A pull-down menu 52 for drawing, a button 53 for resetting drawing, and a button 54 for completing drawing (activity A001).

On the outline generation screen 5 of FIG. 6, the host first selects an appropriate value as the interval between lattice points LP from the pull-down menu 52 in accordance with the size of the real space RS which is grasped by measuring in advance. select. Next, the host creates a drawing line SL by moving the start point SP so as to connect the lattice points LP in the drawing area. Subsequently, the host presses the button 54 after moving the start point SP to the lattice point LP where the end point EP is placed so as to form a closed area with the drawn line SL. When the host presses the button 54 in this way, the input information by the series of operations is transmitted to the information processing device 3 via the network and received by the communication unit 31 . Next, the control unit 33 generates outline data regarding the outline GD of the real space RS based on the input information. (Activity A001).

After that, the storage unit 32 stores the outline data at least temporarily. Here, it is preferable that the management unit 336 manages the outline data for a predetermined period in association with information such as the name and location of the real space RS (activity A001).

Subsequently, the user uses the augmented reality presentation device 2 to access the URL of the service proposed by the information processing system 1 or launch a dedicated mobile application. Next, the user selects an outline data search function on a menu screen (not shown) displaying a list of services, and selects outline data from a plurality of outline data managed by the management unit 336 or stored in the storage unit 32. , the desired outline data is retrieved based on the name, location, etc. of the real space RS. For example, if the real space RS is a room, the user searches for outline data on the outline of the room by entering the building name, section number, room number, etc. (activity A002).

After selecting the outline data, the user selects the function of selecting the object O and specifying the placement location on the transitioned screen (not shown). When the function is selected by the user, information related to the selection is transmitted to the information processing device 3 via the network. After that, in the information processing device 3 , after the communication unit 31 receives the information, the layout input screen display unit 332 of the control unit 33 displays the layout input screen 6 (see FIG. 7 ) on the display unit 24 of the augmented reality presentation device 2 . to the augmented reality presentation device 2 via the communication unit 31 and the network (activity A002).

That is, the layout input screen display unit 332 controls to display the layout input screen 6 . Here, as shown in FIGS. 7 and 8, the placement input screen 6 is configured so that at least one object O can be selected from among a plurality of objects O that can be placed in the real space RS. Position coordinate data representing the position in the coordinate system of the outline data can be generated by designating the position of the object O on the area formed by the outline GD included in the outline data. According to this aspect, it is possible to select articles such as furniture to be displayed in a superimposed manner and to designate the location of arrangement by a simple manual operation (activity A002).

The layout input screen 6 shown in FIGS. 7 and 8 has, for example, a drawing area on the right side thereof in which the outline GD included in the outline data selected by the user is drawn, and a layout input screen 6 on the left side thereof. It includes a title 61 indicating that there is an existing object, an icon IO that allows the object O to be placed on the area formed by the outline GD, and an icon IA that allows the anchor A to be placed on the area formed by the outline GD. area (activity A002).

On the layout input screen 6 of FIG. 7, the user drags and drops the icon IO corresponding to the desired object O to the desired position on the area formed by the outline GD drawn in the drawing area. Next, the user adjusts the orientation of the object O by pressing the button RL or button RR displayed at the bottom of the drawing area. Note that, as shown in FIG. 7, the button RL is, for example, a button that enables the object O to rotate 45 degrees to the left, and the button RR, for example, rotates the object O to the right by 45 degrees. It is a button that allows you to Information input by the user through the series of operations is transmitted to the information processing device 3 via the network and received by the communication unit 31 . Next, based on the input information, the control unit 33 generates position coordinate data representing the position of the selected object O in the coordinate system of the outline data (activity A002).

The user similarly arranges all desired objects O at desired positions on the area formed by the outline GD. Thereby, position coordinate data is generated for all desired objects O, and the storage unit 32 stores the position coordinate data at least temporarily. 7, the user can delete the object O that has been placed once but is desired to be deleted by pressing the button DO displayed at the bottom of the drawing area after the object O is selected. . For the deleted object O, the generated position coordinate data is also deleted from the storage (activity A002).

After that, the user operates the icon IA that enables the placement of the anchor A on the area formed by the outline GD on the placement input screen 6 of FIG. Specifically, the user drags and drops the icon IA to an appropriate position on the area formed by the general shape GD drawn in the drawing area on the layout input screen 6 of FIG. Here, as shown in the layout input screen 6 of FIG. 8, when the anchor A is displayed in the drawing area by the drag-and-drop operation, the icon IA switches to a button for completing the layout input. After placing the anchor A at an appropriate position, the user presses the icon IA. By the operation by the user, a position on the area formed by the general shape GD to be associated with the anchor information is designated, and information related to the designation is transmitted to the information processing device 3 via the network and received by the communication unit 31 . Next, the control unit 33 associates the anchor information with the specified position on the area formed by the general shape GD, and then the anchor information associated with the specified position on the area formed by the general shape GD. is stored at least temporarily by the storage unit 32 (activity A003).

In other words, the layout input screen 6 is configured so that the position on the area formed by the general shape GD included in the general shape data, which is associated with the anchor information, can be designated. According to such an aspect, it is possible to designate a place suitable as an anchor by a simple manual operation. If the real space RS is a room, the place suitable for anchoring is, for example, the corner of the general shape GD corresponding to the corner of the room (activity A003).

On the layout input screen 6 of FIG. 8, when the user presses the icon IA that enables the completion of layout input, the imaging unit of the augmented reality presentation device 2 is activated, and the first image data obtained by imaging the real space RS. is transmitted to the information processing device 3 via the network. After that, in the information processing device 3, the communication unit 31 receives the first image data transmitted from the augmented reality presentation device 2, and the reception unit 331 receives the first image data obtained by imaging the real space RS ( Activity A004).

Next, the acquisition unit 334 acquires outline data on the outline GD of the real space RS from the storage unit 32 . Furthermore, the acquisition unit 334 acquires anchor information about the anchor A set for the outline data from the storage unit 32 . Preferably, the acquisition unit 334 acquires the anchor information associated with the specified position on the area formed by the outline GD from the storage unit 32 . Subsequently, based on the first image data and the anchor information, the anchor setting screen display unit 332 generates information for displaying the anchor setting screen (see FIG. 10) on the display unit 24 of the augmented reality presentation device 2. , to the augmented reality presentation device 2 via the communication unit 31 and the network (activity A005).

That is, the anchor setting screen display unit 332 controls to display the anchor setting screen. Here, as shown in FIG. 10, the anchor setting screen includes the anchor A superimposed and displayed on the first image data, and is configured so that the position of the real space RS associated with the anchor information can be designated ( Activity A005).

The anchor setting screen, a part of which is shown in FIG. 10, includes, for example, an image of the real space RS, an anchor A, and a button BA that enables specification of the position of the real space RS with which the anchor information is associated (activity A005).

After the anchor setting screen (see FIG. 10) is displayed on the display unit 24, the user moves in the real space RS while holding or wearing the augmented reality presentation device 2, thereby capturing an image of the augmented reality presentation device 2. Adjust the range to be imaged by the unit 26, and on the anchor setting screen (see FIG. 10) displayed on the display unit 24, correspond to the position on the area formed by the general shape GD where the anchor A is placed on the placement input screen 6 Anchor A is displayed at the position of the real space RS. The user presses anchor A after confirming that anchor A is displayed at an appropriate position. By the operation by the user, the position of the real space RS associated with the anchor information is designated, the information related to the designation is transmitted to the information processing device 3 via the network, and is stored at least temporarily by the storage unit 32 ( Activity A005).

Subsequently, the control unit 33 transmits information for displaying the outline setting screen (see FIG. 11) on the display unit 24 of the augmented reality presentation device 2 to the augmented reality presentation device 2 via the communication unit 31 and the network. do. Here, as shown in FIG. 11, the outline setting screen includes the image of the real space RS, the outline GD, the slider bar SB and slider SS that enable adjustment of the orientation of the outline GD, and the outline and a button BG that allows the orientation adjustment of GD to be completed. In addition, the outline setting screen (see FIG. 11) is a state in which the points on the area formed by the outline GD associated with the anchor information are fixed at the positions of the real space RS associated with the anchor information. It is constructed so that the orientation of the shape GD can be changed (activity A006).

After the outline setting screen (see FIG. 11) is displayed on the display unit 24, the user moves the slider SS left and right on the slider bar SB to display the outline setting screen (see FIG. 11). Adjust the orientation of the outline GD in which the outline GD is present, and match the outline GD with the shape of the real space RS. Note that the movement of the slider SS is performed by tilting the augmented reality presentation device 2 that the user has, or by the augmented reality presentation device 2 such as the AR glasses worn by the user, and the image of the floor of the room (real space RS) is superimposed and displayed. It becomes possible by the movement of the hand or the like with respect to the slider SS. After confirming that the outline GD matches the shape of the real space RS, the user presses the button BG. By this operation by the user, a signal requesting execution of alignment is sent to the information processing device 3 via the network (activity A006).

After that, in the information processing device 3, the communication unit 31 receives the signal requesting execution of alignment transmitted from the augmented reality presentation device 2, and the alignment unit 335 transmits the coordinate system CG of the outline data and the It is associated with the coordinate system CR of the real space RS included in one image data. Preferably, the alignment unit 335 associates the coordinate system CG of the outline data with the coordinate system CR of the real space RS by associating anchor information with the specified position of the real space RS. According to this aspect, by setting the anchor as a common origin in the coordinate system CG of the outline data and the coordinate system CR of the real space RS, the two coordinate systems can be associated with high accuracy (activity A006).

Here, the alignment performed by the alignment unit 335 will be described with a specific example with reference to FIGS. 9 and 12. FIG. As shown in FIG. 9, the coordinate system CG of outline data is, for example, an XY coordinate system having an anchor A as an origin and an x-axis XG and a y-axis YG as coordinate axes. By associating anchor information with the position of the real space RS specified in activity A005, the positioning unit 335 sets the position as the origin of the coordinate system CR of the real space RS, and sets the x Set the axis XR and the y-axis YG. The coordinate system CR of the real space RS thus obtained is the coordinate system shown in FIG. 12 (activity A006).

Furthermore, taking one point as an example, the correspondence between the coordinate system CG of outline data and the coordinate system CR of real space RS will be described. For convenience of explanation, it is assumed here that the interval between lattice points LP included in the outline data is 1 m. In the coordinate system CG of the outline data shown in FIG. 9, a point P1 is shown as one point on the area formed by the outline GD. The point P1 has an x coordinate of 1 m and a y coordinate of 1 m in the coordinate system CG of the outline data in FIG. Correspondingly, as shown in FIG. 12, the position of the point P1 in the real space RS in the coordinate system CR is 1 m, y The position is 1 m away in the direction of the YR axis (activity A006).

After the alignment is executed as described above, the acquisition unit 334 acquires object information about the object O that can be arranged in the real space RS from the storage unit 32 . Preferably, the acquisition unit 334 acquires the object information of the object O selected by the user in the activity A002 from the storage unit 32 . Here, the object information includes drawing data and position coordinate data representing the position of the outline data in the coordinate system CG. The drawing data preferably includes three-dimensional shape data of the object O, and preferably includes color, pattern and texture data. Also, the position coordinate data is preferably position coordinate data generated in activity A002 (activity A007).

Subsequently, the object display screen display unit 332 displays the object display screen (Fig. 13) is generated on the display unit 24 of the augmented reality presentation device 2, and is transmitted to the augmented reality presentation device 2 via the communication unit 31 and the network (activity A007).

That is, the object display screen display unit 332 controls to display the object display screen. Here, as shown in FIG. 13, the object display screen is displayed on the first image data based on the object information and the coordinate system CR of the real space RS associated with the coordinate system CG of the outline data. This is a screen on which an object O is superimposed. According to this aspect, it is possible to superimpose and display the digital information of an article such as furniture at a desired position with high accuracy (activity A007).

The object display screen, a part of which is shown in FIG. 13, includes, for example, an image of the real space RS, a general shape GD, and an object O that is stereoscopically displayed. It is preferable that the object display screen be configured so that the object O can be rearranged, rotated, deleted, and added. (Activity A007).

In summary, the information processing method of this embodiment includes an acquisition step, a reception step, and an alignment step. The receiving step receives first image data obtained by imaging the real space RS. The acquisition step acquires outline data on the outline GD of the real space RS. In the alignment step, the coordinate system CG of the outline data and the coordinate system CR of the real space RS included in the first image data are associated.

According to the present embodiment, it is possible to superimpose and display digital information of articles such as furniture in a certain spatial area such as a room by using augmented reality (AR). .

4. Second Embodiment In this section, an information processing apparatus 3 according to a second embodiment (hereinafter referred to as the present embodiment) will be described. However, descriptions of configurations and functions that are substantially the same as those of the information processing system 1 according to the first embodiment will be omitted.

The information processing device 3 according to this embodiment includes the following units. The reception unit 331 receives input from the first user regarding the layout information for the general shape data regarding the general shape GD of the room. The placement information includes information about the type of object O placed in the room and coordinate information about the place where the object O is placed. The screen display unit 332 transmits the layout information to the augmented reality presentation device 2 via the network, thereby displaying a screen in which the object O as virtual information is superimposed on the room as the real space RS to the augmented reality presentation device 2. display. The augmented reality presentation device 2 is the second user's terminal located in the room. The flow of information processing executed by the information processing apparatus 3 according to this embodiment will be described below.

Here, the first user is, for example, a real estate agent, and the second user is, for example, an insider. Also, the first user and the second user may be the same person.

The first user selects at least one object O from a plurality of objects O that can be placed in the room that is the real space RS on the placement input screen 6 (see FIG. 8) in the same manner as in activity A002 described above. and specify the position of the selected object O on the area formed by the general shape GD of the room. The information input by the operation by the first user is transmitted to the information processing device 3 via the network, received by the communication unit 31 , and then accepted by the reception unit 331 .

Next, the control unit 33 generates information about the type of the selected object O and coordinate information about the location of the selected object O based on the input information. That is, the control unit 33 generates arrangement information. Subsequently, the screen display unit 332 transmits the arrangement information to the augmented reality presentation device 2 possessed by the second user who is actually in the room via the network. Next, in the augmented reality presentation device 2, after the communication unit 21 receives the placement information, the control unit 23 causes the imaging unit 26 to Control is performed so that the display unit 24 displays a screen in which an object O, which is virtual information, is superimposed on the captured image of the room.

According to the present embodiment, it is possible to provide a virtual model room to a customer who has actually visited a real estate property for the purpose of previewing, etc., and to provide a simulated experience of actual life that is difficult to imagine from a conventional preview alone.

5. Third Embodiment In this section, an augmented reality presentation device 2 according to a third embodiment (hereinafter referred to as this embodiment) will be described. However, descriptions of configurations and functions substantially similar to those of the information processing system 1 according to the first embodiment and the information processing apparatus 3 according to the second embodiment will be omitted.

The augmented reality presentation device 2 according to this embodiment includes the following units. The acquisition unit 231 acquires the arrangement information from the information processing device 3 via the network. The placement information includes information about the type of object O to be placed in the room and coordinate information about the placement location of the object O, which is set for outline data about the outline GD of the room. The management unit 232 manages drawing data regarding a plurality of objects O that can be arranged in a room. The display processing unit 233 superimposes the object O, which is virtual information, on the room, which is the real space RS, based on the acquired layout information and drawing data. The flow of information processing executed by the augmented reality presentation device 2 according to this embodiment will be described below.

The layout information is generated in the control unit 33 of the information processing device 3 through an operation on the layout input screen 6 (see FIG. 8), as described in the second embodiment. The acquisition unit 231 of the augmented reality presentation device 2 acquires the arrangement information from the information processing device 3 via the network.

Next, the display processing unit 233 displays an object O as virtual information superimposed on the image of the room captured by the imaging unit 26 based on the acquired layout information and the drawing data managed by the management unit 232. is displayed on the display unit 24 .

Note that the drawing data managed by the management unit 232 preferably includes the three-dimensional shape data of the object O, and preferably includes color, pattern, and texture data, as described in the first embodiment. In addition, the management unit 232 manages the information related to the type of the object O, such as the name and product number, and the drawing data of the object O in association with each other.

According to the present embodiment, it is possible to provide a virtual model room to a customer who has actually visited a real estate property for the purpose of previewing, etc., and to provide a simulated experience of actual life that is difficult to imagine from a conventional preview alone.

5. Others Regarding the information processing system 1, the following aspects may be adopted.

An aspect of this embodiment may be a program. This program causes the computer to execute each step of the information processing system 1 .

In the activity A001 of the first embodiment, the outline generation screen 5 (see FIG. 6) is displayed on the host terminal 4, but the augmented reality presentation device 2 is caused to display the outline generation screen 5 (see FIG. 6), The user may be made to perform an operation related to the generation of the outline GD.

In the first embodiment, activity A001 is executed by the host, and activities A002 to A007 are executed by the user. A004 to activity A007 may be executed by the user using the augmented reality presentation device 2 . According to this aspect, the real estate broker who is the host or the like creates a two-dimensional model room (see FIG. 8) in advance through activities A001 to A003 so that users who come to see the room can see a virtual room. Preparations can be made in advance to experience the model room (see FIG. 13).

In the activity A001 of the first embodiment, outline data may be generated by the host or the user selecting an outline GD from basic outline GDs registered in advance. When the real space RS is a room, the basic general shape GD is, for example, a rectangle, an L shape, a T shape, a curve shape, a Z shape, a cut shape obtained by cutting corners from a rectangle, and the like. Furthermore, after selecting the basic outline GD, it is preferable to be able to adjust the outline GD on a screen such as the outline generation screen 5 (see FIG. 6) so as to better match the shape of the real space RS.

The outline generator 333 may generate outline data from the second image data from which the outline GD of the real space RS can be grasped. For example, outline data may be generated using second image data obtained by imaging the real space RS with a 360-degree camera. According to this aspect, it is possible to automatically generate outline data of a space such as a room having a certain shape. Note that the generated outline data may be three-dimensional data.

The outline data may be CAD data. When the real space RS is a room, for example, CAD data of a floor plan may be used as outline data. CAD data used as outline data may be three-dimensional CAD data, or outline data may be generated as three-dimensional data from two-dimensional CAD data.

When the outline data is three-dimensional data, in the activity A002 of the first embodiment, an input operation of arranging the three-dimensionally represented object O in the space formed by the three-dimensionally represented outline GD is performed. It is preferable that the position coordinate data of the three-dimensional coordinate system is generated for the object O by performing the above.

In the activity A002 of the first embodiment, the appropriate position on the area formed by the general shape GD included in the general shape data is automatically determined by the control unit 33 without user operation. May be specified. Here, it is preferable that the appropriate position designated for the predetermined object O is optimized according to the general shape GD by machine learning.

In the activity A003 of the first embodiment, the position on the area formed by the general shape GD included in the general shape data, which is associated with the anchor information, may be automatically specified by the control unit 33 regardless of the user's operation. good. For example, as shown in the placement input screen 6 of FIG. 8, the left corner of the general shape GD may be automatically designated as the position associated with the anchor information.

The order of the activities A002 and A003 in the first embodiment may be reversed as a flow of information processing. That is, after specifying the position on the area formed by the outline GD included in the outline data to be associated with the anchor information, the object O to be placed in the real space RS is selected, and the selected object O is placed in the outline GD. It may be possible to specify a position on the region.

The drawing data of object O may be obtained from a database stored in an external device. Here, the database stored in the external device is not limited to the database owned by the service provider, and may be an external database owned by the seller of the object O or the like.

The object display screen displayed on the display unit 24 of the augmented reality presentation device 2 in the activity A007 of the first embodiment can be displayed on the augmented reality presentation device 2 possessed by a person other than the user in the real space RS. may be For example, when the real space RS is a room, the real estate agent who is both the host and the user causes his/her augmented reality presentation device 2 to display the object display screen through activities A001 to A007, and then previews the room. The object display screen may be displayed on the augmented reality presentation device 2 possessed by the person. Here, there may be a plurality of people viewing the room. According to this aspect, the person in charge of real estate brokerage or the like uses a two-dimensional model room (see FIG. 8) created in advance through activities A001 to A003 to instantaneously give virtual A model room (see FIG. 13) can be experienced.

It may be provided in each aspect described below.
The information processing system is configured to further perform the following steps, wherein the outline generating step generates the outline data from second image data from which the outline of the real space can be grasped; thing.
The information processing system is configured to further execute the following steps, wherein the outline generation screen display step controls to display the outline generation screen, wherein the outline generation screen includes a plurality of comprising grid points and connecting the grid points to generate the outline data.
The information processing system is configured to further execute the following steps, wherein the obtaining step further obtains anchor information related to the anchor set for the outline data, and the anchor setting screen displaying step includes: controlling to display an anchor setting screen, wherein the anchor setting screen includes the anchor superimposed and displayed on the first image data and designates a position in the real space associated with the anchor information; The aligning step associates the coordinate system of the outline data with the coordinate system of the real space by associating the anchor information with the specified position in the real space.
In the information processing system, the information processing system is configured to further execute the following steps, and in the step of displaying an arrangement input screen, controlling to display an arrangement input screen, wherein the arrangement input screen displays the anchor information. A position on the area formed by the outline included in the outline data to be associated can be specified, and in the obtaining step, the anchor information associated with the specified position on the area formed by the outline to get, stuff.
The information processing system is configured to further execute the following steps, wherein the obtaining step further obtains object information about an object that can be placed in the real space, wherein the object information is drawing data. and position coordinate data representing the position of the outline data in the coordinate system. In the object display screen displaying step, the object display screen is controlled to be displayed, wherein the object display screen is the object A screen in which the object is superimposed on the first image data based on the information and the coordinate system of the real space associated with the coordinate system of the outline data.
In the information processing system, the information processing system is configured to further execute the following steps, and in a layout input screen display step, control is performed to display a layout input screen, wherein the layout input screen is laid out in the real space. at least one object can be selected from among a plurality of possible objects, and by designating the position of the selected object on the area formed by the outline included in the outline data, The position coordinate data can be generated, and the obtaining step obtains the object information of the selected object.
In the information processing system, the real space is an internal space of a building.
In the information processing system, the real space is a room.
In the information processing system, the outline data is three-dimensional data.
A method of information processing, comprising steps of the information processing system.
A program that causes a computer to perform each step of the information processing system.
An information processing apparatus configured to execute the following steps, in the receiving step, receiving from a first user an input related to layout information for outline data relating to the outline of a room, wherein the layout information is , information about the type of object to be placed in the room and coordinate information about the place where the object is placed, and in the screen display step, by transmitting the placement information to an augmented reality presentation device via a network, real space and causes the augmented reality presentation device to display a screen in which the object, which is virtual information, is superimposed in the room, wherein the augmented reality presentation device is a terminal of a second user located in the room ,thing.
An augmented reality presentation device configured to perform the following steps, wherein the acquisition step acquires layout information from an information processing device via a network, wherein the layout information relates to a general shape of a room; Drawing data about a plurality of objects that can be placed in the room, including information about the types of objects to be placed in the room and coordinate information about the places where the objects are placed, which are set for outline data; and, in a display processing step, superimposing and displaying the object, which is virtual information, in the room, which is real space, based on the obtained layout information and the drawing data.
Of course, this is not the only case.

Finally, while various embodiments of the invention have been described, these have been presented by way of example and are not intended to limit the scope of the invention. The novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

1: Information processing system 2: Augmented reality presentation device 20: Communication bus 21: Communication unit 22: Storage unit 23: Control unit 231: Acquisition unit 232: Management unit 233: Display processing unit 24: Display unit 25: Input unit 26: Imaging unit 3 : Information processing device 30 : Communication bus 31 : Communication unit 32 : Storage unit 33 : Control unit 331 : Reception unit 332 : Screen display unit 333 : Outline generation unit 334 : Acquisition unit 335 : Alignment unit 336 : Management Part 4: Host terminal 5: Outline generation screen 51: Title 52: Pull-down menu 53: Button 54: Button 6: Layout input screen 61: Title A: Anchor AR: Glasses type BG: Button CG: Coordinate system CR: Coordinate system DO : Button EP : End point GD : Outline IA : Icon IO : Icon LP : Grid point O : Object P1 : Point RL : Button RR : Button RS : Real space SB : Slider bar SL : Drawing line SP : Starting point SS : Slider XG : x-axis XR : x-axis YG : y-axis YR : y-axis

Claims (14)

An information processing system,
configured to perform each of the following steps,
The receiving step receives first image data obtained by imaging the real space,
The acquisition step acquires outline data related to the outline of the real space,
The alignment step associates the coordinate system of the outline data with the coordinate system of the real space included in the first image data.
thing. In the information processing system according to claim 1,
configured to perform the following further steps,
In the outline generating step, the outline data is generated from second image data from which the outline of the real space can be grasped.
thing. In the information processing system according to claim 1,
configured to perform the following further steps,
In the outline creation screen display step, control is performed to display an outline creation screen, wherein the outline creation screen includes a plurality of grid points, and the outline data is generated by connecting each grid point. configured to be able to generate
thing. In the information processing system according to any one of claims 1 to 3,
configured to further perform each of the following steps,
The obtaining step further obtains anchor information related to anchors set for the outline data,
The anchor setting screen displaying step controls to display an anchor setting screen, wherein the anchor setting screen includes the anchor superimposed on the first image data and is associated with the anchor information. configured to be able to specify a position in the real space,
the alignment step associates the coordinate system of the outline data with the coordinate system of the real space by associating the anchor information with the specified position in the real space;
thing. In the information processing system according to claim 4,
configured to further perform each of the following steps,
In the layout input screen display step, a layout input screen is displayed, wherein the layout input screen displays a position on the area formed by the outline included in the outline data associated with the anchor information. configured to be specifiable,
The acquisition step acquires the anchor information associated with the specified position on the area formed by the outline.
thing. In the information processing system according to any one of claims 1 to 5,
configured to further perform each of the following steps,
The acquiring step further acquires object information about an object that can be placed in the real space, wherein the object information includes drawing data and position coordinate data representing the position of the outline data in the coordinate system. including
The object display screen displaying step controls to display an object display screen, wherein the object display screen is the coordinate system of the real space associated with the coordinate system of the object information and the outline data. A screen in which the object is superimposed on the first image data based on
thing. In the information processing system according to claim 6,
configured to perform the following further steps,
The placement input screen displaying step controls to display a placement input screen, wherein the placement input screen is configured to enable selection of at least one object from among a plurality of objects that can be placed in the real space. and, for the selected object, the position coordinate data can be generated by specifying the position on the area formed by the outline included in the outline data,
the obtaining step obtains the object information of the selected object;
thing. In the information processing system according to any one of claims 1 to 7,
The real space is an internal space of a building,
thing. In the information processing system according to claim 8,
the real space is a room,
thing. In the information processing system according to any one of claims 1 to 9,
The outline data is three-dimensional data,
thing. An information processing method,
Each step of the information processing system according to any one of claims 1 to 10,
Method. a program,
causing a computer to execute each step of the information processing system according to any one of claims 1 to 10,
thing. An information processing device,
configured to perform each of the following steps,
In the receiving step, input relating to arrangement information for outline data relating to the outline of the room is accepted from the first user, wherein the arrangement information includes information relating to the type of object to be arranged in the room and information relating to the arrangement location of the object. including coordinate information;
In the screen display step, by transmitting the arrangement information to the augmented reality presentation device via a network, the augmented reality presentation device displays a screen in which the object, which is virtual information, is superimposed on the room, which is the real space. where the augmented reality presentation device is a terminal of a second user located in the room;
thing. An augmented reality presentation device,
configured to perform each of the following steps,
In the obtaining step, layout information is obtained from an information processing device via a network, wherein the layout information is information about the types of objects to be placed in the room, set for outline data about the outline of the room. and coordinate information about the placement location of the object,
The managing step manages drawing data relating to a plurality of objects that can be placed in the room;
In the display processing step, the object, which is virtual information, is superimposed on the room, which is a real space, based on the obtained layout information and the drawing data.
thing.

Images