JP2020181420A - Virtual object display and program - Google Patents

Abstract

To improve the usability of organizing a plurality of virtual objects.SOLUTION: A virtual object display includes: a memory that stores arrangement patterns including at least one position relative to a reference position; and a CPU that arranges a plurality of virtual objects on a display area displayed on a display device, selects one of the plurality of virtual objects as a selected object, sets an arrangement pattern stored in the memory using the position of the selected object on the display area as a reference position, for the selected object, and places other virtual objects on the display area based on the arrangement pattern.SELECTED DRAWING: Figure 18

Description

An embodiment of the present invention relates to a virtual object display device and a program that causes a computer to function as a virtual object display device.

Conventionally, for the purpose of creating ideas, a device for displaying a virtual object imitating a memo medium such as a sticky note on a display device so that the user can move it has been proposed. For example, the user can visually classify and organize the information shown in the virtual objects by arranging a plurality of virtual objects together in a group.

Japanese Unexamined Patent Publication No. 2006-31666 Japanese Unexamined Patent Publication No. 2014-229167 JP-A-2015-75925

One issue is to improve the usability of organizing multiple virtual objects.

In one embodiment, the virtual object display device has a memory that stores an arrangement pattern including at least one position relative to a reference position; and a plurality of virtual objects arranged on a display area displayed on the display device. One of the virtual objects of is selected as a selection object, an arrangement pattern stored in the memory is set for the selection object with the position of the selection object on the display area as a reference position, and the arrangement pattern is set to the arrangement pattern. A CPU that arranges other virtual objects on the display area based on the above;

It is the schematic which shows the whole structure of the virtual object display system of embodiment. It is a block diagram which shows the hardware composition of the virtual object display device of embodiment. It is a block diagram which shows the hardware configuration of the terminal apparatus of embodiment. It is a flowchart which shows the processing of the CPU of the virtual object display device of embodiment. It is a figure which shows the image captured by the terminal apparatus of embodiment. It is a figure which shows the sticky note which is automatically extracted of an embodiment. It is a figure which shows the sticky note which is manually extracted of embodiment. It is a figure which shows the photograph which is manually extracted of an embodiment. It is a figure which shows the state which all the sticky notes of embodiment were extracted. It is a flowchart which shows the virtual object display processing of embodiment. It is a figure which shows the virtual object arranged on the panel of embodiment. It is a figure which shows the floating icon of embodiment. It is a figure which shows the panel in which the background image of an embodiment is set. It is a figure which represents the pop-up menu which is displayed for the virtual object of embodiment. It is a figure which shows the edit screen of the virtual object extracted from the captured image of an embodiment. It is a figure which shows the pop-up menu which is displayed with respect to the panel of embodiment. It is a figure which shows the edit screen of the virtual object added to the panel of embodiment. It is a flowchart which shows the set process of embodiment. It is a figure which shows the arrangement pattern set to the selection object of embodiment. It is a figure which shows the locus of other virtual objects arranged in the arrangement pattern of embodiment. It is a figure which represents the other virtual object arranged in the arrangement pattern of embodiment. It is a figure which shows the reduced selection object and arrangement pattern of embodiment.

Hereinafter, embodiments of the virtual object display device will be described with reference to the drawings. In each figure, the same reference numerals are given to the same configurations.

(Configuration of virtual object display system)
The virtual object display system 1 including the virtual object display device 10 of the embodiment will be described. FIG. 1 is a schematic view showing the overall configuration of the virtual object display system 1. 2 and 3 are block diagrams showing the hardware configurations of the virtual object display device 10 and the terminal device 30, respectively.

As shown in FIG. 1, the virtual object display system 1 includes a virtual object display device 10, a display device 20 which is a display having a touch panel sensor, a printer device 21, and a terminal device 30 which is a tablet terminal. .. The virtual object display device 10 is connected to the display device 20 as an input / output device, and is also connected to the terminal device 30 via the wireless router 40.

As shown in FIG. 2, the virtual object display device 10 is equipped with a CPU (Central Processing Unit) 101. The CPU 101 is connected to the ROM (Read Only Memory) 102, the RAM (Random Access Memory) 103, the storage 104, and the host controller 105 by the bus line 106. The ROM 102 stores a program executed by the CPU 101 and data necessary for executing the program. The RAM 103 stores temporary data and the like when the CPU 101 executes a program. The connection interface 107 is a connection adapter connected to the host controller 105 and for connecting the display device 20 and the printer device 21. The communication interface 108 is a network adapter such as a LAN (Local Area Network) connected to the host controller 105.

The CPU 101 causes the display device 20 to output a screen via the host controller 105 and the connection interface 107, causes the printer device 21 to print the output screen by the display device 20, and acquires a touch operation to the display device 20 as an input. .. The CPU 101 controls the communication interface 108 via the host controller 105 in order to configure a LAN with a wireless router 40 that communicates wirelessly with the terminal device 30.

As shown in FIG. 3, the terminal device 30 is equipped with the CPU 301. The CPU 301 is connected to the ROM 302, the RAM 303, the storage 304, and the host controller 305 by the bus line 306. The ROM 302 stores a program executed by the CPU 301 and data necessary for executing the program. The RAM 303 stores temporary data and the like when the CPU 101 executes a program. The host controller 305 is connected to a display 307 having a touch panel sensor, a camera 308, and a communication interface 309. The communication interface 309 is a network adapter for performing wireless communication such as a wireless LAN card.

The CPU 301 displays the screen on the display 307 via the host controller 305, and acquires the input from the touch panel sensor of the display 307. The CPU 301 controls the communication interface 309 via the host controller 305 in order to configure a WLAN that communicates wirelessly with the wireless router 40.

The terminal device 30 transmits the captured image captured by the camera 308 on the bulletin board B to which the sticky note S is attached to the virtual object display device 10. The virtual object display device 10 extracts the sticky note S in the captured image received from the terminal device 30 as the virtual object O, and displays the virtual object O operably on the display device 20. The virtual object O is a virtual object that is movably arranged on a panel that is a two-dimensional display area in which the bulletin board B is virtualized, and is formed in a rectangular shape in the present embodiment. The user who uses the virtual object display system 1 can move, reduce, enlarge, and edit the virtual object O by touching the display device 20.

(Operation of virtual object display device)
The processing of the CPU 101 of the virtual object display device 10 will be described. FIG. 4 is a flowchart showing the processing of the CPU 101 of the virtual object display device 10. In FIG. 4, it is assumed that the captured image captured by the camera 308 of the terminal device 30 has been received in advance by the virtual object display device 10.

As shown in FIG. 4, the CPU 101 of the virtual object display device 10 acquires the captured image by storing the received captured image in the RAM 103 (S101). Here, it is assumed that the captured image is a bulletin board B to which a plurality of sticky notes S and photographs P are attached, as shown in FIG. Further, it is assumed that the sticky note B on the bulletin board B is colored in gray or the like, and each sticky note S is formed in a rectangular shape, and is a piece of paper having at least a contrast difference with respect to the sticky note.

After acquiring the captured image, the CPU 101 automatically extracts the sticky note S in the captured image as the virtual object O (S102), as shown in FIG. FIG. 6 shows a display screen of the display device 20 in a state where the virtual object O is automatically extracted. Each of the extracted virtual objects O is attached with a marker indicating an upper side and a semi-transparent rectangle covering the virtual object O.

The CPU 101 extracts the sticker S from the captured image by extracting a quadrangular region in which pixels having saturation equal to or higher than a predetermined threshold are continuous. For example, based on the contrast difference with the sticking surface as the background. It may be extracted from the captured image. The virtual object O extracted from the captured image includes an image obtained by cropping a part of the captured image.

The CPU 101 also extracts a trapezoidal quadrangular region and an inclined quadrangular region as virtual objects O. As a result, the sticky note S can be automatically extracted even if the captured image is captured from a position inclined with respect to the bulletin board B, that is, an imaging direction that is not substantially perpendicular to the plane of the bulletin board B. By performing keystone correction and rotation correction for each of the extracted virtual objects O, the CPU 101 distorts the upper and lower sides of the virtual object O so as to be horizontal and the left and right sides to be vertical. To correct.

The CPU 101 determines whether or not the extraction instruction has been given by the user (S103), and when the extraction instruction is given (S103, YES), the sticky note S is manually extracted as the virtual object O (S104).

The manual extraction is performed by the user operating the extraction area E as shown in FIG. The extraction area E is displayed as a quadrangular frame as a whole by swiping the display device 20. The extraction area E has a marker indicating the upper side of the frame body and a plurality of handles, and the shape of the extraction area E is deformed, enlarged, or reduced by operating any of the handles. Further, the extraction area E is rotated by performing a rotate operation for moving at least one point while tapping two points with respect to the portion other than the handle in the extraction area E, and the extraction area E is moved by the drag operation. Will be done. According to the manual extraction, since a predetermined area of the captured image is extracted as the virtual object O, not only the sticky note S but also the photograph P attached to the bulletin board B can be extracted as shown in FIG. it can.

If the extraction instruction is not given in step S103 (S103, NO), or after the virtual object O is manually extracted, the CPU 101 determines whether or not the extraction end instruction has been given by the user (S105). Here, the extraction end instruction is given by tapping the extraction end instruction button displayed on the display device 20 as shown in FIG. The user taps the extraction end instruction button after extracting all the parts to be extracted as the virtual object O in the captured image.

When the extraction end instruction is given (S105, YES), the CPU 101 executes the virtual object display process described later (S106). On the other hand, when the extraction end instruction is not given (S105, NO), the CPU 101 again determines whether or not the extraction instruction has been given by the user (S103).

(Virtual object display processing)
The virtual object display process will be described. FIG. 10 is a flowchart showing the virtual object display process.

As shown in FIG. 10, the CPU 101 of the virtual object display device 10 displays the virtual object O on the panel (S201). Here, as described above, the panel is a two-dimensional display area in which the bulletin board B is virtualized, and the virtual object O is managed by the CPU 101 in association with the panel on which it is arranged. As shown in FIG. 11, the CPU 101 newly creates one panel PN for the extraction of the virtual object O for one captured image, superimposes the virtual object O on the panel PN, and displays the virtual object O on the panel PN. Brings a floating icon FI in the foreground that contains multiple instructions to change the state of the panel PN or virtual object O. The upper part of the panel PN is a title area for displaying the name of the panel, and the user can input an arbitrary name.

The floating icon FI will be described. As shown in FIG. 12, the floating icon FI includes a virtual object edit icon, a hand-drawn input icon, a text input icon, a virtual object erase icon, a background setting icon, a new creation / read icon, and a print icon as icons indicating instructions. Includes save icon and undo icon. Of these, the virtual object edit icon and the virtual object delete icon indicate changes to the individual virtual objects O, and the other icons indicate changes to the panel PN.

The hand-drawing input icon instructs the panel PN to shift to the hand-drawing mode in which the hand-drawing input can be performed. In the hand drawing mode, the CPU 101 draws a locus traced by the user on the display device 20 as a line, and generates a hand drawing object DO as shown in FIG. The text input icon indicates that the user shifts to the text mode in which the panel PN can input text. The CPU 101 displays a software keyboard on the display device 20 in the text mode, and generates a text object TO as shown in FIG. 11 from the text input by the software keyboard.

The save icon is an instruction for the user to execute a save process in which the CPU 101 stores the panel PN displayed on the display device 20 as panel information in the storage 104. The panel information indicates the state of the panel PN and the states of all virtual objects O arranged on the panel PN, including the hand-drawn object DO and the text object TO. In the save process, the user is required to give a file name to the panel information, and when the panel name is set in the panel PN as shown in FIG. 11, the panel name is set as the default file name in advance. Entered. The new creation / read icon indicates the user to execute a new creation process in which the CPU 101 creates a new panel PN, or a read process in which the CPU 101 reads the panel information stored in the storage 104.

The print icon indicates the user to execute a printing process in which the CPU 101 causes the printer device 21 to print the screen displayed by the display device 20. The undo icon indicates that the user executes an undo process in which the CPU 101 cancels the change in the state of the panel PN due to the operation immediately performed by the user and returns the panel PN to the original state.

As shown in FIG. 13, the background setting icon indicates that the user instructs the CPU 101 to execute a background setting process for setting an image selected by the user for the panel PN displayed as the background of the virtual object O. .. By making it possible to set an image on the panel PN in this way, it is possible to display an image that serves as an arrangement reference for the virtual object O as the background of the virtual object O. For example, in FIG. 13, an image representing the vertical axis and the horizontal axis is set in the panel PN, and the virtual object O is moved by the panel PN with reference to the vertical axis and the horizontal axis to display a virtual object display device. 10 can be used for idea generation by the matrix method or the like.

The virtual object erase icon instructs the user to shift to the erase mode in which the virtual object O can be erased. The CPU 101 erases the virtual object O selected by the user from the panel PN in the erase mode. The virtual object edit icon indicates a transition to an edit mode that allows the user to make changes to the individual virtual objects O.

As shown in FIG. 10, the CPU 101 determines whether or not the user has instructed to change the state of the panel PN via the floating icon FI described above (S202). In this determination, the CPU 101 determines whether or not any of the hand-drawn input icon, the text input icon, the background setting icon, the new creation / loading icon, the print icon, the save icon, and the undo icon in the floating icon FI is selected. .. When an instruction to change the state of the panel PN is given (S202, YES), the CPU 101 updates the state of the panel PN based on the instruction (S203).

When the instruction to change the state of the panel PN is not given (S202, NO), or after updating the state of the panel PN, the CPU 101 determines whether or not the instruction to change the state of the virtual object O is given (S204). .. In this determination, the CPU 101 determines that an instruction to change the state of the virtual object O has been given when either the virtual object edit icon or the virtual object delete icon in the floating icon FI is selected.

In the edit mode transitioned by selecting the virtual object edit icon, the CPU 101 displays the pop-up menu PM1 (see FIG. 14) or the pop-up menu PM2 (see FIG. 16) when the tap operation is performed for a certain period of time or longer. It is displayed on the front surface of the device 20. The pop-up menu PM1 is displayed on the virtual object O by tapping the virtual object O for a certain period of time or longer, and includes an edit menu, a copy menu, and a set menu as shown in FIG. The pop-up menu PM2 is displayed by performing a tap operation for a certain period of time or longer at a place where the virtual object O is not displayed, and includes an additional menu and a paste menu as shown in FIG.

The copy menu is selected by the user to instruct the CPU 101 to execute a copy process for copying the corresponding virtual object O. The edit menu is selected by the user to instruct the CPU 101 to execute an edit process for editing the corresponding virtual object O. When the execution of the editing process is instructed, the CPU 101 causes the display device 20 to display the editing screen as shown in FIG. According to the edit screen, the user can set the classification tags T1 to T5 for the virtual object O to be edited. Each of the classification tags T1 to T5 is attribute information related to the corresponding virtual object O, and when the attribute information is not added to the virtual object O, the classification tag T0 is set for the virtual object O. By setting the classification tags T1 to T5 on the virtual object O, the user can classify the virtual object O based on the criteria according to a predetermined viewpoint.

When selected by the user, the set menu instructs the CPU 101 to execute a set process for arranging another virtual object having the same classification tag as the virtual object O for the corresponding virtual object O. When the paste menu is selected by the user, the CPU 101 instructs the user to execute the paste process of pasting the virtual object O copied by the copy process into the panel PN.

When the additional menu is selected by the user, the CPU 101 instructs the panel PN to execute an additional process of adding a new virtual object O. When the execution of the additional processing is instructed, the CPU 101 causes the display device 20 to display the additional screen as shown in FIG. According to the additional screen, the user can set the background color of the new virtual object O, and edit the content drawn in the new virtual object O by using the hand drawing input tool and the line erasing tool. Can be done. Further, according to the addition screen, any of the classification tags T1 to T5 can be set for the new virtual object O as in the edit screen.

As shown in FIG. 10, when the instruction to change the state of the virtual object O is not given (S204, NO), the CPU 101 displays the virtual object O on the panel PN again (S201). On the other hand, when an instruction to change the state of the virtual object O is given (S204, YES), the CPU 101 determines whether or not an additional instruction to add a new virtual object O to the panel PN is given (S205). Here, the CPU 101 determines that an additional instruction has been given when the OK button is selected on the additional screen shown in FIG.

When an additional instruction is given (S205, YES), the CPU 101 adds a new virtual object O to the panel PN (S206), and displays the virtual object O on the panel PN again (S201). On the other hand, when the additional instruction is not given (S205, NO), the CPU 101 determines whether or not the tag change instruction for instructing the change of the classification tag of the virtual object O has been given (S207). Here, the CPU 101 gives a tag change instruction when a classification tag different from the classification tag set for the virtual object O to be edited is selected on the edit screen shown in FIG. 15 and the OK button is selected. Judged as

When the tag change instruction is given (S207, YES), the CPU 101 sets a new classification tag for the virtual object O to be edited, updates the classification tag (S208), and again puts the virtual object O on the panel PN. Is displayed (S201). On the other hand, when the tag change instruction is not given (S207, NO), the CPU 101 determines whether or not the set instruction for instructing the execution of the set process has been given (S209). Here, the CPU 101 determines that the set instruction has been given when the set menu in the pop-up menu PM1 shown in FIG. 14 is selected.

When the set instruction is given (S209, YES), the CPU 101 executes the set process described later (S210), and displays the virtual object O on the panel PN again (S201). On the other hand, when the set instruction is not given (S209, NO), the CPU 101 executes the corresponding process (S211) in response to the instruction to execute the copy process and the paste process, and again displays the virtual object O on the panel PN. Display (S201).

(Assembly processing)
The set processing will be described. FIG. 18 is a flowchart showing the set processing.

As shown in FIG. 18, the CPU 101 selects the virtual object O, which is the target of the set processing, as the selection object (S301). Here, the selected object is a virtual object O corresponding to the set menu shown in FIG. As shown in FIG. 19, the CPU 101 sets the arrangement pattern A for the selected object O0 (S302). The arrangement pattern A is stored in the ROM 102 in advance, and includes at least one position relative to the reference position with the position of the selected object O0 as the reference position. The arrangement pattern A shown in FIG. 19 includes a plurality of relative positions surrounding the selected object O0 in a matrix, and the arrangement order shown by 1 to 24 is set for each relative position.

After setting the arrangement pattern A, the CPU 101 arranges the other virtual objects O1 to O6 having the same classification tag as the selected object O0 at the relative positions included in the arrangement pattern A, as shown in FIGS. 20 and 21. (S303). In the present embodiment, as shown in FIG. 20, the CPU 101 uses each of the other virtual objects O1 to O6 so that the movement loci from the arrangement positions of the other virtual objects O1 to O6 to their relative positions do not intersect each other. Is placed in the corresponding relative position. Further, the CPU 101 arranges the other virtual objects O1 to O6 at relative positions in ascending order of arrangement. The order of arranging the virtual object O at the relative position may be based on a predetermined sortable criterion, and such a predetermined criterion is, for example, the number of the ID attached to the virtual object O or Examples include characters, the distance between the placement position of another virtual object O and the reference position, and the like.

When another virtual object O is arranged at a relative position as shown in FIG. 21, the CPU 101 determines whether or not the size of the selected object O0 has been changed (S304). When the size of the selected object O0 is changed (S304, YES), the CPU 101 changes the size of the arrangement pattern A based on the size of the selected object O0 (S305). Here, as shown in FIG. 22, the CPU 101 reduces the size of the arrangement pattern A when the selected object O0 is reduced, and increases the size of the arrangement pattern A when the selected object O0 is enlarged. .. When the size of the arrangement pattern A is changed, the distance of the relative position included in the arrangement pattern A from the reference position is changed.

After the size of the arrangement pattern A is changed, or when the size of the selected object O0 is not changed (S304, NO), the CPU 101 determines whether or not the classification tag set in the selected object O0 has been changed (S306). When the classification tag is changed (S306, YES), the CPU 101 sets another virtual object O in which the same classification tag as the changed classification tag is set in the arrangement pattern A (S303). On the other hand, when the classification tag is not changed (S306, NO), the CPU 101 determines whether or not the edit mode is canceled (S307). When the edit mode is canceled (S307, YES), the CPU 101 ends the set processing, and when the edit mode is not canceled (S307, NO), whether or not the size of the selected object O0 has been changed again. Is determined (S304).

In this way, by automatically arranging other virtual objects O having the same classification tag set at relative positions relative to the position of the selected virtual object O, the other virtual objects O can be manually individually arranged individually. Other virtual objects O can be easily predicted and placed in a visible position without being moved and sorted and arranged according to attributes, improving usability for organizing multiple virtual objects O. Can be made to. Furthermore, when the classification tag of the selected virtual object O is changed, other virtual objects O for which the changed classification tag is set are aggregated, so that even if the classification standard of the virtual object O is changed, There is no need to manually rearrange many virtual objects O individually.

In the embodiment, one classification tag is set for the virtual object O, but a plurality of classification tags may be set. In this case, the CPU 101 presents the classification tag set in the target virtual object O to the user after selecting the set menu in the floating menu FM1, and sets the same classification tag as the selected classification tag. Other virtual objects O may be arranged at relative positions.

In the embodiment, it is assumed that a program for realizing the function of the invention is recorded in advance in a ROM which is a program storage unit inside the device, but the present invention is not limited to this, and a similar program may be downloaded from the network to the device. .. Alternatively, a similar program recorded on the recording medium may be installed on the device. The recording medium may be in any form as long as it can store a program such as a CD-ROM or a memory card and the device can read it. Further, the function obtained by installing or downloading the program may be one that realizes the function in cooperation with the OS (operating system) or the like inside the device.

Although embodiments of the invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 Virtual object display system 10 Virtual object display device 20 Display device 30 Terminal device 101 CPU
102 ROM

Claims (6)

With a memory that stores an arrangement pattern that includes at least one position relative to the reference position;
A plurality of virtual objects are arranged on a display area displayed on a display device, one of the plurality of virtual objects is selected as a selection object, and the selection is made using the position of the selection object on the display area as a reference position. With a CPU that sets an arrangement pattern stored in the memory for an object and arranges another virtual object on the display area based on the arrangement pattern;
A virtual object display device that comprises. The CPU claims that the attributes of the virtual object are set for each of the plurality of virtual objects, and other virtual objects to which the same attribute information as the selected object is added are arranged at the relative positions. The virtual object display device according to 1. The virtual object display device according to claim 1 or 2, wherein the CPU changes the size of the arrangement pattern based on the size change of the virtual object. The arrangement position in the arrangement pattern is associated with the arrangement order, and the CPU arranges other sorted virtual objects at the arrangement position based on the arrangement order, claims 1 to 3. The virtual object display device according to any one of the above. The virtual object display device according to any one of claims 1 to 4, wherein the CPU sets an image for the display area. With the function of arranging multiple virtual objects on the display area displayed on the display device;
With the function of selecting one of the plurality of virtual objects as a selection object;
With the position of the selected object on the display area as a reference position, a function of setting an arrangement pattern including at least one relative position with respect to the selected object;
With the function of arranging other virtual objects on the display area based on the arrangement pattern;
A program for realizing the above in the CPU of a computer.