JP2011022703A - Display control apparatus and display control method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To superimpose auxiliary information for supporting a user's operation on an image in real space, in accordance with information of an object desired by the user. <P>SOLUTION: A display control apparatus is equipped with: an entire image acquisition unit 108 for acquiring an entire image taken with an imaging means; an object specifying unit 105 for specifying an object interested by the user from a plurality of objects depending on the user's operation; an object information acquisition unit (interested object detection unit 109, interested object information acquisition unit 110) for acquiring object information related to one object specified by the object specifying unit; and an auxiliary information display unit (operation auxiliary information display unit 102) for superimposing auxiliary information for acquiring peripheral information of the one object on the entire image acquired by the entire image acquisition unit based on object information of the one object acquired by the object information acquisition unit. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a display control device and a display control method, and more particularly, to a display control device and a display control method for supporting a user operation by displaying a virtual auxiliary line on an image captured by an imaging device.

  Recently, as cameras and microphones become cheaper and easier to use due to technological advances, multiple cameras and microphones are placed in the office space and used for communication support and monitoring in the office. Sometimes. When a plurality of cameras and microphones are arranged, the user needs to obtain desired information using a plurality of information acquisition means. However, the technique about how a user effectively uses a plurality of information means, such as which camera and which microphone can obtain desired information, is not sufficient. There is a problem that the user cannot easily acquire necessary information because of a large amount of information that can be acquired.

  There is also a technique for obtaining desired information by operating a robot having a moving function equipped with a camera and a microphone. However, as described above, there is a problem that the technique for obtaining desired information by operating the robot is not sufficient. Therefore, in order to support the acquisition of information using a moving camera, a technique for displaying auxiliary information indicating the range of field of view in a virtual space using a plurality of screens is disclosed (for example, patents). Reference 1).

JP 2005-230263 A

However, since the above-mentioned patent document 1 is intended for operation in a virtual space, it does not add information to a video shot in an actual space. In addition, since the auxiliary information is uniformly displayed on the display screen, there is a problem that the auxiliary information becomes insufficient depending on the nature of the information desired by the user.
Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to implement auxiliary information for assisting user operations in accordance with information on an object desired by the user. It is an object of the present invention to provide a new and improved display control apparatus and display control method that can be superimposed on a space image.

  In order to solve the above-described problem, according to an aspect of the present invention, an entire image acquisition unit that acquires an entire image captured by an imaging device, and a user is interested from a plurality of objects according to a user operation Object specifying unit for specifying an object, object information acquiring unit for acquiring object information related to the object specified by the object specifying unit, and object information of the object acquired by the object information acquiring unit A display control device is provided that includes an auxiliary information display unit that superimposes auxiliary information for acquiring peripheral information on an object on the entire image acquired by the entire image acquisition unit.

  According to such a configuration, the entire image is acquired, one object specified by the user operation is specified, and auxiliary information is superimposed on the entire image based on the information related to the object. This makes it possible to superimpose auxiliary information for assisting the user operation on the real space image according to the information of the object desired by the user, and there are various positions such as selectable positions and instruction directions by the user operation. It is possible to present various usages.

  Moreover, you may provide the periphery image | video acquisition part which acquires the periphery image | video of the target object imaged by the imaging device located in the periphery of the target object different from an imaging device.

  The peripheral video acquisition unit may acquire a peripheral video of an object captured by an imaging device capable of imaging the selected range selected using auxiliary information superimposed on the entire video in response to a user operation. Good.

  The object information acquisition unit may acquire the object information including the position information, the direction, and the size of the object based on the image of the object included in the entire image acquired by the entire image acquisition unit. .

  The auxiliary information display unit may determine the type of auxiliary information to be superimposed on the entire image according to the position information, direction, and size of the object.

  The auxiliary information display unit may superimpose a rectangular grid on the entire image as auxiliary information. The auxiliary information display unit may superimpose a concentric grid centered on the object on the entire image as auxiliary information.

  In addition, an auxiliary information storage unit that stores the type of auxiliary information corresponding to the target is provided, and the target information acquisition unit is a type of auxiliary information corresponding to the target that the user specified by the target specifying unit is interested in. May be acquired from the auxiliary information storage unit.

  The auxiliary information display unit may select the type of auxiliary information in accordance with a user operation and superimpose the auxiliary information on the entire image.

  In addition, a marker capable of distributing the position, direction, and ID of the target object is arranged on the target object, and the target object information acquisition unit determines the position and direction of the target object based on the information of the target object distributed from the marker. You may get it.

  The peripheral video acquisition unit may acquire a peripheral video of the object imaged by an imaging device provided in a movable body that can be moved.

  In addition, an operation control unit that controls the operation of the moving object is provided, and the operation control unit moves the moving object to a position indicated by the selection range selected using the auxiliary information superimposed on the entire video according to the user operation. It may be moved.

  The auxiliary information display unit may change the display of the auxiliary information according to the arrangement relationship between the moving object and the object specified by the object specifying unit.

  In order to solve the above-mentioned problem, according to another aspect of the present invention, there is provided a display control method for superimposing predetermined auxiliary information on an entire video imaged by an imaging device, the entire image captured by the imaging device. A step of acquiring an image, a step of specifying an object of interest from a plurality of objects in response to a user operation, a step of acquiring object information relating to the specified object, and an acquired object Superimposing auxiliary information for acquiring peripheral information of the object on the entire image acquired by the entire image acquisition unit based on the object information of the object, Is provided.

  As described above, according to the present invention, auxiliary information for assisting a user operation can be superimposed on an image in real space in accordance with information on an object desired by the user.

It is a block diagram which shows the function structure of the display control apparatus concerning the 1st Embodiment of this invention. It is explanatory drawing explaining the implementation image of the display control system concerning the embodiment. It is the image of the terminal which the user who uses the display control system concerning the embodiment uses. It is a display example of the global whole image displayed on the image display unit according to the embodiment. It is the flowchart which showed the detail of the display control process concerning the embodiment. It is explanatory drawing explaining the example of a display of the auxiliary information concerning the embodiment. It is explanatory drawing explaining the example of a display of the auxiliary information concerning the embodiment. It is a block diagram which shows the function structure of the display control apparatus concerning 2nd Embodiment concerning the embodiment. It is explanatory drawing explaining the image of the external shape of the robot concerning the embodiment. It is a flowchart which shows the detail of the display control process concerning the embodiment. It is explanatory drawing explaining the movement of the robot concerning the embodiment. It is explanatory drawing explaining the movement of the robot concerning the embodiment. It is explanatory drawing explaining the movement of the robot concerning the embodiment. It is explanatory drawing explaining the movement of the robot concerning the embodiment. It is explanatory drawing explaining the movement of the conventional robot. It is explanatory drawing explaining the movement of the conventional robot.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

Further, the “detailed description of the embodiments” will be described in the order shown below.
[1] Purpose of this embodiment [2] First embodiment [2-1] Functional configuration of display control device [2-2] Details of display control processing [3] Second embodiment [3-1] Functional configuration of display control device [3-2] Details of display control processing

[1] Purpose of this Embodiment First, the purpose of this embodiment will be described. Recently, as cameras and microphones become cheaper and easier to use due to technological advances, multiple cameras and microphones are placed in the office space and used for communication support and monitoring in the office. Sometimes. When a plurality of cameras and microphones are arranged, the user needs to obtain desired information using a plurality of information acquisition means. However, the technique about how a user effectively uses a plurality of information means, such as which camera and which microphone can obtain desired information, is not sufficient. There is a problem that the user cannot easily acquire necessary information because of a large amount of information that can be acquired.

  There is also a technique for obtaining desired information by operating a robot having a moving function equipped with a camera and a microphone. However, as described above, there is a problem that the technique for obtaining desired information by operating the robot is not sufficient. Therefore, in order to support acquisition of information using a moving camera, a technique for displaying auxiliary information indicating a range of field of view in a virtual space using a plurality of screens is disclosed. However, since the technique is intended for operation in a virtual space, it does not add information to video captured in an actual space. Further, since the auxiliary information is uniformly displayed on the display screen, there is a problem that the auxiliary information becomes insufficient depending on the nature of the information desired by the user.

  Therefore, the display control apparatus 100 according to the embodiment of the present invention has been created with the above circumstances as a focus. According to the display control apparatus 100 according to the present embodiment, it is possible to superimpose auxiliary information for assisting a user operation on an image in real space according to information on an object desired by the user.

[2] First Embodiment The object of the embodiment of the present invention has been described above. Next, a functional configuration of the display control apparatus 100 according to the present embodiment will be described with reference to FIG. In the description of the functional configuration of the display control apparatus 100, FIGS. The display control device 100 can be exemplified by a computer device such as a personal computer (regardless of a notebook type or a desktop type), but is not limited to such an example, and is constituted by a mobile phone or a PDA (Personal Digital Assistant). May be.

[2-1] Functional Configuration of Display Control Device Before describing the functional configuration of FIG. 1, an example of a hardware configuration of the display control device 100 will be described. The display control device 100 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input device, an output device, a storage device (HDD), and the like.

  The CPU functions as an arithmetic processing device and a control device, and controls the overall operation of the display control device 100 according to various programs. The CPU may be a microprocessor. The ROM stores programs used by the CPU, calculation parameters, and the like. The RAM primarily stores programs used in the execution of the CPU, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus including a CPU bus.

  The input device includes, for example, an input means for a user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and input control that generates an input signal based on the input by the user and outputs the input signal to the CPU. It consists of a circuit.

  The output device includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Display) device and a lamp, and an audio output device such as a speaker and headphones. .

  The storage device can include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device is composed of, for example, an HDD (Hard Disk Drive). This storage device drives a hard disk and stores programs executed by the CPU and various data.

  The hardware configuration of the display control apparatus 100 has been described above. Next, the functional configuration of the display control apparatus 100 will be described with reference to FIG. As shown in FIG. 1, the display control apparatus 100 includes a video display unit 101, an operation auxiliary information display unit 102, an operation unit 103, an operation command unit 104, an interest target selection unit 105, an image sound acquisition unit 106, an image sound presentation. Unit 107, entire video acquisition unit 108, interest object detection unit 109, interest object information acquisition unit 110, interest object database 111, and the like.

  Here, with reference to FIG. 2 and FIG. 3, the implementation image of the display control system implement | achieved using the display control apparatus 100 is demonstrated. 2 and 3 are explanatory diagrams for explaining an implementation image of the display control system realized by using the display control apparatus 100. FIG. As shown in FIG. 2, a plurality of omnidirectional cameras (10 to 12) and microphones (1 to 9) are installed on the ceiling of a space such as an office. In addition, a camera 20 that captures the entire image of the office globally is installed at a higher position than the omnidirectional cameras (10-12) and the microphones (1-9). Note that the number of omnidirectional cameras is not limited to three as shown in FIG. 2, and the same number as the microphones may be installed. In this way, by installing a camera that captures the entire video and a plurality of other cameras, the position of the imaging target in the office can be grasped, and detailed information of the imaging target can be acquired. .

  FIG. 3 is an image of the terminal 150 used by a user who uses the display control system 1. The terminal 150 includes, for example, a display 151 and a speaker 152. An image is presented on the display 151 and a sound is presented on the speaker 152. Further, when the user operates the operation unit provided in the terminal 150, the display on the display 151 and the sound from the speaker 152 change. For example, the user can display an image, sound, and the like around the object in the office shown in FIG. 2 via the display 151 by operating an operation unit including a touch panel. .

  Returning to FIG. 1, the description of the functional configuration of the display control apparatus 100 will be continued. The video display unit 101 is configured by the display 151, and has a function of displaying video captured by a camera (camera 20) that captures a global overall video, other cameras, and the like. The global overall video is acquired by the overall video acquisition unit 108 described later. The peripheral video of the object other than the global whole video is acquired by the image sound acquisition unit 106 described later.

  FIG. 4 shows a display example of a global overall video acquired by the overall video acquisition unit 108 and displayed on the video display unit 101. As shown in FIG. 4, the imaging target 41, the imaging target 42, and the like are displayed in a bird's-eye view on the global overall video 401 displayed on the video display unit 101. In addition, the video display unit 101 includes an operation auxiliary information display unit 102. The operation auxiliary information display unit 102 superimposes auxiliary information for acquiring peripheral information on the object on the entire image based on the object information of the object.

  Here, the object is an object of interest of the user. The object information of the object is information such as the position, posture, and size of the object. Information such as the position, orientation, and size of the object is detected by an interest object detection unit 109 described later. The auxiliary information for acquiring the peripheral information of the object is information for easily acquiring the information of the object included in the entire image, and includes, for example, a rectangular grid or a concentric grid. It can be illustrated. By superimposing a rectangular grid or concentric grid on the entire image, and the user performing operations on the auxiliary information, the desired information on the target can be easily operated by indirectly operating an imaging device such as a camera. Can be obtained. The operation auxiliary information display unit 102 is an example of the auxiliary information display unit of the present invention.

  Also, the type of auxiliary information corresponding to the object is stored in the interest object database 111 described later. In this case, the operation auxiliary information display unit 102 displays auxiliary information based on the type of auxiliary information corresponding to the object acquired via the interest object information acquisition unit 110. Further, the auxiliary operation information display unit 102 may select the type of auxiliary information in accordance with a user operation.

  The operation unit 103 includes the above-described input device and has a function of accepting a user operation. The operation unit 103 includes at least an operation command unit 104 and an interest target selection unit 105. Specifically, it may be realized as a touch panel on the display of the video display unit 101. Further, it may be realized using a normal pointing device such as a joystick or a mouse.

  The operation command unit 104 notifies the image / acquisition acquisition unit 106 of an operation command issued by the user, and changes the method of acquiring video and audio. Moreover, the interested object selection part 105 has a function which designates one target object from the several target object contained in the predetermined range according to user operation. The predetermined range is, for example, a range displayed on the video display unit 101. One object specified from a plurality of objects is an object which a user is interested in. In addition, the interest object detection unit 109 is notified of information on the object selected by the user operation. The object of interest selection unit 105 is an example of an object specifying unit of the present invention.

  When the operation unit 103 is realized as a touch panel on the video display unit 101, the user can directly specify and specify an object of interest displayed on the video display unit 101. Further, the object that the user is interested in may be managed by the ID and recorded in the interest object database 111. In this case, the interested object selecting unit 105 notifies the interested object information acquiring unit 110 of the object ID of the interested object selected by the user. The object ID of interest is information acquired by the interest detection unit 109 described later. The object ID will be described in detail later.

  The image / acquisition acquisition unit 106 has a function of acquiring a peripheral image of an object captured by an imaging device such as a microphone (microphones 1 to 9) or a camera (cameras 10 to 13) located around the object. A microphone, a camera, or the like located around the object includes at least one imaging device. The image acquisition unit 106 can change the transmission method of the video and audio captured by the camera or the like based on the command from the operation command unit 104. Specifically, a camera that acquires video and a microphone that transmits audio are selected from a plurality of microphones and cameras, or the directivity of the selected microphone is changed. The image sound acquisition unit 106 is an example of a peripheral video acquisition unit of the present invention.

  The image / sound presentation unit 107 includes the above-described output device and has at least a function of presenting video and audio. Specifically, for example, the video presentation function is realized as a display, and the audio presentation function is realized using a speaker or the like. The image / sound presentation unit 107 displays the video and audio transmitted from the image / sound acquisition unit 106 as appropriate.

  The overall video acquisition unit 108 has a function of acquiring a global overall video of a predetermined range imaged by the imaging device. The whole video acquisition unit 108 acquires, for example, a video captured by the ceiling overhead camera 20 illustrated in FIG. The entire video acquisition unit 108 provides the acquired entire video to the video display unit 101.

  The interest object detection unit 109 has a function of acquiring information on an object of interest of the designated user based on the designation from the interest object selection unit 105. For example, image recognition processing is performed on the global video acquired by the overall video acquisition unit 108, where the object is located in the video, what kind of posture is taken, and how much Gets the display size.

For example, it is assumed that an object in which the user is interested has a marker as shown in Non-Patent Document 2. In this case, the position and direction in the screen can be detected by the frame of the marker, and the type of the object can be detected by the pattern in the marker. The type of object is the above-described object ID or the like. Since the method for acquiring the position, direction, and pattern of the object using the markers is disclosed in Non-Patent Document 2 and Patent Document 2, detailed description thereof is omitted.
(Non-Patent Document 1)
Kato, H. Billinghurst, M. (1999)
Marker Tracking and HMD Calibration for a video-based Augmented Reality Conferencing System, In Proceedings of the 2nd International Workshop on Augumented Reality (IWAR 99). October, San Francisco, USA.
(Patent Document 2)
JP2000-322602A

  The interest object information acquisition unit 110 searches the interest object database 111 using the object ID notified by the interest object selection unit 105 and acquires information on the object to be imaged. The object information acquired by the object-of-interest information acquisition unit 110 is, for example, information related to auxiliary information for acquiring peripheral information of the object. For example, the shape of the lattice superimposed on the entire image, the size of the lattice, This is the number of lattices. The interest object detection unit 109 and the interest object information acquisition unit 110 are examples of the object information acquisition unit of the present invention. That is, the interested object detection unit 109 detects physical information such as the position and direction of the object, and the interest information acquisition unit 110 acquires the type of auxiliary information and the number of grids according to the object. The

  In the interest database 111, for example, the object ID and the type and basic size of the auxiliary information are stored in association with each other. The object ID is an ID for identifying the object as described above, and examples of the auxiliary information include a rectangular lattice or a concentric lattice as described above. The basic size indicates the basic size of an object necessary for displaying auxiliary information. The interest object information acquisition unit 110 provides the operation assistance information display unit 102 with information related to the object acquired from the interest object database 111.

[2-2] Details of Display Control Processing The functional configuration of the display control device 100 has been described above. Next, details of display control processing in the display control apparatus 100 will be described. In describing the details of the display control process, FIGS. 6 and 8 will be referred to as appropriate. FIG. 5 is a flowchart showing details of display control processing in the display control apparatus 100.

  As shown in FIG. 5, first, the user selects a target (S102). In step S102, when the user selects a target, it is assumed that a global entire video is displayed on the video display unit 101. In addition, the global whole image includes a plurality of objects to be imaged. Further, the target selected in step S102 is an object of interest to the user. The interested object selection unit 105 selects a target selected by a user operation.

  Then, the interest object information acquisition unit 110 searches the interest object database 111 from the target type selected in step S102 (S104). The target type is a target object ID. As described above, the object-of-interest detection unit 109 detects the object ID using the marker frame or pattern provided on the object. The interest object information acquisition unit 110 receives the object ID corresponding to the object selected in step S102 from the interest object detection unit 109.

  Next, the interest object information acquisition part 110 acquires the information regarding an object from the interest object database 111 (S106). The information related to the object acquired by the interested object information acquisition unit 110 in step S106 is a type of auxiliary information corresponding to the object. Specifically, the auxiliary information includes the type of grid and the number of grids to be superimposed on the entire video.

  Then, the interest detection unit 109 performs image recognition processing on the video acquired by the entire video acquisition unit 108, acquires the position, direction, and size of the target, and acquires them in the operation auxiliary information display unit 102. Notification is made (S108).

  The operation unit 103 determines whether there is an instruction to change auxiliary information from the designation according to the previous user operation, or whether the designation of auxiliary information is the first time (S110). Here, the auxiliary information change instruction is an instruction to change the displayed grid. In step S110, when there is an instruction to change auxiliary information or when the auxiliary information is specified for the first time, auxiliary information is specified (S112). In step S112, when the auxiliary information is designated for the first time, a value set as a default is designated. Here, for example, a rectangular grid is set as a default. Further, a concentric lattice other than the rectangular lattice may be set.

  Then, the auxiliary operation information display unit 102 superimposes and displays a virtual rectangular grid on the entire image as auxiliary information using the position, direction, and size of the target acquired in step S108 (S114). In step S114, auxiliary information corresponding to the position, direction, and size of the target is displayed. For example, auxiliary information may be displayed only in the vicinity of the target position, or the size of the rectangular grid may be changed according to the size of the target.

  Here, with reference to FIG. 6 and FIG. 7, a display example of the auxiliary information superimposed on the entire video will be described. 6 and 7 are explanatory diagrams for explaining examples of displaying auxiliary information. As shown in FIG. 6, the target 41 and the target 42 are displayed in the display example 402. When the target 41 is designated according to the user operation, the auxiliary information 43 is displayed around the target 41 so as to be superimposed on the entire video. The auxiliary information 42 is a rectangular lattice, and is information for facilitating selection around the target 41 in which the user is interested. The auxiliary information 42 is displayed in a predetermined range of the target 41 and is displayed with a lattice size corresponding to the size of the target 41.

  Returning to FIG. 5, a predetermined area (element) of auxiliary information displayed in a rectangle or the like is selected by the user (S116). Here, a case where a predetermined area of auxiliary information is selected by the user will be described with reference to FIG. As shown in the display example 404 in FIG. 6, the user selects the area 44 that is one of the auxiliary information 42 of the rectangular grid. By selecting the area 44, the user has instructed an operation to acquire video and audio from the lower right side of the target 41.

  Here, a case where a concentric lattice is designated as auxiliary information in step S112 will be described. As shown in FIG. 7, when the auxiliary information concentric lattice is designated in step S112, the concentric lattice is displayed at the center of the target. As shown in the display example 501, when the user selects the target 52, a concentric grid centering on the target 52 is displayed so as to be superimposed on the entire image. Then, as shown in the display example 502, the region 54 is selected from the concentric lattice by the user. When the area 54 is selected, it becomes possible to clarify the direction in which video and audio are acquired, compared to the case where a rectangular grid area is selected. Note that the concentric lattice means, for example, a lattice formed by dividing a plurality of concentric circles into n equal parts (n is an integer of 2 or more).

  From the region (element) selected in step S116, the operation command unit 104 instructs the image and sound acquisition unit 106 to acquire audio and video (S118). In step S118, the image acquisition unit 106 instructed by the operation command unit 105 changes the image acquisition device such as a microphone or a camera to acquire user-desired audio or video.

  For example, it is assumed that microphones and cameras existing at the positions of the numbers shown in FIG. In this case, a camera and speakers M1 and C6 that have a minimum spatial distance from the target position selected by the user are selected. The distance in space with the target position is expressed by, for example, the Euclidean distance. It is possible to acquire information in the vicinity of the target by using a device such as a microphone or a camera selected in this way.

  Then, the image / sound presentation unit 107 presents the audio and video acquired by the image / sound acquisition unit 106 in step S118 (S120). In step S120, the entire video display screen including a plurality of targets is changed to a video in which details of one target selected by the user can be confirmed. Next, it is determined whether or not an end instruction has been given by a user operation (S122). In step S122, if there is an end instruction, the process ends.

  If there is no termination instruction in step S122, it is determined whether or not a target change instruction has been issued by a user operation (S124). If there is a target change instruction in step S124, the processing from step S102 is repeated. If there is no instruction for changing the target in step S124, the processing from step S108 is repeated.

  According to the display control processing according to the present embodiment, in accordance with a user operation, an image that specifies an object that the user is interested in and assists the user operation in accordance with properties such as the shape and size of the object. Present. The user can easily obtain necessary video and audio by operating using the presented auxiliary information. Thereby, more flexible selection is possible than when only the position of the target included in the entire video is designated. Further, by changing the type of auxiliary information according to the object and according to the user operation, various usage methods such as selectable positions and instruction directions can be presented.

[3] Second Embodiment The first embodiment has been described above. Next, a second embodiment according to the present invention will be described. In the first embodiment, a plurality of image acquisition devices such as a camera and a microphone are installed, but these image acquisition devices are fixedly arranged. For this reason, there is a problem in that the information of video and audio that can be acquired is limited depending on the information acquisition range of each image and sound acquisition device. Therefore, in the present embodiment, by configuring the image / acquisition acquisition device as a moving body (robot) having a moving function, it is possible to acquire video and audio with a greater degree of freedom compared to the first embodiment. did.

  Furthermore, in this embodiment, the function of the interface used in the first embodiment is extended to support the change and movement of the viewpoint of the robot camera. Further, the auxiliary information presented to the user is changed depending on the distance and positional relationship between the position of the robot and the object of interest of the user, thereby enabling more flexible operation. In the present embodiment, the robot allows the user to work on a predetermined space by providing a function for presenting some information to the user as well as the image / sound acquisition function.

[3-1] Functional Configuration of Display Control Device The functional configuration of the display control device 200 according to the present embodiment will be described with reference to FIG. Hereinafter, the configuration different from that of the first embodiment will be described in detail, and detailed description of the same configuration as that of the first embodiment will be omitted. As shown in FIG. 8, the display control apparatus 200 according to the present embodiment is greatly different from the first embodiment in that it includes a robot control unit 901 that controls a robot (not shown). The robot control unit 901 is an example of the operation control unit of the present invention.

  Here, the outline of the robot controlled by the robot control unit 901 will be described with reference to FIG. FIG. 9 is an explanatory diagram for explaining an image of the outer shape of the robot. As shown in FIG. 9, the robot 90 includes a movable operating body 91 and includes an image / acquisition acquisition device 92 and an information presentation device 93 that acquire video and audio. The operation body 91 of the robot 90 may be provided with omnidirectional wheels that can operate in all directions. The image acquisition device 92 may be a camera that can change the posture and zoom. The information presentation device 93 may be a display capable of displaying a still image or a moving image, for example.

  Returning to FIG. 8, the functional configuration of the robot control unit 901 will be described. The robot control unit 901 includes an image sound acquisition unit 902, an operation unit 903, an information presentation unit 904, and the like. The image / acquisition acquisition unit 902 receives an object (an object of interest of the user) captured by an imaging device (image / acquisition acquisition device 92) such as a camera provided in the operating body 91 under the instruction of the robot control unit 901. It has a function to acquire peripheral video.

  The operation unit 903 has a function of moving the operation body 91 of the robot 90 to the instructed position and direction under the instruction of the robot control unit 901. The information presentation unit 904 controls information presentation by an information presentation device such as a display, a projector, and a speaker mounted on the robot under the instruction of the robot control unit 901. The information presenting unit 904 may be configured to present preset content, or may acquire video or audio provided from another device different from the display control device 200 and present it on a display or the like. It may be.

  The robot position acquisition unit 905 has a function of acquiring the position of the robot 90 in the space and the direction of the robot 90. The robot position acquisition unit 905 may acquire the position (for example, coordinates) and direction of the robot 90 by performing image recognition processing on the entire image provided by the entire video acquisition unit 108. Further, information on sensors arranged in the space where the robot 90 is located may be integrated to acquire the position and direction of the robot 90 on the screen. The robot position acquisition unit 905 notifies the video display unit 101 of the position and direction of the robot 90 via the operation assistance information display unit 906.

[3-2] Details of Display Control Processing The functional configuration of the display control apparatus 200 according to the present embodiment has been described above. Next, the details of the display control process in the display control apparatus 200 will be described with reference to FIG. FIG. 10 is a flowchart showing details of display control processing in the display control apparatus 200. Regarding the display control process according to the present embodiment, the process different from the first embodiment will be particularly described, and the details of the process similar to the first embodiment will be omitted.

  Since the processing from step S202 to step S210 is the same as the processing from step S102 to step S110 of the first embodiment, detailed description thereof is omitted. In step S210, when there is no instruction to change auxiliary information from the designation according to the previous user operation or the designation of the first auxiliary information is not made, the robot position acquisition unit 905 acquires the position of the robot 90 and performs the operation. The information is provided to the auxiliary information display unit 906 (S214).

  Then, the auxiliary operation information display unit 906 displays the entire image displayed by the video display unit 101 using the position of the robot 90 acquired in step S214 and the position, direction, and size of the target acquired in step S208. Auxiliary information is superimposed and displayed inside (S216). Then, the user selects a predetermined area (element) from the auxiliary information presented in step S216 (S218). The robot control unit 901 moves the robot 90 to the predetermined area (element) selected in step S218, and changes the viewing direction of the robot 90.

  Here, the movement of the robot 90 will be described in detail with reference to FIGS. FIG. 11 is an explanatory diagram for explaining the movement of the robot 90. 11, description will be made assuming that the robot 90 can move on a desk placed in the space in the same space as in FIG. For explanation, the position on the desk is represented by two-dimensional coordinates, the horizontal axis is x, and the vertical axis is y. The position of the robot 90 is also expressed in the same two-dimensional coordinate system if necessary.

  As shown in FIG. 11, in the display example 601, for example, there are posters, presentations, etc. (targets 63) that are objects of interest to the user on the wall surface, and there are also targets 61, 62, etc. Suppose that Assuming that the wall surface target 63 is selected by a user operation, auxiliary information of a rectangular grid such as the display example 601 is displayed superimposed on the entire video.

  Thereafter, when the user selects an area in the display example 601, as shown in the display example 602, the display of the area 64 selected by the user is inverted. Then, the robot 90 moves to the coordinate position represented as the center of the selected region 64. For example, as shown in the display example 603, the robot 90 moves to a desired position. An existing method such as a road map method can be used for generating a movement route to the coordinate position of the robot 90 and for a movement control method.

  Next, with reference to FIG. 12, a case where concentric grids are displayed as auxiliary information in a superimposed manner will be described. As shown in the display example 701 of FIG. 12, it is assumed that the targets 71 to 73 are included in the entire video, and the user selects the target 71 as an object of interest. For example, it is assumed that the target 71 is an object that needs to be observed from around the model, such as a model.

  In this case, a concentric lattice centering on the target 71 is displayed superimposed on the entire image. As shown in the display example 701, the concentric lattice displayed as the auxiliary information may present a region that can be selected by a certain radius every 30 degrees, for example. Further, the angle between the robot 90 and the target 71 on the display screen may be calculated from the positional relationship between the target 71 and the robot 90, and the size and angle of the concentric grid may be changed.

  For example, it is assumed that the robot is present on a two-dimensional coordinate r (Rx, Ry) on the desk and the object is present at o (Ox, Oy). In this case, the angle θ is calculated from the arctan “Rx−Ox, Ry−Oy”. Then, the auxiliary information is rotated by −θ so that r and o are the basic axes in the direction in which the robot faces. As shown in the display example 702, the region 74 is selected by a user operation. Then, as shown in the display example 703, the robot 90 moves to the area selected by the user operation. At this time, the direction of the robot is controlled to face the target 71.

  As described above, when the auxiliary information is displayed by the rectangular lattice, only the positional relationship of the robot is changed. However, when the concentric lattice is displayed as the auxiliary information, the direction in which the robot should face is also determined by the user. Can be intuitively shown. In other words, since the center of the circle is directed toward the robot, the information around the target object that the user wants to acquire, such as the back side of the circular target object, is intuitively presented not only by changing the position but also by changing the direction. It becomes possible. Further, for example, when the target is a person, not only the direction in which the information about the target person is acquired, but also the direction of the information presentation device 93 provided in the robot 90 may be designated.

  Furthermore, as shown in FIG. 13, the size of the area selectable by the user may be changed according to the distance between the robot 90 and the target. For example, when the target 73 is selected by the user, the selectable region is narrowed as the target 73 is closer. Thereby, the user can make finer adjustments as the target 73 is approached.

  Further, as shown in FIG. 14, the size of the selectable area represented by the auxiliary information may be changed according to the size of the target in the display screen. For example, a rectangular grid smaller than the target 75 may be displayed to indicate the width of information that can be acquired by the robot. Further, once one of the areas in the screen display is selected by the user, the position of the robot 90 may be changed by dragging the user.

  In this embodiment, by adding a moving function to the image acquisition device and making it into a robot, more detailed information and more information can be acquired, and user-desired information can be acquired flexibly. It becomes possible. In addition, since it is possible to change the auxiliary information presented according to the relationship between the robot and the target, detailed information can be obtained without complicated operations.

  Here, with reference to FIG. 15 and FIG. 16, the case where the robot 90 is moved in the second embodiment and the case where the robot 90 is moved by a method different from the second embodiment will be compared. FIG. 15 is an explanatory diagram for explaining a case where the robot is moved using an input operation means such as a joystick.

  As shown in FIG. 15, when the robot 90 is moved using the conventional input operation means, the user can continuously move the robot by manual operation so as to appropriately capture a desired portion of the target. There is a need. In this case, the user must make fine position adjustments in order to obtain a desired video or the like. Furthermore, even after moving to a position where a desired portion of the target can be photographed, the robot 90 must be constantly moved manually.

  On the other hand, according to the second embodiment, the user only needs to select a desired region and does not need to move the robot 90 in a manual operation. In addition, once the region is selected and the robot 90 is moved, the robot 90 in the display screen is dragged (or, for example, the robot 90 is selected by touching the screen with a finger or the like on the screen. It is possible to make fine adjustments by moving (tracing).

  FIG. 16 is an explanatory diagram for explaining a case where a robot 90 configured to move between teaching points designated in advance is moved. As shown in FIG. 16, when moving between registration points designated in advance, for example, movement between C-A indicated by arrow R1 and movement between A-B indicated by arrow R2 are possible. In this case, in order to move the robot 90 from the current position C to a position close to the target, it is necessary to first specify the point A and then specify the point B to move the robot 90 to a position close to the target. There is. However, when finely adjusting the position, orientation, etc. thereafter, it is necessary to perform a manual operation as in the case shown in FIG.

  On the other hand, according to the second embodiment, the user can easily move the robot 90 to a desired position and direction simply by selecting a predetermined area using the auxiliary information represented by a rectangular grid or concentric circles as a guide. Is possible.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, in the above embodiment, the process of selecting an object of interest to the user using the touch panel provided on the display screen has been described as an example, but the present invention is not limited to such an example. For example, an object to be noticed may be changed in order by a key operation. For example, every time the user presses a key, the display of the target object and auxiliary information in the screen changes.

  In the above-described embodiment, a global bird's-eye view video has been described as an example of the entire video. However, the present invention is not limited to this example, and any video that can obtain global information may be used. For example, global information may be acquired using a 360 degree camera.

  In the above embodiment, the movement and direction change in the two-dimensional plane are performed. However, the present invention is not limited to this example, and the movement and direction change in the three-dimensional space are performed. Also good.

  Further, an object of interest to the user may be registered in advance, or may be registered and added to the database while the user is using the display control device. Furthermore, the device for acquiring video and audio may be provided in one device, or may be provided exclusively in separate devices.

  For example, each step in the processing of the display control devices 100 and 200 of the present specification does not necessarily have to be processed in time series in the order described in the flowchart. That is, each step in the processes of the display control devices 100 and 200 may be executed in parallel even if they are different processes.

  In addition, it is possible to create a computer program for causing hardware such as CPU, ROM, and RAM incorporated in the display control devices 100 and 200 to perform the same functions as the components of the display control devices 100 and 200 described above. is there. A storage medium storing the computer program is also provided.

DESCRIPTION OF SYMBOLS 100 Display control apparatus 102 Operation assistance information display part 103 Operation part 104 Operation command part 105 Interest object selection part 106 Image sound acquisition part 107 Image sound presentation part 108 Whole image | video acquisition part 109 Interest object detection part 110 Interest object information acquisition part 111 Interest Target Database 200 Display Control Device 901 Robot Control Unit 902 Image Sound Acquisition Unit 903 Operation Unit 904 Information Presentation Unit 905 Robot Position Acquisition Unit 906 Operation Auxiliary Information Display Unit

Claims (14)

An overall video acquisition unit that acquires the entire video captured by the imaging device;
An object designating unit that designates an object of interest to the user from a plurality of objects according to a user operation;
A target information acquisition unit that acquires target information related to the target specified by the target specification unit;
Based on the object information of the object acquired by the object information acquisition unit, auxiliary information for acquiring peripheral information of the object is superimposed on the entire image acquired by the entire image acquisition unit An auxiliary information display unit to
A display control apparatus comprising:   The display control according to claim 1, further comprising: a peripheral video acquisition unit that acquires a peripheral video of the target image captured by an imaging device positioned around the target object different from the imaging device. apparatus.   The peripheral video acquisition unit acquires a peripheral video of an object imaged by an imaging device capable of imaging the selected range selected using the auxiliary information superimposed on the entire video in response to a user operation. The display control apparatus according to claim 2, wherein:   The target information acquisition unit acquires target information including position information, direction, and size of the target based on the video of the target included in the entire video acquired by the video acquisition unit. The display control apparatus according to claim 1, wherein:   The display control device according to claim 4, wherein the auxiliary information display unit determines a type of auxiliary information to be superimposed on the entire image according to position information, a direction, and a size of the object. .   The display control apparatus according to claim 1, wherein the auxiliary information display unit superimposes a rectangular grid on the entire image as the auxiliary information.   The display control apparatus according to claim 1, wherein the auxiliary information display unit superimposes a concentric lattice centered on the object on the entire image as the auxiliary information. An auxiliary information storage unit that stores the type of auxiliary information corresponding to the object;
The said object information acquisition part acquires the kind of the said auxiliary information corresponding to the target object with which the said user specified by the said object designation | designated part is interested from the said auxiliary information storage part, It is characterized by the above-mentioned. The display control apparatus according to 1.   The display control apparatus according to claim 1, wherein the auxiliary information display unit selects a type of the auxiliary information in accordance with a user operation and superimposes the auxiliary information on the entire image. A marker capable of delivering the position, direction and ID of the object is arranged on the object,
The display control apparatus according to claim 1, wherein the object information acquisition unit acquires a position and a direction of the object based on information on the object distributed from the marker.   The display control device according to claim 1, wherein the peripheral video acquisition unit acquires a peripheral video of the object captured by an imaging device provided in a movable body that is movable. An operation control unit for controlling the operation of the moving body;
The operation control unit moves the moving body to a position indicated by a selection range selected using the auxiliary information superimposed on the entire video according to a user operation. The display control apparatus according to 1.   The said auxiliary information display part changes the display of the said auxiliary information according to the arrangement | positioning relationship between the said mobile body and the said object designated by the said object designation | designated part, The said auxiliary | assistant information display part is characterized by the above-mentioned. Display control device. A display control method for superimposing predetermined auxiliary information on an entire video imaged by an imaging device,
Obtaining a whole image captured by the imaging device;
In response to a user operation, specifying a target that the user is interested in from a plurality of targets;
Obtaining object information relating to the specified object;
Superimposing auxiliary information for obtaining peripheral information of the object on the whole image obtained by the whole image obtaining unit based on the object information of the obtained object;
A display control method comprising:

Images