KR102245760B1 - Table top devices and table top systems - Google Patents

Abstract

According to an embodiment of the present invention, a table top device and a table top system including the same comprise: a table top housing including a bottom surface and an inclined surface having a predetermined slope with respect to the bottom surface; a display system disposed on an inclined surface of the table top housing; an image sensor disposed on the display system; and a processor assembly that generates virtual content for a photographed image based on the photographed image output from the display system and a user input image photographed by the image sensor.

Description

Table top devices and table top systems including the same {TABLE TOP DEVICES AND TABLE TOP SYSTEMS}

The present invention relates to a table top device and a table top system including the same. In more detail, it relates to a table top device providing an augmented reality-based communication system and a table top system including the same.

In general, various facilities for the production of a specific product are arranged in an industrial factory, and a worker in charge of this is arranged in each facility to sequentially perform tasks according to a process required for product production.

At this time, the workers who are in charge of each field facility part in the industrial plant greatly affect the efficiency and productivity of work according to the skill level of the assembly production process in charge of them.

In particular, in the case of an unskilled worker, it may be difficult to assemble a process in a part of the facility in which he or she is in charge, and this has a problem that leads to a decrease in the overall product production, that is, productivity.

In addition, even in the case of skilled workers, there is a problem of not immediately recognizing the change of the work order content, and in particular, there is a problem of time delay as the changed work order content is directly instructed by the manager each time.

In particular, it is difficult for workers to communicate between workers due to the soundproof effect of wearing various items such as work helmets and dustproof clothes, and noise in the work factory makes communication between workers more difficult, which soon leads to a decrease in productivity.

Therefore, in the existing industrial factory, a method of managing the state of the work process was applied by attaching a display to each field facility part and outputting the printout at all times.

However, it is difficult to manage organic processes due to the complex system of each field facility part in the industrial plant.

In other words, the limitations of management due to the increasingly complex system increase in industrial plants, a lot of time is required to create documents for information and sharing, it is difficult to grasp the progress of real-time process work, and communication between workers and managers is a problem. Wow, there was a problem of management limitations due to the redundancy of work that is not tracked and managed, and the change history of the work process.

Meanwhile, with the recent development of information and communication technologies (ICT), information provision technology that provides real-time information on various fields through at least one host server through a long-distance data communication network is active. Is being developed.

In addition, while interest in Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) is rapidly increasing recently, extended reality (XR) has also been added. The market for services and devices that provide users' immersive experience is developing day by day, and in recent years, technologies that apply these technologies to factories in various forms are being developed.

KR 10-2019-0076770 A

An object of the present invention is to provide a table top device that provides an augmented reality-based communication system and a table top system including the same.

However, the technical problems to be achieved by the present invention and the embodiments of the present invention are not limited to the technical problems as described above, and other technical problems may exist.

A table top device and a table top system including the same according to an embodiment of the present invention include: a table top housing including a floor surface and an inclined surface having a predetermined inclination with respect to the floor surface; A display system disposed on an inclined surface of the table top housing; An image sensor disposed on the display system; And a processor assembly for generating virtual content for the captured image based on the captured image output from the display system and the user input image captured by the image sensor.

In this case, the table top housing further includes a camera support extending from one side of the inclined surface to support the image sensor so as to be disposed on the display system.

In addition, the camera support includes a first support formed to extend from the inclined surface in a first direction, a second support formed to extend in a second direction from one end of the first support, and the image at one end of the second support. It includes a camera housing that houses the sensor.

In addition, the camera support is disposed so that the photographing direction of the image sensor faces an upper surface of the display system.

In addition, the processor assembly corrects distortion by performing a warping process on the user input image captured by the image sensor.

In addition, the processor assembly separates a region matching the captured image output through the display system from the user input image, and detects a user's gesture input from the user input image in the separated region.

In addition, the processor assembly may detect a gesture image of the user from the user input image, determine image coordinates of the captured image according to the position of the detected gesture image, and add the detected gesture image to the determined image coordinates. Create virtual content according to.

A table-top device and a table-top system including the same according to an embodiment of the present invention provide a table-top device and a system that implements an augmented reality-based communication system, so that a user input for an image acquired from a remote computing device is provided. It is possible to obtain more conveniently, and by providing virtual content based on this, there is an effect that it is possible to provide guidance more easily and clearly to remote workers.

In addition, in the table top device and the table top system including the same according to an embodiment of the present invention, the camera field of view of the table top device includes all of the display area of the table top device, so that a user input for a taken image as a reference is performed. It has an effect that can be detected all without loss.

However, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood from the following description.

1 is a conceptual diagram of an augmented reality-based communication system according to an embodiment of the present invention.
2 is a conceptual diagram of experiencing an extended reality environment through a wearable type computing device according to an embodiment of the present invention.
3 is a conceptual diagram of experiencing an extended reality environment through a mobile type computing device according to an embodiment of the present invention.
4 is a diagram illustrating a table top device according to an embodiment of the present invention.
5 is a view for explaining in more detail a table top device according to an embodiment of the present invention.
6 is an internal block diagram of a table top device according to an embodiment of the present invention.
7 is an internal block diagram of a wearable type computing device according to an embodiment of the present invention.
8 is an internal block diagram of a mobile type computing device according to an embodiment of the present invention.
9 is a flowchart illustrating a method of performing augmented reality-based communication in a table top device according to an embodiment of the present invention.
10 is an example of receiving and displaying a captured image from a remote computing device according to an embodiment of the present invention.
11 is an example of detecting a gesture image according to a user's gesture input according to an embodiment of the present invention.
12 is an example of transmitting and displaying virtual content to a remote computing device according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used for the purpose of distinguishing one constituent element from other constituent elements rather than a limiting meaning. In addition, expressions in the singular include plural expressions unless the context clearly indicates otherwise. In addition, terms such as include or have means that the features or components described in the specification are present, and do not preclude the possibility of adding one or more other features or components in advance. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, and thus the present invention is not necessarily limited to what is shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted. .

-Augmented reality-based communication system

1 is a conceptual diagram of an augmented reality-based communication system according to an embodiment of the present invention, FIG. 2 is a conceptual diagram of experiencing an extended reality environment through a wearable type computing device according to an embodiment of the present invention, and FIG. A conceptual diagram of experiencing an extended reality environment through a mobile type computing device according to an embodiment of FIG.

1 to 3, a network speed responsive extended reality (XR) communication system (hereinafter, “communication system”) according to an embodiment of the present invention allows users to communicate based on extended reality content through a computing device. It is a system that provides.

Here, the extended reality environment refers to an environment generated by inserting virtual contents related to a physical space around a user or a communication counterpart, and may include an augmented reality (AR) environment and a mixed reality environment (MR). The virtual contents are virtual contents generated by a computing device, including labels, text information, images, drawing objects, and 3D entities, and augmented/mixed generated in response to the physical space or an image photographed in the physical space. It's content.

And XR Communication means that different users exchange communication data for mutual communication through a computing device based on immersive media (eg, video, audio) including the virtual content. Means activity.

That is, the communication system according to the embodiment may support multi-party communication within an extended reality environment by additionally communicating virtual content related to a physical space or a photographed image along with voice and image, which are communication mediums of general communication.

Communication data may include audio, video, and/or virtual content that computing devices of different users (eg, remote managers and field workers) exchange through a network. In an embodiment, communication data may be transmitted/received based on a network by a computing device and/or a server or the like.

In an embodiment, the communication system may provide different communication data communication methods according to the communication environment of the user's computing device, thereby providing an extended reality communication environment optimized for the communication environment.

Meanwhile, in an embodiment of the present invention, such a communication system may include a computing device 100, a table top system, and a communication data relay server system 500 (hereinafter, “server system”).

The computing device 100, the table top system, and the communication data relay server system according to an embodiment of the present invention may interwork with each other to provide an augmented reality-based communication service.

In addition, each component of FIG. 1 may be connected through a network. The network refers to a connection structure in which information exchange is possible between each node such as the computing device 100 and the data relay server system 500, and an example of such a network is a 3GPP (3rd Generation Partnership Project) network, an LTE ( Long Term Evolution) Network, World Interoperability for Microwave Access (WIMAX) Network, Internet, Local Area Network (LAN), Wireless Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN) , A Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and a Digital Multimedia Broadcasting (DMB) network, but are not limited thereto.

In detail, the computing device 100 provides an extended reality environment to a user, and executes an extended reality application capable of communicating with other users using virtual contents corresponding to voice, image, and real objects within this extended reality environment. I can.

The computing device 100 may include a computing device 100 of various types (eg, a wearable type, a mobile type, a desktop type, or a table top type) on which an extended reality application is installed.

In addition, for convenience of description, the roles of the computing device 100 will be classified and described according to the subject, environment, and/or purpose of the computing device 100.

In an embodiment, the computing device 100 may be divided into a first user computing device and a second user computing device. In more detail, the first user computing device is a computing device of a remote administrator located remotely from a work site, and may transmit and receive data to and from the second user computing device based on a network.

In the following description, the first user computing device provides an extended reality environment based on a photographed image received from the field computing device of a field worker, and communicates the intention to the field worker according to the first user input within the provided extended reality interface environment. In order to do so, the description will be based on a computing device of a remote administrator that generates communication data.

Here, the remote manager may be a person who provides work guidance as virtual content for instructing or assisting the work of a corresponding site worker at a location remote from the site where the work is performed.

In addition, the field worker may be a person who performs actual work based on the extended reality environment provided through the work guidance received at the work site.

In the following description, the first user is a remote administrator, and the second user is a field worker.

Meanwhile, in an embodiment, the server system 500 may perform a series of processes for providing a table top service that provides an augmented reality-based communication system.

In detail, the server system 500 may relay the exchange of communication data transmitted and received on a network basis between the user computing devices 100 and a communication repeater.

The server system 500 may include a data relay server, a data processing server, a virtual content database, and a spatial information database.

In detail, the data relay server may include a communication facility for data relay, and may relay communication data to be transmitted and received between the computing devices 100 through a wired or wireless communication network. For example, the data relay server may serve as a relay providing communication data transmitted and received between a computing device of a first user and a computing device of a second user according to a network communication speed.

In addition, the data processing server may perform a role of processing communication data differently according to the network speed. Hereinafter, in an embodiment, the role of processing communication data according to a communication mode according to a network speed will be described as being performed by the computing device 100.

In addition, virtual content data for implementing an augmented reality environment or a mixed reality environment may be stored in the virtual content database. The virtual content database may match the virtual content to an actual object (eg, a marker) or a spatial coordinate and store the virtual content as virtual content.

In addition, the virtual content database may serve as a virtual content source that delivers virtual content matched to the surrounding physical space of the computing device 100 upon request of the computing device 100.

In addition, the spatial information database may store information data on the physical space by scanning a physical space of a specific area or modeling a three-dimensional space. In addition, feature information obtained by image learning of an actual object, a marker, etc. may be further stored in the physical space, and the feature information may be matched with the spatial information and stored.

That is, the server system 500 transmits virtual content data and spatial information data for the surrounding physical space of the computing device 100 together, and provides an augmented reality environment or an extended reality environment through the computing device 100. I can.

The server system 500 may include at least one computing server and a database server, and may include processors for processing data and memories storing instructions for providing a communication relay service.

Hereinafter, a table top system of a communication system will be described in detail with reference to the accompanying drawings.

-Table top system

In an embodiment of the present invention, the table-top system provides a user with an augmented reality environment implemented through a virtual object corresponding to an actual object of a photographed image obtained from a remote computing device 100 and a photographed image obtained from the table-top device. You can implement a remote communication service provided to you.

In detail, in an embodiment of the present invention, the table top system may include a table top device 400, a guide tool 200, and a computing device 100.

1. Table Top Device

First, according to an exemplary implementation, the table top device 400 may be implemented in a tabletop shape of a new shape and structure different from the existing desktop type computing device 100, and in this case, it is included in the table top. The extended reality environment can be provided through the system.

That is, with a tabletop device, a remote administrator can easily check an image received from the field worker's computing device 100, and easily input work guidance for an object displayed in the image based on virtual content. It may be a device that provides an interface system to perform.

That is, the table top device 400 displays an augmented/mixed reality image that displays virtual content generated based on the actual object of the image acquired at the current location on the image acquired from the computing device 100 of another user. It may be a system that creates and provides.

4 is a diagram illustrating a table top device 400 according to an embodiment of the present invention. Further, FIG. 5 is a diagram for explaining in more detail the table top device 400 according to an embodiment of the present invention, and FIG. 6 is an internal block diagram of the table top device 400 according to the embodiment of the present invention.

4 and 5, the table top device 400 according to the embodiment may include a table top housing.

In detail, in the embodiment, the table top housing forms the exterior of the table top device 400 and may include various components necessary for driving the table top device 400 inside/outside.

In more detail, the table top housing includes a bottom surface 409 supporting the table top housing from the ground, an inclined surface 401 having a predetermined angle, and connecting the bottom surface 409 and the inclined surface 401 from the upper surface. It may include a connection surface 404 and a side surface 403 connecting the bottom surface 409 and the inclined surface 401 on left/right sides.

In this case, the predetermined angle of the inclined surface 401 may be changed according to the user's viewing standard.

That is, the bottom surface 409 and the inclined surface 401 of the table top housing may be coupled at predetermined intervals through the connection surface 404 and the side surface 403, and through this Space can be formed.

In addition, a computing device 100 providing an augmented reality-based communication service may be disposed in a predetermined space formed inside the table top housing.

In addition, the table top housing may further include a guide tool box 415 capable of accommodating at least one guide tool 200.

In detail, the guide tool box 415 may be implemented in the form of a box extending from one surface of the table top housing, or may be implemented as a separate unit spaced apart from the table top housing.

The guide tool box 415 may accommodate and store at least one guide tool 201, 202, 203,… implemented in various shapes therein. A detailed description of the guide tool 200 will be described later.

In more detail, referring to FIG. 6, a display system 470 including a display 471 and a touch sensor 473 (shown in FIG. 6) may be disposed on the inclined surface 401 of the table top housing. .

In addition, at the lower end of the inclined surface 401, a support jaw 402 capable of placing or supporting the guide tool 200 and/or other devices may be disposed.

At this time, the support jaw 402 may be implemented in a rectangular protruding shape forming a predetermined angle (eg, 90 degrees) with the inclined surface 401 of the table top housing.

In addition, a camera support 408 may be further disposed on the inclined surface 401 of the table top housing.

In detail, the camera support 408 may play a role of disposing the camera 461, which is the image sensor 461, at a predetermined height or more from the inclined surface 401.

Further, the camera support 408 may be a structure in which the camera 461 and the display 471 face each other while the camera 461 is disposed above a predetermined height from the display 471.

The camera support 408 may be disposed in any area on the inclined surface 401 as long as the camera 461 can be separated from the display 471 by a predetermined distance, but the user may input a gesture or a guide tool 200. In order to most efficiently form a movable area when performing an input based on ), it may be the most preferable embodiment to be positioned above the inclined surface 401, that is, in the uppermost central area of the edge areas of the inclined surface 401. Hereinafter, the description will be limited to an embodiment in which the camera support 408 is disposed on the inclined surface 401.

In more detail, the camera support 408 may include a first support 405, a second support 406, and a camera housing 407.

Here, the first support 405 may be implemented in a bar shape extending from one side (eg, the center) of the top of the inclined surface 401 to the other side, which is a direction spaced apart from the inclined surface 401.

The first support 405 may include a predetermined inner groove providing a passage through which the electric wire L can pass.

In addition, in the embodiment of the present invention, the length of the first support 405 may be adjustable through a predetermined method.

For example, the first support 405 may be implemented as a rod-shaped multi-stage shaft that transmits power through a linear reciprocating motion, and its length may be adjusted. At this time, the length of the first support 405 may be changed by manually adjusting the first support 405 by the user, or based on a user input based on a length adjustment interface for setting the length of the first support 405. The length can also be changed. In addition, the first support 405, according to a predetermined criterion (e.g., a pre-defined essential maintenance distance condition between the camera 461 and the display 471) set in the extended reality application of the table top device 400 The length can also be adjusted automatically.

That is, the first support 405 may adjust its length in a predetermined manner to adjust the distance between the camera 461 and the display 471, and through this, an image output through the display 471 at a later time. And it is possible to assist to smoothly detect all user inputs for the corresponding image.

Meanwhile, the second support 406 may be implemented in a bar shape in which one end is bent at a predetermined angle (eg, vertically) from the first support 405 and extends.

The second support 406 may include an inner groove extending from the inner groove of the first support 405, and like the inner groove of the first support 405, the electric wire L can pass. Can provide a passageway.

In addition, at the other end of the second support 406, a camera housing 407 may be formed in which at least two cameras 461 may be disposed.

Here, when the angle of view of the camera 461 photographing the display 471 is twisted, the corresponding camera 461 cannot face the display 471 in front, and in this case, the image taken by the camera 461 is The screen of the display 471 may be distorted (for example, the screen of the rectangular display 471 may be distorted in a trapezoidal shape).

In preparation for the above case, in the embodiment of the present invention, in the camera housing 407, at least one camera 461 disposed in the camera housing 407 faces the display 471 of the table top device 400 in front. It may be disposed on the second support 406 to form a viewing angle of view.

That is, in the embodiment, the camera housing 407 is arranged to have an angle of view facing the display 471 in front, so that an image output through the display 471 and a user input to the image can be detected without fail. can do.

In another embodiment, when the camera 461 field of view of the camera housing 407 does not face the display 471 of the table top device 400 due to the deformation of the camera support 408, the table top device 400 The extended reality application of) can minimize distortion of the screen of the display 471 by performing a warping process. A detailed description of this will be described later.

In addition, the camera housing 407 as described above has at least one camera 461 therein, and may operate to be opened when a control signal exposing the camera 461 is detected.

Here, the camera 461 included in the camera housing 407 may be a stereo camera 461 including at least two cameras 461-1 and 461-2 spaced apart at a predetermined interval in the embodiment. .

In addition, the camera housing 407 may be formed in a shape in which a width in a direction perpendicular to the length direction of the second support 406 is extended to accommodate the stereo camera 461 as described above.

In an embodiment of the present invention, the table top device 400 including the above components may provide a photographed image acquired from a remote computing device 100 based on the display system 470, and the camera A virtual object corresponding to the real object of the captured image obtained based on 461 can be implemented, and a remote communication service that provides virtual content generated by augmenting the implemented virtual object to the user can be implemented.

Meanwhile, the table top device 400 as described above is illustratively a memory 410, a processor assembly 420, a communication module 430, an interface module 440, an input system 450, a sensor system 460, and A display system 470 may be included. In addition, the components may be implemented to be included in the table top housing.

In detail, in the memory 410 , an extended reality application 411 is stored, and in the extended reality application 411, virtual contents, image buffers, location engines, and virtual contents display 471 engines for providing an extended reality environment are provided. Can be included. That is, the memory 410 may store commands and data that can be used to create an extended reality environment.

In an embodiment, the extended reality application 411 may include a communication application for performing communication based on the extended reality environment. The communication application may include various applications, engines, data, and commands for providing a network speed responsive extended reality communication service.

Further, the memory 410 may include at least one non-transitory computer-readable storage medium and a temporary computer-readable storage medium. For example, the memory 410 may be various storage devices such as ROM, EPROM, flash drive, hard drive, etc., and a web storage that performs a storage function of the memory 410 on the Internet. It may include.

The processor assembly 420 may include at least one processor capable of executing instructions of the extended reality application 411 stored in the memory 410 in order to perform various tasks for creating an extended reality environment.

In an embodiment, the processor assembly 420 may control the overall operation of a component through the extended reality application 411 of the memory 410 in order to provide a network speed responsive extended reality (XR) communication service.

For example, the processor assembly 420 can recognize a real object from an image acquired based on the image sensor 461, and computes to generate and display an augmented reality image matching virtual content with the recognized real object. Components of the device 100 can be controlled.

The processor assembly 420 may include a central processing unit (CPU) and/or a graphic processor unit (GPU). In addition, the processor assembly 420 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and electrical units for performing other functions.

The communication module 430 may include one or more devices for communicating with other computing devices (eg, server system 500). The communication module 430 may communicate through a wireless network.

In detail, the communication module 430 may communicate with a computing device storing a virtual content source for implementing an extended reality environment, and may communicate with various user input components such as a controller that receives a user input.

In an embodiment, the communication module 430 may transmit and receive communication data related to a network speed responsive extended reality (XR) communication service with the server system 500 and/or other computing device 100.

These communication modules 430, technical standards or communication methods for mobile communication (e.g., LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G NR (New Radio), WIFI) Alternatively, data may be wirelessly transmitted and received with at least one of a base station, an external terminal, and an arbitrary server on a mobile communication network established through a communication device capable of performing a short-range communication method.

The sensor system 460 may include various sensors such as an image sensor 461, a position sensor (IMU, 463), an audio sensor 465, a distance sensor, a proximity sensor, and a contact sensor.

The image sensor 461 may capture an image and/or an image of a physical space around the computing device 100.

In an embodiment, the image sensor 461 may acquire and capture an image related to a network speed responsive extended reality (XR) communication service.

In addition, the image sensor 461 may be disposed on the front or/and the rear of the computing device 100 to acquire an image in a photographing direction in the disposed direction, and the camera disposed toward the outside of the computing device 100 Through this, it is possible to photograph a physical space such as a work site.

The image sensor 461 may include an image sensor 461 and an image processing module. In detail, the image sensor 461 may process a still image or a moving picture obtained by the image sensor 461 (eg, CMOS or CCD).

In addition, the image sensor 461 may extract necessary information by processing a still image or a moving image acquired through the image sensor 461 using an image processing module, and transmit the extracted information to the processor.

The image sensor 461 may be a camera assembly including at least one camera. The camera assembly may include a general camera for photographing a visible light band, and may further include a special camera such as an infrared camera and a stereo camera.

The IMU 463 may detect at least one or more of motion and acceleration of the computing device 100. For example, it can be made of a combination of various position sensors such as an accelerometer, a gyroscope, and a magnetometer. In addition, by interlocking with the location communication module 430 such as GPS of the communication module 430, spatial information on a physical space around the computing device 100 may be recognized.

Also, the IMU 463 may detect information for detecting and tracking a user's gaze direction and head movement based on the detected position and direction.

In addition, in some implementations, the extended reality application 411 may use these IMUs 463 and image sensors 461 to determine the location and orientation of the user in the physical space or to recognize a feature or object in the physical space. .

The audio sensor 465 may recognize sound around the computing device 100.

In detail, the audio sensor 465 may include a microphone capable of detecting a voice input of a user of the computing device 100.

In an embodiment, the audio sensor 465 may receive voice data of communication data to be transmitted through an extended reality communication service from a user.

The interface module 440 may connect the computing device 100 to communicate with one or more other devices. In detail, the interface module 440 may include wired and/or wireless communication devices compatible with one or more different communication protocols.

The computing device 100 may be connected to various input/output devices through the interface module 440.

For example, the interface module 440 may be connected to an audio output device such as a headset port or a speaker to output audio.

As an example, it has been described that the audio output device is connected through the interface module 440, but an embodiment installed inside the computing device 100 may also be included.

The interface module 440 connects a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and a device equipped with an identification module. Ports, audio input/output (I/O) ports, video input/output (I/O) ports, earphone ports, power amplifiers, RF circuits, transceivers and other communication circuits It may be configured to include at least one of.

The input system 450 may sense a user's input (eg, gesture, voice command, button operation, or other type of input) related to a network speed responsive extended reality (XR) communication service.

In detail, the input system 450 may include a button, a touch sensor 473, and an image sensor 461 that receives a user motion input.

In addition, the input system 450 may be connected to an external controller through the interface module 440 and may receive a user's input.

In an embodiment, the display system 470 may output various information related to a table top service providing an augmented reality-based communication system as a graphic image.

Such a display 471 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display ( flexible display), a 3D display, and an e-ink display.

In addition, the example table top device 400 outputs a photographed image captured by another computing device 100 through the display system 470 and virtual content related to the photographed image to provide an extended reality environment to the user. I can.

In addition, the table top device 400 may provide an input/output interface for receiving a touch input such as a user's pointing or dragging on the captured image through the touch sensor 473 on the display 471 of the display system 470. have.

In addition, the table top device 400 may receive a gesture input of a user's hand (LH, RH) through a sensor system 460 (eg, an image sensor 461) disposed on the display system 470. In addition, a motion input according to the guide tool 200 (GT) may be received. The gesture input and the motion input of the guide tool 200 may also be input to the display 471 in response to the captured image, and the computing device 100 may detect the user input by matching the actual object image displayed on the captured image. I can.

That is, the extended reality application 411 of the table top device 400 according to the embodiment performs a series of processes for implementing a service that visualizes and provides virtual content based on a user input acquired through the sensor system 460. You can do it. In this case, since the user input is performed corresponding to the captured image displayed on the display 471, by providing an input/output interface together, the user can more intuitively input the user for the captured image.

2. Guide tool

Meanwhile, in the embodiment of the present invention, the guide tool 200 may be a medium for augmenting a virtual object in response to a photographed image displayed on the display 471 of the table top device 400.

In detail, the guide tool 200 is matched with a specific virtual object (e.g., a virtual object representing various tools such as a screwdriver, spanner, or plier) matched with the appearance or identification information such as a marker disposed on the guide tool 200 It can be a real object.

The guide tool 200 may be implemented in various shapes according to the type of virtual object to be specified.

For example, the first guide tool 200 may have a'driver' shape, the second guide tool 200 may have a'spanner' shape, and the third guide tool 200 may have a'plier' shape.

In an embodiment, the guide tool 200 as described above may be detected as an actual object by the camera of the table top device 400. In addition, an augmented virtual object may be generated by interacting according to the detected shape, posture, and positional relationship with the photographed image of the guide tool 200. A detailed description of this will be described later.

Returning again, in order to generate the guide tool 200 information in the embodiment, the table top device 400 may first store identification information of the guide tool 200 in a memory in advance.

Here, the identification information is information for identifying the guide tool 200, and is feature information obtained by learning an image of the appearance of the guide tool 200 or/and feature information of a marker attached to the guide tool 200. I can.

For example, the table top device 400 may include identification information of the first guide tool 200 having a'driver' shape, identification information of the second guide tool 200 having a'spanner' shape, or Identification information of the third guide tool 200 having a'flyer' shape may be acquired and stored.

To this end, the table top device 400 acquires by taking a picture of the guide tool 200 through a camera to learn feature information of the appearance, or by learning feature information of a marker disposed on the guide tool 200. I can.

In addition, the table top device 400 may previously store information on at least one virtual object matching the identification information of the guide tool 200 in a memory.

Here, the virtual object may be an object modeled based on an actual shape related to a specific object (e.g., a virtual shape of various tools such as a screwdriver, a spanner, or a plier) symbolized by the shape of the guide tool 200 and visualized virtually. have.

Such a virtual object may be matched with the identification information of the corresponding guide tool 200 and stored and managed in a memory.

For example, the table top device 400 may store a first virtual object, a second virtual object, and a third virtual object matching the identification information of the first guide tool 200.

At this time, when the first guide tool 200 is in the shape of a'flyer', the first virtual object is a virtual object modeled based on the actual shape of the'pliers', which is a kind of pliers, and the second virtual object is another object of the pliers. The virtual object modeled based on the actual shape of the'Long Rose Flyer', which is a type, and the third virtual object, may be virtual objects modeled based on the actual shape of the'Vice Flyer', which is another type of the flyer.

In addition, the table top device 400 may generate the guide tool 200 information including identification information of the guide tool 200 and virtual object information matching the identification information of the guide tool 200 as described above. In addition, the table top device 400 may store and manage the generated guide tool 200 information in a memory.

As described above, in the embodiment, the table top device 400 stores a plurality of virtual objects matching the guide tool 200 having various shapes, so that the virtual object based on the guide tool 200 is visualized on an image. The provided service can be implemented.

3. Computing device

In an embodiment of the present invention, the computing device 100 provides an extended reality environment to a user, and in this extended reality environment, an extension capable of communicating with other users using virtual contents corresponding to voice, image, and real objects You can run real applications.

The computing device 100 may include a computing device 100 of various types (eg, a wearable type, a mobile type, a desktop type, or a table top type) on which an extended reality application is installed.

1. Wearable type computing device (101, 102)

2 is a conceptual diagram illustrating an extended reality environment through wearable type computing devices 101 and 102 according to an embodiment of the present invention, and FIG. 7 is a wearable type computing device 101 and 102 according to an embodiment of the present invention. ) Is an internal block diagram.

2 and 7, the computing devices 101 and 102 according to the embodiment may include wearable type computing devices 101 and 102 such as a smart glasses display or a head-mounted display (HMD). I can.

The smart glass-type computing device 101 is a display system including a glass that transmits light so that the user can see the surrounding physical space while being worn and displays virtual content (eg, virtual object image) on the user's field of view. (470) may be included.

In detail, the computing device 101 of the embodiment is a transparent glass display that transmits light from the surrounding physical space to reach the user's eyes and reflects the virtual content displayed by the display system 470 toward the user's eyes. (471) may be included.

For example, referring to FIG. 2, the extended reality application of the computing device 101 can image recognition of a real object (RO) and a marker (MK) within the real object in the surrounding physical space 20, and It is possible to control to display the virtual content VC1 in the user's field of view corresponding to the marked marker MK.

In addition, the extended reality application may recognize the learned real object and control to display the virtual content VC2 in the user's field of view corresponding to the position of the recognized real object.

In addition, the extended reality application may recognize a physical space through a learned real object or a marker MK, and may display virtual content matched to a specific location of the recognized space so as to correspond to the user's field of view.

Such virtual content may include visual content such as an image or video that may be displayed 471 in a portion of the user's field of view on the computing device 101. For example, the virtual content may include virtual object images that overlay various parts of a physical space. This virtual object image can be rendered as a 2D image or a 3D image.

The computing device 102 of the head-mounted display type may block light to the surrounding physical space so that the displayed image can be viewed only by the display system 470. The head-mounted display type computing device 102 may output a 3D image by outputting different images offset by parallax to the left and right eye displays 471 in order to recognize a 3D scene.

In addition, the head-mounted display type computing device 102 may also provide an extended reality environment by outputting an image of a surrounding physical space and a virtual content generated based on the captured image as a 3D image.

Hereinafter, specific components will be described centering on the smart glass-type computing device 101 among such wearable-type devices.

Referring to FIG. 7, a computing device 101 according to an exemplary implementation includes a memory 110 including an extended reality application, a processor assembly 120, a communication module 130, an interface module 140, an input system ( 150), a sensor system 160 and a display system 470.

In this case, the components may be implemented to be included in the housing of the computing device 101. Among the descriptions of the components of the wearable type computing devices 101 and 102, the overlapping content will be replaced with the description of the above-described components, and the differences will be described below.

Referring to FIG. 7, the display system 470 transmits light from the physical space around the computing device 101 to reach the user's eyes, and at the same time transmits the virtual content displayed by the display system 470 to the user's eyes. It may include a transparent glass display 471 that reflects toward.

The display system 470 may include a left display 171 corresponding to the left eye of a user wearing the computing device 101 and a right display 172 corresponding to the right eye, and the left display 171 and The right display 172 outputs different images offset by parallax as virtual content, so that the user can recognize the virtual content as a 3D image.

These wearable type computing devices 101 and 102 may be advantageous for use by field workers located in a physical space such as a work site.

2. Mobile type computing device (103)

3 is a conceptual diagram of experiencing an extended reality environment through a mobile-type computing device 103 according to an embodiment of the present invention, and FIG. 8 is an internal block diagram of a mobile-type computing device 103 according to an embodiment of the present invention. It is a degree.

In another example, the computing device 103 may be a mobile device such as a smart phone or a tablet PC in which an extended reality application is installed. The mobile type computing device 103 captures an image of the surrounding physical space with an image sensor 461, and displays the image captured through the display system 470 and virtual content matched to the physical space to display extended reality. The environment can be provided to the user.

For example, the computing device 103 may include a smart phone, a mobile phone, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, and the like.

Referring to FIG. 4, the extended reality application of the mobile type computing device 103 captures an image of a real object (RO) and a marker (MK) in the surrounding physical space 20 and controls the display system 470 to display it. can do. In addition, the extended reality application may control to display the virtual content VC1 at a location corresponding to the recognized marker MK. In addition, the extended reality application may learn and recognize a specific real object, and control to display the virtual content VC2 in the user's field of view corresponding to the location of the recognized specific real object.

Referring to FIG. 8, a mobile type computing device 103 according to an exemplary implementation includes a memory 210, a processor assembly 220, a communication module 230, an interface module 240, an input system 250, and a sensor. System 260 and display system 270. These components may be configured to be contained within a housing of computing device 103.

Among the descriptions of the components of the mobile-type computing device 103, the overlapping content will be replaced with a description of the components of the wearable-type computing devices 101 and 102. Hereinafter, the wearable-type computing devices 101 and 102 ) And the difference between them.

The components may be disposed within the housing of such a mobile type computing device 103, and the user interface may include a touch sensor 273 on a display 271 configured to receive a user touch input.

In detail, the display system 270 may include a display 271 for outputting an image and a touch sensor 273 for sensing a user's touch input.

For example, the display 271 may be implemented as a touch screen by forming a layer structure or integrally with the touch sensor 273. Such a touch screen may function as a user input unit that provides an input interface between the computing device 103 and a user, and may provide an output interface between the computing device 103 and the user.

In addition, the sensor system 260 includes an image sensor 261, for example, the image sensor 261 may be disposed on one side and the other side of the housing of the computing device 103.

At this time, the image sensor 461 on one side is oriented toward the physical space to capture an image of the physical space, and the image sensor 461 on the other side is oriented toward the user to control the user's field of view and gestures. You can shoot.

The mobile type computing device 103 may be suitable for field workers located in a physical space such as a work site.

-How to perform augmented reality-based communication on a tabletop device

Hereinafter, a method of performing augmented reality-based communication in the table top device 400 according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings. For convenience of description, the second user computing device will be described based on the mobile type computing device 103.

9 is a flowchart illustrating a method of performing augmented reality-based communication in the table top device 400 according to an embodiment of the present invention.

Referring to FIG. 9, in an embodiment of the present invention, the table top device 400 may execute an extended reality application for performing augmented reality-based extended reality communication with a second user computing device.

The executed extended reality application may measure the distance between the camera of the table top device 400 and the display 471. (S101)

For example, the extended reality application is the distance between the projection origin of the first camera 461-1 and the second camera 461-2 of the stereo camera disposed in the camera housing 407, the first camera 461-1 ) And a distance between the projection origin of the display 471 and a point of the display 471 and the distance between the projection origin of the second camera 461-2 and a point of the display 471 can be calculated, and based on these, the table top device 400 ), the distance between the camera and the display 471 can be calculated.

In addition, the extended reality application may determine whether the camera is disposed within a predetermined area based on the measured distance between the camera and the display 471. (S103)

Here, the predetermined area may be an area in which the input image captured by the camera includes all of the display area 471 of the table top device 400.

In this case, the extended reality application may adjust the position of the camera when the camera is disposed outside a predetermined area. (S105)

In detail, the extended reality application can adjust the position of the camera housing 407 by controlling the length of the first support 405 of the table top device 400 based on a user input or an automatic control method according to a preset standard. have. Through this, the extended reality application may allow the camera built in the camera housing 407 to be located within a predetermined area.

For example, the extended reality application may provide a length adjustment interface for setting the length of the first support 405 implemented as a rod-shaped multistage shaft that transmits power through a linear reciprocating motion, and a user based on the provided interface. The position of the camera may be adjusted within a predetermined area by changing the length of the first support 405 based on the input.

As another example, the extended reality application may automatically adjust the length of the first support 405 according to a preset predetermined criterion (eg, a required maintenance distance condition between a predefined camera and the display 471), and this Through this, the position of the camera can be changed within a predetermined area.

Here, the length of the first support 405 may be adjusted by a user manually adjusting the first support 405.

As described above, in the embodiment, the extended reality application adjusts the distance between the camera and the display 471 so that the camera is placed in a predetermined area, so that the captured image output through the display 471 and the user's Any input can be detected without loss.

On the other hand, in the extended reality application in the embodiment of the present invention, when the angle of view of the camera photographing the display 471 is twisted, that is, due to the deformation of the camera support 408, the camera angle of view of the camera housing 407 is When the display 471 of the device 400 is not viewed in front, the extended reality application of the table top device 400 performs a warping process to minimize the distortion of the screen of the display 471 of the captured image. have.

In detail, for example, the extended reality application may first separate the area occupied by the display 471 of the table top device 400 in the captured image obtained from the camera through image analysis. In this case, the separated area may have a distorted display 471 shape (eg, trapezoidal) different from the original display 471 shape (eg, a rectangle) due to a distorted camera angle of view.

In addition, the extended reality application modulates the image for the separated area (eg, rotates the x/y axis and/or adjusts the scale, etc.) to normalize the original display 471 to a warping process. You can do it.

In this way, the extended reality application provides a function of minimizing distortion to the screen image of the display 471 when the camera angle of view is distorted, so that even if the camera angle of view does not face the display 471 in front, the display in the captured image It is possible to optimize to detect the 471 area as the original shape of the display 471.

Returning again, on the other hand, in the embodiment of the present invention, the extended reality application may receive and display a photographed image captured from a field computing device when the camera is disposed within a predetermined area. (S107)

10 is an example of receiving and displaying a photographed image from a remote computing device 100 according to an embodiment of the present invention.

In detail, referring to FIG. 10, the extended reality application may receive a basic photographed image acquired based on an image sensor 461 of a field computing device located remotely through the communication module 430.

In addition, the extended reality application may output the received basic photographed image through the display 471.

That is, the extended reality application may receive and display an image captured by a field computing device and share it in real time.

In addition, in an embodiment of the present invention, the extended reality application may detect a user input based on the displayed photographed image. (S109)

In detail, the extended reality application may detect a user input according to a gesture input.

Here, the gesture input may be a user's input indicating a specific body part (eg, hand) of the user or a movement (motion) based on the guide tool 200.

In more detail, the extended reality application may acquire a user input image by first photographing a physical space between the camera and the display 471.

In addition, the extended reality application may separate a region matching the display 471 from the acquired user input image.

For example, an extended reality application can perform image preprocessing on a user input image captured through a camera, and perform image segmentation and clustering, so that the background and the foreground are (foreground) can be separated. Thereafter, the extended reality application may detect and separate an object, that is, an area of the display 471 from the foreground separated by the image segment.

11 is an example of detecting a gesture image according to a user's gesture input according to an embodiment of the present invention.

In addition, referring to FIG. 11, the extended reality application may acquire a user gesture input within a separated display 471 area, and may detect a gesture image according to the acquired gesture input.

Here, the gesture image may be information obtained by analyzing an image of a user's gesture input and forming a corresponding gesture into an image.

For example, the extended reality application provides a gesture input representing a movement (movement) based on a user's specific body part (eg, hand) or the guide tool 200 by a neural network, a support vector machine (SVM), or By analyzing an image using a Histograms of Oriented Gradients (HOG) technique, etc., a gesture image for a gesture input can be obtained.

In addition, the extended reality application may acquire a user input for a captured image displayed based on the acquired gesture image.

In this case, the extended reality application may also obtain a user input according to a user's touch input based on the touch sensor 473 of the display system 470.

In this case, the extended reality application determines the two-dimensional coordinates (x, y) for the user input on the basic photographed image obtained from the field computing device in order to obtain an accurate user input, and then the image coordinates of the basic photographed image and By matching, a user input for a captured image displayed on the display 471 may be obtained.

As described above, in the embodiment, the extended reality application may more conveniently acquire a user input based on the table top system.

In addition, in an embodiment of the present invention, the extended reality application may generate and display virtual content in which a virtual object is augmented in a basic photographed image based on the detected user input. (S111)

Here, the virtual content is virtual content generated by the computing device 100, including labels, text information, images, drawing objects, and 3D entities, and is generated in response to the physical space or an image photographed in the physical space. This is augmented/mixed content.

In detail, the extended reality application may detect a virtual object corresponding to the detected user input, and generate virtual content based on the detected virtual object.

In an embodiment, the extended reality application may receive a gesture input of a user's hand (LH, RH) and/or a motion input according to a guide tool (GT) based on a user input.

In this case, the extended reality application may detect at least one real object, that is, the guide tool 200, from a user input based on a gesture image.

In detail, the extended reality application may first detect at least one real object based on a gesture image acquired from a user input image.

For example, an extended reality application may perform image preprocessing on an image captured through a camera, and perform segmentation and clustering to provide a background and a foreground. Can be separated. Thereafter, the extended reality application may detect an object (ie, an actual object) from the foreground separated by the image segment.

In addition, in an embodiment, the extended reality application may determine whether the detected real object is the guide tool 200.

For example, an extended reality application may classify and verify objects separated by image segments (ie, real objects). In detail, the extended reality application uses a neural network identification method, a support vector machine (SVM) technique, a technique identification by AdaBoost using Haar-like features, or a Histograms of Oriented Gradients (HOG) technique. Thus, the guide tool 200 may be identified from the detected at least one or more actual objects.

Thereafter, in the embodiment, in the extended reality application, when the detected real object is determined as the guide tool 200, a virtual object corresponding to the guide tool 200 is superimposed on the basic photographed image obtained from the field computing device. Can be displayed.

In this case, the extended reality application may continuously track the movement (eg, position movement, gesture, and/or motion) of the guide tool 200 photographed based on the camera.

Through this, the extended reality application may cause the virtual object displayed by being superimposed on the corresponding guide tool 200 to perform a corresponding movement according to the movement of the guide tool 200 in the future.

In addition, the extended reality application may determine the overlapping position of the virtual object by reflecting the position of the guide tool 200 with respect to the basic photographed image displayed on the display 471 in order to deliver accurate guidance.

In detail, the extended reality application may determine the two-dimensional coordinates (x, y) of the guide tool 200 on the basic photographed image, and then match the image coordinates of the basic photographed image to determine the overlay position of the virtual object.

In addition, the extended reality application is an augmented reality that can adjust the perspective of the virtual object by changing the size of the virtual object corresponding to the guide tool 200 according to the distance z between the basic photographed image and the guide tool 200. Interface environment can be provided.

In addition, in the embodiment, the extended reality application can continuously track the movement of the guide tool 200 displayed as virtual content (eg, position movement, gesture and/or motion, etc.), and guide When the tool 200 operates in a predetermined manner, a virtual object displayed by being superimposed on the corresponding guide tool 200 may also be operated in the same manner.

For example, in the extended reality application, when the guide tool 200 performs rotational motion, a virtual object matched to the corresponding guide tool 200 may perform rotational motion in the same manner.

To this end, the extended reality application may obtain information on 6 degrees of freedom of the guide tool 200 by analyzing an image photographed by the guide tool 200. In addition, the previously stored virtual object may be pre-designed to change its shape in response to a change of 6 degrees of freedom of the guide tool 200. Here, the six degrees of freedom may be posture estimation information obtained by measuring six rotational and translational motion elements of an object moving in a three-dimensional space.

In addition, the extended reality application may match and store image changes of a virtual object with respect to feature movements for each guide tool 200.

For example, in the case of the plier-type guide tool 200, a pinch-shape and a pinch-off state can be externally formed, and a characteristic operation of moving into two states may be possible.

The extended reality application may acquire information on the feature operation when acquiring identification information by learning the flyer-type guide tool 200.

In addition, the extended reality application may store the virtual object image to change according to the feature operation of the guide tool 200.

Through this, the extended reality application may generate a virtual object such that, when a feature motion of the guide tool 200 is detected in an image captured by the guide tool 200, the image of the virtual object is also matched and changed according to the feature motion.

In addition, the extended reality application may designate a reference point for each guide tool 200 and store a change in a virtual object image corresponding to the reference point.

Here, the reference point means at least one point on the guide tool 200 that is a reference for estimating the posture of the guide tool 200. This reference point may be used as a reference for augmenting a virtual object interacting with the guide tool 200 when implementing augmented reality based on the guide tool 200.

For example, the reference point is a method of designating a pen tip point as a reference point when the guide tool 200 has a'pen' shape, and augmenting a virtual line according to the movement of the pen tip into a virtual object. A virtual drawing augmented reality environment can be provided. As another example, the reference point is an augmented reality in which, when the guide tool 200 is in the shape of a'driver', a position is designated as a reference point at the tip of the driver, and the tip rotates together according to the rotation of the driver. It can provide an environment.

In this extended reality application, since it is difficult to directly recognize 6 degrees of freedom of the reference point of the guide tool 200 in the image, the marker is recognized on the guide tool 200 and the reference point is 6 degrees of freedom can be estimated.

Through this, the extended reality application, when moving the pen-shaped guide tool 200 on the photographed image, estimates 6 degrees of freedom of the reference point of the guide tool 200, and generates a virtual line segment according to the 6 degrees of freedom from the estimated reference points. , It is possible to generate an augmented reality virtual object according to the guide tool 200.

As such, in the embodiment of the present invention, the extended reality application provides a work method using an object (eg, a driver, etc.) specified by the guide tool 200 to a worker performing a work remotely based on a virtual content. Guidance can be made easily and clearly.

Returning again, the extended reality application that generated virtual content in an embodiment of the present invention may transmit the generated virtual content to the field computing device. (S113)

12 is an example of transmitting and displaying virtual content to a remote computing device 100 according to an embodiment of the present invention.

In detail, referring to FIG. 12, the extended reality application may transmit and display virtual content generated from the table top device 400 to a field computing device located remotely.

In this case, the extended reality application may extract and transmit only a virtual object included in the virtual content, or may transmit an augmented image generated by augmenting the virtual object to the basic photographed image.

That is, in the embodiment, the extended reality application selectively generates and provides virtual content with either a virtual object or an augmented image, so that only the virtual object can be transmitted to improve the efficiency of data communication, or the augmented image is transmitted and exchanged. It can also improve the accuracy of the data.

As described above, the table top device 400 and the table top system including the same according to an embodiment of the present invention provide a table top device 400 and a system implementing an augmented reality-based communication system, thereby providing a remote computing device ( It is possible to more conveniently obtain a user input for an image obtained from 100), and by providing virtual contents based on this, there is an effect that it is possible to provide guidance more easily and clearly to a remote operator.

In addition, in the table top device 400 and the table top system including the same according to an embodiment of the present invention, the camera angle of view of the table top device 400 includes all of the display area 471 of the table top device 400. By doing so, there is an effect of being able to detect all user inputs for a reference photographed image without loss.

Further, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. medium), and a hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be changed to one or more software modules to perform the processing according to the present invention, and vice versa.

Specific implementations described in the present invention are examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects 403 of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections that can be replaced or additionally It may be referred to as a connection, or circuit connections. In addition, if there is no specific mention such as “essential” or “importantly”, it may not be an essential component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, the spirit of the present invention described in the claims to be described later if one of ordinary skill in the relevant technical field or those of ordinary skill in the relevant technical field And it will be understood that various modifications and changes can be made to the present invention within a range not departing from the technical field. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

Claims (7)

A table top housing including a bottom surface and an inclined surface having a predetermined inclination with respect to the floor surface;
A display system disposed on an inclined surface of the table top housing;
An image sensor disposed on the display system;
A processor assembly for generating virtual content for the captured image based on a captured image output from the display system and a user input image captured by the image sensor,
The processor assembly,
Based on the distance between the first camera included in the image sensor and the first point of the display system, and the distance between the second camera included in the image sensor and the first point of the display system, Calculate the distance,
The processor assembly,
Based on the calculated distance between the image sensor and the display system, it is determined whether the image sensor is disposed in a predetermined area including all areas of a captured image captured by the image sensor and output from the display system,
The processor assembly,
When the image sensor is disposed outside the predetermined area, the position of the image sensor is adjusted by controlling the table top housing,
The processor assembly,
Separating a region matching the photographed image output from the display system from the user input image photographed by the image sensor,
Detecting a user's gesture input in the user input image corresponding to the separated area,
Detecting a gesture image representing an image matching the gesture input based on the sensed gesture input,
Providing virtual content for the captured image based on the detected gesture image
Table top device. The method of claim 1,
The table top housing,
Further comprising a camera support extending from one side of the inclined surface to support the image sensor to be disposed on the display system
Table top device. The method of claim 2,
The camera support,
A first support formed to extend from the inclined surface in a first direction, a second support formed to extend in a second direction from one end of the first support, and a camera housing accommodating the image sensor at one end of the second support. Inclusive
Table top device. The method of claim 3,
The camera support,
Arranged so that the photographing direction of the image sensor faces the top surface of the display system
Table top device. The method of claim 1,
The processor assembly,
Correcting distortion by performing a warping process on the user input image captured by the image sensor
Table top device. delete The method of claim 1,
The processor assembly,
Determining image coordinates of the captured image according to the position of the detected gesture image,
Generating the virtual content according to the detected gesture image at the determined image coordinates
Table top device.

Images