KR102639050B1 - Vr image content providing ststem - Google Patents

Abstract

A VR image content provision system is disclosed.
A VR image content providing system according to an embodiment includes a memory and a processor, wherein the processor receives user input and messages for a predetermined area of the VR image, and selects one of a plurality of objects included in the VR image. The message may be stored in response to an object corresponding to the predetermined area, and a message configured to be output by mapping the message input by the user to the corresponding object may be output.

Description

VR image content provision system {VR IMAGE CONTENT PROVIDING STSTEM}

The present invention relates to a VR image content service provision system.

In general, VR (virtual reality) is a technology that implements a three-dimensional virtual reality that makes users feel as if they are in that location through a three-dimensional effect using panoramic images in cyberspace on a computer, tablet PC, or smartphone. In particular, through user interactivity such as dragging the mouse or touching the touch screen, the 3D image rotates in the direction of the drag or touch, giving the feeling of actually being there.

It can be used in a variety of fields, including individuals who share their products on social networking service websites, advertising companies to promote products, schools to deliver realistic and lively educational content, and museums to display artifacts or works of art. You can find it.

Republic of Korea Patent Publication No. 10-2022-0073684 (published on June 3, 2022)

The present invention can provide a VR image content providing system that can be registered by matching a message input by a user to objects of VR images input in a grid-shaped map.

A VR image content providing system according to an embodiment includes: memory; and a processor, wherein the processor receives a user input and a message for a predetermined area of the VR image, and stores the message in response to an object corresponding to the predetermined area among a plurality of objects included in the VR image. , a message configured to be output by mapping the message input by the user to the corresponding object can be output.

The processor performs a first labeling process based on Dense Prediction on the pixels of the VR image, performs a second labeling process based on Object Detection on the VR image, and provides a second labeling process result and a second labeling process. Based on the results, the message can be stored corresponding to the object.

The processor sets a segmentation range for each object based on the second labeling processing result, and within the range, determines a segmentation border of each object based on the first labeling processing result, and , the message can be stored based on the determined border.

A message corresponding to each object may be output to the outside based on a specific point predetermined for each object.

The message may be configured to be additionally written in the form of a comment to the previously entered user message.

A method of providing VR image content according to an embodiment includes receiving a user input and message regarding a predetermined area of a VR image; storing the message in response to an object corresponding to the predetermined area among a plurality of objects included in the VR image; and outputting a message configured to map and output a message input by the user to the corresponding object.

The VR image content providing system of the present invention can register a message input by a user by matching it to objects in VR images, so that other users can more easily recognize the message by distinguishing it from messages about other objects. .

1 is a block diagram for explaining a VR image content providing system according to an embodiment of the present invention. Figure 2 is a block diagram for explaining a processor of a VR image content providing system according to an embodiment of the present invention.
Figure 3 is a diagram for explaining a VR image output to a user terminal.
Figure 4 is a flowchart illustrating a method of providing VR image content according to an embodiment of the present invention.
5A to 5D are diagrams for explaining a segmentation method of a VR image content providing system according to an embodiment.
Figures 6A to 6C are diagrams for explaining the process of inputting a user message into a VR image output to a user terminal.
FIG. 7 is a diagram illustrating the structure of a grid map provided when a VR image is registered in a grid format according to an exemplary embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following examples are presented to sufficiently convey the idea of the present invention to those skilled in the art. The present invention is not limited to the embodiments presented herein and may be embodied in other forms. In order to clarify the present invention, the drawings may omit illustrations of parts unrelated to the description and may exaggerate the sizes of components somewhat to aid understanding.

The embodiments described in this specification and the configurations shown in the drawings are only preferred examples of the disclosed invention, and at the time of filing this application, there may be various modifications that can replace the embodiments and drawings in this specification.

The same reference numbers or symbols shown in each drawing of this specification indicate parts or components that perform substantially the same function. The shapes and sizes of elements in the drawings may be exaggerated for clarity.

Terms used herein are used to describe embodiments and are not intended to limit and/or limit the disclosed invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “include” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. The existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not excluded in advance.

Terms containing ordinal numbers, such as “first,” “second,” etc., used in this specification may be used to describe various components, but the components are not limited by the terms, and the terms refer to one It is used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term “and/or” includes any combination of a plurality of related stated items or any of a plurality of related stated items.

The terms “top”, “bottom”, “top”, “bottom”, etc. used below are defined based on the drawings, and the shape and location of each component are not limited by these terms.

Like reference numerals refer to like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present invention pertains is omitted.

The terms 'unit, module, and device' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'units, modules, and devices' may be implemented as one component, or a single 'unit, It is also possible for a 'module or device' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

In an embodiment of the present invention, “device” refers to a device that can collect, read, process, process, store, and display data, such as general-purpose servers, dedicated servers, desktop computers, laptop computers, smartphones, VR viewers, and game consoles. means any means of calculation. In particular, the devices in the embodiment of the present invention can execute software written in decipherable code and have the function of displaying and delivering it to the user. Additionally, if necessary, the software can be stored internally or read together with data from outside.

In addition, the devices in the embodiment of the present invention include not only data processing functions but also functions such as input, output, and storage. For this purpose, the CPU, motherboard, graphics card, hard disk, sound card, and In addition to various elements such as speakers, keyboards, mice, monitors, USB, and communication modems, server computers have CPUs, motherboards, graphics chips, memory chips, sound engines, speakers, touchpads, external connection terminals such as USB, and communications. It may include an antenna, a communication modem capable of implementing communications such as 3G, LTE, LTE-A, 5G, WiFi, and Bluetooth. These various elements can implement one or more functions singly, two or more together, or parts of various elements combined, and in the embodiments of the present invention, a device shown as one or two or more blocks in the drawings or detailed description or The parts may mean that the above various elements are singly or more together, or parts of the various elements are combined to represent one or more functions.

Meanwhile, the devices of the embodiment of the present invention may have a communication function, and in order to implement the communication function, wired Internet, wireless Internet, infrared communication, Bluetooth, WCDMA, Wibro, WiFi, LTE, LTE-A, 5G, wired and wireless telephone network, etc. Various network means can be provided.

1 is a block diagram for explaining a VR image content providing system according to an embodiment of the present invention. Figure 2 is a block diagram for explaining a processor of a VR image content providing system according to an embodiment of the present invention.

Referring to FIGS. 1 and 2 , the VR image content providing system 100 may be an information processing device including a processor 120 and a memory 140 . For example, the electronic device including the VR image content providing system 100 may include an electronic device for performing various operations and/or control. As an example, the information processing device may include at least one of devices capable of processing information, such as a desktop computer, a laptop computer, or a server computer.

The VR image content providing system 100 according to an embodiment of the present invention may be output in the form of an application to the user terminal of a user who wishes to use the VR environment, but is not limited to this.

According to one embodiment, the processor 120 may perform computation, processing, control, and/or program interpretation to execute the overall operation or partial operation of the VR image content providing system 100.

According to one embodiment, the VR image content providing system 100 of the present invention stores the message entered by the user in the object selected by the user in the VR image, and outputs the stored message in a pattern corresponding to the object in the VR image. It can refer to a system or device that can Depending on the embodiment, VR images may be pre-stored in the form of a grid. In this case, the VR image content providing system 100 preferentially provides a grid map including a plurality of VR images to the user, and selects a grid map from the provided grid map. The series of operations described above can be performed based on the object pattern selected by the user. The detailed operation of the VR image content provision system will be described later.

According to one embodiment, the processor 120 may perform a predetermined operation by driving a pre-embedded program and/or a program stored in the memory 140. Here, the program stored in the memory 140 may be stored directly from the designer, or may be obtained through an electronic software distribution network accessible through a wired or wireless communication network. According to an exemplary embodiment, the processor 120 includes a central processing unit (CPU), a micro controller unit (MCU), a micro processor (Micom), and an application processor (AP). , it can be implemented using an electronic control unit (ECU) and/or other electronic devices capable of various calculation processing and generation of control signals. According to some embodiments, the processor 120 may be implemented using one physical device, or may be implemented using two or more physical devices.

According to one embodiment, the memory 140 may be a storage medium that temporarily or non-temporarily stores various data required for the operation of the processor 120.

According to one embodiment, the memory 140 may include at least one of a main memory device or an auxiliary memory device. The main memory device may be implemented using a semiconductor storage medium such as read-only memory (ROM) and/or random access memory (RAM). Auxiliary storage devices include solid state drives (SSDs), flash memory devices, hard disks (HDDs), SD (Secure Digital) cards, magnetic drums, magnetic tapes, compact disks (CDs), and DVDs. ), it may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as a laser disk, magneto-optical disk, and/or floppy disk.

According to one embodiment, the processor 120 includes a grid map providing unit 122 (which may be an element that can be optionally provided depending on the VR image storage type), an image providing unit 124, and a message storage unit ( 126), and may include a message output unit 128. According to one embodiment, the above-described components 122, 124, 126, and 128 may be implemented in software or hardware.

The processor 120 of the VR image content providing system 100 of the present invention includes a receiver (not shown) that receives user input and messages corresponding to selection of a predetermined area of the VR image, and a plurality of VR images to the user. a grid map providing unit 122 (optionally provided) that provides a grid map, an image providing unit 124 that provides a VR image to the user, and a message storage unit that stores the message entered by the user in the object pattern selected by the user. (126), and a message output unit 128 that outputs the stored message to the object pattern of the VR image.

Figure 3 is a diagram for explaining a VR image output to a user terminal.

Referring to FIG. 3, the VR image output to the user terminal shows a plurality of objects (or object patterns) 44 and a user message 45 displayed on each object (or object pattern) 44. . An object pattern may refer to a form in which each object is segmented and has a border.

A user using the VR environment can select any one object 44. The user can input a user message for any one object 44. The user message may be a review, comment, or explanation about the corresponding object 44. The input method of the user message may be selected by the user so that the user's personal information is exposed, or may be entered anonymously.

The message storage unit 126 can store messages entered by the user. The message storage unit 126 may store the message entered by the user in the memory 140 along with the segmentation value of the object pattern of the corresponding object.

The message output unit 128 can display messages stored in the memory 140 having the segmentation value of the corresponding object pattern on the object pattern of the corresponding object, and the displayed message can be displayed on the VR image output from the user terminal. .

Figure 4 is a flowchart illustrating a method of providing VR image content according to an embodiment of the present invention.

Referring to FIG. 4, the processor may receive a user input and message for a predetermined area of the VR image in step S410.

In step S420, the processor may store a message input in response to an object corresponding to a predetermined area selected by the user among a plurality of objects included in the VR image. More specifically, the processor can apply the CV Segmentation algorithm to the VR image, divide each area of the VR image into a plurality of segments, and assign a segment value to a predetermined area selected by the user. The processor can store the input message along with the segment value assigned to the object. The specific method of assigning segment values is explained in more detail through the accompanying drawings below.

In step S430, the processor may map the message input by the user to the corresponding object and output it. In the process of mapping a message to an object and outputting it, the processor may output a comment by aligning the message at a predetermined location determined to be representative of the object. Through this, even if the point selected for the same object is different, a more readable interface can be provided by providing a message through a preset point for the object.

5A to 5D are diagrams for explaining a segmentation method of a VR image content providing system according to an embodiment.

According to one embodiment, the processor 120 may pattern-process the input VR image. For example, pattern processing may refer to the process of forming a pattern to distinguish objects by processing a VR image according to a segmentation algorithm (eg, CV Segmentation algorithm).

As shown in FIG. 5A, first, the processor 120 may perform pixel labeling (or labeling) processing on the VR image. For example, for the VR image 27, pixel labeling 28 can be performed primarily through Dense Prediction based on Depth Prediction. More specifically, depending on the distance of the object corresponding to the pixel of each image, each pixel may be labeled with a different identification element. For example, in the presented drawing, the pixel corresponding to a plant may be identified as 3, and the pixel corresponding to a person may be identified as 1.

As shown in FIG. 5B, the processor 120 may secondarily perform image labeling processing on the VR image on which labeling processing has been performed. For example, a VR image with pixel labeling processing can be subjected to image labeling processing for objects through instance segmentation based on Object Detection. Referring to FIG. 5B, labeling processing (29, 30) using patterns for a plurality of objects is shown.

As shown in FIG. 5C, the processor 120 may organize the coverage of a plurality of segmentations of a VR image that has undergone image labeling processing according to the object. For example, the range of segmentation of objects can be determined, and the segmentation range can be determined based on the previous secondary labeling result. Additionally, the processor 120 may determine the boundaries of the segmentation of objects within a previously set range by pixel labeling of the VR image that has undergone primary pixel labeling (result in FIG. 5A). Accordingly, the edges 31 and 32 of segmentation of objects may be determined. Through this, the processor 120 determines the segmentation range of a larger area through secondary labeling, and refers to the primary labeling for areas that require detailed review corresponding to the border, thereby increasing segmentation efficiency and improving accuracy. It can be raised.

As shown in FIG. 5D, the processor 120 uses pixel labeling of the VR image that has undergone primary pixel labeling for the remaining portions of the VR image, except for the portion where the edges 31 and 32 of the segmentation of objects are determined. The remaining part can be segmented.

As shown in Figure 5d, a VR image with confirmed segmentation can be used as a segmentation map through the border of the confirmed segmentation. Meanwhile, an object with a confirmed segmentation border may be defined as having an object pattern (33, 34). For example, through the segmentation results through the previous process, the VR image content providing system can provide the user with results in which each object is distinguished, as shown in Figure 3, and the message registration in Figure 4 described above through the segmentation results. method can be implemented.

Meanwhile, Figure 5a provides a photograph with a different background from Figures 5b to 5d, but the present invention is implemented through a photograph of the same background as the process of Figures 5a to 5d, and Figure 5a is a photograph for convenience of explanation. You can.

Accordingly, a user message input by a user may be output on another user's user terminal and the corresponding information may be shared. Additionally, the user message may be configured to be additionally written in the form of a comment to the previously entered user message, thereby providing a communication forum for the corresponding object.

Meanwhile, the user message output by the message output unit 128 may be controlled to be exposed in the central area of the object, but is not limited to this. However, messages corresponding to objects may not be provided scatteredly, but may be provided in response to specific points of the object. Meanwhile, when there are multiple messages stored in the object pattern, the message output unit 128 outputs a message determined according to various algorithms, such as the number of message clicks by users or the number of recommendations, to be displayed in the object pattern before clicking the object pattern. You can also control the message output so that the remaining messages are output by clicking on the corresponding object pattern.

Figures 6A to 6C are diagrams for explaining the process of inputting a user message into a VR image output to a user terminal.

Referring to FIG. 6A, a message input window 47 corresponding to the VR image area (or object) selected by the user is shown in a pop-up state.

Referring to Figure 6b, the message input window 48 is shown popping up in which the user inputs a message (there are so many boxes). When the user completes message input, he or she can click the registration button in the message input window 48 to end message input.

Referring to FIG. 6C, the message 48 input by the user is displayed on the VR image 46. Additionally, referring to FIG. 6C, the VR image 46 shows a state in which a button 49 for returning to the grid map is displayed. If the user clicks button 49, the user can return to the grid map showing the base image, grid pattern, and grid blocks. The method in which the message 48 is mapped and stored to the VR image 46 may be the same as the method described in the previous drawing.

Additionally, referring to FIG. 6C, a state in which grid patterns and grid blocks are displayed in the form of a mini map 50 is shown. Among them, the orange grid block indicates the location of the grid block on the grid pattern of the VR image the user is currently viewing.

FIG. 7 is a diagram illustrating the structure of a grid map provided when a VR image is registered in a grid format according to an exemplary embodiment of the present invention.

Referring to FIG. 7, a grid map generated based on registration of a VR image may include a basic image 41, a grid pattern 42, and a grid block 43. The grid map can be used to provide the mini map 50 shown in FIG. 6C.

The basic image 41 may be a basic image (such as a floor plan of a space) input by a user to implement a VR environment.

The grid pattern 42 may be a grid pattern 42 that partitions grid blocks 43 containing coordinates selected by a user to implement a VR environment.

The grid map generated corresponding to the VR image may be output to the user terminal (e.g., smartphone or VR device) of the user using the VR environment.

A user using the VR environment can select any one grid block 43 from among the plurality of grid blocks 43. When the user selects the grid block 43, the processor 120 may output a pre-stored VR image, and the output VR image may be a VR image having coordinate data of the grid block 43.

The VR image of the selected grid block 43 may be segmented to form object patterns.

The VR image content providing system 100 described above outputs user messages on object patterns, so unlike conventional VR image content providing systems where user messages are displayed sporadically here and there, in the present invention, user messages are displayed on the object pattern. It has the advantage of accurately labeling the intended object.

The VR image content providing system 100 according to an embodiment of the present invention may be output in the form of an application to the user terminal of a user who wishes to implement a VR environment.

The systems and methods described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent. Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

As described above, although the present invention has been described with limited examples and drawings, the present invention is not limited thereto, and the technical idea of the present invention and the following will be understood by those skilled in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope of equivalence of the patent claims to be described.

100: VR image content provision system
120: Processor of VR image content provision system
122: Grid map providing unit
124: Image provider
126: Message storage unit
128: Message output unit

Claims (6)

In the VR image content provision system,
Memory; and
processor
Including,
The processor,
Receive user input and messages for a predetermined area of the VR image,
Storing the message in response to an object corresponding to the predetermined area among a plurality of objects included in the VR image,
Outputs a message configured to map and output a message entered by the user to the corresponding object,
The processor,
Performing a first labeling process based on Dense Prediction on the pixels of the VR image,
Performing a second labeling process based on Object Detection on the VR image,
storing the message corresponding to the object based on a second labeling processing result and a second labeling processing result;

A plurality of messages corresponding to each object are:
Each object is aligned to a specific predetermined position and output to the outside,
The plurality of messages are,
It is configured to be additionally written in the form of a comment to the already entered message,
A plurality of messages corresponding to each object are:
Before each object is clicked, a first message determined based on the number of clicks for each message among the plurality of messages corresponding to each object is output corresponding to each object, The second message corresponding to the messages other than the first message is not output,
When each object is clicked, the second message among the plurality of messages is output in addition to the first message,
A plurality of messages corresponding to each object are:
are collectively aligned in the central area of each object and output to the outside,
The processor,
It further provides a mini-map containing a basic image and a plurality of grid blocks corresponding to the space,
When receiving a user input in which one of the plurality of grid blocks is selected, provide a VR image corresponding to the coordinate data of the selected grid block,
The processor,
Based on the results of the second labeling process, set a segmentation range for each object,
Within the set range, a border of segmentation of each object is determined based on the first labeling processing result, and in the case of an area other than the border, segmentation is performed based on the second labeling result, and the determined A VR image content providing system that stores the message based on the border. delete delete delete delete delete