KR20230054236A - Apparatus and method for providing an image object corresponding to image information of an object - Google Patents

Abstract

The present invention relates to a method of operating a server for providing augmented reality content which comprises the steps of: storing, by a server, reference image information about a reference thing; receiving, by the server, first image information about a first thing from a first user terminal; receiving, by the server, an image object related to the first image information from the first user terminal; calculating, by the server, a first probability index that the first thing is the same as the reference thing based on the first image information; storing, by the server, when the first probability index satisfies a predetermined first condition, the image object in a database subordinate to the reference thing; receiving, by the server, second image information about a second thing from a second user terminal; calculating, by the server, a second probability index that the second thing is the same as the reference thing based on the second image information; and providing, by the server, when the second probability index satisfies a predetermined second condition, the image object to allow the second user terminal to combine the image object with the second image information and output augmented reality content. Accordingly, indirect interaction with other users can be performed.

Description

Apparatus and method for providing an image object corresponding to image information of an object}

The present invention relates to augmented reality technology, and more particularly, to an apparatus and method for providing an image object corresponding to image information of an object through augmented reality technology.

BACKGROUND As terminals such as smart phones and tablet computers equipped with high-performance cameras become popular, it is easy to take high-quality pictures or images of surrounding objects. With the development of these technologies, it has become possible to create and share visual information of objects more specifically and abundantly. In addition, it has become possible to quickly and conveniently upload captured images and related contents to a server through the Internet. However, what a user receives through a camera provided in such a terminal is limited to realistic visual information. For example, in order to learn more about objective information such as the function and use of the object or subjective information such as the evaluation of other people about the object, the process of searching for the information of the object through a portal site should be accompanied. . There is a need to obtain various and rich visualized information based on image information directly acquired through a user terminal, but there is no service that implements this until now.

Republic of Korea Patent No. 10-2153990

An object of the present invention is to check other users' specifically visualized information about a corresponding object only with image information acquired about the object using augmented reality technology.

According to an embodiment of the present disclosure, a method of operating a server for providing augmented reality content includes storing, by the server, reference image information for a reference object; receiving, by the server, first image information about a first object from a first user terminal; receiving, by the server, an image object related to the first image information from the first user terminal; calculating, by the server, a first probability indicator indicating that the first object is identical to the reference object based on the first image information; storing, by the server, the image object in a database subordinate to the reference object when the first probability indicator satisfies a first predetermined condition; receiving, by the server, second image information about a second object from the second user terminal; calculating, by the server, a second probability indicator indicating that the second object is the same object as the reference object based on the second image information; And when the second probability indicator satisfies a predetermined second condition, the server provides the image object so that the second user terminal outputs the image object as augmented reality content by combining the image object with the second image information Include steps.

Also, the image object includes comment information related to the first image information.

The providing of the image object may further include providing the second user terminal with information about how the image object is output to the second user terminal.

Also, the information on the output method may be provided by the first user terminal.

The server may further include generating information about the output method based on the reference object and the second image information, wherein the information about the output method includes at least one image previously stored in the server. Can be created based on the output type.

The providing of the image object may include determining, by the server, an area where the image object can be output to the second user terminal based on the second image information; and adjusting the size of the image object based on the determined outputable area.

The providing of the image object may include determining, by the server, an area where the image object can be output to the second user terminal based on the second image information; and changing information about a method in which the image object is output based on the determined outputable area.

In addition, when a plurality of image objects exist in the database, the server determining a priority among the plurality of image objects based on comment information corresponding to each of the plurality of image objects; and providing, by the server, the image object to the second user terminal based on the determined priority.

In addition, the image object includes user identification information associated with the first user terminal.

According to an embodiment of the present disclosure, an apparatus of a server for providing augmented reality content includes: a communication unit; store; and at least one processor operatively connected to the communication unit and the storage device, wherein the at least one processor performs steps of: storing, by the server, reference image information about a reference object; receiving, by the server, first image information about a first object from a first user terminal; receiving, by the server, an image object related to the first image information from the first user terminal; calculating, by the server, a first probability indicator indicating that the first object is identical to the reference object based on the first image information; storing, by the server, the image object in a database subordinate to the reference object when the first probability indicator satisfies a first predetermined condition; receiving, by the server, second image information about a second object from the second user terminal; calculating, by the server, a second probability indicator indicating that the second object is the same object as the reference object based on the second image information; And when the second probability indicator satisfies a predetermined second condition, the server provides the image object so that the second user terminal outputs the image object as augmented reality content by combining the image object with the second image information It can be configured to perform steps.

An apparatus and method for providing an image object corresponding to image information of an object of the present invention obtains information generated by another person in relation to a corresponding object based on image information acquired about the object, thereby indirectly interacting with other users. It has the advantage of enabling interaction.

1 illustrates an exemplary computing system 100 according to one embodiment of the present disclosure.
2 illustrates a server in a computing system according to an embodiment of the present disclosure.
3 illustrates a client device within a computing system according to an embodiment of the present disclosure.
4 illustrates a sequence of operations of a server in a computing system according to an embodiment of the present disclosure.
5 illustrates an operation of each component in a computing system according to an embodiment of the present disclosure.
6 illustrates an operation sequence of a server in a computing system according to an embodiment of the present disclosure.
7 illustrates information transmitted from a client device to a server in a computing system according to an embodiment of the present disclosure.
8 illustrates an operation of creating an image object by a client device in a computing system according to an embodiment of the present disclosure.
9 illustrates an example in which an image object is implemented in a client device according to an embodiment of the present disclosure.
10 illustrates another example in which an image object is implemented in a client device according to an embodiment of the present disclosure.
11 illustrates another example in which an image object is implemented in a client device according to an embodiment of the present disclosure.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, each step described can be performed regardless of the listed order, except for the case where it must be performed in the listed order due to a special causal relationship.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, the present invention will be described with reference to the accompanying drawings.

1 illustrates an exemplary computing system 100 according to one embodiment of the present disclosure. The embodiment of the computing system 100 shown in FIG. 1 is for illustration only. Other embodiments of computing system 100 may be used without departing from the scope of the present disclosure.

As shown in FIG. 1 , system 100 includes network 102 , which facilitates communication between various components within system 100 .

Network 102 facilitates communication between at least one server 104 and various client devices 106-114. Each server 104 includes any suitable computing or processing device capable of providing computing services to at least one client device.

Each client device 106 - 114 represents any suitable computing or processing device that interacts with at least one server or other computing device(s) over network 102 . In this example, some client devices 108-114 communicate indirectly with network 102. For example, client devices 108-110 communicate through at least one base station 116, such as cellular base stations or eNodeBs.

Although FIG. 1 illustrates one example of a computing system 100 , various changes may be made to FIG. 1 . For example, system 100 may include any number of each component in any suitable arrangement. Computing and communication systems generally come in a variety of configurations, and FIG. 1 does not limit the scope of the present disclosure to any particular configuration. 1 illustrates one operating environment in which various features disclosed in this patent document may be used, these features may be used in any other suitable system.

2 and 3 illustrate example devices within a computing system according to one embodiment of the present disclosure. In particular, FIG. 2 illustrates an exemplary server 200 and FIG. 3 illustrates an exemplary client device 300 . Server 200 may represent server 104 of FIG. 1 , and client device 300 may represent one or more client devices 106 - 114 of FIG. 1 .

In the present disclosure, the server 200 may be a server for providing augmented reality content. The server 200 may acquire augmented reality content through the network 102 of FIG. 1 , process augmented reality content stored in the storage devices 215 of the server 200, or transmit the augmented reality content to the outside. In the present disclosure, the client device 300 may generate or process augmented reality content, and transmits the generated or processed augmented reality content to at least one other device selected from the server 200 or the client devices 106-114. can be sent to the client device. In the present disclosure, a client device may be referred to as a 'user terminal'. When there are a plurality of client devices, they may be classified into 'first user terminal' and 'second user terminal'.

As shown in FIG. 2 , the server 200 performs communication between at least one processor 210, at least one storage device 215, at least one communication unit 220, and at least one input/output unit 225. It includes a bus system 205 to support.

Processing unit 210 executes instructions that may be loaded into memory 230 . Processing device 210 may include any suitable number and type(s) of processors or other devices in any suitable configuration.

Memory 230 and persistent storage 235 are any structures that can store and facilitate retrieval of information (eg, data, program code, and/or other suitable information on a temporary or permanent basis). s) are examples of the storage devices 215 . Memory 230 may represent random access memory (RAM) or any other suitable volatile or non-volatile storage device(s). The permanent storage device 235 may include at least one component or device that supports long-term storage of data, such as a read only memory (ROM), hard drive, flash memory, or optical disk.

The communication unit 220 supports communication with other systems or devices. For example, communication unit 220 may include a network interface card or wireless transceiver that facilitates communication over network 102 . The communication unit 220 may support communication over any suitable physical or wireless communication link(s).

An input/output (I/O) unit 225 allows input and output of data. Input/output unit 225 may also send output to a display, printer, or other suitable output device.

Although FIG. 2 has been described as representing the server 104 of FIG. 1, it is noted that the same or similar architecture may be used in one or more of the client devices 106-114. For example, a laptop or desktop computer may have the same or similar structure as shown in FIG. 2 .

As shown in FIG. 3, the client device 300 includes an antenna 305, a radio frequency (RF) transceiver 310, a transmit (TX) processing circuit 315, and a microphone 320. and receive (RX) processing circuitry 325 . In addition, the client device 300 includes a speaker 330, a main processor 340, an input/output (I/O) interface (IF) 345, a keypad 350, a display ( 355) and a memory (360). The memory 360 includes a basic operating system (OS) program 361 and at least one application 362 .

The RF transceiver 310 receives the received RF signals transmitted by other components of the system from the antenna 305 . Receive processing circuitry 325 transmits the processed baseband signal (equivalent to voice data) to speaker 330 or main processor 340 for further processing (equivalent to web browsing data).

Transmit processing circuitry 315 receives analog or digital voice data from microphone 320 and other outgoing baseband data (web data, e-mail, or interactive video game data) from main processor 340.

The main processor 340 may include at least one processor or other processing devices, and may execute a basic OS program 361 stored in the memory 360 to control the entire operation of the client device 300.

Main processor 340 is also coupled to keypad 350 and display unit 355 . An operator of the client device 300 may use the keypad 350 to enter data into the client device 300 . Display 355 may be a liquid crystal display or a display capable of rendering text and/or at least limited graphics from a website. According to an embodiment of the present disclosure, image information about an object detected through an image sensor may be displayed on the display 355 . In another example, image information may be real-time information displayed on display 355 .

2 and 3 show examples of devices within a computing system, various changes may be made to FIGS. 2 and 3 . For example, the various features of FIGS. 2 and 3 can be combined, subdivided, or omitted and additional features can be added according to particular needs. 3 shows a client device 300 configured as a mobile phone and a smart phone, however, the client devices may be configured as other types of mobile or stationary devices. Also, like computing and communications networks, client devices and servers can have a variety of configurations, and FIGS. 2 and 3 do not limit the present disclosure to any particular client device or server.

4 illustrates a sequence of operations of a server in a computing system according to an embodiment of the present disclosure.

In step 401, the server stores reference image information for a reference object.

A reference thing is a thing that can be specified based on at least one piece of information, and may refer to all things that can be expressed from information stored in a server or obtained from the outside by the server. For example, when an object is a 'mobile phone', a reference object for the object may be a 'mobile phone' that can be identified by information such as a manufacturer, model, appearance, and the like.

Reference image information may refer to a set of information extractable from a corresponding object in order to use the object as a reference object. More preferably, the information extractable from the object may be visual information about the object. For example, when an object is a 'mobile phone', the reference image information for the object may include photos or videos of the 'mobile phone' taken at various angles and in various sizes.

The step of storing the reference image information for the reference object by the server is a criterion for comparing the information on each with the reference object when first and second image information (including other image information) to be described later are obtained. It can be used to set and extract the corresponding attribute. Accordingly, in general, step 401 may precede all operations in this disclosure, or may be omitted. Step 401 may be omitted in an environment in which the server can acquire reference objects, reference image information, or information related thereto from an external server, electronic device, or wireless communication network.

In step 403, the server receives first image information about the first object from the first user terminal.

The first user terminal may be one of the client devices 106 - 114 of FIG. 1 or the client device 300 of FIG. 2 , and may be an electronic device including an image sensor for obtaining image information. For example, the first user terminal may be a 'smartphone' carried by the first user, and in this case, the image sensor may be a 'camera'.

The first image information may be image information about a specific object obtainable by the user through the first user terminal, and in this case, the specific object may be referred to as the first object. The first image information may be visual information obtained from the first user terminal. The first image information may be a photograph of one side of the first object or a video file in which part or the entirety of the first object is recorded in a predetermined frame. For example, a user may photograph a specific object (first object) in front of the user's eyes through his or her smartphone (first user terminal). In this case, the user may obtain image information of the first object through an application installed in the first user terminal and connected to the server, and the obtained image information may be transmitted to the server (in real time). Here, the 'acquisition of image information' operation, which is exemplified by capturing a specific object by a smartphone, includes not only directly initiating photo taking or video recording, but also displaying a real-time image of the object on the display screen of the first user terminal. should be understood as

In step 405, the server receives an image object related to the first image information from the first user terminal.

The image object may be information generated about the first object from the perspective of a user using the first user terminal. The image object may be visual information related to image information obtained by the client device. The image object may be text, picture, or animation drawn or created by the user using the client device.

The operation of receiving the image object to the server may include an operation of the user inputting the image object to the first user terminal through a web browser or an application connected to the server. Subsequently, the first user terminal may transmit the input image object to the server through the network.

When the server receives the image object, the server may identify the image object as related to the first image information rather than to the first object, and may store the image object. More specifically, the first user terminal or server may determine that the image object is related to the first image information when the user inputs the image object within a predetermined time after acquiring the first image information.

In step 407, the server calculates a first probability indicator that the first object is the same object as the reference object based on the first image information.

The server may check whether a reference object corresponding to the object corresponding to the first image information received through the first user terminal exists. That is, since the server cannot specify what the first object is based only on the first image information, the server may use the stored information. Specifically, the server may inquire whether reference image information corresponding to the first image information exists in storage devices of the server or is included in data available to the server. When it is determined that the corresponding information exists, the server may determine a reference object corresponding to the corresponding reference image information, and may estimate that the determined reference object is the same object as the first object. However, since the first image information and the stored reference image information cannot be completely identical, the degree of identity can be calculated based on a predetermined probability indicator.

The server may compare the first image information acquired through the first user terminal with the reference image information corresponding to the first image information, and determine whether a probability that both are identical is greater than or equal to a predetermined probability indicator. In this case, a probability indicator that the first image information and the corresponding reference image information are identical may be referred to as a first probability indicator. Similarly, a probability indicator that the first object and the reference object are identical may also be referred to as a first probability indicator. The concept of the first probability indicator may be mixed as needed, or may be used differently when the number of image information to be compared is plural. The first probability indicator may be a numerical value predetermined by the server or may be changed based on a control command of an external server or a value input by a user.

In step 409, if the first probability index satisfies the first condition, the server stores the image object as subordinate to the reference thing.

The first condition may be a value of a minimum probability index capable of recognizing the first object and the reference object as the same, and satisfying the first condition means that the value of the first probability index exceeds the first condition. can do. In this case, the server may determine the first thing and the reference thing as the same thing. Specifically, the server may determine that the first thing corresponds to any one of at least one reference thing stored in the server.

For example, when the first image information is a 'front image of a mobile phone of company B', the server may find a specific reference image that satisfies the identity requirement (first condition) with the corresponding image among at least one stored reference image. there is. Since the reference object corresponding to the found specific reference image is the 'mobile phone of company B', the server may determine that the first object is the 'mobile phone of company B'. However, if a specific reference image that satisfies the identity requirement is not found, the server determines that the image information transmitted by the user through the first user terminal cannot be recognized by the server, and the server sends a corresponding message to the first user. can be transmitted to the terminal.

When the reference thing is specified according to the satisfaction of the identity requirement, the server may store the image object as subordinate to the reference thing. Storing as dependent may mean that an image object is related to a corresponding reference thing and is in a state in which the image object can also be transmitted or processed according to a request of a terminal related to the reference thing.

The server may create a database corresponding to at least one reference object in a storage device of the server. Here, the database may include a database for storing image objects subordinate to corresponding reference objects. Therefore, in step 409 An image object stored according to satisfaction of the first condition may be stored in such a database. However, the operation of storing the image object in the database is exemplary, and the server transmits the image object to the first user terminal by transmitting a request message for the image object corresponding to the reference object to an external storage device. It is possible, and at this time, the control command for transmitting the request message may be stored in the memory.

In step 411, the server receives second image information about a second object from a second user terminal.

The second user terminal may refer to a device that transmits second image information after acquiring first image information from the first user terminal. Accordingly, the first user terminal and the second user terminal are only conceptually distinguished but may be physically the same, and the users of the first user terminal and the second user terminal may also be the same person.

The second object may be physically the same as or of the same type as the first object, or may be completely different from the first object. The second image information may be a photograph of one side of the second object or a video file in which part or the whole of the second object is recorded in a predetermined frame.

An operation for the server to receive the second image information of the second object may be performed similarly to step 403 . Accordingly, the second user terminal may transmit second image information to the server through a web browser connected to the server or an application installed in the second user terminal.

For example, a user of a second user terminal who is curious about other consumers' evaluation or price information of 'drinks', which is a random object, acquires second image information on 'drinks' through the second user terminal and transmits the second image information to a server. can be sent to

In step 413, the server calculates a second probability indicator that the second object is the same object as the reference object based on the second image information.

Similar to step 408, the server may check whether reference image information corresponding to the obtained image information exists in the server. That is, by comparing the second image information with at least one stored reference image information, reference image information that satisfies the same condition may be searched for. Through this, it is possible to check whether a reference thing corresponding to the second thing exists.

For example, when second image information on one side of 'drink', which is an arbitrary object, is transmitted to the server, the server may search for reference image information within a range of identity with a partial image of the corresponding drink. Similar to step 408, the criterion of the search may be whether a predetermined probability index for identity is satisfied.

In step 415, if the second probability indicator satisfies the second condition, the server provides the second user terminal to output the image object as augmented reality content.

The second condition may be a value of a minimum probability index capable of recognizing the second object and the reference object as the same, and satisfying the second condition means that the value of the second probability index exceeds the second condition. can do. In this case, the server may determine the second thing and the reference thing as the same thing. Specifically, the server may determine that the second thing corresponds to any one of at least one reference thing stored in the server.

When the server recognizes the second object and the reference object as the same, the server may provide an image object subordinate to the reference object from a database corresponding to the reference object. Here, the image object may include an image object stored in step 409 . When there are a plurality of image objects subordinate to the reference object, all or part of the subordinate image objects may be provided to the second user terminal.

More specifically, the server may provide the second user terminal to output the image object to the display. For example, the server may transmit to the second user terminal a control message commanding an image object to be output on a display together with the image object. More preferably, the second user terminal may itself control the output of the image object through an application connected to the server and installed in the second user terminal.

An operation of outputting an image object to the second user terminal may be performed while the user is viewing real-time information of the second object through a camera or the like on a display of the second user terminal. In this case, the second user terminal may display the image Objects can be output as augmented reality content. Augmented reality content output here may be a combination of an image object and second image information. For example, the second image information is an image of a second object implemented on a display of a second user terminal, and the image object may be expressed as an augmented reality image around the image of the implemented second object.

5 illustrates an operation of each component in a computing system according to an embodiment of the present disclosure.

A computing system according to an embodiment of the present disclosure may include a first user terminal 510 , a second user terminal 530 and a server 550 . The above components included in the computing system may each perform the following operations as a subject.

- The server stores the reference 501.

The reference may be information including both reference object and reference image information. The reference may be generated based on image information acquired from a user terminal or may be information obtained in advance through an external server, electronic device, or network. The reference may be object-related information generated based on all information capable of specifying a thing in order for the computing system according to the present disclosure to provide a service for augmented reality content. An operation in which the server stores the reference may be continuously performed even while a subsequent operation is performed.

- The first user terminal transmits (502) the first image information to the server.

The first image information may be image information about the appearance of the first object. The first image information may be first image information received about the first object in a state in which the server has not acquired an image object of the first object.

- The first user terminal transmits the image object to the server (503).

The image object may be subjective or objective information generated by the user about the first object. Since it is difficult to specify the first object only with the image object itself, generally, first image information is first received, and after the first object is specified accordingly, the image object may be utilized. However, the user may transmit the image object and the first image information together or transmit them in a different order by selection. This may be controlled through an application installed in the first user terminal and connected to the server.

- The server determines identity based on the received information and stores it (504).

Here, the received information may be first image information and an image object. However, identity determination may be performed based on the first image information. Specifically, the object of identity determination may be the first image information and the reference image information. What is stored may be an image object. That is, when the server compares the first image information and the reference information and determines that they are identical, the image object may be stored in the server. In this case, the image object may be stored as subordinate to the reference object corresponding to the reference image information of the same range as the first image information.

- The second user terminal transmits (505) second image information to the server.

An operation of receiving the second image information by the server may be performed later by the server after the image object is stored. The second image information may be image information about the appearance of the second object. The second object may be the same as, similar to, or of the same kind as the first object. The same kind may mean that the first object and the second object are in a state in which they can be recognized as conceptually the same when comprehensively considering the same manufacturer, model, appearance, etc., even though they are physically different. An operation of transmitting the second image information may be performed similarly to an operation in which the first user terminal transmits the first image information to the server.

- The server determines (506) identity based on the received information.

Here, the received information may mean second image information. The server may determine whether the second image information is identical to at least one of the reference image information stored in the server. When the server determines that reference image information within the identity range corresponding to the second image information exists, the server may specify a reference object corresponding to the corresponding reference image information.

- The server transmits the image object to the second user terminal (507).

When the reference thing corresponding to the second image information is specified, the server may transmit the image object corresponding to the reference thing to the second user terminal. Here, when the first object and the second object are the same, similar, or of the same kind, the image object transmitted to the second user terminal may include an image object related to the first image information.

- The second user terminal outputs (508) augmented reality content based on the image object.

Specifically, the second user terminal may generate augmented reality content based on the image object and output it to a display screen. For example, augmented reality content may be created by combining the second image information already displayed on the display screen with the image object received from the server. Creation and output of augmented reality content may be performed inside the second user terminal or based on a control command received from the server.

6 illustrates an operation sequence of a client device in a computing system according to an embodiment of the present disclosure.

Hereinafter, the operations of FIG. 6 are exemplified as operations performed by a client device, but according to another embodiment of the present disclosure, when the client device passively performs the operation of FIG. 6 based on a control or data message received from the server. It should also be understood to include.

In step 601, the client device creates an image object.

The image object may be visual information related to image information obtained by the client device. The image object may be text, picture, or animation drawn or created by the user using the client device. An example of a process in which a user creates an image object using a client device is as follows.

First, a user may obtain real-time image information of a specific object through a camera provided in a client device. Thereafter, the user can emphasize or explain image information of a specific object, express his or her feelings about the specific object, or create a user's own image regardless of the specific object. Here, the user's own image may be a picture or text that the user draws or writes through an input means provided in the client device.

For example, when a specific object is “a headset from company A”, the user may highlight a part (eg, a cushion part) of the headset in which he or she is particularly interested, and put a highlight around the “cushion part”. In another example, the user may create historical information regarding personal ratings, scores, etc. for the headset. In another example, a user may draw a character image to decorate the headset's periphery. At this time, the highlight mark, record information, and character image may be image objects in the present disclosure.

In another embodiment of the present disclosure, prior to generating an image object, the client device may determine how to create the image object based on a user's input. For example, the client device may provide a message inquiring whether the user directly creates an image object or selects one of pre-stored image objects.

When the user determines to directly create an image object, the client device may switch to an image object input mode so that the user can draw or write the image object according to the user's input. In this case, a user may create an image object such as a picture, a highlight, or a handwriting through an input means exemplified by a display of a client device. At this time, image information about a specific object displayed on the display of the client device may be continuously output in real time or as a still image.

When the user determines to select one of the pre-stored image objects, the client device may output information about the plurality of pre-stored image objects according to the user's input. At this time, the plurality of output image objects may be exposed at the top according to priorities determined by considering factors such as user's preference, frequency of selection, relevance to a specific object, and the like. When a user selects one of a plurality of pre-stored image objects through a client device, the client device may create the selected image object.

In step 603, the client device determines the display type of the image object.

The display type may include information about how the image object created in step 601 is output on the client device. Here, the client device that determines the display type of the image object and the client device that outputs the corresponding image object may be different devices.

A method for outputting the image object may be determined by a method in which a user using the client directly produces or selects one of a predetermined type. For example, after creating an image object, the user can determine how closely the image object will be displayed with image information or how much weight will be displayed on the output screen of the client device that outputs the image object. It can be determined directly using the values. In another example, after creating an image object, a user may select a display method to be used based on a list of predetermined display types provided by a server or a client device.

Information on the output method includes where the image object will be displayed on the client's display, how it will be displayed, what ratio it will display relative to the output screen, under what conditions it will be displayed, how long it will be displayed, When a plurality of image objects exist, it may include information about which one to display first and whether to repeatedly display the image object.

For example, if the image object is an animation including a character that changes over time, the client device may adjust the size of the corresponding character to increase or decrease over time. In addition, the client device may determine that the corresponding character is output in such a manner as to leave a certain distance or move to the periphery based on image information corresponding to a specific object. Also, if the image object is text information composed of letters, the client device may determine to output the corresponding text information letter by letter over time. In addition, the client device may determine to display a special effect or design to emphasize image information corresponding to a specific object using an image object.

In another example, after creating an image object, the client device may output information about a display type preferred by other users, including the user using the corresponding client device, so that the user can select the image object. In addition, the client device may determine a priority among information on a plurality of display types in consideration of a user's preference, frequency of use, and the like, and arbitrarily determine a display type having the highest priority.

In step 605, the client device transmits image object information to the server.

The image object information may be information including information about an image object and a display type of the corresponding image object. More specifically, after step 603, the client device may bundle the image object created in step 601 and information about a display type corresponding to the corresponding image object and transmit the same to the server. The image object information transmitted here may include identification information of a thing corresponding to the corresponding image object information. Accordingly, the server receives image object information from the client device, determines a thing corresponding to the image object information, and transfers information about the image object and display method included in the image object information to the corresponding thing in the database of the server. It can be saved as relevant.

Thereafter, when other image information corresponding to the object is received from another client device, the server may transmit information about the image object and display method to the other client device based on the image object information corresponding to the object.

7 illustrates image object information transmitted from a client device to a server in a computing system according to an embodiment of the present disclosure.

The image object information may include information about the type of object, information about the image object, and information about the display type of the image object.

Information on the type of thing may mean unique information for identifying a thing corresponding to image object information. Since the image object is for specific image information, and since the specific image information is visual information about one side of a specific object, the image object may be information related to the specific object. Accordingly, the client device may transmit information about the object type to the server so as to be able to determine which type of object-related information the image object is.

Although FIG. 6 shows only first to fifth objects as types of objects included in the image object information, this is exemplary and various types of information on identifiable objects may be further included.

Information about an image object may refer to visual information of an image object generated by a client device. For example, it may be information about color, design, proportion, shape, etc. of an image object. More specifically, when a user draws a "solid blue star" on a part of the image information to highlight the image information output on the client device, and the client device creates it as an image object, information about the image object can be i) blue, ii) solid, iii) star-shaped. However, since a part of corresponding image information is information related to a display type, it may not be information about an image object. In addition, "ratio" may mean a ratio between elements constituting an image object, and may have a different meaning from a ratio of an output display space or image information.

Information about the display type of the image object may be information that determines how the image object will be displayed on the client device. For example, the information about the display type of the image object may include information about a location where the image object is displayed, a display method, a ratio, how to display a plurality of image objects, limiting conditions for display, and the like.

The position may mean a position where an image object is displayed in relation to image information corresponding to a specific object. For example, when a specific thing is a "laptop", when a user of a client device "marks a circle" to emphasize "c terminal of a notebook", when another client device acquires image information about the notebook , The image object “marked circle” may be displayed on the “c terminal of the notebook” part on the image information corresponding to the corresponding notebook.

The display method may refer to how the image object is specifically displayed on the client device. For example, the image object may be displayed over a certain period of time in a fade-in manner. Also, when the image object is a dynamic image, the displayed time may be variable according to a user's setting or predetermined information. For example, even if it takes 3 seconds for a user to create an image object, a client device that outputs the image object may display the corresponding image object for 10 seconds.

The proportion may refer to a physical proportion determined in relation to a surrounding environment when an image object is displayed on a client device. For example, even if the ratio of the image object and image information corresponding to a specific object was 1:10 at the time of creation of the image object, if the blank space on the display is sufficient when outputting from another client device, the ratio of the image object is increased. You can make it bigger (1:5). That is, the ratio may be adaptively set differently according to the size of the image information corresponding to the blank space present on the displayed display and the specific object.

The restriction condition may mean a condition for displaying or not displaying an image object. For example, when an image object includes violent or suggestive content, display may be restricted by reflecting information related to a client device outputting the corresponding image object. For example, if the age of the user using the client device is less than 19 years old, display of a specific image object may be restricted. In addition, when it is determined that the user resides abroad based on the location information of the client device, display of a specific image object may be restricted.

In addition, when a plurality of image objects related to a specific object are stored and the client device has insufficient space to display them all, the number of displayed image objects is limited based on a priority determined according to a predetermined criterion. can do. In addition, if the image object includes text information and it is determined that the user resides abroad based on the location information of the client device, the text information is translated into the language used in the country where the user resides and displayed. can do.

8 illustrates an operation of creating an image object by a client device in a computing system according to an embodiment of the present disclosure.

A user of a client device may obtain image information on one side of a specific object and transmit the obtained image information to a server. Thereafter, the server may check reference image information corresponding to the image information received from the client device. The server, in response to determining the reference object corresponding to the reference image information, may identify the image information received from the client device as that of the reference object. Thereafter, the server may transmit a message for notifying that object information corresponding to the image information exists to the client device. When the server fails to identify the reference object corresponding to the image information, the server may transmit a message for notifying the client device that object information corresponding to the image information does not exist.

After obtaining the message from the server, the client device may perform an operation for creating an image object for image information.

An operation of creating an image object may be the same as or similar to the operation shown in FIG. 6 . More specifically, the user may create an image object while the client device captures one side of a specific object. Also, the user may create an image object while the client device moves the focus from one side of a specific object to another. In this case, generating image information by the client device may be performed in real time while image information of a specific object changes over time.

For example, a user may capture a first side of a first object and create a first image object with respect to first image information corresponding to the first side. More specifically, the "right cushion part", which is the first side of the "headset", which is the first object, is photographed, and on the "right cushion part output on the screen of the client device", which is the first image information corresponding to the first surface, 1 As an image object, a "star mark emphasizing the excellent fit of the cushion part" can be drawn.

Thereafter, the user may move the camera focus of the client device to capture the second side of the first object and create a second image object with respect to second image information corresponding to the second side. More specifically, the "left charging terminal", which is the second side of the "headset", which is the first object, is photographed, and on the second image information corresponding to the second side, "the left charging terminal output on the screen of the client device", the first 2 As an image object, "X mark emphasizing poor durability of the charging terminal" can be drawn.

Then, the client device may transmit both the first image object and the second image object to the server. When receiving image information on the first object from another client device, the server may transmit both the first image object and the second image object so that they are displayed on the screen of the other client device.

According to an embodiment of the present disclosure, once an object is specified, even if image information is subsequently changed, a created image object may be generated by reflecting the changed image information, transmitted to a server, and stored therein.

According to another embodiment of the present disclosure, an image object transmitted from a client device to a server may include comment information. The client device may transmit display information together with the comment information to the server.

When the client device transmits both comment information and display information to the server, the transmitted information may be referred to as user input information. That is, the user input information may include comment information and display information.

Display information may refer to information about how comment information is displayed in a client device. Here, the client device may be a client device different from a client device that transmits user input information.

For example, the display information may include information about how the comment information will be displayed on the display of the client device, where it will be displayed, how long it will be displayed, and the like. Also, information on the display method may be information related to second image information obtained and displayed by the client device. More specifically, when the reference object corresponding to the second image information is a "cup" and the comment information includes a "human-shaped character", the "human-shaped character" leans against the "cup" or climbs on the cup. It can be implemented on the display of the client device in a seated form. Also, depending on the shape of the "cup", the display method of the comment information may be different, and the information about this may be included in the display information. For example, in the case of a "long cup", the "human-shaped character" can be implemented in a form leaning on the side of the cup, and in the case of a "flat cup", the "human-shaped character" sits on the cup or , It can be implemented in the form of climbing up a ladder along the side of the cup. That is, a different display method may be applied depending on the property (type, size, shape, color, etc.) of the object corresponding to the image information, and information about this may be included in the display information.

Also, a different display method may be determined based on second image information displayed on the corresponding object on the display of the second user terminal (another client device) as well as the property of the object. For example, if the object is the same as "cup" but the second image information is an image of "the top of the cup", the comment information may be implemented according to a display method related to the top. More specifically, if there are a total of two display methods of "character", which is the comment information about "cup", 'displayed while sitting on top of the cup' and 'displayed while leaning on the side of the cup', the second If the image information is an image of the "upper surface of the cup", the second user terminal may display a "character" that is 'displayed in a form sitting on top of the cup'.

According to an embodiment of the present disclosure, display information may be classified into at least one type according to a display method of an image object.

This disclosure exemplifies size information as a first type of display information.

The size information includes information about the size of the image object displayed in the second user terminal. The displayed size may mean a relative size determined in relation to other objects.

In one example, the size information may be determined differently according to the size of the display screen of the second user terminal on which the image object is displayed. More specifically, the absolute size of an image object displayed on a smartphone having a 4.3-inch display screen and an image object displayed on a tablet PC having a 10.1-inch display screen may be different from each other.

In another example, the size information may be determined differently according to the size of the 'blank area' excluding the area occupied by the image object on the display screen of the second user terminal. More specifically, when the 'blank area' is relatively small because the second object that the second user terminal focuses on is located relatively close to the second user terminal, the size of the image object displayed in the blank area is larger. can be adjusted small. Conversely, when the 'blank area' is relatively large because the second object that the second user terminal focuses on is located relatively far from the second user terminal, the size of the image object displayed in the blank area may be adjusted to be larger. can

In addition, the size information may be determined differently according to the reference object corresponding to the second image information displayed on the second user terminal, preference or usage information of the second user terminal, and the like.

This disclosure exemplifies contact information as the second type of display information.

The contact information includes information about a degree of contact between an image object displayed on the second user terminal and the second image information. The information about the degree of contact may include information about whether the image object and the second image information are in contact, a contact time, a contact location, and a degree of non-contact. Here, the degree of non-contact may be information about how far the image object and the second image information are apart when there is no contact.

In one example, the contact information may be information about the size of a region in which an image object displayed on the second user terminal and the second image information appear to be in contact. More specifically, when the image object is a 'human-shaped character' and the second image information is a 'beverage can', the image object may be displayed in a form leaning against or hugging the second image information. . Information about what posture the image object forms in relation to the second image information may be included in the contact information.

In another example, the contact information may be information about which part of the image object displayed on the second user terminal appears to be in contact with which part of the second image information. More specifically, when the image object is a 'human-shaped character' and the second image information is a 'beverage can', the 'hand' of the image object is displayed in a form holding the 'side' of the second image information, It can be displayed in the form of putting the 'back' of the image object on.

This disclosure exemplifies overlap information as a third type of display information.

The overlap information includes information about the degree of overlap with the second image information on the second user terminal according to the movement of the image object. Information about the degree of overlap may include the degree of overlap between the second image information and the image object, such as displaying an image object separated from the second image information in a completely overlapped state, completely overlapped in a separated state, or partially overlapped with each other. It may be information about.

In one example, the image object may be displayed in such a way that it is separated from the second image information according to a change in time after not overlapping with the second image information. More specifically, when the image object is a 'human-shaped character' and the second image information is a 'beverage can', the image object will be displayed in the form of a 'person' suddenly appearing after hiding behind the second image information. can

This disclosure exemplifies motion information as a fourth type of display information.

The motion information includes information about the degree to which the image object changes over time while being displayed on the display screen of the second user terminal. The information about the degree of change over time may include information about dynamic motion of the image object and dynamic interaction between the image object and the second image information.

In one example, the motion information includes information about motion according to a change in time of image information displayed on the second user terminal in order to emphasize the second image information. More specifically, when the image object is a 'human-shaped character' and the second image information is a 'drink can', the image object may be displayed in the form of a person running around the second image information. When the image object is a 'human-shaped character' and the second image information is a 'flat bowl', the image object may be displayed in the form of a person climbing up a ladder along the side of the second image information.

9 illustrates another example in which an image object is implemented in a client device according to an embodiment of the present disclosure.

9 illustrates a case in which an image obtained from a user's client device is 'a bowl containing food'. The server may cause the client device to output the user's evaluation of the corresponding food (eg, salad), contents on a social network service (SNS) related to the corresponding food, information on the corresponding food, and the like to the display of the client device. Specifically, other users using the computing system according to an embodiment of the present disclosure may display avatars, character images, comments, photos, videos, etc. created in relation to the food. In this case, an ID for identifying a creator, a profile picture, and the like may be displayed together with the expressed picture.

More specifically, in FIG. 9 , it can be confirmed that the second image information is displayed on the display screen of the second user terminal. Here, the second image information may be a screen shot of "salad in a bowl" from one side.

Referring to FIG. 9 , it can be seen that a plurality of comment information is displayed in relation to “salad in a bowl”. The plurality of comment information includes information about the object (the salad in the bowl), evaluation, and the like, such as price information of the corresponding salad, other people's evaluation of the corresponding salad, and how to enjoy the salad. In addition to this, a character interacting with the object may also be included in the comment information. For example, "a character climbing a ladder" along the side of a "bowl of salad" may be included. At this time, the display method of the corresponding character may be determined and displayed in relation to the object (or image information corresponding to the object).

In another example, the character is implemented in the form of a dynamic image appearing from behind an object (or image information corresponding to the object), running around the object, standing next to or leaning on the object, or climbing along the object. , can be implemented in the form of sitting on top of an object. As shown in FIG. 9 , when an object has the shape of a “bowl with a wide width and a low height,” the way the character is displayed is a method of climbing along the object or sitting on top of the object rather than leaning next to the object. This may be displayed with priority, and information about this may be included in display information.

In another example, a plurality of characters may be displayed in a manner of being arranged or moving around the object along the shape of the object.

In the present disclosure, comment information is exemplified as “character,” but more specifically, the character may include information capable of identifying a specific person (specific company) or specific product, such as promotional text, brand, and purchase link. For example, to promote "chicken breast" that goes well with salad, display information that causes company A, a brand specializing in chicken breast, to display its mascot character to interact with 'salad in a bowl' More information may be included.

According to an embodiment of the present disclosure, a certain area displayed on a display of a client device and adjacent to image information may be provided as an area for displaying comment information to a specific person. A plurality of predetermined regions may exist based on a ratio between a relative size of image information and a display size of the image information and the client device. For example, when a specific object is photographed from a distance, there may be more blank areas than when a specific object is photographed from a distance. At this time, since comment information may be displayed in the blank area, more certain areas may exist.

Also, the number or location of certain areas may be determined differently according to the shape of the object. For example, a vertically long object such as a 'beverage can' and a horizontally long object such as a 'bowl' may have different numbers or positions of certain areas.

A specific person may be a business operator for promoting his/her business, brand, or product, and may occupy a certain area related to an object and display a phrase or character for publicity in the certain area. Accordingly, display information provided by the server to the client device may further include display priority information. Information on display priority may include information on whether a specific person has purchased a certain area of corresponding image information for a corresponding object. When there are a plurality of buyers, a list of buyers and purchase information corresponding to a plurality of areas may be included in the display information.

10 illustrates another example in which an image object is implemented in a client device according to an embodiment of the present disclosure.

10 illustrates a case in which an image obtained from a user's client device is a 'drink can'. Referring to FIG. 10 , it can be seen that the external feature of the 'beverage can' is that the vertical length is longer than the horizontal length. Accordingly, a method of displaying the character described in FIG. 9 may be determined based on external information of the 'beverage can', and such information may be included in display information. For example, the character of FIG. 9 prefers displaying a display method leaning on a 'beverage can' rather than sitting on top of a 'beverage can' or climbing a 'beverage can' on a ladder. and this information may be included in display information.

11 illustrates another example in which an image object is implemented in a client device according to an embodiment of the present disclosure.

11 illustrates another example in which an image obtained from a user's client device is 'food'. In the computing system according to an embodiment of the present disclosure, the server may assist the user to rationally consume corresponding food based on abundant information.

For example, price information of corresponding food (eg, ramen) according to a seller may be displayed according to a predetermined criterion. For example, the display may be based on the place where 'ramen' is sold the most, or the place where the 'ramen' can be purchased at the lowest price may be displayed first.

In addition, the client device may display information about users' reviews of the corresponding 'Ramen'. In this case, the information on the review may display a photo of the corresponding 'ramen' and information on the reviewer. In addition, the client device may also display user recommendation information, such as displaying a similar product of 'ramen' to induce purchase. User recommendation information may be provided based on pre-stored information about a user's purchase history, user preference, and the like.

According to an embodiment of the present disclosure, a method of operating a server for providing augmented reality content includes storing, by the server, reference image information about a reference object; receiving, by the server, first image information about a first object from a first user terminal; receiving, by the server, an image object related to the first image information from the first user terminal; calculating, by the server, a first probability indicator indicating that the first object is identical to the reference object based on the first image information; storing, by the server, the image object in a database subordinate to the reference object when the first probability indicator satisfies a first predetermined condition; receiving, by the server, second image information about a second object from the second user terminal; calculating, by the server, a second probability indicator indicating that the second object is the same object as the reference object based on the second image information; And when the second probability index satisfies a predetermined second condition, the server provides the image object so that the second user terminal outputs the image object as augmented reality content by combining the image object with the second image information Include steps.

Also, the image object includes comment information related to the first image information.

The providing of the image object may further include providing the second user terminal with information about how the image object is output to the second user terminal.

Also, the information on the output method may be provided by the first user terminal.

The server may further include generating information about the output method based on the reference object and the second image information, wherein the information about the output method includes at least one image previously stored in the server. Can be created based on the output type.

The providing of the image object may include determining, by the server, an area where the image object can be output to the second user terminal based on the second image information; and adjusting the size of the image object based on the determined outputable area.

The providing of the image object may include determining, by the server, an area where the image object can be output to the second user terminal based on the second image information; and changing information about a method in which the image object is output based on the determined outputable area.

In addition, when a plurality of image objects exist in the database, the server determining a priority among the plurality of image objects based on comment information corresponding to each of the plurality of image objects; and providing, by the server, the image object to the second user terminal based on the determined priority.

In addition, the image object includes user identification information associated with the first user terminal.

According to an embodiment of the present disclosure, an apparatus of a server for providing augmented reality content includes: a communication unit; store; and at least one processor operatively connected to the communication unit and the storage device, wherein the at least one processor performs steps of: storing, by the server, reference image information about a reference object; receiving, by the server, first image information about a first object from a first user terminal; receiving, by the server, an image object related to the first image information from the first user terminal; calculating, by the server, a first probability indicator indicating that the first object is identical to the reference object based on the first image information; storing, by the server, the image object in a database subordinate to the reference object when the first probability indicator satisfies a first predetermined condition; receiving, by the server, second image information about a second object from the second user terminal; calculating, by the server, a second probability indicator indicating that the second object is the same object as the reference object based on the second image information; And when the second probability indicator satisfies a predetermined second condition, the server provides the image object so that the second user terminal outputs the image object as augmented reality content by combining the image object with the second image information It can be configured to perform steps.

The technical features disclosed in each embodiment of the present invention are not limited to the corresponding embodiment, and unless incompatible with each other, the technical features disclosed in each embodiment may be merged and applied to other embodiments.

Therefore, in each embodiment, each technical feature is mainly described, but each technical feature may be merged and applied to each other unless incompatible with each other.

The present invention is not limited to the above-described embodiments and accompanying drawings, and various modifications and variations will be possible from the viewpoint of those skilled in the art to which the present invention belongs. Therefore, the scope of the present invention should be defined by not only the claims of this specification but also those equivalent to these claims.

100: computing system
102: network
104: server
106: desktop computer
108: mobile phone or smartphone
110: PDAs
112: laptop computer
114: tablet computer
116: base station
118: wireless access point
200: server
205: bus system
210: processor(s)
215: storage devices
220: Communication Department
225: input/output unit
230: memory
235: permanent storage
300: client device
305: antenna
310: RF transceiver
315: transmission processing circuit
320: microphone
325: reception processing circuit
330: speaker
340: Processor(s)
345: input/output interface
350: touch screen
355: display
360: memory
361: operating system
362: Applications

Claims (7)

In the operating method of the server for providing augmented reality content,
storing, by the server, reference image information about a reference object;
receiving, by the server, first image information about a first object from a first user terminal;
Receiving, by the server, user input information from the first user terminal
-The user input information includes comment information related to the first image information and display information related to a method of displaying the comment information-;
calculating, by the server, a first probability indicator indicating that the first object is identical to the reference object based on the first image information;
storing, by the server, the user input information in a database subordinate to the reference thing when the first probability indicator satisfies a first predetermined condition;
receiving, by the server, second image information about a second object from the second user terminal;
calculating, by the server, a second probability indicator indicating that the second object is the same object as the reference object based on the second image information; and
When the second probability index satisfies a predetermined second condition, the server provides the user input information so that the second user terminal combines the comment information with the second image information and outputs it as augmented reality content including steps,
The comment information is output from the second user terminal based on the second image information and the display information;
The display information may include a relative size of the comment information with respect to the second image information, a type of the comment information, a time when the comment information is displayed on the second user terminal, and a predetermined display area for the second image information. containing information about at least one of
Server operation method for providing augmented reality content. According to claim 1,
Further comprising determining, by the server, the type of the reference thing;
The comment information is output from the second user terminal based on the determined type of the reference thing.
Server operation method for providing augmented reality content. According to claim 1,
Providing the user input information,
determining, by the server, an area where the image object can be output to the second user terminal based on the second image information; and
Further comprising adjusting the size of the image object based on the determined outputable area.
Server operation method for providing augmented reality content. According to claim 1,
Providing the user input information,
determining, by the server, an area where the user input information can be output to the second user terminal based on the second image information; and
Based on the determined outputable area, further comprising the step of changing the display information for the comment information.
Server operation method for providing augmented reality content. According to claim 1,
if a plurality of pieces of user input information exist in the database, determining, by the server, priorities among the pieces of user input information based on comment information corresponding to the plurality of pieces of user input information; and
Based on the determined priority, further comprising the server providing the user input information to the second user terminal.
Server operation method for providing augmented reality content. According to claim 1,
The user identification information,
Including user identification information associated with the first user terminal,
Server operation method for providing augmented reality content. In the device of a server for providing augmented reality content,
The server,
communications department;
store; and
including at least one processor operatively connected to the communication unit and the storage device;
The at least one processor,
storing, by the server, reference image information about a reference object;
receiving, by the server, first image information about a first object from a first user terminal;
receiving, by the server, user input information from the first user terminal;
-The user input information includes comment information related to the first image information and display information related to a method of displaying the comment information-
calculating, by the server, a first probability indicator indicating that the first object is identical to the reference object based on the first image information;
storing, by the server, the user input information in a database subordinate to the reference thing when the first probability indicator satisfies a first predetermined condition;
receiving, by the server, second image information about a second object from the second user terminal;
calculating, by the server, a second probability indicator indicating that the second object is the same object as the reference object based on the second image information; and
When the second probability index satisfies a predetermined second condition, the server provides the user input information so that the second user terminal combines the comment information with the second image information and outputs it as augmented reality content It is configured to perform steps,
The comment information is output from the second user terminal based on the second image information and the display information;
The display information may include a relative size of the comment information with respect to the second image information, a type of the comment information, a time when the comment information is displayed on the second user terminal, and a predetermined display area for the second image information. containing information about at least one of
Server device for providing augmented reality content.

Images