KR102498781B1 - Method for generating scene graph of objects in images and electronic device using the same - Google Patents

Abstract

A method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure includes receiving a first satellite image, identifying at least one object in the received first satellite image, and identifying at least one object in the received first satellite image. Generating annotation data corresponding to one object, determining relationship data between at least one identified object based on the generated annotation data, and obtaining a second satellite image including the annotation data and the relationship data A step of providing the first region may be included.

Description

A method for generating a scene graph for an object in an image and an electronic device using the same

Various embodiments of the present invention relate to a method for generating annotation data and relation data for an object in a satellite image and an electronic device using the same. In particular, it relates to a user interface and system for creating scene graph annotations for aerial and satellite imagery.

Scene graph data targeting satellite imagery and aerial imagery can identify the trend of a group that is used as an object by utilizing changes in the relationship between objects within the target area. In satellite imagery, the number of objects present in the image is large, and a unique ID is assigned to each individual object, and converting the relationship between IDs into data can consume more time and data than general images. Therefore, there is a need for a technique capable of efficiently reducing time and data by simplifying object data extracted from satellite images.

Embodiments of the present invention are proposed to solve the above problems and relate to a method for generating a scene graph while reducing data consumption for a plurality of objects included in a satellite image and an electronic device using the same. The technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems can be inferred from the following embodiments.

A method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure includes receiving a first satellite image, identifying at least one object in the received first satellite image, and identifying at least one object in the received first satellite image. Generating annotation data corresponding to one object, determining relationship data between at least one identified object based on the generated annotation data, and obtaining a second satellite image including the annotation data and the relationship data A step of providing the first region may be included.

An electronic device providing a method for generating a scene graph according to various embodiments of the present disclosure includes a memory, a communication unit, and a control unit, wherein the control unit receives a first satellite image through the communication unit and receives the received satellite image. Identifying at least one object in the first satellite image, generating annotation data for the identified at least one object, determining relationship data between the identified at least one object based on the generated annotation data, and A second satellite image including the annotation data and the relation data may be provided to the first area.

According to an embodiment of the present invention, one or more of the following effects may exist. If a bounding box for identifying an object in the selected area by selecting a certain area in the image can be rotated, it is possible to select an area that does not contain the object to be identified, consuming data for object detection and identification. can be reduced In addition, a selection area for identifying an object can be minimized through a convenient interface in which a user can utilize a rotatable bounding box. Accordingly, it is possible to effectively simplify the process of identifying objects in an image and providing annotation data and relational data between objects, thereby reducing required time and data consumption, and thus adaptively responding to multiple objects.

The effects of the embodiments are not limited to the effects described above, and other effects not described can be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of internal components of an electronic device according to various embodiments of the present disclosure.
2 is a schematic flowchart of a method for generating a scene graph according to various embodiments of the present invention.
3 is a flowchart illustrating object identification in a satellite image of a method for generating a scene graph according to various embodiments of the present disclosure.
4A to 4D are exemplary diagrams related to the step-by-step process of FIG. 3 .
5 is a flowchart illustrating relationship data determination in a method for generating a scene graph according to various embodiments of the present disclosure.
6A to 6E are exemplary diagrams related to the step-by-step process of FIG. 5 .
7A to 7C are exemplary diagrams of a first area and a second area on an interface of a method for generating a scene graph according to various embodiments.

The terms used in the embodiments have been selected as general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

In the entire specification, when a part is said to "include" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated. In addition, terms such as “~unit” and “~module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

The expression of “at least one of a, b, and c” described throughout the specification means 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c' ', or 'all of a, b, and c'.

A “terminal” referred to below may be implemented as a computer or portable terminal capable of accessing a server or other terminals through a network. Here, the computer includes, for example, a laptop, desktop, laptop, etc. equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution), etc. It may include a handheld-based wireless communication device.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a block diagram of internal components of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 1 , internal components of the electronic device 100 are not limited to the illustrated components. The electronic device 100 may include at least one of a control unit 110 , a communication unit 120 , and a memory 130 . The electronic device 100 may include a computer, a server, a smart phone, and the like, and may include a system configured of a server and a computer to perform the scene graph generation method of the present specification.

The controller 110 may process a series of steps for performing a method for generating a scene graph according to various embodiments of the present disclosure. The controller 110 may control other elements of the electronic device 100 . The control unit 110 described herein may perform repetitive learning through a neural network, and may include a central processing unit (CPU) or a neural network processing unit (NPU). The controller 110 of the present specification provides a scene graph by generating annotation data and relationship data between objects through accumulated storage and application of data on a specific object, name of the object, relationship between objects, etc. that may be included in a satellite image. can

The controller 110 may receive the first satellite image through the communication unit 120 . The controller 110 may identify an object included in the received first satellite image. For example, objects such as a ship and a helipad constituting the ship, a cockpit, a head, a deck crane, or a cargo loading space may be included in the first satellite image. The controller 110 may identify an object included in a certain area selected by the user in the first satellite image. Identifying an object in an image through the controller 110 is to automatically identify an object by repeatedly accumulating data learned through a neural network, or to annotate an object included in a certain area initially selected for such automatic identification. Data may be input and identified by the user. The control unit 110 identifies an object in the image based on the corner coordinates (eg, (x1, y1), (x2, y2), (x3, y3), (x4, y4)) of a certain area selected by the user. It may be to identify from a list of objects that can be included in the included area. For example, if data learned by the neural network does not exist and a certain area is selected, the controller 110 may provide a list of names of objects that may be included in a satellite image to the user so that the user can select the object. . For another example, the controller 110 may, if there are data learned from repeated selections of a plurality of users by a neural network, if a certain area is selected from a satellite image, at least one or more objects included in the certain area can be automatically identified. When user's manual input is repeatedly accumulated and stored, the controller 110 can automatically identify an object included in a selected region of the satellite image.

The controller 110 may generate annotation data for at least one identified object. For example, the controller 110 may automatically generate annotation data for the identified object based on data accumulated through training data for repetitive learning of the neural network. The controller 110 may identify an object included in a certain area selected from the first satellite image as a 'deck crane' and generate annotation data of 'deck crane'. For another example, the controller 110 may manually generate annotation data for the identified object in order to provide data provided for repetitive learning of the neural network. The control unit 110 identifies an object included in a certain area selected from the first satellite image as a 'deck crane', lists and provides names of objects that may be included in the 'ship' to the user, and the user identifies the identified object. Annotation data corresponding to 'deck crane' can be entered manually. When user's manual inputs are repeatedly accumulated and stored, the controller 110 can automatically generate annotation data for an object included in a selected area of a satellite image.

The controller 110 may determine relationship data between the identified objects based on the generated annotation data. The controller 110 may automatically determine relation data between a plurality of objects (eg, between two objects) for which annotation data is generated based on data accumulated through training data for repetitive learning of the neural network. The controller 110 may determine relation data between objects in which annotation data is created within a certain region selected from the first satellite image. For example, the control unit 110 selects a certain area including the 'deck crane' (eg, the first object), obtains a click and drag input, and selects the 'deck crane' as a starting point. ), it is possible to determine relational data between the 'deck crane' and the 'cargo loading space' according to the input of the 'cargo loading space' (eg, the second object) as the end point. Here, the control unit 110 may determine the relationship data that the 'deck crane' is 'on' of the 'cargo shipping space', and repeatedly set the 'first object' as the subject to the 'first object'. It is possible to determine relational data using '2 objects' as an object. For another example, the controller 110 may manually generate relational data between objects in which annotation data is created within a selected area to provide data provided for iterative learning of the neural network. The control unit 110 obtains a click and drag input after selecting a certain area including the 'deck crane', and sets the 'deck crane' as the starting point to the 'cargo loading space' as the end point. and 'cargo shipping space' may be manually determined. Here, the control unit 110 lists and provides terms that may include data related to 'positional relationship' between objects for which footnote data is created to the user according to the acquisition of the click-and-drag input, and the user lists 'deck crane' and Relational data corresponding to the 'positional relationship' between 'cargo shipping spaces' can be selected manually. When manual inputs of users are repeatedly accumulated and stored, the controller 110 may automatically generate relation data between objects in which annotation data is generated in a satellite image.

In this specification, relationship data may include first relationship data, second relationship data, or an indicator. The first relationship data is relationship data having a first object as a start point and a second object different from the first object as an end point, and may mean a subject-object relationship. For example, when a click and drag input from a 'helipad' to a 'ship' is acquired, information that the 'helipad' is 'above' the 'ship' may be determined as relational data. For another example, when a click-and-drag input from the 'cockpit' to the 'ship' is obtained, information that the 'cockpit' is 'above' the 'ship' may be determined as relational data. The second relational data is relational data with the first object as a start point and the first object as an end point, and may mean regression and may include information about the first object. For example, when a click-and-drag input from 'bow' to 'bow' is acquired, information about 'shape is rectangle' as information about the 'bow' may be determined as relational data. For another example, when a click-and-drag input from 'ship' to 'ship' is acquired, information about 'color is gray' as information about 'ship' may be determined as relational data. The indicator may include an arrow from the first object to the first object and an arrow from the first object to the second object.

The communication unit 120 transmits information stored in the memory 130 of the electronic device 100 or information processed by the control unit 110 to another device, or functions to receive information from another device to the electronic device 100. can be done For example, the electronic device 100 may receive the first satellite image through the communication unit 110 . For another example, the electronic device 100 may transmit the second satellite image and data related to the second satellite image to a backup server or another electronic device through the communication unit 110 .

The memory 130 is a data structure implemented in a predetermined storage space of the electronic device 100, and functions such as storing, searching, deleting, editing, or adding data can be freely performed. For example, the memory 130 may include fields or components for processing functions such as data storage, search, deletion, editing, or addition. The memory 130 may store data related to the electronic device 100 performing an information providing method. For example, the memory 130 may store instructions or data for an execution operation of the control unit 110 .

2 is a schematic flowchart of a method for generating a scene graph according to various embodiments of the present invention.

In the method of generating a scene graph, in step S210, a first satellite image may be received. The first satellite image may include a satellite image of a specific place or a satellite image of a specific object.

In the method for generating a scene graph, at step S220, at least one object included in the first satellite image may be identified. The first satellite image may include a plurality of objects, such as ship 1, ship 2, a bridge, or a port. In the scene graph generation method, objects included in the first satellite image may be identified by classifying objects included in the first satellite image into sub-concepts including a range having a large size as an upper concept and components included in the corresponding object as a lower range.

In the method of generating a scene graph, in step S230, annotation data corresponding to the object identified in the first satellite image may be generated. On the interface provided in the scene graph generation method, for example, when a user input for selecting a partial region of the first satellite image is obtained, annotation data corresponding to an identified object existing in the selected region may be generated. According to the scene graph generation method, coordinates of a certain area selected as a rotatable bounding box may be obtained, and annotation data corresponding to an identified object existing at the corresponding coordinates may be generated.

In the method of generating a scene graph, relationship data may be determined based on the annotation data in step S240. For example, it is possible to determine relationship data describing a relationship between two objects for which annotation data is created as a subject-object relationship.

In the method of generating a scene graph, a second satellite image including annotation data and relation data may be overlaid on the first satellite image and provided in step S250. The second satellite image may include annotation data of objects included in the first satellite image, and may include relationship data between objects for which the annotation data is generated. The second satellite image provided overlaid on the first satellite image can be explained as being provided superimposed on the first satellite image, and can be provided in the first area in an interface provided to the user. The first satellite image may be changed to the second satellite image and provided, or the second satellite image may include the first satellite image.

3 is a flowchart illustrating object identification in a satellite image of a method for generating a scene graph according to various embodiments of the present disclosure.

FIG. 3 may be a detailed flowchart of S220 of the method for generating a scene graph of FIG. 2 .

In the method for generating a scene graph according to various embodiments, a user's input for a partial region of the first satellite image may be obtained in step S221. The user's input may be click and drag, and in S221, an input of clicking and dragging the cursor may be acquired to draw a corner (eg, side) on a partial area of the first satellite image. The user may draw a corner from (x1, y1) to (x2, y2) in the first satellite image. In step S221, the user's click-and-drag input ends to finish drawing the corner at (x2, y2), and the user cancels the click (e.g., clicking the mouse button) with the point at (x2, y2) as the end point. can be stopped).

In step S222, if the length of the corner drawn by the user does not exceed 0, it is possible to return to the starting step and perform step S221. The method for generating a scene graph may proceed to step S223 when the length of the corner drawn in step S222 exceeds zero.

In step S223, the method for generating a scene graph generates (x1, y1) and (x2, y2) when an input of moving the cursor corresponding to the cancellation of the click at (x2, y2), which is the end point in step S221, is obtained. The bounding box can be drawn as one corner and expansion or contraction in a direction orthogonal to the corner occurs. The drawn bounding box may be rotated after drawing, and if the corners drawn at (x1, y1) and (x2, y2) are not horizontal to the x-axis in the first satellite image, the bounding rotated by moving the cursor It can be drawn in the form of a box. The bounding box drawing of step S223 is described in detail in FIGS. 4A to 4D.

In step S224, the scene graph generating method may identify an object within the bounding box. For example, when there is a ship within a bounding box that selects at least a portion of the first satellite image, the ship may be identified as an object. When the ship is identified as an object, objects constituting the ship may also be identified together as a lower concept with the ship as a higher concept. Sub-concept objects constituting a ship may include a helipad, a cockpit, a head, a deck crane, or a cargo loading space. In this specification, the objects included in the first satellite image are only examples and are not limited thereto.

4A to 4D are exemplary diagrams related to the step-by-step process of FIG. 3 .

Specifically, FIGS. 4A and 4B may correspond to steps S221 and S222 of FIG. 3 , FIG. 4C may correspond to step S223 of FIG. 3 , and FIG. 4D may correspond to step S224 of FIG. 3 .

Referring to FIG. 4A , a first satellite image provided to a first area on an interface where a scene graph generation method is implemented is shown. The user may select a part of the first satellite image by inputting the cursor. Input through a cursor is an example, and touch input may be possible, and the method is not limited. 4A is an example in which the end of the cursor is positioned at (x1, y1) as a point of time for the user to select a partial area of the first satellite image. In the process from FIG. 4A to FIG. 4B , the user may click (x1, y1) as a viewpoint for drawing the bounding box.

Referring to FIG. 4B , a corner 410 drawn by a user with (x1, y1) as a start point and (x2, y2) as an end point is shown. The user can release the mouse button clicked at (x2, y2) by holding down the click on (x1, y1) and dragging the cursor. In this specification, releasing a mouse button may be described as a combination of canceling a click. The user's click-and-drag input ends at (x2, y2) to finish drawing the corner, and the user takes the point at (x2, y2) as the end point to cancel the click (eg, stop clicking the mouse button). can

Referring to FIG. 4C, the user moves the cursor after canceling the click at (x2, y2), which is the end point shown in FIG. Expansion or contraction in a direction perpendicular to the corner may cause the bounding box 420 to be drawn. The drawn bounding box may be rotated after drawing, and if the corners drawn at (x1, y1) and (x2, y2) are not horizontal to the x-axis in the first satellite image, the bounding rotated by moving the cursor It can be drawn in the form of a box. Four vertices of the bounding box 420 may correspond to (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

Referring to FIG. 4D , a user may select annotation data provided from an interface of a scene graph generation method. For example, when the object in the bounding box 420 drawn through FIGS. 4A to 4C is determined to be a 'cockpit', the user selects annotation data as 'cockpit' (command post) in the name selection window 430 of the object. can The user's selection of annotation data may correspond to a process of accumulating data for neural network training. The neural network may automatically identify an object within the bounding box 420 without providing the object name selection window 430 to the user by storing results of selection of annotation data by a plurality of users.

Referring to FIGS. 4A to 4D , if a bounding box can be rotated in a region set to identify an object included in a region desired by a user in a received satellite image, consumption of data can be reduced. For example, in the satellite image, objects included in the area displayed on the screen may not be positioned parallel to the x-axis. Even in the case of 'ship' in FIGS. 4A to 4D , satellite images positioned diagonally with respect to the x-axis are shown. When a certain area is selected using a bounding box that cannot be rotated to identify an object including a 'ship', an extra area that does not include an object to be identified may be selected together. Accordingly, the selected area together with the redundant area may have a larger capacity including unnecessary data. Operation processing for large-capacity images such as satellite images or aerial images may be delayed when the data volume is large. When an object is identified using a rotatable bounding box to delay such calculation processing and to increase the efficiency of the entire process, the user can select only a certain area that includes the object to be identified, not including the redundant area. As a result, the time of calculation processing can be shortened and the efficiency of the entire process can be increased.

5 is a flowchart illustrating relationship data determination in a method for generating a scene graph according to various embodiments of the present disclosure.

5 may be a detailed flowchart of step S240 of FIG. 2 .

In the method for generating a scene graph according to various embodiments, in step S241, an input for selecting a certain region as a bounding box may be obtained. If an input is obtained indicating that the bounding box is drawn, in step S242, it may be checked whether the number of identified objects in the bounding box is greater than zero. If the number of objects identified in the bounding box drawn in step S242 is equal to or less than 0, the process may return to step S241 again.

The method for generating a scene graph may proceed to step S243 when the number of objects in the drawn bounding box is one or more. In step S243, any one of the objects in the bounding box may be identified as the first object. The object identified as the first object in step S243 may be an object occupying the largest area in the bounding box, and an algorithm identified as the first object among a plurality of objects in the bounding box may be changed by a developer.

In step S244, the method for generating a scene graph may obtain a click and drag input from the first object. The click-and-drag input in step S244 may be performed using a mouse button different from the click-and-drag input in step S221 described in FIG. 4B, but is not limited thereto.

In step S245, the method for generating a scene graph may check whether the number of objects in the dragged area is greater than zero. If the number of objects in the area dragged in step S245 is equal to or smaller than 0, the process may be returned to step S244.

The method for generating a scene graph may proceed to step S246 when the number of objects in the area dragged in step S245 is one or more. In step S246, any one of the objects in the dragged area may be identified as the second object. The object identified as the second object in step S246 may be an object occupying the largest area in the dragged area, and an algorithm identified as the second object among a plurality of objects in the dragged area may be changed by a developer.

In step S247, the method for generating a scene graph may check whether the first object and the second object are the same object. If it is determined in step S247 that the first object and the second object are the same object, step S248 may be performed, and if they are not identical, step S249 may be performed.

In step S248, the scene graph generating method may determine second relationship data. The second relational data is relational data with the first object as a start point and the first object as an end point, and may mean regression and may include information about the first object. For example, when a click-and-drag input from 'bow' to 'bow' is acquired, information about 'shape is rectangle' as information about the 'bow' may be determined as relational data. For another example, when a click-and-drag input from 'ship' to 'ship' is acquired, information about 'color is gray' as information about 'ship' may be determined as relational data.

In step S249, the scene graph generating method may determine first relationship data. The first relationship data is relationship data having a first object as a start point and a second object different from the first object as an end point, and may mean a subject-object relationship. For example, when a click and drag input from a 'helipad' to a 'ship' is acquired, information that the 'helipad' is 'above' the 'ship' may be determined as relational data. For another example, when a click-and-drag input from the 'cockpit' to the 'ship' is obtained, information that the 'cockpit' is 'above' the 'ship' may be determined as relational data.

In step S244, the scene graph generation method may display an indicator from the center point of the first object to the center point of the second object in the dragged area. For example, the indicator may include an arrow pointing from the first object to the first object and an arrow pointing from the first object to the second object.

6A to 6E are exemplary diagrams related to the step-by-step process of FIG. 5 .

In detail, FIG. 6A is an example of the first area 610 on the interface provided in the scene graph generation method and the second area excluding the first area. 6B may correspond to steps S241 to S243 of FIG. 5, FIG. 6C may correspond to steps S244 to S246 of FIG. 5, and FIGS. 6D and 6E may correspond to steps S247 to S249 of FIG. This is an example of what could be. The description of FIGS. 6A to 6E is based on the premise that annotation data for an object of a portion provided in a first region of the first satellite image is generated. In addition, the coordinates in FIGS. 6A to 7C are for four vertices of (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and are not shown at all.

In FIG. 6A , an area other than the first area 610 may be referred to as a second area, and annotation data 620 and relation data 630 may be provided in the second area. Annotation data 620 may be stored in a kind of table form, numbering may be an order in which annotation data 620 is generated, and a name of an identified object 621 may correspond to an individual number. The identified object 621 is identified in the form of a bounding box and has four vertices, and the coordinates 622 of the identified object may be stored for each name. The name of the identified object 621 may vary according to the first satellite image, and the coordinates 622 of the identified object may similarly vary. This may be related to the movement of the image. The movement of the image may be caused by the movement of the object according to the capture time of the satellite image, and when the user wants to look at an area other than the area of the first satellite image provided in the first area 610, the first area 610 may be moved. It may be caused by movement of the satellite image to another area.

In FIG. 6B , the first object may be identified by a selection input resulting from drawing a bounding box for a certain area of the first area 610 . The first object identified in the first area 610 may be provided as '4, deck crane, coordinates' in the annotation data 620 in the second area. In the annotation data 620 of the second area, an object identified as the first object of the first area 610 may be highlighted.

In FIG. 6C , a second object may be identified in an area of the bounding box drawn from the first object of the first area 610 to the end point by clicking and dragging (eg, the dragged area). The second object identified in the first area 610 may be provided as '6, cargo shipping space, coordinates' in the annotation data 620 in the second area. In the annotation data 620 of the second area, an object identified as the second object of the first area 610 may be highlighted.

In FIG. 6D , an object name selection window 640 may be provided on at least a part of the first region 610 . The name selection window 640 of FIG. 6D is not the same as the name selection window 430 of FIG. 4D. The name selection window 640 of FIG. 6D selects a name describing the relationship data between the first object and the second object, and the name selection window 430 of FIG. 4D selects a name for the identified object. The user may select a name describing the relationship between the first object and the second object in the name selection window 640 . The list provided in the name selection window 640 of FIG. 6D is only an example and is not limited thereto.

The relationship data selected in FIG. 6D may be stored and provided in the relationship data 630 in FIG. 6E. For example, the relationship data 630 includes numbered relationships 631 and may provide a relationship between a first object and a second object in a subject-object relationship. The relation data of the first object and the second object provided in FIG. 6E may explain a relation '4 is in 6th place'.

According to various embodiments of the present invention, the processes of FIGS. 6A to 6E may be automatically performed in response to reception of the first satellite image through neural network training. For another example, the processes of FIGS. 6A to 6C may be manually performed by a user and the remaining processes may be automatically performed.

7A to 7C are exemplary diagrams of a first area and a second area on an interface of a method for generating a scene graph according to various embodiments.

7A is an example of the first area 710 on the interface provided in the scene graph generation method and the second area excluding the first area. When a user places a cursor on an area displayed in the first area 710, data on an object identified in the displayed area may be provided to a second area. For example, annotation data 720 and relation data 730 may be provided in the second area. In FIG. 7A , the object identified in the area where the cursor is located in the first area 710 may be highlighted and provided to corresponding data among the annotation data 720 in the second area. The annotation data 720 in the second area may correspond to the annotation data 621 of FIG. 6E, and the name of the identified object 721 may correspond to the name of the identified object 621 of FIG. 6E. , the coordinates 722 of the identified object may correspond to the coordinates 622 of the identified object in FIG. 6E, and the relationship data 730 and numbered relationships 731 correspond to the relationship data 630 and It may correspond to numbered relationships 631 .

Referring to FIGS. 7A and 7B , the annotation data 720 and the relationship data 730 do not overlap with the first region 710 provided with a part of the first satellite image on the interface provided by the scene graph generation method. area can be provided. When a first object is selected from among the at least one object identified in the first area 710 or the cursor is located in the area of the bounding box formed by the coordinates in which the first object is identified, it is selected in the second area or the cursor is correspondingly selected. Annotation data 720 for an object in an area where is located may be highlighted and provided. For example, in FIG. 7A , when the mouse cursor is placed on the first object, '1, star-shaped helipad, coordinates', which is the annotation data 720 for the first object, can be highlighted in the second area. An object for the annotation data selected in the first area 710 may be highlighted and provided according to an input for selecting one of the at least one piece of annotation data 720 generated in the second area. For example, if '2, star-shaped helipad, coordinates' is selected in FIG. 7B , an object corresponding to the object selected in the first area 710 may be highlighted and provided.

In the scene graph generation method according to various embodiments, objects and indicators corresponding to the relation data selected in the first area 710 are highlighted and provided according to an input for selecting one of the relation data 730 determined in the second area. can do. For example, in FIG. 7C , when the user places the mouse cursor on '0, 4, on, 6' and selects them, objects (eg, 4 and 6) corresponding to the selected relationship data 730 in the first area 710 are displayed. ) and an indicator (eg, an arrow from 4 to 6) 711 connecting objects may be highlighted and provided.

A method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure includes receiving a first satellite image, identifying at least one object in the received first satellite image, and identifying at least one object in the received first satellite image. Generating annotation data corresponding to one object, determining relationship data between at least one identified object based on the generated annotation data, and obtaining a second satellite image including the annotation data and the relationship data A step of providing the first region may be included.

The identifying step of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure includes identifying the at least one object in a designated area using a rotatable bounding box. Including, the designated area may be at least a partial area of the first satellite image.

The generating step of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure includes selecting one of a plurality of annotation data selectable in response to an input for selecting one of the identified at least one object. It may include the step of selecting and generating.

The determining step of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure may include a plurality of selectable objects corresponding to an input from a first object to a second object among the identified at least one object. Selecting and determining one of the relational data, and the input from the first object to the second object may include a click-and-drag input with the first object as a starting point and the second object as an end point. there is.

In the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure, the annotation data includes data on the name of the identified at least one object, and the relationship data includes the identified at least one object. It may include data representing the positional relationship between objects.

In the providing of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure, the second satellite image is converted to the first satellite image in response to an input on the identified at least one object. Overlaying and providing may be included.

In the providing step of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure, the annotation data and the relationship data are stored and the first region and the first satellite image are provided on the interface. A step of providing a non-overlapping second area may be included.

The providing of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure may include the second area in response to an input for selecting one of the one or more identified objects in the first area. Annotation data for the selected object is highlighted and provided, and an object for the selected annotation data is highlighted in the first area in response to an input for selecting one of the at least one generated annotation data in the second area. can be provided.

The providing of the method for generating a scene graph in an interface of an electronic device according to various embodiments of the present disclosure may include the first area in response to an input for selecting one of the determined at least one relationship data in the second area. In , objects and indicators corresponding to the selected relationship data may be highlighted and provided.

An electronic device providing a method for generating a scene graph according to various embodiments of the present disclosure, including a communication unit and a control unit, wherein the control unit receives a first satellite image through the communication unit, and displays the received first satellite image. Identifying at least one object in the image, generating annotation data for the identified at least one object, determining relationship data between the at least one identified object based on the generated annotation data, and determining the annotation data and A second satellite image including the relation data may be provided to the first area.

A non-transitory computer-readable storage medium according to various embodiments of the present invention includes a medium configured to store computer-readable instructions, wherein the computer-readable instructions, when executed by a processor, cause the processor to: Receiving an image, identifying at least one object from the received first satellite image, generating annotation data corresponding to the identified at least one object, and identifying at least one object based on the generated annotation data. A method for generating a scene graph may be performed, which includes determining relationship data between objects of and providing a second satellite image including the annotation data and the relationship data to a first region.

An electronic device or terminal according to the above-described embodiments includes a processor, a memory for storing and executing program data, a permanent storage unit such as a disk drive, a communication port for communicating with an external device, a touch panel, and a key. , user interface devices such as buttons, and the like. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, the computer-readable recording medium includes magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM) ), and DVD (Digital Versatile Disc). A computer-readable recording medium may be distributed among computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

This embodiment can be presented as functional block structures and various processing steps. These functional blocks may be implemented with any number of hardware or/and software components that perform specific functions. For example, an embodiment is an integrated circuit configuration such as memory, processing, logic, look-up table, etc., which can execute various functions by control of one or more microprocessors or other control devices. can employ them. Similar to components that can be implemented as software programming or software elements, the present embodiments include data structures, processes, routines, or various algorithms implemented as combinations of other programming constructs, such as C, C++, Java ( It can be implemented in a programming or scripting language such as Java), assembler, or the like. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, this embodiment may employ conventional techniques for electronic environment setting, signal processing, and/or data processing. Terms such as “mechanism”, “element”, “means” and “composition” may be used broadly and are not limited to mechanical and physical components. The term may include a meaning of a series of software routines in association with a processor or the like.

The foregoing embodiments are merely examples and other embodiments may be implemented within the scope of the claims described below.

Claims (11)

A method for generating a scene graph in an interface of an electronic device,
receiving a first satellite image;
identifying at least one object from the received first satellite image;
generating annotation data corresponding to at least one identified object;
determining relation data between the identified at least one object based on the created annotation data; and
Providing a second satellite image including the annotation data and the relationship data to a first area;
The determining step is
Selecting and determining one of a plurality of selectable relational data corresponding to an input from a first object to a second object among the identified at least one object,
The input from the first object to the second object includes a click and drag input with the first object as a starting point and the second object as an end point;
The step of selecting and determining one of the plurality of relationship data
checking whether the first object and the second object are the same object; and
When the first object and the second object are not identical, first relation data for a relationship between the first object and the second object is determined, and when the first object and the second object are identical, the first relation data is determined. A method of generating a scene graph comprising determining second relationship data including information about an object. According to claim 1,
The step of identifying
Identifying the at least one object in a designated area using a rotatable bounding box,
The designated area is at least a partial area of the first satellite image, the scene graph generating method. According to claim 1,
The generating step is
In response to an input for selecting one of the at least one identified object, selecting and generating one of a plurality of selectable annotation data. delete According to claim 1,
the annotation data includes data on the name of the identified at least one object;
Wherein the relationship data includes data representing a positional relationship between the identified at least one object. According to claim 1,
The steps provided above are
and overlaying the second satellite image on the first satellite image in response to an input of the at least one identified object and providing the overlay. According to claim 1,
The steps provided above are
and storing the annotation data and the relationship data and providing them to a second area that does not overlap with a first area provided with the first satellite image on the interface. According to claim 7,
The steps provided above are
Highlighting and providing annotation data for the selected object in the second area in response to an input for selecting one of the at least one object identified in the first area;
In response to an input for selecting one of the at least one generated annotation data in the second area, highlighting and providing an object for the selected annotation data in the first area. According to claim 7,
The steps provided above are
In response to an input for selecting one of the determined at least one relationship data in the second area, objects and indicators corresponding to the selected relationship data in the first area are highlighted and provided. An electronic device providing a method for generating a scene graph,
communications department; and
including a control unit;
The control unit
Receiving a first satellite image through the communication unit;
Identifying at least one object in the received first satellite image;
generating annotation data to the at least one object identified above;
determining relationship data between the identified at least one object based on the generated annotation data;
providing a second satellite image including the annotation data and the relation data to a first region;
The control unit
Selecting and determining one of a plurality of selectable relational data corresponding to an input from a first object to a second object among the identified at least one object;
The input from the first object to the second object includes a click and drag input with the first object as a starting point and the second object as an end point;
The control unit
Check whether the first object and the second object are the same object;
When the first object and the second object are not identical, first relation data for a relationship between the first object and the second object is determined, and when the first object and the second object are identical, the first relation data is determined. An electronic device that determines second relationship data including information about an object. As a non-transitory computer-readable storage medium,
a medium configured to store computer readable instructions;
The computer readable instructions, when executed by a processor, cause the processor to:
receiving a first satellite image;
identifying at least one object from the received first satellite image;
generating annotation data corresponding to at least one identified object;
determining relation data between the identified at least one object based on the created annotation data; and
providing a second satellite image including the annotation data and the relationship data to a first region;
The determining step is
Selecting and determining one of a plurality of selectable relational data corresponding to an input from a first object to a second object among the identified at least one object,
The input from the first object to the second object includes a click and drag input with the first object as a starting point and the second object as an end point;
The step of selecting and determining one of the plurality of relationship data
checking whether the first object and the second object are the same object; and
When the first object and the second object are not identical, first relation data for a relationship between the first object and the second object is determined, and when the first object and the second object are identical, the first relation data is determined. A non-transitory computer-readable storage medium for performing a scene graph generation method comprising determining second relationship data including information about an object.

Images