KR102567496B1 - Control system including neural network that learns visual objects - Google Patents

Abstract

Disclosed is a control system. The control system comprises: at least one display; a first processor controlling the at least one display to divide the at least one display into a plurality of zones; a first electronic device comprising a first communication circuit configured to be connected to other electronic devices through a neural network and a network; a second processor; and a second electronic device comprising a second communication circuit configured to be connected to the first electronic device via the network. The first processor may be configured to receive information processed through the second processor; display the received information on one zone designated among the plurality of zones; based on the received information, identify, through the neural network, first information processed with the high frequency through the second processor or second information recently processed; identify the access to the network through the second communication circuit; and in response to the identification, display the first information or the second information on the designated one zone. Other embodiments may be provided. The control system aims to provide a user environment according to the histories of users accessing the video wall.

Description

Control system including neural network learning visual objects {CONTROL SYSTEM INCLUDING NEURAL NETWORK THAT LEARNS VISUAL OBJECTS}

Various embodiments relate to a control system including a neural network learning a visual object.

A video wall in which a control system is utilized may refer to a large display panel in which a plurality of displays are connected. A system including a video wall can display images on a large screen, and thus can be used in a control system for purposes such as crime prevention, evidence collection, and disaster prevention.

The video wall uses a large display panel and can provide various information. A control system utilizing a large display panel provided by a video wall allows a plurality of users to access and perform user tasks. A control system including a large display seeks to provide a user environment according to a user's history accessing a video wall.

The technical problem to be achieved in this document is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

According to an embodiment, the control system is configured to be connected to at least one display, a first processor controlling the at least one display to be divided into a plurality of zones, a neural network, and other electronic devices through the network. and a second electronic device including a second communication circuit configured to be connected to a second processor and the first electronic device through the network. According to an embodiment, the first processor receives information processed through the second processor, displays the received information in a designated one area among the plurality of areas, and based on the received information In this way, through the neural network, first information that is frequently processed through the second processor or second information that is recently processed is identified, and access to the network is identified through the second communication circuit, In response to the identification, the first information or the second information may be displayed on the designated one area.

According to an embodiment, a method of a control system including an electronic device receives information processed through a processor of another electronic device, and transmits the received information to a designated one of a plurality of areas divided on at least one display. In the area of, based on the received information, first information that is frequently processed through a neural network or processor of the other electronic device or second information that is recently processed is identified, and the other electronic device may be configured to identify access to a network connected to the electronic device, and display the first information or the second information on the designated area in response to the identification.

According to an embodiment, a control system using a neural network or a virtual environment (eg, a meta bus) provides a user environment corresponding to each user to perform tasks in a divided area within a video wall (large display). can be done Each divided area may provide a separate environment according to the user. As the control system is implemented in a virtual environment, it can overcome temporal or spatial limitations.

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below. will be.

1 is a block diagram of a control system according to various embodiments.
2 is an exemplary diagram for explaining a neural network obtained by an electronic device from a set of parameters stored in a memory according to an exemplary embodiment.
3 illustrates an example of a display of a control system using a plurality of regions according to an exemplary embodiment.
4 is a diagram illustrating examples of users performing different tasks, according to an exemplary embodiment.
5 is a diagram illustratively illustrating independence of work in each area according to an embodiment.
6 is a diagram illustrating an example of a user's work accessible to some areas among a plurality of areas, according to an exemplary embodiment.
7A, 7B, and 7C are diagrams illustrating exemplarily controlling a control system by accessing a virtual control system by utilizing an external electronic device, according to an embodiment.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only illustrated for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

Embodiments according to the concept of the present invention can apply various changes and can have various forms, so the embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosures, and includes modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these examples. Like reference numerals in each figure indicate like elements.

1 is a block diagram of a control system according to various embodiments.

Referring to FIG. 1 , the control system 100 may obtain information from cameras including CCTVs in order to maintain public order and/or safety in an area. The control system 100 may obtain information from cameras installed in buildings or factories in order to monitor building maintenance or factory product manufacturing processes. Users using the control system 100 may monitor the display 150 and perform actions based on monitoring results. The action may include calling a rescue team, emergency broadcasting, process modification or maintenance. The control system 100 may be configured to provide an environment related to the user's work on a screen designated according to the user accessing the control system 100 so as to perform tasks assigned to the user in the control system 100. .

According to an embodiment, the control system 100 includes a first electronic device 101, a second electronic device 102, a third electronic device 103, a fourth electronic device 104, and/or a display ( 150) may be included. The first electronic device 101 controls the display 150, and other electronic devices (eg, the second electronic device 102, the third electronic device 103, and/or the fourth electronic device 104). ) may be a device capable of exchanging data from. The first electronic device 101 may be a server managing the control system 100 . According to an embodiment, the first electronic device 101 may include a server of a service provider. A server may include one or more PCs and/or workstations. In one embodiment, the camera connected to the first electronic device 101 may include a Closed Circuit Tele-Vision (CCTV) that captures video from a designated location. The electronic device 101 according to an embodiment may simultaneously receive and process videos captured by one or more cameras disposed at different locations.

The first electronic device 101 , the second electronic device 102 , the third electronic device 103 , and the fourth electronic device 104 may access the network 199 . The network 199 may include a wired network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The network is a wireless network such as LTE (Long Term Evolution), 5g NR (New Radio), WiFi (Wireless Fidelity), Zigbee, NFC (Near Field Communication), Bluetooth, BLE (Bluetooth Low-Energy) or a combination thereof can include Although the first electronic device 101 , the second electronic device 102 , the third electronic device 103 , and the fourth electronic device 104 are shown connected through the network 199 , the first electronic device 101 ), the second electronic device 102, the third electronic device 103, and the fourth electronic device 104 may be directly connected through cables.

According to an embodiment, the first electronic device 101 may include at least one of a memory 110 , a processor 120 , a communication circuit 130 , and a neural network 140 . The memory 110, the processor 120, the communication circuit 130 and the neural network 140, and the display 150 may be electrically or operatively connected to each other by electronic devices such as a communication bus. The type and/or number of hardware included in the first electronic device 101 is not limited to that shown in FIG. 1 . For example, the electronic device 101 may include only some of the hardware components shown in FIG. 1 .

The memory 110 of the first electronic device 101 according to an embodiment may include a hardware component for storing data and/or instructions input and/or output to the processor 120 . The memory 110 may include, for example, volatile memory such as random-access memory (RAM) and/or non-volatile memory such as read-only memory (ROM). there is. The volatile memory may include, for example, at least one of Dynamic RAM (DRAM), Static RAM (SRAM), Cache RAM, and Pseudo SRAM (PSRAM). The nonvolatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disk, and an embedded multi media card (eMMC). can

In the memory 110 , one or more instructions indicating an operation to be performed on data by the processor 120 may be stored. A set of instructions may refer to firmware, operating system, process, routine, sub-routine and/or application. For example, the first electronic device 101 and/or the processor 120 of the electronic device 101 may execute a set of a plurality of instructions distributed in the form of an application.

The first electronic device 101 according to an embodiment may manage information stored in the memory 110 based on a database (DB) stored in the memory 110 . In one embodiment, the database is a set of information systematized to be shared between independent applications running within the electronic device 101 and/or a plurality of electronic devices including the electronic device 101, or It may include at least one of one or more applications that manage the information. In the set of information, different pieces of information may be combined with each other based on units such as type, column, record, and/or table. Combinations of information may be used to add, delete, update, and retrieve information within the database. For example, in a state in which the electronic device 101 searches for information of a specified condition, the electronic device 101 may further identify information satisfying the condition and other information combined with the information using a database. there is.

The processor 120 of the electronic device 101 according to an embodiment may include a hardware component for processing data based on one or more instructions. Hardware components for processing data may include, for example, an Arithmetic and Logic Unit (ALU), a Field Programmable Gate Array (FPGA), and/or a Central Processing Unit (CPU). The number of processors 120 may be one or more. For example, the processor 120 may have a structure of a multi-core processor such as a dual core, quad core, or hexa core.

The communication circuit 130 of the electronic device 101 according to an embodiment includes a first electronic device 101 and another electronic device (eg, a second electronic device 102, a It may include a hardware component for supporting transmission and/or reception of electrical signals between the third electronic device 103 and the fourth electronic device 104. The communication circuit 130 may include, for example, at least one of a modem (MODEM), an antenna, and an optical/electronic (O/E) converter. The communication circuit 130 includes Ethernet, Local Area Network (LAN), Wide Area Network (WAN), Wireless Fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Long Term Evolution (LTE), Transmission and/or reception of electrical signals may be supported based on various types of protocols such as 5G New Radio (NR).

The display 150 of the electronic device 101 according to an embodiment may output visualized information to the user. The number of displays 150 connectable to the electronic device 101 may be one or more. When a plurality of displays 150 are connected to the electronic device 101, the electronic device 101 may, for example, receive video from a plurality of cameras connected to the electronic device 101 on each of the plurality of displays 150. can be displayed.

In one embodiment, the display 150 may be controlled by a controller such as a graphic processing unit (GPU) to output visualized information to a user. In one embodiment, the GPU may be included as at least part of the processor 120 . The display 240 may include a flat panel display (FPD) and/or electronic paper. The FPD may include a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), and/or one or more Light Emitting Diodes (LEDs). The LED may include organic LED (OLED).

The electronic device 101 according to various embodiments may recognize a video received from a camera and identify an event being captured by the camera. If the identified event is a designated event, eg, an event that requires an emergency response, such as an emergency patient, the electronic device 101 may notify the user of the occurrence of the designated event. In response to identification of the designated event, the electronic device 101 according to an embodiment may transmit a signal notifying occurrence of the designated event to an external electronic device corresponding to the designated event. For example, when the external electronic device is a server of an emergency rescue center, the first electronic device 101 responds to identification of a designated event such as an occurrence of an emergency patient, and sends a signal informing the external electronic device of the occurrence of a designated event (eg , emergency rescue team dispatch request) can be transmitted.

The first electronic device 101 according to various embodiments may store at least a portion of a video received from a camera in a database based on the identified event. For example, the first electronic device 101 may store at least a portion of the video and a combination of events corresponding to the portion in a database. The stored combinations may be used for event-based retrieval and/or analysis of video.

According to an embodiment, a set of parameters related to the neural network 140 may be stored in the memory 110 of the first electronic device 101 . The neural network 140 is a software or hardware implemented cognitive model that mimics the computational power of a biological system using a large number of artificial neurons (or nodes). The neural network 140 may perform a human cognitive function or a learning process through artificial neurons. Parameters related to the neural network 140 may represent, for example, weights assigned to a plurality of nodes included in the neural network 140 and/or connections between the plurality of nodes. A structure of the neural network 140 represented by a set of parameters stored in the memory 110 of the first electronic device 101 according to an embodiment will be described later with reference to FIG. 2 . The number of neural networks 140 stored in the memory 110 is not limited to that shown in FIG. 1 , and sets of parameters corresponding to each of a plurality of neural networks may be stored in the memory 110 .

According to an embodiment, the processor 120 may be referred to as a first processor in terms of being mounted in the first electronic device 101 . Each of the second electronic device 102 , the third electronic device 103 , and the fourth electronic device 104 may include a second processor, a third processor, and a fourth processor. The first processor may receive information processed through the second processor. Information processed through the second processor may be information processed by a user accessing the second electronic device. The first processor may display the received information on a designated one area among the plurality of areas in the display 150 . The first processor may identify first information that is frequently processed through the neural network or second information that is recently processed through the second processor, based on the received information. The first processor is configured to identify that the second electronic device 102 accesses the network and, in response to the identification, display the first information or the second information on the designated area. can

According to an embodiment, the first processor may match the identified first information or the second information to correspond to the identifier based on the identifier input through the second electronic device. For example, based on the ID of the user of the second electronic device, the first information or the second information stored in the memory 110 may be matched with the user. If the first processor identifies that the identifier is input through a third electronic device that is different from the second electronic device based on the matched information, the first processor converts the identified first information or the second information to the designated one. It may be configured to display the first information or the second information on an area of . For example, even if the user accesses through the third electronic device 103 in addition to the second electronic device 102, the processor 120, based on the identifier, the third electronic device in the divided area of the display 150 The first information or the second information may be displayed in an area corresponding to (103).

According to an embodiment, the second processor of the second electronic device 102 determines the size of the visual object based on an input for adjusting the size of the visual object included in the first information or the second information. Third information related to adjusting may be transmitted to the first electronic device. The first processor may be configured to display a part of the visual object in the one designated area when the size of the visual object is greater than the size of the one designated area based on receiving the third information. . A specific example related to this will be described later with reference to FIG. 5 .

According to an embodiment, the first processor may identify administrator authority corresponding to the identifier based on the identifier input through the second electronic device. The first processor may be configured to allocate at least one area among the plurality of areas to the identifier based on the manager authority.

The first processor may be configured to display a partial visual object among information obtained from the second electronic device on all of the plurality of regions, based on the manager authority being identified as the first authority. Administrator authority is one of the users of the first electronic device 101, the second electronic device 102, the third electronic device 103, and the fourth electronic device 104, and a person who manages the other users. can be When accessing with a manager ID, the user can provide a visual object on the entire display screen.

The first processor is configured to display some visual objects among information obtained from the second electronic device on some areas including a designated area among the plurality of areas, based on the administrator authority being identified as the second authority. can be configured. The second authority may be authority not to access the entire area of the display, but to access only a part of a designated area. When access is made with the manager ID of the second authority, the user can display a visual object in a designated area of the display.

Control of visual objects by the manager will be described later with reference to FIG. 6 .

According to an embodiment, the first processor connects an external electronic device different from the second electronic device 102 , the third electronic device 103 , and the fourth electronic device 104 through the network 199 . can be identified. The external electronic device may be an electronic device different from an electronic device constituting the control system. The display screen of the display 150 including the plurality of areas may be transmitted to the external electronic device. The display screen of the display 150 is a display screen implemented in a virtual metabus environment and can be displayed on an external electronic device. The external electronic device may include an external processor, and the external processor may identify data input on one area of the transmitted display screen. The external processor may identify data input by the user through external electronic means. The external processor may be configured to display the input in an area corresponding to the one area on the display as the first processor based on the input of the identified data.

According to an embodiment, the first processor, when identifying another input to the area corresponding to the one area through the second electronic device, sends the third electronic device to the display area of the third electronic device. The other input may be displayed in an area corresponding to the one area.

An example of a control system that operates in relation to the metabus environment will be described later with reference to FIGS. 7A, 7B, and 7C.

2 is an exemplary diagram for explaining a neural network 140 obtained from a set of parameters stored in a memory by an electronic device according to an exemplary embodiment. The neural network 140 of FIG. 2 may correspond to the neural network 140 of FIG. 1 .

Referring to FIG. 2 , the neural network 140 may include a plurality of layers. For example, the neural network 140 may include an input layer 141 , one or more hidden layers 142 , and an output layer 143 . The input layer 141 may receive a vector representing input data (eg, a vector having elements corresponding to the number of nodes included in the input layer 141). Signals generated by each of the nodes in the input layer 141 generated by the input data may be transmitted from the input layer 141 to the hidden layers 142 . The output layer 143 may generate output data of the neural network 140 based on one or more signals received from the hidden layers 142 . The output data may include, for example, a vector having elements corresponding to the number of nodes included in the output layer 143 .

Referring to FIG. 2 , one or more hidden layers 142 may be positioned between the input layer 141 and the output layer 143, and input data transmitted through the input layer 141 may be converted into values that are easy to predict. can be converted The input layer 141, one or more hidden layers 142, and the output layer 143 may include a plurality of nodes. The one or more hidden layers 142 may be convolution filters or fully connected layers in a convolutional neural network (CNN), or various types of filters or layers grouped based on special functions or characteristics. can In one embodiment, one or more hidden layers 142 may be a layer based on a recurrent neural network (RNN) whose output values are input back to the hidden layer at the current time. The neural network 140 according to an embodiment may include a plurality of hidden layers 142 to form a deep neural network. Training a deep neural network is called deep learning. Among the nodes of the neural network 140, nodes included in the hidden layers 142 are referred to as hidden nodes.

Nodes included in the input layer 141 and one or more hidden layers 142 may be connected to each other through a connection line having a connection weight, and nodes included in the hidden layer and the output layer may also be connected to each other through a connection line having a connection weight. can be connected Tuning and/or training the neural network 140 is performed on each of the layers (eg, the input layer 141, one or more hidden layers 142, and the output layer 143) included in the neural network 140. It may mean changing the connection weight between included nodes. Tuning of the neural network 140 may be performed based on, for example, supervised learning and/or unsupervised learning.

An electronic device according to an embodiment may tune the neural network 140 based on reinforcement learning in unsupervised learning. For example, the electronic device may change policy information used by the neural network 140 to control the agent based on an interaction between the agent and the environment. The electronic device according to an embodiment may cause the policy information to be changed by the neural network 140 in order to maximize the agent's goal and/or reward through the interaction.

3 illustrates an example of a display of a control system using a plurality of regions according to an exemplary embodiment.

Referring to FIG. 3 , the control system 100 may include a display 150 . The display 150 may include a plurality of regions 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , and 219 . The plurality of regions 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , and 219 may be virtual regions distinguished on the display 150 . The plurality of areas 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , and 219 may be divided by a processor (eg, the processor 120 of FIG. 1 ). The processor 120 may be divided to correspond to users 201a, 201b, 201c, 201d, 201e, 201f, 201g, 201h, and 201i registered to use the display 150 . The divided regions 211, 212, 213, 214, 215, 216, 217, 218, and 219 may have the same size or shape, but are not limited thereto. For example, the shape or size of the first area 211 used by the first user 201a and the second area 212 used by the second user 201b may be the same. For another example, the shape or size of the first area 211 used by the first user 201a and the third area 213 used by the third user 201c may be different. The size or shape of the plurality of regions 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , and 219 may be previously designated for each user. For example, the processor 120 assigns the plurality of regions 211, 212, 213, 214, 215, 216, 217, 218, and 219 to the users 201a, 201b, and 219 based on a predetermined size or shape. 201c, 201d, 201e, 201f, 201g, 201h, 201i).

According to an embodiment, a plurality of areas 211, 212, 213, 214, 215, 216, 217, and 218 are provided according to users 201a, 201b, 201c, 201d, 201e, 201f, 201g, 201h, and 201i. , 219) may be different. For example, the processor 201a, 201b, 201c, 201d, 201e, 201f, 201g, 201h, and 201i through a neural network (eg, the neural network 140 of FIG. 1) that has learned previous work histories. 120 may provide different environments to each of the plurality of areas 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , and 219 . For example, the processor 120 may provide a work environment to the first user 201a through the first area 211 based on the previous use history of the first user 201a. The processor 120 may provide a work environment to the second user 201b through the second area 212 based on the previous use history of the second user 201b.

Regarding the different working environments provided by the processor 120, it will be described based on FIG. 4.

4 is a diagram illustrating examples of users performing different tasks, according to an exemplary embodiment.

Referring to FIG. 4 , users 201a, 201b, 201c, 201d, and 201e may perform assigned tasks. The processor 120 may learn work contents or main tasks of the users 201a, 201b, 201c, 201d, and 201e through the neural network 140 . For example, the first user 201a may perform a task of mainly observing the first visual object 301 in the entire work environment 300 . The second user 201b may perform a task of managing the entire work environment 300 . The third user 201c may manage a part of the environment 310 of the entire work environment 300 . For example, the third user 201c may manage the first visual object 301 and the second visual object 302 within the partial environment 310 . The fourth user 201d may perform a task of observing the second visual object 302 in the entire work environment 300 . The fifth user 201e may perform a task of observing the third visual object 303 in the entire work environment 300 .

The processor 120 may learn the work pattern of the first user 201a through the neural network 140 . For example, the processor 120 may identify a visual object mainly utilized by the first user 201a from among visual objects within the entire work environment 300 . When the first user 201a enlarges the first visual object 301 or performs an operation of clicking the first visual object 301, the neural network 140 performs an operation of clicking or enlarging the first visual object 301 by the first user 201a. Information related to the visual object 301 may be learned. For example, the neural network 140 may convert the first visual object 301 into data including one or more vectors and output output data to an output layer through an input layer. The neural network 140 may identify the first visual object 301 based on the output data. The first visual object 301 may include similar objects as well as identical objects. For example, the visual objects may be the same person or thing. For another example, the visual object may be another person wearing the same costume or a means of transportation of the same type. For example, the neural network 140 may classify a vehicle and a person moving on a road. The neural network 140 may subdivide and classify vehicles or people according to learning stages or user patterns. For example, the neural network 140 may classify vehicles as cars, motorcycles, or bicycles, and the neural network 140 may classify people as children, adults, or genders.

According to an embodiment, the neural network 140 may identify the main observation target of the first user 201a by referring to the work pattern of the first user 201a. The first user 201a may observe the entire work environment 300 , check the first visual object 301 , and provide an event related to the first visual object 301 . The first user 201a may not provide an event while observing other visual objects 302 and 303 . The processor 120 may identify a main task object of the first user 201a from among the visual objects identified through the neural network 140, and store the identified task object in a memory (eg, the memory 110 of FIG. 1 ). )) can be stored. For example, when the number of events provided to the first visual object 301 is greater than the number of events provided to other visual objects by the first user 201a, the processor 120 via the neural network 140 Among the identified visual objects, the first visual object 301 may be identified as the main work object of the first user 201a.

According to an embodiment, the neural network 140 may identify the main observation target of the second user 201b by referring to the work pattern of the second user 201b. The second user 201b observes the entire work environment 300, checks the first visual object 301, the second visual object 302, and/or the third visual object 303, and the entire environment An event related to (300) may be provided. The second user 201b may observe the visual objects 301, 302, and 303 and may not provide an event. The processor 120 may identify that the entire work environment 300 including visual objects identified through the neural network 140 is observed, and the identified entire work environment is the main task of the second user 201b. It can be stored in a memory (eg, memory 110 of FIG. 1) as a pattern. For example, when the number of times the first user 201a observes the work environment including the visual objects 301, 302, and 303 is greater than the number of events provided to other visual objects, the processor 120: Among the visual objects identified through the neural network 140 and the entire working environment including the visual objects, the main task of the second user 201b may be identified as the entire working environment 300 .

According to an embodiment, the neural network 140 may identify the main observation target of the third user 201c by referring to the work pattern of the third user 201c. The third user 201c observes the entire work environment 300, checks the first visual object 301 and the second visual object 302, and the first visual object 301 and the second visual object ( 302) and related events may be provided. The third user 201c observes another third visual object 303 and may not provide an event. The processor 120 may identify a main task object of the third user 201c from among the visual objects identified through the neural network 140, and store the identified task object in a memory (eg, the memory 110 of FIG. 1 ). )) can be stored. For example, when the number of events provided to the first visual object 301 and the second visual object 302 is greater than the number of events provided to other visual objects by the third user 201c, the processor 120 , among the visual objects identified through the neural network 140, the first visual object 301 and the second visual object 302 may be identified as the main task objects of the third user 201c.

According to an embodiment, the neural network 140 may identify the main observation target of the fourth user 201d by referring to the work pattern of the fourth user 201d. The fourth user 201d may observe the entire work environment 300 , check the second visual object 302 , and provide an event related to the second visual object 302 . The fourth user 201d may not provide an event while observing the other visual objects 301 and 303 . The processor 120 may identify a main task object of the fourth user 201d from among the visual objects identified through the neural network 140, and store the identified task object in a memory (eg, the memory 110 of FIG. 1 ). )) can be stored. For example, when the number of events provided to the second visual object 302 is greater than the number of events provided to other visual objects by the fourth user 201d, the processor 120 via the neural network 140 Among the identified visual objects, the second visual object 302 may be identified as the main work object of the fourth user 201d.

According to an embodiment, the neural network 140 may identify the main observation target of the fifth user 201e by referring to the work pattern of the fifth user 201e. The fifth user 201e may observe the entire work environment 300 , check the third visual object 303 , and provide an event related to the third visual object 303 . The fifth user 201e may not provide an event while observing the other visual objects 301 and 302 . The processor 120 may identify a main task object of the fifth user 201e from among the visual objects identified through the neural network 140, and store the identified task object in a memory (eg, the memory 110 of FIG. 1 ). )) can be stored. For example, when the number of events provided to the third visual object 303 is greater than the number of events provided to other visual objects by the fifth user 201e, the processor 120, through the neural network 140 Among the identified visual objects, the third visual object 303 may be identified as the main work object of the fifth user 201e.

According to an embodiment, the processor 120 may provide a work object and a work environment suitable for each user based on identifying the main work object or work environment of the users. According to an embodiment, the processor 120 is connected to the first region 211 of the display 150 (eg, the second electronic device 102 of FIG. 1 , the third electronic device 103 or The fourth electronic device 104) may be identified, and users 201a, 201b, 201c, 201d, and 201e accessed through the external electronic device may be identified. The processor 120 determines an external electronic device connected to the first area 211 based on a work object or work environment for each of the users 201a, 201b, 201c, 201d, and 201e identified through the neural network 140. A work object or work environment corresponding to the first user 201a inputting the identifier (ID) may be provided. For example, the processor 120 assigns the first user 201a to the first area 211 based on identifying that the first user 201a accesses an external electronic device connected to the first area 211 . The first visual object 301 recognized as the main work object of ) may be provided to the first area 211 .

According to an embodiment, the processor 120 determines the second area 211 based on a work object or work environment for each of the users 201a, 201b, 201c, 201d, and 201e identified through the neural network 140. ), a work object or work environment corresponding to the second user 202a inputting an identifier (ID) into an external electronic device connected to the second user 202a may be provided. For example, the processor 120 assigns the second user 201b to the second area 212 based on identifying that the second user 202a accesses an external electronic device connected to the second area 212. The entire work environment 300 recognized as the main work of ) may be provided to the second area 212 .

According to an embodiment, the processor 120 determines the third area 213 based on a work object or work environment for each of the users 201a, 201b, 201c, 201d, and 201e identified through the neural network 140. A work object or work environment corresponding to the third user 201c who inputs an identifier (ID) into an external electronic device connected to ) may be provided. For example, the processor 120 assigns the third user 201c to the third area 213 based on identifying that the third user 201c accesses an external electronic device connected to the third area 213. A partial environment 310 including the first visual object 301 and the second visual object 302 recognized as the main task object of ) may be provided to the third area 211 .

According to an embodiment, the processor 120 determines the fourth area 214 based on a work object or work environment for each of the users 201a, 201b, 201c, 201d, and 201e identified through the neural network 140. A work object or work environment corresponding to the fourth user 201d who inputs an identifier (ID) into an external electronic device connected to ) may be provided. For example, the processor 120 assigns the fourth user 201d to the fourth area 214 based on identifying that the fourth user 201d accesses an external electronic device connected to the fourth area 214 . The second visual object 302 recognized as the main work object of ) may be provided to the fourth area 214 .

According to an embodiment, the processor 120 determines the fifth area 215 based on a work object or work environment for each of the users 201a, 201b, 201c, 201d, and 201e identified through the neural network 140. ), a work object or work environment corresponding to the fifth user 201e who has input an identifier (ID) into an external electronic device connected to the external electronic device may be provided. For example, the processor 120 assigns the fifth user 201e to the fifth area 215 based on identifying that the fifth user 201e accesses an external electronic device connected to the fifth area 215 . The third visual object 303 recognized as the main work object of ) may be provided to the fifth area 215 .

According to an embodiment, an external electronic device used by the first user 201a is connected to the first area 211, and an external electronic device used by the second user 201b is connected to the second area 212. The external electronic device used by the third user 201c is connected to the third area 213, the external electronic device used by the fourth user 201d is connected to the fourth area 214, and the fifth Although it has been described that an external electronic device used by the fifth user 201e is connected to the area 211, it is not limited thereto. For example, when the first user 201a accesses through an external electronic device that is different from the external electronic device connected to the first area 211, the first visual object 301 is displayed in the area connected to the other external electronic device. can be displayed When the first user 201a accesses the control system 100 through an external electronic device connected to the second area 212, the processor 120 converts the first visual object 301 to the second area 212. can be displayed on

According to one embodiment, each visual object has been described as an object observed in the control system by taking a vehicle, a person, etc. as an example, but is not limited thereto. For example, the visual object may represent a process line or assembly parts of a factory, and may include a transportation path and transportation device of a transportation module. The visual object may be variously applied depending on the environment to which the control system 100 is applied.

5 is a diagram illustratively illustrating independence of work in each area according to an embodiment.

Referring to FIG. 5 , each area of the display 150 may be configured to operate independently. An exemplary first screen 401a including various visual objects may be provided in the first region 211, and an exemplary second screen 402 including various visual objects may be provided in the second region 212. there is. When the first user 201a using an external electronic device connected to the first area 211 enlarges a part of the first screen 401a and uses the enlarged screen 401b, the enlarged screen 401b is , may be provided only within the first region 211 . For example, the first screen 401a having a size corresponding to the original area P1 may be enlarged to a size corresponding to the enlarged area P2. The enlarged area P2 may have an area larger than that of the first area 211 . For example, the enlarged area P2 may overlap a portion of the second area 212 . The processor 120 does not display the first screen 401a converted to the enlarged area P2 on the second area 212, and displays only the area overlapping the first area 211 of the enlarged area P2. can do.

According to an embodiment, the display 150 operates as a single display, but each area may be used as a separate device. The processor may be configured to display only tasks of a designated user in areas allocated to each user. The control system 100 can increase the concentration of work and improve work by providing each user with an area in which only their work is displayed.

6 is a diagram illustrating an example of a user's work accessible to some areas among a plurality of areas, according to an exemplary embodiment.

Referring to FIG. 6 , the display 150 may provide a screen including visual objects allocated to regions 211 , 212 , and 213 for each user. When a manager who is distinguished from a plurality of users accesses the control system 100 , the manager may display visual objects in the plurality of areas 211 , 212 , and 213 .

According to one embodiment, the manager may be a general manager who manages the entire control system, or a manager (eg, team leader) who manages some of the users using the control system. The manager may access the control system 100 based on an external electronic device or ID assigned to the manager. A manager may display the visual object 510 on the first area 211 , the second area 212 , and the third area 213 . For example, the visual object 510 may be displayed across the first area 211 , the second area 212 and the third area 213 . For example, a first user 211a using the first area 211, a second user 211b using the second area 212, and a third user 211c using the third area 213, It could be the team that manages the visual object represented by A. An administrator may be an administrator who manages the first user 211a, the second user 211b, and the third user 211c. The manager controls the entire screen of the first area 211, the second area 212, and the third area 213 used by the first user 211a, the second user 211b, and the third user 211c, who are members of the team. A visual object 510 may be provided through. Contiguous areas such as the first area 211, the second area 212, and the third area 213 are displayed as one visual object 510, but each visual object is displayed in areas that are not contiguous with each other. can provide

7A, 7B, and 7C are diagrams illustrating exemplarily controlling a control system by accessing a virtual control system by utilizing an external electronic device, according to an embodiment.

Referring to FIG. 7A , the control system 100 may use a metaverse or a metaverse autonomous twin. For example, through a program that implements the control system 100, the user can select an external electronic device different from the second electronic device 102, the third electronic device 103, and the fourth electronic device 104 of FIG. It is possible to access the program implementing the control system 100 through. The external electronic device may visually provide the meta bus environment 601 to the user. A user may access each area of the actual display 150 within the meta bus environment 601 .

7A shows that the third user 201c observes the second area 212 through an external electronic device. An external electronic device may access an electronic device implementing the metabus environment 601 (eg, the electronic device 101 of FIG. 1 ) through a network (eg, the network 199 of FIG. 1 ). The external electronic device may include a third user's personal terminal (eg, mobile phone, tablet PC) or PC. A third user may click or transform one visual object 611 displayed in the second area 212 through the metaverse environment 601 . For example, the second user 201b performs a task (eg, naming or labeling) on the visual object 611, so that the visual object 611 is displayed as a modified visual object 611'. can Although the transformed visual object 611 ′ was executed in the user's external electronic device, the executed contents may be displayed on the display 150 through the first electronic device 101 . In this regard, it will be described based on FIGS. 7B and 7C.

Referring to FIGS. 7B and 7C , a second user 201b may convert a visual object 611 corresponding to the second area 212 into a modified visual object 611' through an external electronic device. . Upon receiving the information related to the transformed visual object 611', the first electronic device 101 may display related content within the second area 212 of the display 150. For example, based on receiving information related to the transformed visual object 611', the processor 120 may reflect the information to a visual object corresponding to the transformed visual object 611'. The processor 120 may identify the second visual object 302 as a visual object corresponding to the visual object 611 and the transformed visual object 611'. Based on the identification of the second visual object 302, the processor 120 converts the second visual object 302 into a transformed second visual object 302' corresponding to the transformed visual object 611'. can be converted

According to one embodiment, the processor 120 only transforms the visual object of the second area 212 within the metaverse environment 601 (eg, the second visual object 302 within the second area 212). can transform However, the processor 120 is not limited thereto, and the second visual object displayed on the display 150 is based on the transformed visual object (eg, the second visual object 302) within the metaverse environment 601. (302) can be modified. For example, the second visual object 302 may be displayed on the second area 212 , the third area 213 , and the fourth area 214 . Based on recognizing that the transformed visual object 611' is the second visual object 302, the processor 120 converts all second visual objects 302 displayed on the display 150 to the transformed second visual object. (302').

According to the above-described embodiment, the control system 100 implements a metaverse environment 610 that can be accessed from the outside, so that the user can access electronic devices connected to the control system 100 (eg, the first electronic device 101 ), through electronic devices other than the second electronic device 102, the third electronic device 103, and the fourth electronic device 104, it is possible to access the virtually implemented control system. The user may perform a control task through the control system in the metabus environment 601, and data related to the performed control task may be transmitted to the first electronic device 101. Through the implementation of the meta bus environment 601, the user can use the control system 100 without restriction of location and/or time.

According to the above-described embodiment, the control system includes at least one display, a first processor controlling the at least one display to be divided into a plurality of zones, a neural network, and connected to other electronic devices through the network. It may include a first electronic device including a configured first communication circuit, a second processor, and a second electronic device including a second communication circuit configured to be connected to the first electronic device through the network.

According to an embodiment, the first processor receives information processed through the second processor, displays the received information in a designated one area among the plurality of areas, and based on the received information In this way, through the neural network, first information that is frequently processed through the second processor or second information that is recently processed is identified, and access to the network is identified through the second communication circuit, In response to the identification, the first information or the second information may be displayed on the designated one area.

According to an embodiment, the first processor matches the identified first information or the second information to correspond to the identifier based on the identifier input through the second electronic device, and the matched information Based on, if it is identified that the identifier is input through a third electronic device distinct from the second electronic device, the identified first information or the second information is displayed on the designated one area. It may be configured to display the second information.

According to an embodiment, the second processor receives third information related to adjusting the size of the visual object based on an input for adjusting the size of the visual object included in the first information or the second information. transmission to the first electronic device, and the first processor, on the basis of receiving the third information, if the size of the visual object is greater than the size of the one designated area, the first processor converts a part of the visual object to the designated one area. It can be configured to display in the area of.

According to an embodiment, the first processor identifies administrator authority corresponding to the identifier based on the identifier input through the second electronic device, and based on the manager authority, among the plurality of areas. It may be configured to assign at least one region to the identifier.

According to an embodiment, the first processor may be configured to display some visual objects among information acquired from the second electronic device on all of the plurality of areas, based on the administrator authority being identified as the first authority. can

According to an embodiment, the first processor, based on the administrator authority being identified as the second authority, converts some visual objects among information obtained from the second electronic device to a designated area among the plurality of areas. It can be configured to display in some areas.

According to an embodiment, the first processor identifies that a third electronic device different from the second electronic device accesses through the network, and, as the third electronic device, the display including the plurality of regions. The display screen of can be transmitted.

The third electronic device includes a third processor, and the third processor identifies an input of data on one area of the transmitted display screen, and based on the input of the identified data, The first processor may be configured to display the input in an area corresponding to the one area on the display.

According to an embodiment, the first processor, when identifying another input to the area corresponding to the one area through the second electronic device, sends the third electronic device to the display area of the third electronic device. The other input may be displayed in an area corresponding to the one area.

According to an embodiment, the neural network is configured to receive information processed through the second processor from the first processor, and to update the first information or the second information based on the received information. It can be.

According to an embodiment, the first information may be acquired by learning visual objects indicated by the user of the second electronic device.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, an electronic device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeably interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integral part or the smallest unit of a part or part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Claims (10)

In the control system,
at least one display;
a first electronic device including a first processor, a neural network, and a first communication circuit that controls to divide the at least one display into a plurality of zones; and
a second electronic device comprising a second processor and a second communication circuit configured to establish a communication channel with the first communication circuit to be connected to the first electronic device through a network; including,
The first processor,
Receiving information processed through the second processor;
Displaying the received information in a designated area among the plurality of areas;
Based on the received information, identify first information that is frequently processed through the neural network and through the second processor or second information that is recently processed,
Identifying that the second electronic device accesses the network through the second communication circuit;
In response to the identification, configured to display the first information or the second information on the designated one area;
The neural network,
Receiving information processed through the second processor from the first processor;
Based on the received information, configured to update the first information or the second information,
The first information,
Obtained by the user of the second electronic device learning a visual object corresponding to an input by the user among visual objects displayed on the designated area through the neural network,
control system.
According to claim 1,
The first processor,
Based on the identifier input through the second electronic device, matching the identified first information or the second information to correspond to the identifier,
Based on the matched information, if it is identified that the identifier is input through a third electronic device distinct from the second electronic device, the identified first information or the second information is displayed on the designated one area. configured to display the first information or the second information;
control system.
According to claim 1,
The second processor,
Transmitting third information related to adjusting the size of the visual object to the first electronic device based on an input for adjusting the size of the visual object included in the first information or the second information;
The first processor,
Based on receiving the third information, if the size of the visual object is greater than the size of the one designated area, displaying a part of the visual object in the one designated area,
control system.
According to claim 1,
The first processor,
Based on the identifier input through the second electronic device, an administrator authority corresponding to the identifier is identified;
Based on the manager authority, at least one area among the plurality of areas is configured to be allocated to the identifier,
control system.
According to claim 4,
The first processor,
Based on the administrator authority being identified as the first authority, some visual objects among the information obtained from the second electronic device are displayed on all of the plurality of regions.
control system.
According to claim 4,
The first processor,
Based on the administrator authority being identified as the second authority, some visual objects of information obtained from the second electronic device are displayed in some areas including a designated area among the plurality of areas,
control system.
According to claim 1,
The first processor,
Identifying that a third electronic device different from the second electronic device accesses through the network;
transmits a display screen of the display including the plurality of regions to the third electronic device;
The third electronic device,
a third processor;
The third processor,
On an area of one of the transmitted display screens, identify input of data;
Based on the input of the identified data, the first processor is configured to display the input in an area corresponding to the one area on the display,
control system.
According to claim 7,
When the first processor identifies another input to the area corresponding to the one area through the second electronic device, the first processor responds to the one area in the display area of the third electronic device to the third electronic device. configured to display the other input in the area to be,
control system.
delete According to claim 1,
The first information,
Obtained by learning visual objects indicated by the user of the second electronic device,
control system.