KR20230081250A - Electronic device and method for container screen mirroring - Google Patents

Abstract

An electronic device for mirroring a container screen is provided. The electronic device includes a display, a memory, and a processor, and the memory stores a first application operating on a Linux-based first operating system and a second application operating on an Android-based second operating system, and According to the execution, the processor outputs a first object associated with the execution of the first application and a second object associated with the execution of the second application on the screen of the first operating system, the second operating system and a mirroring server module that outputs an execution screen of the second application on at least a part of the screen of the operating system.

Description

Electronic device and method for container screen mirroring {ELECTRONIC DEVICE AND METHOD FOR CONTAINER SCREEN MIRRORING}

The present invention relates to an electronic device and method for mirroring a container screen.

Recently, as the performance of user's portable devices such as smart phones has rapidly improved and various applications operated in portable devices have been developed, the application area of portable devices has been greatly expanded compared to the past.

On the other hand, most vehicles currently have a vehicle monitor for displaying an image output from an automotive audio/video/navigation device (AVN) controlled by the head unit, and the vehicle monitor is relatively to provide a large screen.

In order to increase driver's convenience and safety, interest in mirroring technology, which uses a user's portable device in conjunction with a vehicle's head unit, has recently increased. In particular, as research and development of autonomous vehicles are actively progressing, interest in mirroring technology capable of displaying and processing various information is growing.

Patent Publication No. 10-2014-0054742

The problem to be solved by the present invention is to smoothly provide interworking functions between different operating systems in using or manipulating applications based on different operating systems, and in particular, a container screen that can easily provide a screen mirroring function in a heterogeneous operating system. An electronic device and method for mirroring are provided.

However, the problem to be solved by the present invention is not limited to the above problem, and other problems may exist.

In a first aspect of the present invention for solving the above problems, an electronic device for container screen mirroring includes a display, a memory, and a processor, and the memory includes a first application operating on a Linux-based first operating system and an Android-based device. stores a second application operating on the basis of a second operating system, and according to the execution of the memory, the processor executes the first object associated with the execution of the first application on the screen of the first operating system and the execution of the second application Outputs a second object associated with the second operating system, and the second operating system includes a mirroring server module that outputs an execution screen of the second application on at least a part of the screen of the first operating system.

In some embodiments of the present invention, the mirroring server module of the second operating system is connected to the Surfaceflinger module of the second operating system and the Wayland server module of the first operating system to provide data for mirroring (hereinafter , mirroring data) can be transmitted and received.

In some embodiments of the present invention, the mirroring server module is a Surfaceflinger that requests and receives mirroring data generated by a Hardware Composer (HWC) module of the second operating system through a Surfaceflinger module. A communication unit, a data converter for converting the mirroring data into data in a form processable by the first operating system, a shared memory for storing the converted mirroring data, and a Wayland server of the first operating system for mirroring data stored in the shared memory. It may include a Wayland client unit that transmits to the module.

In some embodiments of the present invention, the Surface Flinger communication unit may receive and store the mirroring data based on frame data of a raw data type at each predetermined frame update timing.

In some embodiments of the present invention, the data conversion unit may include a bitmap converter for converting the mirroring data of the original data type stored at each frame update timing into data in a form processable by the first operating system for each frame unit. .

In some embodiments of the present invention, the mirroring server module may register information of the shared memory on the first operating system as the second operating system boots.

In some embodiments of the present invention, the data conversion unit converts an input event through an execution screen of a second application being mirrored on a screen of the first operating system into data in a form that can be processed by an Inputflinger module of the second operating system. An event converter that converts and transmits the conversion to the input plunger module may be included.

Also, a container screen mirroring method in an electronic device according to a second aspect of the present invention includes outputting a first object associated with execution of a first application on a screen of a Linux-based first operating system; requesting an output of a second object associated with the execution of a second application on a screen of the first operating system as the Android-based second operating system is booted; and outputting an execution screen of the second application on at least a part of the screen of the first operating system through the mirroring server module of the second operating system according to the request.

In addition, the computer program according to another aspect of the present invention is combined with a computer that is hardware to execute the container screen mirroring method in the electronic device, and is stored in a computer-readable recording medium.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention described above, since an Android screen image is acquired and converted through frame buffer data provided by the Surface Plunger of the Android operating system, rather than using a graphic data buffer used by the hardware composer of the Android operating system, separate sharing It is possible to save by designating a shared memory area. In addition, since the graphic library provided by the vendor (SoC, graphic card, etc.) is used, it can be applied to various vehicle hardware environments at a high speed compared to the prior art, the software has high reusability, and as a result, cost reduction is possible. There are advantages.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of an electronic device in a network environment according to an embodiment of the present invention.
2 is a diagram illustrating an example of components of an electronic device for mirroring a container screen according to an embodiment of the present invention.
3 is a diagram illustrating an example of detailed components of an electronic device for mirroring a container screen according to an embodiment of the present invention.
4 is a diagram for explaining a surface plunger communication unit in one embodiment of the present invention.
5 is a diagram for explaining a bitmap converter according to an embodiment of the present invention.
6 is a diagram for explaining a shared memory and Wayland client unit in one embodiment of the present invention.
7 is a flowchart of a container screen mirroring method in an electronic device according to an embodiment of the present invention.
8 is a flowchart illustrating a process of requesting a mirroring screen according to an embodiment of the present invention.
9 is a flowchart illustrating a process of processing an input event according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

1 is a block diagram of an electronic device 100 in a network environment according to an embodiment of the present invention.

Referring to FIG. 1 , an electronic device 100 in a network environment communicates with an electronic device 101 through a first network (eg, a short-range wireless communication network) or through a second network (eg, a short-distance wireless communication network). It is possible to communicate with the electronic device 102 or the server 103 through a long-distance wireless communication network).

According to one embodiment, the electronic device 100 includes a display 110 , a memory 120 and a processor 130 .

The display 110 may visually provide information to the outside of the electronic device 100 (eg, a user). The display 110 may include, for example, a general display device, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display 110 may include a touch circuitry set to detect a touch or a sensor circuit (eg, a pressure sensor) set to measure the intensity of force generated by the touch. .

The memory 120 may store various data used by at least one component of the electronic device 100 . Here, the memory 120 collectively refers to a non-volatile storage device and a volatile storage device that continuously retain stored information even when power is not supplied.

For example, the memory 120 may include a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD), and a micro SD card. NAND flash memory such as cards, magnetic computer storage devices such as hard disk drives (HDD), and optical disc drives such as CD-ROMs and DVD-ROMs. can

Also, the program stored in the memory 120 may be implemented in the form of software or hardware such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), and may perform predetermined roles.

The processor 130, for example, executes software (eg, a program) to at least one other component (eg, hardware or software component) of the electronic device 100 connected to the processor 130. can be controlled, and various data processing or calculations can be performed.

In addition, the electronic device 100 may further include a communication module (not shown). The communication module establishes a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 100 and an external electronic device (eg, the electronic device 101, the electronic device 102, or the server 103). , and can support communication through an established communication channel. The communication module may include one or more communication processors that operate independently of the processor 130 (eg, an application processor) and support wired communication or wireless communication.

For example, the wired communication module may be implemented as a power line communication device, a telephone line communication device, cable home (MoCA), Ethernet, IEEE1294, an integrated wired home network, and an RS-485 control device. In addition, the wireless communication module may be implemented with wireless LAN (WLAN), Bluetooth, HDR WPAN, UWB, ZigBee, Impulse Radio, 60 GHz WPAN, Binary-CDMA, wireless USB technology, wireless HDMI technology, and the like.

2 is a diagram illustrating an example of components of the electronic device 100 for container screen mirroring according to an embodiment of the present invention.

Referring to FIG. 2 , the electronic device 100 includes an operating system (OS) and can simultaneously execute a plurality of operating systems 200 . In one embodiment of the present invention, the operating system may include a first operating system 210 and a second operating system 220 .

For example, the first operating system 210 may be a Linux-based operating system, and the second operating system 220 may be an Android-based operating system. At this time, the second operating system 220 is driven in a virtualized environment using a Linux container (LXC). In an embodiment of the present invention, the first operating system 210 may be referred to as a host operating system, and the second operating system 220 may be referred to as a guest operating system corresponding to the host.

The first operating system 210 and the second operating system 220 may operate and coexist independently of each other, and may be mutually dependent when operated based on Linux.

3 is a diagram illustrating an example of detailed components of the electronic device 100 for container screen mirroring according to an embodiment of the present invention.

Referring to FIG. 3 , the electronic device 100 includes a plurality of operating systems, and the operating system may include a plurality of heterogeneous operating systems 310 and 320 . For example, as shown in FIG. 2 , a first operating system 310 based on Linux and a second operating system 320 based on Android may be included.

As an embodiment, the first operating system 310 may include a first application, which is at least one application operable in the Linux operating system and may be referred to as an application program. In an example of the present invention, the first application may be a program (first object) for an AVN system.

The first operating system 310 is a main operating system that operates when the electronic device 100 boots, and may be referred to as a host operating system.

As an embodiment, the second operating system 320 may include a second application, which is at least one application operable in the Android operating system and may be referred to as an application program. In an example of the present invention, the second application may be a navigation application (second object) mirrored on the AVN system.

The processor 130 outputs a first object associated with the execution of the first application on the screen of the first operating system 310 according to the execution of the memory 120, and displays a second object associated with the execution of the second application 320. print the object

At this time, one embodiment of the present invention is characterized by including the mirroring server module 330 for allowing the second operating system 320 to output the execution screen of the second application on at least a part of the screen of the first operating system.

Meanwhile, in one embodiment of the present invention, the mirroring server module 330 of the second operating system 320 is connected to the Surfaceflinger module 321 of the second operating system 320 and the first operating system 310's way It is connected to the Wayland server module 311 and can transmit/receive data for mirroring (hereinafter referred to as mirroring data).

In the case of the prior art, in order to output mirroring data on a screen driven by a Linux-based operating system, the combined and generated mirroring data in the Hardware Composer of the Android-based operating system is transferred to Weston (Wayland) of the Linux-based operating system. For this, the Wayland Protocol was used. To this end, a separate bridge (HWC-to Wayland Bridge) had to be configured to interconnect the Hardware Composer of the Android operating system and the Wayland server of the Linux operating system.

At this time, the bridge has a Wayland client interface and is connected to a Wayland server of a Linux-based operating system. Accordingly, the host operating system outputs an Android-based application screen and a Linux-based application screen integrated.

This prior art uses the frame buffer of the Linux kernel and the GBM and Mesa libraries (open source graphic libraries) in transferring the Android container screen to the Linux screen. In order to apply to various vehicle hardware environments, the kernel version, device driver, Optimization and modification in consideration of the Mesa library version are essential.

However, the hardware dependence of the Linux kernel's dmabuf, GBM, and Mesa libraries for frame buffer allocation is high, and the conventional method uses the Mesa library, which requires many modifications depending on the device, device driver, and compatible Linux kernel version. The reusability of the software such as the etc. is low, and as a result, it is difficult to reduce costs.

In order to solve this problem, an embodiment of the present invention does not use the graphic data buffer used by the hardware composer of the Android operating system, but through the frame buffer data provided by the Surface Plunger of the Android operating system, the Android screen image is displayed. Since it acquires and converts, it is possible to designate and store a separate shared memory area, and because it uses a graphic library provided by a vendor (SoC, graphic card, etc.), it is fast in various vehicle hardware environments compared to the prior art. It has the advantage of being applicable.

Specifically, the mirroring server module 330 included in the second operating system 320 according to an embodiment of the present invention includes a surface plunger communication unit 331, a data conversion unit 332, a shared memory 335, and Wayland. A client portion 336 is included.

4 is a diagram for explaining the surface plunger communication unit 331 in one embodiment of the present invention. 5 is a diagram for explaining the bitmap converter 333 in one embodiment of the present invention. 6 is a diagram for explaining the shared memory 335 and the Wayland client unit 336 in one embodiment of the present invention.

In one embodiment, the Surface Plunger communication unit 331 requests and receives mirroring data generated by the hardware composer module 322 of the second operating system 320 through the Surface Plunger module 321 . That is, the Surface Plunger communication unit 331 transfers mirroring data (screen data) generated by the hardware composer module 322 from the mirroring server module 330 through the Surface Plunger module 321 of the second operating system 320. Request and receive real-time screen data to enable processing.

At this time, the Surface Plunger communication unit 331 may receive and store mirroring data (image bitmap data) at predetermined frame refresh timing (for example, 60 fps (60 frames per second)). At this time, the received mirroring data is frame buffer data that exists in a raw data (binary form) type rather than a standardized format.

For example, assuming a resolution of 1920*1018, 1920 pixels are stored in an array of 1080 pixels as shown in FIG. 4 .

As an embodiment, the data conversion unit 332 may include a bitmap converter 333 that converts mirroring data of a raw data type stored at each frame update timing into data in a form processable by the first operating system for each frame unit. .

The bitmap converter 333 converts mirroring data, which is real-time screen data received from the Surface Plunger communication unit 331, into a data format that can be processed by the Wayland server module 311 of the first operating system 310, which is the host OS. do. At this time, the bitmap converter 333 converts the frame buffer data stored at each frame update timing one by one, that is, in units of frames.

Referring to FIG. 5, for example, if the frame buffer data input to the bitmap converter 333 is pixel data composed of 4 bytes of A (Alphablending)-numerical (transparency), Blue, Green, and Red types, the output data may be output as pixel data composed of 4 bytes of X (unused, actually filled with 00000000), R, G, and B types.

In one embodiment, the mirroring data converted by the bitmap converter 333 is stored on a shared memory 335. At this time, the types of mirroring data input and output to the shared memory 335 are the same as shown in FIG. 6 .

Meanwhile, after the second operating system is initially booted, information of the shared memory 335 is registered on the first operating system.

As an embodiment, the Wayland client unit 336 transmits the mirroring data stored in the shared memory 335 to the Wayland server module 311 of the first operating system. That is, when the Wayland client unit 336 receives an Android screen mirroring request from the first operating system 310, the mirroring data stored on the shared memory 335 in the mirroring server module 330 is transferred to the Wayland server in response to the request. Send to module 311. At this time, the types of mirroring data input and output to the Wayland client unit 336 are the same as shown in FIG. 6 .

As an embodiment, the Wayland server module 311 of the first operating system 310 is a first application (eg, In-Vehicle Infotainment ( IVI ) application) that runs on the first operating system 310. When requesting the mirroring screen of the Android container, transfer of the Android mirroring screen image (mirroring data) is requested to the Wayland client unit 331 of the mirroring server module 330, and mirroring is performed from the Wayland client 336 in response to the request. When data (the execution screen of the second application) is received, it is displayed in a specific area of the execution screen of the first application.

Meanwhile, in one embodiment of the present invention, the data converter 332 may further include an event converter 334. When the event converter 334 receives an input event through the execution screen of the second application being mirrored on the screen of the first operating system 310, the input event is transmitted through the Wayland server module 311 of the first operating system 310. It may be received, converted into data in a form that can be processed by the input flinger module 323 of the second operating system 320, and transmitted to the input flinger module 323.

At this time, the input data of the event converter 334 may be, for example, coordinate values (X, Y) of the input event, types of touch events (Up, Down, Motion, etc.), multi-touch events, etc., and the output data is, for example, For example, SLOT (input device identification), TRACKING_ID (in case of multi-touch event), touch_xoffset (X coordinate), touch_yoffset (Y coordinate), PRESSURE (pressure, a value required by the Android input plunger module 323, actually Since it is not an input value, it can be converted by filling it with '50' (random value)), SYN_REPORT, etc.

7 is a flowchart of a container screen mirroring method in the electronic device 100 according to an embodiment of the present invention.

Referring to FIG. 7 , the container screen mirroring method may include steps S110 to S130. In this case, each step/operation of the container screen mirroring method may be performed by the electronic device 100 . In this embodiment, at least one of steps S110 to S130 may be omitted, the order of some steps may be changed, or other steps may be added. In addition, even if other omitted contents, the contents of FIGS. 1 to 6 may be applied to the contents of the container screen mirroring method of FIG. 7 .

According to one embodiment of the present invention, in step S110, the first object associated with the execution of the first application is displayed on the screen of the first operating system based on Linux.

In step S120, as the Android-based second operating system is booted, an output of a second object related to the execution of the second application is requested on the screen of the first operating system.

In step S130, an execution screen of the second application is displayed on at least a part of the screen of the first operating system through the mirroring server module of the second operating system according to the request.

8 is a flowchart illustrating a process of requesting a mirroring screen according to an embodiment of the present invention.

Referring to FIG. 8 , the process of requesting a mirroring screen may include steps S205 to S255. In this case, each step/operation of the mirroring screen request process may be performed by the electronic device 100 . In this embodiment, at least one of steps S205 to S255 may be omitted, the order of some steps may be changed, or another step may be added.

First, as the system boots, information on the shared memory of the second operating system 320 is registered on the first operating system 310 (S205).

Next, after displaying the service screen of the second operating system 320 on the hidden area (S210), the screen of the second application running in the second operating system 320 through the first application of the first operating system 310 Mirroring is requested (S215).

According to the request, the Wayland server module of the first operating system 310 requests screen mirroring to the Wayland client unit of the mirroring server module 330 (S220), and according to the request, the mirroring server module 330 performs a second Mirroring data is requested to the Surface Plunger module of the operating system 320 (S225).

Next, the Surface Plunger module of the second operating system 320 transmits mirroring data to the Surface Plunger communication unit of the mirroring server module 330 (S230), and the Surface Plunger communication unit converts the data of the mirroring server module 330. Mirroring data is transmitted to the unit (S235).

Next, the bitmap converter of the data conversion unit converts the mirroring data into a form processable by the first operating system 310 and stores it on a shared memory (S240), and the Wayland client unit reads the converted mirroring data on the shared memory (S245), it is transmitted to the Wayland server module of the first operating system 310 (S250).

Accordingly, the mirroring data is displayed on a part of the screen of the first application of the first operating system 310 (S255).

9 is a flowchart illustrating a process of processing an input event according to an embodiment of the present invention.

Referring to FIG. 9 , input event processing may include steps S305 to S320. In this case, each step/operation of the input event processing process may be performed by the electronic device 100 . In this embodiment, at least one of steps S305 to S325 may be omitted, the order of some steps may be changed, or another step may be added.

First, when an input event through the execution screen of the second application of the second operating system 320 mirrored in the first operating system 310 is received (S305), the Wayland server module of the first operating system 310 receives the input event. It is delivered to the mirroring server module 330 (S310).

Next, the event converter of the mirroring server module 330 converts the corresponding input event into a form that can be processed by the input plunger module of the second operating system 320 (S315), and delivers the converted input event to the input plunger module. Do (S320).

Accordingly, the input plunger module processes an input event in the second application (S325).

The container screen mirroring method in the electronic device 100 according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-mentioned program is C, C++, JAVA, Ruby, C, C++, JAVA, Ruby, which the processor (CPU) of the computer can read through the device interface of the computer so that the computer reads the program and executes the methods implemented as a program. It may include a code coded in a computer language such as machine language. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100: electronic device
110: display
120: memory
130: processor

Claims (8)

In the electronic device for container screen mirroring,
includes a display, memory and processor;
The memory stores a first application operating on a first operating system based on Linux and a second application operating on a second operating system based on Android,
According to the execution of the memory, the processor outputs a first object associated with the execution of the first application and a second object associated with the execution of the second application on the screen of the first operating system,
The second operating system includes a mirroring server module that outputs an execution screen of the second application on at least a part of the screen of the first operating system.
Electronic device for mirroring container screen.
According to claim 1,
The mirroring server module of the second operating system is connected to the Surfaceflinger module of the second operating system and the Wayland server module of the first operating system to transmit and receive data for mirroring (hereinafter referred to as mirroring data) person,
Electronic device for mirroring container screen.
According to claim 1,
The mirroring server module,
A Surfaceflinger communication unit for requesting and receiving mirroring data generated by the Hardware Composer (HWC) module of the second operating system through the Surfaceflinger module;
A data conversion unit for converting the mirroring data into data in a form processable by the first operating system;
A shared memory for storing the converted mirroring data; and
A Wayland client unit transmitting mirroring data stored in the shared memory to a Wayland server module of the first operating system;
Electronic device for mirroring container screen.
According to claim 3,
The surface plunger communication unit receives and stores the mirroring data based on frame data of a raw data type at each predetermined frame update timing,
Electronic device for mirroring container screen.
According to claim 4,
The data conversion unit comprises a bitmap converter for converting the mirroring data of the original data type stored at each frame update timing into data in a form processable by the first operating system for each frame unit,
Electronic device for mirroring container screen.
According to claim 3,
The mirroring server module registers the information of the shared memory on the first operating system as the second operating system boots.
Electronic device for mirroring container screen.
According to claim 1,
The data conversion unit converts an input event through an execution screen of a second application being mirrored on the screen of the first operating system into data in a form that can be processed by an Inputflinger module of the second operating system, and sends the data to the Inputflinger module. Including an event converter that forwards,
Electronic device for mirroring container screen.
In the container screen mirroring method in an electronic device,
outputting a first object associated with execution of a first application on a screen of a first Linux-based operating system;
requesting an output of a second object associated with the execution of a second application on a screen of the first operating system as the Android-based second operating system is booted; and
In response to the request, outputting an execution screen of the second application on at least a part of the screen of the first operating system through a mirroring server module of the second operating system,
Container screen mirroring method in electronic devices.

Images