WO2020088118A1

WO2020088118A1 - Electronic apparatus and method employing mobile device and display device to display duplicate screens

Info

Publication number: WO2020088118A1
Application number: PCT/CN2019/105675
Authority: WO
Inventors: 盛子望
Original assignee: 盛子望
Priority date: 2018-11-02
Filing date: 2019-09-12
Publication date: 2020-05-07
Also published as: CN109308175A

Abstract

The present application provides an electronic apparatus and method employing a mobile device and a display device to display duplicate screens. The electronic apparatus comprises: at least one first interface unit connected to an input device so as to receive an operation instruction output by the input device; a second interface unit outputting the received operation instruction to the mobile device, and receiving image information output by the mobile device, wherein the image information comprises image information in which the mobile device generates an operation event according to the operation instruction and outputs the operation event; and a processing unit configured to connect the second interface unit with the display device so as to decode the received image information and output a duplicate-screen image to be displayed on the display device. The invention thus meets user requirements of operating a mobile device by means of an external input device and achieves use of a large screen for a mobile office or entertainment purposes.

Description

Electronic device and method for performing same-screen display using mobile equipment and display equipment

Technical field

The present application relates to the field of data processing, and in particular to an electronic device, a computer device, a client installed in a mobile device with a display screen, a data processing system, and a mobile device using a mobile device and a display device And display device to display on the same screen.

Background technique

With the rapid development of mobile Internet and cloud computing technologies, mobile smart electronic devices are increasingly affecting all aspects of human life. Currently, common mobile smart electronic devices include smartphones, laptops, tablets, and electronic readers. Although each of the above-mentioned products is very lightweight, it is also a heavy burden to carry them all. If you can provide a computer device in a public place that users can rent anytime and anywhere, it will greatly reduce the burden of travel. .

In addition, with the rapid development of mobile phone chip technology, its processing power will soon be comparable to or even exceed that of laptops. If you store personal data in the cloud, you can share a laptop-like display with a keyboard or tablet-like computer that can quickly connect to your mobile phone anytime, anywhere The touch screen display (not including CPU processor and related circuits, the cost is reduced by orders of magnitude), can not only improve the shortcomings of the mobile phone screen is too small, it is inconvenient to browse web pages and files, but also can solve the problem of virtual keyboard inconvenience typing Greatly facilitate people travel.

Although there are co-screen devices on the market that can meet the needs of users for large screens, these co-screen devices are intended to satisfy consumers' connected devices that display images on mobile phones through a larger display screen. The existing co-screen device only transmits the image on the mobile phone to a larger display screen, no matter whether the display screen comes with a processor or not, it can not be input through an external input device to realize the operation of the mobile phone, such as a TV The specific processing of the image is performed by the processing system in the TV. When the display screen that receives the image transmitted by the same screen does not have its own processing system, if you want to perform related operations on the image through peripherals, such as a keyboard or a touch screen, you can only use the mobile phone that sent the image. However, the same-screen device in the prior art does not have a back-control function, so it cannot meet the user's need to use an external input device to operate the operating system of the mobile phone and realize the use of a large screen to work anywhere.

Summary of the invention

In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide an electronic device, a computer device, a client installed in a mobile device with a display screen, and a data processing device using a mobile device and a display device for co-display The system and the method of using a mobile device and a display device to display on the same screen are used to solve the technical problem in the prior art that the user cannot use an external input device to operate the operating system of the mobile phone and can use the large screen to work anywhere.

To achieve the above and other related objectives, a first aspect of the present application provides an electronic device that uses a mobile device and a display device to perform on-screen display, including: at least a first interface unit for connecting an input device to receive an An operation instruction output by the input device; a second interface unit for outputting the received operation instruction to the mobile device; and receiving image information output by the mobile device; wherein the image information includes the mobile The device generates an operation event according to the operation instruction and executes output image information of the operation event; a processing unit is used to connect the second interface unit and the display device to decode the received image information Then output the same screen image to make the display device display.

A second aspect of the present application also provides a computer device that uses a mobile device for data processing, including: a display device; an input device; a processing device that connects the display device and the input device, including; a connector that connects the mobile device respectively The device and the input device are used for receiving the operation instruction output by the input device and sending it to the mobile device, and outputting the image information output by the mobile device; the image information includes the mobile device according to the The operation instruction generates an operation event and executes output image information of the operation event; a processor, connected to the connector and the display device, is used to decode the received image information and output the same screen image to make the The display device displays.

A third aspect of the present application also provides a client loaded in a mobile device with a display screen, including: an interface module for receiving an operation instruction input by an external input device via the external connection port of the mobile device; event generation A module for generating an operation event for operating the mobile device according to the operation instruction and performing a corresponding operation for the mobile device; an image processing module for capturing the input of the mobile device to the display screen Image information, and output via an external connection port of the mobile device; the image information includes image information that the mobile device generates an operation event according to the operation instruction and performs output of the operation event.

A fourth aspect of the present application also provides a data processing system, including: a computer device, including an input device and a display device, and a connector, connected to the input device, for receiving image information and output by the input device The operation instructions are output; the processor is connected to the connector and the display device for decoding the received image information to output the same screen image to display the display device; the mobile device is connected to the A connector for receiving an operation instruction of the input device and generating an operation event for operating the mobile device according to the operation instruction and performing a corresponding operation for the mobile device; and capturing input of the mobile device Image information for the display screen and transmitted to the processor via the connector; the image information includes the mobile device generating an operation event and outputting the operation event according to the operation instruction Image information.

A fifth aspect of the present application also provides a method for performing on-screen display using a mobile device and a display device. The method includes the following steps: receiving an operation instruction output by an input device; generating the mobile device according to the operation instruction Operate the operation event and perform the corresponding operation to the mobile device; capture the image information input by the mobile device to the display screen and output through the external connection port of the mobile device; the image information includes all The mobile device generates an operation event according to the operation instruction and performs output of the operation event on the image information; after decoding the image information, an on-screen image is output to enable the display device to display.

A sixth aspect of the present application also provides a method for performing on-screen display using a mobile device and a display device. The method includes the following steps: receiving an operation instruction output by an input device; and outputting the received operation instruction to the A mobile device; receiving image information output by the mobile device; wherein the image information includes image information that the mobile device generates an operation event according to the operation instruction and performs output of the operation event; the image to be received After the information is decoded, the same screen image is output so that the display device can display it.

According to a seventh aspect of the present application, a computer-readable storage medium is stored on the storage medium, and when the program is run, the same screen as described above in the fifth or sixth aspect is realized using a mobile device and a display device The method displayed.

As mentioned above, the present application can transfer images from a mobile device such as a mobile phone to an external display device that does not have processing capabilities for display, and can also control the mobile device through a wired connection through peripheral input devices such as a mouse and keyboard , So that the image processed by the mobile device is transmitted to the display device for display, thereby satisfying the user's need to use an external input device to operate the operating system of the mobile phone and realize the use of a large screen for office or entertainment.

BRIEF DESCRIPTION

FIG. 1 shows an application block diagram of an electronic device of this application in an embodiment.

FIG. 2 shows a block diagram of an application of the electronic device of this application in another embodiment.

FIG. 3 shows a block diagram of the application of the electronic device of this application in yet another embodiment.

FIG. 4 shows an application block diagram of a computer device for data processing using a mobile device of the present application.

FIG. 5 shows an application block diagram of an embodiment of the client of this application.

FIG. 6 shows an application block diagram of an embodiment of the data processing system of the present application.

FIG. 7 shows a flowchart of an embodiment of a method for displaying the same screen on the same application.

FIG. 8 shows a flowchart in another embodiment of the same-screen display method of the present application.

detailed description

The following describes the implementation of the present application by specific specific examples. Those familiar with this technology can easily understand other advantages and effects of the present application by the contents disclosed in this specification. In the following description, referring to the drawings, the drawings describe several embodiments of the present application. It should be understood that other embodiments may be used, and that changes in mechanical composition, structure, electrical, and operation may be made without departing from the spirit and scope of the present disclosure. The following detailed description should not be considered limiting, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terminology used herein is for describing particular embodiments only, and is not intended to limit the present application.

Although in some examples the terms first, second, etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, descriptions of the first interface and the second interface. Furthermore, as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "comprising" and "including" indicate the presence of the described features, steps, operations, elements, components, items, categories, and / or groups, but do not exclude one or more other features, steps, operations, The presence, appearance, or addition of elements, components, items, categories, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C" . The exception to this definition only occurs when a combination of elements, functions, steps, or operations are inherently mutually exclusive in certain ways.

The present application provides an electronic device that uses a mobile device and a display device to display on the same screen. In an actual implementation, the mobile device is, for example, a portable electronic device including but not limited to a smartphone, a handheld computer, a tablet computer, a media player, a personal digital assistant (PDA), etc. It should be understood that the present application is in an implementation The portable electronic device described in is just an application example, and the components of the device may have more or fewer components than shown, or have different component configurations. The various components described can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

In the specific implementation of the present application, the mobile device will be taken as a smartphone for example. Common examples are the iPhone series phones loaded with iOS operating system by Apple; and the Android operations loaded with Google developed by Google. Android smartphones of the system are, for example, smartphones of companies such as Samsung, Huawei, and Xiaomi.

The mobile device described as a smartphone includes a memory, a memory controller, one or more processors (CPU), a peripheral interface, an RF circuit, an audio circuit, a speaker, a microphone, an input / output (I / O) system, and a touch screen , Other output or control devices, and external ports, these components communicate through one or more communication buses or signal lines.

In an embodiment, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments, the memory may also include memory remote from one or more processors, such as network-attached memory accessed via RF circuits or external ports and a communication network (not shown), where the communication network may be the Internet , One or more intranet, local area network (LAN), wide area network (WLAN), storage area network (SAN), etc., or an appropriate combination thereof. The memory controller can control other components of the device, such as the CPU and peripheral interface, to access the memory.

In an exemplary embodiment, the display screen of the smart phone is a touch screen, which may use LCD (liquid crystal display) technology or LPD (light emitting polymer display) technology, but other display technologies may be used in other embodiments. Touch screens and touch screen controllers can use any of a variety of touch-sensitive technologies to detect contact and its movement or interruption. These touch-sensitive technologies include, but are not limited to, capacitance, resistance, infrared and surface acoustic wave technologies, and other proximity sensor arrays , Or other techniques for determining one or more points in contact with the touch screen. The touch screen displays visual output from portable devices, while the touch-sensitive panel does not provide visual output. The touch screen may have a resolution higher than 100dpi. In one exemplary embodiment, the touch screen may have a resolution of about 168 dpi. The user can use any suitable object or accessory, such as a stylus, finger, etc., to touch the touch screen. Specifically, the touch screen receives user input based on tactile and / or tactile contact. The touch screen forms a touch-sensitive surface that accepts user input. The touch screen and touch screen controller (together with any associated modules and / or instruction sets in the memory) detect the contact on the touch screen (and any movement or interruption of the touch) and transform the detected contact into a display Interaction of multimedia files (such as pictures or videos) or multimedia sample files (such as screenshots of pictures or videos) on the touch screen.

In an embodiment, the peripheral interface of the mobile device that is a smartphone includes an FFC interface, FPC interface, AT interface, COM interface, MIDI interface, PS / 2 interface, USB interface, or Bluetooth interface; for example, the smartphone The peripheral interface is a USB interface. For example, the interfaces include the following standards: USB A-Type, USB B-Type, USB C-Type, Micro-USB A, Micro-USB B, USB Mini-b (5-pin), USB Mini-b (4-pin), USB 3.0 A-Type, USB 3.0 B-Type, USB 3.0 Micro B, etc.

The peripheral interface is communicatively connected to an input / output (I / O) system of the smartphone, the I / O system includes a touch screen controller and one or more input controllers for other output or control devices. The one or more input controllers receive / transmit electrical signals from / to other input or control devices. The I / O system provides an interface that provides an input interface for an input / output peripheral of the device and a peripheral interface, such as a touch screen and other input / control devices. Specifically, the touch screen may provide input interfaces including but not limited to text, images, icons, soft keys (or "virtual buttons"), drop-down menus, radio buttons, check boxes, selectable lists, and so on. Correspondingly, the displayed user interface objects may include: non-interactive objects for transmitting information or constituting the appearance of the user interface, interactive objects available for user interaction, or a combination thereof.

The mobile device described as a smartphone supports various applications, such as one or more of the following: drawing applications, rendering applications, word processing applications, website creation applications, disk editing applications, electronic Form application, game application, phone application, video conferencing application, email application, instant messaging application, fitness support application, photo management application, digital camera application, digital video camera application, web browsing Applications, digital music player applications and / or digital video player applications.

In an actual embodiment, the display device is, for example, a liquid crystal display screen, or is also called a liquid crystal display, and a common one is an LCD display screen, for example. In this application, the display device is an external expansion display for expanding the touch screen of the mobile device, and is used to display the display content of the mobile device on the external display device in the same screen manner.

The electronic device for displaying on the same screen using a mobile device and a display device includes at least a first interface unit, a second interface unit, and a processing unit.

Please refer to FIG. 1, which shows an application block diagram of an embodiment of the electronic device of the present application. As shown in the figure, the at least one first interface unit 101 is used to connect an input device 11 to receive the output of the input device 11 In the embodiment, when the first interface unit 101 receives the operation instruction output by the input device 11, it performs fusion processing on the operation instruction to convert it into a set of event sequences.

In an embodiment, the first interface unit 101 corresponds to different input devices 11 with different interface types, such as FFC interface, FPC interface, AT interface, COM interface, MIDI interface, PS / 2 interface, USB interface or Bluetooth interface. Kind of. The USB interface includes, for example, physical interfaces of the following standards or specifications: USB A-Type, USB B-Type, USB C-Type, Micro-USB A, Micro-USB B, USB Mini-b (5-pin ), USB Mini-b (4-pin), USB 3.0 A-Type, USB 3.0 B-Type, USB 3.0 Micro B, etc. In an embodiment, the input device 11 includes, for example, one or more of a keyboard, a mouse, a touchpad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board. Of course, based on different application scenarios, the input device 11 may also be a microphone, electronic key, stylus, scanner, or the like.

In an exemplary embodiment, the input device 11 is, for example, a keyboard, and the corresponding output operation instructions are, for example, a variety of operation instructions such as character input, character deletion, line feed, space, and full selection performed by the user by tapping the keyboard. For example, when the input device 11 is a keyboard, the corresponding interface specification of the first interface unit 101 is an FFC interface, an FPC interface, an AT interface, a COM interface, a USB interface, or a PS / 2 interface.

In another exemplary embodiment, the input device 11 is, for example, a mouse, and the corresponding output operation instructions are, for example, the user's mouse left and right clicks, drag and drop to move or select the mouse pointer, etc. Operation instructions. For example, when the input device 11 is a mouse, the corresponding interface specification of the first interface unit 101 is a COM interface, a USB interface, a PS / 2 interface, or a Bluetooth interface. In some implementation states, the mouse may also be a touchpad (TouchPad or TrackPad), which controls the movement of the pointer by sensing the movement of the user's finger.

In yet another exemplary embodiment, the input device 11 such as a handwriting input board (also referred to as a handwriting board or writing board, etc.) correspondingly outputs an operation instruction such as an operation instruction such as a writing track. For example, when the input device 11 is a handwriting input board, the interface specification of the corresponding first interface unit 101 is a USB interface or a Bluetooth interface.

In yet another exemplary embodiment, for example, a gamepad, the corresponding output operation instruction is, for example, a joystick instruction or a key instruction. For example, when the input device 11 is a game controller, the corresponding interface specification of the first interface unit 101 is a MIDI interface, a USB interface, or a Bluetooth interface.

In an application example, there are two first interface units 101, one of the first interface units 101 is connected to the input device 11 such as a keyboard, and the other first interface unit 101 is connected to the input device 11 such as a mouse. Those skilled in the art can understand that in different implementation scenarios, according to actual needs, the corresponding first interface unit 101 can be added according to the number of input devices 11, and of course, different physical configurations can be configured according to the interface requirements of the input device 11. The interface or the first interface unit 101 of different specifications is hereby described.

The second interface unit 102 is electrically connected to the first interface unit 101, and the second interface unit 102 is configured to output the received operation instruction to the mobile device 12; and receive the output of the mobile device 12 Image information; wherein, the image information includes image information that the mobile device 12 generates an operation event according to the operation instruction and performs output of the operation event. In an embodiment, the second interface unit 102 corresponds to different input devices 11 with different interface types, such as FFC interface, FPC interface, AT interface, COM interface, MIDI interface, PS / 2 interface, USB interface or Bluetooth interface. Kind of. The USB interface includes, for example, physical interfaces of the following standards or specifications: USB A-Type, USB B-Type, USB C-Type, Micro-USB A, Micro-USB B, USB Mini-b (5-pin ), USB Mini-b (4-pin), USB 3.0 A-Type, USB 3.0 B-Type, USB 3.0 Micro B, etc.

In an embodiment, the mobile device 12 is, for example, a smart phone, a handheld computer, a tablet computer, a media player, a personal digital assistant (PDA), and other portable electronic devices. In the following embodiments of the present application, the mobile device 12 is, for example, a smartphone as an example, for example, an Android smartphone loaded with an Android operating system, such as smartphones of companies such as Samsung, Huawei, and Xiaomi; However, it is not limited to this. In order to avoid redundant description, those skilled in the art should know that, for example, iPhone series Apple mobile phones loaded with iOS operating system are also suitable for this application.

The second interface unit 102 outputs the operation instruction from the first interface unit 101 to the mobile device 12 that is an Android smartphone; in an embodiment, the mobile device 12 that is a smartphone is connected to the first When the second interface unit 102, the electronic device 10 runs an event implantation service program on the Android smartphone through the ADB (Android Debug Bridge) channel, for example, the event implantation service program is a sagevt program, which is used to Operating system implantation event.

In this embodiment, when the mobile device 12 that is a smartphone is connected to the second interface unit 102, the electronic device 10 runs a screen recording application on the Android smartphone through the ADB (Android Debug Bridge) channel Program, the screen recording application program is used to call the native interface of the Android system to record the display content of the touch screen of the Android smartphone, and decode the image information of the screen recording, that is, compress the image information of the screen recording into a video Stream, for example compressed into H264 video stream.

In an embodiment, when the first interface unit 101 receives the operation instruction output by the input device 11, it performs fusion processing on the operation instruction to convert it into a set of event sequences and sends it to the second interface unit 102. The second interface unit 102 transmits a set of event sequences corresponding to the operation instruction to the smartphone through the ADB channel of the Android smartphone, and the event implantation service program running in the smartphone generates the corresponding group of events The sequence of key maps is converted into touch coordinates for the smartphone, and an operation event corresponding to the operation instruction is generated. The smartphone performs the operation after receiving the operation time and reflects the operation through its touch screen output Image information of time, that is, the image information includes image information that the mobile device 12 generates an operation event according to the operation instruction and performs output of the operation event.

In the embodiment provided by the present application, the second interface unit 102 is further configured to receive image information output by the mobile device 12; the image information includes the standby of the mobile device 12 when an external input event is not performed Image information, execution image information (such as playing a video, previewing a news page, route navigation, etc.) that is running a program, and image information that generates an operation event according to the operation instruction and performs output of the operation event, etc. In an embodiment, when the mobile device 12 is connected to the second interface unit 102 through the ADB channel, the electronic device 10 obtains the image information output by the mobile device 12 through the second interface unit 102.

Please refer to FIG. 2, which shows a block diagram of an application of the electronic device 10 of the present application in another embodiment. As shown in the figure, in one embodiment, the first interface unit 101 and the second interface unit 102 are configured In a HUB module 100, that is, the HUB module 100 integrates at least one first interface unit 101 and a second interface unit 102, for example, the HUB module 100 integrates multiple first interfaces for connecting different input devices 11 An interface unit 101, and a plurality of second interface units 102 for connecting smartphones with different peripheral interfaces. For example, the HUB module 100 integrates an interface unit for connecting to a keyboard and an interface unit for connecting to a mouse, a microUSB interface and a type-c interface for connecting to an Android smartphone, and an integrated unit for connecting to an iPhone series mobile phone. Lightning interface, etc., the HUB module 100 also integrates a Bluetooth chip, for example, RTL8761AU-CG chip. The HUB module 100 includes a USB-HUB chip, for example, a GL850G chip.

The second interface unit 102 is also used to receive the audio information output by the mobile device 12; in an embodiment, the second interface unit 102 receives the audio information sent by the mobile device 12 through an integrated Bluetooth chip, in In a specific embodiment, the second interface unit 102 receives audio information sent by the smart phone via Bluetooth via the HUB module 100 including a USB-HUB chip and a Bluetooth chip.

The processing unit 103 is used to connect the second interface unit 102 and the display device 13 to perform decoding processing on the received image information and output an on-screen image for the display device 13 to display. In an embodiment, the processing unit 103 is connected to the second interface unit 102 through a USB communication interface, and the processing unit 103 is connected to the display device 13 through an HDMI interface communication interface.

In the embodiment provided by the present application, the processing unit 103 is a multimedia data processing chip. In a specific implementation manner, the multimedia data processing chip is, for example, an AM8251 chip, an AM8271 chip, or an AM8272 chip and other high-definition digital media processing Device. The multimedia data formats include, for example, AVI, MPEG / 1/2/4, RM, RMVB, WMV, VCD / SVCD, DAT, VOB, MOV, MP4, MKV, ASF, FLV, etc.

In the embodiment provided by the present application, the processing unit 103 is connected to the display device 13 through a third interface, the third interface includes a DP interface, an eDP interface, an MHL interface, a DiiVA interface, an HDBaseT interface, a Thunderbolt interface, One of USB interface, LVDS interface, HDMI interface, VGA interface, or DVI interface. In an actual embodiment, the display device 13 is, for example, a liquid crystal display screen, or is also called a liquid crystal display, and a common display screen is, for example, an LCD display screen. In this application, the processing unit 103 is connected to the display device 13 through an HDMI interface communication interface, and the display device 13 is an external expansion display for expanding the touch screen of the mobile device 12 described above, and is used to convert the The display content is displayed on the external display device 13 in the same screen.

In the embodiment where the mobile device 12 is an Android smartphone, the processing unit 103 is connected to the Android smartphone through the second interface unit 102 and establishes communication with the Android smartphone through an ADB channel. As described above, when the mobile device 12 that is a smartphone is connected to the second interface unit 102, the electronic device 10 runs a screen recording application on the Android smartphone through the ADB channel, and the screen recording application Used to call the native interface of the Android system to record the display content of the touch screen of the Android smartphone, and decode the recorded image information, that is, compress the recorded image information into a video stream, such as H264 video stream . After receiving the video stream, the processing unit 103 performs decoding processing into an on-screen image and transmits it to the display device 13 for display.

In the embodiment of the present application, the second interface unit 102 is further configured to receive audio information output by the mobile device 12; in an embodiment, the second interface unit 102 receives the mobile through an integrated Bluetooth chip The audio information sent by the device 12, in a specific embodiment, the second interface unit 102 receives the audio information sent by the smartphone through its Bluetooth through the HUB module 100 including a USB-HUB chip and a Bluetooth chip. The processing unit 103 is also used to process the audio information received by the second interface unit 102 via the Bluetooth module, and output audio playback signals after processing the audio information.

Please refer to FIG. 3, which shows a block diagram of the application of the electronic device 10 of this application in another embodiment. As shown in the figure, in this embodiment, the electronic device 10 further includes a cache unit 104 and a storage unit 105. The cache unit 104 is connected to the processing unit 103 for caching image information received by the processing unit 103; in a specific embodiment, the cache unit 104 is, for example, a DDR3 unit, and the storage unit 105 is connected to the processing unit 103, used to store the image processing program of the processing unit 103. The storage unit 105 is, for example, a NADN unit.

In the embodiment of the present application, the processing unit 103 of the electronic device 10 further includes a MICRO-USB interface unit, and the MICRO-USB interface unit is used for an external power supply and a firing program.

The electronic device of the present application can transmit images in a transfer device such as a mobile phone to an external display device without processing capability for display, and can also control the mobile device through peripheral input devices such as a mouse and a keyboard, so that the mobile device can process After the image is transmitted to the display device for display, the user's operating system using an external input device to operate the mobile phone can also meet the needs of using a large screen for office or entertainment. In addition, the input device is connected to the mobile device and display through a wired connection. The device communicates with the advantages of low latency and high reliability compared to wireless connection.

The present application also provides a computer device that uses a mobile device for data processing. In an actual implementation, the mobile device is, for example, a portable electronic device including but not limited to a smart phone, a handheld computer, a tablet computer, a media player, a personal digital assistant (PDA), etc. It should be understood that this application is based on The portable electronic device described in is just an application example, and the components of the device may have more or fewer components than shown, or have different component configurations. The various components described can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

Please refer to FIG. 4, which shows a block diagram of a computer device that uses a mobile device for data processing. As shown in the figure, the computer device 21 includes a display device 211, an input device 212, and a processing device 213.

In an actual embodiment, the display device 211 is, for example, a liquid crystal display, or is also called a liquid crystal display, and a common one is an LCD display, for example. In this application, the display device 211 is an external expansion display for expanding the touch screen of the mobile device 20, and is used to display the display content of the mobile device 20 on the external display device 211 in the same screen. The input device 212 is, for example, one or more of a keyboard, a mouse, a touchpad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board. Of course, based on different application scenarios, the input device 11 may also be a microphone, electronic key, stylus, scanner, or the like.

In an actual embodiment, the input device 212 is, for example, a keyboard, and the corresponding output operation instructions are, for example, various operation instructions such as character input, character deletion, line feed, and space performed by the user by tapping the keyboard. For example, when the input device 212 is a keyboard, the interface specifications of the corresponding first interface unit are FFC interface, FPC interface, AT interface, COM interface, USB interface, or PS / 2 interface.

In an actual embodiment, the input device 212 may also be, for example, a mouse, and the corresponding output operation instructions are, for example, the user ’s left and right mouse clicks and drags to move or select the mouse pointer by operating the mouse. Operation instructions. For example, when the input device 212 is a mouse, the interface specification of the corresponding first interface unit is a COM interface, a USB interface, a PS / 2 interface, or a Bluetooth interface. In some implementation states, the mouse may also be a touchpad (TouchPad or TrackPad), which controls the movement of the pointer by sensing the movement of the user's finger.

The display device 211 and the input device 212 of the computer device 21 are, for example, the physical form of a notebook computer, that is, the host portion provided for the input device 212 of the keyboard, and the display screen portion of the display device 211 is hinged to the host portion so that The display screen can be rotated from 0 ° to 360 ° through the hinge relative to the host to achieve operations such as closing, unfolding, and folding back.

The processing device 213 is connected to the display device 211 and the input device 212, and includes a connector 2131 and a processor 2132.

The connector 2131 is respectively connected to the mobile device 20 and the input device 212, and is used to receive the operation instruction output by the input device 212 and send it to the mobile device 20, and to receive the image information output by the mobile device 20 Output; the image information includes image information that the mobile device 20 generates an operation event according to the operation instruction and performs output of the operation event;

In an embodiment, the connector 2131 is a HUB module configured with at least one first interface for connecting the input device 212 and a second interface for connecting the mobile device 20, that is, all The first interface and the second interface are configured in a HUB module, that is, the HUB module integrates at least one first interface and a second interface, for example, the HUB module integrates multiple for connecting different inputs The first interface of the device 212 and the second interfaces of multiple smart phones for connecting different peripheral interfaces. For example, the HUB module integrates an interface unit for connecting a keyboard and an interface unit for connecting a mouse, a microUSB interface and a type-c interface for connecting an Android smartphone, and Lightning for connecting an iPhone series mobile phone Interfaces, etc. The HUB module also integrates a Bluetooth chip, such as the RTL8761AU-CG chip. The HUB module includes a USB-HUB chip, for example, a GL850G chip.

The at least one first interface is used to connect an input device 212 to receive the operation command output by the input device 212; in an embodiment, when the first interface receives the operation command output by the input device 212, the The operation instructions are fused to be converted into a set of event sequences.

In an embodiment, the first interface corresponds to different input devices 212 with different interface types, such as one of the FFC interface, FPC interface, AT interface, COM interface, MIDI interface, PS / 2 interface, USB interface or Bluetooth interface Species. The USB interface includes, for example, physical interfaces of the following standards or specifications: USB A-Type, USB B-Type, USB C-Type, Micro-USB A, Micro-USB B, USB Mini-b (5-pin ), USB Mini-b (4-pin), USB 3.0 A-Type, USB 3.0 B-Type, USB 3.0 Micro B, etc. In an embodiment, the input device 212 includes, for example, one or more of a keyboard, a mouse, a touchpad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board. Of course, based on different application scenarios, the input device 11 may also be a microphone, electronic key, stylus, scanner, or the like.

The second interface outputs the operation instruction from the first interface to the mobile device 20 that is an Android smartphone; in an embodiment, when the mobile device 20 that is a smartphone is connected to the second interface, The electronic device runs an event implantation service program on the Android smartphone through the ADB (Android Debug Bridge) channel, for example, the event implantation service program is a sagevt program, which is used to implant events into the smartphone's operating system .

In this embodiment, when the mobile device 20 that is a smartphone is connected to the second interface, the electronic device runs a screen recording application on the Android smartphone through the ADB (Android Debug Bridge) channel. The screen recording application program is used to call the native interface of the Android system to record the display content of the touch screen of the Android smartphone, and decode the image information of the screen recording, that is, compress the image information of the screen recording into a video stream, such as Compressed into H264 video stream.

In an embodiment, when the first interface receives the operation instruction output by the input device 212, the operation instruction is fused to be converted into a set of event sequences and sent to the second interface, the second interface The set of event sequences corresponding to the operation instruction is transmitted to the smart phone through the ADB channel of the Android smart phone, and the event implantation service program running in the smart phone generates a key mapping corresponding to the set of event sequences to convert to the corresponding position. The touch coordinates of the smartphone, and then generate an operation event corresponding to the operation instruction, the smartphone executes the operation after receiving the operation time, and outputs image information reflecting the operation time through its touch screen, that is, the image The information includes image information that the mobile device 20 generates an operation event according to the operation instruction and performs output of the operation event.

In the embodiment provided by the present application, the second interface is further used to receive image information output by the mobile device 20; the image information includes standby image information of the mobile device 20 when an external input event is not performed 1. Execution image information (such as playing a video, previewing a news page, path navigation, etc.) that is running a program, and image information that generates an operation event according to the operation instruction and performs output of the operation event, etc. In an embodiment, when the mobile device 20 is connected to the second interface through the ADB channel, the electronic device obtains the image information output by the mobile device 20 through the second interface.

The second interface of the connector 2131 is also used to receive audio information output by the mobile device 20; in an embodiment, the second interface receives audio information sent by the mobile device 20 through an integrated Bluetooth chip, In a specific embodiment, the second interface receives audio information sent by the smartphone through its Bluetooth through a HUB module that includes a USB-HUB chip and a Bluetooth chip.

The processor 2132 is connected to the connector 2131 and the display device 211, and is used for decoding the received image information and outputting the same screen image to display the display device 211. In an embodiment, the processor 2132 is connected to the second interface through a USB communication interface, and the processor 2132 is connected to the display device 211 through an HDMI interface communication interface.

In the embodiment provided by the present application, the processor 2132 is a multimedia data processing chip. In a specific implementation manner, the multimedia data processing chip is, for example, an AM8251 chip, an AM8271 chip, or an AM8272 chip and other high-definition digital media processing chips.器 2132. The multimedia data formats include, for example, AVI, MPEG / 1/2/4, RM, RMVB, WMV, VCD / SVCD, DAT, VOB, MOV, MP4, MKV, ASF, FLV, etc.

In the embodiment provided by the present application, the processor 2132 is connected to the display device 211 through a third interface, the third interface includes a DP interface, an eDP interface, an MHL interface, a DiiVA interface, an HDBaseT interface, a Thunderbolt interface, One of USB interface, LVDS interface, HDMI interface, VGA interface, or DVI interface. In an actual embodiment, the display device 211 is, for example, a liquid crystal display, or is also called a liquid crystal display, and a common one is an LCD display, for example. In this application, the processor 2132 is connected to the display device 211 through an HDMI interface communication interface. The display device 211 is an external expansion display used to expand the touch screen of the mobile device 20, and is used to connect the mobile device 20 to the display device 211. The display content is displayed on the external display device 211 on the same screen.

In the embodiment where the mobile device 20 is an Android smartphone, the processor 2132 connects to the Android smartphone through the second interface and establishes communication with the Android smartphone through an ADB channel, as described above , When the mobile device 20 that is a smartphone is connected to the second interface, the electronic device runs a screen recording application on the Android smartphone through the ADB channel, and the screen recording application is used to call the Android system The native interface of the device records the display content of the touch screen of the Android smartphone, and decodes the image information of the recorded screen, that is, the image information of the recorded screen is compressed into a video stream, such as an H264 video stream. After receiving the video stream, the processor 2132 decodes the same screen image and transmits it to the display device 211 for display.

In the embodiment of the present application, the second interface is also used to receive the audio information output by the mobile device 20; in the embodiment, the second interface receives the mobile device 20 sent by the integrated Bluetooth chip Audio information. In a specific embodiment, the second interface receives audio information sent by the smartphone through its Bluetooth through a HUB module that includes a USB-HUB chip and a Bluetooth chip, and the processor 2132 is also used to process To the second interface, the audio information received by the Bluetooth module, and after processing the audio information, output an audio playback signal. Correspondingly, the computer device 21 further includes an audio playback device for playing the received audio playback signal. In an embodiment, the audio playback device is, for example, an audio player.

The computer device 21 further includes a buffer for buffering the image information and a memory for storing image processing programs. In this embodiment, the electronic device further includes a cache unit and a storage unit, the buffer is connected to the processor 2132, and is used to cache the image information received by the processor 2132; in a specific embodiment, all The buffer is, for example, a DDR3 unit, and the memory is connected to the processor 2132 for storing the image processing program of the processor 2132. The storage unit is, for example, a NADN unit.

In the embodiment of the present application, the processor 2132 of the computer device 21 further includes a MICRO-USB interface unit, and the MICRO-USB interface unit is used for external power supply and firing program.

The computer device of the present application can transfer the image in the transfer device such as a mobile phone to an external display device that does not have processing capabilities for display, and can also control the mobile device through peripheral input devices such as a mouse and a keyboard, so that it can be processed by the mobile device The image after transmission is transmitted to the display device for display, thereby satisfying the user's need to use an external input device to operate the operating system of the mobile phone and to realize the use of a large screen for office or entertainment; in addition, the input device is connected to the mobile device and display through a wired connection The device communicates with the advantages of low latency and high reliability compared to wireless connection.

This application provides a client loaded in a mobile device with a display screen. In an actual embodiment, the mobile device includes, for example, but not limited to, a smartphone, a handheld computer, a tablet computer, a media player, and a personal digital Assistant (PDA) and other portable electronic devices, it should be understood that the portable electronic device described in the embodiments of the present application is only an application example, and the components of the device may have more or less components than the illustration, or have different Component configuration. The various components described can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

Please refer to FIG. 5, which is a block diagram of an embodiment of the client of the present application. As shown in the figure, the client 320 loaded in the mobile device 32 with a display screen includes: an interface module 3201 and an event generation module 3202 , And the image processing module 3203.

The interface module 3201 is used to receive an operation instruction input by an external input device 31 via the external interface of the mobile device 32; in an embodiment, when the interface module 3201 receives the operation instruction output by the input device 31, the The operation instructions are fused to be converted into a set of event sequences. The interface module 3201 is, for example, an ADB (Android Debug Bridge) channel (Channel) interface program. The input device 31 is, for example, one or more of a keyboard, a mouse, a touchpad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board. Of course, based on different application scenarios, the input device 11 may also be a microphone, electronic key, stylus, scanner, or the like.

In an exemplary embodiment, the input device 31 is, for example, a keyboard, and the corresponding output operation instructions are, for example, a variety of operation instructions such as character input, character deletion, line feed, and space performed by the user by tapping the keyboard.

In another exemplary embodiment, the input device 31 is, for example, a mouse, and the corresponding output operation instructions are, for example, a user's mouse left and right clicks, drag and drop to move or select the mouse pointer, etc. Operation instructions. In some implementation states, the mouse may also be a touchpad (TouchPad or TrackPad), which controls the movement of the pointer by sensing the movement of the user's finger.

In yet another exemplary embodiment, the input device 31 such as a handwriting input board (also referred to as a handwriting board or writing board, etc.) corresponds to an output operation instruction such as an operation instruction such as a writing track.

In yet another exemplary embodiment, for example, a gamepad, the corresponding output operation instruction is, for example, a joystick instruction or a key instruction.

In an embodiment, when the interface module 3201 receives an operation command output by the input device 31 through a connector 33, the connector 33 is, for example, a HUB module, and the HUB module is configured with at least one A first interface connected to the input device 31 and a second interface used to connect to the mobile device 32, that is, the HUB module integrates at least one first interface and a second interface, for example, the HUB module is integrated There are multiple first interfaces for connecting different input devices 31, and multiple second interfaces for connecting smartphones of different peripheral interfaces. For example, the HUB module integrates an interface unit for connecting a keyboard and an interface unit for connecting a mouse, a microUSB interface and a type-c interface for connecting an Android smartphone, and Lightning for connecting an iPhone series mobile phone Interfaces, etc. The HUB module also integrates a Bluetooth chip, such as the RTL8761AU-CG chip. The HUB module includes a USB-HUB chip, for example, a GL850G chip.

In the following embodiments of the present application, the mobile device 32 is, for example, a smartphone, for example, for example, an Android smartphone loaded with an Android operating system, for example, smartphones of companies such as Samsung, Huawei, and Xiaomi; However, it is not limited to this. In order to avoid redundant description, those skilled in the art should know that, for example, iPhone series Apple mobile phones loaded with iOS operating system are also suitable for this application.

The event generation module 3202 is used to generate an operation event for operating the mobile device 32 according to the operation instruction and perform a corresponding operation for the mobile device 32; the event generation module 3202 is, for example, an application program, specifically In fact, an event implantation service program is run on the Android smartphone through the ADB (Android Debug Bridge) channel. For example, the event implantation service program is a sagevt program, which is used to implant events into the smartphone's operating system.

The ADB channel (ADB Channel) of the Android smartphone transmits the set of event sequences corresponding to the operation instruction to the smartphone, and the event generation module 3202 running in the smartphone for the event implant service program generates the corresponding group The key sequence of the event sequence is converted into touch coordinates to the smartphone, and then an operation event corresponding to the operation instruction is generated. The smartphone executes the operation after receiving the operation time, and reflects this through its touch screen output. Image information of the operation time, that is, the image information includes image information that the mobile device 32 generates an operation event according to the operation instruction and performs output of the operation event.

The image processing module 3203 is used to capture the image information input by the mobile device 32 to the display screen and output through the external connection port of the mobile device 32; the image information includes the mobile device 32 according to The operation instruction generates an operation event and executes output image information of the operation event. In this embodiment, the image processing module 3203 is, for example, a screen recording application. By running the screen recording application on the Android smartphone, the screen recording application is used to call the native interface of the Android system Recording the display content of the touch screen of the Android smart phone, and decoding the image information of the recorded screen, that is, compressing the image information of the recorded screen into a video stream, such as an H264 video stream. After receiving the video stream, the processing unit decodes the same screen image and transmits it to the display device 30 for display.

In the embodiment provided by the present application, the image information further includes the standby image information of the mobile device 32 when the external input event is not executed, and the execution image information of a program that is running (such as playing a video, previewing a news page) , Path navigation and other programs), and image information that generates an operation event according to the operation instruction and performs output of the operation event.

In an embodiment, the client is an application program (Application, APP for short) loaded in the mobile device, and the application program runs based on a system platform of the mobile device, for example, Apple The iPhone OS (iOS) operating system platform developed by the company, or the Android mobile phone operating system platform developed by Google, and also the Windows Mobile (WP) mobile operating platform developed by Microsoft Corporation, etc. In this application, as described above, the client uses its built-in module to implement event implantation services, screen recording, video stream generation, and compression.

The client also remotely connects to a cloud database, such as a remote server or service platform, when the client detects that other applications in the mobile device are open, such as WeChat and other applications, it is necessary to determine whether It is a known application of the client. If it is a known application, it will obtain mapping data from the cloud database, and generate a virtual keyboard by using the built-in event generation module 3202, and forward the mapping table to sagevt. Touch algorithm processing. If the client detects that other applications in the mobile device are non-known applications, it may prompt the user to open the mapping function of the client to perform touch point mapping.

The client of the present application can transfer images in a mobile device such as a mobile phone to an external display device without processing capability for display, and can also control the mobile device through peripheral input devices such as a mouse and a keyboard, so that the mobile device processes The image after transmission is transmitted to the display device for display, thereby satisfying the user's need to use an external input device to operate the operating system of the mobile phone and to realize the use of a large screen for office or entertainment.

Please refer to FIG. 6, which is a schematic diagram of an embodiment of the data processing system of the present application. As shown in the figure, the data processing system includes: a computer device 40 and a mobile device 50, wherein the computer device 40 includes an input device 401, a display device 402, a processing device 400 including a connector 4001 and a processor 4002.

In an actual embodiment, the display device 402 is, for example, a liquid crystal display screen, or is also called a liquid crystal display, and a common one is, for example, an LCD display screen. In this application, the display device 402 is an external expansion display for expanding the touch screen of the mobile device 50, and is used to display the display content of the mobile device 50 on the external display device 402 in the same screen. The input device 401 is, for example, one or more of a keyboard, a mouse, a touchpad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board. Of course, based on different application scenarios, the input device 11 may also be a microphone, electronic key, stylus, scanner, or the like.

In an actual embodiment, the input device 401 is, for example, a keyboard, and the corresponding output operation instructions are, for example, character input, character deletion, line feed, space, and other operation instructions performed by the user by tapping the keyboard.

In an actual implementation, the input device 401 may also be a mouse, for example, and the corresponding output operation instructions are, for example, a user's mouse left and right clicks, drag and drop to move or select the mouse pointer, etc. Operation instructions. In some implementation states, the mouse may also be a touchpad (TouchPad or TrackPad), which controls the movement of the pointer by sensing the movement of the user's finger.

The display device 402 and the input device of the computer equipment are, for example, the physical form of a notebook computer, that is, the host part provided for the input device of the keyboard, and the display part of the display device 402 is hinged to the host part so that the display can be Relative to the host, it can be rotated from 0 ° to 360 ° through a hinge to achieve operations such as closing, unfolding, and folding back.

In an embodiment, the connector 4001 is a HUB module configured with at least one first interface for connecting the input device 401 and a second interface for connecting the mobile device 50, that is, all The first interface and the second interface are configured in a HUB module, that is, the HUB module integrates at least one first interface and a second interface, for example, the HUB module integrates multiple for connecting different inputs The first interface of the device 401 and the second interfaces of multiple smart phones for connecting different peripheral interfaces. For example, the HUB module integrates an interface unit for connecting a keyboard and an interface unit for connecting a mouse, a microUSB interface and a type-c interface for connecting an Android smartphone, and Lightning for connecting an iPhone series mobile phone Interfaces, etc. The HUB module also integrates a Bluetooth chip, such as the RTL8761AU-CG chip. The HUB module includes a USB-HUB chip, for example, a GL850G chip.

The at least one first interface is used to connect an input device 401 to receive the operation instruction output by the input device 401; in an embodiment, when the first interface receives the operation instruction output by the input device 401, the The operation instructions are fused to be converted into a set of event sequences.

The second interface outputs the operation instruction from the first interface to the mobile device 50 that is an Android smartphone; in an embodiment, when the mobile device 50 that is a smartphone is connected to the second interface, The electronic device runs an event implantation service program on the Android smartphone through an ADB (Android Debug Bridge) channel. For example, the event implantation service program is a sagevt program, which is used to implant events into the smartphone's operating system.

The second interface of the connector 4001 is also used to receive audio information output by the mobile device 50; in an embodiment, the second interface receives audio information sent by the mobile device 50 through an integrated Bluetooth chip, at In a specific embodiment, the second interface receives audio information sent by the smartphone through its Bluetooth through a HUB module that includes a USB-HUB chip and a Bluetooth chip.

The processor 4002 is connected to the connector 4001 and the display device 402, and is used for decoding the received image information and outputting the same screen image for the display device 402 to display. In an embodiment, the processor 4002 is connected to the second interface through a USB communication interface, and the processor 4002 is connected to the display device 402 through an HDMI interface communication interface.

In the embodiment provided in this application, the processor 4002 is a multimedia data processing chip. In a specific implementation manner, the multimedia data processing chip is, for example, an AM8251 chip, an AM8271 chip, or an AM8272 chip and other high-definition digital media processing器 4002. The multimedia data formats include, for example, AVI, MPEG / 1/2/4, RM, RMVB, WMV, VCD / SVCD, DAT, VOB, MOV, MP4, MKV, ASF, FLV, etc.

In the embodiment provided by the present application, the processor 4002 is connected to the display device 402 through a third interface, the third interface includes a DP interface, an eDP interface, an MHL interface, a DiiVA interface, an HDBaseT interface, a Thunderbolt interface, One of USB interface, LVDS interface, HDMI interface, VGA interface, or DVI interface. In an actual embodiment, the display device 402 is, for example, a liquid crystal display screen, or is also called a liquid crystal display, and a common one is, for example, an LCD display screen. In this application, the processor 4002 is connected to the display device 402 through an HDMI interface communication interface. The display device 402 is an external expansion display used to expand the touch screen of the mobile device 50, and is used to connect the mobile device 50 to the display device 402. The display content is displayed on the external display device 402 on the same screen.

In an embodiment where the mobile device 50 is an Android smartphone, the processor 4002 connects to the Android smartphone through the second interface and establishes communication with the Android smartphone through an ADB channel, as described above , When the mobile device 50 that is a smartphone is connected to the second interface, the electronic device runs a screen recording application on the Android smartphone through the ADB channel, and the screen recording application is used to call the Android system The native interface of the device records the display content of the touch screen of the Android smartphone, and decodes the image information of the recorded screen, that is, the image information of the recorded screen is compressed into a video stream, such as an H264 video stream. After receiving the video stream, the processor 4002 decodes the same screen image and transmits it to the display device 402 for display.

In the embodiment of the present application, the second interface is also used to receive audio information output by the mobile device 50; in an embodiment, the second interface receives the mobile device 50 sent by the integrated Bluetooth chip Audio information. In a specific embodiment, the second interface receives audio information sent by the smartphone through its Bluetooth through a HUB module including a USB-HUB chip and a Bluetooth chip, and the processor 4002 is also used to process To the second interface, the audio information received by the Bluetooth module, and after processing the audio information, output an audio playback signal. Correspondingly, the computer device further includes an audio playback device for playing the received audio playback signal. In an embodiment, the audio playback device is, for example, an audio player.

The computer device also includes a buffer for buffering the image information and a memory for storing image processing programs. In this embodiment, the electronic device further includes a buffer and a memory, the buffer is connected to the processor 4002, and is used to buffer the image information received by the processor 4002; in a specific embodiment, the The buffer is, for example, a DDR3 unit, and the memory is connected to the processor 4002 and used to store the image processing program of the processor 4002. The memory is, for example, a NADN unit.

In the embodiment of the present application, the processor 4002 of the computer device further includes a MICRO-USB interface unit, and the MICRO-USB interface unit is used for external power supply and firing program.

In an actual embodiment, the mobile device 50 is, for example, a portable electronic device including, but not limited to, a smartphone, a handheld computer, a tablet computer, a media player, a personal digital assistant (PDA), etc. It should be understood that this application is implemented The portable electronic device described in the manner is just an application example, and the components of the device may have more or fewer components than shown, or have different component configurations. The various components described can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

In the specific implementation of the present application, the mobile device 50 will be taken as a smartphone for example. A common example is an iPhone series mobile phone equipped with an iOS operating system by Apple; and another example is an Android developed by Google. Android smartphones with operating systems, such as those of Samsung, Huawei, Xiaomi, etc.

The mobile device 50 is loaded with at least one client 500, and the client includes an interface module 5001, an event generation module 5002, and an image processing module 5003.

The interface module 5001 is used to receive an operation instruction input by an external input device 401 via an external interface of the mobile device 50; in an embodiment, when the interface module 5001 receives an operation instruction output by the input device 401, the The operation instructions are fused to be converted into a set of event sequences. The interface module 5001 is, for example, an ADB (Android Debug Bridge) channel (Channel) interface program.

The event generation module 5002 is used to generate an operation event to operate the mobile device 50 according to the operation instruction and perform a corresponding operation to the mobile device 50; the event generation module 5002 is, for example, an application program, specifically In fact, an event implantation service program is run on the Android smartphone through the ADB (Android Debug Bridge) channel. For example, the event implantation service program is a sagevt program, which is used to implant events into the smartphone's operating system.

The ADB channel (ADB Channel) of the Android smartphone transmits the set of event sequences corresponding to the operation instruction to the smartphone, and the event generation module 5002 running the event implant service program in the smartphone generates the corresponding group The key sequence of the event sequence is converted into touch coordinates to the smartphone, and then an operation event corresponding to the operation instruction is generated. The smartphone executes the operation after receiving the operation time, and reflects this through its touch screen output. Image information of the operation time, that is, the image information includes image information that the mobile device 50 generates an operation event according to the operation instruction and performs output of the operation event.

The image processing module 5003 is used to capture the image information input by the mobile device 50 to the display screen, and output through the external connection port of the mobile device 50; the image information includes the mobile device 50 according to The operation instruction generates an operation event and executes output image information of the operation event. In this embodiment, the image processing module 5003 is, for example, a screen recording application. By running the screen recording application on the Android smartphone, the screen recording application is used to call the native interface of the Android system Recording the display content of the touch screen of the Android smart phone, and decoding the image information of the recorded screen, that is, compressing the image information of the recorded screen into a video stream, such as an H264 video stream. After receiving the video stream, the processor 4002 decodes the same screen image and transmits it to the display device for display.

In the embodiment provided by the present application, the image information further includes the standby image information of the mobile device 50 when the external input event is not executed, and the execution image information of a program that is running (such as playing a video, previewing a news page) , Path navigation and other programs), and image information that generates an operation event according to the operation instruction and performs output of the operation event.

The data processing system of the present application can transmit images from a mobile device such as a mobile phone to an external display device that does not have processing capabilities for display, and can also control the mobile device through peripheral input devices such as a mouse and keyboard, so that the mobile device The processed image is transmitted to the display device for display, thereby satisfying the user's need to use an external input device to operate the operating system of the mobile phone and to realize the use of a large screen for office or entertainment; The display device communicates and has the advantages of lower latency and higher reliability than wireless connections.

The present application also provides a method for using the mobile device and the display device to perform the same-screen display, which is used to connect the mobile device and the display device to perform the same-screen display through an electronic device including a connector and a processor, respectively. In an embodiment, the mobile device is, for example, a portable electronic device including, but not limited to, a smart phone, a handheld computer, a tablet computer, a media player, a personal digital assistant (PDA), etc. It should be understood that this application is described in the embodiments The portable electronic device is just an application example, and the components of the device may have more or less components than shown, or have different component configurations. The various components described can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

Please refer to FIG. 7, which shows a flow chart of a method for using a mobile device and a display device to perform the same screen display as shown in the figure. As shown in the figure, the method for using a mobile device and a display device to perform the same screen display includes the following steps:

First, step S10 is executed to receive an operation instruction output by an input device; in an embodiment, an operation instruction output by an input device is received through a connector.

In an embodiment, the connector is a HUB module configured with at least one first interface for connecting the input device and a second interface for connecting the mobile device, that is, the first The interface and the second interface are configured in a HUB module, that is, the HUB module integrates at least one first interface and a second interface. For example, the HUB module integrates a plurality of first interfaces for connecting different input devices An interface, and a plurality of second interfaces for connecting smartphones with different peripheral interfaces. For example, the HUB module integrates an interface unit for connecting a keyboard and an interface unit for connecting a mouse, a microUSB interface and a type-c interface for connecting an Android smartphone, and Lightning for connecting an iPhone series mobile phone Interfaces, etc. The HUB module also integrates a Bluetooth chip, such as the RTL8761AU-CG chip. The HUB module includes a USB-HUB chip, for example, a GL850G chip. In an actual embodiment, the input device is, for example, a keyboard, and the corresponding output operation instructions are, for example, a variety of operation instructions such as character input, character deletion, line feed, and space performed by the user by tapping the keyboard.

In an actual embodiment, the input device may also be a mouse, for example, and the corresponding output operation instructions are, for example, a user's mouse left and right mouse clicks, right mouse clicks, and dragging to move or select the mouse pointer. instruction. In some implementation states, the mouse may also be a touchpad (TouchPad or TrackPad), which controls the movement of the pointer by sensing the movement of the user's finger.

Next, step S11 is executed, an operation event for operating the mobile device is generated according to the operation instruction, and the corresponding operation is performed for the mobile device; in an embodiment, the first interface of the connector receives the input device When outputting the operation instruction, the operation instruction is fused to be converted into a set of event sequences. The second interface of the connector outputs the operation instruction from the first interface to a mobile device that is an Android smartphone; in an embodiment, when the mobile device that is a smartphone is connected to the second interface , The electronic device runs an event implantation service program on the Android smartphone through the ADB (Android Debug Bridge) channel, for example, the event implantation service program is a sagevt program for implanting events into the smartphone's operating system .

In an embodiment, the ADB channel (ADB) of the Android smartphone transmits the set of event sequences corresponding to the operation instruction to the smartphone, and the event implantation service program running in the smartphone generates a sequence corresponding to the set of event sequences The key map is converted into the touch coordinates of the smart phone, and an operation event corresponding to the operation instruction is generated. The smart phone performs the operation after receiving the operation time, and the touch screen output reflects the operation time. Image information, that is, the image information includes image information that the mobile device generates an operation event according to the operation instruction and performs output of the operation event.

Then step S12 is executed to capture the image information input by the mobile device to the display screen and output via the external connection port of the mobile device; the image information includes the mobile device generating an operation event according to the operation instruction And execute the output image information on the operation event; in an embodiment, by running the screen recording application on the Android smartphone, the screen recording application is used to call the native interface of the Android system to the The display content of the touch screen of the Android smartphone is recorded, and the image information of the recorded screen is decoded, that is, the image information of the recorded screen is compressed into a video stream, such as an H264 video stream. After receiving the video stream, the processor decodes the video into the same screen image and transmits it to the display device for display.

In the embodiment in which the mobile device is an Android smartphone, the processor connects to the Android smartphone through the second interface and establishes communication with the Android smartphone through the ADB channel, as described above. When a mobile device described as a smartphone is connected to the second interface, the electronic device runs a screen recording application on the Android smartphone through the ADB channel, and the screen recording application is used to call a native interface of the Android system Recording the display content of the touch screen of the Android smart phone, and decoding the image information of the recorded screen, that is, compressing the image information of the recorded screen into a video stream, such as an H264 video stream.

Finally, step S13 is first performed to decode the image information and output the same screen image for the display device to display. In an embodiment, after receiving the video stream, the processor decodes the image into the same screen and transmits it to the display device for display.

In the embodiment provided by the present application, the image information further includes standby image information of the mobile device when an external input event is not executed, execution image information of a program that is running (such as playing a video, previewing a news page, Programs such as route navigation), and image information that generates an operation event according to the operation instruction and performs output of the operation event.

The same-screen method of the present application can transmit images from a mobile device such as a mobile phone to an external display device that does not have processing capabilities for display, and can also control the mobile device through peripheral input devices such as a mouse and keyboard, so that the mobile device The processed image is transmitted to the display device for display, thereby satisfying the user's need to use an external input device to operate the operating system of the mobile phone and realize the use of a large screen for office or entertainment anywhere.

The present application also provides a method for performing on-screen display using a mobile device and a display device, which is used to connect the mobile device and the display device to perform the on-screen display through an electronic device including a connector and a processor, respectively. In an embodiment, the mobile device is, for example, a portable electronic device including, but not limited to, a smart phone, a handheld computer, a tablet computer, a media player, a personal digital assistant (PDA), etc. It should be understood that this application is described in the embodiments The portable electronic device is just an application example, and the components of the device may have more or less components than shown, or have different component configurations. The various components described can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

Please refer to FIG. 8, which shows a flow chart of the method for using the mobile device and the display device to perform the same screen display as shown in the figure. As shown in the figure, the method for using the mobile device and the display device to perform the same screen display includes the following steps:

First, step S20 is executed to receive an operation instruction output by an input device; in an embodiment, an electronic device including a connector and a processor receives an operation instruction output by an input device. The connector is a HUB module configured with at least one first interface for connecting the input device and a second interface for connecting the mobile device, that is, the first interface and the second The interface is configured in a HUB module, that is, the HUB module integrates at least one first interface and a second interface, for example, the HUB module integrates multiple first interfaces for connecting different input devices, and multiple The second interface of the smartphone for connecting different peripheral interfaces. For example, the HUB module integrates an interface unit for connecting a keyboard and an interface unit for connecting a mouse, a microUSB interface and a type-c interface for connecting an Android smartphone, and Lightning for connecting an iPhone series mobile phone Interfaces, etc. The HUB module also integrates a Bluetooth chip, such as the RTL8761AU-CG chip. The HUB module includes a USB-HUB chip, for example, a GL850G chip. The input device is, for example, one or more of a keyboard, a mouse, a touchpad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board. Of course, based on different application scenarios, the input device 11 may also be a microphone, electronic key, stylus, scanner, or the like.

In an actual embodiment, the input device is, for example, a keyboard, and the corresponding output operation instructions are, for example, a variety of operation instructions such as character input, character deletion, line feed, and space performed by the user by tapping the keyboard.

Next, step S21 is executed to output the received operation instruction to the mobile device; in an embodiment, the electronic device including the connector and the processor outputs the received operation instruction to the mobile device, specifically The second interface of the connector outputs the operation instruction from the first interface to a mobile device that is an Android smartphone; in an embodiment, the mobile device that is a smartphone is connected to the second During the interface, the electronic device runs an event implantation service program on the Android smartphone through the ADB (Android Debug Bridge) channel. For example, the event implantation service program is a sagevt program, which is used to implant the operating system of the smartphone Into the event.

Then step S22 is executed to receive the image information output by the mobile device; wherein the image information includes image information that the mobile device generates an operation event according to the operation instruction and performs output of the operation event; in the embodiment , The electronic device including the connector and the processor receives the image information output by the mobile device.

In an embodiment, by running the screen recording application on the Android smartphone, the screen recording application is used to call a native interface of the Android system to record the display content of the touch screen of the Android smartphone, and The image information of the screen recording is decoded, that is, the image information of the screen recording is compressed into a video stream, such as an H264 video stream. After receiving the video stream, the processor decodes the video into the same screen image and transmits it to the display device for display.

Finally, step S23 is executed to decode the received image information and output the same screen image for the display device to display. In an embodiment, after receiving the video stream, the processor decodes the image into the same screen and transmits it to the display device for display.

In addition, it should be noted that, through the description of the above embodiments, those skilled in the art can clearly understand that part or all of the present application can be implemented by using software in combination with a necessary general hardware platform. Based on this understanding, the present application also provides a storage medium for a computer device, where the storage medium stores at least one program, and when the program is called, the mobile medium described in the foregoing embodiment of FIG. 7 or FIG. 8 is executed. Device and display device for displaying on the same screen.

Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or part of the contribution to the existing technology, and the computer software product can include one or more machine executable program code stored on it. Multiple machine-readable media. When these program codes are executed by one or more machines, such as a computer, a computer network, or other electronic devices, the one or more machines may cause the one or more machines to perform operations according to the embodiments of the present application. For example, the steps in the method for performing the same screen display are performed. Machine-readable media may include, but is not limited to, floppy disks, optical disks, CD-ROM (compact disk-read only memory), magneto-optical disks, ROM (read only memory), RAM (random access memory), EPROM (erasable) Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic or optical card, flash memory, or other types of media / machine readable media suitable for storing machine executable program code. The storage medium may be located in a machine or a third-party server, such as a server that provides cloud storage.

In addition, it should be noted that, through the description of the above embodiments, those skilled in the art can clearly understand that part or all of the present application can be implemented by using software in combination with a necessary general hardware platform. Based on this understanding, the present application also provides a storage medium of a computer device, where the storage medium stores at least one program, and when the program is called, any of the foregoing methods for displaying on the same screen are performed.

This application can be described in the general context of computer-executable program code executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

The above-mentioned embodiments merely exemplify the principle and efficacy of the present application, and are not intended to limit the present application. Anyone familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in this application should still be covered by the claims of this application.

Claims

An electronic device for displaying on the same screen using mobile equipment and display equipment, characterized in that it includes:

At least one first interface unit for connecting an input device to receive operation instructions output by the input device;

A second interface unit for outputting the received operation instruction to the mobile device; and receiving image information output by the mobile device; wherein the image information includes the mobile device generating an operation according to the operation instruction Event and image information output for performing the operation event;

The processing unit is configured to connect the second interface unit and the display device to perform decoding processing on the received image information and output an image on the same screen for the display device to display.
The electronic device of claim 1, wherein the first interface unit and the second interface unit are configured in a HUB module.
The electronic device according to claim 1 or 2, wherein the first interface unit and the second interface unit include an FFC interface, an FPC interface, an AT interface, a COM interface, a MIDI interface, a PS / 2 interface, One of USB interface or Bluetooth interface.
The electronic device according to claim 1, wherein the input device includes one or more of a keyboard, a mouse, a touch pad, a trackball, a touch screen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board .
The electronic device according to claim 1, wherein the processing unit is connected to the display device through a third interface, the third interface includes a DP interface, eDP interface, MHL interface, DiiVA interface, HDBaseT interface, One of Thunderbolt interface, USB interface, LVDS interface, HDMI interface, VGA interface, or DVI interface.
The electronic device according to claim 1, further comprising:

A cache unit, connected to the processing unit, for caching image information received by the processing unit;

The storage unit is connected to the processing unit and used for storing the image processing program of the processing unit.
The electronic device according to claim 1, wherein the second interface unit is further configured to receive audio information output by the mobile device; the processing unit is further configured to process the received audio information and output audio playback signal.
A computer device for data processing using a mobile device, which is characterized by comprising:

Display device

Input device

The processing device, connected to the display device and the input device, includes;

The connector is respectively connected to the mobile device and the input device, and is used to receive the operation instruction output by the input device and send it to the mobile device, and receive and output the image information output by the mobile device; then output the image information; Including image information that the mobile device generates an operation event according to the operation instruction and performs output of the operation event;

The processor is connected to the connector and the display device, and is used for decoding the received image information and outputting the same screen image to display the display device.
The computer device according to claim 8, wherein the connector is a HUB module configured with at least one first interface for connecting the input device and a second interface for connecting the mobile device interface.
The computer device according to claim 9, wherein the first interface and the second interface include an FFC interface, an FPC interface, an AT interface, a COM interface, a MIDI interface, a PS / 2 interface, a USB interface, or Bluetooth One of the interfaces.
The computer device according to claim 8, wherein the input device comprises one or more of a keyboard, a mouse, a touch pad, a trackball, a touch screen, an electronic whiteboard, and a joystick.
The computer device according to claim 8, wherein the processor is connected to the display device through a third interface, the third interface includes a DP interface, eDP interface, MHL interface, DiiVA interface, HDBaseT interface, Thunderbolt One of interface, USB interface, LVDS interface, HDMI interface, VGA interface, or DVI interface.
The computer device according to claim 8, further comprising a buffer for buffering the image information and a memory for storing an image processing program.
The computer device according to claim 8, wherein the connector is further used to receive audio information output by the mobile device; the processor is further used to process the received audio information and output an audio playback signal; The computer device also includes an audio playback device for playing the received audio playback signal.
A client installed in a mobile device with a display screen, characterized in that it includes:

The interface module is used to receive an operation instruction input by an external input device via the external connection port of the mobile device;

An event generation module, configured to generate an operation event for operating the mobile device according to the operation instruction and perform a corresponding operation for the mobile device;

The image processing module is used to capture the image information input by the mobile device to the display screen and output via the external connection port of the mobile device; the image information includes the mobile device generating operations according to the operation instructions Event and execute output image information of the operation event.
A data processing system, characterized in that it includes:

Computer equipment, including input devices and display devices, also includes:

A connector, connected to the input device, for outputting image information and operation instructions output by the input device;

A processor, connected to the connector and the display device, for decoding the received image information and outputting the same screen image to display the display device;

A mobile device, connected to the connector, for receiving an operation instruction of the input device, and generating an operation event to operate the mobile device according to the operation instruction and performing a corresponding operation to the mobile device; and capturing Fetch image information input to the display screen by the mobile device and transmit it to the processor via the connector; the image information includes the mobile device generating an operation event according to the operation instruction and The operation event executes output image information.
The data processing system according to claim 16, wherein the connector is a HUB module configured with at least one first interface for connecting the input device and a first interface for connecting the mobile device Second interface.
The data processing system according to claim 16, wherein the first interface and the second interface include an FFC interface, an FPC interface, an AT interface, a COM interface, a MIDI interface, a PS / 2 interface, a USB interface, or One of the Bluetooth interfaces.
The data processing system according to claim 16, wherein the input device includes one or more of a keyboard, a mouse, a touchpad, a trackball, a touchscreen, an electronic whiteboard, a joystick, a gamepad, and a handwriting input board Species.
The data processing system according to claim 16, wherein the processor is connected to the display device through a third interface, the third interface includes a DP interface, eDP interface, MHL interface, DiiVA interface, HDBaseT interface, One of Thunderbolt interface, USB interface, LVDS interface, HDMI interface, VGA interface, or DVI interface.
The data processing system according to claim 16, wherein the computer device further includes a buffer for buffering the image information and a memory for storing an image processing program.
The data processing system according to claim 16, wherein the connector is further configured to receive audio information output by the mobile device; the processor is further configured to process the received audio information and output an audio playback signal The computer equipment also includes an audio playback device for playing the received audio playback signal.
A method for performing on-screen display using a mobile device and a display device, characterized in that the method includes the following steps:

Receive operation instructions output by an input device;

Generate an operation event for operating the mobile device according to the operation instruction and perform a corresponding operation for the mobile device;

Retrieving the image information input by the mobile device to the display screen and outputting through the external connection port of the mobile device; the image information includes the mobile device generating an operation event according to the operation instruction and Image information output by event execution;

After decoding the image information, an on-screen image is output to enable the display device to display.
A method for performing on-screen display using a mobile device and a display device, characterized in that the method includes the following steps:

Receive operation instructions output by an input device;

Output the received operation instruction to the mobile device;

Receiving image information output by the mobile device; wherein the image information includes image information that the mobile device generates an operation event according to the operation instruction and performs output of the operation event;

After decoding the received image information, an on-screen image is output to enable the display device to display.
A computer-readable storage medium, characterized in that a program is stored on the storage medium, and when the program is run, the method for displaying on a same screen using a mobile device and a display device according to claim 23 or 24 is realized.